Datasette had two tracing systems since the OpenTelemetry spans landed. The
hand-rolled one measures the wrong thing - issue 1730, open since 2022, is
about exactly that - and it cannot be rebuilt on top of the new spans without
core owning a TracerProvider, which is the one thing the OTel design refuses
to do. Rather than carry duplicate instrumentation on the db.execute() hot
path indefinitely, the old system goes.
Deleted: datasette/tracer.py, the trace_debug setting, the AsgiTracer
response-rewriting middleware and the ?_trace=1 query-string argument.
- datasette/database.py: the four `with trace(...)` wrappers PR 1 deliberately
nested the OTel spans inside are removed and the bodies dedented. That also
retires the `# noqa: SIM117` comments those wrappers required - a leftover
unnecessary noqa trips ruff's RUF100 - and `kwargs["count"] = count` in
execute_write_many, which fed the old tracer only. `git diff -w` on this file
shows nothing but the deleted lines.
- datasette/views/base.py: stream_csv() still read ?_trace=1 to wrap CSV output
in an HTML <textarea> debug page. That whole branch, including the
EscapeHtmlWriter selection and the conditional content-type, is gone. The
EscapeHtmlWriter class itself stays in datasette.utils - it is an importable
public name and removing it would widen the API break.
- .github/workflows/deploy-latest.yml no longer passes --setting trace_debug 1.
Worth stating precisely, because the ticket claimed otherwise: this would not
have broken the deploy. Setting.convert() in cli.py only rewrites a bare name
to settings.<name> for *known* settings, so `--setting trace_debug 1` would
have been silently accepted as a meaningless top-level config key. The flag is
removed because it is dead, not because it errors.
Tests. tests/test_tracer.py is deleted outright (6 items). Four other tests used
?_trace=1 as an assertion instrument rather than testing tracing:
- test_csv_trace tested the trace mechanism itself - deleted.
- test_table_csv_stream_does_not_calculate_facets,
test_table_csv_stream_does_not_calculate_counts and
test_nocount_nofacet_if_shape_is_object test real behaviour, and are rebuilt
against captured spans. All three had silently stopped being able to fail: the
facets test looked for "select content, count(*) as n", which facet suggestion
has not emitted since it moved to a `with limited as (...)` CTE, and none of
the three requested the count or facet work whose suppression they claim to
check. The rebuilt versions ask for it explicitly, match strings the current
SQL contains, and carry a guard assertion so an empty span list cannot
masquerade as a pass. Each was confirmed to fail with the covered code broken.
- test_trace_correctly_escaped is kept, renamed test_query_page_escapes_sql,
with ?_trace=1 dropped. It ran against ds_client, which has no trace_debug, so
it never exercised the tracer - what it actually covered is the query page
echoing user SQL into HTML, the surface of the two reflected-XSS advisories in
issue 1360, and nothing else in the suite covers it. Deleting it would have
quietly dropped that.
tests/test_utils.py's pairs_to_nested_config case used settings.trace_debug to
check that a later key overrides an earlier one; it now uses template_debug
rather than losing the case.
Docs: the datasette.tracer section of internals.rst, the trace_debug section of
settings.rst, the ?_trace=1 entries in json_api.rst and introspection.rst, and
the regenerated cli-reference.rst. changelog.rst gets a breaking-change entry
and keeps all its historical ?_trace=1 entries - two of them had to lose a
:ref: role pointing at a label this commit deletes, or Sphinx warns on every
build.
2368 passed, 39 skipped, 6 xfailed, 15 xpassed, 140 subtests, against 2375 /
141 before. Net -7 tests, fully accounted for: -6 test_tracer.py, -1
test_csv_trace, -1 test_trace_correctly_escaped, +1 test_query_page_escapes_sql.
The lost subtest is the per-setting case trace_debug generated in
test_settings_are_documented.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Introduce new default /$DB/-/query endpoint
* Fix a lot of tests
* Update pyodide test to use query endpoint
* Link to /fixtures/-/query in a few places
* Documentation for QueryView
---------
Co-authored-by: Simon Willison <swillison@gmail.com>
* _blob_hash= checking plus refactored to use new BadRequest class, refs #1050
* Replace BlobView with new .blob renderer, closes#1050
* .blob downloads on arbitrary queries, closes#1051
I've run the black code formatting tool against everything:
black tests datasette setup.py
I also added a new unit test, in tests/test_black.py, which will fail if the code does not
conform to black's exacting standards.
This unit test only runs on Python 3.6 or higher, because black itself doesn't run on 3.5.
* table.csv?_stream=1 to download all rows - refs #266
This option causes Datasette to serve ALL rows in the table, by internally
following the _next= pagination links and serving everything out as a stream.
Also added new config option, allow_csv_stream, which can be used to disable
this feature.
* New config option max_csv_mb limiting size of CSV export
These new querystring arguments can be used to request expanded foreign keys
in both JSON and CSV formats.
?_labels=on turns on expansions for ALL foreign key columns
?_label=COLUMN1&_label=COLUMN2 can be used to pick specific columns to expand
e.g. `Street_Tree_List.json?_label=qSpecies&_label=qLegalStatus`
{
"rowid": 233,
"TreeID": 121240,
"qLegalStatus": {
"value" 2,
"label": "Private"
}
"qSpecies": {
"value": 16,
"label": "Sycamore"
}
"qAddress": "91 Commonwealth Ave",
...
}
The labels option also works for the HTML and CSV views.
HTML defaults to `?_labels=on`, so if you pass `?_labels=off` you can disable
foreign key expansion entirely - or you can use `?_label=COLUMN` to request
just specific columns.
If you expand labels on CSV you get additional columns in the output:
`/Street_Tree_List.csv?_label=qLegalStatus`
rowid,TreeID,qLegalStatus,qLegalStatus_label...
1,141565,1,Permitted Site...
2,232565,2,Undocumented...
I also refactored the existing foreign key expansion code.
Closes#233. Refs #266.
The test used to expect CSV to come back like this:
hello
world
""
With the final blank value encoded in quotes.
Judging by Travis failures, this behaviour changed between Python 3.6.3 and
3.6.5:
https://travis-ci.org/simonw/datasette/jobs/392586661
Tables and custom SQL query results can now be exported as CSV.
The easiest way to do this is to use the .csv extension, e.g.
/test_tables/facet_cities.csv
By default this is served as Content-Type: text/plain so you can see it in
your browser. If you want to download the file (using text/csv and with an
appropriate Content-Disposition: attachment header) you can do so like this:
/test_tables/facet_cities.csv?_dl=1
We link to the CSV and downloadable CSV URLs from the table and query pages.
The links use ?_size=max and so by default will return 1,000 rows.
Also fixes#303 - table names ending in .json or .csv are now detected and
URLs are generated that look like this instead:
/test_tables/table%2Fwith%2Fslashes.csv?_format=csv
The ?_format= option is available for everything else too, but we link to the
.csv / .json versions in most cases because they are aesthetically pleasing.
