Datasette had two tracing systems since the OpenTelemetry spans landed. The
hand-rolled one measures the wrong thing - issue 1730, open since 2022, is
about exactly that - and it cannot be rebuilt on top of the new spans without
core owning a TracerProvider, which is the one thing the OTel design refuses
to do. Rather than carry duplicate instrumentation on the db.execute() hot
path indefinitely, the old system goes.
Deleted: datasette/tracer.py, the trace_debug setting, the AsgiTracer
response-rewriting middleware and the ?_trace=1 query-string argument.
- datasette/database.py: the four `with trace(...)` wrappers PR 1 deliberately
nested the OTel spans inside are removed and the bodies dedented. That also
retires the `# noqa: SIM117` comments those wrappers required - a leftover
unnecessary noqa trips ruff's RUF100 - and `kwargs["count"] = count` in
execute_write_many, which fed the old tracer only. `git diff -w` on this file
shows nothing but the deleted lines.
- datasette/views/base.py: stream_csv() still read ?_trace=1 to wrap CSV output
in an HTML <textarea> debug page. That whole branch, including the
EscapeHtmlWriter selection and the conditional content-type, is gone. The
EscapeHtmlWriter class itself stays in datasette.utils - it is an importable
public name and removing it would widen the API break.
- .github/workflows/deploy-latest.yml no longer passes --setting trace_debug 1.
Worth stating precisely, because the ticket claimed otherwise: this would not
have broken the deploy. Setting.convert() in cli.py only rewrites a bare name
to settings.<name> for *known* settings, so `--setting trace_debug 1` would
have been silently accepted as a meaningless top-level config key. The flag is
removed because it is dead, not because it errors.
Tests. tests/test_tracer.py is deleted outright (6 items). Four other tests used
?_trace=1 as an assertion instrument rather than testing tracing:
- test_csv_trace tested the trace mechanism itself - deleted.
- test_table_csv_stream_does_not_calculate_facets,
test_table_csv_stream_does_not_calculate_counts and
test_nocount_nofacet_if_shape_is_object test real behaviour, and are rebuilt
against captured spans. All three had silently stopped being able to fail: the
facets test looked for "select content, count(*) as n", which facet suggestion
has not emitted since it moved to a `with limited as (...)` CTE, and none of
the three requested the count or facet work whose suppression they claim to
check. The rebuilt versions ask for it explicitly, match strings the current
SQL contains, and carry a guard assertion so an empty span list cannot
masquerade as a pass. Each was confirmed to fail with the covered code broken.
- test_trace_correctly_escaped is kept, renamed test_query_page_escapes_sql,
with ?_trace=1 dropped. It ran against ds_client, which has no trace_debug, so
it never exercised the tracer - what it actually covered is the query page
echoing user SQL into HTML, the surface of the two reflected-XSS advisories in
issue 1360, and nothing else in the suite covers it. Deleting it would have
quietly dropped that.
tests/test_utils.py's pairs_to_nested_config case used settings.trace_debug to
check that a later key overrides an earlier one; it now uses template_debug
rather than losing the case.
Docs: the datasette.tracer section of internals.rst, the trace_debug section of
settings.rst, the ?_trace=1 entries in json_api.rst and introspection.rst, and
the regenerated cli-reference.rst. changelog.rst gets a breaking-change entry
and keeps all its historical ?_trace=1 entries - two of them had to lose a
:ref: role pointing at a label this commit deletes, or Sphinx warns on every
build.
2368 passed, 39 skipped, 6 xfailed, 15 xpassed, 140 subtests, against 2375 /
141 before. Net -7 tests, fully accounted for: -6 test_tracer.py, -1
test_csv_trace, -1 test_trace_correctly_escaped, +1 test_query_page_escapes_sql.
The lost subtest is the per-setting case trace_debug generated in
test_settings_are_documented.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
next_url became a default table JSON key alongside next, making the
extra a no-op. Requesting ?_extra=next_url now returns the standard
unknown-extra 400 error.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
Table JSON responses previously only included the next pagination token
by default - the ready-to-follow next_url required ?_extra=next_url.
Both keys are now always present (null on the final page), which the
pagination documentation already claimed. The next_url extra remains
valid for backwards compatibility.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
Unknown ?_extra= names (including internal HTML-only extras such as
display_rows) were silently ignored, so a typo returned the default
payload with no signal. Table, row and query data formats now return
400 "Unknown _extra: <names>". HTML pages continue to ignore unknown
names.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
The count extra is computed with a limit subquery, so a count equal to
count_limit + 1 (default 10001) actually means "at least this many" -
but only the HTML view knew that. A public count_truncated extra now
reports the flag and is implicitly included whenever count is
requested, using the same logic the HTML view already used.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
GET /db/-/query.json with no (or blank) ?sql= previously returned 200
with empty rows, masking caller bugs, while the .csv format returned
400 "?sql= is required" for the same request. All data formats now
return the 400; the HTML SQL editor page is unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
All JSON error responses now use a single format built by the new
datasette.utils.error_body() helper:
{"ok": false, "error": "...", "errors": ["..."], "status": 400}
- error is all messages joined with '; ', errors is the full list,
status always matches the HTTP status code
- The exception handler no longer emits the legacy title key in JSON
(it is still available to the HTML error template)
- The permission debug endpoints (/-/allowed, /-/rules, /-/check,
POST /-/permissions) no longer return bare {"error": ...} objects
- JSON renderer SQL errors keep their rows/truncated context keys but
now include the canonical keys as well
- _shape=object misuse (queries or tables without primary keys) now
returns HTTP 400 instead of 200 with an error body
- Method-not-allowed 405 responses use the canonical shape
Adds tests/test_error_shape.py covering all four previous shape
producers, updates affected tests, and documents the format in a new
'Error responses' section of docs/json_api.rst.
Implements section 1 of stable-api-recommendations.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
These three extras return values that exist for the HTML templates -
a Filters instance, an async function and markupsafe/sqlite3.Row data
- so requesting them on a .json page returned a 500 serialization
error, while the generated documentation and ?_extra=extras both
advertised them as API surface. They are now public=False: ignored
like any unknown name on JSON requests, omitted from the docs and the
extras list, and still resolved for the HTML view via the new
include_internal flag on ExtraRegistry.resolve().
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
QueryExtra re-derived named parameters from the SQL with a regex,
which missed parameters declared in a stored query's params list,
reported magic _-prefixed parameters with raw querystring values that
were never bound, and echoed the entire querystring when no SQL was
present. QueryView now passes its named_parameter_values dict - the
parameters it actually bound - through QueryExtraContext.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
QueryView hardcoded private=False unless the request was for a stored
query, so /db/-/query.json?_extra=private reported false even when
execute-sql was restricted to the authenticated actor. Use
check_visibility() like the table and row views do.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The render_cell pks parameter commit added rows to compound_primary_key
(2->3 rows) and no_primary_key (201->202 rows) tables but did not
update existing tests that had hardcoded row count expectations.
https://claude.ai/code/session_01XfPSZfK57bzRRiEa7Kz5n1
* Neater design for PermissionSQL class, refs #2556
- source is now automatically set to the source plugin
- params is optional
* PermissionSQL.allow() and PermissionSQL.deny() shortcuts
Closes#2556
* Filter out temp database from attached_databases()
Refs https://github.com/simonw/datasette/issues/2557#issuecomment-3470510837
This change integrates canned queries with Datasette's new SQL-based
permissions system by making the following changes:
1. **Default canned_queries plugin hook**: Added a new hookimpl in
default_permissions.py that returns canned queries from datasette
configuration. This extracts config-reading logic into a plugin hook,
allowing QueryResource to discover all queries.
2. **Async resources_sql()**: Converted Resource.resources_sql() from a
synchronous class method returning a string to an async method that
receives the datasette instance. This allows QueryResource to call
plugin hooks and query the database.
3. **QueryResource implementation**: Implemented QueryResource.resources_sql()
to gather all canned queries by:
- Querying catalog_databases for all databases
- Calling canned_queries hooks for each database with actor=None
- Building a UNION ALL SQL query of all (database, query_name) pairs
- Properly escaping single quotes in resource names
4. **Simplified get_canned_queries()**: Removed config-reading logic since
it's now handled by the default plugin hook.
5. **Added view-query to default allow**: Added "view-query" to the
default_allow_actions set so canned queries are accessible by default.
6. **Removed xfail markers**: Removed test xfail markers from:
- tests/test_canned_queries.py (entire module)
- tests/test_html.py (2 tests)
- tests/test_permissions.py (1 test)
- tests/test_plugins.py (1 test)
All canned query tests now pass with the new permission system.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Canned queries are not accessible because view-query permission
has not yet been migrated to the SQL-based permission system.
Marks the following tests with xfail:
- test_config_cache_size (test_api.py)
- test_edit_sql_link_not_shown_if_user_lacks_permission (test_html.py)
- test_database_color - removes canned query path (test_html.py)
- test_hook_register_output_renderer_* (test_plugins.py - 3 tests)
- test_hook_query_actions canned query parameter (test_plugins.py)
- test_custom_query_with_unicode_characters (test_table_api.py)
- test_permissions_checked neighborhood_search (test_permissions.py)
- test_padlocks_on_database_page (test_permissions.py)
All reference issue #2510 for tracking view-query migration.
* Introduce new default /$DB/-/query endpoint
* Fix a lot of tests
* Update pyodide test to use query endpoint
* Link to /fixtures/-/query in a few places
* Documentation for QueryView
---------
Co-authored-by: Simon Willison <swillison@gmail.com>
* Refs #2111, closes#2110
* New Context dataclass/subclass mechanism, refs #2127
* Define QueryContext and extract get_tables() method, refs #2127
* Fix OPTIONS bug by porting DaatbaseView to be a View subclass
* Expose async_view_for_class.view_class for test_routes test
* Error/truncated aruments for renderers, closes#2130
* Dash encoding functions, tests and docs, refs #1439
* dash encoding is now like percent encoding but with dashes
* Use dash-encoding for row PKs and ?_next=, refs #1439
* Use dash encoding for table names, refs #1439
* Use dash encoding for database names, too, refs #1439
See also https://simonwillison.net/2022/Mar/5/dash-encoding/
