Datasette had two tracing systems since the OpenTelemetry spans landed. The
hand-rolled one measures the wrong thing - issue 1730, open since 2022, is
about exactly that - and it cannot be rebuilt on top of the new spans without
core owning a TracerProvider, which is the one thing the OTel design refuses
to do. Rather than carry duplicate instrumentation on the db.execute() hot
path indefinitely, the old system goes.
Deleted: datasette/tracer.py, the trace_debug setting, the AsgiTracer
response-rewriting middleware and the ?_trace=1 query-string argument.
- datasette/database.py: the four `with trace(...)` wrappers PR 1 deliberately
nested the OTel spans inside are removed and the bodies dedented. That also
retires the `# noqa: SIM117` comments those wrappers required - a leftover
unnecessary noqa trips ruff's RUF100 - and `kwargs["count"] = count` in
execute_write_many, which fed the old tracer only. `git diff -w` on this file
shows nothing but the deleted lines.
- datasette/views/base.py: stream_csv() still read ?_trace=1 to wrap CSV output
in an HTML <textarea> debug page. That whole branch, including the
EscapeHtmlWriter selection and the conditional content-type, is gone. The
EscapeHtmlWriter class itself stays in datasette.utils - it is an importable
public name and removing it would widen the API break.
- .github/workflows/deploy-latest.yml no longer passes --setting trace_debug 1.
Worth stating precisely, because the ticket claimed otherwise: this would not
have broken the deploy. Setting.convert() in cli.py only rewrites a bare name
to settings.<name> for *known* settings, so `--setting trace_debug 1` would
have been silently accepted as a meaningless top-level config key. The flag is
removed because it is dead, not because it errors.
Tests. tests/test_tracer.py is deleted outright (6 items). Four other tests used
?_trace=1 as an assertion instrument rather than testing tracing:
- test_csv_trace tested the trace mechanism itself - deleted.
- test_table_csv_stream_does_not_calculate_facets,
test_table_csv_stream_does_not_calculate_counts and
test_nocount_nofacet_if_shape_is_object test real behaviour, and are rebuilt
against captured spans. All three had silently stopped being able to fail: the
facets test looked for "select content, count(*) as n", which facet suggestion
has not emitted since it moved to a `with limited as (...)` CTE, and none of
the three requested the count or facet work whose suppression they claim to
check. The rebuilt versions ask for it explicitly, match strings the current
SQL contains, and carry a guard assertion so an empty span list cannot
masquerade as a pass. Each was confirmed to fail with the covered code broken.
- test_trace_correctly_escaped is kept, renamed test_query_page_escapes_sql,
with ?_trace=1 dropped. It ran against ds_client, which has no trace_debug, so
it never exercised the tracer - what it actually covered is the query page
echoing user SQL into HTML, the surface of the two reflected-XSS advisories in
issue 1360, and nothing else in the suite covers it. Deleting it would have
quietly dropped that.
tests/test_utils.py's pairs_to_nested_config case used settings.trace_debug to
check that a later key overrides an earlier one; it now uses template_debug
rather than losing the case.
Docs: the datasette.tracer section of internals.rst, the trace_debug section of
settings.rst, the ?_trace=1 entries in json_api.rst and introspection.rst, and
the regenerated cli-reference.rst. changelog.rst gets a breaking-change entry
and keeps all its historical ?_trace=1 entries - two of them had to lose a
:ref: role pointing at a label this commit deletes, or Sphinx warns on every
build.
2368 passed, 39 skipped, 6 xfailed, 15 xpassed, 140 subtests, against 2375 /
141 before. Net -7 tests, fully accounted for: -6 test_tracer.py, -1
test_csv_trace, -1 test_trace_correctly_escaped, +1 test_query_page_escapes_sql.
The lost subtest is the per-setting case trace_debug generated in
test_settings_are_documented.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Several internal helpers quoted table names using SQLite [bracket]
identifiers built with an f-string, e.g. PRAGMA foreign_key_list([{table}]).
Bracket quoting cannot escape a "]" character, so any table whose name
contains "]" (for example "[foo]" or "foo]") produced
"sqlite3.OperationalError: unrecognized token" - crashing schema
introspection at startup and 500-ing the table page.
Switch these call sites to the existing escape_sqlite() helper, which uses
"double quote" quoting with correct "" escaping (the same approach already
used elsewhere in the codebase and in the test suite):
- utils/internal_db.py: PRAGMA foreign_key_list / index_list
- utils/__init__.py: get_outbound_foreign_keys
- database.py: table_counts count query
- facets.py: default "select * from" SQL
Added a regression test covering table names with "]" characters.
Co-authored-by: Claude <noreply@anthropic.com>
Reverts the object envelope introduced in 1.0a36 for this endpoint -
it once again returns a top-level JSON array of plugin objects.
Closes#2842
Claude-Session: https://claude.ai/code/session_012TYc1NTBK4zEjabB3u2zqu
Co-authored-by: Claude <noreply@anthropic.com>
next_url became a default table JSON key alongside next, making the
extra a no-op. Requesting ?_extra=next_url now returns the standard
unknown-extra 400 error.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
The SQL time limit error embedded an HTML fragment (paragraph, textarea
and script tags) as the error string in JSON responses. DatasetteError
now accepts a plain_message which the exception handler prefers for
JSON error bodies; the HTML error page keeps the rich message with the
SQL textarea.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
JSON endpoints that are not part of the documented API now include
"unstable": "This API is not part of Datasette's stable interface and
may change at any time" in their responses, making the stability tier
machine-readable. Applied to the homepage (/.json and /-/.json),
/db/-/queries/analyze, POST /db/-/queries/store,
/db/<query>/-/definition, /db/-/query/parameters,
/db/-/execute-write/analyze and the POST /-/permissions playground
response. The message lives in datasette.utils.UNSTABLE_API_MESSAGE.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
/.json previously returned databases as an object keyed by database
name, unlike /-/databases.json and every other collection in the API.
It now returns a list of database objects. The HTML template already
consumed the list. This endpoint remains undocumented.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
/-/threads exposes runtime internals - thread idents and asyncio task
reprs including file paths - but only required view-instance. It now
requires permissions-debug, like /-/actions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
/-/actions.json now returns {"ok": true, "actions": [...]} instead of a
bare JSON array, so the response can grow additional keys without a
breaking change. The debug_actions.html template reads data.actions,
and the endpoint is now documented in docs/introspection.rst.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
/-/databases.json now returns {"ok": true, "databases": [...]} instead
of a bare JSON array, so the response can grow additional keys without
a breaking change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
/-/plugins.json now returns {"ok": true, "plugins": [...]} instead of a
bare JSON array, so the response can grow additional keys without a
breaking change. The `datasette plugins` CLI command still outputs a
plain array.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
JsonDataView now injects "ok": true into dict responses, covering
/-/versions, /-/settings, /-/config, /-/threads and /-/actor. The
homepage JSON, /-/jump, the three /-/schema endpoints, /-/allowed,
/-/rules, /-/check, POST /-/permissions and the table /-/autocomplete
endpoint set it explicitly.
The remaining top-level array endpoints (/-/plugins, /-/databases,
/-/actions) will be converted to objects in separate commits.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
All JSON error responses now use a single format built by the new
datasette.utils.error_body() helper:
{"ok": false, "error": "...", "errors": ["..."], "status": 400}
- error is all messages joined with '; ', errors is the full list,
status always matches the HTTP status code
- The exception handler no longer emits the legacy title key in JSON
(it is still available to the HTML error template)
- The permission debug endpoints (/-/allowed, /-/rules, /-/check,
POST /-/permissions) no longer return bare {"error": ...} objects
- JSON renderer SQL errors keep their rows/truncated context keys but
now include the canonical keys as well
- _shape=object misuse (queries or tables without primary keys) now
returns HTTP 400 instead of 200 with an error body
- Method-not-allowed 405 responses use the canonical shape
Adds tests/test_error_shape.py covering all four previous shape
producers, updates affected tests, and documents the format in a new
'Error responses' section of docs/json_api.rst.
Implements section 1 of stable-api-recommendations.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
The pre-1.0 ?_extras= (plural) parameter was kept for backwards
compatibility with the old row JSON API. ?_extra= is the documented
mechanism now that row pages share the extras registry.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`datasette.client.get(path, actor={"id": "root"}` now makes the internal request with that actor as `request.actor` - same for the other HTTP verb methods on `datasette.client`.
Upgraded relevant tests to use the new `actor=` mechanism.
The render_cell pks parameter commit added rows to compound_primary_key
(2->3 rows) and no_primary_key (201->202 rows) tables but did not
update existing tests that had hardcoded row count expectations.
https://claude.ai/code/session_01XfPSZfK57bzRRiEa7Kz5n1
* Fix flaky test_database_page test with deterministic ordering
- Add ORDER BY to table_names() query in database.py
- Sort foreign keys deterministically in get_all_foreign_keys()
- Refactor test_database_page to use property-based assertions instead of
500+ lines of hardcoded expected data
- Run blacken-docs on plugin_hooks.rst
* Update test_row_foreign_key_tables for new deterministic FK ordering
The foreign keys are now sorted by (other_table, column, other_column),
so complex_foreign_keys comes before foreign_key_references alphabetically.
* Update test_table_names for new alphabetical ordering
The table_names() method now returns tables sorted alphabetically.
* Fix for test that fails prior to SQLite 3.37
---------
Co-authored-by: Claude <noreply@anthropic.com>
This adds a new endpoint at /-/actions that lists all registered actions
in the permission system. The endpoint supports both JSON and HTML output.
Changes:
- Added _actions() method to Datasette class to return action list
- Added route for /-/actions with JsonDataView
- Created actions.html template for nice HTML display
- Added template parameter to JsonDataView for custom templates
- Moved respond_json_or_html from BaseView to JsonDataView
- Added test for the new endpoint
The endpoint requires view-instance permission and provides details about
each action including name, abbreviation, description, resource class,
and parent/child requirements.
Closes#2547
Co-Authored-By: Claude <noreply@anthropic.com>
This fixes issues introduced by the ruff commit e57f391a which converted
Optional[x] to x | None:
- Fixed datasette/app.py line 1024: Dict[id | str, Dict] -> Dict[int | str, Dict]
(was using id built-in function instead of int type)
- Fixed datasette/app.py line 1074: Optional["Resource"] -> "Resource" | None
- Added 'from __future__ import annotations' for Python 3.10 compatibility
- Added TYPE_CHECKING blocks to avoid circular imports
- Removed dead code (unused variable assignments) from cli.py and views
- Removed unused imports flagged by ruff across multiple files
- Fixed test fixtures: moved app_client fixture imports to conftest.py
(fixed 71 test errors caused by fixtures not being registered)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
This change integrates canned queries with Datasette's new SQL-based
permissions system by making the following changes:
1. **Default canned_queries plugin hook**: Added a new hookimpl in
default_permissions.py that returns canned queries from datasette
configuration. This extracts config-reading logic into a plugin hook,
allowing QueryResource to discover all queries.
2. **Async resources_sql()**: Converted Resource.resources_sql() from a
synchronous class method returning a string to an async method that
receives the datasette instance. This allows QueryResource to call
plugin hooks and query the database.
3. **QueryResource implementation**: Implemented QueryResource.resources_sql()
to gather all canned queries by:
- Querying catalog_databases for all databases
- Calling canned_queries hooks for each database with actor=None
- Building a UNION ALL SQL query of all (database, query_name) pairs
- Properly escaping single quotes in resource names
4. **Simplified get_canned_queries()**: Removed config-reading logic since
it's now handled by the default plugin hook.
5. **Added view-query to default allow**: Added "view-query" to the
default_allow_actions set so canned queries are accessible by default.
6. **Removed xfail markers**: Removed test xfail markers from:
- tests/test_canned_queries.py (entire module)
- tests/test_html.py (2 tests)
- tests/test_permissions.py (1 test)
- tests/test_plugins.py (1 test)
All canned query tests now pass with the new permission system.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Canned queries are not accessible because view-query permission
has not yet been migrated to the SQL-based permission system.
Marks the following tests with xfail:
- test_config_cache_size (test_api.py)
- test_edit_sql_link_not_shown_if_user_lacks_permission (test_html.py)
- test_database_color - removes canned query path (test_html.py)
- test_hook_register_output_renderer_* (test_plugins.py - 3 tests)
- test_hook_query_actions canned query parameter (test_plugins.py)
- test_custom_query_with_unicode_characters (test_table_api.py)
- test_permissions_checked neighborhood_search (test_permissions.py)
- test_padlocks_on_database_page (test_permissions.py)
All reference issue #2510 for tracking view-query migration.
* `asyncio_default_fixture_loop_scope = function`
* Fix a bunch of BeautifulSoup deprecation warnings
* Fix for PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
* xfail for sql_time_limit tests (these can be flaky in CI)
Refs #2461
* Introduce new default /$DB/-/query endpoint
* Fix a lot of tests
* Update pyodide test to use query endpoint
* Link to /fixtures/-/query in a few places
* Documentation for QueryView
---------
Co-authored-by: Simon Willison <swillison@gmail.com>
