Commit graph

1,443 commits

Author SHA1 Message Date
Alex Garcia
8d32eac895 Name every span and attribute once, in a registry the docs are built from
The span and attribute names were string literals spread across four call
sites in database.py and one in app.py, with a hand-written reference page
that would have been true only on the day it was written. That drift is not
hypothetical: an earlier iteration of this work carried a README asserting
parameter values were never recorded for two branches after that had stopped
being true.

datasette/telemetry_registry.py now holds each name once, with its
documentation. Attribute and SpanName subclass str, so a registry entry *is*
the string OpenTelemetry wants - no wrapper API over the OTel calls, no
parallel structure to keep in step, and a typo becomes an ImportError rather
than a silently misnamed attribute. docs/internals.rst renders the span
reference from it via cog, and `cog --check docs/*.rst` already runs in CI,
so the reference cannot drift from the definitions.

Nothing changes on the wire: the emitted span names and attribute keys are
byte-identical before and after, verified by diffing a dump of both.

tests/test_telemetry_registry.py exercises a real workload and compares it
against the registry in both directions - emitted-but-unregistered catches
instrumentation added without documentation, registered-but-never-emitted
catches documentation that has outlived its code. Because the call sites now
take their names from the registry, neither direction can catch a rename:
move DB_NAMESPACE to "db.namespace2" and code and registry still agree while
every dashboard breaks. So the literal names are also written out in the test
and asserted against the registry and against the wire separately. That pair
is the only comparison in the file not derived from the registry itself.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-01 16:24:35 -07:00
Alex Garcia
e24a2c122f Link block=False write spans to their enqueuer instead of parenting them
A block=False write returns without awaiting the reply future, so the
enclosing db.query span finishes - and exports - before db.write.queue_wait
and db.write.execute even exist. They were still parented to it, which
produced a child bar ending ~50ms after its already-closed parent: legal
OpenTelemetry, but it renders as nonsense in a trace UI.

Parenting asserts containment; a link asserts causation without containment.
The enqueueing request causes the write without containing it, which is
exactly what a span link is for. So for block=False both write spans are now
roots - started with an explicit empty Context, so the write thread's ambient
context cannot supply a parent either - each carrying one link back to the
enqueueing span. block=True is untouched, since there the caller really does
await the reply and containment is accurate.

The link carries no attributes. There is only one kind of link here, so
naming the relationship would be a constant conveying nothing the link's
existence does not already say.

Accepted trade-off: a linked span will not appear inside the request's
waterfall in most trace UIs. It shows up as its own trace with a "linked
from" reference rather than a bar under the request. For a fire-and-forget
write whose latency the request never pays, that is the right trade -
correctness over at-a-glance nesting for a case the request-latency view was
never accurate for anyway.

This does add root traces, which looks like it cuts against the startup span
work that spent its whole diff removing them. The difference is reachability:
those roots were orphans, whereas these are reachable from the request that
caused them via the link.

Nothing in core issues block=False writes today - it is a plugin-facing path
- so this changes no trace Datasette produces on its own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-01 16:24:35 -07:00
Alex Garcia
77b025be28 Make db.query spans match OpenTelemetry semantic conventions
Three corrections to the emitted data, bundled because changing what is on
the wire after operators have built dashboards on it is a breaking change -
so they belong in the first release that ships spans at all, not a later one.

db.query is now SpanKind.CLIENT. Trace UIs key their database rendering off
the span kind rather than off db.system, so the spans rendered as ordinary
internal work despite carrying db.system and db.query.text. The three child
spans stay INTERNAL on purpose: db.query.execute, db.write.execute and
db.write.queue_wait are Datasette's decomposition of one logical query, not
three database calls, and queue_wait touches no database at all - marking
them CLIENT would make one query look like several to anything counting
spans by kind.

The instrumentation scope now carries the Datasette version and a schema
URL, so a backend can tell which Datasette produced a span. The URL is
1.29.0 rather than the latest semconv release because that is the highest
version at which every name emitted here is the current spelling: db.system
was renamed to db.system.name in 1.30.0 and this code still emits the older
form. Claiming a later schema would be false, and would stop a consumer
translating that name forward, since the claim asserts the rename already
happened.

db.operation.name is the statement's leading keyword matched against a fixed
allowlist, not a parse. On a public instance the SQL is attacker-controlled
and this attribute is a candidate metric dimension in a later phase, so
echoing back an arbitrary first token would let a visitor's typo mint a
permanent series. Anything unrecognised gets no attribute rather than a
wrong one. execute_write_script() does not set it at all, since semantic
conventions say not to extract an operation name from query text that can
hold several statements.

db.collection.name comes only from a new table= argument on
Database.execute(), and is never derived from the SQL: deriving it would be
a parse, and on an instance where anyone can create a table the value set
has no ceiling. It is passed from every query in the table and row views
that targets exactly one user table. Internal-catalog reads and the row
view's cross-table foreign key counts are deliberately left without it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-01 16:24:35 -07:00
Alex Garcia
4ebec0b1ea Give startup's ~20 orphan spans somewhere to belong
invoke_startup() runs before any request exists, so nothing it does has an
ambient span to nest under. Measured on a fresh instance: 19 distinct traces,
19 of them single- or few-span roots - the register_* hook dispatches, the
internal catalog's db.query reads and its db.write.* catalog writes. In a
trace UI that is nineteen pieces of noise sitting next to every real trace,
which for an operator opening Jaeger for the first time is the difference
between "this works" and "this is unusable".

Bracketing the whole method body in one datasette.startup span takes that to
1. This is not a propagation fix - ticket 04's context propagation was already
correct, it simply had nothing to propagate. The bulk of the app.py diff is
re-indentation; `git diff -w` shows the real change (plus one line-length
rewrap black applied to the StartupError raise).

register_output_renderer and asgi_wrapper stay orphans deliberately: both are
dispatched from Datasette.__init__ / .app(), before invoke_startup() exists to
be called, and wrapping them would mean holding a span open across object
construction in library code that may never serve a request.

Suppressing instrumentation during warm-up was rejected as an alternative: a
slow prepare_connection runs on every connection, not just at startup, and is
exactly what tracing should reveal.

Also corrects the stale write-thread warm-up comment in database.py. It is
still a root, but for a reason worth stating precisely: a raw
threading.Thread does not inherit the starting thread's context, so the
datasette.startup span current on the event loop does not reach it. Read
connections do warm up under copy_context() and nest correctly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-01 16:24:35 -07:00
Alex Garcia
582d79a148 Propagate otel context across the thread boundaries
Spans created on a worker thread resolve their parent from that thread's
ambient context, so without this every span produced below Database came
back as an unparented root, disconnected from the request that caused it.
Carrying the caller's context across each boundary is also what makes the
thread-pool wait visible: db.query covers the full round trip, the new
db.query.execute covers only the work inside the worker, and the gap
between them is the queueing the old tracer folds invisibly into one
number.

- execute_fn()'s executor.submit() and execute_isolated_fn()'s
  run_in_executor() (immutable databases) now run the callable inside a
  contextvars.copy_context(). A *fresh* copy per submit is required:
  concurrently entering one shared Context raises "RuntimeError: cannot
  enter context ... already entered".
- WriteTask carries the otel Context captured on the event loop at enqueue
  time plus an enqueued_at_ns timestamp (both need __slots__ entries, or
  they fail with AttributeError at runtime). _execute_writes attaches that
  context right after the _SHUTDOWN check and detaches it in a finally
  spanning all three execution branches - the write thread is persistent
  and shared, so a leaked token would grow its context stack for every
  write processed afterwards, and a wrong-token detach only logs rather
  than raising.
- New spans: db.query.execute (read worker thread), db.write.queue_wait
  (explicit start/end timestamps, so its duration is the real enqueue ->
  dequeue wait rather than the microseconds spent building the span) and
  db.write.execute (skipped in the conn_exception branch, where fn never
  runs). db.query.execute honours log_sql_errors for the same reason
  db.query does: facet suggestion probes with log_sql_errors=False and
  would otherwise paint two red spans per text column on every table page.
- The write-thread warm-up prepare_connection is left as a documented
  orphan root - no caller context exists that early.

Tests assert actual parent/child span-id relationships in a shared trace,
not just that spans exist, since an unparented root looks identical to a
correct span if you only check presence.

Note that copy_context() copies every ContextVar, not just OTel's, so
Datasette's own context vars (_skip_permission_checks,
_permission_check_cache, _in_datasette_client) now flow into worker
threads where they previously did not.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-01 16:24:35 -07:00
Alex Garcia
59bfa495cc Emit db.query spans from the three write entry points
execute_write(), execute_write_script() and execute_write_many() were the
only Database methods that ran SQL without producing an OpenTelemetry
span, so any instance doing writes - which is every instance, since
Datasette builds its internal catalog through these methods at startup -
showed reads in a trace and nothing else. The same db.system,
db.namespace and db.query.text attributes the read path already sets now
appear here, with db.query.text going through sql_attribute() so
attacker-supplied SQL cannot put an unbounded string on a span.

execute_write_many() records the parameter-set count as
datasette.param_sets, not datasette.rows_returned. executemany() consumes
parameter sets and returns no rows at all, so a rows_returned name would
be describing something that does not exist - and a consumer building a
"rows written" dashboard on top of it would be charting the wrong number.

These spans only cover the event-loop side of a write. The time actually
spent waiting on the write queue and executing on the write thread is not
attributed yet; that needs context propagation across the thread
boundary and lands separately. Writes with block=False are worse still -
execute_write_fn returns before the write happens, so the span closes
early. Span links fix that later.

As with the read path, the existing `with trace(...)` wrappers stay put
and the new spans nest inside them, so ?_trace=1 keeps working
unchanged - including execute_write_many's `count`, which the old tracer
stashes through the context manager's return value.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-01 16:24:15 -07:00
Alex Garcia
b40b06f1cb Emit a db.query span around Database.execute()
Datasette's existing tracer times a "sql" block that wraps a good deal
more than the query itself - queueing onto the thread pool, the pool
wait, and result marshalling all disappear into one number. That is
simonw/datasette#1730, "SQL tracing should much more closely track the
SQL query execution", open since 2022. A db.query span here is the outer
half of the answer; a later change adds the inner span drawn around the
sqlite3 call itself, and the gap between the two is exactly the thread
pool wait the current tracer folds away.

The span carries OTel semantic-convention attributes (db.system,
db.namespace, db.query.text) plus a few datasette.* ones. db.query.text
goes through sql_attribute(), which caps it at 2048 characters, because
on a public instance the SQL is attacker-supplied and unbounded. Only
len(params) is recorded, never a parameter value.

The existing `with trace(...)` wrapper stays exactly where it is and the
new span nests inside it. This change removes nothing: ?_trace=1 and the
trace_debug setting keep working unchanged. The two systems are
independent code paths.

Exception handling on the span is explicit rather than inherited from
start_as_current_span's defaults, which would record the exception and
set StatusCode.ERROR on anything passing through. That is wrong here
because some SQL failures are the expected answer. ArrayFacet.suggest()
runs json_type(<column>) against every column precisely to discover
which ones raise "malformed JSON", and passes log_sql_errors=False to
say so. Left to the defaults, a table with N text columns marks N
queries per page as failed - burying genuine failures and tripping any
alerting keyed on span status. Measured on a plain table page before
this: 4 error spans out of 225, all expected. Suppressed errors now
leave the status UNSET and set datasette.sql_error_suppressed instead,
so they stay discoverable without reading as failures.

QueryInterrupted still sets ERROR unconditionally. That is not quite
right either - facet suggestion is designed to time out - but the fix
needs its own reasoning and lands separately.

Behaviour change worth calling out: time_limit_ms is hoisted out of
sql_operation_in_thread so the span can record it on the event loop. It
is therefore read at call time rather than at thread-execution time.
Benign in practice, since ds.sql_time_limit_ms is set at startup, but it
is a real change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-01 16:24:15 -07:00
Alex Garcia
8194cb5a1d Add opentelemetry-api dependency and datasette/telemetry.py scaffolding
Datasette core is gaining OpenTelemetry spans alongside the existing
hand-rolled tracer. This commit only lays the groundwork - no span is
emitted yet.

Core takes a runtime dependency on opentelemetry-api and nothing more.
It deliberately never creates a TracerProvider, configures an exporter,
or touches sampling: that belongs to whoever runs Datasette, normally
via an opentelemetry-instrument agent. Owning a provider in core was
tried in an earlier design and produced a cross-request span leak, a
process-global provider that tests could not tear down, and a sampling
env var that silently blanked output. With no provider installed every
span is a NonRecordingSpan and costs approximately nothing.

datasette/telemetry.py exposes the module-level tracer plus
sql_attribute(), which truncates SQL to 2048 characters. On a public
instance the SQL is attacker-controlled and unbounded - someone can
paste a 10MB query into ?sql= - so it must never reach a telemetry
pipeline verbatim.

opentelemetry-sdk goes in the dev dependency group only, because the
test suite needs it to assert on spans while the package itself must
not import it. tests/test_telemetry.py enforces that by importing
datasette in a fresh interpreter and inspecting sys.modules, which
catches a lazy import inside a function body that a grep would miss.

conftest.py gains a session-scoped autouse fixture installing an SDK
provider with an InMemorySpanExporter. It has to be session-scoped
because set_tracer_provider() is effectively once-per-process - a
second call logs a warning and is ignored. SimpleSpanProcessor rather
than BatchSpanProcessor, so assertions made right after a request never
race a background export thread. The otel_spans fixture that later
tickets assert against is added here too.

test_datasette_package_never_imports_the_sdk is moved to the front of
the run. Late in a serial run the pytest process holds enough threads
that the fork half of subprocess' fork+exec segfaults the interpreter
on macOS/CPython 3.13. That reproduces with any subprocess call in that
position on an unmodified tree, so it is a pre-existing hazard rather
than something this commit introduces; the repo already moves its other
subprocess-spawning tests to the front for related reasons.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-01 16:24:15 -07:00
Alex Garcia
3e018bb1b5
Run startup via ASGI lifespan instead of waiting for the first request (#2887)
* Run startup via ASGI lifespan instead of waiting for the first request
* Ensure immutable table counts still precompute when startup ran first
2026-09-01 09:39:25 -07:00
Alex Garcia
e78b8a2e6a
Run datasette serve startup and uvicorn on a single event loop (#2886)
* Run datasette serve startup and uvicorn on a single event loop
* Move the serve-subprocess test plumbing into a conftest fixture
* Fix datasette-litestream URL and trim marker-task test comments
* Explain why serve_with_plugins needs a subprocess and plugin files
* Apply ruff 0.16 and black fixes
* Tweaked some comments
2026-09-01 09:32:37 -07:00
Simon Willison
0337fba234
disable_fts() before dropping table
Closes #2874
2026-08-10 15:03:29 -07:00
Simon Willison
eb6c2b96b9 Fix for SQL injection issue in table filters, refs #2868 2026-08-06 11:19:50 -07:00
Simon Willison
e889403d3b
Upgrade to ruff>=0.16.0 (#2857)
* ruff>=0.16.0

See https://astral.sh/blog/ruff-v0.16.0

* uv run ruff check . --fix --unsafe-fixes

* Ruff fixes by Claude Code Opus 5
2026-07-25 15:47:08 -07:00
TowyTowy
591b909a4d
Escape table names with [square] brackets, refs #2431 (#2846)
Several internal helpers quoted table names using SQLite [bracket]
identifiers built with an f-string, e.g. PRAGMA foreign_key_list([{table}]).
Bracket quoting cannot escape a "]" character, so any table whose name
contains "]" (for example "[foo]" or "foo]") produced
"sqlite3.OperationalError: unrecognized token" - crashing schema
introspection at startup and 500-ing the table page.

Switch these call sites to the existing escape_sqlite() helper, which uses
"double quote" quoting with correct "" escaping (the same approach already
used elsewhere in the codebase and in the test suite):

- utils/internal_db.py: PRAGMA foreign_key_list / index_list
- utils/__init__.py: get_outbound_foreign_keys
- database.py: table_counts count query
- facets.py: default "select * from" SQL

Added a regression test covering table names with "]" characters.

Co-authored-by: Claude <noreply@anthropic.com>
2026-07-14 08:53:45 -07:00
Simon Willison
7f0a8b38ae
Better permission debug tools and documentation
Closes #2841
2026-07-14 08:40:07 -07:00
Simon Willison
10088dfa1d
execute_write(transaction=False) parameter, plus fix for errors inside tasks
Ensure a write inside a failing Datasette task never becomes visible. Refs #2831
2026-07-13 22:42:44 -07:00
Simon Willison
ccace40e5a
/-/plugins.json is now an array of objects again (#2843)
Reverts the object envelope introduced in 1.0a36 for this endpoint -
it once again returns a top-level JSON array of plugin objects.

Closes #2842


Claude-Session: https://claude.ai/code/session_012TYc1NTBK4zEjabB3u2zqu

Co-authored-by: Claude <noreply@anthropic.com>
2026-07-13 21:19:04 -07:00
Simon Willison
54597f22fa A few more SQLite string fixes, refs #2783 2026-07-07 14:26:34 -07:00
JSap0914
bf3e277c98
Fix named_parameters when string literals contain comment markers (#2783)
named_parameters stripped SQL comments before string literals in
separate passes. A string literal such as '-- TODO' would be treated
as the start of a line comment, swallowing the rest of the line and
hiding any named parameters that followed it. For example:

    select * from t where note = '-- TODO' and id = :id

returned [] instead of ['id'], so the query parameter input form
would be missing the :id field.

Match comments and string literals in a single left-to-right pass so
that whichever construct starts first wins, matching how SQL is
actually tokenized.

Co-authored-by: JSap0914 <JSap0914@users.noreply.github.com>
2026-07-07 14:23:31 -07:00
Zain Dana Harper
211e70d4e1
Return 400 not 500 for wrong-arity composite-PK row URLs (#2815)
Fixes #2811

Co-authored-by: Zain Dana Harper <zain@aurora-framework.dev>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 14:19:08 -07:00
Simon Willison
db7ba1d30c Switch to sqlite-utils migrations for internal.db, closes #2827 2026-07-07 13:58:48 -07:00
Simon Willison
96e8b85523 Upgrade to sqlite-utils 4.0 2026-07-07 13:57:06 -07:00
Claude
be25d6e3e4
Remove test_query_list_json_signals_pagination_via_next_only
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-07 05:18:54 +00:00
Claude
4a853cb10c
Use UNSTABLE_API_MESSAGE constant in tests
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-07 05:18:54 +00:00
Claude
b83b12dd7a
Remove params input alias from the query create and update APIs
The alias existed so API payloads could mirror the params key used by
queries defined in datasette.yaml, but it was undocumented and untested,
and the create endpoint is not part of the stable API. The API now only
accepts parameters - sending params is a 400 Invalid keys error. The
documented params key for queries in configuration is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-07 01:13:33 +00:00
Claude
4874c29286
Remove the next_url extra - the key is always present
next_url became a default table JSON key alongside next, making the
extra a no-op. Requesting ?_extra=next_url now returns the standard
unknown-extra 400 error.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-06 23:59:53 +00:00
Claude
8b159144a5
Add Response.error() for JSON errors in the standard format
Response.error(messages, status=400) builds a JSON error response in
Datasette's standard error format, alongside Response.json/html/text.
messages can be a single string or a list. All internal error response
construction now uses it - the private views.base._error() helper is
gone and the verbose Response.json(error_body(...), status=...) sites
are converted. error_body() remains for the cases that merge the error
keys into a larger payload (the JSON renderer, handle_exception and the
permission debug payload builders).

Since Response is public plugin API, plugins that build JSON endpoints
now have an obvious way to return errors in the canonical shape.
Documented in the internals documentation, including the guidance to
raise Forbidden/NotFound/BadRequest/DatasetteError instead when the
error should content-negotiate.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-06 23:48:49 +00:00
Claude
0d962deb05
Plain text SQL Interrupted errors in JSON responses
The SQL time limit error embedded an HTML fragment (paragraph, textarea
and script tags) as the error string in JSON responses. DatasetteError
now accepts a plain_message which the exception handler prefers for
JSON error bodies; the HTML error page keeps the rich message with the
SQL textarea.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-06 23:34:17 +00:00
Claude
87cd695ca3
Write endpoints parse the body as JSON regardless of Content-Type
The insert, upsert, alter and set-column-type endpoints previously
required Content-Type: application/json while /-/create parsed the body
blind - and insert returned a 500 AttributeError when the header was
missing entirely. The lenient rule is now uniform: the body is always
parsed as JSON and invalid JSON is a 400. This makes curl -d and
requests data=json.dumps(...) invocations work without remembering the
header. Cross-site request forgery remains prevented by the Origin and
Sec-Fetch-Site checks in CrossOriginProtectionMiddleware, which is the
defense the strict content-type requirement was historically standing
in for.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-06 23:20:47 +00:00
Claude
e892c686c2
Remove has_more from query list JSON - next: null signals the end
next: null (with next_url: null) is the single end-of-results signal
across the API, keeping default response keys to a minimum. The
StoredQueryPage.has_more attribute on the documented Python API is
unchanged.

Also fixes a bug this uncovered: the query list JSON next_url pointed
at the HTML page (it was built from the query list path, dropping the
.json extension) and was a relative path where the table view next_url
is absolute. It is now built from the request path and absolute, so it
preserves the requested format and can be followed directly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-06 23:02:42 +00:00
Claude
5c418efd7f
Update tests from main for canonical error shape and rows key
Two tests merged from main were written against the pre-merge response
shapes: the max_post_body_bytes 413 error now uses the canonical error
envelope, and row update with return:true responds with a rows list
rather than a singular row.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-06 22:42:12 +00:00
Claude
194ee95ae2
Merge remote-tracking branch 'origin/main' into claude/json-api-docs-1-0-review-a3e83u
# Conflicts:
#	datasette/__init__.py
#	tests/test_api_write.py
2026-07-06 22:25:27 +00:00
Claude
f4dfd6e0f7
Fix unused variable flagged by ruff
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-06 22:19:35 +00:00
Simon Willison
b6f5fd5cd0 test_internal_foreign_key_references() fix for sqlite-utils 4.0rc3
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4889529980
2026-07-05 23:04:51 -07:00
Simon Willison
557e08c6ef Make binary playwright test more robust
Failed here: https://github.com/simonw/datasette/actions/runs/28746832371/job/85239037361
2026-07-05 22:46:00 -07:00
Simon Willison
6d253d10c8 Compatibility with sqlite-utils>=4.0rc2
Runs the tests in CI. Had to add a little bit of code to handle
the difference between [table] and "table" and REAL v.s FLOAT.
2026-07-05 09:09:20 -07:00
Claude
3a0ea58557
Unify page-size parameters on _size with table semantics
The stored query lists silently clamped out-of-range ?_size= values
(a request for 5000 quietly returned 1000) and did not accept the max
keyword. They now share the table view semantics via a new
parse_size_limit() helper: blank means default, "max" means the
maximum (max_returned_rows for query lists), negative or non-integer
values are a 400, and values over the maximum are a 400 instead of
being silently clamped.

The /-/allowed and /-/rules debug endpoints renamed their bare
page/page_size parameters to _page/_size, matching the underscore
grammar used by every other system parameter, with the same validation
(400 instead of silently capping page_size at 200). Their HTML debug
pages and next_url/previous_url builders use the new names.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 17:54:03 +00:00
Claude
afa7b1ba0d
Add unstable marker to undocumented JSON endpoints
JSON endpoints that are not part of the documented API now include
"unstable": "This API is not part of Datasette's stable interface and
may change at any time" in their responses, making the stability tier
machine-readable. Applied to the homepage (/.json and /-/.json),
/db/-/queries/analyze, POST /db/-/queries/store,
/db/<query>/-/definition, /db/-/query/parameters,
/db/-/execute-write/analyze and the POST /-/permissions playground
response. The message lives in datasette.utils.UNSTABLE_API_MESSAGE.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 17:01:53 +00:00
Claude
5cb2bc6909
Homepage JSON returns databases as a list
/.json previously returned databases as an object keyed by database
name, unlike /-/databases.json and every other collection in the API.
It now returns a list of database objects. The HTML template already
consumed the list. This endpoint remains undocumented.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 16:45:19 +00:00
Claude
13dc7a08b7
Fix foreign key label test expectations for default next_url key
Follow-up to the commit adding next_url to the default table JSON keys.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 16:30:04 +00:00
Claude
e5e9aca871
Require permissions-debug for /-/threads
/-/threads exposes runtime internals - thread idents and asyncio task
reprs including file paths - but only required view-instance. It now
requires permissions-debug, like /-/actions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 16:22:41 +00:00
Claude
0bf3a54716
Include next_url in default table JSON keys
Table JSON responses previously only included the next pagination token
by default - the ready-to-follow next_url required ?_extra=next_url.
Both keys are now always present (null on the final page), which the
pagination documentation already claimed. The next_url extra remains
valid for backwards compatibility.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 16:19:03 +00:00
Claude
e0ba8b3c6a
Return 400 for unknown _extra names on data formats
Unknown ?_extra= names (including internal HTML-only extras such as
display_rows) were silently ignored, so a typo returned the default
payload with no signal. Table, row and query data formats now return
400 "Unknown _extra: <names>". HTML pages continue to ignore unknown
names.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 16:16:02 +00:00
Claude
9ee95cab3d
Schema endpoints check permission before database existence
/db/-/schema previously returned 404 for missing databases before
checking view-database, letting unauthorized actors probe for database
existence. The permission check now runs first, so actors without
view-database get a uniform 403. The table schema endpoint also now
returns 404 for an unknown database instead of an unhandled KeyError.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 16:11:30 +00:00
Claude
b958d03c0f
Expose count truncation in table JSON via count_truncated extra
The count extra is computed with a limit subquery, so a count equal to
count_limit + 1 (default 10001) actually means "at least this many" -
but only the HTML view knew that. A public count_truncated extra now
reports the flag and is implicitly included whenever count is
requested, using the same logic the HTML view already used.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 16:09:40 +00:00
Claude
b09dceea88
Row update return:true responds with rows list, matching insert/upsert
Row update previously returned a singular "row" object where insert and
upsert return a "rows" list. All write endpoints now use "rows".

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 16:07:22 +00:00
Claude
6488b7a30e
Remove duplicate params key from stored query JSON objects
Every stored-query object carried the same list of parameter names
twice, as both "params" and "parameters". Output objects now carry only
"parameters", consistent with /-/query/parameters and the two analyze
endpoints (and distinct from the "params" bound-values dictionary used
by the query extra and /-/execute-write). "params" remains an accepted
input alias for query creation, update and datasette.yaml config.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 16:00:25 +00:00
Claude
f3f5e891c9
Block API deletion of trusted stored queries
QueryUpdateView already rejected is_trusted queries but QueryDeleteView
did not, so an actor with delete-query could delete a config-defined
trusted query - which would then silently reappear on restart when the
config re-syncs. Both the POST endpoint and the HTML confirmation page
now return 403, matching update. datasette.remove_query() is unchanged
for internal use.

The docs already claimed this behavior ("Trusted stored queries cannot
be edited or deleted through the web interface or the JSON API") - the
code now matches them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 15:49:49 +00:00
Claude
ea9c1b1524
Return 400 for query data formats when ?sql= is missing
GET /db/-/query.json with no (or blank) ?sql= previously returned 200
with empty rows, masking caller bugs, while the .csv format returned
400 "?sql= is required" for the same request. All data formats now
return the 400; the HTML SQL editor page is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 15:39:11 +00:00
Claude
aaaffe45b8
Return 401 for invalid or expired bearer tokens
Invalid dstok_ tokens - bad signature, malformed payload, expired, or
presented while allow_signed_tokens is off - previously degraded the
request to anonymous, so clients saw a 403 permission error or worse,
a 200 with anonymous-visible data. Token handlers can now raise
TokenInvalid for tokens they recognize but reject; Datasette responds
with 401, the canonical JSON error body and a WWW-Authenticate: Bearer
error="invalid_token" header, even when a valid cookie is also present.

Bearer tokens no registered handler recognizes are still ignored, so
authentication plugins with their own token formats keep working.
TokenInvalid is exported from the datasette package for use by plugin
token handlers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 15:18:12 +00:00