Commit graph

1,363 commits

Author SHA1 Message Date
Alex Garcia
1e44c3dbec Mark in-process datasette.client requests on their SERVER span
An internal datasette.client request runs the full ASGI stack, so it
emits a second SERVER span nested inside the outer request's - which
double-counts requests in any dashboard that counts by span kind. Rather
than downgrading the inner span to INTERNAL (which would diverge from
how httpx-ASGI instrumentation behaves and break the registry's
kind-based dynamic-name matching), the span now carries an optional
datasette.internal_client=True attribute for dashboards to filter on.

The in_datasette_client ContextVar moves to telemetry.py so the
middleware can read it without a circular import; its writers and the
in_client() accessor stay in app.py.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012U7coQfVu8nK2R4q2mCULA
2026-09-02 14:20:32 -07:00
Alex Garcia
4aaf20355b Review polish: changelog entry, comment dedupe, semconv note, 404 route test
- Add the missing changelog bullet for the HTTP request span - the most
  user-visible signal in the stack had no entry.
- State the deliberate deviation on server.address: it is the verbatim
  Host header including any :port, not the semconv address/port split.
- Deduplicate the is_recording()-vs-is_valid rationale: the middleware
  fast path keeps the full telling, request_span() now points at it.
- Trim the http.route regex rationale in the docs intro to a pointer at
  the attribute description, halve the httpx raw_path digression, drop
  a comment that restated the call below it, and leave client-IP policy
  to the privacy section instead of the user_agent description.
- Assert the routed 404 still carries http.route and an enriched span
  name - route enrichment must not be gated on a successful response.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012U7coQfVu8nK2R4q2mCULA
2026-09-02 13:33:14 -07:00
Alex Garcia
8b5956a77b Name the request span after the route it matched
The request span was created at the ASGI edge, before anything knew which
route would match, so it carried nothing but the method: every request in a
trace UI showed up as "GET", and the only URL on it was url.path, which is
unbounded on a public instance and useless as a grouping key. Routing
resolves in DatasetteRouter, so that is where the span gets http.route and
its semconv `{method} {route}` name.

http.route is the compiled route pattern, not a prettified
/{database}/{table} template. Datasette routes with compiled regexes and the
route table is fixed when the app is built, so the pattern is exact, bounded
and needs no parsing; the transform into something prettier accretes edge
cases, and Django's instrumentation ships regex-flavoured routes for the same
reason. A request that matches no route gets no http.route and keeps its bare
method name, which is what semantic conventions ask for.

Two things the obvious implementation gets wrong, both found by testing it:

- The router must not read `get_current_span()`. A plugin asgi_wrapper()
  runs *inside* the request middleware, so an instrumented plugin makes its
  own span current for the whole request - and the route then lands on that
  plugin's INTERNAL span, renaming it, while the actual request span never
  gets the one attribute a trace UI groups by. It reproduces with a five-line
  plugin. The span is passed through the ASGI scope instead, falling back to
  the current span so an externally-created SERVER span is still enriched.

- The method has to be clamped again here. The middleware clamps it for the
  attribute, but the name is rebuilt from request.method, which is the raw
  client string - so an unclamped rename put `FROB /(?P<database>...` back
  into the span name that the middleware had just kept it out of.

Both guards are `is_recording()`, not `get_span_context().is_valid`: with no
provider but an inbound traceparent the API returns a NonRecordingSpan
carrying the remote context, which is valid and records nothing, so an
is_valid guard would do the work on every request from a traced caller.

Tests cover the route and name, the unrouted 404 fallback, the full attribute
set, db.query spans reaching the request span by parent walk, a 500, an
inbound traceparent becoming a remote parent, ?sql= never reaching a span
attribute, and - in a subprocess, because the suite's provider fixture is
session-scoped and unavoidable - the no-provider fast path handing the app
the original `send`. The streaming test uses a table larger than one page so
the export genuinely issues queries during the body send; without that it
passes however early the span ends.

Measured on this branch against fixtures.db: a faceted table page went from
112 spans in 56 traces to 113 spans in 1.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-02 13:33:14 -07:00
Alex Garcia
ad9da9959f Give every span a request to belong to
Nothing in Datasette created a span for the HTTP request itself, so every
span the database layer emits was a root span. Measured on this branch: one
faceted table page produces 70 spans in 36 separate traces, none of which
carries a URL. A trace UI shows that as dozens of unrelated single-span
traces per page, interleaved across concurrent requests - worse than
?_trace=1 at the exact job people reach for tracing to do. With the request
span it is 71 spans in 1 trace.

`opentelemetry-instrument` does not fix this on its own: auto-instrumentation
only picks up frameworks that ship an instrumentor entry point, and
Datasette's raw ASGI app is not one.

TelemetryMiddleware is mounted outermost in Datasette.app(), after the
asgi_wrapper() plugin loop, so plugin middleware and the CSRF layer run
*inside* the span. Putting it in DatasetteRouter instead would leave a span
created by an instrumented plugin as an orphan root - reintroducing the
problem for exactly the code most likely to be instrumented.

It stays at ~90 lines, against roughly 700 for
opentelemetry-instrumentation-asgi, because Datasette's app does not return
before its body is sent: route_path awaits response.asgi_send(send), and a
streaming CSV export runs its generator inline inside AsgiStream.asgi_send.
So a plain `finally` covers the response body and no deferred-end machinery
is needed.

Two decisions worth flagging for review:

- Inbound W3C traceparent and baggage are extracted, using the *global*
  propagator. That is the ecosystem norm (Flask, Django, FastAPI, the ASGI
  instrumentation), and going through the global propagator leaves the
  operator in control with no Datasette setting to invent:
  OTEL_PROPAGATORS=none disables it entirely. A public instance that does
  not want client-influenced traces should strip those headers at the proxy.
- url.query is not recorded, anywhere. Datasette query strings carry
  user-supplied SQL in ?sql= and canned query parameters. client.address is
  not recorded either.

The status code is sniffed from the ASGI http.response.start message rather
than read off a Response, because asgi_static, the favicon route, AsgiStream
and AsgiFileDownload all send that message themselves and never build one.
Only a >= 500 sets an error status - per semantic conventions a 4xx is the
client's mistake, and Datasette 404s are routine enough that treating them
as errors would bury a real 500.

The registry gains a `dynamic` flag, because this span's name is composed at
runtime and so can never equal a fixed registry string. Dynamic entries
resolve by span kind instead, and only after exact and prefix matching has
failed, so they cannot shadow a span that does have a registered name.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-02 13:33:14 -07:00
Alex Garcia
f73128dea6 Trace callback-style calls: execute_fn, execute_write_fn, execute_isolated_fn
The database instrumentation covered the four SQL-string entry points but
not the callback entry points, which are the documented way for plugins to
run arbitrary SQL - so the JSON write API's inserts and deletes, the
startup catalog scan, and every plugin built on execute_fn/execute_write_fn
were invisible to a trace, or worse, showed orphan-looking db.write.* spans
with no db.query above them.

Each callback method now opens the same db.query CLIENT span as its
SQL-string sibling, carrying a new optional datasette.callback attribute
(the callable's qualified name, captured before _wrap_fn_with_hooks() can
rename it) in place of db.query.text, which is now marked optional. A bare
execute_fn() also wraps the callback in a db.query.execute child, so the
"gap between the spans is thread-wait" story holds for plugin callbacks
too. No db.operation.name: there is no statement to take a keyword from,
and the registry says that attribute is omitted rather than guessed.

The previous bodies move to private _execute_fn()/_execute_write_fn() and
the SQL-string methods call those, so an execute() emits exactly the spans
it did before - pinned by test_execute_does_not_double_wrap. Database's own
introspection helpers stay on the public method deliberately: they are real
SQLite round trips, which lifts a table page from ~58 to ~100 (no-op) spans.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012U7coQfVu8nK2R4q2mCULA
2026-09-02 11:48:56 -07:00
Alex Garcia
6916287503 Review polish: drop unused prefix machinery, tighten comments and docs
- Remove the registry's unused prefix=True slot, its span_for() branch,
  its doc-rendering case and its test - nothing in the stack sets it.
- Stop promising a "later phase" query-duration metric dimension in the
  db.operation.name description; the cardinality rationale stands alone.
- Replace baked-in benchmark numbers in the telemetry module docstring
  with the docs' own phrasing (below run-to-run variation).
- Compact the duplicated copy_context() and enqueue-site comments in
  database.py to pointers at their canonical tellings.
- Make the "catch people out" gotchas skimmable as a bullet list and
  give the changelog's "nothing is removed" line a clear antecedent.
- Add a test that a result cut short by max_returned_rows records
  datasette.truncated=True - previously only ever asserted False.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012U7coQfVu8nK2R4q2mCULA
2026-09-02 11:25:10 -07:00
Alex Garcia
1052dc5c7b Document what the database-layer spans emit, and how to turn them on
The span reference itself is generated from the registry, so this adds the
prose the generated list cannot supply: how to actually see a span, what is
deliberately never recorded, and where the instrumentation stops short.

The "how to turn it on" part is the part people get wrong. Core installs no
provider, so OTEL_TRACES_EXPORTER=console against a plain `datasette` process
emits nothing at all - that variable is read by the SDK auto-configuration
which only runs under `opentelemetry-instrument`. Documented as a warning
because it reads like a bug when you hit it. Two more measured facts get the
same treatment: the SDK's BatchSpanProcessor default schedule delay is 5000ms
(checked, not assumed - `BatchSpanProcessor._default_schedule_delay_millis()`
on opentelemetry-sdk 1.44), so nothing appears for five seconds; and without
OTEL_SERVICE_NAME the default resource reports service.name=unknown_service.

Privacy properties are stated positively rather than left implicit: SQL
truncated at 2048 characters, parameter values never recorded, no actor
identifiers, table names only from an explicit `table=` argument. The last of
those is now documented on db.execute() itself, since it is public API.

The limitations section claims only what was measured. An earlier draft said
two traces per process are orphaned by the register_output_renderer and
asgi_wrapper hooks; measuring it showed a default install emits zero spans
from either, because Datasette queries no database there - it is a plugin
that would produce the orphan. Corrected to say that.

It also deliberately does NOT say an embedder must install its provider
before Datasette's first span or get nothing. That claim is false:
ProxyTracer._tracer returns the no-op tracer without caching it when no
provider is set, so early spans are dropped and nothing is poisoned.

The telemetry.py docstring said no-op spans "cost approximately nothing".
The benchmark for this diff does not support a claim that strong - a table
page emits ~58 spans - so it now states the measurement instead: median
9.80ms to 9.98ms across 15 runs, inside a 1.4ms run-to-run spread.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-01 16:24:35 -07:00
Alex Garcia
b74231c190 Stop marking a deliberately-short query budget as a span error
Datasette has a family of callers that run a query under a tiny time limit
and treat "did not finish" as a usable answer. table_counts() is the loudest:
the homepage counts every table with a 10ms budget and stores None for the
ones that blow it. The QueryInterrupted handler on the db.query span was
unconditional, so on a two-table database that produced four ERROR spans -
two db.query and two db.query.execute - on every homepage hit. Measured on a
30MB two-table database: 4 red spans before, 0 after.

Honouring log_sql_errors here would have silenced none of it. Only the three
ArrayFacet json_type() probes pass log_sql_errors=False, and they are not the
queries that time out; table_counts() and ColumnFacet.suggest both leave it at
its True default. The signal that does separate the two cases is the budget
itself: a caller asking for less time than sql_time_limit_ms is saying the
query may not finish. Keying off that needs no new API and no changes outside
database.py. A query that runs out the instance-wide limit is still an error.

datasette.interrupted is still set in every case - it is the signal worth
having, and only the ERROR status becomes conditional. Its registry
description said the status is "also set to ERROR" full stop, which is now
wrong, and that string is published in docs/internals.rst.

The inner db.query.execute span carried the same bug through
set_status_on_exception=log_sql_errors, so its exception handling is now
explicit, matching the db.query span above it. The context manager's flags
apply to every exception type alike and this span has to tell two apart.

test_query_interrupted_sets_error_status forced its timeout with
?_timelimit=5, which is exactly the signal now reclassified as expected. It
now forces one via sql_time_limit_ms so it still tests what it was written to
test.

Also documents, at the copy_context() sites, that context propagation carries
Datasette's non-OTel ContextVars into worker threads too. Verified harmless:
nothing reads _skip_permission_checks, _permission_check_cache or
_in_datasette_client off the event loop, and Context.run() restores the
thread's previous context on return, so no value can reach the next task on
the shared pool.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-01 16:24:35 -07:00
Alex Garcia
8d32eac895 Name every span and attribute once, in a registry the docs are built from
The span and attribute names were string literals spread across four call
sites in database.py and one in app.py, with a hand-written reference page
that would have been true only on the day it was written. That drift is not
hypothetical: an earlier iteration of this work carried a README asserting
parameter values were never recorded for two branches after that had stopped
being true.

datasette/telemetry_registry.py now holds each name once, with its
documentation. Attribute and SpanName subclass str, so a registry entry *is*
the string OpenTelemetry wants - no wrapper API over the OTel calls, no
parallel structure to keep in step, and a typo becomes an ImportError rather
than a silently misnamed attribute. docs/internals.rst renders the span
reference from it via cog, and `cog --check docs/*.rst` already runs in CI,
so the reference cannot drift from the definitions.

Nothing changes on the wire: the emitted span names and attribute keys are
byte-identical before and after, verified by diffing a dump of both.

tests/test_telemetry_registry.py exercises a real workload and compares it
against the registry in both directions - emitted-but-unregistered catches
instrumentation added without documentation, registered-but-never-emitted
catches documentation that has outlived its code. Because the call sites now
take their names from the registry, neither direction can catch a rename:
move DB_NAMESPACE to "db.namespace2" and code and registry still agree while
every dashboard breaks. So the literal names are also written out in the test
and asserted against the registry and against the wire separately. That pair
is the only comparison in the file not derived from the registry itself.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-01 16:24:35 -07:00
Simon Willison
12b25affb5 Release 1.0a38
Refs #2868
2026-08-06 11:20:25 -07:00
Simon Willison
e889403d3b
Upgrade to ruff>=0.16.0 (#2857)
* ruff>=0.16.0

See https://astral.sh/blog/ruff-v0.16.0

* uv run ruff check . --fix --unsafe-fixes

* Ruff fixes by Claude Code Opus 5
2026-07-25 15:47:08 -07:00
Simon Willison
481df7ff6d Shorten link text in changelog 2026-07-14 09:31:28 -07:00
Simon Willison
2ffd8a860e Release 1.0a37
Refs #2831, #2832, #2841, #2842, #2843, #2846
2026-07-14 09:28:29 -07:00
Simon Willison
7f0a8b38ae
Better permission debug tools and documentation
Closes #2841
2026-07-14 08:40:07 -07:00
Simon Willison
10088dfa1d
execute_write(transaction=False) parameter, plus fix for errors inside tasks
Ensure a write inside a failing Datasette task never becomes visible. Refs #2831
2026-07-13 22:42:44 -07:00
Simon Willison
ccace40e5a
/-/plugins.json is now an array of objects again (#2843)
Reverts the object envelope introduced in 1.0a36 for this endpoint -
it once again returns a top-level JSON array of plugin objects.

Closes #2842


Claude-Session: https://claude.ai/code/session_012TYc1NTBK4zEjabB3u2zqu

Co-authored-by: Claude <noreply@anthropic.com>
2026-07-13 21:19:04 -07:00
Simon Willison
52ae7d1b6d Release 1.0a36
Refs
#1983, #1996, #2783, #2806, #2809, #2811, #2812, #2813, #2815, #2818, #2819, #2822, #2823, #2827
2026-07-07 14:32:25 -07:00
Simon Willison
a31673c90b Changelog for #2811, #2815, #2783 2026-07-07 14:26:48 -07:00
Simon Willison
617acedd38 Remove readthedocs/actions/preview
Closes #2828
2026-07-07 14:18:10 -07:00
Simon Willison
7f37205e76 Remove Datasette Desktop from installation guide
Until I have time to fix it up and bring it back.
2026-07-07 14:04:46 -07:00
Simon Willison
a926ab392e Updated internals.rst schema using cog, refs #2827 2026-07-07 14:02:02 -07:00
Claude
b7bbde04be
Link the consistency review release note to PR #2824
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-07 05:18:54 +00:00
Claude
57ce1a059f
Tighten unstable marker release note
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-07 01:13:33 +00:00
Claude
b83b12dd7a
Remove params input alias from the query create and update APIs
The alias existed so API payloads could mirror the params key used by
queries defined in datasette.yaml, but it was undocumented and untested,
and the create endpoint is not part of the stable API. The API now only
accepts parameters - sending params is a 400 Invalid keys error. The
documented params key for queries in configuration is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-07 01:13:33 +00:00
Claude
b23fc4ec48
Unreleased release notes for the JSON API consistency review
Documents the canonical error format, the ok/envelope changes, the
array-to-object endpoint conversions, 401s for invalid tokens, the
pagination and page-size unification, removed legacy keys and formats,
and the new Response.error(), TokenInvalid, count_truncated and
unstable-marker APIs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-07 00:32:29 +00:00
Claude
610c24d59a
/-/jump is a stable documented endpoint, not a debug exemption
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-07 00:25:09 +00:00
Claude
4874c29286
Remove the next_url extra - the key is always present
next_url became a default table JSON key alongside next, making the
extra a no-op. Requesting ?_extra=next_url now returns the standard
unknown-extra 400 error.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-06 23:59:53 +00:00
Claude
8b159144a5
Add Response.error() for JSON errors in the standard format
Response.error(messages, status=400) builds a JSON error response in
Datasette's standard error format, alongside Response.json/html/text.
messages can be a single string or a list. All internal error response
construction now uses it - the private views.base._error() helper is
gone and the verbose Response.json(error_body(...), status=...) sites
are converted. error_body() remains for the cases that merge the error
keys into a larger payload (the JSON renderer, handle_exception and the
permission debug payload builders).

Since Response is public plugin API, plugins that build JSON endpoints
now have an obvious way to return errors in the canonical shape.
Documented in the internals documentation, including the guidance to
raise Forbidden/NotFound/BadRequest/DatasetteError instead when the
error should content-negotiate.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-06 23:48:49 +00:00
Claude
87cd695ca3
Write endpoints parse the body as JSON regardless of Content-Type
The insert, upsert, alter and set-column-type endpoints previously
required Content-Type: application/json while /-/create parsed the body
blind - and insert returned a 500 AttributeError when the header was
missing entirely. The lenient rule is now uniform: the body is always
parsed as JSON and invalid JSON is a 400. This makes curl -d and
requests data=json.dumps(...) invocations work without remembering the
header. Cross-site request forgery remains prevented by the Origin and
Sec-Fetch-Site checks in CrossOriginProtectionMiddleware, which is the
defense the strict content-type requirement was historically standing
in for.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-06 23:20:47 +00:00
Claude
8154f7149f
Advise plugin authors on naming secret configuration keys
/-/config redacts values for keys whose names contain secret, key,
password, token, hash or dsn. Plugins that follow that naming get
automatic redaction; plugins that don't will leak their secrets on
that endpoint.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-06 23:09:10 +00:00
Claude
3322e1f528
Document the boolean query string argument grammar
Boolean arguments parsed by value_as_boolean() accept on/true/1 and
off/false/0 - state this once in the JSON API docs rather than leaving
each argument to imply its own grammar.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-06 23:08:28 +00:00
Claude
6e17c51361
Document why upsert returns 200 where insert returns 201
An upsert may update existing rows without creating anything, so it
deliberately does not claim resource creation with a 201.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-06 23:08:05 +00:00
Claude
194ee95ae2
Merge remote-tracking branch 'origin/main' into claude/json-api-docs-1-0-review-a3e83u
# Conflicts:
#	datasette/__init__.py
#	tests/test_api_write.py
2026-07-06 22:25:27 +00:00
Claude
3a0ea58557
Unify page-size parameters on _size with table semantics
The stored query lists silently clamped out-of-range ?_size= values
(a request for 5000 quietly returned 1000) and did not accept the max
keyword. They now share the table view semantics via a new
parse_size_limit() helper: blank means default, "max" means the
maximum (max_returned_rows for query lists), negative or non-integer
values are a 400, and values over the maximum are a 400 instead of
being silently clamped.

The /-/allowed and /-/rules debug endpoints renamed their bare
page/page_size parameters to _page/_size, matching the underscore
grammar used by every other system parameter, with the same validation
(400 instead of silently capping page_size at 200). Their HTML debug
pages and next_url/previous_url builders use the new names.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 17:54:03 +00:00
Claude
404ee4c3a7
Document the JSON API stability promise
docs/json_api.rst now opens with an API stability section declaring
what the 1.x promise covers: documented endpoints, parameters and
response keys are stable with additive-only changes; pagination tokens
are opaque strings; the error format and token restriction semantics
are stable. It lists the exempt tiers: endpoints carrying the
"unstable" marker key, debug and support endpoints (/-/threads,
/-/actions, /-/jump, the permission debug endpoints, table
autocomplete), and keys explicitly labeled unstable such as the
execute-write analysis block. Cross-referenced from the introspection
and permission-debug documentation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 17:07:04 +00:00
Claude
e5e9aca871
Require permissions-debug for /-/threads
/-/threads exposes runtime internals - thread idents and asyncio task
reprs including file paths - but only required view-instance. It now
requires permissions-debug, like /-/actions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 16:22:41 +00:00
Claude
0bf3a54716
Include next_url in default table JSON keys
Table JSON responses previously only included the next pagination token
by default - the ready-to-follow next_url required ?_extra=next_url.
Both keys are now always present (null on the final page), which the
pagination documentation already claimed. The next_url extra remains
valid for backwards compatibility.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 16:19:03 +00:00
Claude
e0ba8b3c6a
Return 400 for unknown _extra names on data formats
Unknown ?_extra= names (including internal HTML-only extras such as
display_rows) were silently ignored, so a typo returned the default
payload with no signal. Table, row and query data formats now return
400 "Unknown _extra: <names>". HTML pages continue to ignore unknown
names.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 16:16:02 +00:00
Claude
b958d03c0f
Expose count truncation in table JSON via count_truncated extra
The count extra is computed with a limit subquery, so a count equal to
count_limit + 1 (default 10001) actually means "at least this many" -
but only the HTML view knew that. A public count_truncated extra now
reports the flag and is implicitly included whenever count is
requested, using the same logic the HTML view already used.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 16:09:40 +00:00
Claude
b09dceea88
Row update return:true responds with rows list, matching insert/upsert
Row update previously returned a singular "row" object where insert and
upsert return a "rows" list. All write endpoints now use "rows".

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 16:07:22 +00:00
Claude
6488b7a30e
Remove duplicate params key from stored query JSON objects
Every stored-query object carried the same list of parameter names
twice, as both "params" and "parameters". Output objects now carry only
"parameters", consistent with /-/query/parameters and the two analyze
endpoints (and distinct from the "params" bound-values dictionary used
by the query extra and /-/execute-write). "params" remains an accepted
input alias for query creation, update and datasette.yaml config.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 16:00:25 +00:00
Claude
aaaffe45b8
Return 401 for invalid or expired bearer tokens
Invalid dstok_ tokens - bad signature, malformed payload, expired, or
presented while allow_signed_tokens is off - previously degraded the
request to anonymous, so clients saw a 403 permission error or worse,
a 200 with anonymous-visible data. Token handlers can now raise
TokenInvalid for tokens they recognize but reject; Datasette responds
with 401, the canonical JSON error body and a WWW-Authenticate: Bearer
error="invalid_token" header, even when a valid cookie is also present.

Bearer tokens no registered handler recognizes are still ignored, so
authentication plugins with their own token formats keep working.
TokenInvalid is exported from the datasette package for use by plugin
token handlers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 15:18:12 +00:00
Claude
e8048e023f
Return 400 for write canned-query SQL failures
POST to a write canned query previously returned HTTP 200 with
{"ok": false, "message": ...} when the SQL failed to execute, so JSON
clients (and anything that trusts HTTP status) recorded success for
failed writes. SQL failures now return 400 with the canonical error
shape plus the "redirect" context key from on_error_redirect; the
QueryWriteRejected 403 branch uses the canonical shape too. Successful
executions and the HTML flash-message flow are unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 14:35:54 +00:00
Claude
ae10a99811
Return canonical JSON error for Forbidden on JSON requests
The default forbidden() hook previously rendered an HTML error page even
for .json requests. It now returns the canonical JSON error shape with
status 403 when the request path ends in .json or the request sends an
Accept: application/json or Content-Type: application/json header. HTML
requests still get the error page.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 14:09:10 +00:00
Claude
f091b6dab1
Filter /-/databases by view-database permission
/-/databases previously listed every attached database (including
filesystem paths and sizes) to any actor with view-instance, while the
homepage and every other endpoint filtered by view-database. The
endpoint now only lists databases the current actor is allowed to view.

JsonDataView data callbacks may now be async.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 14:00:42 +00:00
Claude
23ccdaeffc
Convert /-/actions.json from top-level array to object
/-/actions.json now returns {"ok": true, "actions": [...]} instead of a
bare JSON array, so the response can grow additional keys without a
breaking change. The debug_actions.html template reads data.actions,
and the endpoint is now documented in docs/introspection.rst.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 13:46:26 +00:00
Claude
19e54b10d4
Convert /-/databases.json from top-level array to object
/-/databases.json now returns {"ok": true, "databases": [...]} instead
of a bare JSON array, so the response can grow additional keys without
a breaking change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 13:43:44 +00:00
Claude
b74a8e5b12
Convert /-/plugins.json from top-level array to object
/-/plugins.json now returns {"ok": true, "plugins": [...]} instead of a
bare JSON array, so the response can grow additional keys without a
breaking change. The `datasette plugins` CLI command still outputs a
plain array.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 13:42:24 +00:00
Claude
089e96a437
Add "ok": true to every JSON object success response
JsonDataView now injects "ok": true into dict responses, covering
/-/versions, /-/settings, /-/config, /-/threads and /-/actor. The
homepage JSON, /-/jump, the three /-/schema endpoints, /-/allowed,
/-/rules, /-/check, POST /-/permissions and the table /-/autocomplete
endpoint set it explicitly.

The remaining top-level array endpoints (/-/plugins, /-/databases,
/-/actions) will be converted to objects in separate commits.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 13:40:05 +00:00
Claude
0679e04bd3
Unify JSON error responses into one canonical shape
All JSON error responses now use a single format built by the new
datasette.utils.error_body() helper:

    {"ok": false, "error": "...", "errors": ["..."], "status": 400}

- error is all messages joined with '; ', errors is the full list,
  status always matches the HTTP status code
- The exception handler no longer emits the legacy title key in JSON
  (it is still available to the HTML error template)
- The permission debug endpoints (/-/allowed, /-/rules, /-/check,
  POST /-/permissions) no longer return bare {"error": ...} objects
- JSON renderer SQL errors keep their rows/truncated context keys but
  now include the canonical keys as well
- _shape=object misuse (queries or tables without primary keys) now
  returns HTTP 400 instead of 200 with an error body
- Method-not-allowed 405 responses use the canonical shape

Adds tests/test_error_shape.py covering all four previous shape
producers, updates affected tests, and documents the format in a new
'Error responses' section of docs/json_api.rst.

Implements section 1 of stable-api-recommendations.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
2026-07-04 03:12:15 +00:00