Adds an internals documentation section describing the task-equals-
transaction model for write connections, the autocommit read path, and
the cross-project constraint that Datasette and sqlite-utils both rely
on the legacy sqlite3 transaction handling - Python 3.12+ autocommit=
style connections are not supported and any migration would need to be
coordinated across both projects.
Refs #2831
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N76afGMhBRQk528VF1LTpR
New journal_mode setting lets deployments opt mutable database files
into WAL mode (or delete/truncate/persist), applied on the write
connection. WAL is paired with PRAGMA synchronous=NORMAL. Datasette
does not change the journal mode of database files by default.
Also fixes an inconsistency: a persistent internal database passed via
--internal now gets WAL enabled, matching the temporary internal
database default (which was moved to a temp disk file specifically so
it could use WAL).
Refs #2831
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N76afGMhBRQk528VF1LTpR
The SQLite busy timeout was previously an implicit policy - every
connection inherited the sqlite3 driver's silent 5 second default. It
is now an explicit, documented setting passed as timeout= to every
sqlite3.connect() call. The default remains 5000ms.
This matters for deployments where external processes write to the
same database files Datasette is serving.
Refs #2831
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N76afGMhBRQk528VF1LTpR
Wrapping a connection in sqlite_utils.Database() runs sqlite-utils
plugins' prepare_connection hooks against it by default. Datasette's
write API views and introspection helpers now pass execute_plugins=False
(matching what utils/internal_db.py already did), so third-party
sqlite-utils plugins no longer touch Datasette's connections.
Also apply PRAGMA recursive_triggers=on in Datasette._prepare_connection
so every connection gets consistent trigger semantics - previously only
the write connection got it, as a side effect of the first sqlite-utils
based write.
Refs #2831
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N76afGMhBRQk528VF1LTpR
populate_schema_tables() previously ran six separate transactions per
database: one task deleting that database's catalog rows followed by
five execute_write_many inserts. Readers of the internal database could
observe the intermediate state where a database had no catalog rows.
Schema details are still collected on a read connection of the target
database first; the delete-and-reinsert now happens inside one
execute_write_fn task so the rebuild is atomic.
Refs #2831
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N76afGMhBRQk528VF1LTpR
Adds regression tests confirming the JSON write API is atomic per
request now that write tasks open an explicit transaction: /db/-/create
with failing initial rows creates no table, a failing operation in
/db/table/-/alter rolls back earlier operations, and insert with
"return": true rolls back all rows if one fails.
Also removes the "with operation_conn:" block from the alter endpoint -
write functions run inside the task transaction and should not manage
transactions themselves (that context manager would commit the task
transaction early on success).
Refs #2831
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N76afGMhBRQk528VF1LTpR
execute_write_script() was documented as running inside a transaction
but actually passed transaction=False and used conn.executescript(),
which commits each statement as it executes - a failing script could
half-apply.
Scripts are now split into complete statements (via
sqlite3.complete_statement) and executed one at a time inside the task
transaction, so a failing script applies nothing. Scripts containing
statements that cannot run in a transaction (VACUUM, ATTACH, DETACH,
PRAGMA) or that manage transactions themselves (BEGIN, COMMIT,
SAVEPOINT etc) keep the previous executescript() autocommit behavior.
Refs #2831
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N76afGMhBRQk528VF1LTpR
Write tasks with transaction=True previously relied on the sqlite3
driver's implicit BEGIN, which only fires on the first raw
data-modifying statement. sqlite-utils 4.0 write methods found no open
transaction, committed their own work mid-task, and Datasette's
commit/rollback at task end was a no-op - so a failing write function
could leave partial writes permanently committed.
The write thread (and the non-threaded write path) now executes BEGIN
IMMEDIATE before invoking each transaction=True task, commits when it
returns and rolls back if it raises. sqlite-utils methods nest inside
that transaction as savepoints, restoring task-level atomicity for
every write path.
execute_write() now detects statements SQLite refuses to run inside a
transaction (VACUUM, ATTACH, DETACH, PRAGMA) and runs those in
autocommit mode, preserving previous behavior for e.g. trusted canned
queries that run VACUUM.
Refs #2831
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N76afGMhBRQk528VF1LTpR
The alias existed so API payloads could mirror the params key used by
queries defined in datasette.yaml, but it was undocumented and untested,
and the create endpoint is not part of the stable API. The API now only
accepts parameters - sending params is a 400 Invalid keys error. The
documented params key for queries in configuration is unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
Documents the canonical error format, the ok/envelope changes, the
array-to-object endpoint conversions, 401s for invalid tokens, the
pagination and page-size unification, removed legacy keys and formats,
and the new Response.error(), TokenInvalid, count_truncated and
unstable-marker APIs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
next_url became a default table JSON key alongside next, making the
extra a no-op. Requesting ?_extra=next_url now returns the standard
unknown-extra 400 error.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
Response.error(messages, status=400) builds a JSON error response in
Datasette's standard error format, alongside Response.json/html/text.
messages can be a single string or a list. All internal error response
construction now uses it - the private views.base._error() helper is
gone and the verbose Response.json(error_body(...), status=...) sites
are converted. error_body() remains for the cases that merge the error
keys into a larger payload (the JSON renderer, handle_exception and the
permission debug payload builders).
Since Response is public plugin API, plugins that build JSON endpoints
now have an obvious way to return errors in the canonical shape.
Documented in the internals documentation, including the guidance to
raise Forbidden/NotFound/BadRequest/DatasetteError instead when the
error should content-negotiate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
The insert, upsert, alter and set-column-type endpoints previously
required Content-Type: application/json while /-/create parsed the body
blind - and insert returned a 500 AttributeError when the header was
missing entirely. The lenient rule is now uniform: the body is always
parsed as JSON and invalid JSON is a 400. This makes curl -d and
requests data=json.dumps(...) invocations work without remembering the
header. Cross-site request forgery remains prevented by the Origin and
Sec-Fetch-Site checks in CrossOriginProtectionMiddleware, which is the
defense the strict content-type requirement was historically standing
in for.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
/-/config redacts values for keys whose names contain secret, key,
password, token, hash or dsn. Plugins that follow that naming get
automatic redaction; plugins that don't will leak their secrets on
that endpoint.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
Boolean arguments parsed by value_as_boolean() accept on/true/1 and
off/false/0 - state this once in the JSON API docs rather than leaving
each argument to imply its own grammar.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
An upsert may update existing rows without creating anything, so it
deliberately does not claim resource creation with a 201.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
The stored query lists silently clamped out-of-range ?_size= values
(a request for 5000 quietly returned 1000) and did not accept the max
keyword. They now share the table view semantics via a new
parse_size_limit() helper: blank means default, "max" means the
maximum (max_returned_rows for query lists), negative or non-integer
values are a 400, and values over the maximum are a 400 instead of
being silently clamped.
The /-/allowed and /-/rules debug endpoints renamed their bare
page/page_size parameters to _page/_size, matching the underscore
grammar used by every other system parameter, with the same validation
(400 instead of silently capping page_size at 200). Their HTML debug
pages and next_url/previous_url builders use the new names.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
docs/json_api.rst now opens with an API stability section declaring
what the 1.x promise covers: documented endpoints, parameters and
response keys are stable with additive-only changes; pagination tokens
are opaque strings; the error format and token restriction semantics
are stable. It lists the exempt tiers: endpoints carrying the
"unstable" marker key, debug and support endpoints (/-/threads,
/-/actions, /-/jump, the permission debug endpoints, table
autocomplete), and keys explicitly labeled unstable such as the
execute-write analysis block. Cross-referenced from the introspection
and permission-debug documentation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
/-/threads exposes runtime internals - thread idents and asyncio task
reprs including file paths - but only required view-instance. It now
requires permissions-debug, like /-/actions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
Table JSON responses previously only included the next pagination token
by default - the ready-to-follow next_url required ?_extra=next_url.
Both keys are now always present (null on the final page), which the
pagination documentation already claimed. The next_url extra remains
valid for backwards compatibility.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
Unknown ?_extra= names (including internal HTML-only extras such as
display_rows) were silently ignored, so a typo returned the default
payload with no signal. Table, row and query data formats now return
400 "Unknown _extra: <names>". HTML pages continue to ignore unknown
names.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
The count extra is computed with a limit subquery, so a count equal to
count_limit + 1 (default 10001) actually means "at least this many" -
but only the HTML view knew that. A public count_truncated extra now
reports the flag and is implicitly included whenever count is
requested, using the same logic the HTML view already used.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
Row update previously returned a singular "row" object where insert and
upsert return a "rows" list. All write endpoints now use "rows".
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
Every stored-query object carried the same list of parameter names
twice, as both "params" and "parameters". Output objects now carry only
"parameters", consistent with /-/query/parameters and the two analyze
endpoints (and distinct from the "params" bound-values dictionary used
by the query extra and /-/execute-write). "params" remains an accepted
input alias for query creation, update and datasette.yaml config.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
Invalid dstok_ tokens - bad signature, malformed payload, expired, or
presented while allow_signed_tokens is off - previously degraded the
request to anonymous, so clients saw a 403 permission error or worse,
a 200 with anonymous-visible data. Token handlers can now raise
TokenInvalid for tokens they recognize but reject; Datasette responds
with 401, the canonical JSON error body and a WWW-Authenticate: Bearer
error="invalid_token" header, even when a valid cookie is also present.
Bearer tokens no registered handler recognizes are still ignored, so
authentication plugins with their own token formats keep working.
TokenInvalid is exported from the datasette package for use by plugin
token handlers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
POST to a write canned query previously returned HTTP 200 with
{"ok": false, "message": ...} when the SQL failed to execute, so JSON
clients (and anything that trusts HTTP status) recorded success for
failed writes. SQL failures now return 400 with the canonical error
shape plus the "redirect" context key from on_error_redirect; the
QueryWriteRejected 403 branch uses the canonical shape too. Successful
executions and the HTML flash-message flow are unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
The default forbidden() hook previously rendered an HTML error page even
for .json requests. It now returns the canonical JSON error shape with
status 403 when the request path ends in .json or the request sends an
Accept: application/json or Content-Type: application/json header. HTML
requests still get the error page.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
/-/databases previously listed every attached database (including
filesystem paths and sizes) to any actor with view-instance, while the
homepage and every other endpoint filtered by view-database. The
endpoint now only lists databases the current actor is allowed to view.
JsonDataView data callbacks may now be async.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
/-/actions.json now returns {"ok": true, "actions": [...]} instead of a
bare JSON array, so the response can grow additional keys without a
breaking change. The debug_actions.html template reads data.actions,
and the endpoint is now documented in docs/introspection.rst.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
/-/databases.json now returns {"ok": true, "databases": [...]} instead
of a bare JSON array, so the response can grow additional keys without
a breaking change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
/-/plugins.json now returns {"ok": true, "plugins": [...]} instead of a
bare JSON array, so the response can grow additional keys without a
breaking change. The `datasette plugins` CLI command still outputs a
plain array.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
JsonDataView now injects "ok": true into dict responses, covering
/-/versions, /-/settings, /-/config, /-/threads and /-/actor. The
homepage JSON, /-/jump, the three /-/schema endpoints, /-/allowed,
/-/rules, /-/check, POST /-/permissions and the table /-/autocomplete
endpoint set it explicitly.
The remaining top-level array endpoints (/-/plugins, /-/databases,
/-/actions) will be converted to objects in separate commits.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
All JSON error responses now use a single format built by the new
datasette.utils.error_body() helper:
{"ok": false, "error": "...", "errors": ["..."], "status": 400}
- error is all messages joined with '; ', errors is the full list,
status always matches the HTTP status code
- The exception handler no longer emits the legacy title key in JSON
(it is still available to the HTML error template)
- The permission debug endpoints (/-/allowed, /-/rules, /-/check,
POST /-/permissions) no longer return bare {"error": ...} objects
- JSON renderer SQL errors keep their rows/truncated context keys but
now include the canonical keys as well
- _shape=object misuse (queries or tables without primary keys) now
returns HTTP 400 instead of 200 with an error body
- Method-not-allowed 405 responses use the canonical shape
Adds tests/test_error_shape.py covering all four previous shape
producers, updates affected tests, and documents the format in a new
'Error responses' section of docs/json_api.rst.
Implements section 1 of stable-api-recommendations.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
