populate_schema_tables() previously ran six separate transactions per
database: one task deleting that database's catalog rows followed by
five execute_write_many inserts. Readers of the internal database could
observe the intermediate state where a database had no catalog rows.
Schema details are still collected on a read connection of the target
database first; the delete-and-reinsert now happens inside one
execute_write_fn task so the rebuild is atomic.
Refs #2831
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N76afGMhBRQk528VF1LTpR
Adds regression tests confirming the JSON write API is atomic per
request now that write tasks open an explicit transaction: /db/-/create
with failing initial rows creates no table, a failing operation in
/db/table/-/alter rolls back earlier operations, and insert with
"return": true rolls back all rows if one fails.
Also removes the "with operation_conn:" block from the alter endpoint -
write functions run inside the task transaction and should not manage
transactions themselves (that context manager would commit the task
transaction early on success).
Refs #2831
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N76afGMhBRQk528VF1LTpR
execute_write_script() was documented as running inside a transaction
but actually passed transaction=False and used conn.executescript(),
which commits each statement as it executes - a failing script could
half-apply.
Scripts are now split into complete statements (via
sqlite3.complete_statement) and executed one at a time inside the task
transaction, so a failing script applies nothing. Scripts containing
statements that cannot run in a transaction (VACUUM, ATTACH, DETACH,
PRAGMA) or that manage transactions themselves (BEGIN, COMMIT,
SAVEPOINT etc) keep the previous executescript() autocommit behavior.
Refs #2831
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N76afGMhBRQk528VF1LTpR
Write tasks with transaction=True previously relied on the sqlite3
driver's implicit BEGIN, which only fires on the first raw
data-modifying statement. sqlite-utils 4.0 write methods found no open
transaction, committed their own work mid-task, and Datasette's
commit/rollback at task end was a no-op - so a failing write function
could leave partial writes permanently committed.
The write thread (and the non-threaded write path) now executes BEGIN
IMMEDIATE before invoking each transaction=True task, commits when it
returns and rolls back if it raises. sqlite-utils methods nest inside
that transaction as savepoints, restoring task-level atomicity for
every write path.
execute_write() now detects statements SQLite refuses to run inside a
transaction (VACUUM, ATTACH, DETACH, PRAGMA) and runs those in
autocommit mode, preserving previous behavior for e.g. trusted canned
queries that run VACUUM.
Refs #2831
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N76afGMhBRQk528VF1LTpR
named_parameters stripped SQL comments before string literals in
separate passes. A string literal such as '-- TODO' would be treated
as the start of a line comment, swallowing the rest of the line and
hiding any named parameters that followed it. For example:
select * from t where note = '-- TODO' and id = :id
returned [] instead of ['id'], so the query parameter input form
would be missing the :id field.
Match comments and string literals in a single left-to-right pass so
that whichever construct starts first wins, matching how SQL is
actually tokenized.
Co-authored-by: JSap0914 <JSap0914@users.noreply.github.com>
The alias existed so API payloads could mirror the params key used by
queries defined in datasette.yaml, but it was undocumented and untested,
and the create endpoint is not part of the stable API. The API now only
accepts parameters - sending params is a 400 Invalid keys error. The
documented params key for queries in configuration is unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
Documents the canonical error format, the ok/envelope changes, the
array-to-object endpoint conversions, 401s for invalid tokens, the
pagination and page-size unification, removed legacy keys and formats,
and the new Response.error(), TokenInvalid, count_truncated and
unstable-marker APIs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
next_url became a default table JSON key alongside next, making the
extra a no-op. Requesting ?_extra=next_url now returns the standard
unknown-extra 400 error.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
Response.error(messages, status=400) builds a JSON error response in
Datasette's standard error format, alongside Response.json/html/text.
messages can be a single string or a list. All internal error response
construction now uses it - the private views.base._error() helper is
gone and the verbose Response.json(error_body(...), status=...) sites
are converted. error_body() remains for the cases that merge the error
keys into a larger payload (the JSON renderer, handle_exception and the
permission debug payload builders).
Since Response is public plugin API, plugins that build JSON endpoints
now have an obvious way to return errors in the canonical shape.
Documented in the internals documentation, including the guidance to
raise Forbidden/NotFound/BadRequest/DatasetteError instead when the
error should content-negotiate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
existing-api.md and stable-api-recommendations.md were working
documents for the 1.0 API consistency review. Their content remains
available in this branch history; they are not intended to merge to
main.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
The SQL time limit error embedded an HTML fragment (paragraph, textarea
and script tags) as the error string in JSON responses. DatasetteError
now accepts a plain_message which the exception handler prefers for
JSON error bodies; the HTML error page keeps the rich message with the
SQL textarea.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
The insert, upsert, alter and set-column-type endpoints previously
required Content-Type: application/json while /-/create parsed the body
blind - and insert returned a 500 AttributeError when the header was
missing entirely. The lenient rule is now uniform: the body is always
parsed as JSON and invalid JSON is a 400. This makes curl -d and
requests data=json.dumps(...) invocations work without remembering the
header. Cross-site request forgery remains prevented by the Origin and
Sec-Fetch-Site checks in CrossOriginProtectionMiddleware, which is the
defense the strict content-type requirement was historically standing
in for.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
/-/config redacts values for keys whose names contain secret, key,
password, token, hash or dsn. Plugins that follow that naming get
automatic redaction; plugins that don't will leak their secrets on
that endpoint.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
Boolean arguments parsed by value_as_boolean() accept on/true/1 and
off/false/0 - state this once in the JSON API docs rather than leaving
each argument to imply its own grammar.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
An upsert may update existing rows without creating anything, so it
deliberately does not claim resource creation with a 201.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
next: null (with next_url: null) is the single end-of-results signal
across the API, keeping default response keys to a minimum. The
StoredQueryPage.has_more attribute on the documented Python API is
unchanged.
Also fixes a bug this uncovered: the query list JSON next_url pointed
at the HTML page (it was built from the query list path, dropping the
.json extension) and was a relative path where the table view next_url
is absolute. It is now built from the request path and absolute, so it
preserves the requested format and can be followed directly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
Two tests merged from main were written against the pre-merge response
shapes: the max_post_body_bytes 413 error now uses the canonical error
envelope, and row update with return:true responds with a rows list
rather than a singular row.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
The stored query lists silently clamped out-of-range ?_size= values
(a request for 5000 quietly returned 1000) and did not accept the max
keyword. They now share the table view semantics via a new
parse_size_limit() helper: blank means default, "max" means the
maximum (max_returned_rows for query lists), negative or non-integer
values are a 400, and values over the maximum are a 400 instead of
being silently clamped.
The /-/allowed and /-/rules debug endpoints renamed their bare
page/page_size parameters to _page/_size, matching the underscore
grammar used by every other system parameter, with the same validation
(400 instead of silently capping page_size at 200). Their HTML debug
pages and next_url/previous_url builders use the new names.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
docs/json_api.rst now opens with an API stability section declaring
what the 1.x promise covers: documented endpoints, parameters and
response keys are stable with additive-only changes; pagination tokens
are opaque strings; the error format and token restriction semantics
are stable. It lists the exempt tiers: endpoints carrying the
"unstable" marker key, debug and support endpoints (/-/threads,
/-/actions, /-/jump, the permission debug endpoints, table
autocomplete), and keys explicitly labeled unstable such as the
execute-write analysis block. Cross-referenced from the introspection
and permission-debug documentation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
JSON endpoints that are not part of the documented API now include
"unstable": "This API is not part of Datasette's stable interface and
may change at any time" in their responses, making the stability tier
machine-readable. Applied to the homepage (/.json and /-/.json),
/db/-/queries/analyze, POST /db/-/queries/store,
/db/<query>/-/definition, /db/-/query/parameters,
/db/-/execute-write/analyze and the POST /-/permissions playground
response. The message lives in datasette.utils.UNSTABLE_API_MESSAGE.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
/.json previously returned databases as an object keyed by database
name, unlike /-/databases.json and every other collection in the API.
It now returns a list of database objects. The HTML template already
consumed the list. This endpoint remains undocumented.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
