Adds an internals documentation section describing the task-equals-
transaction model for write connections, the autocommit read path, and
the cross-project constraint that Datasette and sqlite-utils both rely
on the legacy sqlite3 transaction handling - Python 3.12+ autocommit=
style connections are not supported and any migration would need to be
coordinated across both projects.
Refs #2831
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N76afGMhBRQk528VF1LTpR
New journal_mode setting lets deployments opt mutable database files
into WAL mode (or delete/truncate/persist), applied on the write
connection. WAL is paired with PRAGMA synchronous=NORMAL. Datasette
does not change the journal mode of database files by default.
Also fixes an inconsistency: a persistent internal database passed via
--internal now gets WAL enabled, matching the temporary internal
database default (which was moved to a temp disk file specifically so
it could use WAL).
Refs #2831
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N76afGMhBRQk528VF1LTpR
The SQLite busy timeout was previously an implicit policy - every
connection inherited the sqlite3 driver's silent 5 second default. It
is now an explicit, documented setting passed as timeout= to every
sqlite3.connect() call. The default remains 5000ms.
This matters for deployments where external processes write to the
same database files Datasette is serving.
Refs #2831
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N76afGMhBRQk528VF1LTpR
Wrapping a connection in sqlite_utils.Database() runs sqlite-utils
plugins' prepare_connection hooks against it by default. Datasette's
write API views and introspection helpers now pass execute_plugins=False
(matching what utils/internal_db.py already did), so third-party
sqlite-utils plugins no longer touch Datasette's connections.
Also apply PRAGMA recursive_triggers=on in Datasette._prepare_connection
so every connection gets consistent trigger semantics - previously only
the write connection got it, as a side effect of the first sqlite-utils
based write.
Refs #2831
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N76afGMhBRQk528VF1LTpR
populate_schema_tables() previously ran six separate transactions per
database: one task deleting that database's catalog rows followed by
five execute_write_many inserts. Readers of the internal database could
observe the intermediate state where a database had no catalog rows.
Schema details are still collected on a read connection of the target
database first; the delete-and-reinsert now happens inside one
execute_write_fn task so the rebuild is atomic.
Refs #2831
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N76afGMhBRQk528VF1LTpR
Adds regression tests confirming the JSON write API is atomic per
request now that write tasks open an explicit transaction: /db/-/create
with failing initial rows creates no table, a failing operation in
/db/table/-/alter rolls back earlier operations, and insert with
"return": true rolls back all rows if one fails.
Also removes the "with operation_conn:" block from the alter endpoint -
write functions run inside the task transaction and should not manage
transactions themselves (that context manager would commit the task
transaction early on success).
Refs #2831
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N76afGMhBRQk528VF1LTpR
execute_write_script() was documented as running inside a transaction
but actually passed transaction=False and used conn.executescript(),
which commits each statement as it executes - a failing script could
half-apply.
Scripts are now split into complete statements (via
sqlite3.complete_statement) and executed one at a time inside the task
transaction, so a failing script applies nothing. Scripts containing
statements that cannot run in a transaction (VACUUM, ATTACH, DETACH,
PRAGMA) or that manage transactions themselves (BEGIN, COMMIT,
SAVEPOINT etc) keep the previous executescript() autocommit behavior.
Refs #2831
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N76afGMhBRQk528VF1LTpR
Write tasks with transaction=True previously relied on the sqlite3
driver's implicit BEGIN, which only fires on the first raw
data-modifying statement. sqlite-utils 4.0 write methods found no open
transaction, committed their own work mid-task, and Datasette's
commit/rollback at task end was a no-op - so a failing write function
could leave partial writes permanently committed.
The write thread (and the non-threaded write path) now executes BEGIN
IMMEDIATE before invoking each transaction=True task, commits when it
returns and rolls back if it raises. sqlite-utils methods nest inside
that transaction as savepoints, restoring task-level atomicity for
every write path.
execute_write() now detects statements SQLite refuses to run inside a
transaction (VACUUM, ATTACH, DETACH, PRAGMA) and runs those in
autocommit mode, preserving previous behavior for e.g. trusted canned
queries that run VACUUM.
Refs #2831
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N76afGMhBRQk528VF1LTpR
named_parameters stripped SQL comments before string literals in
separate passes. A string literal such as '-- TODO' would be treated
as the start of a line comment, swallowing the rest of the line and
hiding any named parameters that followed it. For example:
select * from t where note = '-- TODO' and id = :id
returned [] instead of ['id'], so the query parameter input form
would be missing the :id field.
Match comments and string literals in a single left-to-right pass so
that whichever construct starts first wins, matching how SQL is
actually tokenized.
Co-authored-by: JSap0914 <JSap0914@users.noreply.github.com>
The alias existed so API payloads could mirror the params key used by
queries defined in datasette.yaml, but it was undocumented and untested,
and the create endpoint is not part of the stable API. The API now only
accepts parameters - sending params is a 400 Invalid keys error. The
documented params key for queries in configuration is unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
Documents the canonical error format, the ok/envelope changes, the
array-to-object endpoint conversions, 401s for invalid tokens, the
pagination and page-size unification, removed legacy keys and formats,
and the new Response.error(), TokenInvalid, count_truncated and
unstable-marker APIs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
next_url became a default table JSON key alongside next, making the
extra a no-op. Requesting ?_extra=next_url now returns the standard
unknown-extra 400 error.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
Response.error(messages, status=400) builds a JSON error response in
Datasette's standard error format, alongside Response.json/html/text.
messages can be a single string or a list. All internal error response
construction now uses it - the private views.base._error() helper is
gone and the verbose Response.json(error_body(...), status=...) sites
are converted. error_body() remains for the cases that merge the error
keys into a larger payload (the JSON renderer, handle_exception and the
permission debug payload builders).
Since Response is public plugin API, plugins that build JSON endpoints
now have an obvious way to return errors in the canonical shape.
Documented in the internals documentation, including the guidance to
raise Forbidden/NotFound/BadRequest/DatasetteError instead when the
error should content-negotiate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
existing-api.md and stable-api-recommendations.md were working
documents for the 1.0 API consistency review. Their content remains
available in this branch history; they are not intended to merge to
main.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
The SQL time limit error embedded an HTML fragment (paragraph, textarea
and script tags) as the error string in JSON responses. DatasetteError
now accepts a plain_message which the exception handler prefers for
JSON error bodies; the HTML error page keeps the rich message with the
SQL textarea.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
The insert, upsert, alter and set-column-type endpoints previously
required Content-Type: application/json while /-/create parsed the body
blind - and insert returned a 500 AttributeError when the header was
missing entirely. The lenient rule is now uniform: the body is always
parsed as JSON and invalid JSON is a 400. This makes curl -d and
requests data=json.dumps(...) invocations work without remembering the
header. Cross-site request forgery remains prevented by the Origin and
Sec-Fetch-Site checks in CrossOriginProtectionMiddleware, which is the
defense the strict content-type requirement was historically standing
in for.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
/-/config redacts values for keys whose names contain secret, key,
password, token, hash or dsn. Plugins that follow that naming get
automatic redaction; plugins that don't will leak their secrets on
that endpoint.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
Boolean arguments parsed by value_as_boolean() accept on/true/1 and
off/false/0 - state this once in the JSON API docs rather than leaving
each argument to imply its own grammar.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
An upsert may update existing rows without creating anything, so it
deliberately does not claim resource creation with a 201.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
next: null (with next_url: null) is the single end-of-results signal
across the API, keeping default response keys to a minimum. The
StoredQueryPage.has_more attribute on the documented Python API is
unchanged.
Also fixes a bug this uncovered: the query list JSON next_url pointed
at the HTML page (it was built from the query list path, dropping the
.json extension) and was a relative path where the table view next_url
is absolute. It is now built from the request path and absolute, so it
preserves the requested format and can be followed directly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
Two tests merged from main were written against the pre-merge response
shapes: the max_post_body_bytes 413 error now uses the canonical error
envelope, and row update with return:true responds with a rows list
rather than a singular row.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
The stored query lists silently clamped out-of-range ?_size= values
(a request for 5000 quietly returned 1000) and did not accept the max
keyword. They now share the table view semantics via a new
parse_size_limit() helper: blank means default, "max" means the
maximum (max_returned_rows for query lists), negative or non-integer
values are a 400, and values over the maximum are a 400 instead of
being silently clamped.
The /-/allowed and /-/rules debug endpoints renamed their bare
page/page_size parameters to _page/_size, matching the underscore
grammar used by every other system parameter, with the same validation
(400 instead of silently capping page_size at 200). Their HTML debug
pages and next_url/previous_url builders use the new names.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
