New journal_mode setting lets deployments opt mutable database files
into WAL mode (or delete/truncate/persist), applied on the write
connection. WAL is paired with PRAGMA synchronous=NORMAL. Datasette
does not change the journal mode of database files by default.
Also fixes an inconsistency: a persistent internal database passed via
--internal now gets WAL enabled, matching the temporary internal
database default (which was moved to a temp disk file specifically so
it could use WAL).
Refs #2831
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N76afGMhBRQk528VF1LTpR
The SQLite busy timeout was previously an implicit policy - every
connection inherited the sqlite3 driver's silent 5 second default. It
is now an explicit, documented setting passed as timeout= to every
sqlite3.connect() call. The default remains 5000ms.
This matters for deployments where external processes write to the
same database files Datasette is serving.
Refs #2831
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N76afGMhBRQk528VF1LTpR
Wrapping a connection in sqlite_utils.Database() runs sqlite-utils
plugins' prepare_connection hooks against it by default. Datasette's
write API views and introspection helpers now pass execute_plugins=False
(matching what utils/internal_db.py already did), so third-party
sqlite-utils plugins no longer touch Datasette's connections.
Also apply PRAGMA recursive_triggers=on in Datasette._prepare_connection
so every connection gets consistent trigger semantics - previously only
the write connection got it, as a side effect of the first sqlite-utils
based write.
Refs #2831
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N76afGMhBRQk528VF1LTpR
execute_write_script() was documented as running inside a transaction
but actually passed transaction=False and used conn.executescript(),
which commits each statement as it executes - a failing script could
half-apply.
Scripts are now split into complete statements (via
sqlite3.complete_statement) and executed one at a time inside the task
transaction, so a failing script applies nothing. Scripts containing
statements that cannot run in a transaction (VACUUM, ATTACH, DETACH,
PRAGMA) or that manage transactions themselves (BEGIN, COMMIT,
SAVEPOINT etc) keep the previous executescript() autocommit behavior.
Refs #2831
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N76afGMhBRQk528VF1LTpR
Write tasks with transaction=True previously relied on the sqlite3
driver's implicit BEGIN, which only fires on the first raw
data-modifying statement. sqlite-utils 4.0 write methods found no open
transaction, committed their own work mid-task, and Datasette's
commit/rollback at task end was a no-op - so a failing write function
could leave partial writes permanently committed.
The write thread (and the non-threaded write path) now executes BEGIN
IMMEDIATE before invoking each transaction=True task, commits when it
returns and rolls back if it raises. sqlite-utils methods nest inside
that transaction as savepoints, restoring task-level atomicity for
every write path.
execute_write() now detects statements SQLite refuses to run inside a
transaction (VACUUM, ATTACH, DETACH, PRAGMA) and runs those in
autocommit mode, preserving previous behavior for e.g. trusted canned
queries that run VACUUM.
Refs #2831
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N76afGMhBRQk528VF1LTpR
execute_isolated_fn() always opened its temporary connection with
write=True, which is not allowed for immutable databases - so APIs
that rely on it, like SQL analysis when storing a query, failed.
An immutable database can never receive writes, so there is no write
queue to block: in that case the function now opens a read-only
connection and runs it on the executor, bypassing the write thread
entirely. Mutable databases keep the existing write-thread behavior.
Also fixed a latent bug in the write thread where a connect() failure
for an isolated task would crash the thread instead of delivering the
exception back to the caller.
Closes#2768
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix for execute write returning, closes#2762
* Fix stored write returning rowcount message
* Add configurable execute_write returning limit
* Return rows/truncated from execute query if it used RETURNING
* INSERT ... RETURNING shows rows in /-/execute-write
* Skip RETURNING tests if SQLite version does not support it
Screenshot: https://github.com/simonw/datasette/issues/2762#issuecomment-4588111545
Experimental, we may need this for the upcoming canned query
work so that we can tell if a user should be able to save
a writable canned query by confirming they have the right
permissions to update the affected tables.
Refs #2735
After this commit, Database.close() sends a sentinel to the write queue so
the background write thread exits cleanly, closes cached read/write
connections, and marks the instance closed. Subsequent calls to execute*()
raise DatasetteClosedError. close() remains idempotent and one-way.
Refs #2692
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes#2683
* Add is_temp_disk option to Database for temp file-backed databases
Replace the default in-memory internal database with a temporary
file-backed database using WAL mode. This fixes concurrent read/write
locking errors that occur with named in-memory SQLite databases.
The new is_temp_disk parameter on Database creates a temp file via
tempfile.mkstemp, connects to it as a regular file-based database
with WAL mode enabled, and cleans it up on close() and via atexit.
https://claude.ai/code/session_01TteLrUjpDcARjnP1GMRqz2
* Fix flaky test_database_page test with deterministic ordering
- Add ORDER BY to table_names() query in database.py
- Sort foreign keys deterministically in get_all_foreign_keys()
- Refactor test_database_page to use property-based assertions instead of
500+ lines of hardcoded expected data
- Run blacken-docs on plugin_hooks.rst
* Update test_row_foreign_key_tables for new deterministic FK ordering
The foreign keys are now sorted by (other_table, column, other_column),
so complex_foreign_keys comes before foreign_key_references alphabetically.
* Update test_table_names for new alphabetical ordering
The table_names() method now returns tables sorted alphabetically.
* Fix for test that fails prior to SQLite 3.37
---------
Co-authored-by: Claude <noreply@anthropic.com>
This fixes issues introduced by the ruff commit e57f391a which converted
Optional[x] to x | None:
- Fixed datasette/app.py line 1024: Dict[id | str, Dict] -> Dict[int | str, Dict]
(was using id built-in function instead of int type)
- Fixed datasette/app.py line 1074: Optional["Resource"] -> "Resource" | None
- Added 'from __future__ import annotations' for Python 3.10 compatibility
- Added TYPE_CHECKING blocks to avoid circular imports
- Removed dead code (unused variable assignments) from cli.py and views
- Removed unused imports flagged by ruff across multiple files
- Fixed test fixtures: moved app_client fixture imports to conftest.py
(fixed 71 test errors caused by fixtures not being registered)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Test for cross-database join, refs #283
* Warn if --crossdb used with more than 10 DBs, refs #283
* latest.datasette.io demo of --crossdb joins, refs #283
* Show attached databases on /_memory page, refs #283
* Documentation for cross-database queries, refs #283
* Support for generated columns, closes#1116
* Show SQLite version in pytest report header
* Use table_info() if SQLite < 3.26.0
* Cache sqlite_version() rather than re-calculate every time
* Adjust test_database_page for SQLite 3.26.0 or higher
