Commit 6f87527 said SQLite 3.53 refuses VACUUM INTO an existing in-memory
database. That was wrong: the test ran VACUUM INTO from a connection opened
without uri=True, and SQLite builds without SQLITE_USE_URI (the uv/
python-build-standalone 3.14 builds) treat the target as a plain file name,
so the first run created a file literally named
"file:review_fixes_copy?mode=memory&cache=shared" and later runs failed with
"output file already exists". The test already uses a plain connection; only
the comment changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
It fails 3 of 3 runs before the fix on Python 3.11; it had been dropped
after being checked, by mistake, against a Python 3.14 environment, where
the problem does not occur.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
test_new_memory_database_is_not_scanned_at_registration filled the memory
database with VACUUM INTO, which SQLite 3.53 (the free-threaded 3.14 build)
refuses for an existing in-memory database; it now uses a plain connection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
tests/connection_torture.py drives one Datasette from several event loops
(threads) with the smallest pool, 1ms idle timeout and 10ms schema polling
through every connection operation, plus add/remove_database, scratch
databases, external schema changes, replaced/deleted files, cancellation,
Database.close()/Datasette.close() mid-flight and non-threaded mode. One
scenario per process, so crashes and hangs are results. It checks for hangs,
crashes, undocumented exceptions, per-database write order, lost writes,
read-your-writes, catalog freshness, and fds/threads/the instance released
after close(). tests/test_connection_torture.py runs a short fixed-seed
subset of every scenario.
Fixes for what it found:
- A cursor that left the thread owning its connection (the cursor returned
by execute_write_many()/execute_write_script(), one returned by a write
callback, one rejected by execute_fn(), or one held by an exception's
frames) could be freed on the event loop thread while the connection ran
the same SQL again; on Python < 3.12 freeing it resets the shared cached
statement, and the other query failed with "bad parameter or other API
misuse". Such cursors are now closed on the owning thread. Pre-existing on
main (51 failures per 1,800 multi-loop execute_write_many calls).
- invoke_startup() is single-flight across event loops (a concurrent.futures
future, datasette.utils.inflight) instead of _startup_sequence() holding
an asyncio.Lock: a second loop got "bound to a different event loop" or
hung, and two loops could run the startup hooks twice. A request made
from a startup hook no longer re-runs or deadlocks startup.
- The index page takes one snapshot of ds.databases: a database removed,
renamed or closed mid-request, or whose file was deleted, no longer turns
the page into a 500. Database.size is 0 for a deleted file.
The scan-coalescing helpers move to datasette/utils/inflight.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
Schema watcher:
- Scan coalescing is loop-agnostic: in-flight scans are concurrent.futures
futures any event loop can wait for, taken over only when their loop has
closed or stopped; each refresh() resolves only its own scans. A write on
one loop no longer hangs forever when another loop's scan is in flight.
- At most one polling task across event loops; replaced pollers exit.
- A scan or catalog write that fails transiently (internal database locked,
shared-cache lock, cancellation) re-marks the database and retries it from
later requests and sweeps, with back-off; requests are never failed by it.
Write threads mark schema changes directly (under a lock) instead of
relying on a notification reaching a loop that may have closed.
- add_database() file databases and named in-memory databases use the
default_schema_watch mode (external unless configured). Named in-memory
databases can be external: polled with PRAGMA schema_version on their
write connection. They are not scanned at registration time, which raced
plugins filling them (datasette-copy-to-memory).
- close() no longer persists a fingerprint for an owned database whose file
header shows a schema the catalog has not seen.
- The derived-table permission rule (external-content FTS) trusts the
catalog only when a stat and header read prove it is current; otherwise
it asks the database, and schedules a rescan if the catalog is behind.
Connections and write threads:
- A BaseException from a write or isolated callback is delivered to the
caller as a RuntimeError instead of killing the thread silently.
- conn.close() on a write lease raises ConnectionLeaseError, and the write
thread reopens a connection something else closed.
- Write connections pin PRAGMA recursive_triggers=on (sqlite-utils FTS
triggers rely on it; idle reopen lost the old drift).
- execute_fn() accepts generators that no longer hold the connection or a
cursor.
- --crossdb _memory connections count against max_open_connections, are
reaped, and are discarded when databases are added or removed.
Lifecycle:
- The temp internal database file uses weakref.finalize, not atexit, so
closed or abandoned instances can be garbage collected.
- The read-pool reaper no longer keeps the last pool alive in its frame.
- The scratch directory is loaded (and locked) at the end of __init__, so a
failed construction leaves nothing locked.
- config_dir/scratch is used only if empty or created by Datasette; a second
instance on the same config dir starts without it; sidecar files are never
deleted the first time a directory is opened.
Regression tests: tests/test_connection_review_fixes.py (24 of 27 fail on
a509b81; the other 3 guard behaviour that must not change).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
