check_resources() listed /proc/self/fd, which only exists on Linux, so on
macOS every scenario crashed at the end with FileNotFoundError. macOS now
lists /dev/fd and resolves each descriptor with fcntl F_GETPATH, which,
like /proc, still reports files that have been deleted (psutil's
open_files() leaves those out). The temp directory is compared via
realpath(), since macOS reports /var/folders as /private/var/folders.
The test now fails with the harness's stderr when it exits without a
summary line, instead of an IndexError that hid the real traceback.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Commit 6f87527 said SQLite 3.53 refuses VACUUM INTO an existing in-memory
database. That was wrong: the test ran VACUUM INTO from a connection opened
without uri=True, and SQLite builds without SQLITE_USE_URI (the uv/
python-build-standalone 3.14 builds) treat the target as a plain file name,
so the first run created a file literally named
"file:review_fixes_copy?mode=memory&cache=shared" and later runs failed with
"output file already exists". The test already uses a plain connection; only
the comment changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
It fails 3 of 3 runs before the fix on Python 3.11; it had been dropped
after being checked, by mistake, against a Python 3.14 environment, where
the problem does not occur.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
test_new_memory_database_is_not_scanned_at_registration filled the memory
database with VACUUM INTO, which SQLite 3.53 (the free-threaded 3.14 build)
refuses for an existing in-memory database; it now uses a plain connection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
tests/connection_torture.py drives one Datasette from several event loops
(threads) with the smallest pool, 1ms idle timeout and 10ms schema polling
through every connection operation, plus add/remove_database, scratch
databases, external schema changes, replaced/deleted files, cancellation,
Database.close()/Datasette.close() mid-flight and non-threaded mode. One
scenario per process, so crashes and hangs are results. It checks for hangs,
crashes, undocumented exceptions, per-database write order, lost writes,
read-your-writes, catalog freshness, and fds/threads/the instance released
after close(). tests/test_connection_torture.py runs a short fixed-seed
subset of every scenario.
Fixes for what it found:
- A cursor that left the thread owning its connection (the cursor returned
by execute_write_many()/execute_write_script(), one returned by a write
callback, one rejected by execute_fn(), or one held by an exception's
frames) could be freed on the event loop thread while the connection ran
the same SQL again; on Python < 3.12 freeing it resets the shared cached
statement, and the other query failed with "bad parameter or other API
misuse". Such cursors are now closed on the owning thread. Pre-existing on
main (51 failures per 1,800 multi-loop execute_write_many calls).
- invoke_startup() is single-flight across event loops (a concurrent.futures
future, datasette.utils.inflight) instead of _startup_sequence() holding
an asyncio.Lock: a second loop got "bound to a different event loop" or
hung, and two loops could run the startup hooks twice. A request made
from a startup hook no longer re-runs or deadlocks startup.
- The index page takes one snapshot of ds.databases: a database removed,
renamed or closed mid-request, or whose file was deleted, no longer turns
the page into a 500. Database.size is 0 for a deleted file.
The scan-coalescing helpers move to datasette/utils/inflight.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
Schema watcher:
- Scan coalescing is loop-agnostic: in-flight scans are concurrent.futures
futures any event loop can wait for, taken over only when their loop has
closed or stopped; each refresh() resolves only its own scans. A write on
one loop no longer hangs forever when another loop's scan is in flight.
- At most one polling task across event loops; replaced pollers exit.
- A scan or catalog write that fails transiently (internal database locked,
shared-cache lock, cancellation) re-marks the database and retries it from
later requests and sweeps, with back-off; requests are never failed by it.
Write threads mark schema changes directly (under a lock) instead of
relying on a notification reaching a loop that may have closed.
- add_database() file databases and named in-memory databases use the
default_schema_watch mode (external unless configured). Named in-memory
databases can be external: polled with PRAGMA schema_version on their
write connection. They are not scanned at registration time, which raced
plugins filling them (datasette-copy-to-memory).
- close() no longer persists a fingerprint for an owned database whose file
header shows a schema the catalog has not seen.
- The derived-table permission rule (external-content FTS) trusts the
catalog only when a stat and header read prove it is current; otherwise
it asks the database, and schedules a rescan if the catalog is behind.
Connections and write threads:
- A BaseException from a write or isolated callback is delivered to the
caller as a RuntimeError instead of killing the thread silently.
- conn.close() on a write lease raises ConnectionLeaseError, and the write
thread reopens a connection something else closed.
- Write connections pin PRAGMA recursive_triggers=on (sqlite-utils FTS
triggers rely on it; idle reopen lost the old drift).
- execute_fn() accepts generators that no longer hold the connection or a
cursor.
- --crossdb _memory connections count against max_open_connections, are
reaped, and are discarded when databases are added or removed.
Lifecycle:
- The temp internal database file uses weakref.finalize, not atexit, so
closed or abandoned instances can be garbage collected.
- The read-pool reaper no longer keeps the last pool alive in its frame.
- The scratch directory is loaded (and locked) at the end of __init__, so a
failed construction leaves nothing locked.
- config_dir/scratch is used only if empty or created by Datasette; a second
instance on the same config dir starts without it; sidecar files are never
deleted the first time a directory is opened.
Regression tests: tests/test_connection_review_fixes.py (24 of 27 fail on
a509b81; the other 3 guard behaviour that must not change).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
- Startup with 1000 scratch databases (threaded and num_sql_threads=0):
Datasette() opens none of them; _startup_sequence() with a temporary
internal database opens each exactly once (its catalog scan) with no
pooled reads, writes or ATTACH; the index page, /-/databases.json,
/-/api, /-/allowed, /-/jump, /-/check, /-/permissions and idle sweeps
open none.
- Restart with a persisted internal database opens no scratch database,
even for one written after its last scan (closing fingerprints).
- check_databases() only opens each scratch database for the catalog scan
and does not fail on a broken scratch file.
- create, write, read, list, rename and delete touch only the scratch
database concerned (list touches nothing); direct sqlite3.connect()
calls are confined to the scratch directory (registry, WAL setup,
rename checkpoint).
- docs: serve does not check scratch databases; listing pages do not open
them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
Conflicts:
- database.py: keep both class attributes (is_scratch from scratch,
immutable_count_time_limit_ms from budget).
- cli.py check_databases(): take budget's catalog-first check and keep
scratch's rule that scratch databases are never checked (they are
skipped in _databases_needing_check()); the --crossdb warning no longer
counts scratch databases, which are never ATTACHed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
_virtual_table_module() stripped comments character by character from every
table's DDL. Computing derived-table dependencies for each scanned database
made that 1.4s of event-loop time per 17,500 tables at startup. Removing
comments only ever replaces text with spaces, so DDL without the word
"virtual" cannot declare a virtual table: return early.
Also lock SchemaWatcher.catalog_generation increments, since scans can be
stored from several event loops (threads) at once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
- The database page's columns, primary keys, FTS tables, hidden flags and
foreign keys come from the _internal catalog (catalog_all_foreign_keys()
added); only row counts query the database. Removes three round trips
per table and the O(tables^2) hidden-table check from that page.
- PaginatedResources.all() fetches everything after the first page in a
single query: each page re-ran the whole permission query, so iterating
87,000 tables cost O(N^2 / 1000) (21 s of the index page at N=5000).
- catalog_generation is bumped after the per-database state it covers.
- Docs: index page row-count behaviour (pages.rst), immutable counts are
computed on first use (performance.rst), catalog_tables.type and "pages
that cover many databases read the catalog" (internals.rst).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
Adds tests/connection_budget.py, a helper that records which databases a
block of code opens, reads, writes or ATTACHes, and tests/test_connection_budget.py
which asserts budgets for ~60 endpoints, startup, sweeps and telemetry with
50 attached databases.
Fixes for every all-database loop it found:
- Index page: tables, views, hidden tables, columns, primary keys, FTS
tables and foreign-key relationship counts come from the _internal
catalog (a few queries for any number of databases). Row counts are only
run when at most 20 databases need counting (COUNT_MAX_DATABASES);
above that the page shows table counts plus counts already known.
- Derived-table permissions: allowed_resources() no longer queries every
database per page; dependency maps are computed by the SchemaWatcher from
the rows it scans, cached per database, and read back from the catalog
for databases restored from a persisted catalog.
- catalog_tables gains a type column (PRAGMA table_list: table, virtual,
shadow) so hidden shadow tables can be identified without a connection.
- PaginatedResources.all() fetches pages of 1000 after the first.
- /-/api and /-/debug/autocomplete use the catalog; /-/schema does one
permission query and skips databases with nothing visible.
- /-/versions computes SQLite details once per instance, without crossdb
ATTACHes; detect_json1() caches its answer.
- /-/databases only stats/hashes databases the actor can see.
- Immutable table counts are computed when first needed, not at startup.
- check_databases() (serve, --get) reuses the catalog scan instead of
opening every database a second time, and reports "file is not a
database" as a usage error.
- SchemaWatcher: failed scans are retried when the file changes, not on
every sweep; scan connections skip crossdb ATTACH; catalog_generation
counter for caches.
- The SQL write UI no longer forces a stat sweep of every database.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
datasette.create_scratch_database() / delete_scratch_database() /
rename_scratch_database() / list_scratch_databases() manage SQLite files in
a scratch directory (--scratch-dir, config_dir/scratch, or a temporary
directory created on first use).
- Registry: a SQLite sidecar (.datasette-scratch.db) with intent states
(creating/deleting/renaming); startup reconciles it with the directory,
which is the source of truth (hand-copied files are adopted, hand-deleted
ones dropped, orphan -wal/-shm/-journal removed). flock() per directory.
- Scratch databases are attached at startup without opening them, use
schema watch mode "owned", and track last_used in memory (flushed to the
registry at most once a minute and on close).
- Delete/rename: queued reads and writes fail with ScratchDatabaseDeleted,
running ones finish; files are unlinked only once no connection
(including the SchemaWatcher's, now counted) has them open.
- Datasette.close() records final fingerprints of owned file databases in a
persistent internal database, so restarts reuse their catalog rows
without opening them.
- Scratch databases are not attached to crossdb _memory and are skipped by
check_databases().
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
- SchemaWatcher ran its stat sweeps and catalog scans with
asyncio.to_thread(), which starts threads: with num_sql_threads=0 they
now run inline, so Datasette still starts where threads cannot be
started at all (Pyodide). New test simulates that in a subprocess with
Thread.start() raising, covering reads, writes, isolated functions,
owned and external schema detection and a scratch database.
- refresh_schemas(force=True) waited on an asyncio.Lock, which binds to
whichever event loop first contends for it. It is now a threading.Lock
only ever acquired without blocking; a forced refresh that finds one
running runs its own sweep instead of waiting.
- Database.close() no longer copies the set of in-use write connections
(membership tests only), which could race a running write thread.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
- internals.rst: new "Connections and threads" section (read pool, idle
write threads, isolated connections, schema watching, deleted files,
non-threaded mode, prepare_connection) and "Connections are only valid
inside your function", which lists what the lease proxy cannot catch.
execute_fn/execute_write_fn/execute_isolated_fn/close/add_database
(schema_watch=) updated; the internal database is file-backed, and
temporary tables only last as long as their connection.
- plugin_hooks.rst: prepare_connection runs once per new connection and
connections come and go; write_wrapper's conn is leased.
- Metric and span descriptions no longer promise a "single write thread"
that lives forever; connections.open says what it counts. Metric and
span names are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
Callbacks passed to execute_write_fn() - and the internal callbacks of
execute_write(), execute_write_many() and execute_write_script(), plus
write_wrapper hooks - and execute_isolated_fn() now receive a
LeasedConnection that raises ConnectionLeaseError once the callback has
returned, in threaded and non-threaded mode. "with conn as c" yields the
proxy, not the raw connection. Write callbacks may still return a cursor
(lambda conn: conn.execute(...) is common, and the write_many/script
methods return theirs).
The threaded execute_isolated_fn() path for immutable databases now runs
as a tracked executor future, so Database.close() waits for it instead
of closing its connection mid-query (the segfault in the review, P03).
close() also leaves alone the connections of a write thread that is
still running after the 10s join; that thread closes them itself.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
Read path: the leased pool replaces the thread-local cache. Pooled
connections record Database._conn_generation when opened; a stale one is
discarded on checkout or return, and _invalidate_connections() (called
by the SchemaWatcher on a replaced/deleted file) closes that database's
idle connections at once. Leased ones are only ever closed by their own
thread on release.
Write path: the idle-exit write thread keeps its lock protocol; its
connection lives in a _WriteConnection holder that is reopened when the
generation moves, before the next task. Post-write schema_version check
and _after_write catalog await are kept for every write API in both
modes. Non-threaded write connections are reopened on a generation change
and never cached half-prepared.
Deleted files: write connections use mode=rw once Datasette has seen the
file exist (_file_seen), so a deleted file is never silently recreated,
while a database added before its file exists is still created by its
first write.
Settings: max_open_connections (clamped to >= 4 x num_sql_threads),
connection_idle_timeout_ms (30000, read connections and write threads),
schema_watch_interval_ms, default_schema_watch (external|owned). The
spike-only knobs (connection_idle_timeout, connection_pool_wait_ms,
pool_read_connections, write_thread_idle_timeout_ms) are gone, along with
the pool's wait/disable code paths. The reaper falls back to checkout-time
reaping where threads cannot be started (Pyodide).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
Settings-list conflicts resolved as a union; the settings are
consolidated in a later commit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
remove_database() closes the Database, which closed every connection in
_all_connections - including a catalog scan connection still in use on a
worker thread: segfault (crashed an xdist worker in the full suite,
reproduced with an add/remove stress loop). Database.connect(track=False)
leaves the watcher's short-lived connections to the watcher, which always
closes them on the thread that opened them. Regression test added.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
When the fingerprint shows no non-empty -wal or -journal the main file is
the whole database, so the sweep reads the schema cookie (offset 40 of
the header) with pread instead of opening a connection. No SHARED lock
means no busy back-off for a concurrent writer: polling was costing
write-heavy phases ~10% throughput and up to 2x p99. The fingerprint is
still taken first, so a commit in flight is covered by the racy rule.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
- write connections reopened after the watcher saw the file replaced or
deleted use mode=rw, so a write to a deleted database fails instead of
silently creating a new empty file
- derived_table_dependencies() keys its cache on the schema_version the
watcher stored in the catalog (plus its scan count) instead of running
PRAGMA schema_version on a pooled read connection - allowed_resources()
calls it for every database on every page
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
- collect_schema uses pragma_table_xinfo/foreign_key_list/index_list
table-valued functions joined to sqlite_master: 3 queries per database
instead of 3 per table (falls back to the per-table loop on error)
- bulk scans run one chunk at a time (thread concurrency only added GIL
contention); internal-DB writes of chunk k overlap the scan of k+1
- an empty -wal/-journal counts as absent (read-only opens of WAL
databases create an empty -wal) and side files ignore ctime (SQLite
fchown()s them on every open when running as root)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
Per-database schema_watch modes (owned / external / immutable):
- write path reads PRAGMA schema_version after every write task and
rebuilds that one database's catalog before the write returns
- external files are polled with a stat() fingerprint prefilter
(racily-clean fingerprints re-verified), PRAGMA schema_version only on
change, rebuild only when the version moved or the inode changed
- immutable databases are scanned once
- remove_database deletes its catalog rows explicitly; startup deletes
rows for databases that are not attached (no per-refresh full scans)
- catalog reads use one short-lived connection per database inside one
read transaction; shared-cache memory databases go through their
write thread
- replaced/deleted files invalidate cached connections (generation)
- fingerprints persisted in catalog_databases.fingerprint (migration
0002) so restarts skip unchanged files
- new setting schema_watch_interval_ms (default 1000, 0 = no polling)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
A traceback that keeps the frame alive (e.g. QueryInterrupted from a
facet suggestion) kept the statement unfinalized; when the pool later
closed that connection it became a zombie that held its fds until the
next gc. With this, fd cross-check matches in 3/3 N=1000 runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
New setting write_thread_idle_timeout_ms (default 30000, 0 disables).
A database's write thread now waits on queue.get(timeout=...) and, when
it times out, takes _write_thread_lock, re-checks that the queue is
empty, deregisters itself and closes its write connection. Producers
enqueue and check/start the thread under the same lock, so a task can
never be left in the queue with no thread to run it.
Also: close() refuses new tasks under the lock and joins retiring
threads; a failed connect is retried by the next write once the queue
drains; memory databases never idle out; a thread that dies from a
BaseException deregisters (and restarts if tasks are waiting); the
thread is registered only after start() succeeds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
Replace the per-thread cached read connections with a pool. Each
execute_fn()/execute() callback checks a connection out for its duration
and gets a LeasedConnection proxy that raises ConnectionLeaseError if
used after the callback returns.
New settings: max_open_connections (global LRU cap, default 128),
connection_idle_timeout (seconds, default 60, enforced by one
process-wide reaper thread plus opportunistic reaping on checkout),
connection_pool_wait_ms (soft cap: wait then exceed) and
pool_read_connections (off = fresh connection per query, a control).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
