- Startup with 1000 scratch databases (threaded and num_sql_threads=0):
Datasette() opens none of them; _startup_sequence() with a temporary
internal database opens each exactly once (its catalog scan) with no
pooled reads, writes or ATTACH; the index page, /-/databases.json,
/-/api, /-/allowed, /-/jump, /-/check, /-/permissions and idle sweeps
open none.
- Restart with a persisted internal database opens no scratch database,
even for one written after its last scan (closing fingerprints).
- check_databases() only opens each scratch database for the catalog scan
and does not fail on a broken scratch file.
- create, write, read, list, rename and delete touch only the scratch
database concerned (list touches nothing); direct sqlite3.connect()
calls are confined to the scratch directory (registry, WAL setup,
rename checkpoint).
- docs: serve does not check scratch databases; listing pages do not open
them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
- The database page's columns, primary keys, FTS tables, hidden flags and
foreign keys come from the _internal catalog (catalog_all_foreign_keys()
added); only row counts query the database. Removes three round trips
per table and the O(tables^2) hidden-table check from that page.
- PaginatedResources.all() fetches everything after the first page in a
single query: each page re-ran the whole permission query, so iterating
87,000 tables cost O(N^2 / 1000) (21 s of the index page at N=5000).
- catalog_generation is bumped after the per-database state it covers.
- Docs: index page row-count behaviour (pages.rst), immutable counts are
computed on first use (performance.rst), catalog_tables.type and "pages
that cover many databases read the catalog" (internals.rst).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
Adds tests/connection_budget.py, a helper that records which databases a
block of code opens, reads, writes or ATTACHes, and tests/test_connection_budget.py
which asserts budgets for ~60 endpoints, startup, sweeps and telemetry with
50 attached databases.
Fixes for every all-database loop it found:
- Index page: tables, views, hidden tables, columns, primary keys, FTS
tables and foreign-key relationship counts come from the _internal
catalog (a few queries for any number of databases). Row counts are only
run when at most 20 databases need counting (COUNT_MAX_DATABASES);
above that the page shows table counts plus counts already known.
- Derived-table permissions: allowed_resources() no longer queries every
database per page; dependency maps are computed by the SchemaWatcher from
the rows it scans, cached per database, and read back from the catalog
for databases restored from a persisted catalog.
- catalog_tables gains a type column (PRAGMA table_list: table, virtual,
shadow) so hidden shadow tables can be identified without a connection.
- PaginatedResources.all() fetches pages of 1000 after the first.
- /-/api and /-/debug/autocomplete use the catalog; /-/schema does one
permission query and skips databases with nothing visible.
- /-/versions computes SQLite details once per instance, without crossdb
ATTACHes; detect_json1() caches its answer.
- /-/databases only stats/hashes databases the actor can see.
- Immutable table counts are computed when first needed, not at startup.
- check_databases() (serve, --get) reuses the catalog scan instead of
opening every database a second time, and reports "file is not a
database" as a usage error.
- SchemaWatcher: failed scans are retried when the file changes, not on
every sweep; scan connections skip crossdb ATTACH; catalog_generation
counter for caches.
- The SQL write UI no longer forces a stat sweep of every database.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
