check_resources() listed /proc/self/fd, which only exists on Linux, so on
macOS every scenario crashed at the end with FileNotFoundError. macOS now
lists /dev/fd and resolves each descriptor with fcntl F_GETPATH, which,
like /proc, still reports files that have been deleted (psutil's
open_files() leaves those out). The temp directory is compared via
realpath(), since macOS reports /var/folders as /private/var/folders.
The test now fails with the harness's stderr when it exits without a
summary line, instead of an IndexError that hid the real traceback.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Commit 6f87527 said SQLite 3.53 refuses VACUUM INTO an existing in-memory
database. That was wrong: the test ran VACUUM INTO from a connection opened
without uri=True, and SQLite builds without SQLITE_USE_URI (the uv/
python-build-standalone 3.14 builds) treat the target as a plain file name,
so the first run created a file literally named
"file:review_fixes_copy?mode=memory&cache=shared" and later runs failed with
"output file already exists". The test already uses a plain connection; only
the comment changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
It fails 3 of 3 runs before the fix on Python 3.11; it had been dropped
after being checked, by mistake, against a Python 3.14 environment, where
the problem does not occur.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
test_new_memory_database_is_not_scanned_at_registration filled the memory
database with VACUUM INTO, which SQLite 3.53 (the free-threaded 3.14 build)
refuses for an existing in-memory database; it now uses a plain connection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
tests/connection_torture.py drives one Datasette from several event loops
(threads) with the smallest pool, 1ms idle timeout and 10ms schema polling
through every connection operation, plus add/remove_database, scratch
databases, external schema changes, replaced/deleted files, cancellation,
Database.close()/Datasette.close() mid-flight and non-threaded mode. One
scenario per process, so crashes and hangs are results. It checks for hangs,
crashes, undocumented exceptions, per-database write order, lost writes,
read-your-writes, catalog freshness, and fds/threads/the instance released
after close(). tests/test_connection_torture.py runs a short fixed-seed
subset of every scenario.
Fixes for what it found:
- A cursor that left the thread owning its connection (the cursor returned
by execute_write_many()/execute_write_script(), one returned by a write
callback, one rejected by execute_fn(), or one held by an exception's
frames) could be freed on the event loop thread while the connection ran
the same SQL again; on Python < 3.12 freeing it resets the shared cached
statement, and the other query failed with "bad parameter or other API
misuse". Such cursors are now closed on the owning thread. Pre-existing on
main (51 failures per 1,800 multi-loop execute_write_many calls).
- invoke_startup() is single-flight across event loops (a concurrent.futures
future, datasette.utils.inflight) instead of _startup_sequence() holding
an asyncio.Lock: a second loop got "bound to a different event loop" or
hung, and two loops could run the startup hooks twice. A request made
from a startup hook no longer re-runs or deadlocks startup.
- The index page takes one snapshot of ds.databases: a database removed,
renamed or closed mid-request, or whose file was deleted, no longer turns
the page into a 500. Database.size is 0 for a deleted file.
The scan-coalescing helpers move to datasette/utils/inflight.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
Schema watcher:
- Scan coalescing is loop-agnostic: in-flight scans are concurrent.futures
futures any event loop can wait for, taken over only when their loop has
closed or stopped; each refresh() resolves only its own scans. A write on
one loop no longer hangs forever when another loop's scan is in flight.
- At most one polling task across event loops; replaced pollers exit.
- A scan or catalog write that fails transiently (internal database locked,
shared-cache lock, cancellation) re-marks the database and retries it from
later requests and sweeps, with back-off; requests are never failed by it.
Write threads mark schema changes directly (under a lock) instead of
relying on a notification reaching a loop that may have closed.
- add_database() file databases and named in-memory databases use the
default_schema_watch mode (external unless configured). Named in-memory
databases can be external: polled with PRAGMA schema_version on their
write connection. They are not scanned at registration time, which raced
plugins filling them (datasette-copy-to-memory).
- close() no longer persists a fingerprint for an owned database whose file
header shows a schema the catalog has not seen.
- The derived-table permission rule (external-content FTS) trusts the
catalog only when a stat and header read prove it is current; otherwise
it asks the database, and schedules a rescan if the catalog is behind.
Connections and write threads:
- A BaseException from a write or isolated callback is delivered to the
caller as a RuntimeError instead of killing the thread silently.
- conn.close() on a write lease raises ConnectionLeaseError, and the write
thread reopens a connection something else closed.
- Write connections pin PRAGMA recursive_triggers=on (sqlite-utils FTS
triggers rely on it; idle reopen lost the old drift).
- execute_fn() accepts generators that no longer hold the connection or a
cursor.
- --crossdb _memory connections count against max_open_connections, are
reaped, and are discarded when databases are added or removed.
Lifecycle:
- The temp internal database file uses weakref.finalize, not atexit, so
closed or abandoned instances can be garbage collected.
- The read-pool reaper no longer keeps the last pool alive in its frame.
- The scratch directory is loaded (and locked) at the end of __init__, so a
failed construction leaves nothing locked.
- config_dir/scratch is used only if empty or created by Datasette; a second
instance on the same config dir starts without it; sidecar files are never
deleted the first time a directory is opened.
Regression tests: tests/test_connection_review_fixes.py (24 of 27 fail on
a509b81; the other 3 guard behaviour that must not change).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
- Startup with 1000 scratch databases (threaded and num_sql_threads=0):
Datasette() opens none of them; _startup_sequence() with a temporary
internal database opens each exactly once (its catalog scan) with no
pooled reads, writes or ATTACH; the index page, /-/databases.json,
/-/api, /-/allowed, /-/jump, /-/check, /-/permissions and idle sweeps
open none.
- Restart with a persisted internal database opens no scratch database,
even for one written after its last scan (closing fingerprints).
- check_databases() only opens each scratch database for the catalog scan
and does not fail on a broken scratch file.
- create, write, read, list, rename and delete touch only the scratch
database concerned (list touches nothing); direct sqlite3.connect()
calls are confined to the scratch directory (registry, WAL setup,
rename checkpoint).
- docs: serve does not check scratch databases; listing pages do not open
them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
Conflicts:
- database.py: keep both class attributes (is_scratch from scratch,
immutable_count_time_limit_ms from budget).
- cli.py check_databases(): take budget's catalog-first check and keep
scratch's rule that scratch databases are never checked (they are
skipped in _databases_needing_check()); the --crossdb warning no longer
counts scratch databases, which are never ATTACHed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
- The database page's columns, primary keys, FTS tables, hidden flags and
foreign keys come from the _internal catalog (catalog_all_foreign_keys()
added); only row counts query the database. Removes three round trips
per table and the O(tables^2) hidden-table check from that page.
- PaginatedResources.all() fetches everything after the first page in a
single query: each page re-ran the whole permission query, so iterating
87,000 tables cost O(N^2 / 1000) (21 s of the index page at N=5000).
- catalog_generation is bumped after the per-database state it covers.
- Docs: index page row-count behaviour (pages.rst), immutable counts are
computed on first use (performance.rst), catalog_tables.type and "pages
that cover many databases read the catalog" (internals.rst).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
Adds tests/connection_budget.py, a helper that records which databases a
block of code opens, reads, writes or ATTACHes, and tests/test_connection_budget.py
which asserts budgets for ~60 endpoints, startup, sweeps and telemetry with
50 attached databases.
Fixes for every all-database loop it found:
- Index page: tables, views, hidden tables, columns, primary keys, FTS
tables and foreign-key relationship counts come from the _internal
catalog (a few queries for any number of databases). Row counts are only
run when at most 20 databases need counting (COUNT_MAX_DATABASES);
above that the page shows table counts plus counts already known.
- Derived-table permissions: allowed_resources() no longer queries every
database per page; dependency maps are computed by the SchemaWatcher from
the rows it scans, cached per database, and read back from the catalog
for databases restored from a persisted catalog.
- catalog_tables gains a type column (PRAGMA table_list: table, virtual,
shadow) so hidden shadow tables can be identified without a connection.
- PaginatedResources.all() fetches pages of 1000 after the first.
- /-/api and /-/debug/autocomplete use the catalog; /-/schema does one
permission query and skips databases with nothing visible.
- /-/versions computes SQLite details once per instance, without crossdb
ATTACHes; detect_json1() caches its answer.
- /-/databases only stats/hashes databases the actor can see.
- Immutable table counts are computed when first needed, not at startup.
- check_databases() (serve, --get) reuses the catalog scan instead of
opening every database a second time, and reports "file is not a
database" as a usage error.
- SchemaWatcher: failed scans are retried when the file changes, not on
every sweep; scan connections skip crossdb ATTACH; catalog_generation
counter for caches.
- The SQL write UI no longer forces a stat sweep of every database.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
datasette.create_scratch_database() / delete_scratch_database() /
rename_scratch_database() / list_scratch_databases() manage SQLite files in
a scratch directory (--scratch-dir, config_dir/scratch, or a temporary
directory created on first use).
- Registry: a SQLite sidecar (.datasette-scratch.db) with intent states
(creating/deleting/renaming); startup reconciles it with the directory,
which is the source of truth (hand-copied files are adopted, hand-deleted
ones dropped, orphan -wal/-shm/-journal removed). flock() per directory.
- Scratch databases are attached at startup without opening them, use
schema watch mode "owned", and track last_used in memory (flushed to the
registry at most once a minute and on close).
- Delete/rename: queued reads and writes fail with ScratchDatabaseDeleted,
running ones finish; files are unlinked only once no connection
(including the SchemaWatcher's, now counted) has them open.
- Datasette.close() records final fingerprints of owned file databases in a
persistent internal database, so restarts reuse their catalog rows
without opening them.
- Scratch databases are not attached to crossdb _memory and are skipped by
check_databases().
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
- SchemaWatcher ran its stat sweeps and catalog scans with
asyncio.to_thread(), which starts threads: with num_sql_threads=0 they
now run inline, so Datasette still starts where threads cannot be
started at all (Pyodide). New test simulates that in a subprocess with
Thread.start() raising, covering reads, writes, isolated functions,
owned and external schema detection and a scratch database.
- refresh_schemas(force=True) waited on an asyncio.Lock, which binds to
whichever event loop first contends for it. It is now a threading.Lock
only ever acquired without blocking; a forced refresh that finds one
running runs its own sweep instead of waiting.
- Database.close() no longer copies the set of in-use write connections
(membership tests only), which could race a running write thread.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
Callbacks passed to execute_write_fn() - and the internal callbacks of
execute_write(), execute_write_many() and execute_write_script(), plus
write_wrapper hooks - and execute_isolated_fn() now receive a
LeasedConnection that raises ConnectionLeaseError once the callback has
returned, in threaded and non-threaded mode. "with conn as c" yields the
proxy, not the raw connection. Write callbacks may still return a cursor
(lambda conn: conn.execute(...) is common, and the write_many/script
methods return theirs).
The threaded execute_isolated_fn() path for immutable databases now runs
as a tracked executor future, so Database.close() waits for it instead
of closing its connection mid-query (the segfault in the review, P03).
close() also leaves alone the connections of a write thread that is
still running after the 10s join; that thread closes them itself.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
Read path: the leased pool replaces the thread-local cache. Pooled
connections record Database._conn_generation when opened; a stale one is
discarded on checkout or return, and _invalidate_connections() (called
by the SchemaWatcher on a replaced/deleted file) closes that database's
idle connections at once. Leased ones are only ever closed by their own
thread on release.
Write path: the idle-exit write thread keeps its lock protocol; its
connection lives in a _WriteConnection holder that is reopened when the
generation moves, before the next task. Post-write schema_version check
and _after_write catalog await are kept for every write API in both
modes. Non-threaded write connections are reopened on a generation change
and never cached half-prepared.
Deleted files: write connections use mode=rw once Datasette has seen the
file exist (_file_seen), so a deleted file is never silently recreated,
while a database added before its file exists is still created by its
first write.
Settings: max_open_connections (clamped to >= 4 x num_sql_threads),
connection_idle_timeout_ms (30000, read connections and write threads),
schema_watch_interval_ms, default_schema_watch (external|owned). The
spike-only knobs (connection_idle_timeout, connection_pool_wait_ms,
pool_read_connections, write_thread_idle_timeout_ms) are gone, along with
the pool's wait/disable code paths. The reaper falls back to checkout-time
reaping where threads cannot be started (Pyodide).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
Settings-list conflicts resolved as a union; the settings are
consolidated in a later commit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
remove_database() closes the Database, which closed every connection in
_all_connections - including a catalog scan connection still in use on a
worker thread: segfault (crashed an xdist worker in the full suite,
reproduced with an add/remove stress loop). Database.connect(track=False)
leaves the watcher's short-lived connections to the watcher, which always
closes them on the thread that opened them. Regression test added.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
When the fingerprint shows no non-empty -wal or -journal the main file is
the whole database, so the sweep reads the schema cookie (offset 40 of
the header) with pread instead of opening a connection. No SHARED lock
means no busy back-off for a concurrent writer: polling was costing
write-heavy phases ~10% throughput and up to 2x p99. The fingerprint is
still taken first, so a commit in flight is covered by the racy rule.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
- write connections reopened after the watcher saw the file replaced or
deleted use mode=rw, so a write to a deleted database fails instead of
silently creating a new empty file
- derived_table_dependencies() keys its cache on the schema_version the
watcher stored in the catalog (plus its scan count) instead of running
PRAGMA schema_version on a pooled read connection - allowed_resources()
calls it for every database on every page
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
Per-database schema_watch modes (owned / external / immutable):
- write path reads PRAGMA schema_version after every write task and
rebuilds that one database's catalog before the write returns
- external files are polled with a stat() fingerprint prefilter
(racily-clean fingerprints re-verified), PRAGMA schema_version only on
change, rebuild only when the version moved or the inode changed
- immutable databases are scanned once
- remove_database deletes its catalog rows explicitly; startup deletes
rows for databases that are not attached (no per-refresh full scans)
- catalog reads use one short-lived connection per database inside one
read transaction; shared-cache memory databases go through their
write thread
- replaced/deleted files invalidate cached connections (generation)
- fingerprints persisted in catalog_databases.fingerprint (migration
0002) so restarts skip unchanged files
- new setting schema_watch_interval_ms (default 1000, 0 = no polling)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
New setting write_thread_idle_timeout_ms (default 30000, 0 disables).
A database's write thread now waits on queue.get(timeout=...) and, when
it times out, takes _write_thread_lock, re-checks that the queue is
empty, deregisters itself and closes its write connection. Producers
enqueue and check/start the thread under the same lock, so a task can
never be left in the queue with no thread to run it.
Also: close() refuses new tasks under the lock and joins retiring
threads; a failed connect is retried by the next write once the queue
drains; memory databases never idle out; a thread that dies from a
BaseException deregisters (and restarts if tasks are waiting); the
thread is registered only after start() succeeds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
Replace the per-thread cached read connections with a pool. Each
execute_fn()/execute() callback checks a connection out for its duration
and gets a LeasedConnection proxy that raises ConnectionLeaseError if
used after the callback returns.
New settings: max_open_connections (global LRU cap, default 128),
connection_idle_timeout (seconds, default 60, enforced by one
process-wide reaper thread plus opportunistic reaping on checkout),
connection_pool_wait_ms (soft cap: wait then exceed) and
pool_read_connections (off = fresh connection per query, a control).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
Attribute, SpanName and MetricName are str subclasses whose __new__
requires the metadata arguments, so copy.deepcopy could not reconstruct
one - it falls back to cls.__new__(cls) and raises TypeError.
That broke a real path rather than a theoretical one. The SDK's
ConsoleMetricExporter renders data points through dataclasses.asdict(),
which deepcopies mappings, and both core and kit-based plugins pass
registry entries as metric attribute keys - so every console metrics
dump crashed, core's own points included. Found by datasette-paper's
dev harness running opentelemetry-instrument with console exporters.
__reduce__ collapses copies to a plain str, which is what an entry is
everywhere except the registry module itself: the description, values
and buckets describe the single registered instance, and nothing reads
them off a copy. Pickle is fixed by the same change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JwU7BcTnAxUGSJYhBrQaY7
Outcome of a whole-stack review with the kit visible as one system:
- otel_reset: an autouse fixture draining the span exporter and metric
reader after every test. Without it a large suite accumulates hundreds
of thousands of recorded spans in the session-scoped exporter - the
likeliest amplifier of the slow-runner CI flakes - and plugins would
inherit the same leak.
- assert_registry_covered renamed to assert_spans_covered: the old name
read as covering the whole registry, which is exactly wrong next to
assert_metrics_covered. Public API is forever; renamed before anything
ships, no alias.
- The installers now verify their provider actually took: with a
provider installed first (opentelemetry-instrument, an embedding app),
set_*_provider() is silently ignored, and fixtures would assert
against an exporter wired to nothing. They skip clearly instead.
- UPDOWN_COUNTER registry kind, mapped to Sum with monotonicity checked
both ways - a Counter must collect monotonic, an UpDownCounter must
not. Previously an UpDownCounter's kind check was silently skipped.
- The docs page now prescribes naming: scope = import package name
(underscores), signal prefix = a name you own, never bare datasette.*;
its own examples no longer teach the hyphenated outlier. Plus an
observable-gauges pattern section and a prefix-overlap note.
- assert_no_forbidden_values(): the enforcement half of the privacy
rules - plant sentinel secrets in a workload and assert they never
appear in any span name, attribute, event, status description or
metric attribute, across all scopes by default.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012U7coQfVu8nK2R4q2mCULA
test_kit_module_itself_never_imports_the_sdk shells out, and like every
subprocess-spawning test in this suite it crashes the interpreter with
SIGBUS on macOS/CPython 3.13 when it runs late enough that the process
holds many threads - the exact failure conftest already front-loads
test_datasette_package_never_imports_the_sdk for. Move it to the front
too, and note the hazard in assert_package_never_imports_sdk's docstring
since plugin suites will call it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012U7coQfVu8nK2R4q2mCULA
The five surveyed plugin plans all kept a hand-rolled metrics-vs-registry
diff because the kit's conformance helpers covered spans only. This adds
the metric side:
- metric_for() in the registry (the span_for analogue - no prefix/dynamic
machinery, metric names are static), and the attribute helpers are
documented as accepting MetricName entries.
- MetricsCollector.collect() now retains the instrumentation scope per
collected metric, so a plugin is judged against its own meter only.
- assert_metrics_conform(): every collected metric in scope is registered,
was created as the instrument kind and unit its registry entry declares
(drift between the registry entry and the meter.create_*() call was
previously caught by nothing, in core or any plugin), sets only
registered attributes, and respects values= enums - the check that makes
a metric dimension provably bounded.
- assert_metrics_covered(): every registered metric collected at least
once with every non-optional attribute seen. Both *_covered helpers now
exempt optional=True attributes, so a workload is not forced to
manufacture every error path; pin those with targeted tests instead.
- datasette.operation declares values={"read", "write"} - core dogfoods
the enum enforcement on the dimension where it matters most.
- Core's generic metric conformance tests are now calls to the kit
helpers with scope_name="datasette"; the stricter literal-pinning and
optional-attribute-coverage tests stay hand-written on purpose.
- The metric reference docs render attributes through the same helper as
spans, so *(optional)* markers and enum values now appear there too.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012U7coQfVu8nK2R4q2mCULA
A survey of five plugin OTel plans (datasette-paper, -agent, -litestream,
-accounts, -cron) found every one hand-copying the same core machinery:
the registry classes, the conformance-test harness, the pytest fixtures,
the bucket boundaries and the detached-root-with-Link recipe. This makes
that machinery importable instead:
- The registry classes are documented public API. Attribute gains
values= (a closed enum the conformance helpers enforce - what makes an
attribute safe as a metric dimension); SpanName gains prefix=True for
span families like "chat {model}" whose names share a fixed prefix,
matched by span_for() after exact names. span_for()/attribute helpers
accept a spans= tuple so plugin registries can use them.
- datasette.telemetry.linked_root_span_kwargs(): the root-span-with-Link
shape for work a request caused without containing - background jobs,
scheduled ticks, block=False writes. Core's own write thread now uses
it instead of building the kwargs inline.
- datasette.telemetry_testing: the session provider fixtures, otel_spans
/ otel_metrics, a two-way registry conformance checker (including enum
and prefix handling, filtered by instrumentation scope) and an
assert_package_never_imports_sdk() guard. Core's conftest now imports
these instead of defining them, so the suite consumes the kit exactly
as a plugin's would.
- New "Telemetry for plugin authors" docs page: scope discipline,
registry usage, privacy/cardinality rules, named-callable guidance,
request_span(), the background root-with-link convention (one root per
tick, always emitted), provider-ordering facts and known caveats.
request_span() is now documented public API.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012U7coQfVu8nK2R4q2mCULA
The callback entry points gained db.query spans in the database-spans PR;
this adds their other half - the duration histogram measurement, so a
plugin's execute_fn/execute_write_fn work and the JSON write API's inserts
and deletes stop being invisible to the one series that survives trace
sampling. execute_isolated_fn records "write" when the database is mutable
(the call blocks the write queue) and "read" when immutable (it runs on
the read pool). error.type comes from the raised exception class, same as
the SQL-string paths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012U7coQfVu8nK2R4q2mCULA
- The exemplars docs described "the pinned opentelemetry-exporter-prometheus"
and "Datasette's own Prometheus exporter" - context from demo/plugin work
that is no longer part of this stack. Reworded to stand alone.
- Saturate a num_sql_threads=1 pool and assert the queue-depth gauge reads
above zero - the headline alerting metric previously only had an absence
test, and this also pins the private ThreadPoolExecutor._work_queue
attribute it depends on.
- Pin error.type on the write path of db.client.operation.duration - the
write wrappers time a different code path than the read one already tested.
- Isolate the non-threaded-mode gauge test from other live instances instead
of comparing global observation counts, which a GC pass could shift.
- Halve the metrics banner, point conftest's meter note at it, compact the
interrupted-counter call-site comment to a registry pointer, note why
instrument and registry descriptions are separate strings, and stop
calling the metric dimension a "later phase" now that metrics shipped.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012U7coQfVu8nK2R4q2mCULA
Span attributes were checked in both directions; metric attributes were not
checked at all, so the generated reference could publish an incomplete list
with nothing to catch it.
The metric workload lives in an `emitted_metrics` fixture, mirroring the
span side, and error.type is checked like every other attribute rather than
exempted for being optional - the workload reaches it two separate ways.
(Adapted from b30c5341: the old workload's facet-timeout probe belongs to
phase 5 and is dropped, and the interrupted counter now needs a query that
exceeds the *configured* time limit - custom short budgets are excluded from
the count on this lineage - so the fixture runs one against a second
instance configured with sql_time_limit_ms=5.)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F2h9ANGZ7paWSpqs5DUAcG
Both histograms declared unit="s" but inherited OpenTelemetry's default
boundaries, which are tuned for milliseconds - so every SQLite query
landed in the single (0, 5] second bucket and every quantile query
returned noise.
The boundaries are the semantic conventions' recommended set for
db.client.operation.duration plus 0.0001 and 0.0005 at the bottom, since
SQLite is in-process and many real queries take tens of microseconds.
(Adapted from 024f2029: that commit assumed the metrics were already in
telemetry_registry.py, which on this lineage held spans only - so this
commit also brings the MetricName registry machinery, the registry
entries for all eight phase-3 metrics, the cog-generated Metric
reference in internals.rst, and the datasette.operation attribute. The
template and facet histograms it also touched belong to phase 5 and are
not included.)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F2h9ANGZ7paWSpqs5DUAcG
Spans describe requests that have finished. They structurally cannot answer
"am I saturating my 3 SQL threads right now", because that is a level rather
than an event - and with num_sql_threads defaulting to 3, it is usually the
first thing worth knowing about a busy Datasette. This adds the metrics that
answer it.
Five observable gauges, computed only when something is collecting, so an
instance with no MeterProvider installed does no work for them at all:
datasette.sql.threads.limit num_sql_threads
datasette.sql.threads.queue_depth queries waiting for a free thread
datasette.sql.queries.pending in-flight reads, by db.namespace
datasette.write.queue_depth writes behind the single write thread
datasette.connections.open tracked file connections
Three instruments recorded inline, which matters because metrics survive
trace sampling and spans do not - an operator sampling 1% of traces still
gets 100% of the latency distribution:
db.client.operation.duration semconv histogram, with error.type
datasette.write.queue_wait the metric twin of the existing span
datasette.sql.queries.interrupted sql_time_limit_ms kills
The interrupted counter closes a gap the plan called out as unanswerable:
"how often are we killing queries at the limit" is a rate, and a rate cannot
be recovered from sampled spans.
Core still creates no provider of any kind, so the architecture is unchanged;
`grep -rn 'opentelemetry.sdk' datasette/` stays empty. One real difference
from tracing is worth recording: _ProxyMeter and its instruments forward to a
provider installed after they were created, whereas ProxyTracer permanently
caches the first concrete tracer it resolves. Module-level instruments are
therefore safe and the test fixture has no ordering constraint.
Live instances are tracked in a lock-guarded WeakSet so instrumenting an
instance never keeps it alive. The pool gauges carry no attribute saying
which Datasette produced them: production runs one instance per process, and
adding an id to disambiguate the test suite's hundreds of instances would buy
unbounded attribute cardinality to fix a case that does not occur. The
collision is documented instead, and the gauge callbacks are plain generator
functions so tests can assert exact values by calling them directly rather
than through the SDK's last-value aggregation.
demos/otel/metrics_demo.py fires 12 concurrent 40ms queries at a 3-thread
pool and samples the gauges mid-flight: queue_depth peaks at exactly 9, and
the duration histogram reads max=0.1695s for a query whose work is 40ms. That
gap is the queue, and it is the thing traces alone will not show you.
Also corrects the demo README's privacy section, which still claimed
parameter values are never recorded - that stopped being unconditionally true
when trace_sql_parameters landed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from 6ef0dd8c and adapted to the rebuilt phase-1 stack:
attribute names now come from telemetry_registry where entries exist, the
meter carries the instrumentation-scope version and schema URL, and the
interrupted-queries counter skips expected timeouts - callers that opted
into a deliberately short budget, like facet suggestion - matching how
those are excluded from span error status. The internals.rst reference
lands with the registry commit that follows.)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F2h9ANGZ7paWSpqs5DUAcG
