A survey of five plugin OTel plans (datasette-paper, -agent, -litestream,
-accounts, -cron) found every one hand-copying the same core machinery:
the registry classes, the conformance-test harness, the pytest fixtures,
the bucket boundaries and the detached-root-with-Link recipe. This makes
that machinery importable instead:
- The registry classes are documented public API. Attribute gains
values= (a closed enum the conformance helpers enforce - what makes an
attribute safe as a metric dimension); SpanName gains prefix=True for
span families like "chat {model}" whose names share a fixed prefix,
matched by span_for() after exact names. span_for()/attribute helpers
accept a spans= tuple so plugin registries can use them.
- datasette.telemetry.linked_root_span_kwargs(): the root-span-with-Link
shape for work a request caused without containing - background jobs,
scheduled ticks, block=False writes. Core's own write thread now uses
it instead of building the kwargs inline.
- datasette.telemetry_testing: the session provider fixtures, otel_spans
/ otel_metrics, a two-way registry conformance checker (including enum
and prefix handling, filtered by instrumentation scope) and an
assert_package_never_imports_sdk() guard. Core's conftest now imports
these instead of defining them, so the suite consumes the kit exactly
as a plugin's would.
- New "Telemetry for plugin authors" docs page: scope discipline,
registry usage, privacy/cardinality rules, named-callable guidance,
request_span(), the background root-with-link convention (one root per
tick, always emitted), provider-ordering facts and known caveats.
request_span() is now documented public API.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012U7coQfVu8nK2R4q2mCULA
The callback entry points gained db.query spans in the database-spans PR;
this adds their other half - the duration histogram measurement, so a
plugin's execute_fn/execute_write_fn work and the JSON write API's inserts
and deletes stop being invisible to the one series that survives trace
sampling. execute_isolated_fn records "write" when the database is mutable
(the call blocks the write queue) and "read" when immutable (it runs on
the read pool). error.type comes from the raised exception class, same as
the SQL-string paths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012U7coQfVu8nK2R4q2mCULA
- The exemplars docs described "the pinned opentelemetry-exporter-prometheus"
and "Datasette's own Prometheus exporter" - context from demo/plugin work
that is no longer part of this stack. Reworded to stand alone.
- Saturate a num_sql_threads=1 pool and assert the queue-depth gauge reads
above zero - the headline alerting metric previously only had an absence
test, and this also pins the private ThreadPoolExecutor._work_queue
attribute it depends on.
- Pin error.type on the write path of db.client.operation.duration - the
write wrappers time a different code path than the read one already tested.
- Isolate the non-threaded-mode gauge test from other live instances instead
of comparing global observation counts, which a GC pass could shift.
- Halve the metrics banner, point conftest's meter note at it, compact the
interrupted-counter call-site comment to a registry pointer, note why
instrument and registry descriptions are separate strings, and stop
calling the metric dimension a "later phase" now that metrics shipped.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012U7coQfVu8nK2R4q2mCULA
Spans describe requests that have finished. They structurally cannot answer
"am I saturating my 3 SQL threads right now", because that is a level rather
than an event - and with num_sql_threads defaulting to 3, it is usually the
first thing worth knowing about a busy Datasette. This adds the metrics that
answer it.
Five observable gauges, computed only when something is collecting, so an
instance with no MeterProvider installed does no work for them at all:
datasette.sql.threads.limit num_sql_threads
datasette.sql.threads.queue_depth queries waiting for a free thread
datasette.sql.queries.pending in-flight reads, by db.namespace
datasette.write.queue_depth writes behind the single write thread
datasette.connections.open tracked file connections
Three instruments recorded inline, which matters because metrics survive
trace sampling and spans do not - an operator sampling 1% of traces still
gets 100% of the latency distribution:
db.client.operation.duration semconv histogram, with error.type
datasette.write.queue_wait the metric twin of the existing span
datasette.sql.queries.interrupted sql_time_limit_ms kills
The interrupted counter closes a gap the plan called out as unanswerable:
"how often are we killing queries at the limit" is a rate, and a rate cannot
be recovered from sampled spans.
Core still creates no provider of any kind, so the architecture is unchanged;
`grep -rn 'opentelemetry.sdk' datasette/` stays empty. One real difference
from tracing is worth recording: _ProxyMeter and its instruments forward to a
provider installed after they were created, whereas ProxyTracer permanently
caches the first concrete tracer it resolves. Module-level instruments are
therefore safe and the test fixture has no ordering constraint.
Live instances are tracked in a lock-guarded WeakSet so instrumenting an
instance never keeps it alive. The pool gauges carry no attribute saying
which Datasette produced them: production runs one instance per process, and
adding an id to disambiguate the test suite's hundreds of instances would buy
unbounded attribute cardinality to fix a case that does not occur. The
collision is documented instead, and the gauge callbacks are plain generator
functions so tests can assert exact values by calling them directly rather
than through the SDK's last-value aggregation.
demos/otel/metrics_demo.py fires 12 concurrent 40ms queries at a 3-thread
pool and samples the gauges mid-flight: queue_depth peaks at exactly 9, and
the duration histogram reads max=0.1695s for a query whose work is 40ms. That
gap is the queue, and it is the thing traces alone will not show you.
Also corrects the demo README's privacy section, which still claimed
parameter values are never recorded - that stopped being unconditionally true
when trace_sql_parameters landed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from 6ef0dd8c and adapted to the rebuilt phase-1 stack:
attribute names now come from telemetry_registry where entries exist, the
meter carries the instrumentation-scope version and schema URL, and the
interrupted-queries counter skips expected timeouts - callers that opted
into a deliberately short budget, like facet suggestion - matching how
those are excluded from span error status. The internals.rst reference
lands with the registry commit that follows.)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F2h9ANGZ7paWSpqs5DUAcG
- Close in-memory database connections, including reads opened on threads
- Close completed and partial file upload file handles
- Close SQLite connections and file handles owned by tests
execute_write_fn(fn, block=False) is documented to return "a UUID representing
the queued task". Two things stopped that being true.
_send_to_write_thread() derived the id from uuid.uuid5(NAMESPACE_DNS,
"datasette.io"), which is deterministic, so every non-blocking write in every
database in every process returned 3f143baa-4e3d-5842-a36f-4fa2f683b72f. A
constant cannot identify a particular task. Now uuid4(). Refs #2860.
With num_sql_threads=0 there is no write thread, so execute_write_fn took the
synchronous branch and `result` was the write function's return value, normally
None. The block=False path then unpacked it unconditionally and raised
TypeError: cannot unpack non-iterable NoneType object. The non-threaded branch
now returns the same (task_id, reply_future) shape, with the future already
resolved because the write has finished, so both modes share one code path.
Refs #2859.
test_execute_write_fn_block_false only asserted isinstance(task_id, uuid.UUID),
which a constant satisfies. The new test is parametrized over threaded and
non-threaded and asserts two calls return different ids, so either regression
fails it.
Several internal helpers quoted table names using SQLite [bracket]
identifiers built with an f-string, e.g. PRAGMA foreign_key_list([{table}]).
Bracket quoting cannot escape a "]" character, so any table whose name
contains "]" (for example "[foo]" or "foo]") produced
"sqlite3.OperationalError: unrecognized token" - crashing schema
introspection at startup and 500-ing the table page.
Switch these call sites to the existing escape_sqlite() helper, which uses
"double quote" quoting with correct "" escaping (the same approach already
used elsewhere in the codebase and in the test suite):
- utils/internal_db.py: PRAGMA foreign_key_list / index_list
- utils/__init__.py: get_outbound_foreign_keys
- database.py: table_counts count query
- facets.py: default "select * from" SQL
Added a regression test covering table names with "]" characters.
Co-authored-by: Claude <noreply@anthropic.com>
execute_isolated_fn() always opened its temporary connection with
write=True, which is not allowed for immutable databases - so APIs
that rely on it, like SQL analysis when storing a query, failed.
An immutable database can never receive writes, so there is no write
queue to block: in that case the function now opens a read-only
connection and runs it on the executor, bypassing the write thread
entirely. Mutable databases keep the existing write-thread behavior.
Also fixed a latent bug in the write thread where a connect() failure
for an isolated task would crash the thread instead of delivering the
exception back to the caller.
Closes#2768
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix for execute write returning, closes#2762
* Fix stored write returning rowcount message
* Add configurable execute_write returning limit
* Return rows/truncated from execute query if it used RETURNING
* INSERT ... RETURNING shows rows in /-/execute-write
* Skip RETURNING tests if SQLite version does not support it
Screenshot: https://github.com/simonw/datasette/issues/2762#issuecomment-4588111545
Experimental, we may need this for the upcoming canned query
work so that we can tell if a user should be able to save
a writable canned query by confirming they have the right
permissions to update the affected tables.
Refs #2735
Closes:
- #2709
The key behavior change: after close() starts, no new execute work can be submitted, but already-running execute work is allowed to finish before SQLite connections are closed.
After this commit, Database.close() sends a sentinel to the write queue so
the background write thread exits cleanly, closes cached read/write
connections, and marks the instance closed. Subsequent calls to execute*()
raise DatasetteClosedError. close() remains idempotent and one-way.
Refs #2692
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes#2683
* Add is_temp_disk option to Database for temp file-backed databases
Replace the default in-memory internal database with a temporary
file-backed database using WAL mode. This fixes concurrent read/write
locking errors that occur with named in-memory SQLite databases.
The new is_temp_disk parameter on Database creates a temp file via
tempfile.mkstemp, connects to it as a regular file-based database
with WAL mode enabled, and cleans it up on close() and via atexit.
https://claude.ai/code/session_01TteLrUjpDcARjnP1GMRqz2
* Add track_event callback to execute_write_fn and write_wrapper
Allows write functions and write_wrapper generators to queue events
during a write operation that are dispatched after successful commit.
The fn or wrapper can optionally accept a `track_event` parameter
(detected via call_with_supported_arguments). Events are discarded
if the write raises an exception.
Does not yet handle the block=False (non-blocking) case - events
queued during non-blocking writes are currently silently discarded.
Refs https://github.com/simonw/datasette/issues/2681
* Dispatch track_event events for non-blocking (block=False) writes
Spawns a background asyncio task that awaits the write thread's reply
queue and dispatches pending events after a successful non-blocking
write. Events are still discarded if the write raises an exception.
Refs https://github.com/simonw/datasette/issues/2681
* Warn that events won't fire for other processes
Refs https://github.com/simonw/datasette/issues/2681#issuecomment-4157118662
* Implement write_wrapper plugin hook for intercepting database writes
Add a new `write_wrapper` plugin hook that lets plugins wrap write
operations with before/after logic using a generator-based context
manager pattern. The hook receives (datasette, database, request,
transaction) and returns a generator function that takes a conn,
yields once to let the write execute, and can run cleanup after.
The write result is sent back via `generator.send()` and exceptions
are thrown via `generator.throw()`, giving plugins full visibility.
Also adds `request=None` parameter to execute_write, execute_write_fn,
execute_write_script, and execute_write_many, and threads request
through all view-layer call sites (insert, upsert, update, delete,
drop, create table, canned queries).
* Add documentation for wrap_write hook, fix lint issues
Document the wrap_write plugin hook in plugin_hooks.rst with
parameter descriptions and two examples: a simple logging wrapper
and an advanced SQLite authorizer-based table protection pattern.
Also fix black formatting and remove unused variable flagged by ruff.
* Rename wrap_write hook to write_wrapper for consistency with asgi_wrapper
* Move write_wrapper docs to just below prepare_connection
* Refactor write_wrapper tests to use pytest.parametrize
Consolidate duplicate test cases: merge before/after tests for
execute_write_fn and execute_write into one parametrized test, and
merge three parameter-passing tests into one parametrized test.
Claude Code transcript: https://gisthost.github.io/?c4c12079434e69677e4aa8ac664b21b8/index.html
* Fix flaky test_database_page test with deterministic ordering
- Add ORDER BY to table_names() query in database.py
- Sort foreign keys deterministically in get_all_foreign_keys()
- Refactor test_database_page to use property-based assertions instead of
500+ lines of hardcoded expected data
- Run blacken-docs on plugin_hooks.rst
* Update test_row_foreign_key_tables for new deterministic FK ordering
The foreign keys are now sorted by (other_table, column, other_column),
so complex_foreign_keys comes before foreign_key_references alphabetically.
* Update test_table_names for new alphabetical ordering
The table_names() method now returns tables sorted alphabetically.
* Fix for test that fails prior to SQLite 3.37
---------
Co-authored-by: Claude <noreply@anthropic.com>
The execute_write_script() method was causing SQLite database locking
errors when multiple executescript() calls ran in quick succession.
Root cause: SQLite's executescript() method has special behavior - it
implicitly commits any pending transaction and operates in autocommit
mode. However, execute_write_script() was passing these calls through
execute_write_fn() with the default transaction=True, which wrapped
the executescript() call in a transaction context (with conn:).
This created a conflict where sequential executescript() calls would
cause the second call to fail with "OperationalError: database table
is locked: sqlite_master" because the sqlite_master table was still
locked from the first operation's implicit commit.
Fix: Pass transaction=False to execute_write_fn() since executescript()
manages its own transactions and should not be wrapped in an additional
transaction context.
This was causing test_hook_extra_body_script to fail because the
internal database initialization (which calls executescript twice in
succession) would fail, preventing the application from rendering
pages correctly.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Table configuration that was incorrectly placed in metadata is now treated as if it was in config.
New await datasette.table_config() method.
Closes#2247
