The five surveyed plugin plans all kept a hand-rolled metrics-vs-registry
diff because the kit's conformance helpers covered spans only. This adds
the metric side:
- metric_for() in the registry (the span_for analogue - no prefix/dynamic
machinery, metric names are static), and the attribute helpers are
documented as accepting MetricName entries.
- MetricsCollector.collect() now retains the instrumentation scope per
collected metric, so a plugin is judged against its own meter only.
- assert_metrics_conform(): every collected metric in scope is registered,
was created as the instrument kind and unit its registry entry declares
(drift between the registry entry and the meter.create_*() call was
previously caught by nothing, in core or any plugin), sets only
registered attributes, and respects values= enums - the check that makes
a metric dimension provably bounded.
- assert_metrics_covered(): every registered metric collected at least
once with every non-optional attribute seen. Both *_covered helpers now
exempt optional=True attributes, so a workload is not forced to
manufacture every error path; pin those with targeted tests instead.
- datasette.operation declares values={"read", "write"} - core dogfoods
the enum enforcement on the dimension where it matters most.
- Core's generic metric conformance tests are now calls to the kit
helpers with scope_name="datasette"; the stricter literal-pinning and
optional-attribute-coverage tests stay hand-written on purpose.
- The metric reference docs render attributes through the same helper as
spans, so *(optional)* markers and enum values now appear there too.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012U7coQfVu8nK2R4q2mCULA
A survey of five plugin OTel plans (datasette-paper, -agent, -litestream,
-accounts, -cron) found every one hand-copying the same core machinery:
the registry classes, the conformance-test harness, the pytest fixtures,
the bucket boundaries and the detached-root-with-Link recipe. This makes
that machinery importable instead:
- The registry classes are documented public API. Attribute gains
values= (a closed enum the conformance helpers enforce - what makes an
attribute safe as a metric dimension); SpanName gains prefix=True for
span families like "chat {model}" whose names share a fixed prefix,
matched by span_for() after exact names. span_for()/attribute helpers
accept a spans= tuple so plugin registries can use them.
- datasette.telemetry.linked_root_span_kwargs(): the root-span-with-Link
shape for work a request caused without containing - background jobs,
scheduled ticks, block=False writes. Core's own write thread now uses
it instead of building the kwargs inline.
- datasette.telemetry_testing: the session provider fixtures, otel_spans
/ otel_metrics, a two-way registry conformance checker (including enum
and prefix handling, filtered by instrumentation scope) and an
assert_package_never_imports_sdk() guard. Core's conftest now imports
these instead of defining them, so the suite consumes the kit exactly
as a plugin's would.
- New "Telemetry for plugin authors" docs page: scope discipline,
registry usage, privacy/cardinality rules, named-callable guidance,
request_span(), the background root-with-link convention (one root per
tick, always emitted), provider-ordering facts and known caveats.
request_span() is now documented public API.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012U7coQfVu8nK2R4q2mCULA
The callback entry points gained db.query spans in the database-spans PR;
this adds their other half - the duration histogram measurement, so a
plugin's execute_fn/execute_write_fn work and the JSON write API's inserts
and deletes stop being invisible to the one series that survives trace
sampling. execute_isolated_fn records "write" when the database is mutable
(the call blocks the write queue) and "read" when immutable (it runs on
the read pool). error.type comes from the raised exception class, same as
the SQL-string paths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012U7coQfVu8nK2R4q2mCULA
- The exemplars docs described "the pinned opentelemetry-exporter-prometheus"
and "Datasette's own Prometheus exporter" - context from demo/plugin work
that is no longer part of this stack. Reworded to stand alone.
- Saturate a num_sql_threads=1 pool and assert the queue-depth gauge reads
above zero - the headline alerting metric previously only had an absence
test, and this also pins the private ThreadPoolExecutor._work_queue
attribute it depends on.
- Pin error.type on the write path of db.client.operation.duration - the
write wrappers time a different code path than the read one already tested.
- Isolate the non-threaded-mode gauge test from other live instances instead
of comparing global observation counts, which a GC pass could shift.
- Halve the metrics banner, point conftest's meter note at it, compact the
interrupted-counter call-site comment to a registry pointer, note why
instrument and registry descriptions are separate strings, and stop
calling the metric dimension a "later phase" now that metrics shipped.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012U7coQfVu8nK2R4q2mCULA
Histograms recorded inside a sampled span carry trace IDs automatically, so a
latency spike links to a trace that caused it. Nothing said so.
Two things are documented because they were measured rather than assumed: an
exemplar is kept per histogram bucket, so the bucket boundaries fixed earlier
in this stack took the same workload from one reachable trace to four; and the
pinned opentelemetry-exporter-prometheus drops exemplars entirely, so the path
that works is an OTLP collector rather than Datasette's Prometheus exporter.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from 9d066255; section numbering and cross-references
adjusted to this branch's demo README, and the exemplar reference placed
as a subsection of the new Metric reference in internals.rst.)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F2h9ANGZ7paWSpqs5DUAcG
Both histograms declared unit="s" but inherited OpenTelemetry's default
boundaries, which are tuned for milliseconds - so every SQLite query
landed in the single (0, 5] second bucket and every quantile query
returned noise.
The boundaries are the semantic conventions' recommended set for
db.client.operation.duration plus 0.0001 and 0.0005 at the bottom, since
SQLite is in-process and many real queries take tens of microseconds.
(Adapted from 024f2029: that commit assumed the metrics were already in
telemetry_registry.py, which on this lineage held spans only - so this
commit also brings the MetricName registry machinery, the registry
entries for all eight phase-3 metrics, the cog-generated Metric
reference in internals.rst, and the datasette.operation attribute. The
template and facet histograms it also touched belong to phase 5 and are
not included.)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F2h9ANGZ7paWSpqs5DUAcG
Response.error(messages, status=400) builds a JSON error response in
Datasette's standard error format, alongside Response.json/html/text.
messages can be a single string or a list. All internal error response
construction now uses it - the private views.base._error() helper is
gone and the verbose Response.json(error_body(...), status=...) sites
are converted. error_body() remains for the cases that merge the error
keys into a larger payload (the JSON renderer, handle_exception and the
permission debug payload builders).
Since Response is public plugin API, plugins that build JSON endpoints
now have an obvious way to return errors in the canonical shape.
Documented in the internals documentation, including the guidance to
raise Forbidden/NotFound/BadRequest/DatasetteError instead when the
error should content-negotiate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ
Adds a per-request cache for permission check results, plus wiring that
resolves action permissions in bulk before plugin hooks need them:
- New _permission_check_cache contextvar, set to a fresh dict for each
request by DatasetteRouter and reset when the request ends. Keys
include the full serialized actor, so actors differing in any field
(e.g. token restrictions) never share entries. SkipPermissions mode
bypasses the cache entirely.
- datasette.allowed_many() now consults the cache and stores its
results there, so repeated datasette.allowed() checks within one
request resolve without further SQL.
- Table pages resolve all registered table-level actions against the
current table and all database-level actions against its database
(database pages likewise) in batched queries before invoking the
table_actions/database_actions plugin hooks - allowed() calls made
inside those hooks are then served from the cache with no plugin
changes required. Actions with no permission rules from any plugin
are resolved to False without touching the database.
Benchmarks (benchmarks/) with a simulated 12-plugin ecosystem making
18 checks per table page show 34 -> 13 internal-DB queries per page;
with 2ms-per-query internal DB latency (modelling Datasette Cloud)
table page time drops from 77.9ms to 27.6ms - the caching layer
accounts for ~91% of that improvement over allowed_many() alone.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix for execute write returning, closes#2762
* Fix stored write returning rowcount message
* Add configurable execute_write returning limit
* Return rows/truncated from execute query if it used RETURNING
* INSERT ... RETURNING shows rows in /-/execute-write
* Skip RETURNING tests if SQLite version does not support it
Screenshot: https://github.com/simonw/datasette/issues/2762#issuecomment-4588111545
Datasette.close() iterates over every attached Database (including the
internal database), calls Database.close() on each, then shuts down the
ThreadPoolExecutor. Exceptions raised by one Database don't prevent the
others from being closed; the first exception is re-raised afterwards.
Idempotent.
Refs #2692
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After this commit, Database.close() sends a sentinel to the write queue so
the background write thread exits cleanly, closes cached read/write
connections, and marks the instance closed. Subsequent calls to execute*()
raise DatasetteClosedError. close() remains idempotent and one-way.
Refs #2692
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`datasette.client.get(path, actor={"id": "root"}` now makes the internal request with that actor as `request.actor` - same for the other HTTP verb methods on `datasette.client`.
Upgraded relevant tests to use the new `actor=` mechanism.
- New CSRF protection middleware inspired by Go 1.25 and research by Filippo Valsorda - https://words.filippo.io/csrf/ - this replaces the old CSRF token based protection.
- Removes all instances of `<input type="hidden" name="csrftoken" value="{{ csrftoken() }}">` in the templates - they are no longer needed.
- Removes the `def skip_csrf(datasette, scope):` plugin hook defined in `datasette/hookspecs.py` and its documentation and tests.
- Updated CSRF protection documentation to describe the new approach.
- Upgrade guide now describes the CSRF change.
Closes#2683
* Add is_temp_disk option to Database for temp file-backed databases
Replace the default in-memory internal database with a temporary
file-backed database using WAL mode. This fixes concurrent read/write
locking errors that occur with named in-memory SQLite databases.
The new is_temp_disk parameter on Database creates a temp file via
tempfile.mkstemp, connects to it as a regular file-based database
with WAL mode enabled, and cleans it up on close() and via atexit.
https://claude.ai/code/session_01TteLrUjpDcARjnP1GMRqz2
I noticed that VACUUM can update the rootpage for tables in a way that
could confuse our rename table detection logic - but using the
execute_isolated_fn() method to run VACUUM avoids this problem.
Refs #2681
* Add track_event callback to execute_write_fn and write_wrapper
Allows write functions and write_wrapper generators to queue events
during a write operation that are dispatched after successful commit.
The fn or wrapper can optionally accept a `track_event` parameter
(detected via call_with_supported_arguments). Events are discarded
if the write raises an exception.
Does not yet handle the block=False (non-blocking) case - events
queued during non-blocking writes are currently silently discarded.
Refs https://github.com/simonw/datasette/issues/2681
* Dispatch track_event events for non-blocking (block=False) writes
Spawns a background asyncio task that awaits the write thread's reply
queue and dispatches pending events after a successful non-blocking
write. Events are still discarded if the write raises an exception.
Refs https://github.com/simonw/datasette/issues/2681
* Warn that events won't fire for other processes
Refs https://github.com/simonw/datasette/issues/2681#issuecomment-4157118662
* Document call_with_supported_arguments as a supported public API
Mark both call_with_supported_arguments and async_call_with_supported_arguments
with the @documented decorator and add documentation to docs/internals.rst
so plugin authors can use these dependency injection utilities in their own code.
https://claude.ai/code/session_01DKogZpHwzCTrbeG4XjXmNc
