The request span was created at the ASGI edge, before anything knew which
route would match, so it carried nothing but the method: every request in a
trace UI showed up as "GET", and the only URL on it was url.path, which is
unbounded on a public instance and useless as a grouping key. Routing
resolves in DatasetteRouter, so that is where the span gets http.route and
its semconv `{method} {route}` name.
http.route is the compiled route pattern, not a prettified
/{database}/{table} template. Datasette routes with compiled regexes and the
route table is fixed when the app is built, so the pattern is exact, bounded
and needs no parsing; the transform into something prettier accretes edge
cases, and Django's instrumentation ships regex-flavoured routes for the same
reason. A request that matches no route gets no http.route and keeps its bare
method name, which is what semantic conventions ask for.
Two things the obvious implementation gets wrong, both found by testing it:
- The router must not read `get_current_span()`. A plugin asgi_wrapper()
runs *inside* the request middleware, so an instrumented plugin makes its
own span current for the whole request - and the route then lands on that
plugin's INTERNAL span, renaming it, while the actual request span never
gets the one attribute a trace UI groups by. It reproduces with a five-line
plugin. The span is passed through the ASGI scope instead, falling back to
the current span so an externally-created SERVER span is still enriched.
- The method has to be clamped again here. The middleware clamps it for the
attribute, but the name is rebuilt from request.method, which is the raw
client string - so an unclamped rename put `FROB /(?P<database>...` back
into the span name that the middleware had just kept it out of.
Both guards are `is_recording()`, not `get_span_context().is_valid`: with no
provider but an inbound traceparent the API returns a NonRecordingSpan
carrying the remote context, which is valid and records nothing, so an
is_valid guard would do the work on every request from a traced caller.
Tests cover the route and name, the unrouted 404 fallback, the full attribute
set, db.query spans reaching the request span by parent walk, a 500, an
inbound traceparent becoming a remote parent, ?sql= never reaching a span
attribute, and - in a subprocess, because the suite's provider fixture is
session-scoped and unavoidable - the no-provider fast path handing the app
the original `send`. The streaming test uses a table larger than one page so
the export genuinely issues queries during the body send; without that it
passes however early the span ends.
Measured on this branch against fixtures.db: a faceted table page went from
112 spans in 56 traces to 113 spans in 1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nothing in Datasette created a span for the HTTP request itself, so every
span the database layer emits was a root span. Measured on this branch: one
faceted table page produces 70 spans in 36 separate traces, none of which
carries a URL. A trace UI shows that as dozens of unrelated single-span
traces per page, interleaved across concurrent requests - worse than
?_trace=1 at the exact job people reach for tracing to do. With the request
span it is 71 spans in 1 trace.
`opentelemetry-instrument` does not fix this on its own: auto-instrumentation
only picks up frameworks that ship an instrumentor entry point, and
Datasette's raw ASGI app is not one.
TelemetryMiddleware is mounted outermost in Datasette.app(), after the
asgi_wrapper() plugin loop, so plugin middleware and the CSRF layer run
*inside* the span. Putting it in DatasetteRouter instead would leave a span
created by an instrumented plugin as an orphan root - reintroducing the
problem for exactly the code most likely to be instrumented.
It stays at ~90 lines, against roughly 700 for
opentelemetry-instrumentation-asgi, because Datasette's app does not return
before its body is sent: route_path awaits response.asgi_send(send), and a
streaming CSV export runs its generator inline inside AsgiStream.asgi_send.
So a plain `finally` covers the response body and no deferred-end machinery
is needed.
Two decisions worth flagging for review:
- Inbound W3C traceparent and baggage are extracted, using the *global*
propagator. That is the ecosystem norm (Flask, Django, FastAPI, the ASGI
instrumentation), and going through the global propagator leaves the
operator in control with no Datasette setting to invent:
OTEL_PROPAGATORS=none disables it entirely. A public instance that does
not want client-influenced traces should strip those headers at the proxy.
- url.query is not recorded, anywhere. Datasette query strings carry
user-supplied SQL in ?sql= and canned query parameters. client.address is
not recorded either.
The status code is sniffed from the ASGI http.response.start message rather
than read off a Response, because asgi_static, the favicon route, AsgiStream
and AsgiFileDownload all send that message themselves and never build one.
Only a >= 500 sets an error status - per semantic conventions a 4xx is the
client's mistake, and Datasette 404s are routine enough that treating them
as errors would bury a real 500.
The registry gains a `dynamic` flag, because this span's name is composed at
runtime and so can never equal a fixed registry string. Dynamic entries
resolve by span kind instead, and only after exact and prefix matching has
failed, so they cannot shadow a span that does have a registered name.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Simon's review: `table=` was a new parameter on the public `execute()`
signature that had no effect on execution - it existed only to set the
`db.collection.name` span attribute.
Removing it takes the whole attribute out of phase 1. `db.query` spans
keep `db.namespace`, `db.query.text`, `db.operation.name` and the rest;
`db.collection.name` returns in a later PR once there is a mechanism
worth committing to in the public API.
Side effect worth having: this PR no longer touches `views/table.py` or
`views/row.py` at all - both files are now byte-identical to main - so
it is purely the database layer it claims to be.
The registry conformance tests already enforce the rest: an attribute
left in `telemetry_registry` but never emitted fails
`test_every_registered_attribute_is_emitted`, so the registry entry, the
generated docs block and the workload that reached it all come out
together.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The database instrumentation covered the four SQL-string entry points but
not the callback entry points, which are the documented way for plugins to
run arbitrary SQL - so the JSON write API's inserts and deletes, the
startup catalog scan, and every plugin built on execute_fn/execute_write_fn
were invisible to a trace, or worse, showed orphan-looking db.write.* spans
with no db.query above them.
Each callback method now opens the same db.query CLIENT span as its
SQL-string sibling, carrying a new optional datasette.callback attribute
(the callable's qualified name, captured before _wrap_fn_with_hooks() can
rename it) in place of db.query.text, which is now marked optional. A bare
execute_fn() also wraps the callback in a db.query.execute child, so the
"gap between the spans is thread-wait" story holds for plugin callbacks
too. No db.operation.name: there is no statement to take a keyword from,
and the registry says that attribute is omitted rather than guessed.
The previous bodies move to private _execute_fn()/_execute_write_fn() and
the SQL-string methods call those, so an execute() emits exactly the spans
it did before - pinned by test_execute_does_not_double_wrap. Database's own
introspection helpers stay on the public method deliberately: they are real
SQLite round trips, which lifts a table page from ~58 to ~100 (no-op) spans.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012U7coQfVu8nK2R4q2mCULA
- Remove the registry's unused prefix=True slot, its span_for() branch,
its doc-rendering case and its test - nothing in the stack sets it.
- Stop promising a "later phase" query-duration metric dimension in the
db.operation.name description; the cardinality rationale stands alone.
- Replace baked-in benchmark numbers in the telemetry module docstring
with the docs' own phrasing (below run-to-run variation).
- Compact the duplicated copy_context() and enqueue-site comments in
database.py to pointers at their canonical tellings.
- Make the "catch people out" gotchas skimmable as a bullet list and
give the changelog's "nothing is removed" line a clear antecedent.
- Add a test that a result cut short by max_returned_rows records
datasette.truncated=True - previously only ever asserted False.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012U7coQfVu8nK2R4q2mCULA
The span reference itself is generated from the registry, so this adds the
prose the generated list cannot supply: how to actually see a span, what is
deliberately never recorded, and where the instrumentation stops short.
The "how to turn it on" part is the part people get wrong. Core installs no
provider, so OTEL_TRACES_EXPORTER=console against a plain `datasette` process
emits nothing at all - that variable is read by the SDK auto-configuration
which only runs under `opentelemetry-instrument`. Documented as a warning
because it reads like a bug when you hit it. Two more measured facts get the
same treatment: the SDK's BatchSpanProcessor default schedule delay is 5000ms
(checked, not assumed - `BatchSpanProcessor._default_schedule_delay_millis()`
on opentelemetry-sdk 1.44), so nothing appears for five seconds; and without
OTEL_SERVICE_NAME the default resource reports service.name=unknown_service.
Privacy properties are stated positively rather than left implicit: SQL
truncated at 2048 characters, parameter values never recorded, no actor
identifiers, table names only from an explicit `table=` argument. The last of
those is now documented on db.execute() itself, since it is public API.
The limitations section claims only what was measured. An earlier draft said
two traces per process are orphaned by the register_output_renderer and
asgi_wrapper hooks; measuring it showed a default install emits zero spans
from either, because Datasette queries no database there - it is a plugin
that would produce the orphan. Corrected to say that.
It also deliberately does NOT say an embedder must install its provider
before Datasette's first span or get nothing. That claim is false:
ProxyTracer._tracer returns the no-op tracer without caching it when no
provider is set, so early spans are dropped and nothing is poisoned.
The telemetry.py docstring said no-op spans "cost approximately nothing".
The benchmark for this diff does not support a claim that strong - a table
page emits ~58 spans - so it now states the measurement instead: median
9.80ms to 9.98ms across 15 runs, inside a 1.4ms run-to-run spread.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Datasette has a family of callers that run a query under a tiny time limit
and treat "did not finish" as a usable answer. table_counts() is the loudest:
the homepage counts every table with a 10ms budget and stores None for the
ones that blow it. The QueryInterrupted handler on the db.query span was
unconditional, so on a two-table database that produced four ERROR spans -
two db.query and two db.query.execute - on every homepage hit. Measured on a
30MB two-table database: 4 red spans before, 0 after.
Honouring log_sql_errors here would have silenced none of it. Only the three
ArrayFacet json_type() probes pass log_sql_errors=False, and they are not the
queries that time out; table_counts() and ColumnFacet.suggest both leave it at
its True default. The signal that does separate the two cases is the budget
itself: a caller asking for less time than sql_time_limit_ms is saying the
query may not finish. Keying off that needs no new API and no changes outside
database.py. A query that runs out the instance-wide limit is still an error.
datasette.interrupted is still set in every case - it is the signal worth
having, and only the ERROR status becomes conditional. Its registry
description said the status is "also set to ERROR" full stop, which is now
wrong, and that string is published in docs/internals.rst.
The inner db.query.execute span carried the same bug through
set_status_on_exception=log_sql_errors, so its exception handling is now
explicit, matching the db.query span above it. The context manager's flags
apply to every exception type alike and this span has to tell two apart.
test_query_interrupted_sets_error_status forced its timeout with
?_timelimit=5, which is exactly the signal now reclassified as expected. It
now forces one via sql_time_limit_ms so it still tests what it was written to
test.
Also documents, at the copy_context() sites, that context propagation carries
Datasette's non-OTel ContextVars into worker threads too. Verified harmless:
nothing reads _skip_permission_checks, _permission_check_cache or
_in_datasette_client off the event loop, and Context.run() restores the
thread's previous context on return, so no value can reach the next task on
the shared pool.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The span and attribute names were string literals spread across four call
sites in database.py and one in app.py, with a hand-written reference page
that would have been true only on the day it was written. That drift is not
hypothetical: an earlier iteration of this work carried a README asserting
parameter values were never recorded for two branches after that had stopped
being true.
datasette/telemetry_registry.py now holds each name once, with its
documentation. Attribute and SpanName subclass str, so a registry entry *is*
the string OpenTelemetry wants - no wrapper API over the OTel calls, no
parallel structure to keep in step, and a typo becomes an ImportError rather
than a silently misnamed attribute. docs/internals.rst renders the span
reference from it via cog, and `cog --check docs/*.rst` already runs in CI,
so the reference cannot drift from the definitions.
Nothing changes on the wire: the emitted span names and attribute keys are
byte-identical before and after, verified by diffing a dump of both.
tests/test_telemetry_registry.py exercises a real workload and compares it
against the registry in both directions - emitted-but-unregistered catches
instrumentation added without documentation, registered-but-never-emitted
catches documentation that has outlived its code. Because the call sites now
take their names from the registry, neither direction can catch a rename:
move DB_NAMESPACE to "db.namespace2" and code and registry still agree while
every dashboard breaks. So the literal names are also written out in the test
and asserted against the registry and against the wire separately. That pair
is the only comparison in the file not derived from the registry itself.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
www.starlette.io and www.uvicorn.org no longer resolve (NXDOMAIN); both
projects now publish their documentation at starlette.dev and uvicorn.dev
respectively. The Docker Hub link was missing the /r/ path segment, which
returns a 404.
- docs/plugin_hooks.rst: www.starlette.io/middleware/ -> starlette.dev/middleware/
- docs/changelog.rst (x3): www.uvicorn.org/ -> uvicorn.dev/
- docs/changelog.rst: hub.docker.com/datasetteproject/... -> hub.docker.com/r/datasetteproject/...
All four replacement URLs verified to return HTTP 200.
