remote/datasette
1
0
Fork 0
mirror of https://github.com/simonw/datasette.git synced 2026-09-02 22:54:08 +02:00
Code Issues Projects Releases Packages Wiki Activity

Name the request span after the route it matched

Browse source 
The request span was created at the ASGI edge, before anything knew which
route would match, so it carried nothing but the method: every request in a
trace UI showed up as "GET", and the only URL on it was url.path, which is
unbounded on a public instance and useless as a grouping key. Routing
resolves in DatasetteRouter, so that is where the span gets http.route and
its semconv `{method} {route}` name.

http.route is the compiled route pattern, not a prettified
/{database}/{table} template. Datasette routes with compiled regexes and the
route table is fixed when the app is built, so the pattern is exact, bounded
and needs no parsing; the transform into something prettier accretes edge
cases, and Django's instrumentation ships regex-flavoured routes for the same
reason. A request that matches no route gets no http.route and keeps its bare
method name, which is what semantic conventions ask for.

Two things the obvious implementation gets wrong, both found by testing it:

- The router must not read `get_current_span()`. A plugin asgi_wrapper()
  runs *inside* the request middleware, so an instrumented plugin makes its
  own span current for the whole request - and the route then lands on that
  plugin's INTERNAL span, renaming it, while the actual request span never
  gets the one attribute a trace UI groups by. It reproduces with a five-line
  plugin. The span is passed through the ASGI scope instead, falling back to
  the current span so an externally-created SERVER span is still enriched.

- The method has to be clamped again here. The middleware clamps it for the
  attribute, but the name is rebuilt from request.method, which is the raw
  client string - so an unclamped rename put `FROB /(?P<database>...` back
  into the span name that the middleware had just kept it out of.

Both guards are `is_recording()`, not `get_span_context().is_valid`: with no
provider but an inbound traceparent the API returns a NonRecordingSpan
carrying the remote context, which is valid and records nothing, so an
is_valid guard would do the work on every request from a traced caller.

Tests cover the route and name, the unrouted 404 fallback, the full attribute
set, db.query spans reaching the request span by parent walk, a 500, an
inbound traceparent becoming a remote parent, ?sql= never reaching a span
attribute, and - in a subprocess, because the suite's provider fixture is
session-scoped and unavoidable - the no-provider fast path handing the app
the original `send`. The streaming test uses a table larger than one page so
the export genuinely issues queries during the body send; without that it
passes however early the span ends.

Measured on this branch against fixtures.db: a faceted table page went from
112 spans in 56 traces to 113 spans in 1.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Alex Garcia 2026-07-30 19:52:12 -07:00
commit 74cd01860f
7 changed files with 681 additions and 41 deletions
Download patch file Download diff file Expand all files Collapse all files

27
datasette/app.py
View file

@ -49,8 +49,13 @@ from .events import Event
from .plugins import DEFAULT_PLUGINS, get_plugins, pm
from .renderer import json_renderer
from .resources import DatabaseResource, TableResource
from .telemetry import TelemetryMiddleware, tracer
from .telemetry_registry import STARTUP
from .telemetry import (
TelemetryMiddleware,
clamp_http_method,
request_span,
tracer,
)
from .telemetry_registry import HTTP_ROUTE, STARTUP
from .tokens import TokenInvalid
from .tracer import AsgiTracer
from .url_builder import Urls
@ -2958,8 +2963,26 @@ class DatasetteRouter:
match, view = resolve_routes(self.routes, path)
if match is None:
# No route matched, so the span keeps the bare method name it was
# given at the edge and gets no http.route. That is what semantic
# conventions ask for when the route is unknown.
return await self.handle_404(request, send)
# The request span was started at the ASGI edge, before routing, so it
# carries only the method as a name. Now that the route is known, give
# it the `{method} {route}` shape semantic conventions want, and the
# http.route attribute - the low-cardinality counterpart to url.path,
# and so the one to group by.
span = request_span(scope)
if span is not None:
route = match.re.pattern
span.set_attribute(HTTP_ROUTE, route)
# Clamped, for the same reason the middleware clamps it: the method
# is a client-controlled string, and an unclamped one here would
# put attacker-supplied text back into the span name that the
# middleware just kept out of it.
span.update_name(f"{clamp_http_method(request.method)} {route}")
new_scope = dict(scope, url_route={"kwargs": match.groupdict()})
request.scope = new_scope
try:

36
datasette/telemetry.py
View file

@ -201,6 +201,38 @@ def _url_path(scope):
return scope.get("path", "")
# The request span is handed to `DatasetteRouter.route_path` through the ASGI
# scope rather than through `get_current_span()`, because by the time routing
# happens the current span may well be something else: a plugin
# `asgi_wrapper()` runs *inside* this middleware, and an instrumented one makes
# its own span current for the whole request. Reading the current span there
# would set `http.route` on that plugin's span - and rename it - while leaving
# the actual request span without the one attribute a trace UI groups by. Not
# hypothetical: an ordinary tracing plugin triggers it.
#
# Namespaced per the ASGI spec's rules for extension keys. Absent when the span
# is not recording, which is exactly when the router should skip the work too.
REQUEST_SPAN_SCOPE_KEY = "datasette.telemetry.request_span"
def request_span(scope):
"""
The recording request span for an ASGI scope, or None.
Falls back to the current span so that a `DatasetteRouter` running under
some other instrumentation - one that started a SERVER span but of course
knows nothing about this scope key - still gets enriched.
"""
span = scope.get(REQUEST_SPAN_SCOPE_KEY)
if span is None:
span = otel_trace.get_current_span()
# is_recording(), not `get_span_context().is_valid`: with no provider but
# an inbound `traceparent`, the API's NoOpTracer hands back a
# NonRecordingSpan carrying the *remote* context, which is perfectly valid
# and still records nothing.
return span if span.is_recording() else None
class TelemetryMiddleware:
"""
One `SpanKind.SERVER` span per HTTP request.
@ -265,6 +297,10 @@ class TelemetryMiddleware:
if user_agent:
span.set_attribute(USER_AGENT_ORIGINAL, user_agent)
# A copy, not a mutation: the scope belongs to the server, and
# every other layer in Datasette extends it the same way.
scope = dict(scope, **{REQUEST_SPAN_SCOPE_KEY: span})
# The status cannot be read off a Response object: `asgi_static`,
# the favicon route, `AsgiStream` and `AsgiFileDownload` all call
# `send` directly and never build one. Wrapping `send` is the only

31
datasette/telemetry_registry.py
View file

@ -67,10 +67,11 @@ class SpanName(str):
self.prefix = prefix
# True when the emitted name is composed at runtime and shares no
# fixed prefix with the registry entry - the HTTP request span, whose
# name is the request method. There is no substring of the entry that
# could be matched against the wire, so `span_for()` resolves these by
# span kind instead, and the entry's own string is a template written
# for a human reading the generated reference.
# name is the request method followed by the matched route. There is
# no substring of the entry that could be matched against the wire, so
# `span_for()` resolves these by span kind instead, and the entry's own
# string is a template written for a human reading the generated
# reference.
self.dynamic = dynamic
# SpanKind.INTERNAL by default - every span Datasette emits describes
# its own internal work. db.query is the one exception: it is a real
@ -104,6 +105,20 @@ HTTP_RESPONSE_STATUS_CODE = Attribute(
"closed before anything was sent.",
optional=True,
)
HTTP_ROUTE = Attribute(
"http.route",
"The route the request matched, as the compiled regular expression "
"pattern Datasette routes with - for example "
"``/(?P<database>[^\\/\\.]+)/(?P<table>[^\\/\\.]+)(\\.(?P<format>\\w+))?$`` "
"for a table page. It is deliberately the pattern rather than a prettified "
"``/{database}/{table}`` template: the route table is fixed when the app "
"is built, so the pattern is exact, bounded and needs no parsing, whereas "
"the transform into something prettier accretes edge cases. Unlike "
"``url.path`` this is low cardinality, so it is the attribute to group by. "
"Omitted when no route matched - a 404 - which is also when the span name "
"falls back to the bare method.",
optional=True,
)
URL_PATH = Attribute(
"url.path",
"The path portion of the URL. The query string is deliberately **not** "
@ -236,18 +251,22 @@ TRANSACTION = Attribute(
# --- Spans ----------------------------------------------------------------
HTTP_REQUEST = SpanName(
"{http.request.method}",
"{http.request.method} {http.route}",
"One span per HTTP request, created by the outermost layer of the ASGI "
"stack - so plugin ``asgi_wrapper()`` middleware, CSRF protection and "
"every database span raised while serving the request all nest inside "
"it. Without it each of those would be its own root trace. The span name "
"is not a fixed string: it is the value of ``http.request.method``. "
"is not a fixed string: it is the method followed by the matched route, "
"and just the method for a request that matched no route. The span starts "
"at the ASGI edge, before routing has happened, so it is named for the "
"method there and renamed once the route is known. "
"W3C ``traceparent`` and ``baggage`` headers are extracted using the "
"global propagator, so a request arriving from an already-traced caller "
"continues that trace; set ``OTEL_PROPAGATORS=none`` to turn that off, "
"and strip those headers at your proxy if your instance is public.",
(
HTTP_REQUEST_METHOD,
HTTP_ROUTE,
URL_PATH,
URL_SCHEME,
SERVER_ADDRESS,

38
docs/internals.rst
View file

@ -2367,21 +2367,26 @@ This reference is generated from ``datasette/telemetry_registry.py``, the single
Spans are ``SpanKind.INTERNAL`` unless a kind is listed below. Two are not: the request span is ``SERVER``, and ``db.query`` is ``CLIENT`` because it is the one span that represents a call to a database rather than Datasette's own work. Trace UIs use the kind to decide whether to render a span as an inbound request or as a database call. ``db.query``'s children stay ``INTERNAL`` because they are Datasette's decomposition of that one query - marking them ``CLIENT`` too would make a single query look like several database calls to anything counting by kind.
The request span's name is the only one that is not a fixed string - it is composed from the request, so the heading below shows the template rather than a literal you will see in a trace.
The request span's name is the only one that is not a fixed string - it is composed from the request, so the heading below shows the template rather than a literal you will see in a trace. A request to a table page produces a span named, in full::
GET /(?P<database>[^\/\.]+)/(?P<table>[^\/\.]+)(\.(?P<format>\w+))?$
That is the route's compiled regular expression, not a prettified ``/{database}/{table}`` template. It is deliberate: Datasette routes with compiled patterns and the route table is fixed when the app is built, so the pattern is exact, bounded and needs no parsing, while transforming it into something prettier accretes edge cases. Django's own instrumentation ships regex-flavoured routes for the same reason.
.. [[[cog
from telemetry_doc import spans
spans(cog)
.. ]]]
``{http.request.method}``
One span per HTTP request, created by the outermost layer of the ASGI stack - so plugin ``asgi_wrapper()`` middleware, CSRF protection and every database span raised while serving the request all nest inside it. Without it each of those would be its own root trace. The span name is not a fixed string: it is the value of ``http.request.method``. W3C ``traceparent`` and ``baggage`` headers are extracted using the global propagator, so a request arriving from an already-traced caller continues that trace; set ``OTEL_PROPAGATORS=none`` to turn that off, and strip those headers at your proxy if your instance is public.
``{http.request.method} {http.route}``
One span per HTTP request, created by the outermost layer of the ASGI stack - so plugin ``asgi_wrapper()`` middleware, CSRF protection and every database span raised while serving the request all nest inside it. Without it each of those would be its own root trace. The span name is not a fixed string: it is the method followed by the matched route, and just the method for a request that matched no route. The span starts at the ASGI edge, before routing has happened, so it is named for the method there and renamed once the route is known. W3C ``traceparent`` and ``baggage`` headers are extracted using the global propagator, so a request arriving from an already-traced caller continues that trace; set ``OTEL_PROPAGATORS=none`` to turn that off, and strip those headers at your proxy if your instance is public.
Kind: ``SERVER``.
Attributes:
- ``http.request.method`` - The HTTP method, clamped to the nine methods RFC 9110 and RFC 5789 define. Anything else is reported as ``_OTHER``: the method is a client-controlled string, so echoing it back unbounded would be a cardinality hazard.
- ``http.route`` *(optional)* - The route the request matched, as the compiled regular expression pattern Datasette routes with - for example ``/(?P<database>[^\/\.]+)/(?P<table>[^\/\.]+)(\.(?P<format>\w+))?$`` for a table page. It is deliberately the pattern rather than a prettified ``/{database}/{table}`` template: the route table is fixed when the app is built, so the pattern is exact, bounded and needs no parsing, whereas the transform into something prettier accretes edge cases. Unlike ``url.path`` this is low cardinality, so it is the attribute to group by. Omitted when no route matched - a 404 - which is also when the span name falls back to the bare method.
- ``url.path`` - The path portion of the URL. The query string is deliberately **not** recorded, on this or any other span: Datasette puts user-supplied SQL in ``?sql=`` and canned query parameters in the query string, so exporting it by default would export exactly the data the rest of this instrumentation is careful with.
- ``url.scheme`` - ``http`` or ``https``.
- ``server.address`` *(optional)* - The ``Host`` header. Client-controlled, so treat it as untrusted input rather than as the identity of the server.
@ -2436,6 +2441,29 @@ The request span's name is the only one that is not a fixed string - it is compo
.. [[[end]]]
.. _internals_telemetry_requests:
Requests and inbound trace context
----------------------------------
Datasette creates the request span itself, at the outermost layer of the ASGI stack, so a trace is complete out of the box with no plugin and no extra instrumentation package. Everything raised while serving the request - plugin ``asgi_wrapper()`` middleware, CSRF protection, every database query - nests inside it.
**Inbound trace context is trusted by default.** W3C ``traceparent`` and ``baggage`` headers are extracted from every request using the global propagator, so a request arriving from an already-traced caller continues that trace instead of starting a new one. That is what every other framework instrumentation does - Flask, Django, FastAPI and ``opentelemetry-instrumentation-asgi`` all extract unconditionally - but on an instance open to the internet it means an arbitrary client can influence your traces:
- **Trace-ID pollution.** The client chooses the trace ID its request is filed under.
- **Sampling control.** The SDK's default sampler is ``parentbased_always_on``, so under any parent-based sampler a client's sampled flag can force recording - a telemetry-cost denial of service - or suppress it.
- **Baggage injection**, through the default composite propagator.
Because extraction goes through the *global* propagator there is no Datasette setting to configure, and the remedies are the standard OpenTelemetry ones:
- Strip ``traceparent``, ``tracestate`` and ``baggage`` at your reverse proxy, which is the right answer for a public instance fronted by one.
- Set ``OTEL_PROPAGATORS=none`` to disable extraction entirely, or ``OTEL_PROPAGATORS=tracecontext`` to keep trace continuation and drop baggage.
- Use a sampler that is not parent-based, which neutralises the sampling concern on its own.
**Installing an ASGI instrumentation as well is harmless.** If you wire up ``opentelemetry-instrumentation-asgi`` through an ``asgi_wrapper()`` plugin, its middleware lands *inside* Datasette's own, so its span becomes a redundant child ``SERVER`` span in the same trace. Nothing is re-orphaned. There is no setting to turn Datasette's request span off, because "turn it off" is already covered by installing no provider, or by ``OTEL_SDK_DISABLED=true``.
**Where** ``datasette.startup`` **lands depends on how you run Datasette.** ``datasette serve`` calls ``invoke_startup()`` before the server starts accepting connections, so the startup span is its own trace. An ASGI-hosted or programmatic deployment reaches startup lazily, on the first request, so there the startup span nests under that first request - which is honest, since it genuinely is that request's latency.
.. _internals_telemetry_privacy:
Privacy and safety
@ -2447,6 +2475,7 @@ Spans leave your infrastructure whenever you configure an exporter, so what goes
- **SQL parameter values are never recorded.** Only ``datasette.param_count``, a count. Parameter values are the part of a query most likely to hold something sensitive, and separating them from the SQL is the reason bound parameters exist.
- **No actor identifiers are recorded.** No actor ID, no actor JSON, no client IP address. Nothing on a span identifies who made the request.
- **Table names come only from an explicit** ``table=`` **argument.** ``db.collection.name`` is set by callers that already know which table they are working with, and is never derived from the SQL. Deriving it would mean parsing, and on an instance where visitors can create tables the set of possible values has no ceiling.
- **The query string is never recorded.** There is no ``url.query`` attribute on the request span or on any other span. Datasette puts user-supplied SQL in ``?sql=`` and canned query parameters in the query string, so recording it by default would export exactly the class of data the rules above are careful with. Only ``url.path`` and ``http.route`` are recorded.
The SQL itself, though, *is* recorded, and on a public instance that means anything a visitor types into the query editor or passes as ``?sql=`` will be exported along with the span. That is the trade-off tracing a query engine makes.
@ -2455,7 +2484,8 @@ The SQL itself, though, *is* recorded, and on a public instance that means anyth
Known limitations
-----------------
- **Datasette does not create a span for the HTTP request itself.** Every span listed above is therefore a root span unless something above Datasette - an ASGI instrumentation layer, or the web framework embedding it - has already started one for the request, in which case Datasette's spans nest underneath it correctly.
- ``http.route`` **is a compiled regular expression, not a pretty route template.** See :ref:`internals_telemetry_requests` above for why.
- **Inbound trace context is trusted by default**, which on a public instance means a client can influence your trace IDs, your sampling and your baggage. :ref:`internals_telemetry_requests` lists the remedies.
- **Two plugin hooks run outside the** ``datasette.startup`` **span.** ``register_output_renderer`` is dispatched from ``Datasette.__init__()`` and ``asgi_wrapper`` from ``Datasette.app()``, both of which happen before ``invoke_startup()``. Datasette itself queries no database in either, so a default install emits nothing there - but a plugin that does will produce a root trace. Covering these would mean holding a span open across object construction, which is worse than the orphan.
- ``db.operation.name`` **reports** ``WITH`` **for a statement that opens with a common table expression**, rather than the operation inside it, and a substantial share of Datasette's own reads take that form. The attribute is a leading-keyword match against a fixed allowlist, deliberately not a parse.
- **Spans emitted before a provider is installed are not recorded.** If you are embedding Datasette in a host application, install your ``TracerProvider`` before serving traffic. This is ordinary OpenTelemetry behaviour rather than anything Datasette controls; nothing is permanently affected, those particular spans are simply dropped.

1
tests/conftest.py
View file

@ -230,6 +230,7 @@ def pytest_collection_modifyitems(config, items):
# (SIGSEGV/SIGBUS inside _execute_child). Reproduces with any subprocess
# call placed there, on an unmodified tree - running it first avoids it.
move_to_front(items, "test_datasette_package_never_imports_the_sdk")
move_to_front(items, "test_no_provider_takes_the_fast_path")
def move_to_front(items, test_name):

537
tests/test_http_span.py
View file

@ -3,31 +3,53 @@ The HTTP request span.
`tests/test_telemetry_registry.py` already pins the span's name shape, kind
and attribute keys against literals, so this file deliberately does not
repeat that. What it covers is the three properties of the middleware that
the registry conformance test structurally cannot see:
repeat that. What it covers is the properties of the middleware and of the
router's `http.route` enrichment that the registry conformance test
structurally cannot see:
- **where the middleware sits.** Outermost is the entire point - moving it
inside the plugin `asgi_wrapper()` loop leaves plugin middleware creating
orphan root traces, which is the problem this span exists to fix, and every
attribute assertion still passes.
- **which span the route lands on**, which only diverges once something else
has made a span current.
- **method clamping**, which a workload of ordinary GETs can never exercise.
- **the query string never being recorded**, which only fails if a request
actually carries one.
- **the span outliving a streamed response body**, which only a paging export
can distinguish from ending far too early.
"""
import asyncio
import itertools
import json
import subprocess
import sys
import textwrap
import time
import pytest
import pytest_asyncio
pytest.importorskip("opentelemetry.sdk")
from opentelemetry.trace import SpanKind, StatusCode
from opentelemetry.trace import (
NonRecordingSpan,
SpanContext,
SpanKind,
StatusCode,
TraceFlags,
)
from datasette import hookimpl
from datasette.app import Datasette
from datasette.telemetry import TelemetryMiddleware, tracer
from datasette.telemetry import (
REQUEST_SPAN_SCOPE_KEY,
TelemetryMiddleware,
request_span,
tracer,
)
from datasette.utils import resolve_routes
# Named in-memory databases are shared-cache: two Datasette instances given
# the same name share one SQLite database and the second `create table`
@ -110,12 +132,51 @@ async def ds():
instance.close()
@pytest_asyncio.fixture
async def ds_paging():
"""
An instance whose table is bigger than `max_returned_rows`.
That is what makes `?_stream=1` genuinely page: `stream_csv` loops calling
`fetch_data` for each page *inside* the response body send, so the trace
contains `db.query` spans that start after the response has begun. On a
table that fits in one page every query finishes before the body starts
and the span-covers-the-body assertion cannot fail.
"""
name = f"httpspanpaging{next(_names)}"
# Both settings matter. `?_stream=1` forces `_size=max`, which is
# `max_returned_rows` - so lowering only that gives one page of five rows
# and no `next` token, and the export never loops.
instance = Datasette(
memory=True, settings={"max_returned_rows": 5, "default_page_size": 3}
)
instance.add_memory_database(name)
await instance.invoke_startup()
db = instance.get_database(name)
await db.execute_write("create table t (id integer primary key, v text)")
await db.execute_write_many(
"insert into t (id, v) values (?, ?)", [[i, f"v{i}"] for i in range(40)]
)
instance.db_name = name
try:
yield instance
finally:
instance.close()
def _server_spans(otel_spans):
return [
span for span in otel_spans.get_finished_spans() if span.kind is SpanKind.SERVER
]
def _route_for(ds, path):
"The compiled pattern Datasette's own router resolves `path` to."
match, _view = resolve_routes(ds._routes(), path)
assert match is not None, f"{path} matches no route"
return match.re.pattern
@pytest.mark.asyncio
async def test_plugin_asgi_wrapper_middleware_runs_inside_the_request_span(
ds, otel_spans
@ -140,20 +201,20 @@ async def test_plugin_asgi_wrapper_middleware_runs_inside_the_request_span(
spans = otel_spans.get_finished_spans()
server = [span for span in spans if span.kind is SpanKind.SERVER]
assert len(server) == 1, "expected exactly one SERVER span per request"
request_span = server[0]
assert request_span.parent is None, "the request span should be the trace root"
server_span = server[0]
assert server_span.parent is None, "the request span should be the trace root"
plugin_spans = [span for span in spans if span.name == PLUGIN_MIDDLEWARE_SPAN]
assert len(plugin_spans) == 1
assert plugin_spans[0].parent is not None
assert plugin_spans[0].parent.span_id == request_span.context.span_id
assert plugin_spans[0].context.trace_id == request_span.context.trace_id
assert plugin_spans[0].parent.span_id == server_span.context.span_id
assert plugin_spans[0].context.trace_id == server_span.context.trace_id
# And the database work is in the same trace, not off on its own.
queries = [span for span in spans if span.name == "db.query"]
assert queries, "a table page should have issued at least one query"
for query in queries:
assert query.context.trace_id == request_span.context.trace_id
assert query.context.trace_id == server_span.context.trace_id
@pytest.mark.asyncio
@ -161,15 +222,20 @@ async def test_unrecognised_method_is_clamped(ds, otel_spans):
"""
Anyone can send `FROB / HTTP/1.1`. An unclamped method is an unbounded
dimension a client controls, so semantic conventions map anything off the
known list to `_OTHER` - and the span name is the method, so an unclamped
one would put attacker-supplied text in the span name too.
known list to `_OTHER`.
The span name is checked too, and it is the reason the router clamps the
method a second time when it renames the span: the middleware's clamping
protects the attribute, but the name is rebuilt from `request.method` in
`route_path`, which is the raw client string. An unclamped rename would
put attacker-supplied text straight back into the span name.
"""
otel_spans.clear()
await ds.client.request("FROB", f"/{ds.db_name}/t")
server = _server_spans(otel_spans)
assert len(server) == 1
assert server[0].name == "_OTHER"
assert server[0].attributes["http.request.method"] == "_OTHER"
assert server[0].name == f"_OTHER {server[0].attributes['http.route']}"
@pytest.mark.asyncio
@ -179,8 +245,8 @@ async def test_known_method_is_not_clamped(ds, otel_spans):
await ds.client.get(f"/{ds.db_name}/t")
server = _server_spans(otel_spans)
assert len(server) == 1
assert server[0].name == "GET"
assert server[0].attributes["http.request.method"] == "GET"
assert server[0].name == f"GET {server[0].attributes['http.route']}"
@pytest.mark.asyncio
@ -275,6 +341,10 @@ async def test_a_404_is_not_an_error(ds, otel_spans):
`error.type` alone. Datasette 404s are routine - every missing table, and
every bot probing for /wp-login.php - so treating them as errors would
drown a real 500 in noise.
Note this 404 *does* match a route: `/no-such-database-at-all` matches the
database pattern and the view then raises `NotFound`. Most Datasette 404s
are that shape rather than the unrouted one below.
"""
otel_spans.clear()
response = await ds.client.get("/no-such-database-at-all")
@ -286,6 +356,29 @@ async def test_a_404_is_not_an_error(ds, otel_spans):
assert server[0].status.status_code is StatusCode.UNSET
@pytest.mark.asyncio
async def test_an_unrouted_404_has_no_route_and_a_bare_method_name(ds, otel_spans):
"""
When no route matches there is nothing to set `http.route` to, so the span
keeps the bare method name it was given at the edge - which is exactly the
fallback semantic conventions specify for an unknown route.
`/a/b/c/d/e` is used rather than a plausible-looking missing name because
Datasette's route table is greedy: `/no-such-database-at-all` matches the
database pattern, and `/-/nope/deeper` matches the row pattern. Only a
path deeper than any route matches nothing at all.
"""
otel_spans.clear()
response = await ds.client.get("/a/b/c/d/e")
assert response.status_code == 404
server = _server_spans(otel_spans)
assert len(server) == 1
assert server[0].name == "GET"
assert "http.route" not in server[0].attributes
assert server[0].attributes["http.response.status_code"] == 404
assert server[0].status.status_code is StatusCode.UNSET
@pytest.mark.asyncio
async def test_only_the_first_http_response_start_is_recorded(otel_spans):
"""
@ -342,3 +435,421 @@ async def test_lifespan_scope_passes_through_unspanned(otel_spans):
await app({"type": "lifespan"}, receive, send)
assert sent == ["lifespan.startup.complete", "lifespan.shutdown.complete"]
assert not _server_spans(otel_spans)
@pytest.mark.asyncio
async def test_http_route_is_the_compiled_pattern(ds, otel_spans):
"""
`http.route` is the route's compiled regex, not a prettified template.
Asserted against what Datasette's own router resolves rather than against
a copied literal, so this pins the *relationship* - the attribute is the
matched route - and does not break when a core pattern is edited.
"""
path = f"/{ds.db_name}/t"
expected = _route_for(ds, path)
otel_spans.clear()
assert (await ds.client.get(path)).status_code == 200
server = _server_spans(otel_spans)
assert len(server) == 1
assert server[0].attributes["http.route"] == expected
assert server[0].name == f"GET {expected}"
# The pattern really is the ugly one, and that is deliberate - if someone
# adds a prettifier this is the assertion that should make them argue for
# it rather than slip it in.
assert "(?P<database>" in expected
@pytest.mark.asyncio
async def test_the_route_lands_on_the_request_span_not_a_plugins_current_span(
ds, otel_spans
):
"""
The route is set on the span the middleware started, found through the
ASGI scope - not on whatever span happens to be current when routing
resolves.
Those are the same span only until a plugin `asgi_wrapper()` starts one of
its own. A plugin wrapper runs *inside* this middleware, so an instrumented
plugin makes its span current for the whole request: reading the current
span in `route_path` renames that plugin's INTERNAL span to
`GET <route>` and hangs `http.route` off it, while the actual request span
keeps a bare method name and never gets the one attribute a trace UI
groups requests by. Verified by reproducing it, not by reasoning about it.
"""
ds.pm.register(_MiddlewarePlugin(), name="httpspan-middleware")
try:
otel_spans.clear()
path = f"/{ds.db_name}/t"
expected = _route_for(ds, path)
assert (await ds.client.get(path)).status_code == 200
finally:
ds.pm.unregister(name="httpspan-middleware")
spans = otel_spans.get_finished_spans()
server = _server_spans(otel_spans)
assert len(server) == 1
assert server[0].attributes["http.route"] == expected
assert server[0].name == f"GET {expected}"
# And the plugin's span is untouched: same name, no route attribute.
plugin_spans = [span for span in spans if span.name == PLUGIN_MIDDLEWARE_SPAN]
assert len(plugin_spans) == 1
assert "http.route" not in (plugin_spans[0].attributes or {})
@pytest.mark.asyncio
async def test_request_span_attributes(ds, otel_spans):
"The whole attribute set on one ordinary request."
path = f"/{ds.db_name}/t"
otel_spans.clear()
assert (await ds.client.get(path)).status_code == 200
server = _server_spans(otel_spans)
assert len(server) == 1
attributes = server[0].attributes
assert attributes["http.request.method"] == "GET"
assert attributes["url.path"] == path
assert attributes["url.scheme"] == "http"
assert attributes["http.response.status_code"] == 200
assert attributes["http.route"] == _route_for(ds, path)
assert server[0].status.status_code is StatusCode.UNSET
# Never, on any span: an IP is borderline PII and the query string carries
# user-supplied SQL.
assert "client.address" not in attributes
assert "url.query" not in attributes
@pytest.mark.asyncio
async def test_db_query_spans_are_children_of_the_request_span(ds, otel_spans):
"""
The point of the whole PR.
Not just "same trace ID" - every `db.query` span must reach the request
span by walking parents, and the request span must be the only root. A
stray root would show up in a trace UI as its own single-span trace, which
is the state this replaces.
"""
otel_spans.clear()
assert (await ds.client.get(f"/{ds.db_name}/t?_facet=v")).status_code == 200
spans = otel_spans.get_finished_spans()
server = _server_spans(otel_spans)
assert len(server) == 1
server_span = server[0]
assert server_span.parent is None
by_span_id = {span.context.span_id: span for span in spans}
roots = [span for span in spans if span.parent is None]
assert [span.name for span in roots] == [server_span.name], (
"every span from a request should hang off the request span, but these "
f"are roots: {sorted(span.name for span in roots)}"
)
queries = [span for span in spans if span.name == "db.query"]
assert queries, "a faceted table page should have issued queries"
for query in queries:
assert query.context.trace_id == server_span.context.trace_id
# Walk up to the root, which must be the request span.
current = query
seen = 0
while current.parent is not None:
current = by_span_id[current.parent.span_id]
seen += 1
assert seen < 20, "parent chain did not terminate"
assert current is server_span
@pytest.mark.asyncio
async def test_500_sets_error_status_and_error_type(ds, otel_spans):
"""
A plain 500 - no exception escaping the app, because `route_path` converts
it into a response itself. The status is the only signal the middleware
gets, so `error.type` is the status as a string.
"""
ds.pm.register(_BoomPlugin(), name="httpspan-boom")
try:
otel_spans.clear()
response = await ds.client.get("/-/http-span-boom")
assert response.status_code == 500
finally:
ds.pm.unregister(name="httpspan-boom")
server = _server_spans(otel_spans)
assert len(server) == 1
assert server[0].attributes["http.response.status_code"] == 500
assert server[0].attributes["error.type"] == "500"
assert server[0].status.status_code is StatusCode.ERROR
@pytest.mark.asyncio
async def test_csv_stream_span_covers_the_body_send(ds_paging, otel_spans):
"""
The span must not end when the handler returns - it has to cover the
response body.
`stream_csv` runs its generator inline inside `AsgiStream.asgi_send`, and
that call happens inside the single `await self.app(...)` the middleware
makes, so a plain `finally` is enough and no deferred-end machinery is
needed. This is the assertion that holds that claim up: a `db.query` that
starts during the body send must still finish before the request span
does.
Only meaningful on an export that actually pages, hence `ds_paging` - on a
single-page table every query is over before the body begins and this
passes however early the span ends. The middle assertion below, that some
query *started* after `http.response.start` went out, is what keeps the
test honest about that; it is why the app is driven as raw ASGI rather
than through `ds.client`, which cannot timestamp the response start.
`time.time_ns()` is the same clock the SDK stamps spans with, so the two
are directly comparable.
"""
app = ds_paging.app()
body = []
response_started_at = None
async def receive():
return {"type": "http.request", "body": b"", "more_body": False}
async def send(message):
nonlocal response_started_at
if message["type"] == "http.response.start":
assert message["status"] == 200
response_started_at = time.time_ns()
else:
body.append(message.get("body") or b"")
otel_spans.clear()
await app(
{
"type": "http",
"http_version": "1.1",
"method": "GET",
"path": f"/{ds_paging.db_name}/t.csv",
"raw_path": f"/{ds_paging.db_name}/t.csv".encode("latin-1"),
"query_string": b"_stream=1",
"scheme": "http",
"headers": [(b"host", b"localhost")],
},
receive,
send,
)
# 40 rows plus a header - the export really did read past one page
assert len(b"".join(body).decode("utf-8").strip().splitlines()) == 41
assert response_started_at is not None
spans = otel_spans.get_finished_spans()
server = _server_spans(otel_spans)
assert len(server) == 1
server_span = server[0]
queries = [span for span in spans if span.name == "db.query"]
assert len(queries) > 1
during_body = [span for span in queries if span.start_time > response_started_at]
assert during_body, (
"no query ran after the response started, so this workload cannot "
"distinguish a span that covers the body send from one that ends when "
"the handler returns - the export is not paging"
)
last_query_end = max(span.end_time for span in queries)
assert server_span.end_time > last_query_end, (
"the request span ended before the last query of a streaming export - "
"it is not covering the response body"
)
for query in queries:
assert query.context.trace_id == server_span.context.trace_id
@pytest.mark.asyncio
async def test_inbound_traceparent_becomes_the_parent(ds, otel_spans):
"""
W3C trace context is extracted with the global propagator, so a request
from an already-traced caller continues that trace.
The sampled flag has to be set: the SDK's default sampler is
parentbased_always_on, so a `-00` flag would drop the span and the test
would fail for a reason that has nothing to do with propagation.
"""
trace_id = "4bf92f3577b34da6a3ce929d0e0e4736"
parent_span_id = "00f067aa0ba902b7"
otel_spans.clear()
response = await ds.client.get(
f"/{ds.db_name}/t",
headers={"traceparent": f"00-{trace_id}-{parent_span_id}-01"},
)
assert response.status_code == 200
server = _server_spans(otel_spans)
assert len(server) == 1
server_span = server[0]
assert f"{server_span.context.trace_id:032x}" == trace_id
assert server_span.parent is not None
assert f"{server_span.parent.span_id:016x}" == parent_span_id
assert server_span.parent.is_remote
# And the database spans joined the caller's trace too, not a new one.
queries = [
span for span in otel_spans.get_finished_spans() if span.name == "db.query"
]
assert queries
for query in queries:
assert f"{query.context.trace_id:032x}" == trace_id
@pytest.mark.asyncio
async def test_user_supplied_sql_in_the_query_string_is_never_recorded(ds, otel_spans):
"""
The `?sql=` case specifically, which is the one that matters: this is the
request where the query string *is* user-supplied SQL, and it reaches a
view that runs it. The marker is searched for across every attribute of
every span in the trace, not just for a `url.query` key, so recording it
under some other name fails too.
`db.query.text` legitimately contains the SQL - that is documented and
deliberate - so the marker is checked against the request span's own
attributes, and against `url.*` and `http.*` keys everywhere.
"""
marker = "secret_marker_5b1f"
otel_spans.clear()
# `/{db}?sql=` 302s to the query view, so go straight there - a redirect
# would leave the SQL only on a span for a request that never ran it.
response = await ds.client.get(f"/{ds.db_name}/-/query?sql=select+'{marker}'")
assert response.status_code == 200
spans = otel_spans.get_finished_spans()
server = _server_spans(otel_spans)
assert len(server) == 1
leaked = [
f"{span.name} -> {key}={value!r}"
for span in spans
for key, value in (span.attributes or {}).items()
if (span is server[0] or str(key).startswith(("url.", "http.")))
and (marker in str(value) or str(key) == "url.query")
]
assert not leaked, "the query string reached a span attribute: " + ", ".join(leaked)
# The request really did carry the marker, so the search above had
# something to find.
assert marker in response.text
def test_request_span_skips_a_valid_but_non_recording_span():
"""
`request_span()` is guarded on `is_recording()`, not on
`get_span_context().is_valid`, and this is the case that separates them.
With no provider installed but an inbound `traceparent`, the API's
NoOpTracer hands back a `NonRecordingSpan` carrying the *remote* span
context - valid, sampled, and recording nothing. An `is_valid` guard would
wave that through and the router would build the name string and call
`set_attribute`/`update_name` on a span that discards both.
Tested at this level deliberately: through a real request the two guards
are indistinguishable, because every call the router makes on a
NonRecordingSpan is already a no-op. The only difference is the work done
to get there, so the guard itself is what has to be asserted on.
"""
remote = SpanContext(
trace_id=0x4BF92F3577B34DA6A3CE929D0E0E4736,
span_id=0x00F067AA0BA902B7,
is_remote=True,
trace_flags=TraceFlags(TraceFlags.SAMPLED),
)
assert remote.is_valid
non_recording = NonRecordingSpan(remote)
assert non_recording.is_recording() is False
assert request_span({REQUEST_SPAN_SCOPE_KEY: non_recording}) is None
# Nothing current, nothing in the scope: the INVALID_SPAN fallback.
assert request_span({}) is None
# And the case it must not skip.
with tracer.start_as_current_span("test.request_span.recording") as span:
assert request_span({REQUEST_SPAN_SCOPE_KEY: span}) is span
# Falling back to the current span is how an externally installed
# SERVER span still gets enriched.
assert request_span({}) is span
NO_PROVIDER_PROGRAM = textwrap.dedent("""
import asyncio, json, sys
from datasette.telemetry import TelemetryMiddleware
seen = {}
async def inner(scope, receive, send):
seen.setdefault("sends", []).append(send)
seen.setdefault("scopes", []).append(scope)
await send({"type": "http.response.start", "status": 200, "headers": []})
await send({"type": "http.response.body", "body": b""})
async def real_send(message):
pass
async def main():
middleware = TelemetryMiddleware(inner)
for headers in ([], [(b"traceparent", b"00-" + b"a" * 32 + b"-" + b"b" * 16 + b"-01")]):
await middleware(
{
"type": "http",
"method": "GET",
"path": "/",
"raw_path": b"/",
"scheme": "http",
"headers": headers,
},
None,
real_send,
)
print(
json.dumps(
{
"unwrapped": [send is real_send for send in seen["sends"]],
"scope_keys": [
"datasette.telemetry.request_span" in scope
for scope in seen["scopes"]
],
"sdk_imported": any(
name.startswith("opentelemetry.sdk") for name in sys.modules
),
}
)
)
asyncio.run(main())
""")
def test_no_provider_takes_the_fast_path():
"""
With no `TracerProvider` installed the middleware must hand the
application the *original* `send`, not a wrapper - a default Datasette
install should pay essentially nothing for instrumentation it is not
using.
This has to run in a subprocess. The suite's `_otel_provider` fixture is
session-scoped and autouse, and `set_tracer_provider()` is effectively
once-per-process, so in-process every span is recording and the fast path
is unreachable.
The second case, with an inbound `traceparent`, is the one that pins the
check itself. With no provider the API's NoOpTracer returns a
NonRecordingSpan carrying the *remote* span context: its
`get_span_context().is_valid` is True while `is_recording()` is False. A
fast path guarded on `is_valid` would therefore silently stop working for
exactly the requests that arrive from an already-traced caller - which on
a real deployment behind an instrumented proxy is all of them.
conftest.py's pytest_collection_modifyitems() moves this test to the front
of the run by name - if you rename it, rename it there too.
"""
result = subprocess.run(
[sys.executable, "-c", NO_PROVIDER_PROGRAM],
capture_output=True,
text=True,
check=True,
)
report = json.loads(result.stdout)
assert report["sdk_imported"] is False, "the SDK loaded in a fresh interpreter"
assert report["unwrapped"] == [True, True], (
"the middleware wrapped `send` with no provider installed; the second "
"entry is the inbound-traceparent case, which fails if the fast path "
"is guarded on is_valid instead of is_recording()"
)
# Same fast path, other observable: nothing is stashed in the scope either.
assert report["scope_keys"] == [False, False]

52
tests/test_telemetry_registry.py
View file

@ -70,15 +70,22 @@ EXPECTED_ATTRIBUTES = {
EXPECTED_SPANS = set(EXPECTED_ATTRIBUTES)
# The HTTP request span is handled separately because its name is composed at
# runtime - it is the request method - so there is no fixed string to pin it
# to. What can still be pinned, and is what a dashboard depends on, is the
# shape of the name and the attribute keys. The workload below only issues
# GETs, so a change that stopped clamping the method, or that started naming
# the span after the path, fails here.
EXPECTED_HTTP_SPAN_NAME = "{http.request.method}"
EXPECTED_HTTP_SPAN_NAMES = {"GET"}
# runtime - the request method, then the route it matched - so there is no
# fixed string to pin it to. What can still be pinned, and is what a dashboard
# depends on, is the shape of that name and the attribute keys.
#
# The route half is deliberately not spelled out as a literal: it is a core
# route regex, and pinning those here would make an unrelated routing change
# fail the telemetry conformance test. What is pinned instead is that the name
# is exactly the method, a space, and the span's own `http.route` value - the
# `{method} {route}` shape semantic conventions specify. The workload below
# only issues GETs, so a change that stopped clamping the method, or that
# started naming the span after the path, fails here.
EXPECTED_HTTP_SPAN_NAME = "{http.request.method} {http.route}"
EXPECTED_HTTP_METHOD_NAMES = {"GET"}
EXPECTED_HTTP_ATTRIBUTES = {
"http.request.method",
"http.route",
"url.path",
"url.scheme",
"server.address",
@ -193,10 +200,12 @@ async def exercise():
@pytest_asyncio.fixture
async def emitted(otel_spans):
"""
Every (span name, span kind, attribute keys) triple a broad workload emits.
Every (span name, span kind, attributes) triple a broad workload emits.
The kind is carried because the request span's name is composed at
runtime, so `span_for()` resolves it by kind instead.
runtime, so `span_for()` resolves it by kind instead. The attributes are
carried as a mapping rather than a set of keys because the request span's
name has to be checked against its own `http.route` value.
"""
# otel_spans has already cleared the exporter, and nothing is cleared
# after this point: the workload's own startup emits datasette.startup.
@ -210,7 +219,7 @@ async def emitted(otel_spans):
(
str(span.name),
span.kind,
frozenset(str(key) for key in span.attributes or {}),
{str(key): value for key, value in (span.attributes or {}).items()},
)
for span in spans
)
@ -227,8 +236,8 @@ def _partition(emitted):
def _keys_by_span(records):
by_span = {}
for name, _kind, keys in records:
by_span.setdefault(name, set()).update(keys)
for name, _kind, attributes in records:
by_span.setdefault(name, set()).update(attributes)
return by_span
@ -246,11 +255,22 @@ async def test_workload_emits_exactly_the_expected_names(emitted):
assert by_span == EXPECTED_ATTRIBUTES
assert server, "the workload made HTTP requests but no SERVER span was emitted"
server_keys = _keys_by_span(server)
assert set(server_keys) == EXPECTED_HTTP_SPAN_NAMES
union = set()
for keys in server_keys.values():
union |= keys
methods = set()
for name, _kind, attributes in server:
union |= set(attributes)
route = attributes.get("http.route")
# Every request in the workload matches a route, so every one of these
# names must be `{method} {route}`. A 404 would be a bare method - the
# http_route tests cover that case with a real request.
assert route, f"the request span {name!r} carries no http.route"
method, _, name_route = name.partition(" ")
assert name_route == route, (
f"the request span is named {name!r}, which is not the "
f"`{{method}} {{route}}` of {method!r} and {route!r}"
)
methods.add(method)
assert methods == EXPECTED_HTTP_METHOD_NAMES
assert union == EXPECTED_HTTP_ATTRIBUTES