datasette/demos/otel

OpenTelemetry demo

Datasette core depends on opentelemetry-api only. It emits spans and metrics and nothing else — it never creates a TracerProvider or a MeterProvider, never configures an exporter, and never sets a sampler. With no SDK installed every span and every instrument is a no-op and costs approximately nothing.

That means "turning telemetry on" is entirely the job of whoever runs Datasette. This directory shows several ways to do it, none of which needs Docker:

otlp_receiver.py	A ~150 line pure-Python OTLP/HTTP receiver — a real protobuf export, summarized in your terminal
just jaeger	The same export into Jaeger's own binary, for a real trace UI
metrics_demo.py	Saturates the SQL thread pool and prints the gauges — the question spans cannot answer

Both listen for OTLP/HTTP on port 4318, so the Datasette side is identical — run one or the other, not both. The Justfile in this directory wraps every command below; bare just lists the recipes.

1. A real OTLP export, pure Python

Terminal 1 — the receiver (just receiver):

uv run --with opentelemetry-proto python demos/otel/otlp_receiver.py

Terminal 2 — Datasette under the OpenTelemetry agent (just serve, which also generates a 200-row demo.db on first run):

OTEL_TRACES_EXPORTER=otlp \
OTEL_METRICS_EXPORTER=none \
OTEL_LOGS_EXPORTER=none \
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \
OTEL_SERVICE_NAME=datasette \
OTEL_BSP_SCHEDULE_DELAY=1000 \
  uv run --with opentelemetry-distro \
         --with opentelemetry-exporter-otlp-proto-http \
    opentelemetry-instrument datasette demo.db -p 8001

Load a page (just request), wait a second for the batch flush, then Ctrl-C the receiver. Real output from startup plus one request against the 200-row demo table:

received 93 spans (93 total)

=== 93 spans ===
  db.query                                        43      18.55ms total
  db.query.execute                                42       7.60ms total
  db.write.queue_wait                              3       0.40ms total
  db.write.execute                                 3       4.99ms total
  datasette.startup                                1       7.17ms total
  GET /(?P<database>[^\/\.]+)/(?P<table>[^\/\.]+)(\.(?P<format>\w+))?$    1      59.46ms total

slowest spans:
      59.46ms  GET /(?P<database>[^\/\.]+)/(?P<table>[^\/\.]+)(\.(?P<format>\w+))?$  -> 200
       7.17ms  datasette.startup
       4.58ms  db.write.execute
       1.81ms  db.query  with limited as (select * from (select id, name, height_cm f

2 root spans:
  datasette.startup x1
  GET /(?P<database>[^\/\.]+)/(?P<table>[^\/\.]+)(\.(?P<format>\w+))?$ x1

Things worth noticing:

• Exactly two root spans. Every span belongs to either the request that caused it or to datasette.startup — there are no orphans. A block=False write would be a third kind of root, carrying a link back to the request that enqueued it rather than a parent, because the write can outlive that request.
• Request spans are named after the matched route, which in Datasette is a regex — that string is ugly but it is the honest low-cardinality name. The pretty path is in the url.path attribute.
• db.query vs db.query.execute. The gap between the two is time spent waiting for a thread. Likewise db.write.queue_wait vs db.write.execute is how you tell "the write was slow" apart from "the write waited behind another writer".

The receiver is a debugging aid, not a backend: nothing is persisted, it speaks OTLP/HTTP only (not gRPC), and it ignores metrics and logs.

2. The same thing with a real UI: Jaeger, no Docker

Jaeger ingests OTLP directly on the same port 4318, so the Datasette command does not change. With the jaeger binary on your PATH (https://www.jaegertracing.io/download/):

just jaeger    # terminal 1 — UI on http://localhost:16686
just serve     # terminal 2 — identical to above
just request   # terminal 3

Then open http://localhost:16686, pick service datasette, and Find Traces. Verified: one request produces exactly two traces — the request trace (~67 spans, rooted at the GET ... span, with every db.query nested inside it across thread boundaries) and the datasette.startup trace (~26 spans of catalog queries and connection warm-up).

3. The question spans cannot answer

uv run python demos/otel/metrics_demo.py

A trace tells you a query took 170ms. It does not tell you that 130ms of that was spent waiting for one of only three threads — "how many threads are busy right now" is a level, not an event, so no span can carry it. That is what metrics are for.

The script registers a SQL function that sleeps, fires 12 concurrent 40ms queries at a pool of 3 threads, and samples the gauges while they are in flight. Real output:

num_sql_threads = 3, firing 12 concurrent 40ms queries

wall clock                     : 170ms
  if fully serialised          : 480ms
  with 3 threads perfectly used : 160ms

Peak values sampled while the queries were in flight:

  datasette.sql.queries.pending  {db.namespace=demo} = 12
  datasette.sql.threads.queue_depth = 9

Final collection:

  datasette.sql.threads.limit
      3
  db.client.operation.duration  {datasette.operation=read, db.namespace=demo, db.system=sqlite}
      count=16 sum=1.2605s min=0.0001s max=0.1695s
  datasette.write.queue_wait  {db.namespace=demo}
      count=1 sum=0.0002s min=0.0002s max=0.0002s

queue_depth = 9 is the whole point: 12 queries, 3 threads, 9 of them sitting in a queue. Sustained above zero in production means requests are backing up on num_sql_threads, and no amount of reading traces would have told you that.

Note also max=0.1695s on the duration histogram against a query whose actual work is 40ms. The gap is queue time. The two numbers together — 170ms observed, 40ms of work — are what distinguishes "my queries are slow" from "my pool is too small".

Notes on the environment variables

• opentelemetry-instrument is required. Setting OTEL_TRACES_EXPORTER and running plain datasette produces nothing at all: that variable is read by the SDK's auto-configuration, which only runs under the agent — core never installs a provider itself.
• OTEL_SERVICE_NAME=datasette is what you pick from Jaeger's Service dropdown. Leave it out and the SDK defaults to unknown_service:<executable>.
• OTEL_BSP_SCHEDULE_DELAY=1000 drops the batch flush from its ~10 second default to ~1 second. For a demo this is the difference between "it works" and "it looks broken". Do not use it in production — it trades export efficiency for latency.
• OTEL_METRICS_EXPORTER=none OTEL_LOGS_EXPORTER=none because opentelemetry-distro defaults every signal to OTLP, and a traces-only backend like Jaeger answers the metrics and logs exports with a stream of StatusCode.UNIMPLEMENTED noise.

4. Exemplars: linking a metric spike to a trace

Once both signals are on — an SDK tracer provider as well as a metrics one, which is what the agent in sections 1 and 2 installs (drop OTEL_METRICS_EXPORTER=none to get both) — each histogram measurement also carries the trace of the request that produced it, with no extra configuration on Datasette's side. The SDK attaches the current trace ID and span ID to any measurement recorded inside a sampled span, and every metric on the query path is recorded inside one. (The metrics demo in section 3 installs no tracer provider, so it shows none of this.) Four queries of increasing cost, each in its own span, produced one exemplar per query on db.client.operation.duration:

db.client.operation.duration count=4
  exemplars: 4
    value=0.001564s trace_id=ddfaf45fd4e14913497d7efeac95f381 span_id=fd5792bdbb01e533
    value=0.006320s trace_id=34aea775ade11a3c5f716695731000fe span_id=25ed9e29dd84dbee
    value=0.045253s trace_id=a65cb58d1460a179f0d04046ff51ed0d span_id=7f34d6378c85d062
    value=0.305240s trace_id=6089f4c515c221c0ca7bb53667b37ac8 span_id=0516f4a6641eaa0b

Exemplars are kept one per histogram bucket, so bucket boundaries decide how many distinct traces a metric can point at. The same four queries, run against an earlier set of bucket boundaries under which all four fell into a single (0, 5] second bucket, produced one exemplar instead of four — fixing the boundaries changed more than the quantiles, it also multiplied the traces reachable from this metric.

The pinned opentelemetry-exporter-prometheus (0.65b0) does not emit exemplars at all — the word exemplar does not appear anywhere in its source, and rendering the workload above through that exporter in the OpenMetrics format — the only exposition format that can carry an exemplar — produced zero exemplar markers. The OTLP exporter carries exemplars through unchanged, so if they need to reach Prometheus, route them through an OTLP collector rather than through Datasette's own Prometheus exporter. On that path, the Prometheus server needs --enable-feature=exemplar-storage and a scrape in the OpenMetrics format — its default text format has no syntax for exemplars — and Grafana needs the Prometheus data source's exemplar configuration (exemplarTraceIdDestinations) pointed at a tracing data source before it draws one as a clickable point. See the internals_telemetry section of the main docs for both, with links to the primary sources.

An exemplar only exists for a trace that was sampled. With the tracer provider's sampler set to ALWAYS_OFF, the same workload produced exemplars: 0 on every data point rather than a link to a trace that was never kept — at low sampling rates most measurements carry no exemplar, but the ones that do always resolve to a real trace.

Privacy

db.query.text is recorded, truncated. SQL parameter values are never recorded — only a parameter count. On a public Datasette instance the SQL text is user-supplied; if you export to a third-party vendor, that text leaves your infrastructure.

See the telemetry section of the Datasette documentation for the full span and attribute reference.