remote/datasette
1
0
Fork 0
mirror of https://github.com/simonw/datasette.git synced 2026-09-02 22:54:08 +02:00
Code Issues Projects Releases Packages Wiki Activity

Merge 76893cb8d3 into 745b7872b8

Browse source
This commit is contained in:
Alex Garcia 2026-09-02 00:09:52 +00:00 • committed by GitHub
commit e7c506cd8a
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 341 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

30
demos/otel/Justfile Normal file
View file

@ -0,0 +1,30 @@
# OpenTelemetry demo - see README.md in this directory.
#
# just receiver + just serve -> span summary in the terminal
# just jaeger + just serve -> real trace UI at http://localhost:16686
#
# Both listen for OTLP/HTTP on port 4318, so `just serve` works with either
# (but not both at once).
otel_env := "OTEL_TRACES_EXPORTER=otlp OTEL_METRICS_EXPORTER=none OTEL_LOGS_EXPORTER=none OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 OTEL_SERVICE_NAME=datasette OTEL_BSP_SCHEDULE_DELAY=1000"
default:
@just --list --unsorted
# Terminal 1, option A: the ~150 line pure-Python receiver. Ctrl-C for a summary
receiver:
uv run --with opentelemetry-proto python otlp_receiver.py
# Terminal 1, option B: Jaeger from its own binary - no Docker. UI on :16686
jaeger:
@command -v jaeger >/dev/null || { echo "No jaeger binary on PATH. Grab one from https://www.jaegertracing.io/download/"; exit 1; }
jaeger
# Terminal 2: Datasette under the OpenTelemetry agent, exporting to :4318
serve db="demo.db":
@[ "{{ db }}" != "demo.db" ] || [ -e demo.db ] || sqlite3 demo.db "create table plants(id integer primary key, name text, height_cm real); with recursive n(i) as (select 1 union all select i + 1 from n where i < 200) insert into plants select i, 'plant ' || i, abs(random() % 300) from n;"
{{ otel_env }} uv run --with opentelemetry-distro --with opentelemetry-exporter-otlp-proto-http opentelemetry-instrument datasette {{ db }} -p 8001
# Terminal 3: make a traced request (defaults to the generated demo table)
request path="/demo/plants":
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8001{{ path }}

121
demos/otel/README.md Normal file
View file

@ -0,0 +1,121 @@
# OpenTelemetry demo
Datasette core depends on `opentelemetry-api` only. It emits spans and nothing else — it never
creates a `TracerProvider`, never configures an exporter, and never sets a sampler. With no SDK
installed every span is a no-op and costs approximately nothing.
That means "turning tracing on" is entirely the job of whoever runs Datasette. This directory
shows two ways to do it, **neither of which needs Docker**:
| | |
|---|---|
| `otlp_receiver.py` | A ~150 line pure-Python OTLP/HTTP receiver — a real protobuf export, summarized in your terminal |
| `just jaeger` | The same export into Jaeger's own binary, for a real trace UI |
Both listen for OTLP/HTTP on port 4318, so the Datasette side is identical — run one or the
other, not both. The `Justfile` in this directory wraps every command below; bare `just` lists
the recipes.
## 1. A real OTLP export, pure Python
Terminal 1 — the receiver (`just receiver`):
```bash
uv run --with opentelemetry-proto python demos/otel/otlp_receiver.py
```
Terminal 2 — Datasette under the OpenTelemetry agent (`just serve`, which also generates a
200-row `demo.db` on first run):
```bash
OTEL_TRACES_EXPORTER=otlp \
OTEL_METRICS_EXPORTER=none \
OTEL_LOGS_EXPORTER=none \
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \
OTEL_SERVICE_NAME=datasette \
OTEL_BSP_SCHEDULE_DELAY=1000 \
uv run --with opentelemetry-distro \
--with opentelemetry-exporter-otlp-proto-http \
opentelemetry-instrument datasette demo.db -p 8001
```
Load a page (`just request`), wait a second for the batch flush, then Ctrl-C the receiver.
Real output from startup plus one request against the 200-row demo table:
```
received 93 spans (93 total)
=== 93 spans ===
db.query 43 18.55ms total
db.query.execute 42 7.60ms total
db.write.queue_wait 3 0.40ms total
db.write.execute 3 4.99ms total
datasette.startup 1 7.17ms total
GET /(?P<database>[^\/\.]+)/(?P<table>[^\/\.]+)(\.(?P<format>\w+))?$ 1 59.46ms total
slowest spans:
59.46ms GET /(?P<database>[^\/\.]+)/(?P<table>[^\/\.]+)(\.(?P<format>\w+))?$ -> 200
7.17ms datasette.startup
4.58ms db.write.execute
1.81ms db.query with limited as (select * from (select id, name, height_cm f
2 root spans:
datasette.startup x1
GET /(?P<database>[^\/\.]+)/(?P<table>[^\/\.]+)(\.(?P<format>\w+))?$ x1
```
Things worth noticing:
- **Exactly two root spans.** Every span belongs to either the request that caused it or to
`datasette.startup` — there are no orphans. A `block=False` write would be a third kind of
root, carrying a *link* back to the request that enqueued it rather than a parent, because
the write can outlive that request.
- **Request spans are named after the matched route**, which in Datasette is a regex — that
string is ugly but it is the honest low-cardinality name. The pretty path is in the
`url.path` attribute.
- **`db.query` vs `db.query.execute`.** The gap between the two is time spent waiting for a
thread. Likewise `db.write.queue_wait` vs `db.write.execute` is how you tell "the write was
slow" apart from "the write waited behind another writer".
The receiver is a debugging aid, not a backend: nothing is persisted, it speaks OTLP/HTTP only
(not gRPC), and it ignores metrics and logs.
## 2. The same thing with a real UI: Jaeger, no Docker
Jaeger ingests OTLP directly on the same port 4318, so the Datasette command does not change.
With the `jaeger` binary on your PATH (<https://www.jaegertracing.io/download/>):
```bash
just jaeger # terminal 1 — UI on http://localhost:16686
just serve # terminal 2 — identical to above
just request # terminal 3
```
Then open <http://localhost:16686>, pick service `datasette`, and Find Traces. Verified: one
request produces exactly two traces — the request trace (~67 spans, rooted at the `GET ...`
span, with every `db.query` nested inside it across thread boundaries) and the
`datasette.startup` trace (~26 spans of catalog queries and connection warm-up).
## Notes on the environment variables
- **`opentelemetry-instrument` is required.** Setting `OTEL_TRACES_EXPORTER` and running plain
`datasette` produces nothing at all: that variable is read by the SDK's auto-configuration,
which only runs under the agent — core never installs a provider itself.
- **`OTEL_SERVICE_NAME=datasette`** is what you pick from Jaeger's Service dropdown. Leave it
out and the SDK defaults to `unknown_service:<executable>`.
- **`OTEL_BSP_SCHEDULE_DELAY=1000`** drops the batch flush from its ~10 second default to ~1
second. For a demo this is the difference between "it works" and "it looks broken". Do not
use it in production — it trades export efficiency for latency.
- **`OTEL_METRICS_EXPORTER=none OTEL_LOGS_EXPORTER=none`** because `opentelemetry-distro`
defaults every signal to OTLP, and a traces-only backend like Jaeger answers the metrics
and logs exports with a stream of `StatusCode.UNIMPLEMENTED` noise.
## Privacy
`db.query.text` **is** recorded, truncated. SQL **parameter values are never recorded** — only
a parameter count. On a public Datasette instance the SQL text is user-supplied; if you export
to a third-party vendor, that text leaves your infrastructure.
See the telemetry section of the Datasette documentation for the full span and attribute
reference.

190
demos/otel/otlp_receiver.py Normal file
View file

@ -0,0 +1,190 @@
"""
A minimal OTLP/HTTP trace receiver, in about a hundred lines of Python.
Run it, point Datasette's OpenTelemetry agent at it, and get a real end-to-end
export - over the wire, in the real protobuf wire format - without Docker, a
collector, or Jaeger:
# terminal 1
uv run --with opentelemetry-proto python demos/otel/otlp_receiver.py
# terminal 2
OTEL_TRACES_EXPORTER=otlp \\
OTEL_METRICS_EXPORTER=none \\
OTEL_LOGS_EXPORTER=none \\
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \\
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \\
OTEL_SERVICE_NAME=datasette \\
OTEL_BSP_SCHEDULE_DELAY=1000 \\
uv run --with opentelemetry-distro \\
--with opentelemetry-exporter-otlp-proto-http \\
opentelemetry-instrument datasette mydb.db
(or `just receiver` and `just serve mydb.db` from this directory.)
Load a page, wait a second for the batch processor to flush, then press
Ctrl-C here for a summary.
This is a debugging aid, not a tracing backend: it does not persist anything,
speaks only OTLP/HTTP (not gRPC), and ignores metrics and logs. For anything
real, export to an actual backend instead.
"""
import gzip
import signal
import sys
import threading
from collections import Counter
from http.server import BaseHTTPRequestHandler, HTTPServer
try:
from opentelemetry.proto.collector.trace.v1.trace_service_pb2 import (
ExportTraceServiceRequest,
ExportTraceServiceResponse,
)
except ImportError:
sys.exit(
"This receiver needs the OpenTelemetry protobuf definitions:\n uv run --with opentelemetry-proto python demos/otel/otlp_receiver.py"
)
HOST = "127.0.0.1"
PORT = 4318
received = []
def attribute_value(value):
for field in ("string_value", "int_value", "double_value", "bool_value"):
if value.HasField(field):
return getattr(value, field)
return None
class OTLPHandler(BaseHTTPRequestHandler):
def log_message(self, *args):
pass # the default handler logs every request to stderr
def do_GET(self):
# For the person who opens http://localhost:4318 in a browser
# expecting a UI: there isn't one here, on Jaeger either - 4318 is
# where exporters POST protobuf. Jaeger's UI lives on :16686.
body = (
f"This is an OTLP/HTTP ingestion endpoint ({len(received)} spans "
"received so far).\n\n"
"There is no UI on this port - OpenTelemetry exporters POST "
"protobuf to /v1/traces here.\nThe span summary appears in the "
"terminal running this receiver when you Ctrl-C it.\n"
"For a real UI, run Jaeger instead (`just jaeger`) and open "
"http://localhost:16686\n"
).encode()
self.send_response(200)
self.send_header("Content-Type", "text/plain")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def do_POST(self):
body = self.rfile.read(int(self.headers.get("Content-Length", 0)))
if self.headers.get("Content-Encoding") == "gzip":
body = gzip.decompress(body)
request = ExportTraceServiceRequest()
request.ParseFromString(body)
batch = 0
for resource_spans in request.resource_spans:
for scope_spans in resource_spans.scope_spans:
for span in scope_spans.spans:
batch += 1
received.append(
{
"name": span.name,
"parent_id": span.parent_span_id.hex() or None,
"duration_ms": (
span.end_time_unix_nano - span.start_time_unix_nano
)
/ 1e6,
"attributes": {
a.key: attribute_value(a.value) for a in span.attributes
},
}
)
# flush=True so the live feedback survives being piped or redirected
print(f"received {batch} spans ({len(received)} total)", flush=True)
payload = ExportTraceServiceResponse().SerializeToString()
self.send_response(200)
self.send_header("Content-Type", "application/x-protobuf")
self.send_header("Content-Length", str(len(payload)))
self.end_headers()
self.wfile.write(payload)
def span_detail(span):
"The one attribute most worth showing next to a span name."
attributes = span["attributes"]
query = attributes.get("db.query.text")
if query:
return f" {query[:60]}"
status = attributes.get("http.response.status_code")
if status is not None:
return f" -> {status}"
return ""
def summarise():
if not received:
print("\nNo spans received.")
print("Remember: `opentelemetry-instrument` is required - Datasette core")
print("installs no provider - and the BatchSpanProcessor flushes about")
print("every 10s unless OTEL_BSP_SCHEDULE_DELAY says otherwise.")
return
print(f"\n=== {len(received)} spans ===")
for name, count in Counter(span["name"] for span in received).most_common():
total_ms = sum(s["duration_ms"] for s in received if s["name"] == name)
print(f" {name:<45}{count:>5} {total_ms:>9.2f}ms total")
slowest = sorted(received, key=lambda s: -s["duration_ms"])[:5]
print("\nslowest spans:")
for span in slowest:
print(f" {span['duration_ms']:>9.2f}ms {span['name']}{span_detail(span)}")
# Roots are one span per request (named "GET <route>"), datasette.startup,
# and any block=False write - those link back to their enqueuer rather than
# nesting under it, because the write can outlive the request that queued it.
roots = Counter(span["name"] for span in received if not span["parent_id"])
print(f"\n{sum(roots.values())} root spans:")
for name, count in roots.most_common():
print(f" {name} x{count}")
if __name__ == "__main__":
try:
server = HTTPServer((HOST, PORT), OTLPHandler)
except OSError as error:
sys.exit(
f"Could not listen on {HOST}:{PORT} ({error}).\n"
"Something else is already using the OTLP port - most likely an "
"earlier copy of this receiver, or Jaeger, still running."
)
print(f"OTLP/HTTP receiver listening on http://{HOST}:{PORT}")
print("Press Ctrl-C for a summary.")
# serve_forever() runs on a worker thread and the main thread just waits,
# so the summary still prints when this is launched through a wrapper such
# as `uv run`, where relying on KeyboardInterrupt alone is unreliable.
stop = threading.Event()
threading.Thread(target=server.serve_forever, daemon=True).start()
def request_stop(signum, frame):
stop.set()
signal.signal(signal.SIGINT, request_stop)
signal.signal(signal.SIGTERM, request_stop)
try:
stop.wait()
except KeyboardInterrupt:
pass
server.shutdown()
summarise()