Commit graph

3,460 commits

Author SHA1 Message Date
Alex Garcia
46961efee2 Add opentelemetry-api dependency and datasette/telemetry.py scaffolding
Datasette core is gaining OpenTelemetry spans alongside the existing
hand-rolled tracer. This commit only lays the groundwork - no span is
emitted yet.

Core takes a runtime dependency on opentelemetry-api and nothing more.
It deliberately never creates a TracerProvider, configures an exporter,
or touches sampling: that belongs to whoever runs Datasette, normally
via an opentelemetry-instrument agent. Owning a provider in core was
tried in an earlier design and produced a cross-request span leak, a
process-global provider that tests could not tear down, and a sampling
env var that silently blanked output. With no provider installed every
span is a NonRecordingSpan and costs approximately nothing.

datasette/telemetry.py exposes the module-level tracer plus
sql_attribute(), which truncates SQL to 2048 characters. On a public
instance the SQL is attacker-controlled and unbounded - someone can
paste a 10MB query into ?sql= - so it must never reach a telemetry
pipeline verbatim.

opentelemetry-sdk goes in the dev dependency group only, because the
test suite needs it to assert on spans while the package itself must
not import it. tests/test_telemetry.py enforces that by importing
datasette in a fresh interpreter and inspecting sys.modules, which
catches a lazy import inside a function body that a grep would miss.

conftest.py gains a session-scoped autouse fixture installing an SDK
provider with an InMemorySpanExporter. It has to be session-scoped
because set_tracer_provider() is effectively once-per-process - a
second call logs a warning and is ignored. SimpleSpanProcessor rather
than BatchSpanProcessor, so assertions made right after a request never
race a background export thread. The otel_spans fixture that later
tickets assert against is added here too.

test_datasette_package_never_imports_the_sdk is moved to the front of
the run. Late in a serial run the pytest process holds enough threads
that the fork half of subprocess' fork+exec segfaults the interpreter
on macOS/CPython 3.13. That reproduces with any subprocess call in that
position on an unmodified tree, so it is a pre-existing hazard rather
than something this commit introduces; the repo already moves its other
subprocess-spawning tests to the front for related reasons.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-14 08:54:15 -07:00
Simon Willison
b338c6f5f6
Migrate from httpx to httpx2, closes #2879
https://claude.ai/code/session_01Xdqoneq8ddvruVZETo6rFf
2026-09-10 19:44:49 -07:00
Simon Willison
61400fba1a Fix Docker release builds on main with Bookworm base image
Apply the Dockerfile fix from 0.65.x to avoid expired Bullseye security repository metadata.

Original fix: 91fa786de9
2026-09-10 18:52:19 -07:00
Simon Willison
186be52863 Publish stable and latest documentation databases to S3 2026-09-10 18:29:54 -07:00
Simon Willison
36acd1ea92 Publish packages when releases are published 1.0a39 2026-09-10 16:53:13 -07:00
Simon Willison
5e7cdaabbd Release 1.0a39 2026-09-10 16:52:52 -07:00
Simon Willison
92c7d4b608 Limit derived-table permissions to one source hop
Simplify the solution to 5de0c1724e - avoid contextvar.
2026-09-10 16:52:52 -07:00
Simon Willison
f70edbfa60 Filter incoming foreign-key relationships by view permission 2026-09-10 16:52:52 -07:00
Simon Willison
b97bb5f016 Reconcile write-timeout regression with the per-call limit 2026-09-10 16:52:26 -07:00
Simon Willison
e036907fc3 Reject structured row writes to virtual and shadow tables 2026-09-10 16:52:25 -07:00
Simon Willison
3f8d8417f6 Inherit source permissions for FTS vocabulary tables 2026-09-10 16:52:25 -07:00
Simon Willison
d334539a1e Deny SQLite statistics table access through a default hook 2026-09-10 16:52:25 -07:00
Simon Willison
628cec8f0c Block framing of stored-query mutation forms 2026-09-09 08:39:03 -07:00
Simon Willison
506c4bb522 Match table permission identities using SQLite case semantics 2026-09-09 08:39:03 -07:00
Simon Willison
e429bd2efa Reuse trusted magic parameter bindings for CSV exports 2026-09-09 08:39:03 -07:00
Alex Garcia
c6ba7b3298 Refuse API token creation from restricted actors 2026-09-09 08:38:40 -07:00
Simon Willison
ac2a9a43a5 Add a default execution time limit to writes 2026-09-08 21:17:03 -07:00
Simon Willison
9d3d741620 Require view permission before using row labels in flash messages 2026-09-08 21:16:36 -07:00
Simon Willison
ceef351622 Protect personalized dynamic responses from shared caching 2026-09-08 21:16:36 -07:00
Simon Willison
7e6039b8df Normalize URL column schemes consistently 2026-09-08 21:16:35 -07:00
Simon Willison
d43a04eb54 Authorize row resources before resolving primary keys 2026-09-08 21:16:35 -07:00
Simon Willison
8b10f58e1b Reject untrusted table-valued PRAGMA reads 2026-09-08 21:16:35 -07:00
Simon Willison
4b8f3b484d Keep private row and table responses out of shared caches 2026-09-08 21:16:35 -07:00
Simon Willison
1be4df77ac Reject invalid token expiry input 2026-09-08 21:16:35 -07:00
Simon Willison
6aa58bf4e5 Authorize configured full-text search targets 2026-09-08 21:16:35 -07:00
Simon Willison
22c601b3d0 Redact configuration keys case-insensitively 2026-09-08 21:16:35 -07:00
Simon Willison
e949ae46de Use unshadowable table classification 2026-09-08 21:16:35 -07:00
Simon Willison
a365903d56 Require view permission before returning written rows 2026-09-08 21:16:35 -07:00
Simon Willison
4c56ce2103 Escape identifiers in upsert row readback 2026-09-08 21:16:35 -07:00
Simon Willison
158c88f259 Escape primary-key cell values in row pages 2026-09-08 21:16:34 -07:00
Simon Willison
35232b5c37 Escape primary-key identifiers in row queries 2026-09-08 21:16:34 -07:00
Simon Willison
c01e95f3bd Filter foreign-key helper targets by view permission 2026-09-08 21:14:10 -07:00
Simon Willison
bf348a22fc Escape LIKE metacharacters in FTS detection 2026-09-08 21:12:23 -07:00
Simon Willison
59618371e9 Validate URL before rendering column links 2026-09-08 21:11:44 -07:00
Simon Willison
5de0c1724e Viewing derived table requires permission for both table and its source
Co-authored-by: Alex Garcia <15178711+asg017@users.noreply.github.com>
2026-09-08 18:09:23 -07:00
Simon Willison
d06737b6f4 Fix CREATE VIEW analysis on Python 3.10 2026-09-08 10:49:52 -07:00
Simon Willison
6473a7ecb0 Clearly document relationship between execute-sql and facets
Refs GHSA-5fff-xcm9-q6vh

Co-authored-by: Alex Garcia <15178711+asg017@users.noreply.github.com>
2026-09-08 10:49:52 -07:00
Simon Willison
f6d0f9bd38 detect_fts() now uses parameterized SQL
Refs GHSA-jcvx-2fh3-pjfp

Co-authored-by: Alex Garcia <15178711+asg017@users.noreply.github.com>
2026-09-08 10:48:20 -07:00
Simon Willison
c899beaebe escape_sqlite() against column names
Refs GHSA-jcvx-2fh3-pjfp

Co-authored-by: Alex Garcia <15178711+asg017@users.noreply.github.com>
2026-09-08 10:48:20 -07:00
Simon Willison
3ae092896d Only allow /db/name/-/schema against tables and views
Refs GHSA-926p-cw2f-643h

Co-authored-by: Alex Garcia <15178711+asg017@users.noreply.github.com>
2026-09-08 10:48:20 -07:00
Simon Willison
5d9a74f370 Check view-table as part of /-/schema and /db/-/schema
Refs GHSA-926p-cw2f-643h

Co-authored-by: Alex Garcia <15178711+asg017@users.noreply.github.com>
2026-09-08 10:48:19 -07:00
Simon Willison
01bf476d51 Require view-instance permission for /-/allowed
Refs GHSA-hp2x-vx2r-6vxg

Co-authored-by: Alex Garcia <15178711+asg017@users.noreply.github.com>
2026-09-08 10:48:19 -07:00
Simon Willison
4904249025 Use .textContent for column names in column chooser
Refs GHSA-hp2x-vx2r-6vxg

Co-authored-by: Alex Garcia <15178711+asg017@users.noreply.github.com>
2026-09-08 10:48:19 -07:00
Simon Willison
435e55ff0a Remove JSON syntax highlighting
Refs GHSA-hp2x-vx2r-6vxg

Co-authored-by: Alex Garcia <15178711+asg017@users.noreply.github.com>
2026-09-08 10:48:19 -07:00
Simon Willison
f8e8e65af7 actor cookie respects expire_after
Refs GHSA-53fc-rhfg-h7qp

Co-authored-by: Alex Garcia <15178711+asg017@users.noreply.github.com>
2026-09-08 10:48:19 -07:00
Simon Willison
577aeb73f0 Disallow ?_through= if user lacks view-table permission
Refs GHSA-53fc-rhfg-h7qp

Co-authored-by: Alex Garcia <15178711+asg017@users.noreply.github.com>
2026-09-08 10:48:19 -07:00
Simon Willison
c280c47424 POST /db/-/create checks table-level insert/update/alter permissions
Refs GHSA-53fc-rhfg-h7qp

Co-authored-by: Alex Garcia <15178711+asg017@users.noreply.github.com>
2026-09-08 10:48:19 -07:00
Simon Willison
4d0a2f2e84 execute-write: Check view-table for every table in a CREATE VIEW
Refs GHSA-53fc-rhfg-h7qp

Co-authored-by: Alex Garcia <15178711+asg017@users.noreply.github.com>
2026-09-08 10:48:19 -07:00
Simon Willison
c7944fc454 Skip deploy if environment variables are missing 2026-09-08 10:48:19 -07:00
Simon Willison
bdaa8cc76c Disable extension loading once --load-extension extensions are loaded
Refs GHSA-2mvv-ffvc-q5p6

Co-authored-by: Alex Garcia <15178711+asg017@users.noreply.github.com>
2026-09-08 10:48:19 -07:00