Commit graph

1,440 commits

Author SHA1 Message Date
Simon Willison
5d797878b9
Merge 8bf6e19643 into d0836ddedc 2026-10-12 18:18:59 +00:00
Simon Willison
e1a4e03635
Require Python 3.11, remove SQLite authorizer workaround
PR #2998
2026-10-12 10:36:07 -07:00
Simon Willison
8bf6e19643 Document connection budgets, resource errors, and lifecycle changes 2026-10-08 16:18:27 -07:00
Simon Willison
2e4bc5bed0 Expire SQLite blob handles and dump iterators with their leases 2026-10-08 15:36:09 -07:00
Simon Willison
c9e01076ea Bound database reads, writers, and queued work 2026-10-07 13:08:54 -07:00
Simon Willison
fb2b4a3ad8 Remove scratch database APIs from the connection proposal 2026-10-07 12:25:44 -07:00
Claude
da523c30e5
Torture harness, and the fixes it found
tests/connection_torture.py drives one Datasette from several event loops
(threads) with the smallest pool, 1ms idle timeout and 10ms schema polling
through every connection operation, plus add/remove_database, scratch
databases, external schema changes, replaced/deleted files, cancellation,
Database.close()/Datasette.close() mid-flight and non-threaded mode. One
scenario per process, so crashes and hangs are results. It checks for hangs,
crashes, undocumented exceptions, per-database write order, lost writes,
read-your-writes, catalog freshness, and fds/threads/the instance released
after close(). tests/test_connection_torture.py runs a short fixed-seed
subset of every scenario.

Fixes for what it found:
- A cursor that left the thread owning its connection (the cursor returned
  by execute_write_many()/execute_write_script(), one returned by a write
  callback, one rejected by execute_fn(), or one held by an exception's
  frames) could be freed on the event loop thread while the connection ran
  the same SQL again; on Python < 3.12 freeing it resets the shared cached
  statement, and the other query failed with "bad parameter or other API
  misuse". Such cursors are now closed on the owning thread. Pre-existing on
  main (51 failures per 1,800 multi-loop execute_write_many calls).
- invoke_startup() is single-flight across event loops (a concurrent.futures
  future, datasette.utils.inflight) instead of _startup_sequence() holding
  an asyncio.Lock: a second loop got "bound to a different event loop" or
  hung, and two loops could run the startup hooks twice. A request made
  from a startup hook no longer re-runs or deadlocks startup.
- The index page takes one snapshot of ds.databases: a database removed,
  renamed or closed mid-request, or whose file was deleted, no longer turns
  the page into a 500. Database.size is 0 for a deleted file.

The scan-coalescing helpers move to datasette/utils/inflight.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
2026-10-04 09:07:48 +00:00
Claude
d7a2ce91cc
Fix concurrency, lifecycle and contract problems from the second review
Schema watcher:
- Scan coalescing is loop-agnostic: in-flight scans are concurrent.futures
  futures any event loop can wait for, taken over only when their loop has
  closed or stopped; each refresh() resolves only its own scans. A write on
  one loop no longer hangs forever when another loop's scan is in flight.
- At most one polling task across event loops; replaced pollers exit.
- A scan or catalog write that fails transiently (internal database locked,
  shared-cache lock, cancellation) re-marks the database and retries it from
  later requests and sweeps, with back-off; requests are never failed by it.
  Write threads mark schema changes directly (under a lock) instead of
  relying on a notification reaching a loop that may have closed.
- add_database() file databases and named in-memory databases use the
  default_schema_watch mode (external unless configured). Named in-memory
  databases can be external: polled with PRAGMA schema_version on their
  write connection. They are not scanned at registration time, which raced
  plugins filling them (datasette-copy-to-memory).
- close() no longer persists a fingerprint for an owned database whose file
  header shows a schema the catalog has not seen.
- The derived-table permission rule (external-content FTS) trusts the
  catalog only when a stat and header read prove it is current; otherwise
  it asks the database, and schedules a rescan if the catalog is behind.

Connections and write threads:
- A BaseException from a write or isolated callback is delivered to the
  caller as a RuntimeError instead of killing the thread silently.
- conn.close() on a write lease raises ConnectionLeaseError, and the write
  thread reopens a connection something else closed.
- Write connections pin PRAGMA recursive_triggers=on (sqlite-utils FTS
  triggers rely on it; idle reopen lost the old drift).
- execute_fn() accepts generators that no longer hold the connection or a
  cursor.
- --crossdb _memory connections count against max_open_connections, are
  reaped, and are discarded when databases are added or removed.

Lifecycle:
- The temp internal database file uses weakref.finalize, not atexit, so
  closed or abandoned instances can be garbage collected.
- The read-pool reaper no longer keeps the last pool alive in its frame.
- The scratch directory is loaded (and locked) at the end of __init__, so a
  failed construction leaves nothing locked.
- config_dir/scratch is used only if empty or created by Datasette; a second
  instance on the same config dir starts without it; sidecar files are never
  deleted the first time a directory is opened.

Regression tests: tests/test_connection_review_fixes.py (24 of 27 fail on
a509b81; the other 3 guard behaviour that must not change).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
2026-10-04 08:32:09 +00:00
Claude
a509b81f2c
Connection budget tests for scratch databases
- Startup with 1000 scratch databases (threaded and num_sql_threads=0):
  Datasette() opens none of them; _startup_sequence() with a temporary
  internal database opens each exactly once (its catalog scan) with no
  pooled reads, writes or ATTACH; the index page, /-/databases.json,
  /-/api, /-/allowed, /-/jump, /-/check, /-/permissions and idle sweeps
  open none.
- Restart with a persisted internal database opens no scratch database,
  even for one written after its last scan (closing fingerprints).
- check_databases() only opens each scratch database for the catalog scan
  and does not fail on a broken scratch file.
- create, write, read, list, rename and delete touch only the scratch
  database concerned (list touches nothing); direct sqlite3.connect()
  calls are confined to the scratch directory (registry, WAL setup,
  rename checkpoint).
- docs: serve does not check scratch databases; listing pages do not open
  them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
2026-10-04 07:13:03 +00:00
Claude
c786d91cf9
Merge spike/budget into spike/final
Conflicts:
- database.py: keep both class attributes (is_scratch from scratch,
  immutable_count_time_limit_ms from budget).
- cli.py check_databases(): take budget's catalog-first check and keep
  scratch's rule that scratch databases are never checked (they are
  skipped in _databases_needing_check()); the --crossdb warning no longer
  counts scratch databases, which are never ATTACHed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
2026-10-04 07:00:25 +00:00
Claude
fc87ad9ccb
Database page from the catalog; allowed_resources().all() in one query
- The database page's columns, primary keys, FTS tables, hidden flags and
  foreign keys come from the _internal catalog (catalog_all_foreign_keys()
  added); only row counts query the database. Removes three round trips
  per table and the O(tables^2) hidden-table check from that page.
- PaginatedResources.all() fetches everything after the first page in a
  single query: each page re-ran the whole permission query, so iterating
  87,000 tables cost O(N^2 / 1000) (21 s of the index page at N=5000).
- catalog_generation is bumped after the per-database state it covers.
- Docs: index page row-count behaviour (pages.rst), immutable counts are
  computed on first use (performance.rst), catalog_tables.type and "pages
  that cover many databases read the catalog" (internals.rst).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
2026-10-04 06:22:57 +00:00
Claude
1bc15caef7
Connection budget: pages, startup and sweeps only touch the databases they need
Adds tests/connection_budget.py, a helper that records which databases a
block of code opens, reads, writes or ATTACHes, and tests/test_connection_budget.py
which asserts budgets for ~60 endpoints, startup, sweeps and telemetry with
50 attached databases.

Fixes for every all-database loop it found:

- Index page: tables, views, hidden tables, columns, primary keys, FTS
  tables and foreign-key relationship counts come from the _internal
  catalog (a few queries for any number of databases). Row counts are only
  run when at most 20 databases need counting (COUNT_MAX_DATABASES);
  above that the page shows table counts plus counts already known.
- Derived-table permissions: allowed_resources() no longer queries every
  database per page; dependency maps are computed by the SchemaWatcher from
  the rows it scans, cached per database, and read back from the catalog
  for databases restored from a persisted catalog.
- catalog_tables gains a type column (PRAGMA table_list: table, virtual,
  shadow) so hidden shadow tables can be identified without a connection.
- PaginatedResources.all() fetches pages of 1000 after the first.
- /-/api and /-/debug/autocomplete use the catalog; /-/schema does one
  permission query and skips databases with nothing visible.
- /-/versions computes SQLite details once per instance, without crossdb
  ATTACHes; detect_json1() caches its answer.
- /-/databases only stats/hashes databases the actor can see.
- Immutable table counts are computed when first needed, not at startup.
- check_databases() (serve, --get) reuses the catalog scan instead of
  opening every database a second time, and reports "file is not a
  database" as a usage error.
- SchemaWatcher: failed scans are retried when the file changes, not on
  every sweep; scan connections skip crossdb ATTACH; catalog_generation
  counter for caches.
- The SQL write UI no longer forces a stat sweep of every database.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
2026-10-04 05:58:31 +00:00
Claude
1f154f1aa4
Scratch databases: file-based, crash-consistent, survive restart
datasette.create_scratch_database() / delete_scratch_database() /
rename_scratch_database() / list_scratch_databases() manage SQLite files in
a scratch directory (--scratch-dir, config_dir/scratch, or a temporary
directory created on first use).

- Registry: a SQLite sidecar (.datasette-scratch.db) with intent states
  (creating/deleting/renaming); startup reconciles it with the directory,
  which is the source of truth (hand-copied files are adopted, hand-deleted
  ones dropped, orphan -wal/-shm/-journal removed). flock() per directory.
- Scratch databases are attached at startup without opening them, use
  schema watch mode "owned", and track last_used in memory (flushed to the
  registry at most once a minute and on close).
- Delete/rename: queued reads and writes fail with ScratchDatabaseDeleted,
  running ones finish; files are unlinked only once no connection
  (including the SchemaWatcher's, now counted) has them open.
- Datasette.close() records final fingerprints of owned file databases in a
  persistent internal database, so restarts reuse their catalog rows
  without opening them.
- Scratch databases are not attached to crossdb _memory and are skipped by
  check_databases().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
2026-10-04 05:56:03 +00:00
Claude
927cecaca9
Docs and telemetry descriptions for the integrated connection model
- internals.rst: new "Connections and threads" section (read pool, idle
  write threads, isolated connections, schema watching, deleted files,
  non-threaded mode, prepare_connection) and "Connections are only valid
  inside your function", which lists what the lease proxy cannot catch.
  execute_fn/execute_write_fn/execute_isolated_fn/close/add_database
  (schema_watch=) updated; the internal database is file-backed, and
  temporary tables only last as long as their connection.
- plugin_hooks.rst: prepare_connection runs once per new connection and
  connections come and go; write_wrapper's conn is leased.
- Metric and span descriptions no longer promise a "single write thread"
  that lives forever; connections.open says what it counts. Metric and
  span names are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
2026-10-04 04:53:50 +00:00
Claude
990c8782db
Merge spike/schema-watcher into spike/integrated, reconciling the three spikes
Read path: the leased pool replaces the thread-local cache. Pooled
connections record Database._conn_generation when opened; a stale one is
discarded on checkout or return, and _invalidate_connections() (called
by the SchemaWatcher on a replaced/deleted file) closes that database's
idle connections at once. Leased ones are only ever closed by their own
thread on release.

Write path: the idle-exit write thread keeps its lock protocol; its
connection lives in a _WriteConnection holder that is reopened when the
generation moves, before the next task. Post-write schema_version check
and _after_write catalog await are kept for every write API in both
modes. Non-threaded write connections are reopened on a generation change
and never cached half-prepared.

Deleted files: write connections use mode=rw once Datasette has seen the
file exist (_file_seen), so a deleted file is never silently recreated,
while a database added before its file exists is still created by its
first write.

Settings: max_open_connections (clamped to >= 4 x num_sql_threads),
connection_idle_timeout_ms (30000, read connections and write threads),
schema_watch_interval_ms, default_schema_watch (external|owned). The
spike-only knobs (connection_idle_timeout, connection_pool_wait_ms,
pool_read_connections, write_thread_idle_timeout_ms) are gone, along with
the pool's wait/disable code paths. The reaper falls back to checkout-time
reaping where threads cannot be started (Pyodide).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
2026-10-04 04:49:04 +00:00
Claude
3ec2c1a6a8 Merge spike/write-idle-thread into spike/integrated
Settings-list conflicts resolved as a union; the settings are
consolidated in a later commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
2026-10-04 04:40:21 +00:00
Claude
7a3c534ced
Spike: SchemaWatcher keeps the _internal catalog current off the request path
Per-database schema_watch modes (owned / external / immutable):
- write path reads PRAGMA schema_version after every write task and
  rebuilds that one database's catalog before the write returns
- external files are polled with a stat() fingerprint prefilter
  (racily-clean fingerprints re-verified), PRAGMA schema_version only on
  change, rebuild only when the version moved or the inode changed
- immutable databases are scanned once
- remove_database deletes its catalog rows explicitly; startup deletes
  rows for databases that are not attached (no per-refresh full scans)
- catalog reads use one short-lived connection per database inside one
  read transaction; shared-cache memory databases go through their
  write thread
- replaced/deleted files invalidate cached connections (generation)
- fingerprints persisted in catalog_databases.fingerprint (migration
  0002) so restarts skip unchanged files
- new setting schema_watch_interval_ms (default 1000, 0 = no polling)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
2026-10-04 03:18:16 +00:00
Claude
70326179a8
Spike: write threads exit after an idle timeout
New setting write_thread_idle_timeout_ms (default 30000, 0 disables).
A database's write thread now waits on queue.get(timeout=...) and, when
it times out, takes _write_thread_lock, re-checks that the queue is
empty, deregisters itself and closes its write connection. Producers
enqueue and check/start the thread under the same lock, so a task can
never be left in the queue with no thread to run it.

Also: close() refuses new tasks under the lock and joins retiring
threads; a failed connect is retried by the next write once the queue
drains; memory databases never idle out; a thread that dies from a
BaseException deregisters (and restarts if tasks are waiting); the
thread is registered only after start() succeeds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
2026-10-04 01:27:57 +00:00
Claude
791ce7a3d9
Spike: leased read-connection pool with global cap and idle reaper
Replace the per-thread cached read connections with a pool. Each
execute_fn()/execute() callback checks a connection out for its duration
and gets a LeasedConnection proxy that raises ConnectionLeaseError if
used after the callback returns.

New settings: max_open_connections (global LRU cap, default 128),
connection_idle_timeout (seconds, default 60, enforced by one
process-wide reaper thread plus opportunistic reaping on checkout),
connection_pool_wait_ms (soft cap: wait then exceed) and
pool_read_connections (off = fresh connection per query, a control).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
2026-10-04 01:26:01 +00:00
Simon Willison
cec5e6b2ef
Screenshots for reusable modal dialogs docs, plus just shots (#2958)
docs/shots.yml defines documentation screenshots for shot-scraper

"Just shots" runs them.
2026-09-24 15:06:57 -07:00
Simon Willison
8e17729ff3
Simplify code comments added since 1.0a40 (#2957)
Refs #2867
2026-09-24 13:52:29 -07:00
Simon Willison
90f2f1910f Improve OpenTelemetry security suggestions
Refs #2867
2026-09-24 12:12:01 -07:00
Simon Willison
0361afb12e Typo fixes 2026-09-24 12:09:29 -07:00
Simon Willison
aa8c2e36f7 Release 1.0a41
Refs #1515, #1730, #1980, #2012, #2446, #2790, #2858, #2867, #2948, #2949, #2951, #2952
2026-09-24 11:58:30 -07:00
Simon Willison
83640cf6e6 Simplified testing docs 2026-09-24 11:45:02 -07:00
Simon Willison
4d361cadcd More documentation simplification, refs #2867 2026-09-24 11:28:56 -07:00
Simon Willison
96d2baec32 More documentation work, refs #2867 2026-09-24 11:16:17 -07:00
Simon Willison
287e8ba3b2 Simplified telemetry descriptions in telemetry_registry.py
Told GPT-6 Astra to apply similar simplifications to the ones I used
when I hand-edited other parts of the docs.

Refs #2867
2026-09-24 11:04:41 -07:00
Simon Willison
1bba4d9a67 Simplified the docs for #2867 2026-09-24 10:47:31 -07:00
Alex Garcia
2588634358 Capstone review fixes: per-test reset, rename, provider guard, UpDownCounter, naming rules, privacy walk
Outcome of a whole-stack review with the kit visible as one system:

- otel_reset: an autouse fixture draining the span exporter and metric
  reader after every test. Without it a large suite accumulates hundreds
  of thousands of recorded spans in the session-scoped exporter - the
  likeliest amplifier of the slow-runner CI flakes - and plugins would
  inherit the same leak.
- assert_registry_covered renamed to assert_spans_covered: the old name
  read as covering the whole registry, which is exactly wrong next to
  assert_metrics_covered. Public API is forever; renamed before anything
  ships, no alias.
- The installers now verify their provider actually took: with a
  provider installed first (opentelemetry-instrument, an embedding app),
  set_*_provider() is silently ignored, and fixtures would assert
  against an exporter wired to nothing. They skip clearly instead.
- UPDOWN_COUNTER registry kind, mapped to Sum with monotonicity checked
  both ways - a Counter must collect monotonic, an UpDownCounter must
  not. Previously an UpDownCounter's kind check was silently skipped.
- The docs page now prescribes naming: scope = import package name
  (underscores), signal prefix = a name you own, never bare datasette.*;
  its own examples no longer teach the hyphenated outlier. Plus an
  observable-gauges pattern section and a prefix-overlap note.
- assert_no_forbidden_values(): the enforcement half of the privacy
  rules - plant sentinel secrets in a workload and assert they never
  appear in any span name, attribute, event, status description or
  metric attribute, across all scopes by default.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012U7coQfVu8nK2R4q2mCULA
2026-09-24 09:58:58 -07:00
Alex Garcia
29a2e6605d Extend the kit with metric-side conformance: kind, unit and enum checks
The five surveyed plugin plans all kept a hand-rolled metrics-vs-registry
diff because the kit's conformance helpers covered spans only. This adds
the metric side:

- metric_for() in the registry (the span_for analogue - no prefix/dynamic
  machinery, metric names are static), and the attribute helpers are
  documented as accepting MetricName entries.
- MetricsCollector.collect() now retains the instrumentation scope per
  collected metric, so a plugin is judged against its own meter only.
- assert_metrics_conform(): every collected metric in scope is registered,
  was created as the instrument kind and unit its registry entry declares
  (drift between the registry entry and the meter.create_*() call was
  previously caught by nothing, in core or any plugin), sets only
  registered attributes, and respects values= enums - the check that makes
  a metric dimension provably bounded.
- assert_metrics_covered(): every registered metric collected at least
  once with every non-optional attribute seen. Both *_covered helpers now
  exempt optional=True attributes, so a workload is not forced to
  manufacture every error path; pin those with targeted tests instead.
- datasette.operation declares values={"read", "write"} - core dogfoods
  the enum enforcement on the dimension where it matters most.
- Core's generic metric conformance tests are now calls to the kit
  helpers with scope_name="datasette"; the stricter literal-pinning and
  optional-attribute-coverage tests stay hand-written on purpose.
- The metric reference docs render attributes through the same helper as
  spans, so *(optional)* markers and enum values now appear there too.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012U7coQfVu8nK2R4q2mCULA
2026-09-24 09:58:58 -07:00
Alex Garcia
05ca128c02 Add a plugin telemetry kit: public registry API, linked_root_span_kwargs, test helpers, docs
A survey of five plugin OTel plans (datasette-paper, -agent, -litestream,
-accounts, -cron) found every one hand-copying the same core machinery:
the registry classes, the conformance-test harness, the pytest fixtures,
the bucket boundaries and the detached-root-with-Link recipe. This makes
that machinery importable instead:

- The registry classes are documented public API. Attribute gains
  values= (a closed enum the conformance helpers enforce - what makes an
  attribute safe as a metric dimension); SpanName gains prefix=True for
  span families like "chat {model}" whose names share a fixed prefix,
  matched by span_for() after exact names. span_for()/attribute helpers
  accept a spans= tuple so plugin registries can use them.
- datasette.telemetry.linked_root_span_kwargs(): the root-span-with-Link
  shape for work a request caused without containing - background jobs,
  scheduled ticks, block=False writes. Core's own write thread now uses
  it instead of building the kwargs inline.
- datasette.telemetry_testing: the session provider fixtures, otel_spans
  / otel_metrics, a two-way registry conformance checker (including enum
  and prefix handling, filtered by instrumentation scope) and an
  assert_package_never_imports_sdk() guard. Core's conftest now imports
  these instead of defining them, so the suite consumes the kit exactly
  as a plugin's would.
- New "Telemetry for plugin authors" docs page: scope discipline,
  registry usage, privacy/cardinality rules, named-callable guidance,
  request_span(), the background root-with-link convention (one root per
  tick, always emitted), provider-ordering facts and known caveats.
  request_span() is now documented public API.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012U7coQfVu8nK2R4q2mCULA
2026-09-24 09:58:58 -07:00
Alex Garcia
2cefab7b4c
Count callback-style calls in db.client.operation.duration
The callback entry points gained db.query spans in the database-spans PR;
this adds their other half - the duration histogram measurement, so a
plugin's execute_fn/execute_write_fn work and the JSON write API's inserts
and deletes stop being invisible to the one series that survives trace
sampling. execute_isolated_fn records "write" when the database is mutable
(the call blocks the write queue) and "read" when immutable (it runs on
the read pool). error.type comes from the raised exception class, same as
the SQL-string paths.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012U7coQfVu8nK2R4q2mCULA
2026-09-24 09:14:06 -07:00
Alex Garcia
8f49def8cb
Review polish: fix stale exemplar context, dedupe rationales, close test gaps
- The exemplars docs described "the pinned opentelemetry-exporter-prometheus"
  and "Datasette's own Prometheus exporter" - context from demo/plugin work
  that is no longer part of this stack. Reworded to stand alone.
- Saturate a num_sql_threads=1 pool and assert the queue-depth gauge reads
  above zero - the headline alerting metric previously only had an absence
  test, and this also pins the private ThreadPoolExecutor._work_queue
  attribute it depends on.
- Pin error.type on the write path of db.client.operation.duration - the
  write wrappers time a different code path than the read one already tested.
- Isolate the non-threaded-mode gauge test from other live instances instead
  of comparing global observation counts, which a GC pass could shift.
- Halve the metrics banner, point conftest's meter note at it, compact the
  interrupted-counter call-site comment to a registry pointer, note why
  instrument and registry descriptions are separate strings, and stop
  calling the metric dimension a "later phase" now that metrics shipped.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012U7coQfVu8nK2R4q2mCULA
2026-09-24 09:14:06 -07:00
Alex Garcia
22cdcd39a6
Document metric exemplars, which Datasette already emits
Histograms recorded inside a sampled span carry trace IDs automatically, so a
latency spike links to a trace that caused it. Nothing said so.

Two things are documented because they were measured rather than assumed: an
exemplar is kept per histogram bucket, so the bucket boundaries fixed earlier
in this stack took the same workload from one reachable trace to four; and the
pinned opentelemetry-exporter-prometheus drops exemplars entirely, so the path
that works is an OTLP collector rather than Datasette's Prometheus exporter.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

(cherry picked from 9d066255; section numbering and cross-references
adjusted to this branch's demo README, and the exemplar reference placed
as a subsection of the new Metric reference in internals.rst.)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F2h9ANGZ7paWSpqs5DUAcG
2026-09-24 09:14:06 -07:00
Alex Garcia
383fa1ad0a
Register metrics and give histograms bucket boundaries suited to seconds
Both histograms declared unit="s" but inherited OpenTelemetry's default
boundaries, which are tuned for milliseconds - so every SQLite query
landed in the single (0, 5] second bucket and every quantile query
returned noise.

The boundaries are the semantic conventions' recommended set for
db.client.operation.duration plus 0.0001 and 0.0005 at the bottom, since
SQLite is in-process and many real queries take tens of microseconds.

(Adapted from 024f2029: that commit assumed the metrics were already in
telemetry_registry.py, which on this lineage held spans only - so this
commit also brings the MetricName registry machinery, the registry
entries for all eight phase-3 metrics, the cog-generated Metric
reference in internals.rst, and the datasette.operation attribute. The
template and facet histograms it also touched belong to phase 5 and are
not included.)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F2h9ANGZ7paWSpqs5DUAcG
2026-09-24 09:14:06 -07:00
Alex Garcia
90b727db73
Add OpenTelemetry metrics for SQL thread pool saturation and query latency
Spans describe requests that have finished. They structurally cannot answer
"am I saturating my 3 SQL threads right now", because that is a level rather
than an event - and with num_sql_threads defaulting to 3, it is usually the
first thing worth knowing about a busy Datasette. This adds the metrics that
answer it.

Five observable gauges, computed only when something is collecting, so an
instance with no MeterProvider installed does no work for them at all:

  datasette.sql.threads.limit         num_sql_threads
  datasette.sql.threads.queue_depth   queries waiting for a free thread
  datasette.sql.queries.pending       in-flight reads, by db.namespace
  datasette.write.queue_depth         writes behind the single write thread
  datasette.connections.open          tracked file connections

Three instruments recorded inline, which matters because metrics survive
trace sampling and spans do not - an operator sampling 1% of traces still
gets 100% of the latency distribution:

  db.client.operation.duration        semconv histogram, with error.type
  datasette.write.queue_wait          the metric twin of the existing span
  datasette.sql.queries.interrupted   sql_time_limit_ms kills

The interrupted counter closes a gap the plan called out as unanswerable:
"how often are we killing queries at the limit" is a rate, and a rate cannot
be recovered from sampled spans.

Core still creates no provider of any kind, so the architecture is unchanged;
`grep -rn 'opentelemetry.sdk' datasette/` stays empty. One real difference
from tracing is worth recording: _ProxyMeter and its instruments forward to a
provider installed after they were created, whereas ProxyTracer permanently
caches the first concrete tracer it resolves. Module-level instruments are
therefore safe and the test fixture has no ordering constraint.

Live instances are tracked in a lock-guarded WeakSet so instrumenting an
instance never keeps it alive. The pool gauges carry no attribute saying
which Datasette produced them: production runs one instance per process, and
adding an id to disambiguate the test suite's hundreds of instances would buy
unbounded attribute cardinality to fix a case that does not occur. The
collision is documented instead, and the gauge callbacks are plain generator
functions so tests can assert exact values by calling them directly rather
than through the SDK's last-value aggregation.

demos/otel/metrics_demo.py fires 12 concurrent 40ms queries at a 3-thread
pool and samples the gauges mid-flight: queue_depth peaks at exactly 9, and
the duration histogram reads max=0.1695s for a query whose work is 40ms. That
gap is the queue, and it is the thing traces alone will not show you.

Also corrects the demo README's privacy section, which still claimed
parameter values are never recorded - that stopped being unconditionally true
when trace_sql_parameters landed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

(cherry picked from 6ef0dd8c and adapted to the rebuilt phase-1 stack:
attribute names now come from telemetry_registry where entries exist, the
meter carries the instrumentation-scope version and schema URL, and the
interrupted-queries counter skips expected timeouts - callers that opted
into a deliberately short budget, like facet suggestion - matching how
those are excluded from span error status. The internals.rst reference
lands with the registry commit that follows.)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F2h9ANGZ7paWSpqs5DUAcG
2026-09-24 09:14:06 -07:00
Alex Garcia
9bce2eec7c
Emit an OpenTelemetry span for each HTTP request
PR #2863
2026-09-24 09:14:03 -07:00
Alex Garcia
daf1989ebe
OpenTelemetry tracing for the database layer (#2862)
PR #2862
2026-09-24 09:11:08 -07:00
Simon Willison
70be140197 Human edit round of dialog JS docs, refs #2790 2026-09-17 16:11:31 -07:00
Simon Willison
3e9356dde0 Rename modal trigger option to returnFocusTo, refs #2790 2026-09-17 15:46:34 -07:00
Simon Willison
ad1d0bf6f9 Rename modal dismissal reason to source, refs #2790 2026-09-17 15:32:47 -07:00
Simon Willison
269c043da3 Share scrolling dialog body styles with modal-body, refs #2790 2026-09-17 14:42:14 -07:00
Simon Willison
0ad118ba26 Clarify focus restoration when reopening a modal, refs #2790 2026-09-17 14:18:22 -07:00
Simon Willison
8220413a8a Keep modal documentation in the JavaScript plugin docs, refs #2790 2026-09-17 14:13:17 -07:00
Simon Willison
71600f1c0a Simplify shared modals now that all dialogs use the document, refs #2790 2026-09-17 14:10:45 -07:00
Simon Willison
e1f89494f1 Add reusable modal component, documentation and lifecycle tests, refs #2790 2026-09-17 13:28:23 -07:00
Simon Willison
3239d10b9b Use cog to show default values in settings.rst 2026-09-17 12:30:47 -07:00
Simon Willison
ffd3b8cada Refactor internal docs to add Background tasks section
Previously information about background tasks was mixed in with
method descriptions for the Datasette class.
2026-09-16 17:47:23 -07:00
Simon Willison
b661889b62 Adjust headings so Request object has three sub-headings 2026-09-16 17:46:16 -07:00