From 8985ecf4387164e76147e7188693a08dc1a7dd1e Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 1 Jul 2026 22:32:28 +0000 Subject: [PATCH 001/131] Add JSON API reference and 1.0 stability review documents - existing-api.md: complete reference for the JSON API as implemented, derived from source code (routes, views, renderer, permissions) - stable-api-recommendations.md: consistency and completeness review with prioritized recommendations for the 1.0 stable release Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- existing-api.md | 1211 +++++++++++++++++++++++++++++++++ stable-api-recommendations.md | 367 ++++++++++ 2 files changed, 1578 insertions(+) create mode 100644 existing-api.md create mode 100644 stable-api-recommendations.md diff --git a/existing-api.md b/existing-api.md new file mode 100644 index 00000000..b3eda9d4 --- /dev/null +++ b/existing-api.md @@ -0,0 +1,1211 @@ +# Datasette JSON API — As Implemented + +This document describes the JSON API of this Datasette codebase (version `1.0a35`) as +derived directly from the source code. It intentionally ignores the existing `docs/` +directory: every claim below is based on the route table in `datasette/app.py` +(`Datasette._routes()`, app.py:2507-2767) and the view implementations in +`datasette/views/`. + +## Contents + +- [Cross-cutting behavior](#cross-cutting-behavior) +- [Instance endpoints](#instance-endpoints) +- [Database endpoints](#database-endpoints) +- [Table and row read endpoints](#table-and-row-read-endpoints) +- [The write API](#the-write-api) +- [Stored (canned) queries API](#stored-canned-queries-api) +- [Authentication and tokens](#authentication-and-tokens) +- [Appendix: registered actions (permissions)](#appendix-registered-actions-permissions) + +--- + +## Cross-cutting behavior + +### URL formats and content negotiation + +- Most read endpoints are registered with an optional format suffix: + `/(...)(\.(?Pjson))?$`. The bare path returns HTML; the `.json` + extension returns JSON. The homepage additionally accepts the legacy + `.jsono` extension, which returns identical JSON (app.py:2517-2518). +- Table, row and query routes accept any `\w+` format extension; formats other + than the built-in `html`, `json`, `csv`, `blob` must be provided by a plugin + via `register_output_renderer`, otherwise the request 404s. +- HTML responses include a `Link: <...>; rel="alternate"; + type="application/json+datasette"` header pointing at the `.json` variant + (views/base.py:141-159), unless the view opts out with + `has_json_alternate = False`. +- Database, table, row and query names in paths are **tilde-encoded** + (a percent-encoding variant using `~` as the escape character; + utils/__init__.py `_TILDE_ENCODING_SAFE`). Multi-column primary keys in row + URLs are comma-separated. +- JSON responses are always compact `json.dumps` output serialized by + `CustomJSONEncoder`; there is no pretty-printing query parameter. Binary + values are serialized as `{"$base64": true, "encoded": "..."}`. +- Success content type: `application/json; charset=utf-8` + (`_shape=array&_nl=on` responses use `text/plain`). + +### Error shapes (there are several) + +The codebase produces **four distinct JSON error shapes**, depending on which +layer generates the error: + +1. **Exception handler** (handle_exception.py:21-59) — used when a view raises + `NotFound`, `Forbidden` (JSON paths only — see below), `DatasetteError`, + `BadRequest` etc. and the request path ends in `.json`: + + ```json + {"ok": false, "error": "message", "status": 404, "title": null} + ``` + +2. **The `_error()` helper** (views/base.py:183-184) — used by the write API, + stored-query API, execute-write and several permission-denied paths: + + ```json + {"ok": false, "errors": ["message", "..."]} + ``` + + Note: plural `errors`, a list, and no `status`/`title` keys. + +3. **JSON renderer errors** (renderer.py:52-56) — SQL errors on table/query + endpoints return HTTP 400 with the error embedded in the data envelope: + + ```json + {"ok": false, "error": "no such table: x", "rows": [], "truncated": false} + ``` + + An invalid `_shape=` value produces `{"ok": false, "error": "Invalid _shape: x", + "status": 400, "title": null}` (renderer.py:101-108). + +4. **Ad-hoc `{"error": ...}` objects** — the permission debug endpoints + (`/-/allowed`, `/-/rules`, `/-/check`, POST `/-/permissions`) return e.g. + `{"error": "Unknown action: x"}` with no `ok` key (views/special.py). + +Method-not-allowed responses return HTTP 405 +`{"ok": false, "error": "Method not allowed"}` when the path ends in `.json` +or the request content type is `application/json`; plain text otherwise +(views/base.py:53, 88-98). + +**`Forbidden` is special:** when a view raises `Forbidden` (e.g. via +`ensure_permission`), the default `forbidden()` plugin hook renders an **HTML +error page with status 403 even for `.json` requests** +(forbidden.py:4-19, app.py:2895-2904). Endpoints that check permissions +themselves and return `_error(..., 403)` produce JSON instead. So a JSON +client may receive either an HTML 403 page or a JSON 403 body depending on +the endpoint. + +### CORS + +When Datasette is started with `--cors`, responses gain +(utils/__init__.py:1297-1302): + +``` +Access-Control-Allow-Origin: * +Access-Control-Allow-Headers: Authorization, Content-Type +Access-Control-Expose-Headers: Link +Access-Control-Allow-Methods: GET, POST, HEAD, OPTIONS +Access-Control-Max-Age: 3600 +``` + +### CSRF / cross-origin protection + +Datasette uses header-based cross-origin protection +(`CrossOriginProtectionMiddleware`, csrf.py:67-178) rather than CSRF tokens +for API calls. For non-GET/HEAD/OPTIONS requests: + +1. Requests carrying `Authorization: Bearer ...` **and no `Cookie` header** + bypass the check entirely (csrf.py:98-110). +2. Otherwise `Sec-Fetch-Site` must be `same-origin` or `none`; other values → 403. +3. If neither `Sec-Fetch-Site` nor `Origin` is present (curl, API clients), + the request passes. +4. Fallback: `Origin` must exactly match the request scheme/host/port → else 403. + +Plain JSON API clients (no cookies, no browser headers) are never blocked; +`Content-Type: application/json` itself plays no role in the CSRF decision. + +### Settings that govern the API + +From `SETTINGS` (app.py:197-287): `default_page_size` (100), +`max_returned_rows` (1000), `max_insert_rows` (100), `sql_time_limit_ms` +(1000), `default_facet_size` (30), `facet_time_limit_ms` (200), +`allow_facet` (true), `allow_download` (true), `allow_signed_tokens` (true), +`default_allow_sql` (true), `max_signed_tokens_ttl` (0), `default_cache_ttl` +(5), `allow_csv_stream` (true), `max_csv_mb` (100), `force_https_urls` +(false), `trace_debug` (false), `base_url` ("/"). + +### The JSON renderer: `_shape`, `_nl`, `_json`, `_json_infinity` + +`json_renderer` (renderer.py:31-126) processes `.json` output for table, row +and query views (but **not** for the instance/database/debug endpoints, which +build JSON directly): + +- **`_shape`** (default `objects`): + - `objects` — `{"ok": true, "rows": [{col: val}, ...], "truncated": false, ...}` + - `arrays` — same envelope, each row a list of values + - `array` — response body is a bare JSON array of row objects + - `arrayfirst` — bare JSON array of the first column's values + - `object` — table views only: an object keyed by primary-key string. + On queries: `{"ok": false, "error": "_shape=object is only available on + tables"}` (with HTTP status 200); on tables without primary keys a similar + error. + - anything else — HTTP 400 `{"ok": false, "error": "Invalid _shape: x", + "status": 400, "title": null}` +- **`_nl=on`** — with `_shape=array` only: newline-delimited JSON, `text/plain`. +- **`_json=COLUMN`** (repeatable) — parse that column's string values with + `json.loads` so they nest as JSON; parse failures leave the value unchanged. +- **`_json_infinity=1`** — preserve `Infinity`/`-Infinity`; by default they + are replaced with `null`. +- `columns` is stripped from dict-shaped output unless `?_extra=columns` was + requested (renderer.py:110-113). +- If a SQL error occurred, `_shape` is ignored, HTTP status is 400 and the + envelope carries `"ok": false, "error": ...` (renderer.py:52-56). + +### The `?_extra=` system + +Table, row and query JSON responses support `?_extra=` (repeatable and/or +comma-separated, extras.py:9-14) to add keys to the response. Extras are +scope-registered (`ExtraScope.TABLE` / `ROW` / `QUERY`) and only **public** +extras are available over JSON (extras.py:73-92). Unknown extra names are +silently ignored. The available names per scope are listed with the relevant +endpoints below. + +--- + +## Instance endpoints + +Most of these are implemented with `JsonDataView` (views/special.py:30-79): +GET-only; bare path renders an HTML page (`show_json.html`), `.json` returns +the data; permission defaults to `view-instance` and denial raises +`Forbidden` → **HTML** 403 page. + +### GET / + +Routes: `/(\.(?Pjsono?))?$` and `/-/(\.(?Pjsono?))?$` +(app.py:2517-2518); `/-` permanently redirects to `/-/`. `IndexView` +(views/index.py:22-189). `GET /.json`, `/.jsono` and `/-/.json` return JSON. + +- **Permission:** `view-instance` (denied → 403). Databases and tables are + further filtered by `view-database` / `view-table` for the actor. +- **Parameters:** `_sort=relationships` sorts each database's truncated table + list by foreign-key relationship count. +- **JSON response** (index.py:147-161): + - `databases` — an **object keyed by database name** (not a list). Each + value: `name`, `hash` (or null), `color`, `path`, + `tables_and_views_truncated` (up to 5 items: `name`, `columns`, + `primary_keys`, `count` (int or null), `hidden`, `fts_table`, + `num_relationships_for_sorting`, `private`; view items are just + `{"name", "private"}`), `tables_and_views_more` (bool), `tables_count`, + `table_rows_sum`, `show_table_row_counts`, `hidden_table_rows_sum`, + `hidden_tables_count`, `views_count`, `private`. + - `metadata` — instance metadata object. + +### GET /-/versions(.json) + +`JsonDataView` over `Datasette._versions` (app.py:2548-2551, 2171-2245). +Permission `view-instance`. No parameters. + +Response keys: `python` (`{version, full}`), `datasette` (`{version}` plus +optional `note`), `asgi` (`"3.0"`), `uvicorn` (string or null), `sqlite` +(`{version, fts_versions, extensions, compile_options}`; `extensions` +includes `json1` and optionally `spatialite`), `pysqlite3` (only when +running under pysqlite3). + +### GET /-/plugins(.json) + +app.py:2552-2557, `Datasette._plugins` (app.py:2247-2266). Permission +`view-instance`. + +- **Parameters:** `?all=1` — include Datasette's built-in default plugins + (filtered out by default). +- **Response:** a JSON **array**, sorted by name, of + `{"name", "static", "templates", "version", "hooks"}`. + +### GET /-/settings(.json) + +app.py:2558-2561. Permission `view-instance`. No parameters. Returns a flat +object mapping every setting name (see [Settings](#settings-that-govern-the-api)) +to its effective value. + +### GET /-/config(.json) + +app.py:2562-2565. Permission `view-instance`. No parameters. Returns the full +`datasette.yaml` configuration dict passed through +`redact_keys(config, ("secret", "key", "password", "token", "hash", "dsn"))` +(app.py:2502-2505) — any dict key containing one of those substrings has its +value replaced by `"***"` (utils/__init__.py:1532-1556). + +### GET /-/threads(.json) + +app.py:2566-2569, `Datasette._threads` (app.py:2268-2285). Permission +`view-instance`. No parameters. + +Response: `num_threads`, `threads` (list of `{name, ident, daemon}`), +`num_tasks`, `tasks` (asyncio task repr strings). When the +`num_sql_threads` setting is 0 the response is exactly +`{"num_threads": 0, "threads": []}`. + +### GET /-/databases(.json) + +app.py:2570-2573, `Datasette._connected_databases` (app.py:2157-2169). +Permission `view-instance`. No parameters. + +Response: a JSON array of `{"name", "route", "path", "size", "is_mutable", +"is_memory", "hash"}` — **all attached databases are listed regardless of +per-database `view-database` permissions**. + +### GET /-/actor(.json) + +app.py:2574-2579, registered with `permission=None` — **accessible to any +request including anonymous**. No parameters. + +Response: `{"actor": {...}}` or `{"actor": null}` (app.py:2287-2288). + +### GET /-/actions(.json) + +app.py:2580-2589. Permission **`permissions-debug`**. No parameters. + +Response: a JSON array, sorted by name, of `{"name", "abbr", "description", +"takes_parent", "takes_child", "resource_class", "also_requires"}` +(app.py:2290-2304). + +### GET /-/auth-token + +`AuthTokenView` (app.py:2590-2593, views/special.py:198-217). GET only, no +`.json` variant, HTML/redirect only. + +- **Parameter:** `token` — the one-time secret printed by `datasette --root`. +- Match → invalidates the token, sets the signed `ds_actor` cookie to + `{"id": "root"}` and 302-redirects to the homepage. Mismatch or reuse → + `Forbidden` → 403 HTML. + +### GET/POST /-/create-token + +`CreateTokenView` (app.py:2594-2597, views/special.py:727-856). **HTML form +endpoint only — there is no JSON request/response mode in this codebase** +(`has_json_alternate = False`; the POST body must be form-encoded, a JSON +content type raises `BadRequest` → 400). + +- **Gates** (each failure → `Forbidden` → 403): `allow_signed_tokens` must be + on; request must have an actor with an `id`; the actor must not itself be + token-derived. +- **POST fields:** `expire_type` (`""`/`minutes`/`hours`/`days`), + `expire_duration` (positive int), plus restriction checkboxes named + `all:`, `database::`, + `resource:::`. +- **Response:** HTML page containing the new `dstok_` token. +- Programmatic alternatives: `datasette create-token` CLI or + `datasette.create_token()`. + +### GET /-/api + +`ApiExplorerView` (app.py:2598-2601, views/special.py:859-1020). HTML API +explorer, GET only. Permission `view-instance` (403 on denial). + +### GET /-/jump(.json) + +`JumpView` (app.py:2602-2605, views/special.py:1023-1201). The route allows +an optional `.json` suffix but the view **always returns JSON**. + +- **Permission:** none checked directly; results are filtered via + `allowed_resources_sql` for the current actor (default items come from the + `jump_items_sql` plugin hook). +- **Parameter:** `q` — whitespace-split terms matched as a case-insensitive + `%term1%term2%` LIKE pattern. +- **Response:** `{"matches": [...], "truncated": bool}`; each match: + `name`, `url`, `type` (`database`/`table`/`view`/`query`/plugin-defined), + `description`, optional `display_name`. Capped at 100 matches. + +### GET /-/schema(.json|.md) + +`InstanceSchemaView` (app.py:2610-2613, views/special.py:1257-1293). + +- **Permission:** no explicit check; only databases the actor can + `view-database` are included (others silently omitted). +- **Formats:** no extension → HTML; `.json` → + `{"schemas": [{"database": name, "schema": "..."}]}`; `.md` → + `text/markdown` rendering. + +### GET/POST /-/logout + +`LogoutView` (app.py:2614-2617, views/special.py:220-238). HTML endpoint. +GET renders a confirmation page (or redirects if anonymous); POST deletes the +`ds_actor` cookie and 302-redirects to `/`. + +### GET/POST /-/permissions + +`PermissionsDebugView` (app.py:2618-2621, views/special.py:241-295). No +`.json` route. Both methods require `view-instance` **and** +`permissions-debug` (403 on denial). + +- **GET** — HTML permission-check log; `?filter=all|exclude-yours|only-yours`. +- **POST** — form-encoded `actor` (JSON string), `permission`, optional + `resource_1`, `resource_2`; returns **JSON** + `{"action", "allowed", "resource": {"parent", "child", "path"}}` plus + `actor_id` when present. Errors: unknown action → 404 `{"error": ...}`; + child without parent → 400 `{"error": ...}`. + +### GET /-/allowed(.json) + +`AllowedResourcesView` (app.py:2622-2625, views/special.py:298-460). Bare +path always renders the HTML form; `.json` returns JSON. + +- **Permission:** none — reports the **current actor's own** allowed + resources. Items gain a `reason` field if the actor also holds + `permissions-debug`. +- **Parameters:** `action` (required; missing → 400 `{"error": ...}`, unknown + → 404), `parent`, `child` (requires `parent`), `page` (default 1), + `page_size` (default 50, silently capped at 200). +- **Response:** `{"action", "actor_id", "page", "page_size", "total", + "items": [{"parent", "child", "resource"}]}` with optional `next_url` / + `previous_url`. + +### GET /-/rules(.json) + +`PermissionRulesView` (app.py:2626-2629, views/special.py:463-584). +Permission `view-instance` **and** `permissions-debug`. Parameters and error +shapes as `/-/allowed`. Response items: +`{"parent", "child", "resource", "allow" (1|0), "reason", "source_plugin"}`. + +### GET /-/check(.json) + +`PermissionCheckView` (app.py:2630-2633, views/special.py:633-662). +Permission `permissions-debug`. Parameters `action` (required), `parent`, +`child`. Checks the **current request's actor**; response +`{"action", "allowed", "resource": {...}}` plus `actor_id`. + +### GET/POST /-/messages + +`MessagesDebugView` (app.py:2634-2637, views/special.py:703-724). HTML debug +tool for flash messages; permission `view-instance`; POST is form-encoded +(`message`, `message_type` = INFO/WARNING/ERROR/all) and 302-redirects. + +### GET /-/allow-debug + +`AllowDebugView` (app.py:2638-2641, views/special.py:665-700). GET only, HTML +only, **no permission required**. Parameters `actor` and `allow` (JSON +strings); renders the result of `actor_matches_allow()` in the page. + +### GET /-/patterns + +Pattern portfolio page (app.py:2642-2645). HTML only; not part of the JSON API. + +### GET /-/debug/autocomplete + +`AutocompleteDebugView` (app.py:2646-2649, views/special.py:94-195). HTML +debug page for the table autocomplete API; permission `view-instance` plus +`view-table` when `?database=&table=` are supplied. + +--- + +## Database endpoints + +### GET /\.db + +Downloads the raw SQLite file. Route → `database_download` +(app.py:2650-2653; views/database.py:533-570). + +- **Permission:** `view-database-download` (denied → `Forbidden` → 403 HTML). +- **Other gates:** unknown database → 404 `"Invalid database"`; in-memory + database → 404; `allow_download` off **or** mutable database → + `Forbidden("Database download is forbidden")`; no file path → 404. +- **Response:** streamed `application/octet-stream` with a + `content-disposition` attachment; immutable databases with a known hash set + `Etag` and honor `If-None-Match` → 304. + +### GET /\(.json) + +`DatabaseView` (app.py:2654-2657; views/database.py:71-277). Only `html` and +`json` formats are accepted; any other extension → 404 `"Invalid format: ..."`. + +- **Permission:** `view-database` via `check_visibility` (denied → + `Forbidden` → 403 HTML). Table/view listings are filtered by `view-table`; + stored queries by `view-query`. +- **Parameters:** + - `?sql=` — non-blank value 302-redirects to `//-/query?...` + preserving the query string and format. + - No `?_extra=` and no `_shape` support — the JSON is built directly and + returned via `Response.json`, bypassing the JSON renderer + (views/database.py:189-212). +- **JSON response** (all keys always present): + - `ok` — always `true` + - `database` — name; `private` — bool; `path` — URL path; `size` — bytes + - `tables` — list (includes hidden tables), each: + `name`, `columns` (names), `primary_keys`, `count` (int or null, + time-boxed), `count_truncated` (bool — count is a capped lower bound), + `hidden`, `fts_table`, `foreign_keys` (`{incoming: [...], outgoing: [...]}` + of `{other_table, column, other_column}`), `private` + - `hidden_count` — number of hidden tables + - `views` — list of `{name, private}` + - `queries` — **up to 5** stored queries (canonical stored-query objects, + see the stored-queries section); `queries_more` (bool); + `queries_count` (total visible) + - `allow_execute_sql` — bool for this actor + - `table_columns` — `{table: [columns]}`, empty `{}` unless + `allow_execute_sql` (views map to `[]`) + - `metadata` — database metadata dict + +### GET /\/-/query(.json) — arbitrary SQL + +`QueryView` (app.py:2691-2694; views/database.py:573-1130). The same class +also executes stored queries dispatched from the table route (see stored +queries section). + +- **Permission:** `execute-sql` on the database via `check_visibility` + (denied → `Forbidden` → 403 HTML). +- **Parameters:** + - `sql` — SQL to run. Must pass `validate_sql_select` + (utils/__init__.py:345-354): after stripping `--` comment lines it must + start with `select`, `with` or an `explain` variant, and must not contain + `pragma` (except allowlisted `pragma_*()` table-valued functions). + Failure → 400 `DatasetteError` titled `"Invalid SQL"` → JSON + `{"ok": false, "error": "Statement must be a SELECT", "status": 400, + "title": "Invalid SQL"}`. + - Any other `name=value` pair supplies the `:name` named parameter; missing + parameters default to `""`. Names starting with `_` are excluded. + - `_timelimit` — per-request SQL time limit in ms. + - `_shape`, `_nl`, `_json`, `_json_infinity` — see the JSON renderer section. + - `_extra` — QUERY-scope extras: `columns`, `debug`, `request`, + `render_cell`, `query` (`{"sql", "params"}`), `metadata`, `database`, + `database_color`, `private`, `extras`. +- **Response** (default shape): + `{"ok": true, "rows": [{col: val}, ...], "truncated": false}` plus any + requested extras. `truncated: true` when the result hit `max_returned_rows`. +- **Errors:** + - SQLite errors (e.g. `no such table`) are **not** raised — they surface as + HTTP 400 `{"ok": false, "error": "", "rows": [], "truncated": false}`. + - Time limit → 400 titled `"SQL Interrupted"` (the `error` value contains + an HTML fragment). + - `?sql=` omitted → 200 `{"ok": true, "rows": [], "truncated": false}` + (the CSV format instead errors 400 `"?sql= is required"`). +- `.csv` streams CSV; unknown extensions → 404. + +### GET /\/-/query/parameters + +`QueryParametersView` (app.py:2687-2690; views/stored_queries.py:26-51). + +- **Permission:** `execute-sql` → 403 JSON + `{"ok": false, "errors": ["Permission denied: need execute-sql"]}`. +- **Parameters:** only `sql` (default `""`); any other key → 400 + `"Invalid keys: ..."`. +- **Response:** 200 `{"ok": true, "parameters": ["name1", ...]}`. SQL with a + parameter beginning `_` → 400 `"Magic parameters are not allowed"`. +- Responses carry `Content-Security-Policy: frame-ancestors 'none'` and + `X-Frame-Options: DENY`. + +### POST /\/-/create + +`TableCreateView` (app.py:2658; views/table_create_alter.py:785-962). +GET → 405. Body is parsed as JSON regardless of content type; invalid JSON → +400 `{"ok": false, "errors": ["Invalid JSON: ..."]}`. + +- **Permissions** (all denials → 403 `{"ok": false, "errors": [...]}`, + all checked at the **database** level): + - `create-table` — always required (`["Permission denied"]`) + - `insert-row` — if `rows`/`row` provided (`need insert-row`) + - `update-row` — if `replace: true` (`need update-row`) + - `alter-table` — if `alter: true` on an **existing** table + (`need alter-table`); when the table does not exist yet and rows are + supplied, alter is enabled automatically. +- **Request schema** (pydantic `CreateTableRequest`, extra keys forbidden → + 400 `"Invalid keys: a, b"`): + - `table` (required) — must match `^(?!sqlite_)[^\n]+$` + - `rows` (list of objects) / `row` (single object) — mutually exclusive + - `columns` — list of `{name, type, fk_table, fk_column, not_null, + default, default_expr}`; mutually exclusive with `rows`/`row`; `type` one + of `text`/`integer`/`float`/`blob` (default `text`); `default` and + `default_expr` mutually exclusive; `default_expr` one of + `current_timestamp`, `current_date`, `current_time`, `current_unixtime`, + `current_unixtime_ms`. At least one of `columns`/`rows`/`row` required. + - `pk` (string) / `pks` (list) — mutually exclusive. For an existing table + a differing pk → 400 `"pk cannot be changed for existing table"`. + - `ignore` / `replace` (bools) — mutually exclusive; require `row`/`rows` + and `pk`/`pks`. + - `alter` (bool) — add missing columns when inserting into an existing table. +- **Success** — **201**: + ```json + {"ok": true, "database": "...", "table": "...", + "table_url": "https://.../db/table", "table_api_url": "https://.../db/table.json", + "schema": "CREATE TABLE ...", "row_count": 2} + ``` + `row_count` only when rows were inserted. Write failures → 400 + `{"ok": false, "errors": [""]}`. Emits `create-table` / + `insert-rows` / `alter-table` events. + +### POST /\/-/execute-write + +`ExecuteWriteView` (app.py:2679-2682; views/execute_write.py:236-476). GET on +the same path renders an HTML form (requires `execute-write-sql`). + +- **Permission (POST):** `execute-write-sql` → 403 + `{"ok": false, "errors": ["Permission denied: need execute-write-sql"]}`; + immutable database → 403 `["Database is immutable"]`. +- **Per-statement permissions:** the SQL is analyzed + (`decision_for_write_sql_operation`, write_sql.py:63-189) and each + operation must pass: + + | Operation | Requirement | + |---|---| + | `select` / internal ops / function calls | ignored | + | read of a table | `view-table` on that table | + | `insert` or `update` | **all of** `insert-row`, `update-row`, `delete-row` on the table | + | `delete` | `delete-row` | + | `create table` | `create-table` on the database | + | `alter table`, `create index`, `drop index` | `alter-table` on the table | + | `drop table` | `drop-table` | + | `VACUUM`, virtual-table writes, shadow-table writes | rejected outright (403) | + | statements touching attached databases | rejected (403) | + +- **Body:** JSON (`{"sql": ..., "params": {...}}` — only those two keys) or + form-encoded (`sql` plus one field per parameter, `_sql_param_` prefix + stripped). Validation errors (400): `"SQL is required"`, + `"params must be a dictionary"`, `"Unknown parameters: a, b"`, + `"Magic parameters are not allowed"`, `"Could not analyze query: ..."`, + `"Use /-/query for read-only SQL; this endpoint only executes writes"`. +- **JSON is returned when** the body was JSON, `Accept: application/json`, or + a truthy `_json` field is present; otherwise HTML. +- **Success** — 200: + ```json + {"ok": true, "message": "Query executed, 1 row affected", "rowcount": 1, + "rows": [], "truncated": false, + "analysis": [{"operation": "insert", "database": "db", "table": "t", + "required_permission": "insert-row, update-row, delete-row", + "source": null}]} + ``` + `rows` is populated by `RETURNING` clauses. SQLite errors → 400 + `{"ok": false, "errors": [""]}`. Anti-framing headers on all + responses. + +### GET /\/-/execute-write/analyze + +`ExecuteWriteAnalyzeView` (app.py:2675-2678; views/execute_write.py:479-507). + +- **Permission:** `execute-write-sql` → 403 `errors` JSON. +- **Parameters:** only `sql` allowed (else 400 `"Invalid keys: ..."`). +- **Response** — 200 even when analysis fails (`ok: false` in body): + `{"ok", "parameters", "analysis_error", "analysis_rows": + [{operation, database, table, required_permission, source, allowed}], + "execute_disabled", "execute_disabled_reason"}`. `allowed` is a per-actor + permission check result (true/false/null). + +### GET /\/-/foreign-key-targets + +`DatabaseForeignKeyTargetsView` (app.py:2659-2662; +views/table_create_alter.py:965-1005). + +- **Parameter:** `table` (optional) — only used for the permission check. +- **Permission:** `create-table` on the database, **or** `alter-table` on + `?table=` when it names an existing table. Neither → 403 + `{"ok": false, "errors": ["Permission denied: need create-table"]}`. +- **Response:** 200 `{"ok": true, "database": "...", "targets": + [{"fk_table", "fk_column", "type"}]}` — every non-hidden table with exactly + one primary-key column; `type` is the pk's SQLite type affinity. + +### GET /\/-/schema(.json|.md) + +`DatabaseSchemaView` (app.py:2683-2686; views/special.py:1296-1329). + +- **Permission:** `view-database` (denied → `Forbidden` → 403 HTML). +- **Unknown database** → 404; for `.json`: + `{"ok": false, "error": "Database not found"}`. (The existence check runs + before the permission check.) +- **Responses:** `.json` → 200 `{"database": "", "schema": ""}` + (concatenated `sqlite_master.sql` joined with `;\n`); `.md` → + `text/markdown`; no extension → HTML. Note the JSON has **no `ok` key** on + success. + +--- + +## Table and row read endpoints + +### GET /\/\.json + +Route `r"/(?P[^\/\.]+)/(?P
[^\/\.]+)(\.(?P\w+))?$"` → +`table_view` (app.py:2711-2714; views/table.py:1670). Serves both tables and +SQL views. GET/HEAD only — POST returns a plain-text 405. If the name is +neither a table nor a view but matches a stored query, the request is +dispatched to `QueryView` (views/table.py:1703-1712). + +**Permission:** `view-table` via `check_visibility`; denial raises +`Forbidden` → **HTML** 403 page even for `.json`. Unknown table → +`TableNotFound` → 404 (JSON error shape for `.json` paths). + +**Default JSON keys** (views/table.py:2308-2332 + renderer): + +| Key | Meaning | +|---|---| +| `ok` | `true` when data was retrieved without error | +| `next` | pagination token string, or `null` on the last page | +| `rows` | list of row objects `{column: value}` (default `_shape=objects`) | +| `truncated` | always present; `false` for table pages | + +`columns` is computed but removed unless `?_extra=columns` was requested. +When there is a next page the response carries a +`Link: ; rel="next"` header (views/table.py:1911-1912). + +**`?_extra=` options** (TABLE scope; registry +views/table_extras.py:1197-1235; unknown names silently ignored): + +| `_extra=` | Returns | +|---|---| +| `count` | total matching-row count, computed with a `limit 10001` subquery so it caps at 10001; `null` with `_nocount` or on count timeout | +| `count_sql` | the SQL used for the count | +| `facet_results` | `{"results": {name: facet}, "timed_out": [...]}`; each facet: `{name, type, hideable, toggle_url, results: [{value, label, count, toggle_url, selected}], truncated}` | +| `facets_timed_out` | facet names that exceeded `facet_time_limit_ms` | +| `suggested_facets` | `[{name, toggle_url, (type)}]`; empty when suggestion is disabled or paginating | +| `human_description_en` | English description of filters + sort | +| `next_url` | absolute URL of the next page or `null` | +| `columns` | column names of the returned rows | +| `all_columns` | all table columns regardless of `_col`/`_nocol` | +| `primary_keys` | pk column names (empty for rowid tables and views) | +| `display_columns` | HTML-oriented column metadata | +| `render_cell` | per-row plugin-rendered HTML strings | +| `debug` | `{url_vars, resolved, nofacet, nosuggest}` — explicitly unstable | +| `request` | `{url, path, full_path, host, args}` | +| `query` | `{sql, params}` of the main query | +| `column_types` | `{column: {type, config}}` assigned column types | +| `set_column_type_ui` | UI helper, `null` unless actor has `set-column-type` | +| `metadata` | table metadata dict including column descriptions | +| `extras` | self-describing list of all available extras | +| `database`, `table`, `database_color` | identity/display values | +| `renderers` | `{format_name: url}` of formats that can render this data | +| `custom_table_templates` | template lookup list | +| `sorted_facet_results` | facets as a display-ordered list | +| `table_definition` | `CREATE TABLE` SQL | +| `view_definition` | `CREATE VIEW` SQL, `null` for tables | +| `is_view` | boolean | +| `private` | `true` if visible to this actor but not anonymously | +| `expandable_columns` | `[[foreign_key, label_column_or_null], ...]` | +| `form_hidden_args` | pairs of `_`-prefixed args for HTML forms | + +Non-public extras (`actions`, `filters`, `display_rows`) are HTML-only and +never appear in JSON. `_extra=_html` expands to the full HTML bundle +(views/table_extras.py:1162-1194). Any `_facet*` argument implicitly adds +`facet_results`; `_shape=object` implicitly adds `primary_keys` +(views/table.py:2252-2256). There is **no** `filtered_table_rows_count` +extra — it was replaced by `count`. + +**Column filters `?__=`** (filters.py:260-427). Any +querystring key not starting with `_` is a filter; bare `?column=value` means +`exact`. Columns whose names start with `_` can be filtered as +`?_col__exact=`. Operators: + +| op | SQL | +|---|---| +| `exact` | `"col" = :p` (default) | +| `not` | `"col" != :p` | +| `contains` / `notcontains` | `like '%v%'` / `not like '%v%'` | +| `endswith` / `startswith` | `like '%v'` / `like 'v%'` | +| `gt` / `gte` / `lt` / `lte` | `>` `>=` `<` `<=` (numeric strings cast to int) | +| `like` / `notlike` | raw `like` / `not like` pattern | +| `glob` | `glob` | +| `in` / `notin` | comma-separated list, or JSON array if the value starts with `[` | +| `arraycontains` / `arraynotcontains` | `[not] in (select value from json_each("col"))` (requires JSON1) | +| `date` | `date("col") = :p` | +| `isnull` / `notnull` | `is null` / `is not null` (no value) | +| `isblank` / `notblank` | `(is null or = '')` / opposite (no value) | + +**Special (underscore) parameters:** + +| Param | Behavior | +|---|---| +| `_where=SQL` | extra raw where clause (repeatable); requires `execute-sql` else 403 `"_where= is not allowed"` | +| `_search=q` | FTS against the table's FTS table | +| `_search_=q` | FTS restricted to one column; 400 if invalid | +| `_searchmode=raw` | pass the query straight to `match` | +| `_fts_table=` / `_fts_pk=` | override the FTS table / pk used for joins | +| `_through={"table","column","value"}` | filter via an incoming foreign key (repeatable, JSON value) | +| `_sort=col` / `_sort_desc=col` | sort; 400 if both given or column not sortable | +| `_next=token` | pagination token | +| `_size=N\|max` | page size; default `default_page_size` (100); `max` = `max_returned_rows` (1000); 400 on invalid | +| `_col=name` (repeatable) | return only pks + these columns; 400 on invalid | +| `_nocol=name` (repeatable) | exclude columns; 400 if invalid or a pk | +| `_labels=on` | expand every FK column into `{"value", "label"}` | +| `_label=col` (repeatable) | expand only the named FK column(s) | +| `_facet=col` | request a facet; 400 `"_facet= is not allowed"` when `allow_facet` off | +| `_facet_array=col` / `_facet_date=col` | typed facets | +| `_facet_size=N\|max` | facet bucket count, default 30, capped at `max_returned_rows` | +| `_nocount=1` | skip count (`count` extra → null) | +| `_nofacet=1` | skip facets and suggestions | +| `_nosuggest=1` | skip facet suggestions only | +| `_shape=` | see renderer section; `array`/`object` also force `_nocount` and `_nofacet` | +| `_nl=on` | NDJSON with `_shape=array` | +| `_json=col` / `_json_infinity=1` | renderer options | +| `_timelimit=ms` | custom SQL time limit | +| `_ttl=seconds` | `Cache-Control: max-age=N` (`0` → `no-cache`); default `default_cache_ttl` (5) | +| `_trace=1` | append `_trace` key (requires `trace_debug` setting) | +| `_extra=` | see above | + +**Pagination** is keyset-based for tables: `page_size + 1` rows are fetched; +`next` is built from the last row of the page — comma-joined tilde-encoded +primary-key values, prefixed by the sort value when sorted (`$null` for null +sort values) (views/table.py:2041-2111, 2421-2482). `next_url` is the +absolute URL with `_next` replaced. + +### GET /\/\.json (SQL views) + +Same code path with `is_view=True`. Differences: + +- No primary keys: `primary_keys` → `[]`; `_shape=object` fails; base query + has no `order by`. +- **Pagination is offset-based**: `_next` is an integer offset applied as + `limit N offset M` (views/table.py:2047-2049, 2438-2439) — unlike the + keyset tokens used for tables. +- `view_definition` returns the `CREATE VIEW` SQL; `table_definition` is null. + +### GET /\/\/\.json + +`RowView` (app.py:2715-2718; views/row.py:137). `` is comma-separated +tilde-encoded primary key values (rowid for rowid tables). + +- **Permission:** `view-table` (denied → `Forbidden` → 403 HTML). Missing row + → 404 `"Record not found: [...]"`. +- **Default JSON keys:** `ok`, `database`, `table`, `rows` (single-element + list), `primary_keys`, `primary_key_values`, `query_ms`, + `truncated: false`; `columns` only with `?_extra=columns`. +- **`?_extra=` (ROW scope):** `columns`, `primary_keys`, `render_cell`, + `debug`, `request`, `query`, `column_types`, `metadata`, `extras`, + `database`, `table`, `database_color`, `private`, `foreign_key_tables` + (incoming FKs with `count` and `link`; single-pk rows only). +- **Foreign-key label expansion does not apply to row JSON** — `_labels` has + no effect here; expansion happens only in the HTML path + (views/row.py:445-475). +- `_shape`, `_json`, `_nl`, `_json_infinity`, `_ttl` apply. A `.jsono` + request redirects to `.json?_shape=objects`. + +### The .blob format + +`//
/.blob?_blob_column=col` (also on query pages) — +fetches raw binary bytes (blob_renderer.py:10-61). `_blob_column` required +(400 if missing/invalid); optional `_blob_hash` must equal the value's +SHA-256 (else 400 `"Link has expired..."`). Returns `application/binary` as a +download attachment. In JSON output, binary cells appear as +`{"$base64": true, "encoded": "..."}`. + +### GET /\/\/-/schema(.json|.md) + +`TableSchemaView` (app.py:2751-2754; views/special.py:1332-1378). + +- **Permission:** `view-table` via `ensure_permission` (denied → 403 HTML). +- **Responses:** `.json` → 200 `{"database", "table", "schema"}` (no `ok` + key); `.md` → `text/markdown`; no extension → HTML. Missing table → 404 + `{"ok": false, "error": "Table not found"}` for `.json`. + +### GET /\/\/-/fragment + +`TableFragmentView` (app.py:2739-2742; views/table.py:1385-1418). +**HTML-only** — returns the `_table.html` partial; no JSON variant. Accepts +table querystring parameters plus `_row=` to render a single row. + +### GET /\/\/-/autocomplete + +`TableAutocompleteView` (app.py:2743-2746; views/table.py:1492-1595). Tables +only — views get 400 `"Autocomplete is only available for tables"`. + +- **Permission:** `view-table` (denied → `Forbidden` → 403). +- **Parameters:** `q` (matched with escaped `LIKE %q%` against pk columns and + the label column) and `_initial` (truthy: with empty `q`, return the 10 + most recent rows). Neither → `{"rows": []}`. +- **Response:** `{"rows": [{"pks": {pk_name: value}, "label": "..."}]}` — max + 10 items; 500 ms query budget with fallbacks, timing out to + `{"rows": []}`. + +--- + +## The write API + +All write endpoints return errors via `_error()` +(`{"ok": false, "errors": [...]}`) and check permissions with +`datasette.allowed()` directly, so their 403s are JSON (unlike the +`Forbidden`-raising read endpoints). Routes: app.py:2719-2762. + +### POST /\/\/-/insert + +`TableInsertView` (views/table.py:907-1194). + +- **Permissions:** `insert-row` on the table (denied → 403 + `["Permission denied"]`); `update-row` additionally required for + `replace: true` (403 `need update-row to use "replace"`); `alter-table` + additionally required for `alter: true` (403 + `Permission denied for alter-table`). Immutable database → 403 + `Database is immutable`. +- **Request** — requires `Content-Type: application/json` (else 400 + `"Invalid content-type, must be application/json"`). Body: + + | Field | Rules | + |---|---| + | `row` | single object; mutually exclusive with `rows`; forces `return: true` | + | `rows` | list of objects; max `max_insert_rows` (default 100), else 400 `"Too many rows, maximum allowed is 100"` | + | `ignore` | skip rows whose pk already exists; mutually exclusive with `replace` | + | `replace` | replace rows with matching pks (needs `update-row`) | + | `alter` | add missing columns (needs `alter-table`) | + | `return` | include inserted rows in the response | + + One of `row`/`rows` required. Unknown keys → 400 `"Invalid parameter: ..."`. + Unless `alter`, row keys must be existing columns → per-row 400 + `"Row 0 has invalid columns: x, y"`. Values are validated against assigned + column types. +- **Response** — **201** `{"ok": true}`; with `return: true` also `rows` + (the rows as stored, re-fetched by rowid). SQLite errors during the write → + 400 with the message. Emits `insert-rows` (and possibly `alter-table`) + events. + +### POST /\/\/-/upsert + +`TableUpsertView` — subclasses insert (views/table.py:1197-1201). + +- **Permissions:** **both** `insert-row` and `update-row` (403 + `need both insert-row and update-row`); `alter: true` needs `alter-table`. +- **Request:** same as insert, except `ignore`/`replace` are rejected (400 + `"Upsert does not support ignore or replace"`) and **every row must contain + the table's primary key(s)** (per-row 400 + `Row 0 is missing primary key column(s): "id"` / `has null primary key`). +- **Response** — **200** (note: insert returns 201) `{"ok": true}`; with + `return: true`, `rows` re-fetched by pk. Emits `upsert-rows`. + +### POST /\/\/-/alter + +`TableAlterView` (views/table_create_alter.py:1130-1353). + +- **Permission:** `alter-table` (403 `need alter-table`); immutable → 403. +- **Request:** `{"operations": [{"op": ..., "args": {...}}, ...]}` — a + non-empty list, validated by pydantic (extra keys forbidden anywhere; + errors → 400 `location: message`): + + | `op` | `args` | + |---|---| + | `add_column` | `name` (required), `type` (`text`/`integer`/`float`/`blob`, default `text`), `not_null`, `default` xor `default_expr`; `not_null: true` requires a default | + | `rename_column` | `name`, `to` | + | `rename_table` | `to` (must not start `sqlite_`) | + | `alter_column` | `name` + at least one of `type`, `not_null`, `default`, `default_expr` | + | `drop_column` | `name` | + | `set_primary_key` | `columns` (non-empty list) | + | `reorder_columns` | `columns` (non-empty list) | + | `add_foreign_key` | `column`, `fk_table`, optional `fk_column` | + | `drop_foreign_key` | `column` | + | `set_foreign_keys` | `foreign_keys`: list of `{column, fk_table, fk_column?}` | + + `default_expr` must be one of the five `current_*` keywords. Operations are + applied in a single write transaction; any failure → 400. +- **Response** — 200: + ```json + {"ok": true, "database": "...", "table": "", + "table_url": "...", "table_api_url": "...", + "altered": true, "schema": "...", "before_schema": "...", + "operations_applied": 2} + ``` + +### POST /\/\/-/drop + +`TableDropView` (views/table.py:1320-1382). + +- **Permission:** `drop-table` (403 `Permission denied`); immutable → 403. +- **Confirmation flow:** without `{"confirm": true}` in the body, nothing is + dropped and a 200 preview is returned: + `{"ok": true, "database", "table", "row_count", + "message": "Pass \"confirm\": true to confirm"}`. With `confirm: true` → + 200 `{"ok": true}`. Emits `drop-table`. + +### POST /\/\/-/set-column-type + +`TableSetColumnTypeView` (views/table.py:1204-1317). Assigns a Datasette +*column type* (metadata stored in the internal `column_types` table) — it +does not change the SQLite schema. + +- **Permission:** `set-column-type` (403 `Permission denied`). +- **Request** (JSON content type required): `{"column": "name", + "column_type": {"type": "url", "config": {...}?} | null}`. Unknown + keys/invalid structure → detailed 400 errors; unknown type → 400 + `"Unknown column type: x"`. Default registered types (via the + `register_column_types` hook): `url`, `email`, `json`, `textarea`. +- **Response** — 200 `{"ok": true, "database", "table", "column", + "column_type": {...} | null}`. + +### GET /\/\/-/foreign-key-suggestions + +`TableForeignKeySuggestionsView` (views/table_create_alter.py:1008-1127). +**GET only** (read-only despite living beside the write endpoints). + +- **Permission:** `alter-table` (403 `need alter-table`); views → 400 + `"Cannot suggest foreign keys for a view"`. +- **Response** — 200: `{"ok": true, "database", "table", + "row_check": {attempted, status, row_limit, sampled_rows, checked_options}, + "columns": [{column, type, affinity, current, + "suggestions": [{fk_table, fk_column, confidence, sampled_values, reasons}], + "options": [...]}]}`. Samples up to 500 rows within 50 ms/200 ms budgets. + +### POST /\/\/\/-/update + +`RowUpdateView` (views/row.py:781-870). + +- **Permissions:** `update-row` (403 `Permission denied`); `alter: true` + additionally requires `alter-table` (403 + `Permission denied for alter-table`). +- **404s:** `Database not found: x` / `Table not found: x` / + `Record not found: [pks]`. +- **Request:** `{"update": {column: value, ...}, "return"?: true, + "alter"?: true}`. Missing/non-dict `update` → 400 + `"JSON must contain an update dictionary"`; unknown keys → 400 + `"Invalid keys: ..."`; write failures (bad column, constraint violation) → + 400 with the message. +- **Response** — 200 `{"ok": true}`; with `return: true`, + `{"ok": true, "row": {...}}` (singular `row`, unlike insert/upsert's + `rows`). Emits `update-row`. + +### POST /\/\/\/-/delete + +`RowDeleteView` (views/row.py:738-778). + +- **Permission:** `delete-row` (403 `Permission denied`). 404s as update. +- **Request:** no body required (any body is ignored — there is no + confirmation step, unlike table drop). +- **Response** — 200 `{"ok": true}`; with `?_redirect_to_table` a `redirect` + key is added. A failure during the write returns **500** with the message + (unlike update's 400). Emits `delete-row`. + +--- + +## Stored (canned) queries API + +Stored queries live in the internal database's `queries` table +(utils/internal_db.py:116-133). Queries defined in `datasette.yaml` are +synced in at startup with `source="config"` and `is_trusted` defaulting to +true; queries created via the API get `source="user"`, `is_trusted=false`, +`owner_id` = actor id. + +**Canonical stored-query JSON object** (`stored_query_to_dict`, +stored_queries.py:55-80): + +```json +{ + "database": "...", "name": "...", "sql": "...", + "title": null, "description": null, "description_html": null, + "hide_sql": false, "fragment": null, + "params": ["p"], "parameters": ["p"], + "is_write": false, "is_private": true, "is_trusted": false, + "source": "user", "owner_id": "...", + "on_success_message": null, "on_success_message_sql": null, + "on_success_redirect": null, + "on_error_message": null, "on_error_redirect": null, + "private": true +} +``` + +`params` and `parameters` are identical lists, both always present. +`private` appears only in list responses. + +**Default permission rules for queries** (default_permissions/defaults.py): +`view-query` is default-allow, but private queries are visible only to their +owner; the owner may `update-query`/`delete-query` their `source='user'` +queries. + +### GET /-/queries(.json) and GET /\/-/queries(.json) + +`GlobalQueryListView` / `QueryListView` (app.py:2606-2609, 2663-2666; +views/stored_queries.py:69-238). The global variant lists queries across all +databases (`database`/`database_color` are null, `show_database` true). + +- **Permissions:** no single gate; results filtered per query by + `view-query` (private queries appear only for their owner). +- **Parameters:** `_size` (default 20 HTML / **50 JSON**, clamped 1–1000; + non-integer → 400), `_next` (cursor), `q` (substring search over + name/title/description/sql), `is_write` / `is_private` (booleans; invalid → + 400 `"is_write must be 0 or 1"`), `source`, `owner_id`. +- **Response** — 200: + `{"ok": true, "database", "database_color", "queries": [...], "next", + "next_url", "has_more", "limit", "show_private_note", + "show_trusted_note", "query_list_path", "show_database", + "facets": [{title, items: [{label, count, href, active}]}], + "filters": {q, is_write, is_private, source, owner_id}}`. + +### GET /\/-/queries/analyze + +`QueryCreateAnalyzeView` (app.py:2667-2670; views/stored_queries.py:290-322). +**GET only** despite being an "analyze" action — POST → 405. + +- **Permissions:** `execute-sql` then `store-query` (each denial → 403 + `errors` JSON). +- **Parameters:** only `sql` (others → 400 `"Invalid keys: ..."`). +- **Response** — 200: `{"ok", "parameters", "analysis_error", + "analysis_rows": [{operation, database, table, required_permission, + source, allowed}], "has_sql", "analysis_is_write", "save_disabled"}`. + +### POST /\/-/queries/store + +`QueryStoreView` (app.py:2671-2674; views/stored_queries.py:325-388). GET on +the same path renders the HTML create form. + +- **Permissions:** `execute-sql` + `store-query` (403 `errors` JSON). +- **Request:** JSON bodies must wrap the fields: + `{"query": {...fields...}}`; form bodies pass fields flat. Fields: + `name` (required; `^[^/\.\n]+$`; conflicts with tables/views or existing + queries → 400), `sql` (required; read SQL must pass `validate_sql_select`; + write SQL must pass per-operation permission checks), `title`, + `description`, `hide_sql`, `fragment`, `parameters`/`params` (must exactly + match the SQL's named parameters; magic parameters rejected), + `is_private` (**default true**), and — only for write SQL — + `on_success_message`, `on_success_redirect`, `on_error_message`, + `on_error_redirect`. `is_write` is derived from SQL analysis; + `is_trusted`, `description_html` and `on_success_message_sql` cannot be + set through this API. +- **Response:** JSON request → **201** `{"ok": true, "query": {...}}`; form + request → 302 redirect. + +### GET /\/\/-/definition + +`QueryDefinitionView` (app.py:2695-2698; views/stored_queries.py:391-408). + +- **Permission:** `view-query` (403 `["Permission denied"]`). +- **Response:** 200 `{"ok": true, "query": {...}}`; 404 + `["Query not found: x"]`. + +### GET/POST /\/\/-/edit + +`QueryEditView` (app.py:2699-2702) — **HTML form endpoint** +(`has_json_alternate = False`), not part of the JSON API. Programmatic +updates use `/-/update`. + +### POST /\/\/-/update + +`QueryUpdateView` (app.py:2703-2706; views/stored_queries.py:411-465). + +- **Permissions:** `update-query` (403 `need update-query`); trusted queries + → 403 `"Trusted queries cannot be updated using the API"`; changing `sql` + additionally requires `execute-sql`. +- **Request:** `{"update": {...partial fields...}, "return"?: true}` — other + top-level keys → 400. Updatable fields: `sql`, `title`, `description`, + `hide_sql`, `fragment`, `parameters`/`params`, `is_private`, `on_*` + fields (write SQL only). New SQL is re-analyzed and `is_write` recomputed. +- **Response:** 200 `{"ok": true}` (plus `query` with `return: true`); 404 + `"Query not found: x"`. + +### POST /\/\/-/delete + +`QueryDeleteView` (app.py:2707-2710; views/stored_queries.py:594-644). GET +renders an HTML confirmation page. + +- **Permission:** `delete-query` (403 `need delete-query`). Unlike update, + **trusted queries are not blocked** from API deletion. +- **Response:** JSON request → 200 `{"ok": true}`; form → 302; 404 + `"Query not found: x"`. No `confirm` field required (unlike table drop). + +### GET/POST /\/\(.json) — executing a stored query + +No dedicated route: the table route resolves the name, and on `TableNotFound` +the request is dispatched to `QueryView` when a stored query matches +(views/table.py:1698-1712). Covers both config-defined and API-stored +queries. + +**GET (read queries)** — `QueryView.get` (views/database.py:695-1130): + +- **Permissions:** `view-query` (denied → `Forbidden` → 403 HTML). Read + queries then require `execute-sql` unless `is_trusted`. Write queries are + **not executed** on GET — JSON returns empty `rows`; HTML shows a POST form. +- **Parameters:** each named `:param` is read from the query string (missing + → `""`); `_timelimit`; renderer options (`_shape`, `_nl`, `_json`, + `_json_infinity`); `_extra` (QUERY scope). +- **Response:** `{"ok": true, "rows": [...], "truncated": false}` + extras. + SQL errors → 400 with `error` in the envelope. + +**POST (write queries)** — `QueryView.post` (views/database.py:574-693): + +- **Permissions:** `view-query`; then, unless `is_trusted`: + `execute-write-sql` on the database **plus** per-operation write + permissions (same table as `/-/execute-write`). Rejection → 403 + `{"ok": false, "message": "...", "redirect": null}` for JSON clients. + Immutable database → 403. +- **Body:** form-encoded or JSON `param=value` pairs (values coerced to + strings). +- **JSON is returned when** `Accept: application/json`, `?_json=1`, or a + `_json` body field is present; otherwise 302 + flash message. +- **Magic parameters** (`:__`, resolved server-side; registered + via `register_magic_parameters`, default_magic_parameters.py): + `_now_epoch`, `_now_date_utc`, `_now_datetime_utc`, `_actor_`, + `_random_chars_`, `_cookie_`, `_header_` (underscores → + hyphens). User-stored queries cannot contain magic parameters — they are a + feature of config/trusted queries. +- **Response — 200 for both success and SQL failure** (only permission + rejection is 403): + `{"ok": true|false, "message": "...", "redirect": "..."|null}` — + `message` honors `on_success_message_sql` / `on_success_message` / + `on_error_message`, falling back to `"Query executed"` or + `"Query executed, N rows affected"`. + +--- + +## Authentication and tokens + +### Bearer tokens (`dstok_`) + +Signed API tokens are sent as `Authorization: Bearer dstok_...`. The +`actor_from_signed_api_token` hook (default_permissions/tokens.py:25-40) +passes the token to `datasette.verify_token()`, which tries every handler +registered via `register_token_handler`; the default is +`SignedTokenHandler` (tokens.py:117-193). + +- **Format:** `dstok_` + itsdangerous-signed payload (namespace `token`) + containing `a` (actor id), `t` (creation Unix time), optional `d` + (duration seconds), optional `_r` (restrictions). +- **Verification** returns no actor when: `allow_signed_tokens` is off, the + signature is invalid, `t` is missing/non-integer, or the token is expired. + The effective duration is `d` capped by `max_signed_tokens_ttl` (default 0 + = no cap; a non-zero setting also imposes a TTL on tokens without `d`). +- **Resulting actor:** `{"id": , "token": "dstok"}` plus `"_r"` and + `"token_expires"` when applicable. Invalid/expired tokens silently produce + an anonymous request (no 401) — the failure then surfaces as a 403 from + whatever permission check the request hits. + +**Restrictions (`_r`)** (default_permissions/restrictions.py): + +- `"a"`: list of actions allowed on any resource +- `"d"`: `{database_name: [actions]}` +- `"r"`: `{database_name: {table_name: [actions]}}` + +Actions are stored as abbreviations when available (see appendix); checks +accept either the full name or the abbreviation. Restrictions are an +allowlist filter layered on top of normal permission resolution — a +restricted token can never do more than its allowlist, and never more than +the underlying actor could do anyway. + +### Token creation + +- **`/-/create-token`** is an HTML form endpoint only (see the instance + section) — there is no JSON API to mint tokens in this codebase. +- Programmatic alternatives: the `datasette create-token` CLI command and + the `datasette.create_token()` Python API. +- `/-/auth-token` is the one-time `--root` login mechanism, unrelated to API + tokens. + +### Cookie authentication + +Browser sessions use the signed `ds_actor` cookie (set by `/-/auth-token`, +plugins, or login flows; cleared by `/-/logout`). API POSTs from browsers are +subject to the cross-origin checks described in +[CSRF](#csrf--cross-origin-protection). + +--- + +## Appendix: registered actions (permissions) + +From `datasette/default_actions.py` (registered via the `register_actions` +hook). Token restrictions store the abbreviation when available. + +| Action | Abbr | Resource level | Notes | +|---|---|---|---| +| `view-instance` | `vi` | global | | +| `permissions-debug` | `pd` | global | gates the debug endpoints | +| `debug-menu` | `dm` | global | UI only | +| `view-database` | `vd` | database | | +| `view-database-download` | `vdd` | database | `also_requires="view-database"` | +| `execute-sql` | `es` | database | `also_requires="view-database"`; denied when the `default_allow_sql` setting is off | +| `execute-write-sql` | `ews` | database | `also_requires="view-database"` | +| `create-table` | `ct` | database | | +| `store-query` | `sq` | database | `also_requires="execute-sql"` | +| `view-table` | `vt` | table | | +| `insert-row` | `ir` | table | | +| `delete-row` | `dr` | table | | +| `update-row` | `ur` | table | | +| `alter-table` | `at` | table | | +| `set-column-type` | `sct` | table | | +| `drop-table` | `dt` | table | | +| `view-query` | `vq` | query | default-allow; private queries restricted to their owner | +| `update-query` | `uq` | query | query owner allowed by default (source=`user` only) | +| `delete-query` | `dq` | query | query owner allowed by default (source=`user` only) | diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md new file mode 100644 index 00000000..e630ae4b --- /dev/null +++ b/stable-api-recommendations.md @@ -0,0 +1,367 @@ +# Datasette 1.0 Stable API — Consistency and Completeness Review + +This review is based on `existing-api.md`, which documents the JSON API as +actually implemented in this codebase (`1.0a35`), derived from source. The +goal here is to identify everything that should be made consistent, fixed, or +explicitly scoped out **before** the 1.0 stability promise takes effect — +because after 1.0, every inconsistency below becomes a compatibility +commitment. + +Findings are grouped by theme. Each carries a priority: + +- **P1 — should block 1.0**: breaking to fix later, or a correctness/security + concern. +- **P2 — strongly recommended**: fixable later only via awkward additive + changes. +- **P3 — nice to have / documentation decision**: can be resolved by + documenting the behavior as intentional. + +--- + +## 1. Error responses: four shapes is three too many (P1) + +The API currently produces four distinct JSON error shapes depending on which +internal layer generates the error: + +| Shape | Producer | Example endpoints | +|---|---|---| +| `{"ok": false, "error", "status", "title"}` | exception handler (handle_exception.py:50-53) | 404s and `DatasetteError`s on any `.json` path | +| `{"ok": false, "errors": [...]}` | `_error()` helper (views/base.py:183-184) | all write endpoints, stored-query endpoints, execute-write | +| `{"ok": false, "error", "rows": [], "truncated": false}` | JSON renderer (renderer.py:52-56) | SQL errors on table/query reads | +| `{"error": "..."}` (no `ok`) | permission debug views (views/special.py) | `/-/allowed`, `/-/rules`, `/-/check`, POST `/-/permissions` | + +Additionally, write canned queries report failure via a **fifth** vocabulary: +`{"ok": false, "message": ..., "redirect": ...}` with HTTP **200** +(views/database.py:678-690). + +A 1.0 client cannot write a single error handler today. **Recommendation:** +pick one canonical error object — the singular/plural tension is easiest to +resolve as: + +```json +{"ok": false, "error": "human-readable summary", "errors": ["detail", "..."], "status": 400} +``` + +where `errors` is optional and `error` is always present — and route every +error path through it (including the `forbidden` and `handle_exception` +defaults). At minimum, eliminate the bare `{"error": ...}` shape and the +`status`/`title` keys nobody else emits (`title` is a template-rendering +concern that leaked into the API). + +### 1a. `Forbidden` returns an HTML 403 to JSON clients (P1) + +Read endpoints that deny access via `ensure_permission`/`check_visibility` +raise `Forbidden`, and the default `forbidden()` hook renders an **HTML error +page even for `.json` requests** (forbidden.py:4-19, app.py:2895-2904). So: + +- `GET /db/table.json` without `view-table` → 403 **HTML** +- `POST /db/table/-/insert` without `insert-row` → 403 **JSON** + +A JSON client gets unparseable output precisely when it most needs a +machine-readable answer. **Recommendation:** the default forbidden handler +must return the canonical JSON error when the path ends in `.json` or the +request prefers JSON, mirroring `handle_exception`. + +### 1b. Errors that return HTTP 200 (P1) + +- `_shape=object` on a query or pk-less table → `{"ok": false, "error": + "_shape=object is only available on tables"}` with **200** + (renderer.py:73-90), while an unknown `_shape` value returns **400** + (renderer.py:101-108). Same class of error, different status. +- Write canned-query SQL failure → **200** `{"ok": false, "message": ...}` + (views/database.py:683-690), while the equivalent failure on + `/-/execute-write` returns **400**. + +**Recommendation:** all `ok: false` responses should carry a 4xx/5xx status. +(`/-/execute-write/analyze` returning `ok: false` with 200 for "analysis +completed, SQL is invalid" is defensible but should then not reuse the `ok` +key — see §2.) + +### 1c. Wrong-status outliers (P2) + +- Row **delete** write failures return **500** (views/row.py:757) while row + **update** write failures return **400** (views/row.py:832-835). Same + failure class, different status; pick 400 (or 409 for constraint + violations) for both. +- Invalid or expired bearer tokens silently degrade the request to anonymous, + so clients see a 403 permission error (or worse, anonymous-permitted data) + rather than a 401 (tokens.py:147-193). For 1.0, a malformed/expired + `Authorization: Bearer dstok_...` header should produce **401** with a + distinguishable error, so clients can tell "renew your token" apart from + "you lack permission". + +--- + +## 2. Success envelope: `ok` is not universal, arrays are not extensible (P1/P2) + +Endpoints disagree about the success envelope: + +- **Have `ok: true`:** table/row/query reads, database view, all write + endpoints, stored-query endpoints, `/-/allowed`-style debug data. +- **No `ok` key:** `/-/versions`, `/-/settings`, `/-/config`, `/-/threads`, + `/-/actor`, `/-/jump`, `/-/schema` variants (`{"database", "schema"}`, + `{"schemas": [...]}`), table `/-/schema.json`, `/-/autocomplete` + (`{"rows": []}`), homepage `/.json`. +- **Top-level JSON arrays:** `/-/plugins`, `/-/databases`, `/-/actions` + (app.py:2247-2304). A top-level array can never grow a sibling key + (pagination, warnings, `ok`) without a breaking change. + +**Recommendations:** + +1. (P1) Wrap the three array endpoints in objects before 1.0: + `{"ok": true, "plugins": [...]}` etc. This is the single cheapest + future-proofing fix in this list. +2. (P2) Add `ok: true` to every JSON-object success response, or explicitly + document that `ok` only exists on data endpoints. Half-consistency is the + worst outcome. +3. (P2) `/db/-/schema.json` (`{"database", "schema"}`) and + `/db/table/-/schema.json` should match the envelope style of their sibling + endpoints (they are also the only data endpoints whose 404 uses the + exception shape but whose success has no `ok`). + +### 2a. Collection representations disagree (P2) + +- Homepage `/.json` returns `databases` as an **object keyed by name** + (index.py:147-161); `/-/databases.json` returns an **array**; the database + page returns `tables` as an array. Choose arrays-of-objects everywhere + (objects-keyed-by-name break when names need ordering or pagination). +- Insert/upsert with `return: true` respond with `rows` (plural, list); row + update with `return: true` responds with `row` (singular, object) + (views/row.py:837-844). Pick one (`rows` everywhere, even for one row, + matches the read API). + +### 2b. `_extra`/`_shape` support is uneven (P2) + +The extras system (`?_extra=`, scope-registered) is the 1.0 mechanism for +response shaping — but it only exists on table, row and query endpoints. The +database view builds JSON by hand and supports **neither `_extra` nor +`_shape`** (views/database.py:189-212); the homepage likewise. Either extend +extras to database/instance scope before 1.0 or document clearly that shaping +is a table/row/query feature. Also decide the contract for **unknown +`_extra` names, which are currently silently ignored** (extras.py:116-122) — +silent ignoring means typos return the default payload with no signal; +recommend a 400 or a `warnings` key. + +### 2c. Count truncation is invisible in JSON (P2) + +The `count` extra is computed with a `limit 10001` subquery, so `count: +10001` actually means "at least 10001" — the `count_truncated` flag exists +but only in the HTML template context, never in JSON (views/table.py: +2334-2337). Expose it (e.g. make `count` be `null` + add `count_estimate`, +or add `count_truncated` to the JSON) before clients start trusting the +number. + +--- + +## 3. Pagination: three mechanisms, two contracts (P2) + +| Endpoint | Mechanism | Token | Extras | +|---|---|---|---| +| Table `.json` | keyset | tilde-encoded pk/sort values in `_next` | `next` always in body, `next_url` via `_extra`, `Link: rel=next` header | +| SQL view `.json` | **offset** | integer in the same `_next` parameter | same envelope | +| `/-/queries` lists | keyset | cursor in `_next` | `next`, `next_url`, **`has_more`** in body | +| `/-/allowed`, `/-/rules` | **page numbers** | `page`/`page_size` | `total`, `next_url`, `previous_url` | + +Concerns: + +1. The same `_next` parameter means "start after key" on tables but "row + offset" on views. Offset pagination over views is also O(n) and skews + under concurrent writes. If unifiable, unify; if not, document loudly. +2. `has_more` exists on query lists but not table pages; `total` exists on + debug endpoints but not elsewhere. Standardize the pagination block + (suggest: `next`, `next_url` — nullable — everywhere; treat `has_more` as + `next != null`). +3. Page-size parameters: `_size` (default 100, `max` keyword allowed) on + tables; `_size` (default 50 JSON, clamped 1–1000, no `max` keyword) on + query lists; `page_size` (default 50, silently capped at 200) on debug + endpoints. Align names, defaults and the cap behavior (silent capping vs + 400) as far as practical. + +--- + +## 4. HTTP semantics (P2) + +- **201 vs 200:** insert → 201, upsert → 200 (views/table.py:1194), create + table → 201, store query → 201. Insert-201/upsert-200 is defensible + (upsert may not create) but it is undocumented subtlety; state it, or + return 200 for both with an explicit `created` count. +- **Destructive-action confirmation is asymmetric:** table drop requires + `{"confirm": true}` and has a preview response (views/table.py:1346-1365); + row delete executes immediately and ignores the body; query delete + executes immediately. Decide the 1.0 rule (suggestion: confirmation only + for schema-destroying operations, i.e. keep as is — but document it as a + deliberate contract). +- **Content-type enforcement is inconsistent:** `/-/insert`, `/-/upsert`, + `/-/alter`, `/-/set-column-type` demand `Content-Type: application/json` + (400 otherwise); `/-/create` parses the body as JSON regardless of + content type; execute-write and the query CRUD endpoints accept both JSON + and form encodings. Pick one rule for JSON-only endpoints. +- **JSON-vs-HTML negotiation on POST differs per endpoint:** execute-write + and canned queries key off `Accept: application/json` / a `_json` body + field; the write API keys off nothing (always JSON); query store keys off + request content type. A single documented rule ("responses are JSON if the + request body was JSON or `Accept: application/json`") would cover all of + them. +- **Endpoints named like actions but served over GET:** + `/-/queries/analyze`, `/-/execute-write/analyze`, + `/-/foreign-key-suggestions`, `/-/query/parameters` are all GET (correct, + they are reads) — fine, but `analyze` under a POST-shaped path invites + wrong calls; make sure 405 responses for POST on these return the JSON 405 + shape (they do only when the path ends `.json` or content type is JSON — + a JSON POST to `/-/queries/analyze` gets JSON, a form POST gets text). + +--- + +## 5. Naming and parameter conventions (P2/P3) + +- **`params` and `parameters` are duplicate keys** in every stored-query + object (stored_queries.py:55-80). Delete one before 1.0 (suggest keeping + `parameters`; the write side already accepts both on input). +- **Three names for the same concept across error/message payloads:** + `error`, `errors`, `message`. See §1. +- **Boolean query parameters have at least three grammars:** `_nl=on`, + `_labels=on/off`, `?all=1`, `is_write=1|0|true|false|t|f|yes|no|on|off`, + `_nocount=1`. Adopt one accepted set (the query-list parser at + query_helpers.py:81-94 is a good candidate) and apply it everywhere. +- **`.jsono`** survives on the homepage route (identical output to `.json`) + and as a row-view redirect. Remove it at 1.0; it is pure legacy. +- **`_json` is overloaded:** on GET it is a renderer option naming a column + to parse as JSON (repeatable); on canned-query POST a `_json` body field + forces a JSON response. Two unrelated meanings for one name. +- The reserved `/-/` namespace is applied consistently across routes — this + is in good shape. The one gap: table names matching `^-$`-adjacent shapes + are protected by tilde-encoding; keep a test asserting `/-/` can never be + shadowed by user data. + +--- + +## 6. Permissions and security consistency (P1/P2) + +- **(P1) `/-/databases.json` ignores per-database permissions** — it lists + every attached database (name, path on disk, size) to any actor holding + `view-instance` (app.py:2157-2169), while the homepage and every other + endpoint filter by `view-database`. On a public instance with private + databases this leaks filesystem paths and database names. Filter it, or + gate it behind `permissions-debug`. +- **(P2) `/db/-/schema` checks existence before permission** + (views/special.py:1308-1317): an actor without `view-database` can + distinguish "database exists" (403) from "does not exist" (404). + Standardize on permission-check-first (as the table view does) so + unauthorized actors get a uniform response. +- **(P2) `/-/threads` exposes runtime internals** (thread idents, asyncio + task reprs including file paths) behind only `view-instance`. Consider + `permissions-debug`, alongside `/-/actions` which already requires it. +- **(P3) `/-/config` redaction is substring-based** on six key names + (app.py:2502-2505); plugins storing secrets under other names leak. Worth + a note in plugin authoring docs plus a `redact_keys` plugin hook. +- **(P3) Database-level checks on `/-/create`** (insert-row/update-row + checked against `DatabaseResource`, not the about-to-exist table — + table_create_alter.py:819-856) vs table-level checks on `/-/insert`. + Correct by necessity, but document that a token restricted to + table-level `ir` cannot use `/-/create` with rows. + +--- + +## 7. Completeness gaps for a 1.0 JSON API (P2/P3) + +1. **(P2) No JSON API to create tokens.** `/-/create-token` is an HTML form + only (`has_json_alternate = False`, form-encoded POST). Any automation + that wants to mint scoped tokens must shell out to `datasette + create-token`. An intentional JSON mode (actor-authenticated, same + restriction vocabulary) rounds out the write API story — or explicitly + document token minting as CLI/Python-only. +2. **(P2) Row JSON cannot expand foreign-key labels.** `_labels` works on + table JSON but is silently ignored on row JSON (views/row.py:445-475 + expands only for HTML). Either support it or return 400 for unsupported + parameters; silent ignoring is the worst option (see also §2b on unknown + `_extra` values). +3. **(P2) No machine-readable "which write features does this instance/table + support" endpoint.** Clients must probe (`/-/insert` on an immutable + database → 403). The API explorer computes exactly this data for HTML + (views/special.py:863-990); exposing it as JSON would let clients degrade + gracefully. (`/-/allowed.json` covers the permission half already.) +4. **(P3) Table list pagination.** `/db.json` inlines all tables (with + counts) and the homepage truncates to 5 per database; a 10,000-table + database has no paginated table listing. Acceptable for 1.0 if + documented; the internal catalog tables would support a real endpoint + later. +5. **(P3) `Link: rel=next` header** exists on table JSON only. Harmless, but + either add it to the other paginated endpoints or drop it from the + contract (`Access-Control-Expose-Headers: Link` suggests it is meant to + be part of the API). + +--- + +## 8. Behavior that looks like a bug and should be resolved before freezing + +1. **Trusted queries: update is blocked, delete is not.** + `QueryUpdateView` rejects `is_trusted` queries with 403 + (stored_queries.py:426-427) but `QueryDeleteView.post` never checks + `is_trusted` — an actor with `delete-query` can delete a config-defined + trusted query via the API (it will resync on restart, making the + behavior confusing rather than catastrophic). Align delete with update. +2. **GET `/db/-/query` with no `?sql=` returns 200 `{"ok": true, "rows": + []}`** while `.csv` on the same request returns 400 `"?sql= is + required"`. The JSON behavior masks caller bugs; return 400 on both. +3. **`_shape=object` HTTP 200 error** (§1b) — almost certainly unintended. +4. **Row delete 500** (§1c) — inconsistent with every sibling endpoint. +5. **The "SQL Interrupted" error embeds an HTML fragment in the JSON `error` + value** (views/database.py:805-820). Error strings in the JSON API should + be plain text. + +--- + +## 9. Define stability tiers explicitly (P1 — documentation, not code) + +Not everything under `/-/` can or should carry a 1.0 guarantee. Recommend +shipping 1.0 with an explicit three-tier contract, per endpoint: + +- **Stable (semver-protected):** table/row/query reads (`.json`, `_shape`, + `_extra` public names, filters, pagination tokens as opaque strings), the + write API (`/-/insert`, `/-/upsert`, `/-/alter`, `/-/drop`, + `/-/set-column-type`, row `/-/update`, `/-/delete`, `/-/create`, + `/-/execute-write`), stored-query CRUD + execution, `/-/versions`, + `/-/plugins`, `/-/settings`, `/-/actor`, `/-/databases`, schema endpoints, + token format & restriction semantics (`_r` abbreviations are wire format + now — they are stored inside issued tokens and cannot change silently). +- **Unstable/debug (documented as exempt):** `/-/threads`, `/-/actions`, + `/-/permissions`, `/-/allowed`, `/-/rules`, `/-/check`, `/-/messages`, + `/-/allow-debug`, `/-/patterns`, `/-/debug/autocomplete`, the `debug` and + `request` extras (the `debug` extra already self-describes as unstable), + `/-/api` and `/-/jump` (UI support endpoints), `/-/autocomplete` and + `/-/fragment` (UI support), `/-/foreign-key-suggestions` and + `/-/foreign-key-targets` (heuristic outputs). +- **Internal:** anything HTML-only (`/-/edit`, `/-/create-token`, + `/-/logout`, `/-/auth-token`). + +Two details make tiering urgent rather than optional: + +- **Extras are enumerable by clients** (`?_extra=extras` self-describes the + registry), so every public extra name is de-facto API. Mark each extra + stable or unstable in its class definition and surface that in the + `extras` output. +- **Pagination tokens leak implementation** (tilde-encoded pk values for + tables, plain integers for views). Declare them opaque now so the view + token can become keyset later without a "breaking" change. + +--- + +## 10. Summary of P1 items (the pre-1.0 checklist) + +1. One canonical JSON error shape; retire the other three (§1). +2. `Forbidden` → JSON 403 for JSON requests (§1a). +3. No `ok: false` with HTTP 200 (§1b: `_shape=object`, write canned-query + SQL errors). +4. Wrap `/-/plugins`, `/-/databases`, `/-/actions` top-level arrays in + objects (§2). +5. Filter `/-/databases.json` by `view-database` or gate it behind + `permissions-debug` (§6). +6. 401 (not silent-anonymous) for invalid/expired bearer tokens (§1c). +7. Publish explicit stability tiers, including extras and pagination-token + opacity (§9). +8. Resolve the looks-like-a-bug list (§8), especially trusted-query delete + and row-delete 500. + +Everything in P2 is worth doing now because each item is breaking-to-fix +later; each P3 can be resolved by a sentence of documentation declaring the +current behavior intentional. From 0679e04bd3c293dd3d0efd45aefd6ae57133f68d Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 03:12:15 +0000 Subject: [PATCH 002/131] Unify JSON error responses into one canonical shape All JSON error responses now use a single format built by the new datasette.utils.error_body() helper: {"ok": false, "error": "...", "errors": ["..."], "status": 400} - error is all messages joined with '; ', errors is the full list, status always matches the HTTP status code - The exception handler no longer emits the legacy title key in JSON (it is still available to the HTML error template) - The permission debug endpoints (/-/allowed, /-/rules, /-/check, POST /-/permissions) no longer return bare {"error": ...} objects - JSON renderer SQL errors keep their rows/truncated context keys but now include the canonical keys as well - _shape=object misuse (queries or tables without primary keys) now returns HTTP 400 instead of 200 with an error body - Method-not-allowed 405 responses use the canonical shape Adds tests/test_error_shape.py covering all four previous shape producers, updates affected tests, and documents the format in a new 'Error responses' section of docs/json_api.rst. Implements section 1 of stable-api-recommendations.md. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/handle_exception.py | 41 ++++---- datasette/renderer.py | 14 +-- datasette/utils/__init__.py | 21 ++++ datasette/views/base.py | 11 +- datasette/views/special.py | 44 ++++---- docs/json_api.rst | 59 +++++++++-- existing-api.md | 94 ++++++++--------- stable-api-recommendations.md | 14 ++- tests/test_api.py | 25 ++--- tests/test_api_write.py | 80 +++++++-------- tests/test_base_view.py | 2 + tests/test_column_types.py | 4 +- tests/test_error_shape.py | 185 ++++++++++++++++++++++++++++++++++ tests/test_table_api.py | 8 +- tests/test_table_html.py | 2 +- 15 files changed, 432 insertions(+), 172 deletions(-) create mode 100644 tests/test_error_shape.py diff --git a/datasette/handle_exception.py b/datasette/handle_exception.py index 2b311644..fc290dbc 100644 --- a/datasette/handle_exception.py +++ b/datasette/handle_exception.py @@ -1,5 +1,5 @@ from datasette import hookimpl, Response -from .utils import add_cors_headers +from .utils import add_cors_headers, error_body from .utils.asgi import ( Base400, ) @@ -45,6 +45,13 @@ def handle_exception(datasette, request, exception): message = str(exception) traceback.print_exc() templates = [f"{status}.html", "error.html"] + headers = {} + if datasette.cors: + add_cors_headers(headers) + if request.path.split("?")[0].endswith(".json"): + body = dict(info) + body.update(error_body(message, status)) + return Response.json(body, status=status, headers=headers) info.update( { "ok": False, @@ -53,24 +60,18 @@ def handle_exception(datasette, request, exception): "title": title, } ) - headers = {} - if datasette.cors: - add_cors_headers(headers) - if request.path.split("?")[0].endswith(".json"): - return Response.json(info, status=status, headers=headers) - else: - environment = datasette.get_jinja_environment(request) - template = environment.select_template(templates) - return Response.html( - await template.render_async( - dict( - info, - urls=datasette.urls, - menu_links=lambda: [], - ) - ), - status=status, - headers=headers, - ) + environment = datasette.get_jinja_environment(request) + template = environment.select_template(templates) + return Response.html( + await template.render_async( + dict( + info, + urls=datasette.urls, + menu_links=lambda: [], + ) + ), + status=status, + headers=headers, + ) return inner diff --git a/datasette/renderer.py b/datasette/renderer.py index f40e3dbb..7c94f6ee 100644 --- a/datasette/renderer.py +++ b/datasette/renderer.py @@ -1,6 +1,7 @@ import json from datasette.extras import extra_names_from_request from datasette.utils import ( + error_body, value_as_boolean, remove_infinites, CustomJSONEncoder, @@ -52,8 +53,7 @@ def json_renderer(request, args, data, error, truncated=None): if error: shape = "objects" status_code = 400 - data["error"] = error - data["ok"] = False + data.update(error_body(error, status_code)) if truncated is not None: data["truncated"] = truncated @@ -87,7 +87,8 @@ def json_renderer(request, args, data, error, truncated=None): object_rows[pk_string] = row data = object_rows if shape_error: - data = {"ok": False, "error": shape_error} + status_code = 400 + data = error_body(shape_error, status_code) elif shape == "array": data = data["rows"] @@ -100,12 +101,7 @@ def json_renderer(request, args, data, error, truncated=None): data["rows"] = [list(row.values()) for row in data["rows"]] else: status_code = 400 - data = { - "ok": False, - "error": f"Invalid _shape: {shape}", - "status": 400, - "title": None, - } + data = error_body(f"Invalid _shape: {shape}", status_code) # Don't include "columns" in output # https://github.com/simonw/datasette/issues/2136 diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index 1caff3a8..17c8702f 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -1294,6 +1294,27 @@ async def derive_named_parameters(db: "Database", sql: str) -> List[str]: return named_parameters(sql) +def error_body(messages, status): + """ + The canonical JSON error body used by every Datasette JSON error response: + + {"ok": False, "error": "...", "errors": ["...", ...], "status": 400} + + "error" is all of the messages joined with "; ", "errors" is the full + list, "status" matches the HTTP status code. Callers may add extra + context keys to the returned dictionary but must not remove these four. + """ + if isinstance(messages, str): + messages = [messages] + messages = [str(message) for message in messages] + return { + "ok": False, + "error": "; ".join(messages), + "errors": messages, + "status": status, + } + + def add_cors_headers(headers): headers["Access-Control-Allow-Origin"] = "*" headers["Access-Control-Allow-Headers"] = "Authorization, Content-Type" diff --git a/datasette/views/base.py b/datasette/views/base.py index 30026f4b..a12ae050 100644 --- a/datasette/views/base.py +++ b/datasette/views/base.py @@ -5,6 +5,7 @@ import sys from datasette.utils.asgi import Request from datasette.utils import ( add_cors_headers, + error_body, EscapeHtmlWriter, InvalidSql, LimitedWriter, @@ -49,9 +50,7 @@ class View: request.path.endswith(".json") or request.headers.get("content-type") == "application/json" ): - response = Response.json( - {"ok": False, "error": "Method not allowed"}, status=405 - ) + response = Response.json(error_body("Method not allowed", 405), status=405) else: response = Response.text("Method not allowed", status=405) return response @@ -90,9 +89,7 @@ class BaseView: request.path.endswith(".json") or request.headers.get("content-type") == "application/json" ): - response = Response.json( - {"ok": False, "error": "Method not allowed"}, status=405 - ) + response = Response.json(error_body("Method not allowed", 405), status=405) else: response = Response.text("Method not allowed", status=405) return response @@ -181,7 +178,7 @@ class BaseView: def _error(messages, status=400): - return Response.json({"ok": False, "errors": messages}, status=status) + return Response.json(error_body(messages, status), status=status) async def stream_csv(datasette, fetch_data, request, database): diff --git a/datasette/views/special.py b/datasette/views/special.py index 3245bc13..602ec5ea 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -9,6 +9,7 @@ from datasette.utils import ( actor_matches_allow, add_cors_headers, await_me_maybe, + error_body, tilde_encode, tilde_decode, ) @@ -348,26 +349,29 @@ class AllowedResourcesView(BaseView): async def _allowed_payload(self, request, has_debug_permission): action = request.args.get("action") if not action: - return {"error": "action parameter is required"}, 400 + return error_body("action parameter is required", 400), 400 if action not in self.ds.actions: - return {"error": f"Unknown action: {action}"}, 404 + return error_body(f"Unknown action: {action}", 404), 404 actor = request.actor if isinstance(request.actor, dict) else None actor_id = actor.get("id") if actor else None parent_filter = request.args.get("parent") child_filter = request.args.get("child") if child_filter and not parent_filter: - return {"error": "parent must be provided when child is specified"}, 400 + return ( + error_body("parent must be provided when child is specified", 400), + 400, + ) try: page = int(request.args.get("page", "1")) page_size = int(request.args.get("page_size", "50")) except ValueError: - return {"error": "page and page_size must be integers"}, 400 + return error_body("page and page_size must be integers", 400), 400 if page < 1: - return {"error": "page must be >= 1"}, 400 + return error_body("page must be >= 1", 400), 400 if page_size < 1: - return {"error": "page_size must be >= 1"}, 400 + return error_body("page_size must be >= 1", 400), 400 max_page_size = 200 if page_size > max_page_size: page_size = max_page_size @@ -485,9 +489,13 @@ class PermissionRulesView(BaseView): # JSON API - action parameter is required action = request.args.get("action") if not action: - return Response.json({"error": "action parameter is required"}, status=400) + return Response.json( + error_body("action parameter is required", 400), status=400 + ) if action not in self.ds.actions: - return Response.json({"error": f"Unknown action: {action}"}, status=404) + return Response.json( + error_body(f"Unknown action: {action}", 404), status=404 + ) actor = request.actor if isinstance(request.actor, dict) else None @@ -496,12 +504,12 @@ class PermissionRulesView(BaseView): page_size = int(request.args.get("page_size", "50")) except ValueError: return Response.json( - {"error": "page and page_size must be integers"}, status=400 + error_body("page and page_size must be integers", 400), status=400 ) if page < 1: - return Response.json({"error": "page must be >= 1"}, status=400) + return Response.json(error_body("page must be >= 1", 400), status=400) if page_size < 1: - return Response.json({"error": "page_size must be >= 1"}, status=400) + return Response.json(error_body("page_size must be >= 1", 400), status=400) max_page_size = 200 if page_size > max_page_size: page_size = max_page_size @@ -587,15 +595,15 @@ class PermissionRulesView(BaseView): async def _check_permission_for_actor(ds, action, parent, child, actor): """Shared logic for checking permissions. Returns a dict with check results.""" if action not in ds.actions: - return {"error": f"Unknown action: {action}"}, 404 + return error_body(f"Unknown action: {action}", 404), 404 if child and not parent: - return {"error": "parent is required when child is provided"}, 400 + return error_body("parent is required when child is provided", 400), 400 # Use the action's properties to create the appropriate resource object action_obj = ds.actions.get(action) if not action_obj: - return {"error": f"Unknown action: {action}"}, 400 + return error_body(f"Unknown action: {action}", 400), 400 # Global actions (no resource_class) don't have a resource if action_obj.resource_class is None: @@ -610,7 +618,7 @@ async def _check_permission_for_actor(ds, action, parent, child, actor): resource_obj = action_obj.resource_class(parent) else: # This shouldn't happen given validation in Action.__post_init__ - return {"error": f"Invalid action configuration: {action}"}, 500 + return error_body(f"Invalid action configuration: {action}", 500), 500 allowed = await ds.allowed(action=action, resource=resource_obj, actor=actor) @@ -651,7 +659,9 @@ class PermissionCheckView(BaseView): # JSON API - action parameter is required action = request.args.get("action") if not action: - return Response.json({"error": "action parameter is required"}, status=400) + return Response.json( + error_body("action parameter is required", 400), status=400 + ) parent = request.args.get("parent") child = request.args.get("child") @@ -1229,7 +1239,7 @@ class SchemaBaseView(BaseView): if self.ds.cors: add_cors_headers(headers) return Response.json( - {"ok": False, "error": error_message}, status=status, headers=headers + error_body(error_message, status), status=status, headers=headers ) else: return Response.text(error_message, status=status) diff --git a/docs/json_api.rst b/docs/json_api.rst index eca22fdc..3df66a8e 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -50,6 +50,37 @@ The ``"truncated"`` key lets you know if the query was truncated. This can happe For table pages, an additional key ``"next"`` may be present. This indicates that the next page in the pagination set can be retrieved using ``?_next=VALUE``. +.. _json_api_errors: + +Error responses +--------------- + +Every JSON error response from Datasette uses the same format: + +.. code-block:: json + + { + "ok": false, + "error": "Table not found", + "errors": [ + "Table not found" + ], + "status": 404 + } + +- ``"ok"`` is always ``false`` for an error. +- ``"errors"`` is a list of one or more error message strings. Endpoints that + validate multiple things at once - such as the :ref:`insert API ` - + may return several messages here. +- ``"error"`` is all of those messages joined with ``"; "``, for + convenience when displaying a single string. +- ``"status"`` matches the HTTP status code of the response. + +Some endpoints add extra context keys. For example, a SQL error from a +:ref:`custom query ` also includes the empty +``"rows"`` and ``"truncated"`` keys of the response it was unable to +produce. + .. _json_api_custom_sql: Executing custom SQL @@ -1625,15 +1656,17 @@ the execute-write returning row limit, which defaults to 10: ] } -Errors use the standard Datasette error format: +Errors use the :ref:`standard Datasette error format `: .. code-block:: json { "ok": false, + "error": "Permission denied: need execute-write-sql", "errors": [ "Permission denied: need execute-write-sql" - ] + ], + "status": 403 } .. _TableInsertView: @@ -1727,9 +1760,11 @@ If any of your rows have a primary key that is already in use, you will get an e { "ok": false, + "error": "UNIQUE constraint failed: new_table.id", "errors": [ "UNIQUE constraint failed: new_table.id" - ] + ], + "status": 400 } Pass ``"ignore": true`` to ignore these errors and insert the other rows: @@ -1859,9 +1894,11 @@ When using upsert you must provide the primary key column (or columns if the tab { "ok": false, + "error": "Row 0 is missing primary key column(s): \"id\"", "errors": [ "Row 0 is missing primary key column(s): \"id\"" - ] + ], + "status": 400 } If your table does not have an explicit primary key you should pass the SQLite ``rowid`` key instead. @@ -1921,7 +1958,7 @@ The returned JSON will look like this: } } -Any errors will return ``{"errors": ["... descriptive message ..."], "ok": false}``, and a ``400`` status code for a bad input or a ``403`` status code for an authentication or permission error. +Any errors will use the :ref:`standard error format `, with a ``400`` status code for a bad input or a ``403`` status code for an authentication or permission error. Pass ``"alter: true`` to automatically add any missing columns to the table. This requires the :ref:`actions_alter_table` permission. @@ -1942,7 +1979,7 @@ To delete a row, make a ``POST`` to ``//
//-/delete``. If successful, this will return a ``200`` status code and a ``{"ok": true}`` response body. -Any errors will return ``{"errors": ["... descriptive message ..."], "ok": false}``, and a ``400`` status code for a bad input or a ``403`` status code for an authentication or permission error. +Any errors will use the :ref:`standard error format `, with a ``400`` status code for a bad input or a ``403`` status code for an authentication or permission error. .. _TableCreateView: @@ -2122,9 +2159,11 @@ If you pass a row to the create endpoint with a primary key that already exists { "ok": false, + "error": "UNIQUE constraint failed: creatures.id", "errors": [ "UNIQUE constraint failed: creatures.id" - ] + ], + "status": 400 } You can avoid this error by passing the same ``"ignore": true`` or ``"replace": true`` options to the create endpoint as you can to the :ref:`insert endpoint `. @@ -2360,7 +2399,7 @@ A successful response returns the new schema and the previous schema. If the req "operations_applied": 11 } -Any errors will return ``{"errors": ["... descriptive message ..."], "ok": false}``, and a ``400`` status code for a bad input or a ``403`` status code for an authentication or permission error. +Any errors will use the :ref:`standard error format `, with a ``400`` status code for a bad input or a ``403`` status code for an authentication or permission error. .. _TableSetColumnTypeView: @@ -2424,7 +2463,7 @@ To clear an existing column type assignment, set ``column_type`` to ``null``: This API stores the assignment in Datasette's internal database, so it can be used with immutable databases as well as mutable ones. -Any errors will return ``{"errors": ["... descriptive message ..."], "ok": false}``, and a ``400`` status code for a bad input or a ``403`` status code for an authentication or permission error. +Any errors will use the :ref:`standard error format `, with a ``400`` status code for a bad input or a ``403`` status code for an authentication or permission error. .. _TableDropView: @@ -2461,4 +2500,4 @@ If you pass the following POST body: Then the table will be dropped and a status ``200`` response of ``{"ok": true}`` will be returned. -Any errors will return ``{"errors": ["... descriptive message ..."], "ok": false}``, and a ``400`` status code for a bad input or a ``403`` status code for an authentication or permission error. +Any errors will use the :ref:`standard error format `, with a ``400`` status code for a bad input or a ``403`` status code for an authentication or permission error. diff --git a/existing-api.md b/existing-api.md index b3eda9d4..1933d287 100644 --- a/existing-api.md +++ b/existing-api.md @@ -44,46 +44,51 @@ directory: every claim below is based on the route table in `datasette/app.py` - Success content type: `application/json; charset=utf-8` (`_shape=array&_nl=on` responses use `text/plain`). -### Error shapes (there are several) +### Error shape (canonical) -The codebase produces **four distinct JSON error shapes**, depending on which -layer generates the error: +Every JSON error response uses one canonical shape, built by `error_body()` +(utils/__init__.py): -1. **Exception handler** (handle_exception.py:21-59) — used when a view raises - `NotFound`, `Forbidden` (JSON paths only — see below), `DatasetteError`, - `BadRequest` etc. and the request path ends in `.json`: +```json +{ + "ok": false, + "error": "all messages joined with '; '", + "errors": ["message", "..."], + "status": 404 +} +``` + +- `errors` is a list of one or more message strings (multi-message + validation errors, e.g. per-row insert errors, list them all). +- `error` is the messages joined with `"; "`. +- `status` always matches the HTTP status code. + +The shape is produced by four code paths, all delegating to `error_body()`: + +1. **Exception handler** (handle_exception.py) — `NotFound`, + `DatasetteError`, `BadRequest` etc. on `.json` paths. `DatasetteError` + `error_dict` context keys are merged in; the legacy `title` key is no + longer emitted in JSON (it survives in the HTML error template context). +2. **The `_error()` helper** (views/base.py:183-184) — the write API, + stored-query API, execute-write and permission-denied paths. +3. **JSON renderer errors** (renderer.py) — SQL errors on table/query + endpoints return HTTP 400 with the canonical keys **plus** the context + keys of the response it could not produce: ```json - {"ok": false, "error": "message", "status": 404, "title": null} + {"ok": false, "error": "no such table: x", "errors": ["no such table: x"], + "status": 400, "rows": [], "truncated": false} ``` -2. **The `_error()` helper** (views/base.py:183-184) — used by the write API, - stored-query API, execute-write and several permission-denied paths: + Invalid `_shape=` values and `_shape=object` misuse (on queries or + pk-less tables) also return canonical 400 errors. +4. **Permission debug endpoints** (`/-/allowed`, `/-/rules`, `/-/check`, + POST `/-/permissions`) — canonical shape (previously bare + `{"error": ...}` objects). - ```json - {"ok": false, "errors": ["message", "..."]} - ``` - - Note: plural `errors`, a list, and no `status`/`title` keys. - -3. **JSON renderer errors** (renderer.py:52-56) — SQL errors on table/query - endpoints return HTTP 400 with the error embedded in the data envelope: - - ```json - {"ok": false, "error": "no such table: x", "rows": [], "truncated": false} - ``` - - An invalid `_shape=` value produces `{"ok": false, "error": "Invalid _shape: x", - "status": 400, "title": null}` (renderer.py:101-108). - -4. **Ad-hoc `{"error": ...}` objects** — the permission debug endpoints - (`/-/allowed`, `/-/rules`, `/-/check`, POST `/-/permissions`) return e.g. - `{"error": "Unknown action: x"}` with no `ok` key (views/special.py). - -Method-not-allowed responses return HTTP 405 -`{"ok": false, "error": "Method not allowed"}` when the path ends in `.json` -or the request content type is `application/json`; plain text otherwise -(views/base.py:53, 88-98). +Method-not-allowed responses return HTTP 405 with the canonical shape when +the path ends in `.json` or the request content type is `application/json`; +plain text otherwise (views/base.py). **`Forbidden` is special:** when a view raises `Forbidden` (e.g. via `ensure_permission`), the default `forbidden()` plugin hook renders an **HTML @@ -144,11 +149,10 @@ build JSON directly): - `array` — response body is a bare JSON array of row objects - `arrayfirst` — bare JSON array of the first column's values - `object` — table views only: an object keyed by primary-key string. - On queries: `{"ok": false, "error": "_shape=object is only available on - tables"}` (with HTTP status 200); on tables without primary keys a similar - error. - - anything else — HTTP 400 `{"ok": false, "error": "Invalid _shape: x", - "status": 400, "title": null}` + On queries or tables without primary keys: a canonical 400 error + (`_shape=object is only available on tables` / + `_shape=object not available for tables with no primary keys`). + - anything else — canonical HTTP 400 error `Invalid _shape: x` - **`_nl=on`** — with `_shape=array` only: newline-delimited JSON, `text/plain`. - **`_json=COLUMN`** (repeatable) — parse that column's string values with `json.loads` so they nest as JSON; parse failures leave the value unchanged. @@ -157,7 +161,7 @@ build JSON directly): - `columns` is stripped from dict-shaped output unless `?_extra=columns` was requested (renderer.py:110-113). - If a SQL error occurred, `_shape` is ignored, HTTP status is 400 and the - envelope carries `"ok": false, "error": ...` (renderer.py:52-56). + envelope carries the canonical error keys alongside `rows`/`truncated`. ### The `?_extra=` system @@ -340,8 +344,8 @@ GET renders a confirmation page (or redirects if anonymous); POST deletes the - **POST** — form-encoded `actor` (JSON string), `permission`, optional `resource_1`, `resource_2`; returns **JSON** `{"action", "allowed", "resource": {"parent", "child", "path"}}` plus - `actor_id` when present. Errors: unknown action → 404 `{"error": ...}`; - child without parent → 400 `{"error": ...}`. + `actor_id` when present. Errors: unknown action → 404; child without + parent → 400 (both canonical error shape). ### GET /-/allowed(.json) @@ -351,7 +355,7 @@ path always renders the HTML form; `.json` returns JSON. - **Permission:** none — reports the **current actor's own** allowed resources. Items gain a `reason` field if the actor also holds `permissions-debug`. -- **Parameters:** `action` (required; missing → 400 `{"error": ...}`, unknown +- **Parameters:** `action` (required; missing → 400 canonical error, unknown → 404), `parent`, `child` (requires `parent`), `page` (default 1), `page_size` (default 50, silently capped at 200). - **Response:** `{"action", "actor_id", "page", "page_size", "total", @@ -497,7 +501,7 @@ queries section). GET → 405. Body is parsed as JSON regardless of content type; invalid JSON → 400 `{"ok": false, "errors": ["Invalid JSON: ..."]}`. -- **Permissions** (all denials → 403 `{"ok": false, "errors": [...]}`, +- **Permissions** (all denials → 403 canonical error JSON, all checked at the **database** level): - `create-table` — always required (`["Permission denied"]`) - `insert-row` — if `rows`/`row` provided (`need insert-row`) @@ -812,8 +816,8 @@ only — views get 400 `"Autocomplete is only available for tables"`. ## The write API -All write endpoints return errors via `_error()` -(`{"ok": false, "errors": [...]}`) and check permissions with +All write endpoints return errors via `_error()` (the canonical error +shape) and check permissions with `datasette.allowed()` directly, so their 403s are JSON (unlike the `Forbidden`-raising read endpoints). Routes: app.py:2719-2762. diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index e630ae4b..9eac543f 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -18,7 +18,17 @@ Findings are grouped by theme. Each carries a priority: --- -## 1. Error responses: four shapes is three too many (P1) +## 1. Error responses: four shapes is three too many (P1) — ✅ IMPLEMENTED + +> **Status:** implemented. All four shapes now delegate to a shared +> `error_body()` helper (`datasette/utils/__init__.py`) producing +> `{"ok": false, "error": "", "errors": [...], "status": }`. +> The `title` key is no longer emitted in JSON; the bare `{"error": ...}` +> debug-endpoint shape is gone; `_shape=object` misuse now returns HTTP 400 +> (part of §1b). Covered by `tests/test_error_shape.py` and documented in +> the "Error responses" section of `docs/json_api.rst`. Still open from +> this section's sub-items: §1a (`Forbidden` → HTML), the write +> canned-query 200 (§1b), and the §1c status outliers. The API currently produces four distinct JSON error shapes depending on which internal layer generates the error: @@ -348,7 +358,7 @@ Two details make tiering urgent rather than optional: ## 10. Summary of P1 items (the pre-1.0 checklist) -1. One canonical JSON error shape; retire the other three (§1). +1. ~~One canonical JSON error shape; retire the other three (§1).~~ ✅ Done. 2. `Forbidden` → JSON 403 for JSON requests (§1a). 3. No `ok: false` with HTTP 200 (§1b: `_shape=object`, write canned-query SQL errors). diff --git a/tests/test_api.py b/tests/test_api.py index f57d0206..e5ed1d23 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -323,20 +323,21 @@ def test_sql_time_limit(app_client_shorter_time_limit): "/fixtures/-/query.json?sql=select+sleep(0.5)", ) assert 400 == response.status + expected_message = ( + "

SQL query took too long. The time limit is controlled by the\n" + 'sql_time_limit_ms\n' + "configuration option.

\n" + '\n' + "" + ) assert response.json == { "ok": False, - "error": ( - "

SQL query took too long. The time limit is controlled by the\n" - 'sql_time_limit_ms\n' - "configuration option.

\n" - '\n' - "" - ), + "error": expected_message, + "errors": [expected_message], "status": 400, - "title": "SQL Interrupted", } @@ -350,7 +351,7 @@ async def test_custom_sql_time_limit(ds_client): "/fixtures/-/query.json?sql=select+sleep(0.01)&_timelimit=5", ) assert response.status_code == 400 - assert response.json()["title"] == "SQL Interrupted" + assert response.json()["error"].startswith("

SQL query took too long.") @pytest.mark.asyncio diff --git a/tests/test_api_write.py b/tests/test_api_write.py index 76797742..17542d4b 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -1,6 +1,6 @@ from datasette.app import Datasette from datasette.events import RenameTableEvent -from datasette.utils import escape_sqlite, sqlite3 +from datasette.utils import error_body, escape_sqlite, sqlite3 from .utils import last_event import pytest import time @@ -788,7 +788,12 @@ async def test_update_row_invalid_key(ds_write): headers=_headers(token), ) assert response.status_code == 400 - assert response.json() == {"ok": False, "errors": ["Invalid keys: bad_key"]} + assert response.json() == { + "ok": False, + "error": "Invalid keys: bad_key", + "errors": ["Invalid keys: bad_key"], + "status": 400, + } @pytest.mark.asyncio @@ -1103,10 +1108,9 @@ async def test_alter_table_foreign_key_requires_fk_table_for_fk_column(ds_write) headers=_headers(write_token(ds_write, permissions=["at"])), ) assert response.status_code == 400 - assert response.json() == { - "ok": False, - "errors": ["operations.0.add_foreign_key.args: fk_column requires fk_table"], - } + assert response.json() == error_body( + ["operations.0.add_foreign_key.args: fk_column requires fk_table"], 400 + ) @pytest.mark.asyncio @@ -1130,10 +1134,9 @@ async def test_alter_table_foreign_key_without_fk_column_requires_single_pk(ds_w headers=_headers(token), ) assert response.status_code == 400 - assert response.json() == { - "ok": False, - "errors": ["Could not detect single primary key for table 'accounts'"], - } + assert response.json() == error_body( + ["Could not detect single primary key for table 'accounts'"], 400 + ) @pytest.mark.asyncio @@ -1199,10 +1202,7 @@ async def test_foreign_key_suggestions_permission_denied(ds_write): headers=_headers(token), ) assert response.status_code == 403 - assert response.json() == { - "ok": False, - "errors": ["Permission denied: need alter-table"], - } + assert response.json() == error_body(["Permission denied: need alter-table"], 403) @pytest.mark.asyncio @@ -1313,10 +1313,7 @@ async def test_foreign_key_targets_permission_denied(ds_write): headers=_headers(token), ) assert response.status_code == 403 - assert response.json() == { - "ok": False, - "errors": ["Permission denied: need create-table"], - } + assert response.json() == error_body(["Permission denied: need create-table"], 403) @pytest.mark.asyncio @@ -1339,10 +1336,7 @@ async def test_alter_table_permission_denied(ds_write): headers=_headers(token), ) assert response.status_code == 403 - assert response.json() == { - "ok": False, - "errors": ["Permission denied: need alter-table"], - } + assert response.json() == error_body(["Permission denied: need alter-table"], 403) @pytest.mark.asyncio @@ -2021,6 +2015,9 @@ async def test_create_table( ) assert response.status_code == expected_status data = response.json() + if expected_response.get("ok") is False: + # Error expectations list their messages; derive the canonical envelope + expected_response = error_body(expected_response["errors"], expected_status) assert data == expected_response # Should have tracked the expected events events = ds_write._tracked_events @@ -2218,13 +2215,12 @@ async def test_create_table_column_validation(ds_write, column, expected_error): ) if expected_error: assert response.status_code == 400 - assert response.json() == {"ok": False, "errors": [expected_error]} + assert response.json() == error_body([expected_error], 400) else: assert response.status_code == 400 - assert response.json() == { - "ok": False, - "errors": ["Could not detect single primary key for table 'owners'"], - } + assert response.json() == error_body( + ["Could not detect single primary key for table 'owners'"], 400 + ) @pytest.mark.asyncio @@ -2262,10 +2258,9 @@ async def test_create_table_foreign_key_without_fk_column_requires_single_pk(ds_ headers=_headers(token), ) assert response.status_code == 400 - assert response.json() == { - "ok": False, - "errors": ["Could not detect single primary key for table 'accounts'"], - } + assert response.json() == error_body( + ["Could not detect single primary key for table 'accounts'"], 400 + ) @pytest.mark.asyncio @@ -2415,10 +2410,9 @@ async def test_create_table_error_if_pk_changed(ds_write): headers=_headers(token), ) assert second_response.status_code == 400 - assert second_response.json() == { - "ok": False, - "errors": ["pk cannot be changed for existing table"], - } + assert second_response.json() == error_body( + ["pk cannot be changed for existing table"], 400 + ) @pytest.mark.asyncio @@ -2442,10 +2436,9 @@ async def test_create_table_error_rows_twice_with_duplicates(ds_write): headers=_headers(token), ) assert second_response.status_code == 400 - assert second_response.json() == { - "ok": False, - "errors": ["UNIQUE constraint failed: test_create_twice.id"], - } + assert second_response.json() == error_body( + ["UNIQUE constraint failed: test_create_twice.id"], 400 + ) @pytest.mark.asyncio @@ -2468,6 +2461,8 @@ async def test_method_not_allowed(ds_write, path): assert response.json() == { "ok": False, "error": "Method not allowed", + "errors": ["Method not allowed"], + "status": 405, } @@ -2535,10 +2530,9 @@ async def test_create_using_alter_against_existing_table( ) if not has_alter_permission: assert response2.status_code == 403 - assert response2.json() == { - "ok": False, - "errors": ["Permission denied: need alter-table"], - } + assert response2.json() == error_body( + ["Permission denied: need alter-table"], 403 + ) else: assert response2.status_code == 201 diff --git a/tests/test_base_view.py b/tests/test_base_view.py index 2cd4d601..c1b0cf20 100644 --- a/tests/test_base_view.py +++ b/tests/test_base_view.py @@ -53,6 +53,8 @@ async def test_get_view(): assert json.loads(post_json_response.body) == { "ok": False, "error": "Method not allowed", + "errors": ["Method not allowed"], + "status": 405, } assert post_json_response.status == 405 diff --git a/tests/test_column_types.py b/tests/test_column_types.py index 45a9e7d1..4e553771 100644 --- a/tests/test_column_types.py +++ b/tests/test_column_types.py @@ -9,7 +9,7 @@ from datasette.column_types import ( ) from datasette.hookspecs import hookimpl from datasette.plugins import pm -from datasette.utils import sqlite3 +from datasette.utils import error_body, sqlite3 from datasette.utils import StartupError import markupsafe import pytest @@ -426,7 +426,7 @@ async def test_set_column_type_api_errors( kwargs["json"] = body response = await ds_ct.client.post("/data/posts/-/set-column-type", **kwargs) assert response.status_code == expected_status - assert response.json() == {"ok": False, "errors": expected_errors} + assert response.json() == error_body(expected_errors, expected_status) @pytest.mark.asyncio diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py new file mode 100644 index 00000000..6747e47a --- /dev/null +++ b/tests/test_error_shape.py @@ -0,0 +1,185 @@ +""" +Tests for the canonical JSON error shape. + +Every JSON error response from Datasette should use one shape: + + { + "ok": false, + "error": "", + "errors": ["", ...], + "status": + } + +Additional context keys (for example "rows" and "truncated" on SQL errors) +are permitted, but "ok", "error", "errors" and "status" must always be +present and the legacy "title" key must not be. + +https://github.com/simonw/datasette/issues - 1.0 API consistency +""" + +import pytest +from datasette.app import Datasette +from datasette.utils import sqlite3 + + +def assert_canonical_error(response, expected_status): + assert response.status_code == expected_status + data = response.json() + assert data["ok"] is False + assert isinstance(data["error"], str) + assert data["error"] + assert isinstance(data["errors"], list) + assert data["errors"] + assert all(isinstance(message, str) for message in data["errors"]) + assert data["error"] == "; ".join(data["errors"]) + assert data["status"] == expected_status + assert "title" not in data + return data + + +@pytest.fixture +def ds_error_shape(tmp_path_factory): + db_directory = tmp_path_factory.mktemp("dbs") + db_path = str(db_directory / "data.db") + conn = sqlite3.connect(db_path) + conn.execute("vacuum") + conn.execute("create table docs (id integer primary key, title text)") + conn.close() + ds = Datasette([db_path]) + ds.root_enabled = True + yield ds + ds.close() + + +# Shape 1: the exception handler (handle_exception.py) + + +@pytest.mark.asyncio +async def test_not_found_error_shape(ds_client): + response = await ds_client.get("/fixtures/no_such_table.json") + assert_canonical_error(response, 404) + + +@pytest.mark.asyncio +async def test_datasette_error_with_title_omits_title_key(ds_client): + # DatasetteError(title="Invalid SQL") previously leaked a "title" key + response = await ds_client.get( + "/fixtures/-/query.json?sql=update+facetable+set+state+=+1" + ) + data = assert_canonical_error(response, 400) + assert data["errors"] == ["Statement must be a SELECT"] + + +# Shape 2: the _error() helper (views/base.py) - write API and friends + + +@pytest.mark.asyncio +async def test_write_api_validation_error_shape(ds_error_shape): + token = "dstok_{}".format( + ds_error_shape.sign( + {"a": "root", "token": "dstok", "t": 0}, + namespace="token", + ) + ) + response = await ds_error_shape.client.post( + "/data/docs/-/insert", + json={"rows": [{"nope": 1}, {"also_nope": 2}]}, + headers={ + "Authorization": "Bearer {}".format(token), + "Content-Type": "application/json", + }, + ) + data = assert_canonical_error(response, 400) + # Multiple messages: errors keeps them all, error joins them + assert len(data["errors"]) == 2 + assert data["errors"][0].startswith("Row 0") + assert data["errors"][1].startswith("Row 1") + + +@pytest.mark.asyncio +async def test_write_api_permission_denied_shape(ds_error_shape): + response = await ds_error_shape.client.post( + "/data/docs/-/insert", + json={"rows": [{"title": "hello"}]}, + headers={"Content-Type": "application/json"}, + ) + assert_canonical_error(response, 403) + + +# Shape 3: the JSON renderer (renderer.py) + + +@pytest.mark.asyncio +async def test_sql_error_shape_keeps_context_keys(ds_client): + response = await ds_client.get( + "/fixtures/-/query.json?sql=select+*+from+no_such_table" + ) + data = assert_canonical_error(response, 400) + # Renderer errors keep their context keys + assert data["rows"] == [] + assert "truncated" in data + + +@pytest.mark.asyncio +async def test_invalid_shape_error_shape(ds_client): + response = await ds_client.get("/fixtures/-/query.json?sql=select+1&_shape=bananas") + data = assert_canonical_error(response, 400) + assert data["errors"] == ["Invalid _shape: bananas"] + + +@pytest.mark.asyncio +async def test_shape_object_on_query_is_a_400_error(ds_client): + # Previously returned HTTP 200 with an ok: false body + response = await ds_client.get("/fixtures/-/query.json?sql=select+1&_shape=object") + data = assert_canonical_error(response, 400) + assert data["errors"] == ["_shape=object is only available on tables"] + + +# Shape 4: bare {"error": ...} from the permission debug endpoints + + +@pytest.mark.asyncio +async def test_allowed_missing_action_error_shape(ds_client): + response = await ds_client.get("/-/allowed.json") + data = assert_canonical_error(response, 400) + assert data["errors"] == ["action parameter is required"] + + +@pytest.mark.asyncio +async def test_allowed_unknown_action_error_shape(ds_client): + response = await ds_client.get("/-/allowed.json?action=no_such_action") + assert_canonical_error(response, 404) + + +@pytest.mark.asyncio +async def test_check_unknown_action_error_shape(ds_error_shape): + response = await ds_error_shape.client.get( + "/-/check.json?action=no_such_action", + actor={"id": "root"}, + ) + assert_canonical_error(response, 404) + + +@pytest.mark.asyncio +async def test_rules_missing_action_error_shape(ds_error_shape): + response = await ds_error_shape.client.get( + "/-/rules.json", + actor={"id": "root"}, + ) + data = assert_canonical_error(response, 400) + assert data["errors"] == ["action parameter is required"] + + +# Other stragglers + + +@pytest.mark.asyncio +async def test_method_not_allowed_error_shape(ds_client): + response = await ds_client.post("/fixtures.json") + assert_canonical_error(response, 405) + + +@pytest.mark.asyncio +async def test_schema_unknown_database_error_shape(ds_client): + response = await ds_client.get("/no_such_db/-/schema.json") + assert_canonical_error(response, 404) diff --git a/tests/test_table_api.py b/tests/test_table_api.py index 272e39e3..c8ba31b7 100644 --- a/tests/test_table_api.py +++ b/tests/test_table_api.py @@ -31,8 +31,8 @@ async def test_table_not_exists_json(ds_client): assert (await ds_client.get("/fixtures/blah.json")).json() == { "ok": False, "error": "Table not found", + "errors": ["Table not found"], "status": 404, - "title": None, } @@ -242,8 +242,8 @@ async def test_table_shape_invalid(ds_client): assert response.json() == { "ok": False, "error": "Invalid _shape: invalid", + "errors": ["Invalid _shape: invalid"], "status": 400, - "title": None, } @@ -635,8 +635,8 @@ async def test_searchable_invalid_column(ds_client): assert response.json() == { "ok": False, "error": "Cannot search by that column", + "errors": ["Cannot search by that column"], "status": 400, - "title": None, } @@ -775,7 +775,7 @@ async def test_table_filter_extra_where_invalid(ds_client): "/fixtures/facetable.json?_where=_neighborhood=Dogpatch'" ) assert response.status_code == 400 - assert "Invalid SQL" == response.json()["title"] + assert "unrecognized token" in response.json()["error"] def test_table_filter_extra_where_disabled_if_no_sql_allowed(): diff --git a/tests/test_table_html.py b/tests/test_table_html.py index 3af2bb08..46d43c6c 100644 --- a/tests/test_table_html.py +++ b/tests/test_table_html.py @@ -1979,8 +1979,8 @@ async def test_sort_errors(ds_client, json, params, error): assert response.json() == { "ok": False, "error": error, + "errors": [error], "status": 400, - "title": None, } else: assert error in response.text From bc51c00724e09f71cd452139153f29eb528fd637 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 05:10:26 +0000 Subject: [PATCH 003/131] Remove legacy .jsono format extension The homepage routes now only accept .json, and the row view no longer redirects .jsono to .json?_shape=objects. The .jsono extension was superseded by ?_shape= and returned output identical to .json. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/app.py | 4 ++-- datasette/views/row.py | 13 ------------- existing-api.md | 6 ++---- stable-api-recommendations.md | 6 ++++-- 4 files changed, 8 insertions(+), 21 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 9c9b7de4..e13f0731 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -2514,8 +2514,8 @@ class Datasette: def add_route(view, regex): routes.append((regex, view)) - add_route(IndexView.as_view(self), r"/(\.(?Pjsono?))?$") - add_route(IndexView.as_view(self), r"/-/(\.(?Pjsono?))?$") + add_route(IndexView.as_view(self), r"/(\.(?Pjson))?$") + add_route(IndexView.as_view(self), r"/-/(\.(?Pjson))?$") add_route(permanent_redirect("/-/"), r"/-$") add_route(favicon, "/favicon.ico") diff --git a/datasette/views/row.py b/datasette/views/row.py index 129216b9..c99b75d1 100644 --- a/datasette/views/row.py +++ b/datasette/views/row.py @@ -21,7 +21,6 @@ from datasette.utils import ( InvalidSql, make_slot_function, path_from_row_pks, - path_with_added_args, path_with_format, path_with_removed_args, to_css_class, @@ -209,18 +208,6 @@ class RowView(BaseView): end = time.perf_counter() data["query_ms"] = (end - start) * 1000 - # Special case for .jsono extension - redirect to _shape=objects - if format_ == "jsono": - return self.redirect( - request, - path_with_added_args( - request, - {"_shape": "objects"}, - path=request.path.rsplit(".jsono", 1)[0] + ".json", - ), - forward_querystring=False, - ) - if format_ in self.ds.renderers.keys(): # Dispatch request to the correct output format renderer # (CSV is not handled here due to streaming) diff --git a/existing-api.md b/existing-api.md index 1933d287..d7678613 100644 --- a/existing-api.md +++ b/existing-api.md @@ -25,8 +25,7 @@ directory: every claim below is based on the route table in `datasette/app.py` - Most read endpoints are registered with an optional format suffix: `/(...)(\.(?Pjson))?$`. The bare path returns HTML; the `.json` - extension returns JSON. The homepage additionally accepts the legacy - `.jsono` extension, which returns identical JSON (app.py:2517-2518). + extension returns JSON. - Table, row and query routes accept any `\w+` format extension; formats other than the built-in `html`, `json`, `csv`, `blob` must be provided by a plugin via `register_output_renderer`, otherwise the request 404s. @@ -772,8 +771,7 @@ tilde-encoded primary key values (rowid for rowid tables). - **Foreign-key label expansion does not apply to row JSON** — `_labels` has no effect here; expansion happens only in the HTML path (views/row.py:445-475). -- `_shape`, `_json`, `_nl`, `_json_infinity`, `_ttl` apply. A `.jsono` - request redirects to `.json?_shape=objects`. +- `_shape`, `_json`, `_nl`, `_json_infinity`, `_ttl` apply. ### The .blob format diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index 9eac543f..8d8a87b4 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -233,8 +233,10 @@ Concerns: `_labels=on/off`, `?all=1`, `is_write=1|0|true|false|t|f|yes|no|on|off`, `_nocount=1`. Adopt one accepted set (the query-list parser at query_helpers.py:81-94 is a good candidate) and apply it everywhere. -- **`.jsono`** survives on the homepage route (identical output to `.json`) - and as a row-view redirect. Remove it at 1.0; it is pure legacy. +- ~~**`.jsono`** survives on the homepage route (identical output to `.json`) + and as a row-view redirect. Remove it at 1.0; it is pure legacy.~~ + ✅ Removed: the homepage routes only accept `.json` and the row-view + redirect is gone. - **`_json` is overloaded:** on GET it is a renderer option naming a column to parse as JSON (repeatable); on canned-query POST a `_json` body field forces a JSON response. Two unrelated meanings for one name. From 089e96a43763c5f215218498335d81958c59a414 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 13:40:05 +0000 Subject: [PATCH 004/131] Add "ok": true to every JSON object success response JsonDataView now injects "ok": true into dict responses, covering /-/versions, /-/settings, /-/config, /-/threads and /-/actor. The homepage JSON, /-/jump, the three /-/schema endpoints, /-/allowed, /-/rules, /-/check, POST /-/permissions and the table /-/autocomplete endpoint set it explicitly. The remaining top-level array endpoints (/-/plugins, /-/databases, /-/actions) will be converted to objects in separate commits. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/views/index.py | 1 + datasette/views/special.py | 10 ++- datasette/views/table.py | 9 ++- docs/introspection.rst | 9 +++ docs/json_api.rst | 2 +- docs/pages.rst | 6 +- existing-api.md | 30 +++++---- stable-api-recommendations.md | 10 ++- tests/test_api.py | 8 ++- tests/test_auth.py | 8 +-- tests/test_autocomplete.py | 25 +++++--- tests/test_cli_serve_get.py | 8 ++- tests/test_docs.py | 4 +- tests/test_internals_datasette.py | 2 +- tests/test_internals_datasette_client.py | 6 +- tests/test_permissions.py | 3 +- tests/test_plugins.py | 8 ++- tests/test_success_envelope.py | 78 ++++++++++++++++++++++++ 18 files changed, 180 insertions(+), 47 deletions(-) create mode 100644 tests/test_success_envelope.py diff --git a/datasette/views/index.py b/datasette/views/index.py index 6a9462ac..86f8ae7d 100644 --- a/datasette/views/index.py +++ b/datasette/views/index.py @@ -151,6 +151,7 @@ class IndexView(BaseView): return Response( json.dumps( { + "ok": True, "databases": {db["name"]: db for db in databases}, "metadata": await self.ds.get_instance_metadata(), }, diff --git a/datasette/views/special.py b/datasette/views/special.py index 602ec5ea..c289a240 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -63,6 +63,8 @@ class JsonDataView(BaseView): headers = {} if self.ds.cors: add_cors_headers(headers) + if isinstance(data, dict): + data = {"ok": True, **data} return Response.json(data, headers=headers) else: context = { @@ -414,6 +416,7 @@ class AllowedResourcesView(BaseView): # If catalog tables don't exist yet, return empty results return ( { + "ok": True, "action": action, "actor_id": actor_id, "page": page, @@ -448,6 +451,7 @@ class AllowedResourcesView(BaseView): return f"{request.path}?{query}" response = { + "ok": True, "action": action, "actor_id": actor_id, "page": page, @@ -573,6 +577,7 @@ class PermissionRulesView(BaseView): return f"{request.path}?{query}" response = { + "ok": True, "action": action, "actor_id": (actor or {}).get("id") if actor else None, "page": page, @@ -623,6 +628,7 @@ async def _check_permission_for_actor(ds, action, parent, child, actor): allowed = await ds.allowed(action=action, resource=resource_obj, actor=actor) response = { + "ok": True, "action": action, "allowed": bool(allowed), "resource": { @@ -1208,7 +1214,7 @@ class JumpView(BaseView): match["display_name"] = row["display_name"] matches.append(match) - return Response.json({"matches": matches, "truncated": truncated}) + return Response.json({"ok": True, "matches": matches, "truncated": truncated}) class SchemaBaseView(BaseView): @@ -1230,7 +1236,7 @@ class SchemaBaseView(BaseView): headers = {} if self.ds.cors: add_cors_headers(headers) - return Response.json(data, headers=headers) + return Response.json({"ok": True, **data}, headers=headers) def format_error_response(self, error_message, format_, status=404): """Format error response based on requested format.""" diff --git a/datasette/views/table.py b/datasette/views/table.py index 1fc151e6..ef9831b2 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1525,7 +1525,7 @@ class TableAutocompleteView(BaseView): and value_as_boolean(initial_arg) ) if not q and not initial: - return Response.json({"rows": []}) + return Response.json({"ok": True, "rows": []}) params = { "q": q, "like": "%{}%".format(_escape_like(q)), @@ -1588,10 +1588,13 @@ class TableAutocompleteView(BaseView): custom_time_limit=AUTOCOMPLETE_TIME_LIMIT_MS, ) except QueryInterrupted: - return Response.json({"rows": []}) + return Response.json({"ok": True, "rows": []}) return Response.json( - {"rows": _autocomplete_response_rows(results.rows, pks, label_column)} + { + "ok": True, + "rows": _autocomplete_response_rows(results.rows, pks, label_column), + } ) diff --git a/docs/introspection.rst b/docs/introspection.rst index 4834f441..d0780763 100644 --- a/docs/introspection.rst +++ b/docs/introspection.rst @@ -7,6 +7,8 @@ Datasette includes some pages and JSON API endpoints for introspecting the curre Each of these pages can be viewed in your browser. Add ``.json`` to the URL to get back the contents as JSON. +JSON responses that return an object include an ``"ok": true`` key, consistent with the rest of the :ref:`JSON API `. + .. _JsonDataView_metadata: /-/metadata @@ -37,6 +39,7 @@ Shows the version of Datasette, Python and SQLite. `Versions example ` for this instance of Datasette. T .. code-block:: json { + "ok": true, "settings": { "template_debug": true, "trace_debug": true, @@ -160,6 +165,7 @@ The endpoint supports a ``?q=`` query parameter for filtering items by name. .. code-block:: json { + "ok": true, "matches": [ { "name": "fixtures", @@ -188,6 +194,7 @@ Search example with ``?q=facet`` returns only items matching ``.*facet.*``: .. code-block:: json { + "ok": true, "matches": [ { "name": "fixtures: facetable", @@ -220,6 +227,7 @@ Shows details of threads and ``asyncio`` tasks. `Threads example jsono?))?$` and `/-/(\.(?Pjsono?))?$` further filtered by `view-database` / `view-table` for the actor. - **Parameters:** `_sort=relationships` sorts each database's truncated table list by foreign-key relationship count. -- **JSON response** (index.py:147-161): +- **JSON response** (index.py:147-161) — includes `ok: true` plus: - `databases` — an **object keyed by database name** (not a list). Each value: `name`, `hash` (or null), `color`, `path`, `tables_and_views_truncated` (up to 5 items: `name`, `columns`, @@ -260,7 +269,7 @@ per-database `view-database` permissions**. app.py:2574-2579, registered with `permission=None` — **accessible to any request including anonymous**. No parameters. -Response: `{"actor": {...}}` or `{"actor": null}` (app.py:2287-2288). +Response: `{"ok": true, "actor": {...}}` or `{"ok": true, "actor": null}` (app.py:2287-2288). ### GET /-/actions(.json) @@ -313,7 +322,7 @@ an optional `.json` suffix but the view **always returns JSON**. `jump_items_sql` plugin hook). - **Parameter:** `q` — whitespace-split terms matched as a case-insensitive `%term1%term2%` LIKE pattern. -- **Response:** `{"matches": [...], "truncated": bool}`; each match: +- **Response:** `{"ok": true, "matches": [...], "truncated": bool}`; each match: `name`, `url`, `type` (`database`/`table`/`view`/`query`/plugin-defined), `description`, optional `display_name`. Capped at 100 matches. @@ -324,7 +333,7 @@ an optional `.json` suffix but the view **always returns JSON**. - **Permission:** no explicit check; only databases the actor can `view-database` are included (others silently omitted). - **Formats:** no extension → HTML; `.json` → - `{"schemas": [{"database": name, "schema": "..."}]}`; `.md` → + `{"ok": true, "schemas": [{"database": name, "schema": "..."}]}`; `.md` → `text/markdown` rendering. ### GET/POST /-/logout @@ -610,10 +619,9 @@ views/table_create_alter.py:965-1005). - **Unknown database** → 404; for `.json`: `{"ok": false, "error": "Database not found"}`. (The existence check runs before the permission check.) -- **Responses:** `.json` → 200 `{"database": "", "schema": ""}` +- **Responses:** `.json` → 200 `{"ok": true, "database": "", "schema": ""}` (concatenated `sqlite_master.sql` joined with `;\n`); `.md` → - `text/markdown`; no extension → HTML. Note the JSON has **no `ok` key** on - success. + `text/markdown`; no extension → HTML. --- @@ -787,8 +795,8 @@ download attachment. In JSON output, binary cells appear as `TableSchemaView` (app.py:2751-2754; views/special.py:1332-1378). - **Permission:** `view-table` via `ensure_permission` (denied → 403 HTML). -- **Responses:** `.json` → 200 `{"database", "table", "schema"}` (no `ok` - key); `.md` → `text/markdown`; no extension → HTML. Missing table → 404 +- **Responses:** `.json` → 200 `{"ok": true, "database", "table", "schema"}`; + `.md` → `text/markdown`; no extension → HTML. Missing table → 404 `{"ok": false, "error": "Table not found"}` for `.json`. ### GET /\/\/-/fragment @@ -805,10 +813,10 @@ only — views get 400 `"Autocomplete is only available for tables"`. - **Permission:** `view-table` (denied → `Forbidden` → 403). - **Parameters:** `q` (matched with escaped `LIKE %q%` against pk columns and the label column) and `_initial` (truthy: with empty `q`, return the 10 - most recent rows). Neither → `{"rows": []}`. + most recent rows). Neither → `{"ok": true, "rows": []}`. - **Response:** `{"rows": [{"pks": {pk_name: value}, "label": "..."}]}` — max 10 items; 500 ms query budget with fallbacks, timing out to - `{"rows": []}`. + `{"ok": true, "rows": []}`. --- diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index 8d8a87b4..48a864af 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -102,7 +102,15 @@ key — see §2.) --- -## 2. Success envelope: `ok` is not universal, arrays are not extensible (P1/P2) +## 2. Success envelope: `ok` is not universal, arrays are not extensible (P1/P2) — ✅ PARTIALLY IMPLEMENTED + +> **Status:** recommendation 2 and 3 are implemented — every JSON-object +> success response now includes `"ok": true` (`JsonDataView` injects it for +> dict responses; homepage, jump, schema, permission-debug and autocomplete +> views set it explicitly; covered by `tests/test_success_envelope.py`). +> Recommendation 1 (wrapping the `/-/plugins`, `/-/databases`, `/-/actions` +> top-level arrays in objects) is being landed as separate per-endpoint +> commits. §2a/2b/2c remain open. Endpoints disagree about the success envelope: diff --git a/tests/test_api.py b/tests/test_api.py index e5ed1d23..4635b236 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -250,6 +250,7 @@ def test_no_files_uses_memory_database(app_client_no_files): response = app_client_no_files.get("/.json") assert response.status == 200 assert { + "ok": True, "databases": { "_memory": { "name": "_memory", @@ -525,7 +526,7 @@ def test_databases_json(app_client_two_attached_databases_one_immutable): @pytest.mark.asyncio async def test_threads_json(ds_client): response = await ds_client.get("/-/threads.json") - expected_keys = {"threads", "num_threads"} + expected_keys = {"ok", "threads", "num_threads"} if sys.version_info >= (3, 7, 0): expected_keys.update({"tasks", "num_tasks"}) data = response.json() @@ -610,6 +611,7 @@ async def test_actions_json(ds_client): async def test_settings_json(ds_client): response = await ds_client.get("/-/settings.json") assert response.json() == { + "ok": True, "default_page_size": 50, "default_facet_size": 30, "default_allow_sql": True, @@ -884,7 +886,7 @@ async def test_config_json(config, expected): "/-/config.json should return redacted configuration" ds = Datasette(config=config) response = await ds.client.get("/-/config.json") - assert response.json() == expected + assert response.json() == {"ok": True, **expected} @pytest.mark.asyncio @@ -980,7 +982,7 @@ async def test_config_json(config, expected): async def test_upgrade_metadata(metadata, expected_config, expected_metadata): ds = Datasette(metadata=metadata) response = await ds.client.get("/-/config.json") - assert response.json() == expected_config + assert response.json() == {"ok": True, **expected_config} response2 = await ds.client.get("/-/metadata.json") assert response2.json() == expected_metadata diff --git a/tests/test_auth.py b/tests/test_auth.py index 5868a21c..8e83d397 100644 --- a/tests/test_auth.py +++ b/tests/test_auth.py @@ -294,7 +294,7 @@ async def test_auth_with_dstok_token(ds_client, scenario, should_work): try: if should_work: data = response.json() - assert data.keys() == {"actor"} + assert data.keys() == {"ok", "actor"} actor = data["actor"] expected_keys = {"id", "token"} if scenario != "valid_unlimited_token": @@ -305,7 +305,7 @@ async def test_auth_with_dstok_token(ds_client, scenario, should_work): if scenario != "valid_unlimited_token": assert isinstance(actor["token_expires"], int) else: - assert response.json() == {"actor": None} + assert response.json() == {"ok": True, "actor": None} finally: ds_client.ds._settings["allow_signed_tokens"] = True @@ -337,10 +337,10 @@ def test_cli_create_token(app_client, expires): } if expires and expires > 0: expected_actor["token_expires"] = details["t"] + expires - assert response.json == {"actor": expected_actor} + assert response.json == {"ok": True, "actor": expected_actor} else: expected_actor = None - assert response.json == {"actor": expected_actor} + assert response.json == {"ok": True, "actor": expected_actor} @pytest.mark.asyncio diff --git a/tests/test_autocomplete.py b/tests/test_autocomplete.py index 76b9c902..194fcf01 100644 --- a/tests/test_autocomplete.py +++ b/tests/test_autocomplete.py @@ -25,13 +25,14 @@ async def test_autocomplete_single_pk_exact_match_and_label_order(): assert response.status_code == 200 assert response.json() == { + "ok": True, "rows": [ {"pks": {"id": 2}, "label": "Longer non-label pk match"}, {"pks": {"id": 20}, "label": "2"}, {"pks": {"id": 21}, "label": "22"}, {"pks": {"id": 3}, "label": "A label containing 2"}, {"pks": {"id": 200}, "label": "A"}, - ] + ], } @@ -52,12 +53,12 @@ async def test_autocomplete_blank_q_returns_no_results(): response = await ds.client.get("/autocomplete_blank/people/-/autocomplete?q=") assert response.status_code == 200 - assert response.json() == {"rows": []} + assert response.json() == {"ok": True, "rows": []} response = await ds.client.get("/autocomplete_blank/people/-/autocomplete") assert response.status_code == 200 - assert response.json() == {"rows": []} + assert response.json() == {"ok": True, "rows": []} @pytest.mark.asyncio @@ -81,11 +82,12 @@ async def test_autocomplete_initial_returns_latest_rows(): assert response.status_code == 200 assert response.json() == { + "ok": True, "rows": [ {"pks": {"id": 3}, "label": "Cleo"}, {"pks": {"id": 2}, "label": "Bob"}, {"pks": {"id": 1}, "label": "Alice"}, - ] + ], } response = await ds.client.get( @@ -94,11 +96,12 @@ async def test_autocomplete_initial_returns_latest_rows(): assert response.status_code == 200 assert response.json() == { + "ok": True, "rows": [ {"pks": {"id": 3}, "label": "Cleo"}, {"pks": {"id": 2}, "label": "Bob"}, {"pks": {"id": 1}, "label": "Alice"}, - ] + ], } @@ -121,9 +124,10 @@ async def test_autocomplete_escapes_like_characters(): assert response.status_code == 200 assert response.json() == { + "ok": True, "rows": [ {"pks": {"id": 1}, "label": "100% real"}, - ] + ], } @@ -149,11 +153,12 @@ async def test_autocomplete_compound_pk_searches_all_pk_columns(): assert response.status_code == 200 assert response.json() == { + "ok": True, "rows": [ {"pks": {"country": "mx", "code": "ca"}, "label": "Campeche"}, {"pks": {"country": "us", "code": "ca"}, "label": "California"}, {"pks": {"country": "ca", "code": "bc"}, "label": "British Columbia"}, - ] + ], } @@ -184,9 +189,10 @@ async def test_autocomplete_primary_key_called_label(): assert response.status_code == 200 assert response.json() == { + "ok": True, "rows": [ {"pks": {"label": "abc"}, "label": "Display value"}, - ] + ], } @@ -246,8 +252,9 @@ async def test_autocomplete_timeout_uses_prefix_fallback(monkeypatch): assert timeout_was_simulated data = response.json() assert data == { + "ok": True, "rows": [ {"pks": {"id": f"item-1999{i:02d}"}, "label": f"name 1999{i:02d}"} for i in range(10) - ] + ], } diff --git a/tests/test_cli_serve_get.py b/tests/test_cli_serve_get.py index dc852201..fe9416d6 100644 --- a/tests/test_cli_serve_get.py +++ b/tests/test_cli_serve_get.py @@ -95,7 +95,10 @@ def test_serve_with_get_and_token(): ], ) assert 0 == result2.exit_code, result2.output - assert json.loads(result2.output) == {"actor": {"id": "root", "token": "dstok"}} + assert json.loads(result2.output) == { + "ok": True, + "actor": {"id": "root", "token": "dstok"}, + } def test_serve_with_get_exit_code_for_error(): @@ -130,8 +133,9 @@ def test_serve_get_actor(): ) assert result.exit_code == 0 assert json.loads(result.output) == { + "ok": True, "actor": { "id": "root", "extra": "x", - } + }, } diff --git a/tests/test_docs.py b/tests/test_docs.py index 13b3a549..0bcb5e62 100644 --- a/tests/test_docs.py +++ b/tests/test_docs.py @@ -248,7 +248,7 @@ async def test_homepage(): async def test_actor_is_null(): ds = Datasette(memory=True) response = await ds.client.get("/-/actor.json") - assert response.json() == {"actor": None} + assert response.json() == {"ok": True, "actor": None} # -- end test_actor_is_null -- @@ -258,5 +258,5 @@ async def test_signed_cookie_actor(): ds = Datasette(memory=True) cookies = {"ds_actor": ds.client.actor_cookie({"id": "root"})} response = await ds.client.get("/-/actor.json", cookies=cookies) - assert response.json() == {"actor": {"id": "root"}} + assert response.json() == {"ok": True, "actor": {"id": "root"}} # -- end test_signed_cookie_actor -- diff --git a/tests/test_internals_datasette.py b/tests/test_internals_datasette.py index 2eaee3f9..1de394a5 100644 --- a/tests/test_internals_datasette.py +++ b/tests/test_internals_datasette.py @@ -188,7 +188,7 @@ async def test_num_sql_threads_zero(): await db.execute_write("create table t(id integer primary key)") await db.execute_write("insert into t (id) values (1)") response = await ds.client.get("/-/threads.json") - assert response.json() == {"num_threads": 0, "threads": []} + assert response.json() == {"ok": True, "num_threads": 0, "threads": []} response2 = await ds.client.get("/test_num_sql_threads_zero/t.json?_shape=array") assert response2.json() == [{"id": 1}] diff --git a/tests/test_internals_datasette_client.py b/tests/test_internals_datasette_client.py index 543077a5..e9aaaae8 100644 --- a/tests/test_internals_datasette_client.py +++ b/tests/test_internals_datasette_client.py @@ -318,7 +318,7 @@ async def test_actor_parameter_sets_cookie(datasette): """Passing actor= should sign a ds_actor cookie and authenticate the request.""" response = await datasette.client.get("/-/actor.json", actor={"id": "root"}) assert response.status_code == 200 - assert response.json() == {"actor": {"id": "root"}} + assert response.json() == {"ok": True, "actor": {"id": "root"}} @pytest.mark.asyncio @@ -327,7 +327,7 @@ async def test_actor_parameter_works_with_request_method(datasette): "GET", "/-/actor.json", actor={"id": "root"} ) assert response.status_code == 200 - assert response.json() == {"actor": {"id": "root"}} + assert response.json() == {"ok": True, "actor": {"id": "root"}} @pytest.mark.asyncio @@ -362,7 +362,7 @@ async def test_actor_parameter_merges_with_other_cookies(datasette): cookies={"unrelated": "value"}, ) assert response.status_code == 200 - assert response.json() == {"actor": {"id": "root"}} + assert response.json() == {"ok": True, "actor": {"id": "root"}} @pytest.mark.asyncio diff --git a/tests/test_permissions.py b/tests/test_permissions.py index 8323fe92..7d99213b 100644 --- a/tests/test_permissions.py +++ b/tests/test_permissions.py @@ -739,6 +739,7 @@ async def test_actor_restricted_permissions( "path": expected_path, } expected = { + "ok": True, "action": permission, "allowed": expected_result, "resource": expected_resource, @@ -1115,7 +1116,7 @@ def test_cli_create_token(options, expected): ], ) assert 0 == result2.exit_code, result2.output - assert json.loads(result2.output) == {"actor": expected} + assert json.loads(result2.output) == {"ok": True, "actor": expected} _visible_tables_re = re.compile(r">\/((\w+)\/(\w+))\.json<\/a> - Get rows for") diff --git a/tests/test_plugins.py b/tests/test_plugins.py index da14c714..59b1c0bf 100644 --- a/tests/test_plugins.py +++ b/tests/test_plugins.py @@ -783,9 +783,13 @@ async def test_hook_permission_resources_sql(): @pytest.mark.asyncio async def test_actor_json(ds_client): - assert (await ds_client.get("/-/actor.json")).json() == {"actor": None} + assert (await ds_client.get("/-/actor.json")).json() == { + "ok": True, + "actor": None, + } assert (await ds_client.get("/-/actor.json?_bot2=1")).json() == { - "actor": {"id": "bot2", "1+1": 2} + "ok": True, + "actor": {"id": "bot2", "1+1": 2}, } diff --git a/tests/test_success_envelope.py b/tests/test_success_envelope.py new file mode 100644 index 00000000..48407660 --- /dev/null +++ b/tests/test_success_envelope.py @@ -0,0 +1,78 @@ +""" +Tests for the canonical JSON success envelope. + +Every JSON object returned by a Datasette endpoint on success should include +"ok": true. (Endpoints that return a top-level array are being converted to +objects separately - see /-/plugins, /-/databases, /-/actions.) +""" + +import pytest +from datasette.app import Datasette +from datasette.utils import sqlite3 + + +@pytest.fixture +def ds_envelope(tmp_path_factory): + db_directory = tmp_path_factory.mktemp("dbs") + db_path = str(db_directory / "data.db") + conn = sqlite3.connect(db_path) + conn.execute("vacuum") + conn.execute("create table docs (id integer primary key, title text)") + conn.close() + ds = Datasette([db_path]) + ds.root_enabled = True + yield ds + ds.close() + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "path", + ( + "/.json", + "/-/.json", + "/-/versions.json", + "/-/settings.json", + "/-/config.json", + "/-/threads.json", + "/-/actor.json", + "/-/jump.json", + "/-/schema.json", + "/fixtures/-/schema.json", + "/fixtures/facetable/-/schema.json", + "/-/allowed.json?action=view-instance", + "/fixtures/facet_cities/-/autocomplete?_initial=1", + ), +) +async def test_success_object_has_ok_true(ds_client, path): + response = await ds_client.get(path) + assert response.status_code == 200 + data = response.json() + assert isinstance(data, dict) + assert data["ok"] is True + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "path", + ( + "/-/rules.json?action=view-instance", + "/-/check.json?action=view-instance", + ), +) +async def test_permission_debug_success_has_ok_true(ds_envelope, path): + response = await ds_envelope.client.get(path, actor={"id": "root"}) + assert response.status_code == 200 + data = response.json() + assert data["ok"] is True + + +@pytest.mark.asyncio +async def test_permissions_post_success_has_ok_true(ds_envelope): + response = await ds_envelope.client.post( + "/-/permissions", + data={"actor": '{"id": "root"}', "permission": "view-instance"}, + actor={"id": "root"}, + ) + assert response.status_code == 200 + assert response.json()["ok"] is True From b74a8e5b12b6e69a8717d0b0b7a37f12162f8ffc Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 13:42:24 +0000 Subject: [PATCH 005/131] Convert /-/plugins.json from top-level array to object /-/plugins.json now returns {"ok": true, "plugins": [...]} instead of a bare JSON array, so the response can grow additional keys without a breaking change. The `datasette plugins` CLI command still outputs a plain array. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/app.py | 5 ++++- docs/introspection.rst | 21 ++++++++++++--------- existing-api.md | 4 ++-- tests/test_api.py | 4 ++-- tests/test_config_dir.py | 7 ++++--- tests/test_plugins.py | 2 +- tests/test_success_envelope.py | 14 ++++++++++++++ 7 files changed, 39 insertions(+), 18 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index e13f0731..b554f9bc 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -2551,7 +2551,10 @@ class Datasette: ) add_route( JsonDataView.as_view( - self, "plugins.json", self._plugins, needs_request=True + self, + "plugins.json", + lambda request: {"plugins": self._plugins(request)}, + needs_request=True, ), r"/-/plugins(\.(?Pjson))?$", ) diff --git a/docs/introspection.rst b/docs/introspection.rst index d0780763..99238204 100644 --- a/docs/introspection.rst +++ b/docs/introspection.rst @@ -78,15 +78,18 @@ Shows a list of currently installed plugins and their versions. `Plugins example .. code-block:: json - [ - { - "name": "datasette_cluster_map", - "static": true, - "templates": false, - "version": "0.10", - "hooks": ["extra_css_urls", "extra_js_urls", "extra_body_script"] - } - ] + { + "ok": true, + "plugins": [ + { + "name": "datasette_cluster_map", + "static": true, + "templates": false, + "version": "0.10", + "hooks": ["extra_css_urls", "extra_js_urls", "extra_body_script"] + } + ] + } Add ``?all=1`` to include details of the default plugins baked into Datasette. diff --git a/existing-api.md b/existing-api.md index 0e7ec907..db613d30 100644 --- a/existing-api.md +++ b/existing-api.md @@ -228,8 +228,8 @@ app.py:2552-2557, `Datasette._plugins` (app.py:2247-2266). Permission - **Parameters:** `?all=1` — include Datasette's built-in default plugins (filtered out by default). -- **Response:** a JSON **array**, sorted by name, of - `{"name", "static", "templates", "version", "hooks"}`. +- **Response:** `{"ok": true, "plugins": [...]}` — each plugin is + `{"name", "static", "templates", "version", "hooks"}`, sorted by name. ### GET /-/settings(.json) diff --git a/tests/test_api.py b/tests/test_api.py index 4635b236..3583503c 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -541,13 +541,13 @@ async def test_plugins_json(ds_client): response = await ds_client.get("/-/plugins.json") # Filter out TrackEventPlugin actual_plugins = sorted( - [p for p in response.json() if p["name"] != "TrackEventPlugin"], + [p for p in response.json()["plugins"] if p["name"] != "TrackEventPlugin"], key=lambda p: p["name"], ) assert EXPECTED_PLUGINS == actual_plugins # Try with ?all=1 response = await ds_client.get("/-/plugins.json?all=1") - names = {p["name"] for p in response.json()} + names = {p["name"] for p in response.json()["plugins"]} assert names.issuperset(p["name"] for p in EXPECTED_PLUGINS) assert names.issuperset(DEFAULT_PLUGINS) diff --git a/tests/test_config_dir.py b/tests/test_config_dir.py index 0a9b30d8..74407e20 100644 --- a/tests/test_config_dir.py +++ b/tests/test_config_dir.py @@ -109,9 +109,10 @@ def test_settings(config_dir_client): def test_plugins(config_dir_client): response = config_dir_client.get("/-/plugins.json") assert 200 == response.status - assert "hooray.py" in {p["name"] for p in response.json} - assert "non_py_file.txt" not in {p["name"] for p in response.json} - assert "mypy_cache" not in {p["name"] for p in response.json} + plugins = response.json["plugins"] + assert "hooray.py" in {p["name"] for p in plugins} + assert "non_py_file.txt" not in {p["name"] for p in plugins} + assert "mypy_cache" not in {p["name"] for p in plugins} def test_templates_and_plugin(config_dir_client): diff --git a/tests/test_plugins.py b/tests/test_plugins.py index 59b1c0bf..5c4034db 100644 --- a/tests/test_plugins.py +++ b/tests/test_plugins.py @@ -1482,7 +1482,7 @@ async def test_plugin_is_installed(): datasette.pm.register(DummyPlugin(), name="DummyPlugin") response = await datasette.client.get("/-/plugins.json") assert response.status_code == 200 - installed_plugins = {p["name"] for p in response.json()} + installed_plugins = {p["name"] for p in response.json()["plugins"]} assert "DummyPlugin" in installed_plugins finally: diff --git a/tests/test_success_envelope.py b/tests/test_success_envelope.py index 48407660..cecc8d70 100644 --- a/tests/test_success_envelope.py +++ b/tests/test_success_envelope.py @@ -76,3 +76,17 @@ async def test_permissions_post_success_has_ok_true(ds_envelope): ) assert response.status_code == 200 assert response.json()["ok"] is True + + +@pytest.mark.asyncio +async def test_plugins_json_is_object(ds_client): + response = await ds_client.get("/-/plugins.json") + assert response.status_code == 200 + data = response.json() + assert set(data.keys()) == {"ok", "plugins"} + assert data["ok"] is True + assert isinstance(data["plugins"], list) + # ?all=1 should include Datasette's default plugins in the same shape + response_all = await ds_client.get("/-/plugins.json?all=1") + all_plugins = response_all.json()["plugins"] + assert len(all_plugins) > len(data["plugins"]) From 19e54b10d4d955e22346339a38f34190e3a1b36b Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 13:43:44 +0000 Subject: [PATCH 006/131] Convert /-/databases.json from top-level array to object /-/databases.json now returns {"ok": true, "databases": [...]} instead of a bare JSON array, so the response can grow additional keys without a breaking change. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/app.py | 6 +++++- docs/introspection.rst | 23 +++++++++++++---------- existing-api.md | 7 ++++--- tests/test_api.py | 6 ++++-- tests/test_cli.py | 6 +++--- tests/test_config_dir.py | 2 +- tests/test_internals_datasette.py | 2 +- tests/test_routes.py | 2 +- tests/test_success_envelope.py | 11 +++++++++++ 9 files changed, 43 insertions(+), 22 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index b554f9bc..bc35669f 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -2571,7 +2571,11 @@ class Datasette: r"/-/threads(\.(?Pjson))?$", ) add_route( - JsonDataView.as_view(self, "databases.json", self._connected_databases), + JsonDataView.as_view( + self, + "databases.json", + lambda: {"databases": self._connected_databases()}, + ), r"/-/databases(\.(?Pjson))?$", ) add_route( diff --git a/docs/introspection.rst b/docs/introspection.rst index 99238204..ab47c0a5 100644 --- a/docs/introspection.rst +++ b/docs/introspection.rst @@ -141,16 +141,19 @@ Shows currently attached databases. `Databases example len(data["plugins"]) + + +@pytest.mark.asyncio +async def test_databases_json_is_object(ds_client): + response = await ds_client.get("/-/databases.json") + assert response.status_code == 200 + data = response.json() + assert set(data.keys()) == {"ok", "databases"} + assert data["ok"] is True + assert isinstance(data["databases"], list) + assert "fixtures" in {db["name"] for db in data["databases"]} From 23ccdaeffc1d403563933af5fcfa90be6947f7f4 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 13:46:26 +0000 Subject: [PATCH 007/131] Convert /-/actions.json from top-level array to object /-/actions.json now returns {"ok": true, "actions": [...]} instead of a bare JSON array, so the response can grow additional keys without a breaking change. The debug_actions.html template reads data.actions, and the endpoint is now documented in docs/introspection.rst. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/app.py | 2 +- datasette/templates/debug_actions.html | 4 ++-- docs/introspection.rst | 24 ++++++++++++++++++++++++ existing-api.md | 12 ++++++------ stable-api-recommendations.md | 19 +++++++++++-------- tests/test_api.py | 2 +- tests/test_success_envelope.py | 11 +++++++++++ 7 files changed, 56 insertions(+), 18 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index bc35669f..5afca2a2 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -2588,7 +2588,7 @@ class Datasette: JsonDataView.as_view( self, "actions.json", - self._actions, + lambda: {"actions": self._actions()}, template="debug_actions.html", permission="permissions-debug", ), diff --git a/datasette/templates/debug_actions.html b/datasette/templates/debug_actions.html index 0ef7b329..c9dccaaa 100644 --- a/datasette/templates/debug_actions.html +++ b/datasette/templates/debug_actions.html @@ -9,7 +9,7 @@ {% include "_permissions_debug_tabs.html" %}

- This Datasette instance has registered {{ data|length }} action{{ data|length != 1 and "s" or "" }}. + This Datasette instance has registered {{ data.actions|length }} action{{ data.actions|length != 1 and "s" or "" }}. Actions are used by the permission system to control access to different features.

@@ -26,7 +26,7 @@ - {% for action in data %} + {% for action in data.actions %} diff --git a/docs/introspection.rst b/docs/introspection.rst index ab47c0a5..0010e8b7 100644 --- a/docs/introspection.rst +++ b/docs/introspection.rst @@ -155,6 +155,30 @@ Shows currently attached databases. `Databases example **Status:** recommendation 2 and 3 are implemented — every JSON-object +> **Status:** recommendations 1-3 are implemented. Every JSON-object > success response now includes `"ok": true` (`JsonDataView` injects it for > dict responses; homepage, jump, schema, permission-debug and autocomplete -> views set it explicitly; covered by `tests/test_success_envelope.py`). -> Recommendation 1 (wrapping the `/-/plugins`, `/-/databases`, `/-/actions` -> top-level arrays in objects) is being landed as separate per-endpoint -> commits. §2a/2b/2c remain open. +> views set it explicitly), and the three top-level-array endpoints now +> return objects: `/-/plugins` → `{"ok": true, "plugins": [...]}`, +> `/-/databases` → `{"ok": true, "databases": [...]}`, `/-/actions` → +> `{"ok": true, "actions": [...]}`. Covered by +> `tests/test_success_envelope.py`. The sub-findings §2a (collection +> representations), §2b (`_extra`/`_shape` coverage) and §2c (count +> truncation) remain open. Endpoints disagree about the success envelope: @@ -372,8 +375,8 @@ Two details make tiering urgent rather than optional: 2. `Forbidden` → JSON 403 for JSON requests (§1a). 3. No `ok: false` with HTTP 200 (§1b: `_shape=object`, write canned-query SQL errors). -4. Wrap `/-/plugins`, `/-/databases`, `/-/actions` top-level arrays in - objects (§2). +4. ~~Wrap `/-/plugins`, `/-/databases`, `/-/actions` top-level arrays in + objects (§2).~~ ✅ Done. 5. Filter `/-/databases.json` by `view-database` or gate it behind `permissions-debug` (§6). 6. 401 (not silent-anonymous) for invalid/expired bearer tokens (§1c). diff --git a/tests/test_api.py b/tests/test_api.py index 3263a88c..b035edb9 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -579,7 +579,7 @@ async def test_actions_json(ds_client): try: ds_client.ds.root_enabled = True response = await ds_client.get("/-/actions.json", actor={"id": "root"}) - data = response.json() + data = response.json()["actions"] finally: ds_client.ds.root_enabled = original_root_enabled assert isinstance(data, list) diff --git a/tests/test_success_envelope.py b/tests/test_success_envelope.py index 290a51e0..22ce6580 100644 --- a/tests/test_success_envelope.py +++ b/tests/test_success_envelope.py @@ -101,3 +101,14 @@ async def test_databases_json_is_object(ds_client): assert data["ok"] is True assert isinstance(data["databases"], list) assert "fixtures" in {db["name"] for db in data["databases"]} + + +@pytest.mark.asyncio +async def test_actions_json_is_object(ds_envelope): + response = await ds_envelope.client.get("/-/actions.json", actor={"id": "root"}) + assert response.status_code == 200 + data = response.json() + assert set(data.keys()) == {"ok", "actions"} + assert data["ok"] is True + assert isinstance(data["actions"], list) + assert "view-instance" in {action["name"] for action in data["actions"]} From f091b6dab165ec9f3a1d7689604abe97e102e60f Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 14:00:30 +0000 Subject: [PATCH 008/131] Filter /-/databases by view-database permission /-/databases previously listed every attached database (including filesystem paths and sizes) to any actor with view-instance, while the homepage and every other endpoint filtered by view-database. The endpoint now only lists databases the current actor is allowed to view. JsonDataView data callbacks may now be async. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/app.py | 15 ++++++++++++++- datasette/views/special.py | 4 ++-- docs/introspection.rst | 2 +- existing-api.md | 3 +-- stable-api-recommendations.md | 9 +++++---- tests/test_permissions.py | 32 ++++++++++++++++++++++++++++++++ 6 files changed, 55 insertions(+), 10 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 5afca2a2..dd5d7e8c 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -2168,6 +2168,18 @@ class Datasette: for name, d in self.databases.items() ] + async def _connected_databases_for_actor(self, actor): + page = await self.allowed_resources("view-database", actor) + allowed_names = {resource.parent async for resource in page.all()} + return [ + database + for database in self._connected_databases() + if database["name"] in allowed_names + ] + + async def _databases_data(self, request): + return {"databases": await self._connected_databases_for_actor(request.actor)} + def _versions(self): conn = sqlite3.connect(":memory:") self._prepare_connection(conn, "_memory") @@ -2574,7 +2586,8 @@ class Datasette: JsonDataView.as_view( self, "databases.json", - lambda: {"databases": self._connected_databases()}, + self._databases_data, + needs_request=True, ), r"/-/databases(\.(?Pjson))?$", ) diff --git a/datasette/views/special.py b/datasette/views/special.py index c289a240..82a76e76 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -53,9 +53,9 @@ class JsonDataView(BaseView): if self.permission: await self.ds.ensure_permission(action=self.permission, actor=request.actor) if self.needs_request: - data = self.data_callback(request) + data = await await_me_maybe(self.data_callback(request)) else: - data = self.data_callback() + data = await await_me_maybe(self.data_callback()) # Return JSON or HTML depending on format parameter as_format = request.url_vars.get("format") diff --git a/docs/introspection.rst b/docs/introspection.rst index 0010e8b7..37f258d7 100644 --- a/docs/introspection.rst +++ b/docs/introspection.rst @@ -137,7 +137,7 @@ Any keys that include the one of the following substrings in their names will be /-/databases ------------ -Shows currently attached databases. `Databases example `_: +Shows currently attached databases that the current actor is allowed to view, based on the ``view-database`` permission. `Databases example `_: .. code-block:: json diff --git a/existing-api.md b/existing-api.md index 5168819a..722331cd 100644 --- a/existing-api.md +++ b/existing-api.md @@ -262,8 +262,7 @@ Permission `view-instance`. No parameters. Response: `{"ok": true, "databases": [...]}` — each database is `{"name", "route", "path", "size", "is_mutable", "is_memory", "hash"}`. -**All attached databases are listed regardless of per-database -`view-database` permissions**. +Only databases the actor is allowed to `view-database` are listed. ### GET /-/actor(.json) diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index bee4b414..a8690f51 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -260,12 +260,13 @@ Concerns: ## 6. Permissions and security consistency (P1/P2) -- **(P1) `/-/databases.json` ignores per-database permissions** — it lists +- ~~**(P1) `/-/databases.json` ignores per-database permissions** — it lists every attached database (name, path on disk, size) to any actor holding `view-instance` (app.py:2157-2169), while the homepage and every other endpoint filter by `view-database`. On a public instance with private databases this leaks filesystem paths and database names. Filter it, or - gate it behind `permissions-debug`. + gate it behind `permissions-debug`.~~ ✅ **Done** — the endpoint now + filters through `allowed_resources("view-database", actor)`. - **(P2) `/db/-/schema` checks existence before permission** (views/special.py:1308-1317): an actor without `view-database` can distinguish "database exists" (403) from "does not exist" (404). @@ -377,8 +378,8 @@ Two details make tiering urgent rather than optional: SQL errors). 4. ~~Wrap `/-/plugins`, `/-/databases`, `/-/actions` top-level arrays in objects (§2).~~ ✅ Done. -5. Filter `/-/databases.json` by `view-database` or gate it behind - `permissions-debug` (§6). +5. ~~Filter `/-/databases.json` by `view-database` or gate it behind + `permissions-debug` (§6).~~ ✅ Done. 6. 401 (not silent-anonymous) for invalid/expired bearer tokens (§1c). 7. Publish explicit stability tiers, including extras and pagination-token opacity (§9). diff --git a/tests/test_permissions.py b/tests/test_permissions.py index 7d99213b..32606789 100644 --- a/tests/test_permissions.py +++ b/tests/test_permissions.py @@ -1800,3 +1800,35 @@ async def test_root_allow_block_with_table_restricted_actor(): actor=admin_actor, ) assert result is True + + +@pytest.mark.asyncio +async def test_databases_json_respects_view_database(tmp_path_factory): + # https://github.com/simonw/datasette - /-/databases should not list + # databases the actor is not allowed to view + db_directory = tmp_path_factory.mktemp("dbs") + from datasette.utils import sqlite3 as _sqlite3 + + paths = [] + for name in ("public", "private"): + path = str(db_directory / "{}.db".format(name)) + conn = _sqlite3.connect(path) + conn.execute("vacuum") + conn.close() + paths.append(path) + ds = Datasette( + paths, + config={"databases": {"private": {"allow": {"id": "root"}}}}, + ) + ds.root_enabled = True + await ds.invoke_startup() + try: + anon_response = await ds.client.get("/-/databases.json") + assert anon_response.status_code == 200 + anon_names = {db["name"] for db in anon_response.json()["databases"]} + assert anon_names == {"public"} + root_response = await ds.client.get("/-/databases.json", actor={"id": "root"}) + root_names = {db["name"] for db in root_response.json()["databases"]} + assert root_names == {"public", "private"} + finally: + ds.close() From ae10a99811111ff81413cc8da1210b6e824d54ff Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 14:09:01 +0000 Subject: [PATCH 009/131] Return canonical JSON error for Forbidden on JSON requests The default forbidden() hook previously rendered an HTML error page even for .json requests. It now returns the canonical JSON error shape with status 403 when the request path ends in .json or the request sends an Accept: application/json or Content-Type: application/json header. HTML requests still get the error page. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/forbidden.py | 10 +++++++ docs/json_api.rst | 5 ++++ docs/plugin_hooks.rst | 2 ++ existing-api.md | 14 +++++----- stable-api-recommendations.md | 14 ++++++---- tests/test_cli.py | 9 ++++--- tests/test_error_shape.py | 49 +++++++++++++++++++++++++++++++++++ 7 files changed, 88 insertions(+), 15 deletions(-) diff --git a/datasette/forbidden.py b/datasette/forbidden.py index 41c48396..3a81ac4f 100644 --- a/datasette/forbidden.py +++ b/datasette/forbidden.py @@ -1,9 +1,19 @@ from datasette import hookimpl, Response +from .utils import add_cors_headers, error_body @hookimpl(trylast=True) def forbidden(datasette, request, message): async def inner(): + if ( + request.path.split("?")[0].endswith(".json") + or "application/json" in (request.headers.get("accept") or "") + or request.headers.get("content-type") == "application/json" + ): + headers = {} + if datasette.cors: + add_cors_headers(headers) + return Response.json(error_body(message, 403), status=403, headers=headers) return Response.html( await datasette.render_template( "error.html", diff --git a/docs/json_api.rst b/docs/json_api.rst index 27d2d705..a561aa9c 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -81,6 +81,11 @@ Some endpoints add extra context keys. For example, a SQL error from a ``"rows"`` and ``"truncated"`` keys of the response it was unable to produce. +Permission errors use the same format: a request that fails a permission +check receives a ``403`` with this JSON error body when the URL ends in +``.json`` or the request sends an ``Accept: application/json`` or +``Content-Type: application/json`` header. + .. _json_api_custom_sql: Executing custom SQL diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index 81ef4acd..8614a15c 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -1685,6 +1685,8 @@ forbidden(datasette, request, message) Plugins can use this to customize how Datasette responds when a 403 Forbidden error occurs - usually because a page failed a permission check, see :ref:`authentication_permissions`. +Datasette's default behavior returns the :ref:`standard JSON error format ` with a 403 status when the request path ends in ``.json`` or the request has an ``Accept: application/json`` or ``Content-Type: application/json`` header; other requests get an HTML error page. + If a plugin hook wishes to react to the error, it should return a :ref:`Response object `. This example returns a redirect to a ``/-/login`` page: diff --git a/existing-api.md b/existing-api.md index 722331cd..90a931d1 100644 --- a/existing-api.md +++ b/existing-api.md @@ -98,13 +98,13 @@ Method-not-allowed responses return HTTP 405 with the canonical shape when the path ends in `.json` or the request content type is `application/json`; plain text otherwise (views/base.py). -**`Forbidden` is special:** when a view raises `Forbidden` (e.g. via -`ensure_permission`), the default `forbidden()` plugin hook renders an **HTML -error page with status 403 even for `.json` requests** -(forbidden.py:4-19, app.py:2895-2904). Endpoints that check permissions -themselves and return `_error(..., 403)` produce JSON instead. So a JSON -client may receive either an HTML 403 page or a JSON 403 body depending on -the endpoint. +**`Forbidden` handling:** when a view raises `Forbidden` (e.g. via +`ensure_permission`), the default `forbidden()` plugin hook returns the +canonical JSON error with status 403 when the path ends in `.json` or the +request carries an `Accept: application/json` / `Content-Type: +application/json` header; other requests get an HTML error page +(forbidden.py). Endpoints that check permissions themselves return +`_error(..., 403)` JSON directly. ### CORS diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index a8690f51..d68bd32e 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -26,9 +26,9 @@ Findings are grouped by theme. Each carries a priority: > The `title` key is no longer emitted in JSON; the bare `{"error": ...}` > debug-endpoint shape is gone; `_shape=object` misuse now returns HTTP 400 > (part of §1b). Covered by `tests/test_error_shape.py` and documented in -> the "Error responses" section of `docs/json_api.rst`. Still open from -> this section's sub-items: §1a (`Forbidden` → HTML), the write -> canned-query 200 (§1b), and the §1c status outliers. +> the "Error responses" section of `docs/json_api.rst`. §1a (`Forbidden` → +> JSON) is now also implemented. Still open from this section's sub-items: +> the write canned-query 200 (§1b) and the §1c status outliers. The API currently produces four distinct JSON error shapes depending on which internal layer generates the error: @@ -58,7 +58,11 @@ defaults). At minimum, eliminate the bare `{"error": ...}` shape and the `status`/`title` keys nobody else emits (`title` is a template-rendering concern that leaked into the API). -### 1a. `Forbidden` returns an HTML 403 to JSON clients (P1) +### 1a. `Forbidden` returns an HTML 403 to JSON clients (P1) — ✅ IMPLEMENTED + +> **Status:** implemented — the default `forbidden()` hook now returns the +> canonical JSON error for requests whose path ends in `.json` or that send +> `Accept: application/json` / `Content-Type: application/json`. Read endpoints that deny access via `ensure_permission`/`check_visibility` raise `Forbidden`, and the default `forbidden()` hook renders an **HTML error @@ -373,7 +377,7 @@ Two details make tiering urgent rather than optional: ## 10. Summary of P1 items (the pre-1.0 checklist) 1. ~~One canonical JSON error shape; retire the other three (§1).~~ ✅ Done. -2. `Forbidden` → JSON 403 for JSON requests (§1a). +2. ~~`Forbidden` → JSON 403 for JSON requests (§1a).~~ ✅ Done. 3. No `ok: false` with HTTP 200 (§1b: `_shape=object`, write canned-query SQL errors). 4. ~~Wrap `/-/plugins`, `/-/databases`, `/-/actions` top-level arrays in diff --git a/tests/test_cli.py b/tests/test_cli.py index 7b528a8e..cbd8edad 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -385,7 +385,9 @@ def test_setting_boolean_validation_false_values(value): ) # Should be forbidden (setting is false) assert result.exit_code == 1, result.output - assert "Forbidden" in result.output + error = json.loads(result.output) + assert error["ok"] is False + assert error["status"] == 403 @pytest.mark.parametrize("value", ("on", "true", "1")) @@ -425,8 +427,9 @@ def test_setting_default_allow_sql(default_allow_sql): assert json.loads(result.output)["rows"][0] == {"21": 21} else: assert result.exit_code == 1, result.output - # This isn't JSON at the moment, maybe it should be though - assert "Forbidden" in result.output + error = json.loads(result.output) + assert error["ok"] is False + assert error["status"] == 403 def test_sql_errors_logged_to_stderr(): diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index 6747e47a..7f12abc6 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -183,3 +183,52 @@ async def test_method_not_allowed_error_shape(ds_client): async def test_schema_unknown_database_error_shape(ds_client): response = await ds_client.get("/no_such_db/-/schema.json") assert_canonical_error(response, 404) + + +# Forbidden responses (the default forbidden() hook) + + +@pytest.fixture +def ds_forbidden(tmp_path_factory): + db_directory = tmp_path_factory.mktemp("dbs") + db_path = str(db_directory / "data.db") + conn = sqlite3.connect(db_path) + conn.execute("vacuum") + conn.execute("create table docs (id integer primary key, title text)") + conn.close() + ds = Datasette( + [db_path], + config={"databases": {"data": {"tables": {"docs": {"allow": {"id": "root"}}}}}}, + ) + ds.root_enabled = True + yield ds + ds.close() + + +@pytest.mark.asyncio +async def test_forbidden_json_path_returns_canonical_json(ds_forbidden): + response = await ds_forbidden.client.get("/data/docs.json") + data = assert_canonical_error(response, 403) + assert "permission" in data["error"].lower() + + +@pytest.mark.asyncio +async def test_forbidden_accept_json_returns_canonical_json(ds_forbidden): + response = await ds_forbidden.client.get( + "/data/docs", headers={"Accept": "application/json"} + ) + assert_canonical_error(response, 403) + + +@pytest.mark.asyncio +async def test_forbidden_html_path_still_returns_html(ds_forbidden): + response = await ds_forbidden.client.get("/data/docs") + assert response.status_code == 403 + assert response.headers["content-type"].startswith("text/html") + + +@pytest.mark.asyncio +async def test_forbidden_json_path_allowed_actor_still_works(ds_forbidden): + response = await ds_forbidden.client.get("/data/docs.json", actor={"id": "root"}) + assert response.status_code == 200 + assert response.json()["ok"] is True From e8048e023fa22971647ae38bfdf24b21b1853ffd Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 14:35:54 +0000 Subject: [PATCH 010/131] Return 400 for write canned-query SQL failures POST to a write canned query previously returned HTTP 200 with {"ok": false, "message": ...} when the SQL failed to execute, so JSON clients (and anything that trusts HTTP status) recorded success for failed writes. SQL failures now return 400 with the canonical error shape plus the "redirect" context key from on_error_redirect; the QueryWriteRejected 403 branch uses the canonical shape too. Successful executions and the HTML flash-message flow are unchanged. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/views/database.py | 22 +++--- docs/sql_queries.rst | 18 ++++- existing-api.md | 14 ++-- stable-api-recommendations.md | 15 ++-- tests/test_error_shape.py | 131 ++++++++++++++++++++++++++++++++++ tests/test_queries.py | 4 +- 6 files changed, 179 insertions(+), 25 deletions(-) diff --git a/datasette/views/database.py b/datasette/views/database.py index e02de657..cf7a6db3 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -16,6 +16,7 @@ from datasette.write_sql import QueryWriteRejected from datasette.utils import ( add_cors_headers, await_me_maybe, + error_body, call_with_supported_arguments, named_parameters as derive_named_parameters, format_bytes, @@ -607,11 +608,7 @@ class QueryView(View): "_json" ): return Response.json( - { - "ok": False, - "message": ex.message, - "redirect": None, - }, + dict(error_body([ex.message], 403), redirect=None), status=403, ) datasette.add_message(request, ex.message, datasette.ERROR) @@ -681,12 +678,17 @@ class QueryView(View): redirect_url = stored_query.on_error_redirect ok = False if should_return_json: + if ok: + return Response.json( + { + "ok": True, + "message": message, + "redirect": redirect_url, + } + ) return Response.json( - { - "ok": ok, - "message": message, - "redirect": redirect_url, - } + dict(error_body([message], 400), redirect=redirect_url), + status=400, ) else: datasette.add_message(request, message, message_type) diff --git a/docs/sql_queries.rst b/docs/sql_queries.rst index 371348fb..4c6e4426 100644 --- a/docs/sql_queries.rst +++ b/docs/sql_queries.rst @@ -657,7 +657,7 @@ There are three options for specifying that you would like the response to your - Include ``?_json=1`` in the URL that you POST to - Include ``"_json": 1`` in your JSON body, or ``&_json=1`` in your form encoded body -The JSON response will look like this: +A successful JSON response will look like this: .. code-block:: json @@ -667,7 +667,21 @@ The JSON response will look like this: "redirect": "/data/add_name" } -The ``"message"`` and ``"redirect"`` values here will take into account ``on_success_message``, ``on_success_message_sql``, ``on_success_redirect``, ``on_error_message`` and ``on_error_redirect``, if they have been set. +If the SQL fails to execute - for example a constraint violation - the response uses the :ref:`standard error format ` with a ``400`` status, plus the ``"redirect"`` key from the query configuration: + +.. code-block:: json + + { + "ok": false, + "error": "UNIQUE constraint failed: docs.id", + "errors": [ + "UNIQUE constraint failed: docs.id" + ], + "status": 400, + "redirect": null + } + +The ``"message"``, ``"error"`` and ``"redirect"`` values here take into account ``on_success_message``, ``on_success_message_sql``, ``on_success_redirect``, ``on_error_message`` and ``on_error_redirect``, if they have been set. .. _pagination: diff --git a/existing-api.md b/existing-api.md index 90a931d1..e0842804 100644 --- a/existing-api.md +++ b/existing-api.md @@ -1132,12 +1132,14 @@ queries. `_random_chars_`, `_cookie_`, `_header_` (underscores → hyphens). User-stored queries cannot contain magic parameters — they are a feature of config/trusted queries. -- **Response — 200 for both success and SQL failure** (only permission - rejection is 403): - `{"ok": true|false, "message": "...", "redirect": "..."|null}` — - `message` honors `on_success_message_sql` / `on_success_message` / - `on_error_message`, falling back to `"Query executed"` or - `"Query executed, N rows affected"`. +- **Response:** success → 200 + `{"ok": true, "message": "...", "redirect": "..."|null}` — `message` + honors `on_success_message_sql` / `on_success_message`, falling back to + `"Query executed"` or `"Query executed, N rows affected"`. SQL failure → + **400** canonical error (message honors `on_error_message`) plus a + `redirect` context key from `on_error_redirect`. Operation rejection + (`QueryWriteRejected`, e.g. VACUUM) → 403 canonical error plus + `redirect: null`. --- diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index d68bd32e..c878cb70 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -27,8 +27,8 @@ Findings are grouped by theme. Each carries a priority: > debug-endpoint shape is gone; `_shape=object` misuse now returns HTTP 400 > (part of §1b). Covered by `tests/test_error_shape.py` and documented in > the "Error responses" section of `docs/json_api.rst`. §1a (`Forbidden` → -> JSON) is now also implemented. Still open from this section's sub-items: -> the write canned-query 200 (§1b) and the §1c status outliers. +> JSON) and §1b (write canned-query 200) are now also implemented. Still +> open from this section's sub-items: the §1c status outliers. The API currently produces four distinct JSON error shapes depending on which internal layer generates the error: @@ -76,7 +76,12 @@ machine-readable answer. **Recommendation:** the default forbidden handler must return the canonical JSON error when the path ends in `.json` or the request prefers JSON, mirroring `handle_exception`. -### 1b. Errors that return HTTP 200 (P1) +### 1b. Errors that return HTTP 200 (P1) — ✅ IMPLEMENTED + +> **Status:** implemented. `_shape=object` misuse returns 400 (done with +> §1), and write canned-query SQL failures now return **400** with the +> canonical error shape (plus the `redirect` context key); the +> `QueryWriteRejected` 403 branch also uses the canonical shape. - `_shape=object` on a query or pk-less table → `{"ok": false, "error": "_shape=object is only available on tables"}` with **200** @@ -378,8 +383,8 @@ Two details make tiering urgent rather than optional: 1. ~~One canonical JSON error shape; retire the other three (§1).~~ ✅ Done. 2. ~~`Forbidden` → JSON 403 for JSON requests (§1a).~~ ✅ Done. -3. No `ok: false` with HTTP 200 (§1b: `_shape=object`, write canned-query - SQL errors). +3. ~~No `ok: false` with HTTP 200 (§1b: `_shape=object`, write canned-query + SQL errors).~~ ✅ Done. 4. ~~Wrap `/-/plugins`, `/-/databases`, `/-/actions` top-level arrays in objects (§2).~~ ✅ Done. 5. ~~Filter `/-/databases.json` by `view-database` or gate it behind diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index 7f12abc6..dd1fcfc9 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -232,3 +232,134 @@ async def test_forbidden_json_path_allowed_actor_still_works(ds_forbidden): response = await ds_forbidden.client.get("/data/docs.json", actor={"id": "root"}) assert response.status_code == 200 assert response.json()["ok"] is True + + +# Write canned queries: SQL failures must not return HTTP 200 + + +@pytest.fixture +def ds_write_query(tmp_path_factory): + db_directory = tmp_path_factory.mktemp("dbs") + db_path = str(db_directory / "data.db") + conn = sqlite3.connect(db_path) + conn.execute("vacuum") + conn.execute("create table docs (id integer primary key, title text)") + conn.close() + ds = Datasette( + [db_path], + config={ + "databases": { + "data": { + "queries": { + "add_doc": { + "sql": ( + "insert into docs (id, title)" " values (:id, :title)" + ), + "write": True, + }, + "add_doc_custom_error": { + "sql": ( + "insert into docs (id, title)" " values (:id, :title)" + ), + "write": True, + "on_error_message": "Custom error message", + "on_error_redirect": "/data", + }, + } + } + } + }, + ) + yield ds + ds.close() + + +@pytest.mark.asyncio +async def test_write_query_success_returns_200(ds_write_query): + response = await ds_write_query.client.post( + "/data/add_doc", + json={"id": 1, "title": "One"}, + headers={"Accept": "application/json"}, + ) + assert response.status_code == 200 + data = response.json() + assert data["ok"] is True + assert data["message"] == "Query executed, 1 row affected" + assert data["redirect"] is None + + +@pytest.mark.asyncio +async def test_write_query_sql_failure_returns_400(ds_write_query): + for _ in range(2): + response = await ds_write_query.client.post( + "/data/add_doc", + json={"id": 1, "title": "One"}, + headers={"Accept": "application/json"}, + ) + data = assert_canonical_error(response, 400) + assert "UNIQUE constraint failed" in data["error"] + # The redirect context key from the canned query flow is preserved + assert data["redirect"] is None + + +@pytest.mark.asyncio +async def test_write_query_failure_uses_on_error_message_and_redirect( + ds_write_query, +): + for _ in range(2): + response = await ds_write_query.client.post( + "/data/add_doc_custom_error", + json={"id": 1, "title": "One"}, + headers={"Accept": "application/json"}, + ) + data = assert_canonical_error(response, 400) + assert data["error"] == "Custom error message" + assert data["redirect"] == "/data" + + +@pytest.mark.asyncio +async def test_write_query_forbidden_is_canonical_403(ds_write_query): + # An untrusted write query run by an actor without execute-write-sql + # raises Forbidden, handled by the forbidden() hook + await ds_write_query.invoke_startup() + await ds_write_query.add_query( + "data", + name="untrusted_add", + sql="insert into docs (id, title) values (:id, :title)", + is_write=True, + is_trusted=False, + source="user", + owner_id="someone", + ) + response = await ds_write_query.client.post( + "/data/untrusted_add", + json={"id": 5, "title": "Five"}, + headers={"Accept": "application/json"}, + actor={"id": "someone"}, + ) + assert_canonical_error(response, 403) + + +@pytest.mark.asyncio +async def test_write_query_rejected_operation_is_canonical_403(ds_write_query): + # A rejected operation (VACUUM) raises QueryWriteRejected, handled by + # the dedicated branch in QueryView.post - root has execute-write-sql + ds_write_query.root_enabled = True + await ds_write_query.invoke_startup() + await ds_write_query.add_query( + "data", + name="vacuum_it", + sql="vacuum", + is_write=True, + is_trusted=False, + source="user", + owner_id="root", + ) + response = await ds_write_query.client.post( + "/data/vacuum_it", + json={}, + headers={"Accept": "application/json"}, + actor={"id": "root"}, + ) + data = assert_canonical_error(response, 403) + assert data["redirect"] is None diff --git a/tests/test_queries.py b/tests/test_queries.py index 6dfcc8b7..b79a9af4 100644 --- a/tests/test_queries.py +++ b/tests/test_queries.py @@ -3093,9 +3093,9 @@ async def test_untrusted_stored_write_query_rejects_virtual_table_control_insert ) assert denied_response.status_code == 403 - assert denied_response.json()["message"] == ( + assert denied_response.json()["errors"] == [ "Writes to virtual tables are not allowed in user-supplied SQL" - ) + ] assert ( await db.execute("select count(*) from docs where docs match 'hello'") ).first()[0] == 1 From b2cdc81d3495a497adf8a911d1171cf4609c9da4 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 14:55:41 +0000 Subject: [PATCH 011/131] Return 400 instead of 500 for row delete write failures Row delete previously returned 500 when the write failed (for example a constraint violation raised by a trigger or foreign key), while row update and every other write endpoint report the same failure class as 400. Delete now matches. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/views/row.py | 2 +- existing-api.md | 4 ++-- stable-api-recommendations.md | 8 +++++--- tests/test_error_shape.py | 32 ++++++++++++++++++++++++++++++++ 4 files changed, 40 insertions(+), 6 deletions(-) diff --git a/datasette/views/row.py b/datasette/views/row.py index c99b75d1..bdd78ed4 100644 --- a/datasette/views/row.py +++ b/datasette/views/row.py @@ -742,7 +742,7 @@ class RowDeleteView(BaseView): try: await resolved.db.execute_write_fn(delete_row, request=request) except Exception as e: - return _error([str(e)], 500) + return _error([str(e)], 400) await self.ds.track_event( DeleteRowEvent( diff --git a/existing-api.md b/existing-api.md index e0842804..d600179e 100644 --- a/existing-api.md +++ b/existing-api.md @@ -968,8 +968,8 @@ does not change the SQLite schema. - **Request:** no body required (any body is ignored — there is no confirmation step, unlike table drop). - **Response** — 200 `{"ok": true}`; with `?_redirect_to_table` a `redirect` - key is added. A failure during the write returns **500** with the message - (unlike update's 400). Emits `delete-row`. + key is added. A failure during the write returns 400 with the message, + matching update. Emits `delete-row`. --- diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index c878cb70..4fec5a00 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -98,10 +98,11 @@ key — see §2.) ### 1c. Wrong-status outliers (P2) -- Row **delete** write failures return **500** (views/row.py:757) while row +- ~~Row **delete** write failures return **500** (views/row.py:757) while row **update** write failures return **400** (views/row.py:832-835). Same failure class, different status; pick 400 (or 409 for constraint - violations) for both. + violations) for both.~~ ✅ **Done** — delete now returns 400, matching + update and the rest of the write API. - Invalid or expired bearer tokens silently degrade the request to anonymous, so clients see a 403 permission error (or worse, anonymous-permitted data) rather than a 401 (tokens.py:147-193). For 1.0, a malformed/expired @@ -337,7 +338,8 @@ Concerns: []}`** while `.csv` on the same request returns 400 `"?sql= is required"`. The JSON behavior masks caller bugs; return 400 on both. 3. **`_shape=object` HTTP 200 error** (§1b) — almost certainly unintended. -4. **Row delete 500** (§1c) — inconsistent with every sibling endpoint. +4. ~~**Row delete 500** (§1c) — inconsistent with every sibling endpoint.~~ + ✅ Done — now 400. 5. **The "SQL Interrupted" error embeds an HTML fragment in the JSON `error` value** (views/database.py:805-820). Error strings in the JSON API should be plain text. diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index dd1fcfc9..21d37cc2 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -363,3 +363,35 @@ async def test_write_query_rejected_operation_is_canonical_403(ds_write_query): ) data = assert_canonical_error(response, 403) assert data["redirect"] is None + + +# Row delete write failures must be 400, matching row update + + +@pytest.mark.asyncio +async def test_row_delete_write_failure_is_400(tmp_path_factory): + db_directory = tmp_path_factory.mktemp("dbs") + db_path = str(db_directory / "data.db") + conn = sqlite3.connect(db_path) + conn.execute("vacuum") + conn.execute("create table docs (id integer primary key, title text)") + conn.execute("insert into docs (id, title) values (1, 'One')") + conn.execute( + "create trigger no_delete before delete on docs " + "begin select raise(abort, 'deletes are blocked'); end" + ) + conn.commit() + conn.close() + ds = Datasette([db_path]) + ds.root_enabled = True + try: + response = await ds.client.post( + "/data/docs/1/-/delete", + json={}, + headers={"Content-Type": "application/json"}, + actor={"id": "root"}, + ) + data = assert_canonical_error(response, 400) + assert "deletes are blocked" in data["error"] + finally: + ds.close() From aaaffe45b851c974fde9ce894191192718f601e5 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 15:18:12 +0000 Subject: [PATCH 012/131] Return 401 for invalid or expired bearer tokens Invalid dstok_ tokens - bad signature, malformed payload, expired, or presented while allow_signed_tokens is off - previously degraded the request to anonymous, so clients saw a 403 permission error or worse, a 200 with anonymous-visible data. Token handlers can now raise TokenInvalid for tokens they recognize but reject; Datasette responds with 401, the canonical JSON error body and a WWW-Authenticate: Bearer error="invalid_token" header, even when a valid cookie is also present. Bearer tokens no registered handler recognizes are still ignored, so authentication plugins with their own token formats keep working. TokenInvalid is exported from the datasette package for use by plugin token handlers. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/__init__.py | 2 +- datasette/app.py | 31 +++++++++++- datasette/tokens.py | 44 +++++++++++++----- docs/authentication.rst | 4 +- docs/plugin_hooks.rst | 4 ++ existing-api.md | 17 ++++--- stable-api-recommendations.md | 13 ++++-- tests/test_api_write.py | 17 +++---- tests/test_auth.py | 19 ++++++-- tests/test_error_shape.py | 88 +++++++++++++++++++++++++++++++++++ tests/test_token_handler.py | 20 +++++--- 11 files changed, 214 insertions(+), 45 deletions(-) diff --git a/datasette/__init__.py b/datasette/__init__.py index eb18e59e..de46861c 100644 --- a/datasette/__init__.py +++ b/datasette/__init__.py @@ -1,7 +1,7 @@ from datasette.permissions import Permission # noqa from datasette.version import __version_info__, __version__ # noqa from datasette.events import Event # noqa -from datasette.tokens import TokenHandler, TokenRestrictions # noqa +from datasette.tokens import TokenHandler, TokenInvalid, TokenRestrictions # noqa from datasette.utils.asgi import Forbidden, NotFound, Request, Response # noqa from datasette.utils import actor_matches_allow # noqa from datasette.views import Context # noqa diff --git a/datasette/app.py b/datasette/app.py index dd5d7e8c..9982c58e 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -111,7 +111,9 @@ from .utils import ( baseconv, call_with_supported_arguments, detect_json1, + add_cors_headers, display_actor, + error_body, escape_css_string, escape_sqlite, find_spatialite, @@ -130,6 +132,7 @@ from .utils import ( redact_keys, row_sql_params_pks, ) +from .tokens import TokenInvalid from .utils.asgi import ( AsgiLifespan, Forbidden, @@ -905,7 +908,9 @@ class Datasette: Verify an API token by trying all registered token handlers. Returns an actor dict from the first handler that recognizes the - token, or None if no handler accepts it. + token, or None if no handler accepts it. A handler may raise + TokenInvalid for a token it recognizes but rejects (bad signature, + expired) - Datasette turns that into a 401 response. """ for token_handler in self._token_handlers(): result = await token_handler.verify_token(self, token) @@ -2887,13 +2892,24 @@ class DatasetteRouter: # Handle authentication default_actor = scope.get("actor") or None actor = None + token_error = None results = pm.hook.actor_from_request(datasette=self.ds, request=request) for result in results: - result = await await_me_maybe(result) + try: + result = await await_me_maybe(result) + except TokenInvalid as ex: + # A presented token was recognized but rejected - fail the + # request with a 401 even if another credential is valid, + # but keep awaiting the remaining coroutines first + if token_error is None: + token_error = ex + continue if result and actor is None: actor = result # Don't break — we must await all coroutines to avoid # "coroutine was never awaited" warnings + if token_error is not None: + return await self.handle_401(request, send, token_error) scope_modifications["actor"] = actor or default_actor scope = dict(scope, **scope_modifications) @@ -2925,6 +2941,17 @@ class DatasetteRouter: except Exception as exception: return await self.handle_exception(request, send, exception) + async def handle_401(self, request, send, exception): + # A presented bearer token was recognized by a handler but rejected. + # Bearer tokens are API credentials, so this is always JSON. + headers = {"www-authenticate": 'Bearer error="invalid_token"'} + if self.ds.cors: + add_cors_headers(headers) + response = Response.json( + error_body([str(exception)], 401), status=401, headers=headers + ) + await response.asgi_send(send) + async def handle_404(self, request, send, exception=None): # If path contains % encoding, redirect to tilde encoding if "%" in request.path: diff --git a/datasette/tokens.py b/datasette/tokens.py index 38a55529..4f905339 100644 --- a/datasette/tokens.py +++ b/datasette/tokens.py @@ -18,6 +18,21 @@ if TYPE_CHECKING: from datasette.app import Datasette +class TokenInvalid(Exception): + """ + Raised by a TokenHandler when a token it recognizes is invalid - + for example a bad signature, malformed payload or expired token. + + Datasette responds to this with an HTTP 401 error. Handlers should + return None instead for tokens they do not recognize at all, so that + other registered handlers get a chance to verify them. + """ + + def __init__(self, message="Invalid token"): + self.message = message + super().__init__(message) + + @dataclasses.dataclass class TokenRestrictions: """ @@ -108,8 +123,12 @@ class TokenHandler: async def verify_token(self, datasette: "Datasette", token: str) -> Optional[dict]: """ - Verify a token and return an actor dict, or None if this handler - does not recognize the token. + Verify a token and return an actor dict. + + Return None if this handler does not recognize the token at all, + so other handlers can try it. Raise TokenInvalid if the token is + recognized but invalid (bad signature, malformed, expired) - the + request will fail with a 401 error. """ raise NotImplementedError @@ -147,29 +166,32 @@ class SignedTokenHandler(TokenHandler): async def verify_token(self, datasette: "Datasette", token: str) -> Optional[dict]: prefix = "dstok_" - if not datasette.setting("allow_signed_tokens"): + if not token.startswith(prefix): + # Not one of our tokens - leave it for other handlers return None + if not datasette.setting("allow_signed_tokens"): + raise TokenInvalid( + "Signed tokens are not enabled for this Datasette instance" + ) + max_signed_tokens_ttl = datasette.setting("max_signed_tokens_ttl") - if not token.startswith(prefix): - return None - raw = token[len(prefix) :] try: decoded = datasette.unsign(raw, namespace="token") except itsdangerous.BadSignature: - return None + raise TokenInvalid("Invalid token signature") if "t" not in decoded: - return None + raise TokenInvalid("Invalid token: no timestamp") created = decoded["t"] if not isinstance(created, int): - return None + raise TokenInvalid("Invalid token: invalid timestamp") duration = decoded.get("d") if duration is not None and not isinstance(duration, int): - return None + raise TokenInvalid("Invalid token: invalid duration") if (duration is None and max_signed_tokens_ttl) or ( duration is not None @@ -180,7 +202,7 @@ class SignedTokenHandler(TokenHandler): if duration: if time.time() - created > duration: - return None + raise TokenInvalid("Token has expired") actor = {"id": decoded["a"], "token": "dstok"} diff --git a/docs/authentication.rst b/docs/authentication.rst index 8101699c..72ac5fa6 100644 --- a/docs/authentication.rst +++ b/docs/authentication.rst @@ -991,7 +991,9 @@ The ``/-/create-token`` page cannot be accessed by actors that are authenticated Datasette plugins that implement their own form of API token authentication should follow this convention. -You can disable the signed token feature entirely using the :ref:`allow_signed_tokens ` setting. +If a request presents a token that a token handler recognizes but rejects - an invalid signature, a malformed payload or an expired token - Datasette responds with a ``401`` status, the :ref:`standard JSON error format ` and a ``WWW-Authenticate: Bearer error="invalid_token"`` header. This means API clients can distinguish "your token needs to be renewed" (``401``) from "your token does not grant this permission" (``403``). A ``Bearer`` token that no registered handler recognizes at all is ignored, since it may be intended for an authentication plugin. + +You can disable the signed token feature entirely using the :ref:`allow_signed_tokens ` setting. Requests presenting a ``dstok_`` token while the feature is disabled receive a ``401``. .. _authentication_cli_create_token: diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index 8614a15c..049cb292 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -2546,6 +2546,10 @@ The default ``SignedTokenHandler`` uses itsdangerous signed tokens (``dstok_`` p async def verify_token(self, datasette, token): # Look up token in database, return actor dict or None + # if this handler does not recognize the token. Raise + # datasette.TokenInvalid for a token this handler + # recognizes but rejects (revoked, expired) - Datasette + # will respond with a 401 error. ... diff --git a/existing-api.md b/existing-api.md index d600179e..8c6e1c4e 100644 --- a/existing-api.md +++ b/existing-api.md @@ -1156,14 +1156,17 @@ registered via `register_token_handler`; the default is - **Format:** `dstok_` + itsdangerous-signed payload (namespace `token`) containing `a` (actor id), `t` (creation Unix time), optional `d` (duration seconds), optional `_r` (restrictions). -- **Verification** returns no actor when: `allow_signed_tokens` is off, the - signature is invalid, `t` is missing/non-integer, or the token is expired. - The effective duration is `d` capped by `max_signed_tokens_ttl` (default 0 - = no cap; a non-zero setting also imposes a TTL on tokens without `d`). +- **Verification:** a `dstok_`-prefixed token that fails verification — + `allow_signed_tokens` off, invalid signature, missing/non-integer `t`, + malformed `d`, or expired — raises `TokenInvalid`, and the request fails + with **401**, the canonical error body and a + `WWW-Authenticate: Bearer error="invalid_token"` header (even if a valid + `ds_actor` cookie is also present). Tokens with prefixes no registered + handler recognizes are ignored (they may belong to an auth plugin). The + effective duration is `d` capped by `max_signed_tokens_ttl` (default 0 = + no cap; a non-zero setting also imposes a TTL on tokens without `d`). - **Resulting actor:** `{"id": , "token": "dstok"}` plus `"_r"` and - `"token_expires"` when applicable. Invalid/expired tokens silently produce - an anonymous request (no 401) — the failure then surfaces as a 403 from - whatever permission check the request hits. + `"token_expires"` when applicable. **Restrictions (`_r`)** (default_permissions/restrictions.py): diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index 4fec5a00..273a272c 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -96,19 +96,23 @@ request prefers JSON, mirroring `handle_exception`. completed, SQL is invalid" is defensible but should then not reuse the `ok` key — see §2.) -### 1c. Wrong-status outliers (P2) +### 1c. Wrong-status outliers (P2) — ✅ IMPLEMENTED - ~~Row **delete** write failures return **500** (views/row.py:757) while row **update** write failures return **400** (views/row.py:832-835). Same failure class, different status; pick 400 (or 409 for constraint violations) for both.~~ ✅ **Done** — delete now returns 400, matching update and the rest of the write API. -- Invalid or expired bearer tokens silently degrade the request to anonymous, +- ~~Invalid or expired bearer tokens silently degrade the request to anonymous, so clients see a 403 permission error (or worse, anonymous-permitted data) rather than a 401 (tokens.py:147-193). For 1.0, a malformed/expired `Authorization: Bearer dstok_...` header should produce **401** with a distinguishable error, so clients can tell "renew your token" apart from - "you lack permission". + "you lack permission".~~ ✅ **Done** — token handlers can raise + `TokenInvalid`; Datasette responds 401 with the canonical body and a + `WWW-Authenticate: Bearer error="invalid_token"` header. Unrecognized + token prefixes still fall through to anonymous so auth plugins keep + working. --- @@ -391,7 +395,8 @@ Two details make tiering urgent rather than optional: objects (§2).~~ ✅ Done. 5. ~~Filter `/-/databases.json` by `view-database` or gate it behind `permissions-debug` (§6).~~ ✅ Done. -6. 401 (not silent-anonymous) for invalid/expired bearer tokens (§1c). +6. ~~401 (not silent-anonymous) for invalid/expired bearer tokens (§1c).~~ + ✅ Done. 7. Publish explicit stability tiers, including extras and pagination-token opacity (§9). 8. Resolve the looks-like-a-bug list (§8), especially trusted-query delete diff --git a/tests/test_api_write.py b/tests/test_api_write.py index 17542d4b..a29f5c99 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -202,8 +202,8 @@ async def test_insert_rows(ds_write, return_rows): "/data/docs/-/insert", {"rows": [{"title": "Test"} for i in range(10)]}, "bad_token", - 403, - ["Permission denied"], + 401, + ["Invalid token signature"], ), ( "/data/docs/-/insert", @@ -410,12 +410,13 @@ async def test_insert_or_upsert_row_errors( }, ) - actor_response = ( - await ds_write.client.get("/-/actor.json", headers=kwargs["headers"]) - ).json() - assert set((actor_response["actor"] or {}).get("_r", {}).get("a") or []) == set( - token_permissions - ) + if special_case != "bad_token": + actor_response = ( + await ds_write.client.get("/-/actor.json", headers=kwargs["headers"]) + ).json() + assert set((actor_response["actor"] or {}).get("_r", {}).get("a") or []) == set( + token_permissions + ) if special_case == "invalid_json": del kwargs["json"] diff --git a/tests/test_auth.py b/tests/test_auth.py index 8e83d397..d2913ecc 100644 --- a/tests/test_auth.py +++ b/tests/test_auth.py @@ -236,7 +236,9 @@ def test_auth_create_token( @pytest.mark.asyncio async def test_auth_create_token_not_allowed_for_tokens(ds_client): - ds_tok = ds_client.ds.sign({"a": "test", "token": "dstok"}, "token") + ds_tok = ds_client.ds.sign( + {"a": "test", "token": "dstok", "t": int(time.time())}, "token" + ) response = await ds_client.get( "/-/create-token", headers={"Authorization": "Bearer dstok_{}".format(ds_tok)}, @@ -304,8 +306,16 @@ async def test_auth_with_dstok_token(ds_client, scenario, should_work): assert actor["token"] == "dstok" if scenario != "valid_unlimited_token": assert isinstance(actor["token_expires"], int) - else: + elif scenario == "no_token": + # No credentials presented - request proceeds as anonymous assert response.json() == {"ok": True, "actor": None} + else: + # Invalid credentials presented - hard 401 + assert response.status_code == 401 + data = response.json() + assert data["ok"] is False + assert data["status"] == 401 + assert response.headers["www-authenticate"].startswith("Bearer") finally: ds_client.ds._settings["allow_signed_tokens"] = True @@ -339,8 +349,9 @@ def test_cli_create_token(app_client, expires): expected_actor["token_expires"] = details["t"] + expires assert response.json == {"ok": True, "actor": expected_actor} else: - expected_actor = None - assert response.json == {"ok": True, "actor": expected_actor} + # Expired token - hard 401 + assert response.status == 401 + assert response.json["ok"] is False @pytest.mark.asyncio diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index 21d37cc2..49b2c314 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -18,6 +18,7 @@ https://github.com/simonw/datasette/issues - 1.0 API consistency """ import pytest +import time from datasette.app import Datasette from datasette.utils import sqlite3 @@ -395,3 +396,90 @@ async def test_row_delete_write_failure_is_400(tmp_path_factory): assert "deletes are blocked" in data["error"] finally: ds.close() + + +# Invalid bearer tokens must produce 401, not silent anonymous access + + +@pytest.mark.asyncio +async def test_expired_token_returns_401(ds_error_shape): + token = "dstok_{}".format( + ds_error_shape.sign( + {"a": "root", "t": int(time.time()) - 2000, "d": 1000}, + namespace="token", + ) + ) + response = await ds_error_shape.client.get( + "/-/actor.json", headers={"Authorization": "Bearer {}".format(token)} + ) + data = assert_canonical_error(response, 401) + assert "expired" in data["error"].lower() + assert response.headers["www-authenticate"].startswith("Bearer") + + +@pytest.mark.asyncio +async def test_bad_signature_token_returns_401(ds_error_shape): + response = await ds_error_shape.client.get( + "/-/actor.json", headers={"Authorization": "Bearer dstok_garbage"} + ) + data = assert_canonical_error(response, 401) + assert response.headers["www-authenticate"].startswith("Bearer") + + +@pytest.mark.asyncio +async def test_unrecognized_token_prefix_stays_anonymous(ds_error_shape): + # No registered handler claims this token - it might belong to a + # plugin's actor_from_request hook, so it must not hard-fail + response = await ds_error_shape.client.get( + "/-/actor.json", headers={"Authorization": "Bearer sometoken_abc"} + ) + assert response.status_code == 200 + assert response.json() == {"ok": True, "actor": None} + + +@pytest.mark.asyncio +async def test_valid_token_still_authenticates(ds_error_shape): + token = "dstok_{}".format( + ds_error_shape.sign( + {"a": "root", "t": int(time.time())}, + namespace="token", + ) + ) + response = await ds_error_shape.client.get( + "/-/actor.json", headers={"Authorization": "Bearer {}".format(token)} + ) + assert response.status_code == 200 + assert response.json()["actor"]["id"] == "root" + + +@pytest.mark.asyncio +async def test_bad_token_beats_valid_cookie(ds_error_shape): + # A malformed Authorization header is a hard error even if a valid + # ds_actor cookie is also present + response = await ds_error_shape.client.get( + "/-/actor.json", + headers={"Authorization": "Bearer dstok_garbage"}, + cookies={"ds_actor": ds_error_shape.client.actor_cookie({"id": "root"})}, + ) + assert_canonical_error(response, 401) + + +@pytest.mark.asyncio +async def test_token_when_signed_tokens_disabled_returns_401(tmp_path_factory): + db_directory = tmp_path_factory.mktemp("dbs") + db_path = str(db_directory / "data.db") + conn = sqlite3.connect(db_path) + conn.execute("vacuum") + conn.close() + ds = Datasette([db_path], settings={"allow_signed_tokens": False}) + try: + token = "dstok_{}".format( + ds.sign({"a": "root", "t": int(time.time())}, namespace="token") + ) + response = await ds.client.get( + "/-/actor.json", headers={"Authorization": "Bearer {}".format(token)} + ) + data = assert_canonical_error(response, 401) + assert "not enabled" in data["error"] + finally: + ds.close() diff --git a/tests/test_token_handler.py b/tests/test_token_handler.py index 5c87f577..f5bbfead 100644 --- a/tests/test_token_handler.py +++ b/tests/test_token_handler.py @@ -5,7 +5,12 @@ Tests for the register_token_handler plugin hook. from datasette.app import Datasette from datasette.hookspecs import hookimpl from datasette.plugins import pm -from datasette.tokens import TokenHandler, TokenRestrictions, SignedTokenHandler +from datasette.tokens import ( + TokenHandler, + TokenInvalid, + TokenRestrictions, + SignedTokenHandler, +) import pytest @@ -66,10 +71,10 @@ async def test_verify_token_unknown_returns_none(datasette): @pytest.mark.asyncio -async def test_verify_token_bad_signature_returns_none(datasette): - """verify_token() should return None for tokens with bad signatures.""" - result = await datasette.verify_token("dstok_tampered_data_here") - assert result is None +async def test_verify_token_bad_signature_raises(datasette): + """verify_token() should raise TokenInvalid for tokens with bad signatures.""" + with pytest.raises(TokenInvalid): + await datasette.verify_token("dstok_tampered_data_here") @pytest.mark.asyncio @@ -334,5 +339,6 @@ async def test_signed_tokens_disabled(): ds = Datasette(settings={"allow_signed_tokens": False}) with pytest.raises(ValueError, match="Signed tokens are not enabled"): await ds.create_token("test_actor", handler="signed") - # verify_token should return None rather than raising - assert await ds.verify_token("dstok_anything") is None + # verify_token should raise TokenInvalid for a dstok_ token + with pytest.raises(TokenInvalid, match="not enabled"): + await ds.verify_token("dstok_anything") From ea9c1b1524279c03f0368c714438c5aacebcbec3 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 15:39:11 +0000 Subject: [PATCH 013/131] Return 400 for query data formats when ?sql= is missing GET /db/-/query.json with no (or blank) ?sql= previously returned 200 with empty rows, masking caller bugs, while the .csv format returned 400 "?sql= is required" for the same request. All data formats now return the 400; the HTML SQL editor page is unchanged. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/views/database.py | 2 ++ existing-api.md | 5 +++-- stable-api-recommendations.md | 6 ++++-- tests/test_error_shape.py | 24 ++++++++++++++++++++++++ tests/test_table_api.py | 6 ++++-- 5 files changed, 37 insertions(+), 6 deletions(-) diff --git a/datasette/views/database.py b/datasette/views/database.py index cf7a6db3..99e85d2b 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -851,6 +851,8 @@ class QueryView(View): return await stream_csv(datasette, fetch_data_for_csv, request, db.name) elif format_ in datasette.renderers.keys(): + if not sql: + raise DatasetteError("?sql= is required", status=400) data = {"ok": True, "rows": rows, "columns": columns} extras = extra_names_from_request(request) if extras: diff --git a/existing-api.md b/existing-api.md index 8c6e1c4e..9947edeb 100644 --- a/existing-api.md +++ b/existing-api.md @@ -486,8 +486,9 @@ queries section). HTTP 400 `{"ok": false, "error": "", "rows": [], "truncated": false}`. - Time limit → 400 titled `"SQL Interrupted"` (the `error` value contains an HTML fragment). - - `?sql=` omitted → 200 `{"ok": true, "rows": [], "truncated": false}` - (the CSV format instead errors 400 `"?sql= is required"`). + - `?sql=` omitted or blank → 400 `"?sql= is required"` for all data + formats (`.json`, `.csv`, plugin formats). The HTML page remains the + SQL editor. - `.csv` streams CSV; unknown extensions → 404. ### GET /\/-/query/parameters diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index 273a272c..11f0a75d 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -338,9 +338,11 @@ Concerns: `is_trusted` — an actor with `delete-query` can delete a config-defined trusted query via the API (it will resync on restart, making the behavior confusing rather than catastrophic). Align delete with update. -2. **GET `/db/-/query` with no `?sql=` returns 200 `{"ok": true, "rows": +2. ~~**GET `/db/-/query` with no `?sql=` returns 200 `{"ok": true, "rows": []}`** while `.csv` on the same request returns 400 `"?sql= is - required"`. The JSON behavior masks caller bugs; return 400 on both. + required"`. The JSON behavior masks caller bugs; return 400 on both.~~ + ✅ **Done** — all data formats now return 400; the HTML SQL editor page + is unchanged. 3. **`_shape=object` HTTP 200 error** (§1b) — almost certainly unintended. 4. ~~**Row delete 500** (§1c) — inconsistent with every sibling endpoint.~~ ✅ Done — now 400. diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index 49b2c314..9d4a566c 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -483,3 +483,27 @@ async def test_token_when_signed_tokens_disabled_returns_401(tmp_path_factory): assert "not enabled" in data["error"] finally: ds.close() + + +# GET /db/-/query without SQL: 400 for data formats, HTML editor stays 200 + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "path", + ( + "/fixtures/-/query.json", + "/fixtures/-/query.json?sql=", + ), +) +async def test_query_json_without_sql_is_400(ds_client, path): + response = await ds_client.get(path) + data = assert_canonical_error(response, 400) + assert data["errors"] == ["?sql= is required"] + + +@pytest.mark.asyncio +async def test_query_html_without_sql_is_still_the_editor(ds_client): + response = await ds_client.get("/fixtures/-/query") + assert response.status_code == 200 + assert response.headers["content-type"].startswith("text/html") diff --git a/tests/test_table_api.py b/tests/test_table_api.py index c8ba31b7..41c89f39 100644 --- a/tests/test_table_api.py +++ b/tests/test_table_api.py @@ -180,9 +180,11 @@ def test_query_extra_query_reports_bound_params(): assert response.json["query"]["params"] == {} -def test_query_extra_query_does_not_echo_querystring_without_sql(): +def test_query_extra_query_does_not_echo_querystring(): with make_app_client() as client: - response = client.get("/fixtures/-/query.json?_extra=query&foo=bar") + response = client.get( + "/fixtures/-/query.json?sql=select+1&_extra=query&foo=bar" + ) assert response.status == 200 assert response.json["query"]["params"] == {} From f3f5e891c9f3ad644a0b78c592ddd9d4f511d4b9 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 15:49:49 +0000 Subject: [PATCH 014/131] Block API deletion of trusted stored queries QueryUpdateView already rejected is_trusted queries but QueryDeleteView did not, so an actor with delete-query could delete a config-defined trusted query - which would then silently reappear on restart when the config re-syncs. Both the POST endpoint and the HTML confirmation page now return 403, matching update. datasette.remove_query() is unchanged for internal use. The docs already claimed this behavior ("Trusted stored queries cannot be edited or deleted through the web interface or the JSON API") - the code now matches them. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/views/stored_queries.py | 4 +++ existing-api.md | 5 ++-- stable-api-recommendations.md | 11 ++++--- tests/test_queries.py | 48 +++++++++++++++++++++++++++++++ 4 files changed, 62 insertions(+), 6 deletions(-) diff --git a/datasette/views/stored_queries.py b/datasette/views/stored_queries.py index 2753f876..d1f151dd 100644 --- a/datasette/views/stored_queries.py +++ b/datasette/views/stored_queries.py @@ -610,6 +610,8 @@ class QueryDeleteView(BaseView): resource=QueryResource(db.name, query_name), actor=request.actor, ) + if existing.is_trusted: + return _error(["Trusted queries cannot be deleted using the API"], 403) return await self.render( ["query_delete.html"], request, @@ -631,6 +633,8 @@ class QueryDeleteView(BaseView): actor=request.actor, ): return _error(["Permission denied: need delete-query"], 403) + if existing.is_trusted: + return _error(["Trusted queries cannot be deleted using the API"], 403) data, is_json = await _json_or_form_payload(request) await self.ds.remove_query(db.name, query_name) diff --git a/existing-api.md b/existing-api.md index 9947edeb..e11491fe 100644 --- a/existing-api.md +++ b/existing-api.md @@ -1093,8 +1093,9 @@ updates use `/-/update`. `QueryDeleteView` (app.py:2707-2710; views/stored_queries.py:594-644). GET renders an HTML confirmation page. -- **Permission:** `delete-query` (403 `need delete-query`). Unlike update, - **trusted queries are not blocked** from API deletion. +- **Permission:** `delete-query` (403 `need delete-query`). Trusted + queries → 403 `"Trusted queries cannot be deleted using the API"`, + matching update. - **Response:** JSON request → 200 `{"ok": true}`; form → 302; 404 `"Query not found: x"`. No `confirm` field required (unlike table drop). diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index 11f0a75d..8c3131e9 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -332,12 +332,15 @@ Concerns: ## 8. Behavior that looks like a bug and should be resolved before freezing -1. **Trusted queries: update is blocked, delete is not.** +1. ~~**Trusted queries: update is blocked, delete is not.** `QueryUpdateView` rejects `is_trusted` queries with 403 (stored_queries.py:426-427) but `QueryDeleteView.post` never checks `is_trusted` — an actor with `delete-query` can delete a config-defined trusted query via the API (it will resync on restart, making the - behavior confusing rather than catastrophic). Align delete with update. + behavior confusing rather than catastrophic). Align delete with update.~~ + ✅ **Done** — both the POST endpoint and the HTML confirmation page now + return 403 `"Trusted queries cannot be deleted using the API"`; + `datasette.remove_query()` remains available for internal use. 2. ~~**GET `/db/-/query` with no `?sql=` returns 200 `{"ok": true, "rows": []}`** while `.csv` on the same request returns 400 `"?sql= is required"`. The JSON behavior masks caller bugs; return 400 on both.~~ @@ -401,8 +404,8 @@ Two details make tiering urgent rather than optional: ✅ Done. 7. Publish explicit stability tiers, including extras and pagination-token opacity (§9). -8. Resolve the looks-like-a-bug list (§8), especially trusted-query delete - and row-delete 500. +8. Resolve the looks-like-a-bug list (§8), especially ~~trusted-query delete + and row-delete 500~~ (both done). Everything in P2 is worth doing now because each item is breaking-to-fix later; each P3 can be resolved by a sentence of documentation declaring the diff --git a/tests/test_queries.py b/tests/test_queries.py index b79a9af4..c5c1c3cd 100644 --- a/tests/test_queries.py +++ b/tests/test_queries.py @@ -3654,3 +3654,51 @@ async def test_stored_write_query_with_truncated_returning_message(): assert response.status_code == 200 assert response.json()["ok"] is True assert response.json()["message"] == "Query executed" + + +@pytest.mark.asyncio +async def test_query_delete_api_rejects_trusted_queries(): + ds = Datasette( + memory=True, + default_deny=True, + config={ + "databases": { + "data": { + "permissions": { + "view-query": {"id": "editor"}, + "delete-query": {"id": "editor"}, + }, + "queries": { + "trusted_report": { + "sql": "select 1 as one", + }, + }, + } + } + }, + ) + ds.add_memory_database("query_delete_trusted_api", name="data") + await ds.invoke_startup() + + response = await ds.client.post( + "/data/trusted_report/-/delete", + actor={"id": "editor"}, + json={}, + ) + assert response.status_code == 403 + assert response.json()["errors"] == [ + "Trusted queries cannot be deleted using the API" + ] + # The query must still exist + assert await ds.get_query("data", "trusted_report") is not None + + # The HTML confirmation page refuses too + get_response = await ds.client.get( + "/data/trusted_report/-/delete", + actor={"id": "editor"}, + ) + assert get_response.status_code == 403 + + # datasette.remove_query() remains available for internal use + await ds.remove_query("data", "trusted_report") + assert await ds.get_query("data", "trusted_report") is None From 6488b7a30e792c6b1af0a596ce81f7e1a0f179a3 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 16:00:25 +0000 Subject: [PATCH 015/131] Remove duplicate params key from stored query JSON objects Every stored-query object carried the same list of parameter names twice, as both "params" and "parameters". Output objects now carry only "parameters", consistent with /-/query/parameters and the two analyze endpoints (and distinct from the "params" bound-values dictionary used by the query extra and /-/execute-write). "params" remains an accepted input alias for query creation, update and datasette.yaml config. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/stored_queries.py | 1 - docs/json_api.rst | 1 - existing-api.md | 6 +++--- stable-api-recommendations.md | 7 +++++-- tests/test_queries.py | 30 ++++++++++++++++++++++++++++++ 5 files changed, 38 insertions(+), 7 deletions(-) diff --git a/datasette/stored_queries.py b/datasette/stored_queries.py index a6123daa..d7e1ec99 100644 --- a/datasette/stored_queries.py +++ b/datasette/stored_queries.py @@ -62,7 +62,6 @@ def stored_query_to_dict(query: StoredQuery) -> dict[str, Any]: "description_html": query.description_html, "hide_sql": query.hide_sql, "fragment": query.fragment, - "params": list(query.parameters), "parameters": list(query.parameters), "is_write": query.is_write, "is_private": query.is_private, diff --git a/docs/json_api.rst b/docs/json_api.rst index a561aa9c..54f0b6bd 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -1173,7 +1173,6 @@ The following extras are available for arbitrary SQL query responses and stored, "description_html": null, "hide_sql": false, "fragment": null, - "params": [], "parameters": [], "is_write": false, "is_private": false, diff --git a/existing-api.md b/existing-api.md index e11491fe..abb82e2c 100644 --- a/existing-api.md +++ b/existing-api.md @@ -990,7 +990,7 @@ stored_queries.py:55-80): "database": "...", "name": "...", "sql": "...", "title": null, "description": null, "description_html": null, "hide_sql": false, "fragment": null, - "params": ["p"], "parameters": ["p"], + "parameters": ["p"], "is_write": false, "is_private": true, "is_trusted": false, "source": "user", "owner_id": "...", "on_success_message": null, "on_success_message_sql": null, @@ -1000,8 +1000,8 @@ stored_queries.py:55-80): } ``` -`params` and `parameters` are identical lists, both always present. -`private` appears only in list responses. +`private` appears only in list responses. On input (create/update and +`datasette.yaml`), `params` is accepted as an alias for `parameters`. **Default permission rules for queries** (default_permissions/defaults.py): `view-query` is default-allow, but private queries are visible only to their diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index 8c3131e9..d1388549 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -249,9 +249,12 @@ Concerns: ## 5. Naming and parameter conventions (P2/P3) -- **`params` and `parameters` are duplicate keys** in every stored-query +- ~~**`params` and `parameters` are duplicate keys** in every stored-query object (stored_queries.py:55-80). Delete one before 1.0 (suggest keeping - `parameters`; the write side already accepts both on input). + `parameters`; the write side already accepts both on input).~~ + ✅ **Done** — output objects carry only `parameters` (matching + `/-/query/parameters` and the analyze endpoints); `params` remains an + accepted input alias for API creation and `datasette.yaml` config. - **Three names for the same concept across error/message payloads:** `error`, `errors`, `message`. See §1. - **Boolean query parameters have at least three grammars:** `_nl=on`, diff --git a/tests/test_queries.py b/tests/test_queries.py index c5c1c3cd..11828c4e 100644 --- a/tests/test_queries.py +++ b/tests/test_queries.py @@ -3702,3 +3702,33 @@ async def test_query_delete_api_rejects_trusted_queries(): # datasette.remove_query() remains available for internal use await ds.remove_query("data", "trusted_report") assert await ds.get_query("data", "trusted_report") is None + + +@pytest.mark.asyncio +async def test_stored_query_json_uses_parameters_not_params(): + ds = Datasette( + memory=True, + config={ + "databases": { + "data": { + "queries": { + "with_params": { + "sql": "select :name as name, :age as age", + "params": ["name", "age"], + }, + }, + } + } + }, + ) + ds.add_memory_database("query_parameters_key", name="data") + await ds.invoke_startup() + + definition = (await ds.client.get("/data/with_params/-/definition")).json() + assert definition["query"]["parameters"] == ["name", "age"] + assert "params" not in definition["query"] + + listing = (await ds.client.get("/data/-/queries.json")).json() + query = [q for q in listing["queries"] if q["name"] == "with_params"][0] + assert query["parameters"] == ["name", "age"] + assert "params" not in query From d41fc0c0760b07a300f3208783e1cbcb893d7f29 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 16:03:07 +0000 Subject: [PATCH 016/131] Design review of the SQL-based permission system Comprehensive review covering architecture, verified defects (homepage 500 on 2,000-table instances, also_requires divergence between allowed() and allowed_resources(), unbound automatic SQL parameters, rule source misattribution), benchmark data, incremental recommendations, and radical alternative designs including a compiled grants ledger. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Me9iBLLK2cbnY3Gx78f26V --- sql-permissions-design-review.md | 771 +++++++++++++++++++++++++++++++ 1 file changed, 771 insertions(+) create mode 100644 sql-permissions-design-review.md diff --git a/sql-permissions-design-review.md b/sql-permissions-design-review.md new file mode 100644 index 00000000..d44d9b0d --- /dev/null +++ b/sql-permissions-design-review.md @@ -0,0 +1,771 @@ +# Design review: Datasette's SQL-based permission system + +*Reviewed at commit `58c07cc` (main, July 2026). All benchmark numbers and bug +reproductions in this document were verified against this checkout; repro +scripts are in Appendix B.* + +## 1. Executive summary + +The core architectural bet — **compile permission rules from every source into +one SQL query against the internal catalog, so that "list everything this +actor can see" is a single query rather than N Python checks** — is the right +bet. It solves the historic `permission_allowed()` N+1 problem, it gives +plugins real expressive power, and the `reason`/`source_plugin` columns bake +explainability into the data model rather than bolting it on. + +The current execution of that bet has three classes of problems: + +1. **The primary goal is not yet met.** On a vanilla 2,000-table instance with + *zero* custom rules, `allowed_resources("view-table", include_is_private=True)` + exceeds the internal database's time limit and the homepage returns + **HTTP 500**. Point checks degrade from 0.6ms to 76ms each with 200 config + rules. The generated SQL shape — three `LEFT JOIN` + `GROUP BY` passes over + `resources × rules`, JSON reason aggregation always on, rule rows inlined as + `UNION ALL` text — is the cause, and it is fixable without changing the + architecture (§4.1, §6-R3). + +2. **There are three parallel implementations of the resolution semantics** + (listing, point-check, plus a third used only by tests), and they have + already diverged: `datasette.allowed()` and `datasette.allowed_resources()` + give **different answers** for actions with chained `also_requires` + (§4.2). Several smaller contract bugs — automatic parameters that + aren't always bound, silent parameter collisions, misattributed rule + sources — all stem from the same root: the plugin contract is *SQL strings + plus conventions*, and each code path re-implements the conventions + slightly differently (§4.3–4.6, §5.2). + +3. **Auditability is designed-in but under-delivered.** Reasons and source + attribution exist, and there are five debug endpoints — but the tools are + fragmented, one of them paginates in Python contradicting the SQL design, + the trace only shows the *winning* rules, the `restriction_sql` half of the + plugin contract is completely undocumented, and the central docs section + explaining resolution contains typos and no worked example (§5.6, §5.7). + +Recommended path: fix the verified defects now (§4), consolidate to **one rule +compiler with data-first rules and a temp-table execution strategy** (§6), and +seriously evaluate the "compiled grants ledger" model (§7.1) — which keeps the +SQL execution model but moves rule *evaluation* from request time to +write time, making permissions indexed, diffable, and auditable as a table. + +--- + +## 2. The system as built + +### 2.1 Concepts + +| Concept | Where | Role | +|---|---|---| +| `Action` | `datasette/permissions.py` | Named operation (`view-table`), optional `abbr` (`vt`), optional `resource_class`, optional `also_requires` chain | +| `Resource` | `datasette/permissions.py`, `datasette/resources.py` | Typed `(parent, child)` pair; hierarchy hard-capped at 2 levels; subclasses supply `resources_sql()` returning *all* resources of that type from the catalog | +| `PermissionSQL` | `datasette/permissions.py` | A plugin's contribution: SQL yielding `(parent, child, allow, reason)` rows, bound `params`, and/or a `restriction_sql` allowlist filter | +| `permission_resources_sql` hook | `hookspecs.py` | Called per `(actor, action)`; returns `PermissionSQL` objects | +| Internal catalog | `catalog_databases`, `catalog_tables`, `catalog_views`, `queries` in the internal DB | The "base" set that rules are joined against | + +### 2.2 Rule sources shipped in core + +All of core's own behavior goes through the same hook +(`datasette/default_permissions/`): + +- `defaults.py` — root-level allow rows for the default-public actions + (`view-instance`, `view-table`, …) unless `--default-deny`; the + `default_allow_sql` deny; query-ownership rules for stored queries. +- `config.py` — `ConfigPermissionProcessor` walks `datasette.yaml` + (`permissions:` blocks at root/db/table/query level, `allow:` /`allow_sql:` + blocks), evaluates each allow block against the actor **in Python** + (`actor_matches_allow`), and emits the verdicts as constant + `SELECT :p AS parent, …` rows. +- `restrictions.py` — the `_r` actor key (API-token restrictions) becomes a + `restriction_sql` allowlist, `INTERSECT`ed across providers and applied as a + final `EXISTS` filter that can only *remove* results. +- `root.py` — a root-level allow row for the `--root` user. + +### 2.3 Resolution semantics + +1. **Specificity cascade:** child-level rules beat parent-level rules beat + global rules. +2. **Deny beats allow within the same level.** +3. **Implicit deny** if no rule matches. +4. **Restrictions** filter the result set afterwards; they can never grant. +5. `also_requires` composes actions (`execute-sql` also requires + `view-database`). + +### 2.4 The three resolvers + +The semantics above are implemented **three times**: + +| Path | File | Strategy | +|---|---|---| +| Listing (`allowed_resources[_sql]`) | `utils/actions_sql.py::_build_single_action_sql` | Three `LEFT JOIN`+`GROUP BY` passes (`child_lvl`/`parent_lvl`/`global_lvl`) over `base × rules`, `CASE` cascade, JSON reason aggregation; duplicated again for anonymous when `include_is_private=True` | +| Point check (`allowed`/`allowed_many`) | `utils/actions_sql.py::check_permissions_for_actions` | Per-action rules CTE, depth-ranked `ORDER BY … LIMIT 1` verdict | +| `resolve_permissions_from_catalog` | `utils/permissions.py` | `ROW_NUMBER()` window-ranked winner — **only referenced by tests**; ships in the package as dead weight | + +`allowed_many` batches several actions into one query, expands +`also_requires` transitively in Python, and consults a request-scoped +`contextvars` cache. The listing path handles `also_requires` differently — by +`INNER JOIN`ing two independently built listing queries (see §4.2). + +### 2.5 Debug and audit surface + +- `/-/permissions` — recent-checks log (in-memory `deque(maxlen=200)`) plus a + playground for checking an arbitrary actor/action/resource. +- `/-/allowed` — list resources for an action for the *current* actor, with + reasons if you hold `permissions-debug`. +- `/-/rules` — dump the assembled rule rows (`parent, child, allow, reason, + source_plugin`) per action. +- `/-/check` — point-check API for the current actor. +- `/-/allow-debug` — test an allow block against an actor document. + +This is a genuinely better debug surface than most permission systems ship +with. Its problems are fragmentation and depth, not absence (§5.6). + +--- + +## 3. Assessment against the stated goals + +### Goal: efficiently list all resources an actor can act on + +**Not currently met.** Measured on this checkout (one database, 2,000 tables, +in-memory internal DB, default settings; script in Appendix B): + +| Scenario | Result | +|---|---| +| `allowed_resources("view-table")`, 0 config rules, first 1,000 rows | 839 ms | +| Same with 200 table-level config rules | 934 ms | +| Same with `include_is_private=True` (any rule count, even zero) | **`QueryInterrupted` — exceeds the 1s internal time limit** | +| `GET /` (homepage uses `include_is_private=True`) | **HTTP 500 in 1.2s** | +| Single `allowed()` point check, 0 config rules | 0.6 ms | +| Single `allowed()` point check, 50 config rules | 3.9 ms | +| Single `allowed()` point check, 200 config rules | **75.9 ms** | + +Why (all fixable, see §6-R2/R3): + +- The rules CTE is inlined **SQL text** — one `SELECT :cfg_N_parent …` per + rule joined with `UNION ALL`. 200 config rules ≈ 100KB of SQL and 800+ bound + parameters *per check*, re-generated and re-parsed on every call. SQL text + size — not query execution — dominates the point-check numbers. +- CTE results have no indexes, so each of the three level-joins is a nested + loop over `2,000 tables × R rules`, and the cascade does that three times + (six with `include_is_private`). +- `json_group_array(...)` reason aggregation runs on every row of every level + even when the caller never asked for reasons. +- `include_is_private=True` rebuilds the entire anonymous-actor cascade inside + the same query rather than reusing anything. +- Pagination (`LIMIT`) is applied *after* the full cascade is computed, so + every page pays the full O(N×R) cost; `PaginatedResources.all()` re-runs + the whole thing per page with default `limit=100` — the homepage on the + 2,000-table instance would run the failing query 20 times even if each + succeeded. + +### Goal: flexibility for plugins + +**Strong — the best part of the design.** Arbitrary SQL against the catalog +means a plugin can express "tables whose name starts with `temp_`", "rows in +my own grants table", "databases tagged in a metadata table" without core +anticipating any of it. Custom `Resource` subclasses + `resources_sql()` let +plugins bring entirely new resource types (documents, models) into the same +machinery, including listing. `restriction_sql` gives token-scoping plugins a +sound "can only narrow" primitive. + +The flexibility has sharp edges, though: the contract is stringly-typed +(column names, parameter conventions, "please prefix your params" in the docs) +and core cannot inspect, validate, optimize, or safely compose what plugins +hand it (§5.2). Everything is possible; nothing is checkable. + +### Goal: understandable and auditable by administrators + +**Mixed.** Right instincts — reasons attached to every verdict, source +attribution, a debug playground, `--default-deny`. But: + +- An administrator cannot answer "*who* can see table X and *why*" in one + place; they must mentally join five debug tools, and none shows losing + rules, restriction filtering, or the `also_requires` chain (§5.6). +- The precedence rule that a **more-specific allow overrides a broader deny** + surprises anyone with AWS-IAM/Postgres expectations, and it means *any + installed plugin can grant access to anything* — config has no "final deny" + (§5.3). +- Actor restrictions are a second, parallel permission mini-language (`_r`, + `a`/`d`/`r` keys, action abbreviations) with its own semantics and its own + code paths (§5.4). +- The docs' central "How permissions are resolved" section is thin, contains + typos ("actor cas access", "permission chucks", "replying True"), and + `restriction_sql` — half the plugin contract — is documented **nowhere** + (§5.7). + +--- + +## 4. Verified defects + +Each of these was reproduced against this checkout (Appendix B). + +### 4.1 Homepage 500 / listing performance cliff + +As measured above: `GET /` on a 2,000-table instance returns HTTP 500 because +the `view-table` + `include_is_private` listing query exceeds the internal +database time limit. Note the failure mode compounds: a permission query that +times out surfaces as an unhandled `QueryInterrupted` → 500, rather than a +clear "permission resolution timed out" error. + +### 4.2 `allowed()` and `allowed_resources()` disagree on chained `also_requires` + +`allowed_many()` expands `also_requires` **transitively** in Python +(`store-query` → `execute-sql` → `view-database`). The listing path +(`build_allowed_resources_sql`) combines only the *first* hop: it INNER JOINs +`store-query` with `execute-sql` and never consults `view-database`. + +Verified: with a plugin that denies `view-database` but allows `store-query` +and `execute-sql` globally: + +``` +datasette.allowed(action="store-query", resource=DatabaseResource("_memory")) # False +datasette.allowed_resources("store-query", actor) # ['_memory'] ← disagrees +``` + +This is the drift risk of three resolvers made concrete. It is +security-relevant: any code that trusts the listing path (menus, plugin UIs, +the `/-/allowed` API) will advertise — and potentially act on — permissions +the enforcement path denies. The same divergence class will reappear unless +the resolvers are unified (§6-R1). + +### 4.3 The documented "automatic" parameters are not reliably bound + +`internals.rst` promises `:actor`, `:actor_id` and `:action` are "automatically +available" in `PermissionSQL` SQL. The implementation +(`gather_permission_sql_from_hooks`) does: + +```python +params = permission_sql.params or {} # fresh dict if params is None… +params.setdefault("actor", actor_json) # …mutated… +``` + +…and the fresh dict is then **discarded** (never assigned back to +`permission_sql.params`). The promise only holds if *some other* collected +rule happens to carry a non-None params dict into the shared merge. Core's +default-allow rules usually do — so it works by accident. Under +`--default-deny` with no config rules, a plugin using `:actor_id` with +`params=None` crashes every check with +`ProgrammingError: You did not supply a value for binding parameter :actor_id.` +(verified). + +### 4.4 Rule sources are misattributed + +`gather_permission_sql_from_hooks` pairs hook results with hook +implementations by index: + +```python +hookimpls = hook_caller.get_hookimpls() +hook_results = list(hook_caller(...)) +for index, result in enumerate(hook_results): + hookimpl = hookimpls[index] +``` + +But pluggy **omits `None` results** from `hook_results` while `hookimpls` +retains every implementation, so the lists misalign whenever any hook returns +`None` — which is the normal case (most hooks return `None` for most actions). +Verified: a third-party plugin's rules were attributed to +`datasette.default_permissions` in the generated SQL. This silently corrupts +exactly the metadata (`source_plugin`, shown in `/-/rules` and in reasons) +that the auditability story depends on. + +### 4.5 Parameter namespacing is inconsistent; collisions are silent (by inspection) + +- Listing path: `all_params.update(p.params)` — **no namespacing**. Two + plugins that both bind `:user_id` (or one plugin returning two + `PermissionSQL`s reusing a name) silently last-write-wins, changing the + *other* plugin's rule semantics. The docs handle this by asking plugins to + prefix their params — a convention, unenforced. +- Point-check path: params are rewritten with a regex **per action** + (`a0_user_id`) — but still not per plugin, so cross-plugin collisions + survive there too. +- The `include_is_private` anonymous-rules branch rewrites params with plain + `str.replace(":key", ":anon_key")` — no word boundary, so `:p` corrupts + `:p2` — while the point-check path uses a correct + regex-with-lookahead. Same job, three implementations, one of them wrong. + +Related: `PermissionSQL.allow()/deny()` mint parameter names from a global +module-level counter (`_reason_id`) — global mutable state where content +hashing or per-gather counters would do; and `p.source` is interpolated into +the SQL as `'{p.source}'` unescaped, so a plugin name containing `'` breaks +every query it participates in (robustness, not injection — the value comes +from the plugin itself). + +### 4.6 Assorted smaller issues (by inspection) + +- `build_permission_rules_sql` docstring says it returns a 2-tuple; it returns + a 3-tuple. +- Keyset pagination encodes a `NULL` child as the literal string `"None"` + (`tilde_encode(str(None))`) and its `WHERE (parent > :p OR (parent = :p AND + child > :c))` silently drops rows with `NULL` children on continuation + pages. It happens to work for the built-in resource types (databases have + unique parents; tables/queries always have children) but is a trap for any + plugin resource type with NULL children. +- `defaults.py` still does `reason.replace("'", "''")` on a value that is + passed as a bound parameter — leftover from a string-interpolation era; + reads as if interpolation might still happen somewhere. +- The obsolete `Permission` dataclass ships with a comment saying it is + obsolete; `resolve_permissions_from_catalog` / `resolve_permissions_with_candidates` + (~300 lines including a third copy of the cascade) are exercised only by + tests. + +--- + +## 5. Design concerns + +### 5.1 Three resolvers, one intended semantics + +§4.2 is the proof that this is not hypothetical. Cascade precedence, +`also_requires`, restriction filtering, param handling, and skip-checks each +exist in 2–3 variants. There is no test asserting the core invariant: + +> for every actor, action, resource: +> `allowed(action, r, actor)` ⇔ `r ∈ allowed_resources(action, actor)` + +That property test would have caught §4.2 and will catch the next drift. + +### 5.2 The plugin contract is "SQL strings + conventions" + +Column names, parameter naming, reserved parameters, source attribution, +quoting — all conventions enforced by nothing. Because rules arrive as opaque +SQL text, core cannot: + +- validate a rule at registration time (typos surface as runtime SQL errors + inside a 100KB generated query); +- index or pre-aggregate rules (root cause of §4.1); +- show an administrator "the rules" in any form other than *executing* + everything (`/-/rules` runs the SQL to show its output — correct, but + policies can't be reviewed statically); +- statically analyze policies (find shadowed rules, contradictions, or answer + "which rules mention table X?"). + +The telling detail: core's own `config.py` doesn't want the SQL flexibility — +it evaluates everything in Python and emits *constant rows* through +`PermissionRowCollector`. The majority use case is rows, not SQL; the design +taxes the common case with the escape hatch's costs. (§6-R2 proposes inverting +this.) + +### 5.3 "Specific allow beats broader deny" + "any plugin can grant" needs guardrails + +The cascade's child-allow-overrides-parent-deny rule is a defensible design +choice (it's what makes "deny the db, allow one table" expressible), but it +combines badly with the open hook: an administrator who writes a root-level +deny in `datasette.yaml` has **no way to make it final**. Any installed +plugin can emit a child-level allow row that silently wins. For an +administrator, "what can Alice see?" is only answerable by trusting every +installed plugin's rule emission. + +Options worth considering, in increasing strength: + +1. Document it loudly ("installing a plugin extends the set of parties who can + grant access") and surface *which plugin granted* prominently in every + debug view (blocked today by §4.4). +2. Rule *tiers*: config rules could optionally be marked `final`, evaluated + after plugin rules with deny-wins. +3. A `--paranoid` mode where config is the ceiling: plugins may only narrow. + +### 5.4 Restrictions are a second permission language + +The `_r` mechanism has its own vocabulary (`a`/`d`/`r`, action +abbreviations), its own resolution semantics (pure allowlist + `INTERSECT` +across providers), its own Python fast path (`restrictions_allow_action`), and +special-case interplay with config (`_add_restriction_gate_denies`, the +hardest ~40 lines in `config.py`, exist solely to stop a child-level config +allow from defeating a restriction). The config processor's +`is_in_restriction_allowlist` additionally has a "parent proceeds if any child +is allowlisted" special case that the SQL `EXISTS` filter does not mirror — +another place semantics live twice, subtly differently. + +The concept is right (attenuated tokens must never escalate). The +implementation would be simpler as a first-class post-filter stage in the one +canonical compiler, with a documented wire format — and §7.3 argues +restrictions and grants may want to become the *same* algebra. + +Also: action abbreviations (`vt`, `es`) exist to keep tokens small, but they +leak into every comparison via `get_action_name_variants` — dual-name matching +in at least four call sites. Consider making abbreviation expansion a +token-decode concern, so the rest of the system only ever sees full names. + +### 5.5 The two-level hierarchy is a hard cap + +`Resource.__init_subclass__` raises on a third level. Fine for +instance/database/table, but plugins with deeper models +(collection/document/section) must flatten, and a future column-level +permission would break the world. The `(parent, child)` schema also leaks +generic names into every API response and debug view where +`database`/`table` would read better. Not urgent — but this is exactly the +kind of decision that becomes unfixable after a 1.0 API freeze, so it deserves +an explicit "yes, forever" or a path-style key design (§7.1's ledger uses +one) now. + +### 5.6 Debug tooling: right pieces, missing the whole + +- Five endpoints with overlapping-but-different capabilities and no + cross-links; an admin must already understand the system to know which tool + answers which question. +- `/-/allowed` fetches **all** rows into Python, then applies the `child` + filter and offset pagination in Python — quietly contradicting (and + bypassing) the keyset-pagination design directly underneath it, and turning + the debug tool into the least scalable consumer of the API it demonstrates. +- Reasons only surface the winning level's rules. "Why *can't* Alice see + X?" — the auditor's most common question — has no answer today: you can't + see the losing allow that was beaten by a deny, the restriction that + filtered a granted row out, or the `also_requires` link that failed. +- The `/-/permissions` check log is a process-local `deque(maxlen=200)` — + gone on restart, per-process on multi-worker deploys. + +### 5.7 Documentation and naming residue + +- `restriction_sql`: undocumented (zero occurrences under `docs/`). +- `internals.rst` documents the automatic parameters unconditionally (§4.3 + makes that false), and documents `PermissionSQL` with a stale field order. +- `authentication.rst`'s "How permissions are resolved" — the section an + auditor most needs — has typos ("actor cas access", "permission chucks", + "replying ``True`` to all permission chucks") and describes the mechanism in + prose without a precedence table or a single worked multi-rule example. +- Terminology drift: the hook is `permission_resources_sql`, the registry is + `datasette.actions`, registered by `register_actions`, holding `Action` + objects, documented under "Permissions"; the obsolete `Permission` class is + still importable. Pick "action" everywhere and finish the migration before + 1.0 freezes the names. + +--- + +## 6. Recommendations (incremental — keep the architecture) + +Ordered so that each unlocks the next; R1–R5 are pre-1.0 material because they +change plugin-visible behavior. + +**R1. One rule compiler, one semantics, one parity test.** +Extract a single module that owns: gathering hook results, param namespacing, +`also_requires` expansion (transitive, in one place — or better, resolve the +chain into a frozen set per action at registration time), restriction +collection, and cascade compilation. Both `allowed_many` and +`allowed_resources_sql` consume it; delete the test-only third resolver and +the obsolete `Permission` class. Add the property test from §5.1 (hypothesis +over random rule sets, or brute-force over fixture matrices) so listing and +point-check can never disagree again. This closes §4.2 structurally, not just +locally. + +**R2. Make rules data-first; SQL becomes the escape hatch.** +`PermissionRowCollector` already proves core wants rows. Let +`permission_resources_sql` (or a successor hook name like `permission_rules`) +return row objects (`Rule(parent, child, allow, reason)`) as the primary form, +with `PermissionSQL` still accepted for genuinely dynamic cases. Then the +compiler can: + +- insert row-rules into an **indexed temp table** once per request (or cache + by `(actor-hash, action)`), instead of generating O(rules) SQL text — this + alone removes the 76ms point-check pathology (§4.1), which is dominated by + SQL parse size, and gives the listing query indexed joins; +- validate rules at collection time (types, unknown actions, reserved names) + with plugin-attributed errors; +- namespace parameters automatically per (plugin, hook-result) for the SQL + escape hatch, using the one correct regex implementation (§4.5), and always + bind `:actor`/`:actor_id`/`:action` at the query level rather than + per-rule-params (§4.3); +- fix source attribution by carrying the plugin name from the hookimpl at + gather time, matched correctly (§4.4 — pluggy's + `hook_caller.call_extra`/wrapper mechanisms or simply wrapping each impl can + give exact pairing). + +**R3. Fix the listing query shape.** +Replace the three `LEFT JOIN`+`GROUP BY` level passes with the single +depth-ranked pass that already exists in the codebase (the `ROW_NUMBER()` +winner CTE), computed off the indexed rules table from R2: + +- compute reasons only when `include_reasons=True` (the JSON aggregation is + pure overhead otherwise); +- for `include_is_private`, evaluate the anonymous verdict from the *same* + rules table (anon rules are a second small rule set, not a reason to + duplicate the whole query); +- keyset-paginate with NULL-safe comparisons and an explicit NULL token + encoding rather than `"None"` (§4.6); +- add a CI benchmark fixture (e.g. 5,000 tables × 500 rules) with a budget + assertion, so the homepage-500 class of regression (§4.1) is caught by + tests, not users. + +The point of the original three-pass shape was clarity of the cascade; that +clarity should live in the one compiler's tests, not in the runtime query +plan. + +**R4. Decide the trust model and say it out loud.** +Whichever option from §5.3 is chosen (even "option 1: document it"), the +decision belongs in `authentication.rst` next to a precedence table and a +worked example: rules from three sources, one resource, showing exactly which +row wins and why. Fail closed *gracefully*: a permission query that errors or +times out should produce a clear "permission resolution failed" 500 with the +action named, not a raw `QueryInterrupted` (§4.1) — and the internal DB may +deserve a higher/separate time limit for permission queries than user-facing +SQL. + +**R5. Fold restriction handling into the compiler.** +One implementation of the allowlist semantics (SQL `EXISTS` version), used by +both paths; `restrictions_allow_action` and the config restriction-gate become +thin delegations or disappear. Expand abbreviations at token decode. Document +the `_r` format as a reference table. + +**R6. Unify the debug tools around "explain".** +One endpoint (and matching CLI) that answers the auditor's actual questions: + +``` +/-/permissions/explain?actor={...}&action=view-table&parent=db&child=t +``` + +returning the full trace: every candidate rule from every source (winning +*and* losing, with source plugin — fixed by R2), the specificity level at +which the decision was made, restriction filtering before/after, the +`also_requires` chain with each link's verdict, and the final answer. The +existing five pages become views over this one trace. Add: + +- `datasette permissions list|explain|diff|dump` CLI (works offline against + config + plugins; `diff actor-a.json actor-b.json` for "what does this role + change?"; `dump --csv` for compliance export); +- a persistent, opt-in check log (internal DB table with a cap) replacing the + in-memory deque for multi-process deployments. + +**R7. Documentation pass.** +Fix the typos in the resolution section; document `restriction_sql`, the +automatic-parameter contract (after R2 makes it true), the trust model (R4), +the `_r` reference; add a "Debugging permissions" guide that walks one +scenario through the explain tool; add a cookbook (default-deny + groups +plugin, public-except-one-table, token-scoped API access). + +--- + +## 7. Radically different approaches + +The stated goals pull in different directions: *arbitrary per-request SQL* +(flexibility) fights *indexed lookup* (listing speed) fights *static +reviewability* (audit). The current design sits at the "maximum flexibility" +corner and pays for it at the other two. Both alternatives below deliberately +move the trade-off point. + +### 7.1 The compiled grants ledger (recommended candidate) + +**Idea: stop evaluating rules at request time. Evaluate them when they +*change*, into a physical table; requests just read the table.** "Compile, +don't interpret." + +Split the problem in two: + +**Phase A — actor → principals (request time, Python, cheap).** +A new hook resolves an actor into a set of principal strings: + +```python +@hookimpl +def actor_principals(datasette, actor): + # e.g. ["anyone", "authenticated", "id:alice", "team:analytics", "role:admin"] + ... +``` + +This is where per-request dynamism lives (group membership, IdP claims, +"business hours"). It is pure Python, trivially testable, and — crucially — +plugins express *identity*, not *policy*. + +**Phase B — grants ledger (write time, SQL, indexed).** +A real table in the internal database: + +```sql +CREATE TABLE grants ( + principal TEXT NOT NULL, -- "team:analytics", "anyone", … + action TEXT NOT NULL, -- full names only + parent TEXT, -- NULL = all + child TEXT, -- NULL = all at parent level + child_like TEXT, -- optional pattern grant: 'temp_%' + allow INTEGER NOT NULL, -- 1 grant / 0 deny + tier INTEGER NOT NULL DEFAULT 0, -- e.g. config-final > plugin > default + source TEXT NOT NULL, -- plugin/config attribution + reason TEXT NOT NULL, + created_at TEXT, expires_at TEXT +); +CREATE INDEX idx_grants_lookup ON grants (action, principal, parent, child); +``` + +Populated by: the config compiler at startup; plugins via a +`register_grants` hook or by writing rows directly and emitting an +invalidation event; token restrictions as deny-tier rows scoped to a +`token:` principal. Rules that today are dynamic SQL over the catalog +("all tables starting with `temp_`") become pattern rows or are re-expanded by +a catalog-change listener (Datasette already has `refresh_schemas` as the +natural hook point). + +**Reads become trivial and fast:** + +```sql +-- Point check: microseconds, fully indexed +SELECT allow FROM grants +WHERE action = :action AND principal IN (:p1, :p2, :p3) + AND (parent IS NULL OR parent = :parent) + AND (child IS NULL OR child = :child OR :child LIKE child_like) +ORDER BY tier DESC, + (child IS NOT NULL OR child_like IS NOT NULL) DESC, + (parent IS NOT NULL) DESC, + allow ASC +LIMIT 1; + +-- Listing: one indexed join against the catalog — same cascade, same +-- deny-beats-allow, but O(matching grants) instead of O(resources × rules), +-- and the query text is CONSTANT SIZE regardless of rule count. +``` + +**What this buys, measured against the three goals:** + +- *Listing*: indexed join, constant-size SQL. Thousands of tables × + hundreds of grants is interactive by construction. Pagination is ordinary + SQL pagination. +- *Flexibility*: preserved but relocated — plugins do identity (Phase A) + and grant management (writes), instead of per-request policy SQL. A + compatibility shim can run legacy `PermissionSQL` plugins by materializing + their output into session-scoped grants, with a deprecation warning on + divergence. +- *Auditability — the transformative win*: **the policy is a table.** + `SELECT * FROM grants WHERE parent='accounting'` *is* the audit. Dump it, + diff it between deploys, keep a `grants_history` trigger for "who could see + this table last March?", review it in a PR when config changes. The `tier` + column gives administrators the "final deny" that §5.3 cannot express + today. Explain-tooling becomes a `SELECT`, not a query-plan archaeology + session. + +**Costs and open problems, honestly:** + +- Rules conditioned on arbitrary actor JSON must be expressible as principals; + pathological cases ("actors whose email domain matches a table naming + scheme") get awkward. Keeping a narrow `PermissionSQL` escape hatch that is + documented as *slow path, unindexed* is probably the right release valve. +- Cache invalidation is now a real subsystem (schema changes × plugin grant + changes × config reloads). Datasette's catalog-refresh machinery is the + precedent, but it must be airtight because staleness here is a security bug + — an *allow* that outlives its revocation. Mitigations: version-stamp the + ledger and rebuild on any registered source's version bump; deny-tier rows + take effect immediately by also being checked from Phase A. +- Ephemeral principals (a token minted per request) need session-scoped grant + overlays — which is what `restriction_sql` is today, kept as a read-time + `EXISTS` filter against a small per-request set. + +### 7.2 Policies as data (Cedar-style), compiled to SQL + +A middle path that keeps request-time evaluation but replaces *SQL strings* +with *declarative policy objects*: + +```python +Rule( + effect="allow", + principals={"team": "analytics"}, # allow-block-style actor matcher + actions=["view-table", "view-query"], + resources=ResourceMatch(parent="analytics", child_like="*"), + priority=10, + reason="analytics team reads analytics DB", +) +``` + +Core compiles these to exactly the SQL it generates today — but because it +*understands* the rules, it can also: statically list all policies touching a +resource, detect shadowed/contradictory rules at startup, render +human-readable policy summaries for admins, and generate the explain trace +without executing anything. This is essentially R2 taken to its logical +conclusion (the allow-block language generalized and given to plugins), and it +composes with either the current engine or the ledger of §7.1 — policy objects +are what you'd *write*, the ledger is what they'd *compile to*. If §7.1 feels +too big for one step, §7.2 is the radical change with a migration path +measured in weeks: the hook keeps its shape, but returns data instead of SQL. + +### 7.3 Authentication-time capabilities + +Invert the lookup entirely: resolve permissions **once, when the actor is +established**, and carry them in the actor — a generalization of the existing +`_r` restrictions from "attenuation only" to the full grant set: + +```json +{"id": "alice", "_caps": {"view-table": ["analytics/*", "prod/orders"], + "execute-sql": ["analytics"]}} +``` + +Checks become pure functions of `(actor, catalog)` — no rule gathering, no +per-request SQL. Listing is one indexed match of patterns against the catalog. +Signed tokens make the whole thing stateless across processes and even across +services (a companion API can verify capabilities without running Datasette). + +Honest assessment: revocation latency (capabilities live until the +cookie/token expires), token size pressure (hence patterns, hence the +abbreviation problem again), and login-time cost make this wrong as *the* +system. But it is worth naming because Datasette already has half of it +(`_r`), and the current design's most confusing aspect is that grants and +restrictions are *different algebras*. A unified capability algebra — grants +computed per §7.1, attenuated by tokens using the *same* representation — +would delete an entire subsystem's worth of special cases (§5.4). + +### Comparison + +| | Current (SQL-per-request) | §7.1 Grants ledger | §7.2 Policy objects | §7.3 Capabilities | +|---|---|---|---|---| +| List 10k tables | Seconds / times out (today) | ms, indexed | Same as current unless compiled to ledger | ms, pattern match | +| Point check | ms→tens of ms, scales with rules | µs | ms | µs | +| Plugin flexibility | Maximal (arbitrary SQL) | Identity + grant writes; SQL escape hatch | Declarative matchers | Login-time resolution | +| Admin audit | Execute-and-inspect only | **Policy is a diffable table** | Statically analyzable | Read the token | +| Revocation | Immediate | Immediate (invalidation must be airtight) | Immediate | Token lifetime | +| Migration cost | — | High (shim possible) | Moderate | High | + +--- + +## 8. Suggested sequencing + +1. **Now (bugfix, no API change):** §4.3 param binding, §4.4 attribution, §4.5 + anon-rewrite regex, §4.6 nits; graceful failure for interrupted permission + queries; parity property test (will initially fail on §4.2). +2. **Pre-1.0 (contract-affecting):** R1 single compiler (fixes §4.2), R2 + data-first rules + auto-namespacing, R5 restriction unification, R4 trust + model decision — these change what plugins are promised, so they must land + before the 1.0 freeze. +3. **Performance:** R3 query shape + temp-table rules + CI benchmark. Success + criterion: 2,000-table homepage under 100ms; point check under 2ms at 500 + rules. +4. **Audit surface:** R6 explain endpoint + CLI, R7 docs pass. +5. **Post-1.0 exploration:** prototype §7.1 (optionally expressed via §7.2 + policy objects) as a plugin first — the hook architecture is flexible + enough to host its own successor, which is itself a good sign about the + hook architecture. + +--- + +## Appendix A: benchmark detail + +Setup: one SQLite database with 2,000 tables (`t00000`…`t01999`), default +settings, in-memory internal database, config granting `allow: {id: alice}` on +the first N tables. Times are steady-state (after warm-up) on this review +container; absolute numbers will vary but the *shape* (linear SQL-text growth, +O(N×R) joins, time-limit interrupt) is structural. + +| Config rules | First page (1,000) `view-table` | + `include_is_private` | Point check | +|---:|---:|---|---:| +| 0 | 839 ms | `QueryInterrupted` | 0.6 ms | +| 50 | 875 ms | `QueryInterrupted` | 3.9 ms | +| 200 | 934 ms | `QueryInterrupted` | 75.9 ms | +| 1,000 | `QueryInterrupted` | — | — | + +`GET /` (which calls `allowed_resources("view-table", include_is_private=True)`): +HTTP 500 in 1.2s at every tested rule count. + +## Appendix B: reproduction scripts + +**B.1 — `also_requires` divergence (§4.2):** register a plugin returning +`PermissionSQL.deny()` for `view-database` and `PermissionSQL.allow()` for +`store-query`/`execute-sql`; compare +`await ds.allowed(action="store-query", resource=DatabaseResource("_memory"), actor={"id":"bob"})` +(→ `False`) with +`await ds.allowed_resources("store-query", {"id":"bob"})` (→ contains `_memory`). + +**B.2 — unbound automatic params (§4.3):** run `Datasette(memory=True, +default_deny=True)` with a plugin returning +`PermissionSQL(sql="SELECT NULL, NULL, CASE WHEN :actor_id='alice' THEN 1 ELSE 0 END, 'r'")` +(no `params`); any `allowed_resources("view-table", {"id":"alice"})` call +raises `ProgrammingError: You did not supply a value for binding parameter :actor_id`. +Remove `default_deny` and it "works" because core's default rules smuggle the +binding in. + +**B.3 — source misattribution (§4.4):** with the B.2 plugin registered under +name `no_params_plugin`, inspect +`(await ds.allowed_resources_sql(action="view-table", actor={"id":"alice"})).sql` +— the plugin's SELECT appears tagged +`'datasette.default_permissions' AS source_plugin`. + +**B.4 — performance (§4.1, Appendix A):** create 2,000 tables, start +`Datasette(["bench.db"])`, `await ds.client.get("/")` → HTTP 500 with +`QueryInterrupted` from `views/index.py:41`. From b09dceea889bf076c3d35af406c5b0ae50654731 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 16:03:05 +0000 Subject: [PATCH 017/131] Row update return:true responds with rows list, matching insert/upsert Row update previously returned a singular "row" object where insert and upsert return a "rows" list. All write endpoints now use "rows". Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/views/row.py | 2 +- docs/json_api.rst | 12 +++++++----- existing-api.md | 4 ++-- stable-api-recommendations.md | 5 +++-- tests/test_api_write.py | 4 ++-- tests/test_error_shape.py | 24 ++++++++++++++++++++++++ tests/test_table_html.py | 2 +- 7 files changed, 40 insertions(+), 13 deletions(-) diff --git a/datasette/views/row.py b/datasette/views/row.py index bdd78ed4..71539f34 100644 --- a/datasette/views/row.py +++ b/datasette/views/row.py @@ -828,7 +828,7 @@ class RowUpdateView(BaseView): resolved.sql, resolved.params, truncate=True ) returned_row = results.dicts()[0] - result["row"] = returned_row + result["rows"] = [returned_row] await self.ds.track_event( UpdateRowEvent( diff --git a/docs/json_api.rst b/docs/json_api.rst index 54f0b6bd..b8287a64 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -1955,11 +1955,13 @@ The returned JSON will look like this: { "ok": true, - "row": { - "id": 1, - "title": "New title", - "other_column": "Will be present here too" - } + "rows": [ + { + "id": 1, + "title": "New title", + "other_column": "Will be present here too" + } + ] } Any errors will use the :ref:`standard error format `, with a ``400`` status code for a bad input or a ``403`` status code for an authentication or permission error. diff --git a/existing-api.md b/existing-api.md index abb82e2c..eb7b5c20 100644 --- a/existing-api.md +++ b/existing-api.md @@ -958,8 +958,8 @@ does not change the SQLite schema. `"Invalid keys: ..."`; write failures (bad column, constraint violation) → 400 with the message. - **Response** — 200 `{"ok": true}`; with `return: true`, - `{"ok": true, "row": {...}}` (singular `row`, unlike insert/upsert's - `rows`). Emits `update-row`. + `{"ok": true, "rows": [{...}]}` — a single-item list, matching + insert/upsert. Emits `update-row`. ### POST /\/\/\/-/delete diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index d1388549..a4dbb344 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -160,10 +160,11 @@ Endpoints disagree about the success envelope: (index.py:147-161); `/-/databases.json` returns an **array**; the database page returns `tables` as an array. Choose arrays-of-objects everywhere (objects-keyed-by-name break when names need ordering or pagination). -- Insert/upsert with `return: true` respond with `rows` (plural, list); row +- ~~Insert/upsert with `return: true` respond with `rows` (plural, list); row update with `return: true` responds with `row` (singular, object) (views/row.py:837-844). Pick one (`rows` everywhere, even for one row, - matches the read API). + matches the read API).~~ ✅ **Done** — row update now returns + `rows: [{...}]`. ### 2b. `_extra`/`_shape` support is uneven (P2) diff --git a/tests/test_api_write.py b/tests/test_api_write.py index a29f5c99..e3363db0 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -1481,9 +1481,9 @@ async def test_update_row(ds_write, input, expected_errors, use_return): assert response.json()["ok"] is True if not use_return: - assert "row" not in response.json() + assert "rows" not in response.json() else: - returned_row = response.json()["row"] + returned_row = response.json()["rows"][0] assert returned_row["id"] == pk for k, v in input.items(): assert returned_row[k] == v diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index 9d4a566c..29f4e8fc 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -507,3 +507,27 @@ async def test_query_html_without_sql_is_still_the_editor(ds_client): response = await ds_client.get("/fixtures/-/query") assert response.status_code == 200 assert response.headers["content-type"].startswith("text/html") + + +# Write API return:true responses use "rows" consistently + + +@pytest.mark.asyncio +async def test_row_update_return_uses_rows_list(ds_error_shape): + await ds_error_shape.client.post( + "/data/docs/-/insert", + json={"row": {"id": 1, "title": "One"}}, + headers={"Content-Type": "application/json"}, + actor={"id": "root"}, + ) + response = await ds_error_shape.client.post( + "/data/docs/1/-/update", + json={"update": {"title": "Updated"}, "return": True}, + headers={"Content-Type": "application/json"}, + actor={"id": "root"}, + ) + assert response.status_code == 200 + data = response.json() + assert data["ok"] is True + assert "row" not in data + assert data["rows"] == [{"id": 1, "title": "Updated"}] diff --git a/tests/test_table_html.py b/tests/test_table_html.py index 46d43c6c..86fb4254 100644 --- a/tests/test_table_html.py +++ b/tests/test_table_html.py @@ -1629,7 +1629,7 @@ async def test_row_update_sets_message(): json={"update": {"name": long_name}, "return": True}, ) assert response.status_code == 200 - assert response.json()["row"]["name"] == long_name + assert response.json()["rows"][0]["name"] == long_name assert ds.unsign(response.cookies["ds_messages"], "messages") == [ ["Updated row 1 ({})".format(truncated_name), ds.INFO] ] From b958d03c0f0f322eeb4135e141e7877734211eef Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 16:09:40 +0000 Subject: [PATCH 018/131] Expose count truncation in table JSON via count_truncated extra The count extra is computed with a limit subquery, so a count equal to count_limit + 1 (default 10001) actually means "at least this many" - but only the HTML view knew that. A public count_truncated extra now reports the flag and is implicitly included whenever count is requested, using the same logic the HTML view already used. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/views/table.py | 23 ++++---------------- datasette/views/table_extras.py | 34 ++++++++++++++++++++++++++++++ docs/json_api.rst | 9 ++++++++ existing-api.md | 3 ++- stable-api-recommendations.md | 5 ++++- tests/test_table_api.py | 37 +++++++++++++++++++++++++++++++++ 6 files changed, 90 insertions(+), 21 deletions(-) diff --git a/datasette/views/table.py b/datasette/views/table.py index ef9831b2..44919281 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -62,6 +62,7 @@ from .table_create_alter import ( from .table_extras import ( TABLE_EXTRA_BUNDLES, TableExtraContext, + count_is_truncated, precompute_database_action_permissions, precompute_table_action_permissions, resolve_table_extras, @@ -2257,6 +2258,8 @@ async def table_view_data( extras.add("facet_results") if request.args.get("_shape") == "object": extras.add("primary_keys") + if "count" in extras: + extras.add("count_truncated") if extra_extras: extras.update(extra_extras) @@ -2335,7 +2338,7 @@ async def table_view_data( data["rows"] = transformed_rows if context_for_html_hack: - data["count_truncated"] = _count_truncated_for_table_page( + data["count_truncated"] = count_is_truncated( datasette, db, database_name, table_name, count_sql, data.get("count") ) data.update(extra_context_from_filters) @@ -2403,24 +2406,6 @@ async def table_view_data( return data, rows[:page_size], columns, expanded_columns, sql, next_url -def _count_truncated_for_table_page( - datasette, db, database_name, table_name, count_sql, count -): - if count != db.count_limit + 1: - return False - if ( - not db.is_mutable - and datasette.inspect_data - and count_sql == f"select count(*) from {table_name} " - ): - try: - datasette.inspect_data[database_name]["tables"][table_name]["count"] - return False - except KeyError: - pass - return True - - async def _next_value_and_url( datasette, db, diff --git a/datasette/views/table_extras.py b/datasette/views/table_extras.py index db659c80..f8ece55b 100644 --- a/datasette/views/table_extras.py +++ b/datasette/views/table_extras.py @@ -140,6 +140,39 @@ class CountExtra(Extra): return count +def count_is_truncated(datasette, db, database_name, table_name, count_sql, count): + if count != db.count_limit + 1: + return False + if ( + not db.is_mutable + and datasette.inspect_data + and count_sql == f"select count(*) from {table_name} " + ): + try: + datasette.inspect_data[database_name]["tables"][table_name]["count"] + return False + except KeyError: + pass + return True + + +class CountTruncatedExtra(Extra): + description = "True if the count hit Datasette's counting limit, meaning the real number of matching rows is at least the reported count." + example = ExtraExample("/fixtures/facetable.json?_extra=count,count_truncated") + scopes = {ExtraScope.TABLE} + expensive = True + + async def resolve(self, context, count): + return count_is_truncated( + context.datasette, + context.db, + context.database_name, + context.table_name, + context.count_sql, + count, + ) + + class FacetInstancesProvider(Provider): scopes = {ExtraScope.TABLE} @@ -1196,6 +1229,7 @@ TABLE_EXTRA_BUNDLES = { TABLE_EXTRA_CLASSES = [ CountExtra, + CountTruncatedExtra, CountSqlExtra, FacetResultsExtra, FacetsTimedOutExtra, diff --git a/docs/json_api.rst b/docs/json_api.rst index b8287a64..2a859ba2 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -302,6 +302,15 @@ The available table extras are listed below. 15 +``count_truncated`` + True if the count hit Datasette's counting limit, meaning the real number of matching rows is at least the reported count. (May execute additional queries.) + + ``GET /fixtures/facetable.json?_extra=count,count_truncated`` + + .. code-block:: json + + false + ``count_sql`` SQL query string used to calculate the total count for the current table view, including active filters. diff --git a/existing-api.md b/existing-api.md index eb7b5c20..a8a2237f 100644 --- a/existing-api.md +++ b/existing-api.md @@ -658,7 +658,8 @@ views/table_extras.py:1197-1235; unknown names silently ignored): | `_extra=` | Returns | |---|---| -| `count` | total matching-row count, computed with a `limit 10001` subquery so it caps at 10001; `null` with `_nocount` or on count timeout | +| `count` | total matching-row count, computed with a `limit 10001` subquery so it caps at 10001; `null` with `_nocount` or on count timeout. Requesting `count` implicitly includes `count_truncated` | +| `count_truncated` | `true` when `count` hit the counting limit (the real count is at least the reported value) | | `count_sql` | the SQL used for the count | | `facet_results` | `{"results": {name: facet}, "timed_out": [...]}`; each facet: `{name, type, hideable, toggle_url, results: [{value, label, count, toggle_url, selected}], truncated}` | | `facets_timed_out` | facet names that exceeded `facet_time_limit_ms` | diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index a4dbb344..64b2acbc 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -178,7 +178,10 @@ is a table/row/query feature. Also decide the contract for **unknown silent ignoring means typos return the default payload with no signal; recommend a 400 or a `warnings` key. -### 2c. Count truncation is invisible in JSON (P2) +### 2c. Count truncation is invisible in JSON (P2) — ✅ IMPLEMENTED + +> **Status:** implemented — a public `count_truncated` extra now exists and +> is implicitly included whenever `count` is requested. The `count` extra is computed with a `limit 10001` subquery, so `count: 10001` actually means "at least 10001" — the `count_truncated` flag exists diff --git a/tests/test_table_api.py b/tests/test_table_api.py index 41c89f39..e3737ee6 100644 --- a/tests/test_table_api.py +++ b/tests/test_table_api.py @@ -1504,6 +1504,7 @@ async def test_col_nocol_errors(ds_client, path, expected_error): "rows": [{"id": "1", "content": "hey", "content2": "world"}], "truncated": False, "count": 1, + "count_truncated": False, }, ), ), @@ -1590,3 +1591,39 @@ async def test_extra_render_cell(): finally: ds.pm.unregister(name="TestRenderCellPlugin") + + +@pytest.mark.asyncio +async def test_count_truncated_included_with_count_extra(tmp_path_factory): + from datasette.app import Datasette + from datasette.utils import sqlite3 + + db_directory = tmp_path_factory.mktemp("dbs") + db_path = str(db_directory / "counts.db") + conn = sqlite3.connect(db_path) + conn.execute("vacuum") + conn.execute("create table big (id integer primary key)") + conn.execute("create table small (id integer primary key)") + conn.executemany("insert into big (id) values (?)", [(i,) for i in range(10)]) + conn.executemany("insert into small (id) values (?)", [(i,) for i in range(3)]) + conn.commit() + conn.close() + ds = Datasette([db_path]) + ds.get_database("counts").count_limit = 5 + try: + response = await ds.client.get("/counts/big.json?_extra=count") + data = response.json() + # Count is capped at count_limit + 1 and flagged as truncated + assert data["count"] == 6 + assert data["count_truncated"] is True + + response = await ds.client.get("/counts/small.json?_extra=count") + data = response.json() + assert data["count"] == 3 + assert data["count_truncated"] is False + + # count_truncated can also be requested on its own + response = await ds.client.get("/counts/big.json?_extra=count_truncated") + assert response.json()["count_truncated"] is True + finally: + ds.close() From 9ee95cab3d9f451dd3529a83bd54a3998be436e0 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 16:11:30 +0000 Subject: [PATCH 019/131] Schema endpoints check permission before database existence /db/-/schema previously returned 404 for missing databases before checking view-database, letting unauthorized actors probe for database existence. The permission check now runs first, so actors without view-database get a uniform 403. The table schema endpoint also now returns 404 for an unknown database instead of an unhandled KeyError. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/views/special.py | 13 +++++++----- existing-api.md | 4 ++-- stable-api-recommendations.md | 6 ++++-- tests/test_error_shape.py | 39 +++++++++++++++++++++++++++++++++++ 4 files changed, 53 insertions(+), 9 deletions(-) diff --git a/datasette/views/special.py b/datasette/views/special.py index 82a76e76..6afb6437 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -1321,17 +1321,17 @@ class DatabaseSchemaView(SchemaBaseView): database_name = request.url_vars["database"] format_ = request.url_vars.get("format") or "html" - # Check if database exists - if database_name not in self.ds.databases: - return self.format_error_response("Database not found", format_) - - # Check view-database permission + # Permission check comes first, so actors without view-database + # cannot distinguish existing databases from missing ones await self.ds.ensure_permission( action="view-database", resource=DatabaseResource(database=database_name), actor=request.actor, ) + if database_name not in self.ds.databases: + return self.format_error_response("Database not found", format_) + schema = await self.get_database_schema(database_name) if format_ == "json": @@ -1365,6 +1365,9 @@ class TableSchemaView(SchemaBaseView): actor=request.actor, ) + if database_name not in self.ds.databases: + return self.format_error_response("Database not found", format_) + # Get schema for the table db = self.ds.databases[database_name] result = await db.execute( diff --git a/existing-api.md b/existing-api.md index a8a2237f..6ce29bf5 100644 --- a/existing-api.md +++ b/existing-api.md @@ -618,8 +618,8 @@ views/table_create_alter.py:965-1005). - **Permission:** `view-database` (denied → `Forbidden` → 403 HTML). - **Unknown database** → 404; for `.json`: - `{"ok": false, "error": "Database not found"}`. (The existence check runs - before the permission check.) + `{"ok": false, "error": "Database not found"}`. The permission check runs + first, so unauthorized actors cannot probe for database existence. - **Responses:** `.json` → 200 `{"ok": true, "database": "", "schema": ""}` (concatenated `sqlite_master.sql` joined with `;\n`); `.md` → `text/markdown`; no extension → HTML. diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index 64b2acbc..a92fbfbd 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -288,11 +288,13 @@ Concerns: databases this leaks filesystem paths and database names. Filter it, or gate it behind `permissions-debug`.~~ ✅ **Done** — the endpoint now filters through `allowed_resources("view-database", actor)`. -- **(P2) `/db/-/schema` checks existence before permission** +- ~~**(P2) `/db/-/schema` checks existence before permission** (views/special.py:1308-1317): an actor without `view-database` can distinguish "database exists" (403) from "does not exist" (404). Standardize on permission-check-first (as the table view does) so - unauthorized actors get a uniform response. + unauthorized actors get a uniform response.~~ ✅ **Done** — permission is + checked first; the table schema view also now 404s (instead of a 500 + KeyError) for an unknown database. - **(P2) `/-/threads` exposes runtime internals** (thread idents, asyncio task reprs including file paths) behind only `view-instance`. Consider `permissions-debug`, alongside `/-/actions` which already requires it. diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index 29f4e8fc..55370c18 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -531,3 +531,42 @@ async def test_row_update_return_uses_rows_list(ds_error_shape): assert data["ok"] is True assert "row" not in data assert data["rows"] == [{"id": 1, "title": "Updated"}] + + +# Schema endpoints: no existence oracle, no 500 on unknown database + + +@pytest.mark.asyncio +async def test_schema_endpoints_no_existence_oracle(tmp_path_factory): + db_directory = tmp_path_factory.mktemp("dbs") + db_path = str(db_directory / "data.db") + conn = sqlite3.connect(db_path) + conn.execute("vacuum") + conn.execute("create table docs (id integer primary key)") + conn.close() + ds = Datasette([db_path], default_deny=True) + ds.root_enabled = True + try: + # An actor without view-database cannot distinguish an existing + # database from a missing one + denied_existing = await ds.client.get("/data/-/schema.json") + denied_missing = await ds.client.get("/nope/-/schema.json") + assert denied_existing.status_code == denied_missing.status_code == 403 + + # An authorized actor sees the real thing + root_existing = await ds.client.get( + "/data/-/schema.json", actor={"id": "root"} + ) + assert root_existing.status_code == 200 + root_missing = await ds.client.get( + "/nope/-/schema.json", actor={"id": "root"} + ) + assert root_missing.status_code == 404 + finally: + ds.close() + + +@pytest.mark.asyncio +async def test_table_schema_unknown_database_is_404_not_500(ds_client): + response = await ds_client.get("/no_such_db/some_table/-/schema.json") + assert_canonical_error(response, 404) From e0ba8b3c6a3d7aa2e4be9866a3dce712196f9a73 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 16:16:02 +0000 Subject: [PATCH 020/131] Return 400 for unknown _extra names on data formats Unknown ?_extra= names (including internal HTML-only extras such as display_rows) were silently ignored, so a typo returned the default payload with no signal. Table, row and query data formats now return 400 "Unknown _extra: ". HTML pages continue to ignore unknown names. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/extras.py | 13 +++++++++++++ datasette/views/database.py | 3 ++- datasette/views/row.py | 3 +++ datasette/views/table.py | 4 ++++ docs/json_api.rst | 2 ++ existing-api.md | 7 ++++--- stable-api-recommendations.md | 6 +++++- tests/test_error_shape.py | 34 ++++++++++++++++++++++++++++++++++ tests/test_table_api.py | 4 ++-- 9 files changed, 69 insertions(+), 7 deletions(-) diff --git a/datasette/extras.py b/datasette/extras.py index 36014185..fb8c2e06 100644 --- a/datasette/extras.py +++ b/datasette/extras.py @@ -5,6 +5,8 @@ from typing import ClassVar from asyncinject import Registry +from datasette.utils.asgi import BadRequest + def extra_names_from_request(request): extra_bits = request.args.getlist("_extra") @@ -113,6 +115,17 @@ class ExtraRegistry: self._allowed_names[key] = names return names + def validate_requested(self, requested, scope): + """ + Raise BadRequest if any requested extra name is not a public extra + for this scope. Used by data formats such as .json - HTML pages + silently ignore unknown names instead. + """ + allowed = self._allowed_names_for_scope(scope, include_internal=False) + unknown = sorted(name for name in requested if name not in allowed) + if unknown: + raise BadRequest("Unknown _extra: {}".format(", ".join(unknown))) + async def resolve(self, requested, context, scope, include_internal=False): allowed_names = self._allowed_names_for_scope(scope, include_internal) requested_names = [name for name in requested if name in allowed_names] diff --git a/datasette/views/database.py b/datasette/views/database.py index 99e85d2b..b7131a05 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -8,7 +8,7 @@ import markupsafe import os import textwrap -from datasette.extras import extra_names_from_request +from datasette.extras import extra_names_from_request, ExtraScope from datasette.database import QueryInterrupted from datasette.resources import DatabaseResource, QueryResource from datasette.stored_queries import StoredQuery, stored_query_to_dict @@ -855,6 +855,7 @@ class QueryView(View): raise DatasetteError("?sql= is required", status=400) data = {"ok": True, "rows": rows, "columns": columns} extras = extra_names_from_request(request) + table_extra_registry.validate_requested(extras, ExtraScope.QUERY) if extras: query_extra_context = QueryExtraContext( datasette=datasette, diff --git a/datasette/views/row.py b/datasette/views/row.py index 71539f34..296a0ac1 100644 --- a/datasette/views/row.py +++ b/datasette/views/row.py @@ -596,6 +596,9 @@ class RowView(BaseView): } extras = extra_names_from_request(request) + if request.url_vars.get("format"): + # Data formats reject unknown extras; HTML ignores them + table_extra_registry.validate_requested(extras, ExtraScope.ROW) # Process extras row_extra_context = RowExtraContext( diff --git a/datasette/views/table.py b/datasette/views/table.py index 44919281..ed3b1276 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -2254,6 +2254,10 @@ async def table_view_data( # Resolve extras extras = extra_names_from_request(request) + if not extra_extras: + # Data formats reject unknown extras; the HTML path (which passes + # extra_extras={"_html"}) resolves internal extras of its own + table_extra_registry.validate_requested(extras, ExtraScope.TABLE) if any(k for k in request.args.keys() if k == "_facet" or k.startswith("_facet_")): extras.add("facet_results") if request.args.get("_shape") == "object": diff --git a/docs/json_api.rst b/docs/json_api.rst index 2a859ba2..0069055b 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -283,6 +283,8 @@ These can be repeated or comma-separated: ?_extra=columns&_extra=count,next_url +Requesting an ``_extra`` name that does not exist returns a ``400`` error in the :ref:`standard error format `, for example ``{"ok": false, "error": "Unknown _extra: nope", ...}``. + .. [[[cog from json_api_doc import table_extras table_extras(cog) diff --git a/existing-api.md b/existing-api.md index 6ce29bf5..cf456921 100644 --- a/existing-api.md +++ b/existing-api.md @@ -176,9 +176,10 @@ build JSON directly): Table, row and query JSON responses support `?_extra=` (repeatable and/or comma-separated, extras.py:9-14) to add keys to the response. Extras are scope-registered (`ExtraScope.TABLE` / `ROW` / `QUERY`) and only **public** -extras are available over JSON (extras.py:73-92). Unknown extra names are -silently ignored. The available names per scope are listed with the relevant -endpoints below. +extras are available over JSON (extras.py:73-92). Unknown extra names (and +internal HTML-only names) on data formats return 400 +`Unknown _extra: `; HTML pages ignore them. The available names per +scope are listed with the relevant endpoints below. --- diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index a92fbfbd..1bc2412e 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -166,7 +166,11 @@ Endpoints disagree about the success envelope: matches the read API).~~ ✅ **Done** — row update now returns `rows: [{...}]`. -### 2b. `_extra`/`_shape` support is uneven (P2) +### 2b. `_extra`/`_shape` support is uneven (P2) — partially implemented + +> **Status:** unknown `_extra` names on data formats now return 400 +> `Unknown _extra: ` (HTML pages still ignore them). Extending +> extras/shaping to database/instance scope remains open. The extras system (`?_extra=`, scope-registered) is the 1.0 mechanism for response shaping — but it only exists on table, row and query endpoints. The diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index 55370c18..f0ff0a42 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -570,3 +570,37 @@ async def test_schema_endpoints_no_existence_oracle(tmp_path_factory): async def test_table_schema_unknown_database_is_404_not_500(ds_client): response = await ds_client.get("/no_such_db/some_table/-/schema.json") assert_canonical_error(response, 404) + + +# Unknown _extra names are a 400, not silently ignored + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "path", + ( + "/fixtures/facetable.json?_extra=nope", + "/fixtures/facetable.json?_extra=count,nope", + "/fixtures/simple_primary_key/1.json?_extra=nope", + "/fixtures/-/query.json?sql=select+1&_extra=nope", + ), +) +async def test_unknown_extra_is_400(ds_client, path): + response = await ds_client.get(path) + data = assert_canonical_error(response, 400) + assert data["errors"] == ["Unknown _extra: nope"] + + +@pytest.mark.asyncio +async def test_html_only_extra_via_json_is_400(ds_client): + # display_rows exists for the HTML view but is not part of the JSON API + response = await ds_client.get("/fixtures/facetable.json?_extra=display_rows") + data = assert_canonical_error(response, 400) + assert data["errors"] == ["Unknown _extra: display_rows"] + + +@pytest.mark.asyncio +async def test_unknown_extra_ignored_on_html_pages(ds_client): + response = await ds_client.get("/fixtures/facetable?_extra=nope") + assert response.status_code == 200 + assert response.headers["content-type"].startswith("text/html") diff --git a/tests/test_table_api.py b/tests/test_table_api.py index e3737ee6..ff413be8 100644 --- a/tests/test_table_api.py +++ b/tests/test_table_api.py @@ -123,8 +123,8 @@ async def test_html_only_extras_are_not_available_via_json(ds_client, extra): # These extras exist for the HTML view; their values are not JSON # serializable so they are internal, not part of the JSON API response = await ds_client.get(f"/fixtures/facetable.json?_extra={extra}") - assert response.status_code == 200 - assert extra not in response.json() + assert response.status_code == 400 + assert response.json()["errors"] == [f"Unknown _extra: {extra}"] @pytest.mark.asyncio From 0bf3a54716a18521332aa55dac6a3e420851c82a Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 16:19:03 +0000 Subject: [PATCH 021/131] Include next_url in default table JSON keys Table JSON responses previously only included the next pagination token by default - the ready-to-follow next_url required ?_extra=next_url. Both keys are now always present (null on the final page), which the pagination documentation already claimed. The next_url extra remains valid for backwards compatibility. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/views/table.py | 1 + docs/json_api.rst | 3 ++- existing-api.md | 1 + stable-api-recommendations.md | 9 ++++++++- tests/test_table_api.py | 21 +++++++++++++++++++++ 5 files changed, 33 insertions(+), 2 deletions(-) diff --git a/datasette/views/table.py b/datasette/views/table.py index ed3b1276..69d5a075 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -2318,6 +2318,7 @@ async def table_view_data( data = { "ok": True, "next": next_value and str(next_value) or None, + "next_url": next_url, } data.update( await resolve_table_extras( diff --git a/docs/json_api.rst b/docs/json_api.rst index 0069055b..dc611a39 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -48,7 +48,7 @@ The ``"rows"`` key is a list of objects, each one representing a row. The ``"truncated"`` key lets you know if the query was truncated. This can happen if a SQL query returns more than 1,000 results (or the :ref:`setting_max_returned_rows` setting). -For table pages, an additional key ``"next"`` may be present. This indicates that the next page in the pagination set can be retrieved using ``?_next=VALUE``. +For table pages, two additional keys are present: ``"next"``, an opaque token that can be used to retrieve the next page using ``?_next=TOKEN``, and ``"next_url"``, the full URL of that next page. Both are ``null`` on the final page. See :ref:`json_api_pagination`. .. _json_api_errors: @@ -128,6 +128,7 @@ options: { "ok": true, "next": null, + "next_url": null, "rows": [ [3, "Detroit"], [2, "Los Angeles"], diff --git a/existing-api.md b/existing-api.md index cf456921..f7f161f3 100644 --- a/existing-api.md +++ b/existing-api.md @@ -647,6 +647,7 @@ dispatched to `QueryView` (views/table.py:1703-1712). |---|---| | `ok` | `true` when data was retrieved without error | | `next` | pagination token string, or `null` on the last page | +| `next_url` | absolute URL of the next page, or `null` on the last page | | `rows` | list of row objects `{column: value}` (default `_shape=objects`) | | `truncated` | always present; `false` for table pages | diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index 1bc2412e..d3d5b6a6 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -196,7 +196,14 @@ number. --- -## 3. Pagination: three mechanisms, two contracts (P2) +## 3. Pagination: three mechanisms, two contracts (P2) — partially implemented + +> **Status:** `next_url` now accompanies `next` in the default table JSON +> keys (previously it required `?_extra=next_url`), so every response with +> a `next` token also carries the ready-to-follow URL. Pagination tokens +> are deliberately left undocumented as to their internal structure. The +> `_size`/`page_size` naming and `has_more`/`total` differences remain +> open. | Endpoint | Mechanism | Token | Extras | |---|---|---|---| diff --git a/tests/test_table_api.py b/tests/test_table_api.py index ff413be8..0a593ff8 100644 --- a/tests/test_table_api.py +++ b/tests/test_table_api.py @@ -1491,6 +1491,7 @@ async def test_col_nocol_errors(ds_client, path, expected_error): { "ok": True, "next": None, + "next_url": None, "columns": ["id", "content", "content2"], "rows": [{"id": "1", "content": "hey", "content2": "world"}], "truncated": False, @@ -1501,6 +1502,7 @@ async def test_col_nocol_errors(ds_client, path, expected_error): { "ok": True, "next": None, + "next_url": None, "rows": [{"id": "1", "content": "hey", "content2": "world"}], "truncated": False, "count": 1, @@ -1627,3 +1629,22 @@ async def test_count_truncated_included_with_count_extra(tmp_path_factory): assert response.json()["count_truncated"] is True finally: ds.close() + + +@pytest.mark.asyncio +async def test_next_url_included_by_default(ds_client): + response = await ds_client.get("/fixtures/compound_three_primary_keys.json") + data = response.json() + assert data["next"] is not None + assert data["next_url"].endswith( + "/fixtures/compound_three_primary_keys.json?_next=" + + urllib.parse.quote(data["next"], safe="") + ) + # Follow to the last page - next and next_url are both null there + while data["next"]: + response = await ds_client.get( + "/fixtures/compound_three_primary_keys.json?_next=" + data["next"] + ) + data = response.json() + assert data["next"] is None + assert data["next_url"] is None From e5e9aca871bed96398d4702b7ae25abcfbb96ac8 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 16:22:31 +0000 Subject: [PATCH 022/131] Require permissions-debug for /-/threads /-/threads exposes runtime internals - thread idents and asyncio task reprs including file paths - but only required view-instance. It now requires permissions-debug, like /-/actions. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/app.py | 4 +++- docs/introspection.rst | 2 +- existing-api.md | 2 +- stable-api-recommendations.md | 5 +++-- tests/test_api.py | 6 +++++- tests/test_error_shape.py | 20 ++++++++++++++------ tests/test_internals_datasette.py | 3 ++- tests/test_success_envelope.py | 2 +- 8 files changed, 30 insertions(+), 14 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 9982c58e..daa3848b 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -2584,7 +2584,9 @@ class Datasette: r"/-/config(\.(?Pjson))?$", ) add_route( - JsonDataView.as_view(self, "threads.json", self._threads), + JsonDataView.as_view( + self, "threads.json", self._threads, permission="permissions-debug" + ), r"/-/threads(\.(?Pjson))?$", ) add_route( diff --git a/docs/introspection.rst b/docs/introspection.rst index 37f258d7..2d13f68b 100644 --- a/docs/introspection.rst +++ b/docs/introspection.rst @@ -252,7 +252,7 @@ Without those query string arguments, the page lists up to five tables with dete /-/threads ---------- -Shows details of threads and ``asyncio`` tasks. `Threads example `_: +Shows details of threads and ``asyncio`` tasks. This endpoint requires the ``permissions-debug`` permission, since it exposes runtime internals. `Threads example `_: .. code-block:: json diff --git a/existing-api.md b/existing-api.md index f7f161f3..af7ca3b9 100644 --- a/existing-api.md +++ b/existing-api.md @@ -249,7 +249,7 @@ value replaced by `"***"` (utils/__init__.py:1532-1556). ### GET /-/threads(.json) app.py:2566-2569, `Datasette._threads` (app.py:2268-2285). Permission -`view-instance`. No parameters. +**`permissions-debug`** (exposes runtime internals). No parameters. Response: `num_threads`, `threads` (list of `{name, ident, daemon}`), `num_tasks`, `tasks` (asyncio task repr strings). When the diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index d3d5b6a6..3c216c04 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -306,9 +306,10 @@ Concerns: unauthorized actors get a uniform response.~~ ✅ **Done** — permission is checked first; the table schema view also now 404s (instead of a 500 KeyError) for an unknown database. -- **(P2) `/-/threads` exposes runtime internals** (thread idents, asyncio +- ~~**(P2) `/-/threads` exposes runtime internals** (thread idents, asyncio task reprs including file paths) behind only `view-instance`. Consider - `permissions-debug`, alongside `/-/actions` which already requires it. + `permissions-debug`, alongside `/-/actions` which already requires it.~~ + ✅ **Done** — `/-/threads` now requires `permissions-debug`. - **(P3) `/-/config` redaction is substring-based** on six key names (app.py:2502-2505); plugins storing secrets under other names leak. Worth a note in plugin authoring docs plus a `redact_keys` plugin hook. diff --git a/tests/test_api.py b/tests/test_api.py index b035edb9..7a575144 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -525,7 +525,11 @@ def test_databases_json(app_client_two_attached_databases_one_immutable): @pytest.mark.asyncio async def test_threads_json(ds_client): - response = await ds_client.get("/-/threads.json") + ds_client.ds.root_enabled = True + try: + response = await ds_client.get("/-/threads.json", actor={"id": "root"}) + finally: + ds_client.ds.root_enabled = False expected_keys = {"ok", "threads", "num_threads"} if sys.version_info >= (3, 7, 0): expected_keys.update({"tasks", "num_tasks"}) diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index f0ff0a42..a34fbed7 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -554,13 +554,9 @@ async def test_schema_endpoints_no_existence_oracle(tmp_path_factory): assert denied_existing.status_code == denied_missing.status_code == 403 # An authorized actor sees the real thing - root_existing = await ds.client.get( - "/data/-/schema.json", actor={"id": "root"} - ) + root_existing = await ds.client.get("/data/-/schema.json", actor={"id": "root"}) assert root_existing.status_code == 200 - root_missing = await ds.client.get( - "/nope/-/schema.json", actor={"id": "root"} - ) + root_missing = await ds.client.get("/nope/-/schema.json", actor={"id": "root"}) assert root_missing.status_code == 404 finally: ds.close() @@ -604,3 +600,15 @@ async def test_unknown_extra_ignored_on_html_pages(ds_client): response = await ds_client.get("/fixtures/facetable?_extra=nope") assert response.status_code == 200 assert response.headers["content-type"].startswith("text/html") + + +# /-/threads exposes runtime internals and requires permissions-debug + + +@pytest.mark.asyncio +async def test_threads_requires_permissions_debug(ds_error_shape): + denied = await ds_error_shape.client.get("/-/threads.json") + assert_canonical_error(denied, 403) + allowed = await ds_error_shape.client.get("/-/threads.json", actor={"id": "root"}) + assert allowed.status_code == 200 + assert allowed.json()["ok"] is True diff --git a/tests/test_internals_datasette.py b/tests/test_internals_datasette.py index 2af069c9..85598c05 100644 --- a/tests/test_internals_datasette.py +++ b/tests/test_internals_datasette.py @@ -184,10 +184,11 @@ async def test_datasette_constructor(): @pytest.mark.asyncio async def test_num_sql_threads_zero(): ds = Datasette([], memory=True, settings={"num_sql_threads": 0}) + ds.root_enabled = True db = ds.add_database(Database(ds, memory_name="test_num_sql_threads_zero")) await db.execute_write("create table t(id integer primary key)") await db.execute_write("insert into t (id) values (1)") - response = await ds.client.get("/-/threads.json") + response = await ds.client.get("/-/threads.json", actor={"id": "root"}) assert response.json() == {"ok": True, "num_threads": 0, "threads": []} response2 = await ds.client.get("/test_num_sql_threads_zero/t.json?_shape=array") assert response2.json() == [{"id": 1}] diff --git a/tests/test_success_envelope.py b/tests/test_success_envelope.py index 22ce6580..97cfd72a 100644 --- a/tests/test_success_envelope.py +++ b/tests/test_success_envelope.py @@ -34,7 +34,6 @@ def ds_envelope(tmp_path_factory): "/-/versions.json", "/-/settings.json", "/-/config.json", - "/-/threads.json", "/-/actor.json", "/-/jump.json", "/-/schema.json", @@ -58,6 +57,7 @@ async def test_success_object_has_ok_true(ds_client, path): ( "/-/rules.json?action=view-instance", "/-/check.json?action=view-instance", + "/-/threads.json", ), ) async def test_permission_debug_success_has_ok_true(ds_envelope, path): From 13dc7a08b79c7b62e6eb2c54befa5e6c02915bd3 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 16:30:04 +0000 Subject: [PATCH 023/131] Fix foreign key label test expectations for default next_url key Follow-up to the commit adding next_url to the default table JSON keys. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- tests/test_table_html.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/test_table_html.py b/tests/test_table_html.py index 86fb4254..87c48b6b 100644 --- a/tests/test_table_html.py +++ b/tests/test_table_html.py @@ -2313,6 +2313,7 @@ async def test_foreign_key_labels_obey_permissions(config): assert root_b.json() == { "ok": True, "next": None, + "next_url": None, "rows": [{"id": 1, "name": "world", "a_id": {"value": 1, "label": "hello"}}], "truncated": False, } @@ -2320,6 +2321,7 @@ async def test_foreign_key_labels_obey_permissions(config): assert anon_b.json() == { "ok": True, "next": None, + "next_url": None, "rows": [{"id": 1, "name": "world", "a_id": 1}], "truncated": False, } From 5cb2bc6909be9a9f9cbd23e7f19f142091cbb5af Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 16:45:19 +0000 Subject: [PATCH 024/131] Homepage JSON returns databases as a list /.json previously returned databases as an object keyed by database name, unlike /-/databases.json and every other collection in the API. It now returns a list of database objects. The HTML template already consumed the list. This endpoint remains undocumented. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/views/index.py | 2 +- existing-api.md | 4 ++-- stable-api-recommendations.md | 6 ++++-- tests/test_api.py | 18 ++++++++++-------- 4 files changed, 17 insertions(+), 13 deletions(-) diff --git a/datasette/views/index.py b/datasette/views/index.py index 86f8ae7d..8c5534cb 100644 --- a/datasette/views/index.py +++ b/datasette/views/index.py @@ -152,7 +152,7 @@ class IndexView(BaseView): json.dumps( { "ok": True, - "databases": {db["name"]: db for db in databases}, + "databases": databases, "metadata": await self.ds.get_instance_metadata(), }, cls=CustomJSONEncoder, diff --git a/existing-api.md b/existing-api.md index af7ca3b9..8d390f76 100644 --- a/existing-api.md +++ b/existing-api.md @@ -201,8 +201,8 @@ Routes: `/(\.(?Pjsono?))?$` and `/-/(\.(?Pjsono?))?$` - **Parameters:** `_sort=relationships` sorts each database's truncated table list by foreign-key relationship count. - **JSON response** (index.py:147-161) — includes `ok: true` plus: - - `databases` — an **object keyed by database name** (not a list). Each - value: `name`, `hash` (or null), `color`, `path`, + - `databases` — a **list** of database objects (undocumented API, subject + to change). Each item: `name`, `hash` (or null), `color`, `path`, `tables_and_views_truncated` (up to 5 items: `name`, `columns`, `primary_keys`, `count` (int or null), `hidden`, `fts_table`, `num_relationships_for_sorting`, `private`; view items are just diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index 3c216c04..d64b3fb8 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -156,10 +156,12 @@ Endpoints disagree about the success envelope: ### 2a. Collection representations disagree (P2) -- Homepage `/.json` returns `databases` as an **object keyed by name** +- ~~Homepage `/.json` returns `databases` as an **object keyed by name** (index.py:147-161); `/-/databases.json` returns an **array**; the database page returns `tables` as an array. Choose arrays-of-objects everywhere - (objects-keyed-by-name break when names need ordering or pagination). + (objects-keyed-by-name break when names need ordering or pagination).~~ + ✅ **Done** — the homepage returns a list, matching `/-/databases.json`. + The homepage JSON remains deliberately undocumented. - ~~Insert/upsert with `return: true` respond with `rows` (plural, list); row update with `return: true` responds with `row` (singular, object) (views/row.py:837-844). Pick one (`rows` everywhere, even for one row, diff --git a/tests/test_api.py b/tests/test_api.py index 7a575144..17edc0e5 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -26,8 +26,9 @@ async def test_homepage(ds_client): "title", ] databases = data.get("databases") - assert databases.keys() == {"fixtures": 0}.keys() - d = databases["fixtures"] + assert isinstance(databases, list) + assert [d["name"] for d in databases] == ["fixtures"] + d = databases[0] assert d["name"] == "fixtures" assert isinstance(d["tables_count"], int) assert isinstance(len(d["tables_and_views_truncated"]), int) @@ -43,7 +44,7 @@ async def test_homepage_sort_by_relationships(ds_client): assert response.status_code == 200 tables = [ t["name"] - for t in response.json()["databases"]["fixtures"]["tables_and_views_truncated"] + for t in response.json()["databases"][0]["tables_and_views_truncated"] ] assert tables == [ "simple_primary_key", @@ -251,8 +252,8 @@ def test_no_files_uses_memory_database(app_client_no_files): assert response.status == 200 assert { "ok": True, - "databases": { - "_memory": { + "databases": [ + { "name": "_memory", "hash": None, "color": "a6c7b9", @@ -267,7 +268,7 @@ def test_no_files_uses_memory_database(app_client_no_files): "views_count": 0, "private": False, }, - }, + ], "metadata": {}, } == response.json # Try that SQL query @@ -852,8 +853,9 @@ async def test_tilde_encoded_database_names(db_name): ds = Datasette() ds.add_memory_database(db_name) response = await ds.client.get("/.json") - assert db_name in response.json()["databases"].keys() - path = response.json()["databases"][db_name]["path"] + databases_by_name = {d["name"]: d for d in response.json()["databases"]} + assert db_name in databases_by_name + path = databases_by_name[db_name]["path"] # And the JSON for that database response2 = await ds.client.get(path + ".json") assert response2.status_code == 200 From afa7b1ba0db8e5084889a15261559bcf21435171 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 17:01:53 +0000 Subject: [PATCH 025/131] Add unstable marker to undocumented JSON endpoints JSON endpoints that are not part of the documented API now include "unstable": "This API is not part of Datasette's stable interface and may change at any time" in their responses, making the stability tier machine-readable. Applied to the homepage (/.json and /-/.json), /db/-/queries/analyze, POST /db/-/queries/store, /db//-/definition, /db/-/query/parameters, /db/-/execute-write/analyze and the POST /-/permissions playground response. The message lives in datasette.utils.UNSTABLE_API_MESSAGE. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/utils/__init__.py | 5 +++ datasette/views/execute_write.py | 10 ++--- datasette/views/index.py | 2 + datasette/views/special.py | 7 ++++ datasette/views/stored_queries.py | 35 +++++++++++++----- existing-api.md | 14 ++++++- stable-api-recommendations.md | 8 +++- tests/test_api.py | 7 +++- tests/test_permissions.py | 4 ++ tests/test_queries.py | 9 ++++- tests/test_success_envelope.py | 61 +++++++++++++++++++++++++++++++ 11 files changed, 142 insertions(+), 20 deletions(-) diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index 17c8702f..1d921f02 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -1294,6 +1294,11 @@ async def derive_named_parameters(db: "Database", sql: str) -> List[str]: return named_parameters(sql) +UNSTABLE_API_MESSAGE = ( + "This API is not part of Datasette's stable interface and may change at any time" +) + + def error_body(messages, status): """ The canonical JSON error body used by every Datasette JSON error response: diff --git a/datasette/views/execute_write.py b/datasette/views/execute_write.py index b7e8288e..6806e69d 100644 --- a/datasette/views/execute_write.py +++ b/datasette/views/execute_write.py @@ -2,7 +2,7 @@ import re from urllib.parse import urlencode from datasette.resources import DatabaseResource -from datasette.utils import sqlite3 +from datasette.utils import UNSTABLE_API_MESSAGE, sqlite3 from datasette.utils.asgi import Response from .base import BaseView, _error @@ -500,8 +500,6 @@ class ExecuteWriteAnalyzeView(BaseView): ) ) sql = request.args.get("sql") or "" - return _block_framing( - Response.json( - await _execute_write_analysis_data(self.ds, db, sql, request.actor) - ) - ) + analysis = await _execute_write_analysis_data(self.ds, db, sql, request.actor) + analysis["unstable"] = UNSTABLE_API_MESSAGE + return _block_framing(Response.json(analysis)) diff --git a/datasette/views/index.py b/datasette/views/index.py index 8c5534cb..67296cd1 100644 --- a/datasette/views/index.py +++ b/datasette/views/index.py @@ -6,6 +6,7 @@ from datasette.utils import ( await_me_maybe, make_slot_function, CustomJSONEncoder, + UNSTABLE_API_MESSAGE, ) from datasette.utils.asgi import Response from datasette.version import __version__ @@ -152,6 +153,7 @@ class IndexView(BaseView): json.dumps( { "ok": True, + "unstable": UNSTABLE_API_MESSAGE, "databases": databases, "metadata": await self.ds.get_instance_metadata(), }, diff --git a/datasette/views/special.py b/datasette/views/special.py index 6afb6437..28d4d6a1 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -6,6 +6,7 @@ from datasette.events import LogoutEvent, LoginEvent, CreateTokenEvent from datasette.resources import DatabaseResource, TableResource from datasette.utils.asgi import Response, Forbidden from datasette.utils import ( + UNSTABLE_API_MESSAGE, actor_matches_allow, add_cors_headers, await_me_maybe, @@ -295,6 +296,12 @@ class PermissionsDebugView(BaseView): response, status = await _check_permission_for_actor( self.ds, permission, parent, child, actor ) + if response.get("ok"): + response = { + "ok": True, + "unstable": UNSTABLE_API_MESSAGE, + **response, + } return Response.json(response, status=status) diff --git a/datasette/views/stored_queries.py b/datasette/views/stored_queries.py index d1f151dd..ea49d983 100644 --- a/datasette/views/stored_queries.py +++ b/datasette/views/stored_queries.py @@ -2,7 +2,7 @@ from urllib.parse import parse_qsl, urlencode from datasette.resources import DatabaseResource, QueryResource from datasette.stored_queries import stored_query_to_dict -from datasette.utils import sqlite3, tilde_decode +from datasette.utils import UNSTABLE_API_MESSAGE, sqlite3, tilde_decode from datasette.utils.asgi import Response from .base import BaseView, _error @@ -48,7 +48,15 @@ class QueryParametersView(BaseView): parameters = _derived_query_parameters(request.args.get("sql") or "") except QueryValidationError as ex: return _block_framing(_error([ex.message], ex.status)) - return _block_framing(Response.json({"ok": True, "parameters": parameters})) + return _block_framing( + Response.json( + { + "ok": True, + "unstable": UNSTABLE_API_MESSAGE, + "parameters": parameters, + } + ) + ) def _query_list_url(path, query_string, *, set_args=None, remove_args=None): @@ -315,11 +323,9 @@ class QueryCreateAnalyzeView(BaseView): ) ) sql = request.args.get("sql") or "" - return _block_framing( - Response.json( - await _query_create_analysis_data(self.ds, db, sql, request.actor) - ) - ) + analysis = await _query_create_analysis_data(self.ds, db, sql, request.actor) + analysis["unstable"] = UNSTABLE_API_MESSAGE + return _block_framing(Response.json(analysis)) class QueryStoreView(QueryCreateView): @@ -384,7 +390,12 @@ class QueryStoreView(QueryCreateView): assert query is not None if is_json: return Response.json( - {"ok": True, "query": stored_query_to_dict(query)}, status=201 + { + "ok": True, + "unstable": UNSTABLE_API_MESSAGE, + "query": stored_query_to_dict(query), + }, + status=201, ) self.ds.add_message(request, "Query saved", self.ds.INFO) return Response.redirect(self.ds.urls.path(self.ds.urls.table(db.name, name))) @@ -405,7 +416,13 @@ class QueryDefinitionView(BaseView): actor=request.actor, ): return _error(["Permission denied"], 403) - return Response.json({"ok": True, "query": stored_query_to_dict(query)}) + return Response.json( + { + "ok": True, + "unstable": UNSTABLE_API_MESSAGE, + "query": stored_query_to_dict(query), + } + ) class QueryUpdateView(BaseView): diff --git a/existing-api.md b/existing-api.md index 8d390f76..2f9996dc 100644 --- a/existing-api.md +++ b/existing-api.md @@ -43,7 +43,7 @@ directory: every claim below is based on the route table in `datasette/app.py` - Success content type: `application/json; charset=utf-8` (`_shape=array&_nl=on` responses use `text/plain`). -### Success envelope +### Success envelope and stability marker Every JSON endpoint that returns an object includes `"ok": true` on success. `JsonDataView` injects it automatically for dict responses @@ -52,6 +52,18 @@ autocomplete views add it explicitly. The former top-level-array endpoints (`/-/plugins`, `/-/databases`, `/-/actions`) now return objects wrapping their arrays (`{"ok": true, "plugins": [...]}` etc.). +JSON endpoints that are **not part of the documented API** include a +marker key (`UNSTABLE_API_MESSAGE` in utils/__init__.py): + +```json +"unstable": "This API is not part of Datasette's stable interface and may change at any time" +``` + +Currently: the homepage (`/.json`, `/-/.json`), `/db/-/queries/analyze`, +`POST /db/-/queries/store`, `/db//-/definition`, +`/db/-/query/parameters`, `/db/-/execute-write/analyze` and the +`POST /-/permissions` playground response. + ### Error shape (canonical) Every JSON error response uses one canonical shape, built by `error_body()` diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index d64b3fb8..baac80f7 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -378,7 +378,13 @@ Concerns: --- -## 9. Define stability tiers explicitly (P1 — documentation, not code) +## 9. Define stability tiers explicitly (P1 — documentation, not code) — partially implemented + +> **Status:** undocumented JSON endpoints now self-describe with an +> `"unstable": "This API is not part of Datasette's stable interface and +> may change at any time"` key (homepage, queries analyze/store/definition, +> query parameters, execute-write analyze, permissions playground POST). +> The written tier documentation remains to be done. Not everything under `/-/` can or should carry a 1.0 guarantee. Recommend shipping 1.0 with an explicit three-tier contract, per endpoint: diff --git a/tests/test_api.py b/tests/test_api.py index 17edc0e5..8ce38fad 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -43,8 +43,7 @@ async def test_homepage_sort_by_relationships(ds_client): response = await ds_client.get("/.json?_sort=relationships") assert response.status_code == 200 tables = [ - t["name"] - for t in response.json()["databases"][0]["tables_and_views_truncated"] + t["name"] for t in response.json()["databases"][0]["tables_and_views_truncated"] ] assert tables == [ "simple_primary_key", @@ -252,6 +251,10 @@ def test_no_files_uses_memory_database(app_client_no_files): assert response.status == 200 assert { "ok": True, + "unstable": ( + "This API is not part of Datasette's stable interface" + " and may change at any time" + ), "databases": [ { "name": "_memory", diff --git a/tests/test_permissions.py b/tests/test_permissions.py index 32606789..f8d2c808 100644 --- a/tests/test_permissions.py +++ b/tests/test_permissions.py @@ -740,6 +740,10 @@ async def test_actor_restricted_permissions( } expected = { "ok": True, + "unstable": ( + "This API is not part of Datasette's stable interface" + " and may change at any time" + ), "action": permission, "allowed": expected_result, "resource": expected_resource, diff --git a/tests/test_queries.py b/tests/test_queries.py index 11828c4e..7f29a3fb 100644 --- a/tests/test_queries.py +++ b/tests/test_queries.py @@ -2140,7 +2140,14 @@ async def test_query_parameters_endpoint_uses_get_sql_only(): ) assert response.status_code == 200 - assert response.json() == {"ok": True, "parameters": ["name", "id"]} + assert response.json() == { + "ok": True, + "unstable": "{}".format( + "This API is not part of Datasette's stable interface" + " and may change at any time" + ), + "parameters": ["name", "id"], + } assert permission_denied_response.status_code == 403 assert permission_denied_response.json()["errors"] == [ "Permission denied: need execute-sql" diff --git a/tests/test_success_envelope.py b/tests/test_success_envelope.py index 97cfd72a..68b042e5 100644 --- a/tests/test_success_envelope.py +++ b/tests/test_success_envelope.py @@ -112,3 +112,64 @@ async def test_actions_json_is_object(ds_envelope): assert data["ok"] is True assert isinstance(data["actions"], list) assert "view-instance" in {action["name"] for action in data["actions"]} + + +UNSTABLE_MESSAGE = ( + "This API is not part of Datasette's stable interface and may change at any time" +) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "path", + ( + "/.json", + "/-/.json", + "/fixtures/-/queries/analyze?sql=select+1", + "/fixtures/-/query/parameters?sql=select+:name", + "/fixtures/-/execute-write/analyze?sql=delete+from+facetable", + ), +) +async def test_undocumented_endpoints_report_unstable(ds_client, path): + ds_client.ds.root_enabled = True + try: + response = await ds_client.get(path, actor={"id": "root"}) + finally: + ds_client.ds.root_enabled = False + assert response.status_code == 200 + assert response.json()["unstable"] == UNSTABLE_MESSAGE + + +@pytest.mark.asyncio +async def test_query_store_and_definition_report_unstable(ds_envelope): + store = await ds_envelope.client.post( + "/data/-/queries/store", + json={"query": {"name": "unstable_check", "sql": "select 1"}}, + actor={"id": "root"}, + ) + assert store.status_code == 201 + assert store.json()["unstable"] == UNSTABLE_MESSAGE + definition = await ds_envelope.client.get( + "/data/unstable_check/-/definition", actor={"id": "root"} + ) + assert definition.status_code == 200 + assert definition.json()["unstable"] == UNSTABLE_MESSAGE + + +@pytest.mark.asyncio +async def test_permissions_post_reports_unstable(ds_envelope): + response = await ds_envelope.client.post( + "/-/permissions", + data={"actor": '{"id": "root"}', "permission": "view-instance"}, + actor={"id": "root"}, + ) + assert response.status_code == 200 + assert response.json()["unstable"] == UNSTABLE_MESSAGE + + +@pytest.mark.asyncio +async def test_documented_endpoints_do_not_report_unstable(ds_client): + for path in ("/-/versions.json", "/fixtures.json", "/fixtures/facetable.json"): + response = await ds_client.get(path) + assert response.status_code == 200 + assert "unstable" not in response.json() From 404ee4c3a79c213d47e263e796364f8b471bde05 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 17:07:04 +0000 Subject: [PATCH 026/131] Document the JSON API stability promise docs/json_api.rst now opens with an API stability section declaring what the 1.x promise covers: documented endpoints, parameters and response keys are stable with additive-only changes; pagination tokens are opaque strings; the error format and token restriction semantics are stable. It lists the exempt tiers: endpoints carrying the "unstable" marker key, debug and support endpoints (/-/threads, /-/actions, /-/jump, the permission debug endpoints, table autocomplete), and keys explicitly labeled unstable such as the execute-write analysis block. Cross-referenced from the introspection and permission-debug documentation. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- docs/authentication.rst | 2 ++ docs/introspection.rst | 2 ++ docs/json_api.rst | 49 ++++++++++++++++++++++++++++++++++- stable-api-recommendations.md | 20 ++++++++------ 4 files changed, 64 insertions(+), 9 deletions(-) diff --git a/docs/authentication.rst b/docs/authentication.rst index 72ac5fa6..7d4d019d 100644 --- a/docs/authentication.rst +++ b/docs/authentication.rst @@ -1151,6 +1151,8 @@ It also provides an interface for running hypothetical permission checks against This is designed to help administrators and plugin authors understand exactly how permission checks are being carried out, in order to effectively configure Datasette's permission system. +These debug endpoints are exempt from the :ref:`JSON API stability promise ` - their JSON shapes may change in future releases. + .. _AllowedResourcesView: Allowed resources view diff --git a/docs/introspection.rst b/docs/introspection.rst index 2d13f68b..ea94b70d 100644 --- a/docs/introspection.rst +++ b/docs/introspection.rst @@ -9,6 +9,8 @@ Each of these pages can be viewed in your browser. Add ``.json`` to the URL to g JSON responses that return an object include an ``"ok": true`` key, consistent with the rest of the :ref:`JSON API `. +The introspection endpoints documented on this page are covered by the :ref:`JSON API stability promise `, with the exception of the debug endpoints ``/-/threads``, ``/-/actions`` and ``/-/jump``, whose shapes may change in future releases. + .. _JsonDataView_metadata: /-/metadata diff --git a/docs/json_api.rst b/docs/json_api.rst index dc611a39..4b1f26d2 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -9,6 +9,53 @@ through the Datasette user interface can also be accessed as JSON via the API. To access the API for a page, either click on the ``.json`` link on that page or edit the URL and add a ``.json`` extension to it. +.. _json_api_stability: + +API stability +------------- + +Datasette 1.0 makes a stability promise for its JSON API: the endpoints, +parameters and response keys documented here and on the pages this +documentation links to will not change in backwards-incompatible ways for +the duration of the 1.x release series. + +Stability means: + +- Documented endpoints will keep their URLs, methods, parameters and + permission requirements. +- Documented response keys will keep their names and types. New keys may be + **added** in any release - clients should ignore keys they do not + recognize. +- The documented ``?_extra=`` names, ``?_shape=`` values and + :ref:`column filter operators ` are stable. +- Pagination tokens - the ``"next"`` key and ``?_next=`` parameter - are + **opaque strings**. Pass them back exactly as you received them; their + internal structure is not part of the API and can change at any time. +- The :ref:`standard error format ` and the + :ref:`API token format and restriction semantics ` are + stable, including the action abbreviations stored inside signed tokens. + +Some JSON endpoints are **exempt** from this promise: + +- Endpoints that are not documented include this marker key in their + responses and can change at any time:: + + "unstable": "This API is not part of Datasette's stable interface and may change at any time" + + This currently covers the instance homepage (``/.json``), the stored + query ``analyze``/``store``/``definition`` endpoints, ``/-/query/parameters``, + ``/-/execute-write/analyze`` and the JSON returned by the ``/-/permissions`` + debug playground. +- Debug and support endpoints are documented so you can use them, but their + JSON shapes are not frozen: :ref:`/-/threads `, + :ref:`/-/actions `, :ref:`/-/jump `, + the :ref:`permission debug endpoints ` + (``/-/allowed``, ``/-/rules``, ``/-/check``) and the + :ref:`table autocomplete endpoint `. +- Response keys explicitly labeled as unstable in this documentation, such + as the ``"analysis"`` block returned by :ref:`execute-write ` + and the ``debug`` and ``request`` extras. + .. _json_api_default: Default representation @@ -1612,7 +1659,7 @@ Unsupported SQL operations are rejected by default. ``VACUUM`` is not allowed in A successful response includes a message, the SQLite ``rowcount``, a ``"rows"`` list, a ``"truncated"`` flag and a summary of the operations that were executed: -The shape of the ``"analysis"`` block is not yet considered a stable API and may change in future Datasette releases. +The shape of the ``"analysis"`` block is not part of the :ref:`stable API ` and may change in future Datasette releases. .. code-block:: json diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index baac80f7..b86d8228 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -378,13 +378,17 @@ Concerns: --- -## 9. Define stability tiers explicitly (P1 — documentation, not code) — partially implemented +## 9. Define stability tiers explicitly (P1 — documentation, not code) — ✅ IMPLEMENTED -> **Status:** undocumented JSON endpoints now self-describe with an -> `"unstable": "This API is not part of Datasette's stable interface and -> may change at any time"` key (homepage, queries analyze/store/definition, -> query parameters, execute-write analyze, permissions playground POST). -> The written tier documentation remains to be done. +> **Status:** implemented. Undocumented JSON endpoints self-describe with +> an `"unstable"` marker key, and `docs/json_api.rst` now opens with an +> "API stability" section (`json_api_stability`) declaring the 1.x +> promise: documented endpoints/keys are stable with additive-only +> changes, pagination tokens are opaque, the error format and token +> restriction semantics are stable, and the exempt tiers (marker-key +> endpoints, debug/support endpoints, explicitly-unstable keys) are +> listed. Cross-referenced from the introspection and permission-debug +> docs. Not everything under `/-/` can or should carry a 1.0 guarantee. Recommend shipping 1.0 with an explicit three-tier contract, per endpoint: @@ -431,8 +435,8 @@ Two details make tiering urgent rather than optional: `permissions-debug` (§6).~~ ✅ Done. 6. ~~401 (not silent-anonymous) for invalid/expired bearer tokens (§1c).~~ ✅ Done. -7. Publish explicit stability tiers, including extras and pagination-token - opacity (§9). +7. ~~Publish explicit stability tiers, including extras and pagination-token + opacity (§9).~~ ✅ Done. 8. Resolve the looks-like-a-bug list (§8), especially ~~trusted-query delete and row-delete 500~~ (both done). From 3a0ea585572ce56576b25ba3a843c587b98891c0 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 17:54:03 +0000 Subject: [PATCH 027/131] Unify page-size parameters on _size with table semantics The stored query lists silently clamped out-of-range ?_size= values (a request for 5000 quietly returned 1000) and did not accept the max keyword. They now share the table view semantics via a new parse_size_limit() helper: blank means default, "max" means the maximum (max_returned_rows for query lists), negative or non-integer values are a 400, and values over the maximum are a 400 instead of being silently clamped. The /-/allowed and /-/rules debug endpoints renamed their bare page/page_size parameters to _page/_size, matching the underscore grammar used by every other system parameter, with the same validation (400 instead of silently capping page_size at 200). Their HTML debug pages and next_url/previous_url builders use the new names. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/templates/debug_allowed.html | 10 ++--- datasette/templates/debug_rules.html | 10 ++--- datasette/utils/__init__.py | 22 +++++++++++ datasette/views/query_helpers.py | 9 ++--- datasette/views/special.py | 53 +++++++++++++------------- datasette/views/stored_queries.py | 1 + docs/authentication.rst | 4 +- docs/pages.rst | 2 +- existing-api.md | 9 +++-- stable-api-recommendations.md | 10 +++-- tests/test_error_shape.py | 50 ++++++++++++++++++++++++ tests/test_html.py | 6 +-- tests/test_permission_endpoints.py | 14 +++---- 13 files changed, 137 insertions(+), 63 deletions(-) diff --git a/datasette/templates/debug_allowed.html b/datasette/templates/debug_allowed.html index 4f8106b8..83cc1ae6 100644 --- a/datasette/templates/debug_allowed.html +++ b/datasette/templates/debug_allowed.html @@ -49,7 +49,7 @@
- + Number of results per page (max 200)
@@ -88,7 +88,7 @@ const hasDebugPermission = {{ 'true' if has_debug_permission else 'false' }}; (function() { const params = populateFormFromURL(); const action = params.get('action'); - const page = params.get('page'); + const page = params.get('_page'); if (action) { fetchResults(page ? parseInt(page) : 1); } @@ -102,14 +102,14 @@ async function fetchResults(page = 1) { const params = new URLSearchParams(); for (const [key, value] of formData.entries()) { - if (value && key !== 'page_size') { + if (value && key !== '_size' && key !== '_page') { params.append(key, value); } } const pageSize = document.getElementById('page_size').value || '50'; - params.append('page', page.toString()); - params.append('page_size', pageSize); + params.append('_page', page.toString()); + params.append('_size', pageSize); try { const response = await fetch('{{ urls.path("-/allowed.json") }}?' + params.toString(), { diff --git a/datasette/templates/debug_rules.html b/datasette/templates/debug_rules.html index aafa755d..d00ba9cc 100644 --- a/datasette/templates/debug_rules.html +++ b/datasette/templates/debug_rules.html @@ -37,7 +37,7 @@
- + Number of results per page (max 200)
@@ -75,7 +75,7 @@ const submitBtn = document.getElementById('submit-btn'); (function() { const params = populateFormFromURL(); const action = params.get('action'); - const page = params.get('page'); + const page = params.get('_page'); if (action) { fetchResults(page ? parseInt(page) : 1); } @@ -89,14 +89,14 @@ async function fetchResults(page = 1) { const params = new URLSearchParams(); for (const [key, value] of formData.entries()) { - if (value && key !== 'page_size') { + if (value && key !== '_size' && key !== '_page') { params.append(key, value); } } const pageSize = document.getElementById('page_size').value || '50'; - params.append('page', page.toString()); - params.append('page_size', pageSize); + params.append('_page', page.toString()); + params.append('_size', pageSize); try { const response = await fetch('{{ urls.path("-/rules.json") }}?' + params.toString(), { diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index 1d921f02..19de31ff 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -1294,6 +1294,28 @@ async def derive_named_parameters(db: "Database", sql: str) -> List[str]: return named_parameters(sql) +def parse_size_limit(value, default, maximum, name="_size"): + """ + Parse a page-size parameter using the same semantics as the table + view's ?_size=: blank means default, "max" means maximum, integers + must be 0 or greater and no larger than maximum. Raises ValueError + with a message suitable for a 400 response. + """ + if value in (None, ""): + return default + if value == "max": + return maximum + try: + size = int(value) + if size < 0: + raise ValueError + except ValueError: + raise ValueError("{} must be a positive integer".format(name)) + if size > maximum: + raise ValueError("{} must be <= {}".format(name, maximum)) + return size + + UNSTABLE_API_MESSAGE = ( "This API is not part of Datasette's stable interface and may change at any time" ) diff --git a/datasette/views/query_helpers.py b/datasette/views/query_helpers.py index 026a999f..e9f85b6d 100644 --- a/datasette/views/query_helpers.py +++ b/datasette/views/query_helpers.py @@ -13,6 +13,7 @@ from datasette.write_sql import ( operation_is_write, ) from datasette.utils import ( + parse_size_limit, named_parameters as derive_named_parameters, escape_sqlite, path_from_row_pks, @@ -94,13 +95,11 @@ def _as_optional_bool(value, name): raise QueryValidationError("{} must be 0 or 1".format(name)) -def _query_list_limit(value, default=50): - if value in (None, ""): - return default +def _query_list_limit(value, default, maximum): try: - return min(max(1, int(value)), 1000) + return parse_size_limit(value, default, maximum) except ValueError as ex: - raise QueryValidationError("_size must be an integer") from ex + raise QueryValidationError(str(ex)) from ex def _derived_query_parameters(sql): diff --git a/datasette/views/special.py b/datasette/views/special.py index 28d4d6a1..9386440c 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -8,6 +8,7 @@ from datasette.utils.asgi import Response, Forbidden from datasette.utils import ( UNSTABLE_API_MESSAGE, actor_matches_allow, + parse_size_limit, add_cors_headers, await_me_maybe, error_body, @@ -373,17 +374,17 @@ class AllowedResourcesView(BaseView): ) try: - page = int(request.args.get("page", "1")) - page_size = int(request.args.get("page_size", "50")) + page = int(request.args.get("_page", "1")) + if page < 1: + raise ValueError except ValueError: - return error_body("page and page_size must be integers", 400), 400 - if page < 1: - return error_body("page must be >= 1", 400), 400 - if page_size < 1: - return error_body("page_size must be >= 1", 400), 400 - max_page_size = 200 - if page_size > max_page_size: - page_size = max_page_size + return error_body("_page must be a positive integer", 400), 400 + try: + page_size = parse_size_limit( + request.args.get("_size"), default=50, maximum=200 + ) + except ValueError as ex: + return error_body(str(ex), 400), 400 offset = (page - 1) * page_size # Use the simplified allowed_resources method @@ -448,12 +449,12 @@ class AllowedResourcesView(BaseView): def build_page_url(page_number): pairs = [] for key in request.args: - if key in {"page", "page_size"}: + if key in {"_page", "_size"}: continue for value in request.args.getlist(key): pairs.append((key, value)) - pairs.append(("page", str(page_number))) - pairs.append(("page_size", str(page_size))) + pairs.append(("_page", str(page_number))) + pairs.append(("_size", str(page_size))) query = urllib.parse.urlencode(pairs) return f"{request.path}?{query}" @@ -511,19 +512,19 @@ class PermissionRulesView(BaseView): actor = request.actor if isinstance(request.actor, dict) else None try: - page = int(request.args.get("page", "1")) - page_size = int(request.args.get("page_size", "50")) + page = int(request.args.get("_page", "1")) + if page < 1: + raise ValueError except ValueError: return Response.json( - error_body("page and page_size must be integers", 400), status=400 + error_body("_page must be a positive integer", 400), status=400 ) - if page < 1: - return Response.json(error_body("page must be >= 1", 400), status=400) - if page_size < 1: - return Response.json(error_body("page_size must be >= 1", 400), status=400) - max_page_size = 200 - if page_size > max_page_size: - page_size = max_page_size + try: + page_size = parse_size_limit( + request.args.get("_size"), default=50, maximum=200 + ) + except ValueError as ex: + return Response.json(error_body(str(ex), 400), status=400) offset = (page - 1) * page_size from datasette.utils.actions_sql import build_permission_rules_sql @@ -574,12 +575,12 @@ class PermissionRulesView(BaseView): def build_page_url(page_number): pairs = [] for key in request.args: - if key in {"page", "page_size"}: + if key in {"_page", "_size"}: continue for value in request.args.getlist(key): pairs.append((key, value)) - pairs.append(("page", str(page_number))) - pairs.append(("page_size", str(page_size))) + pairs.append(("_page", str(page_number))) + pairs.append(("_size", str(page_size))) query = urllib.parse.urlencode(pairs) return f"{request.path}?{query}" diff --git a/datasette/views/stored_queries.py b/datasette/views/stored_queries.py index ea49d983..bb23a6bd 100644 --- a/datasette/views/stored_queries.py +++ b/datasette/views/stored_queries.py @@ -90,6 +90,7 @@ class QueryListView(BaseView): limit = _query_list_limit( request.args.get("_size"), default=20 if format_ == "html" else 50, + maximum=self.ds.max_returned_rows, ) is_write = _as_optional_bool(request.args.get("is_write"), "is_write") is_private = _as_optional_bool(request.args.get("is_private"), "is_private") diff --git a/docs/authentication.rst b/docs/authentication.rst index 7d4d019d..0a476c1c 100644 --- a/docs/authentication.rst +++ b/docs/authentication.rst @@ -1162,7 +1162,7 @@ The ``/-/allowed`` endpoint displays resources that the current actor can access This endpoint provides an interactive HTML form interface. Add ``.json`` to the URL path (e.g. ``/-/allowed.json``) to get the raw JSON response instead. -Pass ``?action=view-table`` (or another action) to select the action. Optional ``parent=`` and ``child=`` query parameters can narrow the results to a specific database/table pair. +Pass ``?action=view-table`` (or another action) to select the action. Optional ``parent=`` and ``child=`` query parameters can narrow the results to a specific database/table pair. Results are paginated: ``?_size=`` sets the page size (default 50, maximum 200, ``max`` for the maximum) and ``?_page=`` selects a page. This endpoint is publicly accessible to help users understand their own permissions. The potentially sensitive ``reason`` field is only shown to users with the ``permissions-debug`` permission - it shows the plugins and explanatory reasons that were responsible for each decision. @@ -1175,7 +1175,7 @@ The ``/-/rules`` endpoint displays all permission rules (both allow and deny) fo This endpoint provides an interactive HTML form interface. Add ``.json`` to the URL path (e.g. ``/-/rules.json?action=view-table``) to get the raw JSON response instead. -Pass ``?action=`` as a query parameter to specify which action to check. +Pass ``?action=`` as a query parameter to specify which action to check. The ``?_size=`` and ``?_page=`` pagination parameters work the same as on ``/-/allowed``. This endpoint requires the ``permissions-debug`` permission. diff --git a/docs/pages.rst b/docs/pages.rst index 67fe390b..65c03a49 100644 --- a/docs/pages.rst +++ b/docs/pages.rst @@ -95,7 +95,7 @@ Use the :ref:`ExecuteWriteView` JSON API to execute writable SQL programmaticall Stored query browsers --------------------- -The ``/-/queries`` page lists stored queries across every database visible to the current actor. The ``/database-name/-/queries`` page lists stored queries for a single database. +The ``/-/queries`` page lists stored queries across every database visible to the current actor. The ``/database-name/-/queries`` page lists stored queries for a single database. The JSON versions accept ``?_size=`` (default 50, ``max`` for the :ref:`setting_max_returned_rows` limit) and a ``?_next=`` pagination token. These pages support search, pagination and filters for read-only or writable queries and private or public queries. Adding a ``.json`` extension to either URL returns the same list as JSON. diff --git a/existing-api.md b/existing-api.md index 2f9996dc..c0c8e442 100644 --- a/existing-api.md +++ b/existing-api.md @@ -377,8 +377,8 @@ path always renders the HTML form; `.json` returns JSON. resources. Items gain a `reason` field if the actor also holds `permissions-debug`. - **Parameters:** `action` (required; missing → 400 canonical error, unknown - → 404), `parent`, `child` (requires `parent`), `page` (default 1), - `page_size` (default 50, silently capped at 200). + → 404), `parent`, `child` (requires `parent`), `_page` (default 1), + `_size` (default 50, maximum 200, accepts `max`; out-of-range → 400). - **Response:** `{"action", "actor_id", "page", "page_size", "total", "items": [{"parent", "child", "resource"}]}` with optional `next_url` / `previous_url`. @@ -1031,8 +1031,9 @@ databases (`database`/`database_color` are null, `show_database` true). - **Permissions:** no single gate; results filtered per query by `view-query` (private queries appear only for their owner). -- **Parameters:** `_size` (default 20 HTML / **50 JSON**, clamped 1–1000; - non-integer → 400), `_next` (cursor), `q` (substring search over +- **Parameters:** `_size` (default 20 HTML / **50 JSON**; accepts `max`; + values over `max_returned_rows` or non-integers → 400, matching table + `_size` semantics), `_next` (cursor), `q` (substring search over name/title/description/sql), `is_write` / `is_private` (booleans; invalid → 400 `"is_write must be 0 or 1"`), `source`, `owner_id`. - **Response** — 200: diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index b86d8228..ba837204 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -203,9 +203,13 @@ number. > **Status:** `next_url` now accompanies `next` in the default table JSON > keys (previously it required `?_extra=next_url`), so every response with > a `next` token also carries the ready-to-follow URL. Pagination tokens -> are deliberately left undocumented as to their internal structure. The -> `_size`/`page_size` naming and `has_more`/`total` differences remain -> open. +> are deliberately left undocumented as to their internal structure. +> `_size` is now the single page-size parameter with uniform table-style +> semantics everywhere: query lists accept `max` and 400 on out-of-range +> values (previously silently clamped), and the `/-/allowed` and +> `/-/rules` debug endpoints renamed `page`/`page_size` to +> `_page`/`_size` with the same validation (400 instead of silent +> capping at 200). The `has_more`/`total` differences remain open. | Endpoint | Mechanism | Token | Extras | |---|---|---|---| diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index a34fbed7..2398940b 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -612,3 +612,53 @@ async def test_threads_requires_permissions_debug(ds_error_shape): allowed = await ds_error_shape.client.get("/-/threads.json", actor={"id": "root"}) assert allowed.status_code == 200 assert allowed.json()["ok"] is True + + +# _size is the one page-size parameter, with uniform validation + + +@pytest.mark.asyncio +async def test_query_list_size_supports_max_keyword(ds_client): + response = await ds_client.get("/fixtures/-/queries.json?_size=max") + assert response.status_code == 200 + # ds_client runs with max_returned_rows=100 + assert response.json()["limit"] == 100 + + +@pytest.mark.asyncio +async def test_query_list_size_rejects_out_of_range(ds_client): + response = await ds_client.get("/fixtures/-/queries.json?_size=5000") + data = assert_canonical_error(response, 400) + assert data["errors"] == ["_size must be <= 100"] + + +@pytest.mark.asyncio +async def test_query_list_size_rejects_non_integer(ds_client): + response = await ds_client.get("/fixtures/-/queries.json?_size=bananas") + data = assert_canonical_error(response, 400) + assert data["errors"] == ["_size must be a positive integer"] + + +@pytest.mark.asyncio +@pytest.mark.parametrize("endpoint", ("allowed", "rules")) +async def test_debug_endpoints_use_size_and_page_parameters(ds_error_shape, endpoint): + base = "/-/{}.json?action=view-instance".format(endpoint) + ok = await ds_error_shape.client.get( + base + "&_size=1&_page=1", actor={"id": "root"} + ) + assert ok.status_code == 200 + assert ok.json()["page_size"] == 1 + + max_size = await ds_error_shape.client.get( + base + "&_size=max", actor={"id": "root"} + ) + assert max_size.status_code == 200 + assert max_size.json()["page_size"] == 200 + + too_big = await ds_error_shape.client.get(base + "&_size=500", actor={"id": "root"}) + data = assert_canonical_error(too_big, 400) + assert data["errors"] == ["_size must be <= 200"] + + bad_page = await ds_error_shape.client.get(base + "&_page=0", actor={"id": "root"}) + data = assert_canonical_error(bad_page, 400) + assert data["errors"] == ["_page must be a positive integer"] diff --git a/tests/test_html.py b/tests/test_html.py index 22943300..b4c47d80 100644 --- a/tests/test_html.py +++ b/tests/test_html.py @@ -1363,12 +1363,12 @@ async def test_permission_debug_tabs_with_query_string(ds_client): # Test /-/allowed with query string response = await ds_client.get( - "/-/allowed?action=view-table&page_size=50", actor=actor + "/-/allowed?action=view-table&_size=50", actor=actor ) assert response.status_code == 200 # Check that Rules and Check tabs have the query string - assert 'href="/-/rules?action=view-table&page_size=50"' in response.text - assert 'href="/-/check?action=view-table&page_size=50"' in response.text + assert 'href="/-/rules?action=view-table&_size=50"' in response.text + assert 'href="/-/check?action=view-table&_size=50"' in response.text # Playground and Actions should not have query string assert 'href="/-/permissions"' in response.text assert 'href="/-/actions"' in response.text diff --git a/tests/test_permission_endpoints.py b/tests/test_permission_endpoints.py index e25be23e..8726ab62 100644 --- a/tests/test_permission_endpoints.py +++ b/tests/test_permission_endpoints.py @@ -137,9 +137,7 @@ async def test_allowed_json_pagination(): await ds.refresh_schemas() # Test page 1 - response = await ds.client.get( - "/-/allowed.json?action=view-table&page_size=10&page=1" - ) + response = await ds.client.get("/-/allowed.json?action=view-table&_size=10&_page=1") assert response.status_code == 200 data = response.json() assert data["page"] == 1 @@ -147,9 +145,7 @@ async def test_allowed_json_pagination(): assert len(data["items"]) == 10 # Test page 2 - response = await ds.client.get( - "/-/allowed.json?action=view-table&page_size=10&page=2" - ) + response = await ds.client.get("/-/allowed.json?action=view-table&_size=10&_page=2") assert response.status_code == 200 data = response.json() assert data["page"] == 2 @@ -157,10 +153,10 @@ async def test_allowed_json_pagination(): # Verify items are different between pages response1 = await ds.client.get( - "/-/allowed.json?action=view-table&page_size=10&page=1" + "/-/allowed.json?action=view-table&_size=10&_page=1" ) response2 = await ds.client.get( - "/-/allowed.json?action=view-table&page_size=10&page=2" + "/-/allowed.json?action=view-table&_size=10&_page=2" ) items1 = {(item["parent"], item["child"]) for item in response1.json()["items"]} items2 = {(item["parent"], item["child"]) for item in response2.json()["items"]} @@ -323,7 +319,7 @@ async def test_rules_json_pagination(): # Test basic pagination structure - just verify it returns paginated results response = await ds.client.get( - "/-/rules.json?action=view-table&page_size=2&page=1", + "/-/rules.json?action=view-table&_size=2&_page=1", actor={"id": "root"}, ) assert response.status_code == 200 From 6d253d10c8cf187d7c06812d4fbbd3337780d939 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 5 Jul 2026 09:09:20 -0700 Subject: [PATCH 028/131] Compatibility with sqlite-utils>=4.0rc2 Runs the tests in CI. Had to add a little bit of code to handle the difference between [table] and "table" and REAL v.s FLOAT. --- .github/workflows/test.yml | 22 +++++++++++++++++++ pyproject.toml | 2 +- tests/test_api_write.py | 45 ++++++++++++++++++++++++++++++++------ 3 files changed, 61 insertions(+), 8 deletions(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 9e47db6f..ba5b1611 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -51,3 +51,25 @@ jobs: run: | pip install datasette-init datasette-json-html tests/test-datasette-load-plugins.sh + test-sqlite-utils-4: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v6 + - name: Set up Python + uses: actions/setup-python@v6 + with: + python-version: "3.13" + cache: pip + cache-dependency-path: pyproject.toml + - name: Build extension for --load-extension test + run: |- + (cd tests && gcc ext.c -fPIC -shared -o ext.so) + - name: Install dependencies + run: | + pip install . --group dev + pip install --pre 'sqlite-utils>=4.0rc2' + pip freeze + - name: Run tests + run: | + pytest -n auto -m "not serial" + pytest -m "serial" diff --git a/pyproject.toml b/pyproject.toml index 215b2cca..38776b2c 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -35,7 +35,7 @@ dependencies = [ "PyYAML>=5.3", "mergedeep>=1.1.1", "itsdangerous>=1.1", - "sqlite-utils>=3.30,<4.0", + "sqlite-utils>=3.30", "asyncinject>=0.7", "setuptools", "pip", diff --git a/tests/test_api_write.py b/tests/test_api_write.py index 1154176f..af1aeb89 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -3,9 +3,31 @@ from datasette.events import RenameTableEvent from datasette.utils import escape_sqlite, sqlite3 from .utils import last_event import pytest +import re import time +def schema_variants(schema): + # sqlite-utils < 4 quotes identifiers [like_this] and uses FLOAT; + # sqlite-utils >= 4 quotes them "like_this" and uses REAL. Given a + # schema fragment in the old format, return both variants so tests + # can pass against either version. + converted = re.sub(r"\[([^\]]+)\]", r'"\1"', schema).replace("FLOAT", "REAL") + return (schema, converted) + + +def assert_schema_contains(fragment, schema): + assert any( + variant in schema for variant in schema_variants(fragment) + ), "Expected schema to contain {!r}, got {!r}".format(fragment, schema) + + +def assert_schema_not_contains(fragment, schema): + assert not any( + variant in schema for variant in schema_variants(fragment) + ), "Expected schema not to contain {!r}, got {!r}".format(fragment, schema) + + @pytest.fixture def ds_write(tmp_path_factory): db_directory = tmp_path_factory.mktemp("dbs") @@ -72,8 +94,8 @@ async def test_base64_write_api_create_table_infers_blob_and_raw_escapes(ds_writ headers=_headers(token), ) assert response.status_code == 201 - assert "[data] BLOB" in response.json()["schema"] - assert "[literal] TEXT" in response.json()["schema"] + assert_schema_contains("[data] BLOB", response.json()["schema"]) + assert_schema_contains("[literal] TEXT", response.json()["schema"]) rows = (await ds_write.get_database("data").execute(""" select @@ -1197,7 +1219,9 @@ async def test_alter_table_foreign_key_operations(ds_write): assert response.status_code == 200, response.text data = response.json() assert data["operations_applied"] == 2 - assert "[owner_id] INTEGER REFERENCES [owners]([id])" in data["schema"] + assert_schema_contains( + "[owner_id] INTEGER REFERENCES [owners]([id])", data["schema"] + ) response = await ds_write.client.post( "/data/docs/-/alter", @@ -1208,7 +1232,7 @@ async def test_alter_table_foreign_key_operations(ds_write): ) assert response.status_code == 200, response.text data = response.json() - assert "[owner_id] INTEGER REFERENCES" not in data["schema"] + assert_schema_not_contains("[owner_id] INTEGER REFERENCES", data["schema"]) response = await ds_write.client.post( "/data/docs/-/alter", @@ -1232,7 +1256,9 @@ async def test_alter_table_foreign_key_operations(ds_write): ) assert response.status_code == 200, response.text data = response.json() - assert "[owner_id] INTEGER REFERENCES [categories]([id])" in data["schema"] + assert_schema_contains( + "[owner_id] INTEGER REFERENCES [categories]([id])", data["schema"] + ) response = await ds_write.client.post( "/data/docs/-/alter", @@ -1241,7 +1267,7 @@ async def test_alter_table_foreign_key_operations(ds_write): ) assert response.status_code == 200, response.text data = response.json() - assert "[owner_id] INTEGER REFERENCES" not in data["schema"] + assert_schema_not_contains("[owner_id] INTEGER REFERENCES", data["schema"]) @pytest.mark.asyncio @@ -2177,6 +2203,9 @@ async def test_create_table( ) assert response.status_code == expected_status data = response.json() + if isinstance(expected_response, dict) and "schema" in expected_response: + assert data.get("schema") in schema_variants(expected_response["schema"]) + expected_response = dict(expected_response, schema=data.get("schema")) assert data == expected_response # Should have tracked the expected events events = ds_write._tracked_events @@ -2219,7 +2248,9 @@ async def test_create_table_with_foreign_key(ds_write): ) assert response.status_code == 201 data = response.json() - assert "[owner_id] INTEGER REFERENCES [owners]([id])" in data["schema"] + assert_schema_contains( + "[owner_id] INTEGER REFERENCES [owners]([id])", data["schema"] + ) @pytest.mark.asyncio From 557e08c6ef02ba1ddee264400868d0295e11c734 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 5 Jul 2026 22:46:00 -0700 Subject: [PATCH 029/131] Make binary playwright test more robust Failed here: https://github.com/simonw/datasette/actions/runs/28746832371/job/85239037361 --- tests/test_playwright.py | 25 +++++++++++-------------- 1 file changed, 11 insertions(+), 14 deletions(-) diff --git a/tests/test_playwright.py b/tests/test_playwright.py index 15c642fe..eb1edb57 100644 --- a/tests/test_playwright.py +++ b/tests/test_playwright.py @@ -356,6 +356,14 @@ def binary_file_blob(datasette_server, pk): return response.content +def wait_for_binary_control_file(binary_control, size, name=None): + binary_control.locator( + ".row-edit-binary-size", has_text=f"Binary: {size} bytes" + ).wait_for() + if name: + binary_control.locator(".row-edit-binary-name", has_text=name).wait_for() + + def bulk_default_rows(datasette_server, **filters): params = { "_shape": "objects", @@ -1516,11 +1524,7 @@ def test_edit_row_binary_control_replaces_blob_from_file(page, datasette_server) "buffer": replacement, } ) - assert ( - binary_control.locator(".row-edit-binary-size").inner_text() - == f"Binary: {len(replacement)} bytes" - ) - assert "replacement.bin" in binary_control.inner_text() + wait_for_binary_control_file(binary_control, len(replacement), "replacement.bin") dialog.locator(".row-edit-save").click() page.locator(".row-mutation-status", has_text="Updated row 1").wait_for() @@ -1547,10 +1551,7 @@ def test_edit_row_binary_control_handles_null_blob(page, datasette_server): "buffer": replacement, } ) - assert ( - binary_control.locator(".row-edit-binary-size").inner_text() - == f"Binary: {len(replacement)} bytes" - ) + wait_for_binary_control_file(binary_control, len(replacement), "from-null.bin") dialog.locator(".row-edit-save").click() page.locator(".row-mutation-status", has_text="Updated row 3").wait_for() @@ -1583,11 +1584,7 @@ def test_insert_row_binary_control_accepts_pasted_file(page, datasette_server): }""", list(pasted), ) - assert ( - binary_control.locator(".row-edit-binary-size").inner_text() - == f"Binary: {len(pasted)} bytes" - ) - assert "pasted.bin" in binary_control.inner_text() + wait_for_binary_control_file(binary_control, len(pasted), "pasted.bin") dialog.locator(".row-edit-save").click() page.locator(".row-mutation-status", has_text="Inserted row 4").wait_for() From c833217401898d0b2076f468f557da525c7e2daa Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 5 Jul 2026 22:47:05 -0700 Subject: [PATCH 030/131] Test with sqlite-utils>=4.0rc3 Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4889420844 --- .github/workflows/test.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index ba5b1611..acc2d6b6 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -67,7 +67,7 @@ jobs: - name: Install dependencies run: | pip install . --group dev - pip install --pre 'sqlite-utils>=4.0rc2' + pip install --pre 'sqlite-utils>=4.0rc3' pip freeze - name: Run tests run: | From 9a0b78b76cb45ab0f065f7f65ca528eec3ebf57f Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 5 Jul 2026 22:53:12 -0700 Subject: [PATCH 031/131] Vendor setup-sqlite-version Same solution as https://github.com/simonw/sqlite-utils/pull/775 --- .../actions/setup-sqlite-version/action.yml | 39 +++++ .../setup-sqlite-version.sh | 144 ++++++++++++++++++ .github/workflows/test-sqlite-support.yml | 2 +- 3 files changed, 184 insertions(+), 1 deletion(-) create mode 100644 .github/actions/setup-sqlite-version/action.yml create mode 100644 .github/actions/setup-sqlite-version/setup-sqlite-version.sh diff --git a/.github/actions/setup-sqlite-version/action.yml b/.github/actions/setup-sqlite-version/action.yml new file mode 100644 index 00000000..fdbc71c9 --- /dev/null +++ b/.github/actions/setup-sqlite-version/action.yml @@ -0,0 +1,39 @@ +name: "Setup SQLite version" +description: "Build and activate a specific SQLite version from its amalgamation archive" +inputs: + version: + description: "The SQLite version to install" + required: true + cflags: + description: "CFLAGS to use when compiling SQLite" + required: false + default: "" + skip-activate: + description: "Set to true to skip modifying the library path" + required: false + default: "false" + fallback-urls: + description: "Whitespace-separated fallback download URLs to try after sqlite.org" + required: false + default: "" +outputs: + sqlite-location: + description: "Directory containing the compiled SQLite library" + value: ${{ steps.build.outputs.sqlite-location }} +runs: + using: "composite" + steps: + - shell: bash + run: mkdir -p "$RUNNER_TEMP/sqlite-versions/downloads" + - uses: actions/cache@v6 + with: + path: ${{ runner.temp }}/sqlite-versions/downloads + key: setup-sqlite-version-${{ inputs.version }}-amalgamation-v1 + - id: build + shell: bash + run: bash "$GITHUB_ACTION_PATH/setup-sqlite-version.sh" + env: + SQLITE_VERSION: ${{ inputs.version }} + SQLITE_CFLAGS: ${{ inputs.cflags }} + SQLITE_SKIP_ACTIVATE: ${{ inputs.skip-activate }} + SQLITE_EXTRA_FALLBACK_URLS: ${{ inputs.fallback-urls }} diff --git a/.github/actions/setup-sqlite-version/setup-sqlite-version.sh b/.github/actions/setup-sqlite-version/setup-sqlite-version.sh new file mode 100644 index 00000000..03d6a68f --- /dev/null +++ b/.github/actions/setup-sqlite-version/setup-sqlite-version.sh @@ -0,0 +1,144 @@ +#!/usr/bin/env bash +set -euo pipefail + +version_spec="${SQLITE_VERSION:?SQLITE_VERSION is required}" +cflags="${SQLITE_CFLAGS:-}" +skip_activate="${SQLITE_SKIP_ACTIVATE:-false}" +extra_fallback_urls="${SQLITE_EXTRA_FALLBACK_URLS:-}" + +case "$version_spec" in + 3.46 | 3.46.0) + sqlite_version="3.46.0" + sqlite_year="2024" + amalgamation_id="3460000" + builtin_fallback_urls="https://static.simonwillison.net/static/2026/sqlite-amalgamation-3460000.zip" + ;; + 3.25 | 3.25.0) + sqlite_version="3.25.0" + sqlite_year="2018" + amalgamation_id="3250000" + builtin_fallback_urls="https://static.simonwillison.net/static/2026/sqlite-amalgamation-3250000.zip?v=1" + ;; + *) + echo "::error::Unsupported SQLite version '$version_spec'. Add its release year and amalgamation id to $GITHUB_ACTION_PATH/setup-sqlite-version.sh." + exit 1 + ;; +esac + +case "$(uname -s)" in + Linux) + library_name="libsqlite3.so.0" + library_path_var="LD_LIBRARY_PATH" + ;; + Darwin) + library_name="libsqlite3.dylib" + library_path_var="DYLD_LIBRARY_PATH" + ;; + *) + echo "::error::Unsupported platform $(uname -s)" + exit 1 + ;; +esac + +runner_temp="${RUNNER_TEMP:-}" +if [ -z "$runner_temp" ]; then + runner_temp="$(mktemp -d)" +fi + +filename="sqlite-amalgamation-${amalgamation_id}" +official_url="https://www.sqlite.org/${sqlite_year}/${filename}.zip" +download_dir="${runner_temp}/sqlite-versions/downloads" +source_root="${runner_temp}/sqlite-versions/source" +source_dir="${source_root}/${filename}" +build_dir="${runner_temp}/sqlite-versions/build/${sqlite_version}" +archive_path="${download_dir}/${filename}.zip" + +mkdir -p "$download_dir" "$source_root" "$build_dir" + +download_archive() { + local url + local candidate_path="${archive_path}.tmp" + local urls=("$official_url") + + for url in $builtin_fallback_urls $extra_fallback_urls; do + urls+=("$url") + done + + rm -f "$candidate_path" + for url in "${urls[@]}"; do + echo "Downloading SQLite ${sqlite_version} amalgamation from ${url}" + if curl \ + --fail \ + --location \ + --show-error \ + --retry 5 \ + --retry-delay 2 \ + --retry-max-time 180 \ + --retry-all-errors \ + --connect-timeout 20 \ + --max-time 240 \ + --output "$candidate_path" \ + "$url"; then + mv "$candidate_path" "$archive_path" + return 0 + fi + + echo "::warning::Download failed from ${url}" + rm -f "$candidate_path" + done + + echo "::error::Could not download SQLite ${sqlite_version} amalgamation" + return 1 +} + +if [ ! -f "${source_dir}/sqlite3.c" ]; then + if [ ! -f "$archive_path" ]; then + download_archive + fi + + rm -rf "$source_dir" + unzip -q "$archive_path" -d "$source_root" +fi + +if [ ! -f "${source_dir}/sqlite3.c" ]; then + echo "::error::Expected ${source_dir}/sqlite3.c after extracting ${archive_path}" + exit 1 +fi + +read -r -a cflag_args <<< "$cflags" + +echo "Compiling SQLite ${sqlite_version} to ${build_dir}/${library_name}" +gcc \ + -fPIC \ + -shared \ + "${cflag_args[@]}" \ + "${source_dir}/sqlite3.c" \ + "-I${source_dir}" \ + -o "${build_dir}/${library_name}" + +if [ "$library_name" = "libsqlite3.so.0" ]; then + ln -sf "$library_name" "${build_dir}/libsqlite3.so" +fi + +if [ -n "${GITHUB_OUTPUT:-}" ]; then + echo "sqlite-location=${build_dir}" >> "$GITHUB_OUTPUT" +else + echo "sqlite-location=${build_dir}" +fi + +case "$(printf '%s' "$skip_activate" | tr '[:upper:]' '[:lower:]')" in + true | 1 | yes) + echo "Skipping ${library_path_var} activation" + ;; + *) + existing_value="${!library_path_var:-}" + if [ -n "${GITHUB_ENV:-}" ]; then + if [ -n "$existing_value" ]; then + echo "${library_path_var}=${build_dir}:${existing_value}" >> "$GITHUB_ENV" + else + echo "${library_path_var}=${build_dir}" >> "$GITHUB_ENV" + fi + fi + echo "Added ${build_dir} to ${library_path_var}" + ;; +esac diff --git a/.github/workflows/test-sqlite-support.yml b/.github/workflows/test-sqlite-support.yml index 23fce459..d86000bf 100644 --- a/.github/workflows/test-sqlite-support.yml +++ b/.github/workflows/test-sqlite-support.yml @@ -34,7 +34,7 @@ jobs: cache: pip cache-dependency-path: pyproject.toml - name: Set up SQLite ${{ matrix.sqlite-version }} - uses: asg017/sqlite-versions@71ea0de37ae739c33e447af91ba71dda8fcf22e6 + uses: ./.github/actions/setup-sqlite-version with: version: ${{ matrix.sqlite-version }} cflags: "-DSQLITE_ENABLE_DESERIALIZE -DSQLITE_ENABLE_FTS5 -DSQLITE_ENABLE_FTS4 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_RTREE -DSQLITE_ENABLE_JSON1" From b6f5fd5cd0fff5432cd74774d5385cb657180959 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 5 Jul 2026 23:04:51 -0700 Subject: [PATCH 032/131] test_internal_foreign_key_references() fix for sqlite-utils 4.0rc3 Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4889529980 --- tests/test_internal_db.py | 75 +++++++++++++++++++++++++++++++++------ 1 file changed, 64 insertions(+), 11 deletions(-) diff --git a/tests/test_internal_db.py b/tests/test_internal_db.py index 26d63a92..340c4813 100644 --- a/tests/test_internal_db.py +++ b/tests/test_internal_db.py @@ -1,5 +1,6 @@ import pytest -import sqlite_utils + +from datasette.utils import escape_sqlite # ensure refresh_schemas() gets called before interacting with internal_db @@ -76,19 +77,71 @@ async def test_internal_foreign_key_references(ds_client): internal_db = await ensure_internal(ds_client) def inner(conn): - db = sqlite_utils.Database(conn) - table_names = db.table_names() - for table in db.tables: - for fk in table.foreign_keys: - other_table = fk.other_table - other_column = fk.other_column - message = 'Column "{}.{}" references other column "{}.{}" which does not exist'.format( - table.name, fk.column, other_table, other_column + table_names = [ + row[0] + for row in conn.execute( + "select name from sqlite_master where type = 'table'" + ).fetchall() + ] + + def columns_for_table(table_name): + return { + row[1] + for row in conn.execute( + "PRAGMA table_info({})".format(escape_sqlite(table_name)) + ).fetchall() + } + + def primary_keys_for_table(table_name): + return [ + name + for _, name in sorted( + (row[5], row[1]) + for row in conn.execute( + "PRAGMA table_info({})".format(escape_sqlite(table_name)) + ).fetchall() + if row[5] + ) + ] + + columns_by_table = { + table_name: columns_for_table(table_name) for table_name in table_names + } + + for table_name in table_names: + foreign_key_rows = conn.execute( + "PRAGMA foreign_key_list({})".format(escape_sqlite(table_name)) + ).fetchall() + foreign_keys_by_id = {} + for foreign_key in foreign_key_rows: + foreign_keys_by_id.setdefault(foreign_key[0], []).append(foreign_key) + + for foreign_key_rows in foreign_keys_by_id.values(): + foreign_key_rows.sort(key=lambda row: row[1]) + other_table = foreign_key_rows[0][2] + other_columns = [row[4] for row in foreign_key_rows] + message = 'Column "{}.{}" references other table "{}" which does not exist'.format( + table_name, foreign_key_rows[0][3], other_table ) assert other_table in table_names, message + " (bad table)" - assert other_column in db[other_table].columns_dict, ( - message + " (bad column)" + if all(other_column is None for other_column in other_columns): + other_columns = primary_keys_for_table(other_table) + length_message = 'Foreign key from "{}" to "{}" has {} columns but references {} columns'.format( + table_name, + other_table, + len(foreign_key_rows), + len(other_columns), ) + assert len(other_columns) == len(foreign_key_rows), length_message + + for foreign_key, other_column in zip(foreign_key_rows, other_columns): + column = foreign_key[3] + message = 'Column "{}.{}" references other column "{}.{}" which does not exist'.format( + table_name, column, other_table, other_column + ) + assert other_column in columns_by_table[other_table], ( + message + " (bad column)" + ) await internal_db.execute_fn(inner) From f4dfd6e0f73a441ad0aa28b31bcdcf1c73942b29 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 22:19:35 +0000 Subject: [PATCH 033/131] Fix unused variable flagged by ruff Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- tests/test_error_shape.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index 2398940b..b5f16b30 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -422,7 +422,7 @@ async def test_bad_signature_token_returns_401(ds_error_shape): response = await ds_error_shape.client.get( "/-/actor.json", headers={"Authorization": "Bearer dstok_garbage"} ) - data = assert_canonical_error(response, 401) + assert_canonical_error(response, 401) assert response.headers["www-authenticate"].startswith("Bearer") From 5c418efd7f18f9b469ccf14c1c8f86fd69d4427b Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 22:42:12 +0000 Subject: [PATCH 034/131] Update tests from main for canonical error shape and rows key Two tests merged from main were written against the pre-merge response shapes: the max_post_body_bytes 413 error now uses the canonical error envelope, and row update with return:true responds with a rows list rather than a singular row. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- tests/test_api_write.py | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/tests/test_api_write.py b/tests/test_api_write.py index 7145a859..14dedb73 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -169,7 +169,10 @@ async def test_base64_write_api_insert_upsert_update_decode_blobs(ds_write): headers=_headers(token), ) assert update_response.status_code == 200 - assert update_response.json()["row"]["data"] == {"$base64": True, "encoded": "/wAB"} + assert update_response.json()["rows"][0]["data"] == { + "$base64": True, + "encoded": "/wAB", + } rows = (await db.execute(""" select @@ -344,10 +347,9 @@ async def test_insert_rows_post_body_too_large(tmp_path_factory): headers=_headers(token), ) assert response.status_code == 413 - assert response.json() == { - "ok": False, - "errors": ["Request body exceeded maximum size of 100 bytes"], - } + assert response.json() == error_body( + ["Request body exceeded maximum size of 100 bytes"], 413 + ) # A small body should still work response2 = await ds.client.post( "/data/docs/-/insert", From e892c686c20ab21dd9b3d3307a8b2af36149a685 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 23:02:42 +0000 Subject: [PATCH 035/131] Remove has_more from query list JSON - next: null signals the end next: null (with next_url: null) is the single end-of-results signal across the API, keeping default response keys to a minimum. The StoredQueryPage.has_more attribute on the documented Python API is unchanged. Also fixes a bug this uncovered: the query list JSON next_url pointed at the HTML page (it was built from the query list path, dropping the .json extension) and was a relative path where the table view next_url is absolute. It is now built from the request path and absolute, so it preserves the requested format and can be followed directly. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/stored_queries.py | 1 - datasette/views/stored_queries.py | 7 +++---- existing-api.md | 2 +- stable-api-recommendations.md | 8 +++++++- tests/test_queries.py | 26 +++++++++++++++++++++++++- 5 files changed, 36 insertions(+), 8 deletions(-) diff --git a/datasette/stored_queries.py b/datasette/stored_queries.py index d7e1ec99..f5f977d9 100644 --- a/datasette/stored_queries.py +++ b/datasette/stored_queries.py @@ -83,7 +83,6 @@ def stored_query_page_to_dict(page: StoredQueryPage) -> dict[str, Any]: return { "queries": [stored_query_to_dict(query) for query in page.queries], "next": page.next, - "has_more": page.has_more, "limit": page.limit, } diff --git a/datasette/views/stored_queries.py b/datasette/views/stored_queries.py index bb23a6bd..3273d13c 100644 --- a/datasette/views/stored_queries.py +++ b/datasette/views/stored_queries.py @@ -120,9 +120,9 @@ class QueryListView(BaseView): if key != "_next" ] pairs.append(("_next", page.next)) - next_url = "{}?{}".format( - query_list_path, - urlencode(pairs), + next_url = self.ds.absolute_url( + request, + "{}?{}".format(request.path, urlencode(pairs)), ) current_filters = { @@ -208,7 +208,6 @@ class QueryListView(BaseView): "queries": page.queries, "next": page.next, "next_url": next_url, - "has_more": page.has_more, "limit": page.limit, "show_private_note": any(query.is_private for query in page.queries), "show_trusted_note": any(query.is_trusted for query in page.queries), diff --git a/existing-api.md b/existing-api.md index c0c8e442..55e9d6a2 100644 --- a/existing-api.md +++ b/existing-api.md @@ -1038,7 +1038,7 @@ databases (`database`/`database_color` are null, `show_database` true). 400 `"is_write must be 0 or 1"`), `source`, `owner_id`. - **Response** — 200: `{"ok": true, "database", "database_color", "queries": [...], "next", - "next_url", "has_more", "limit", "show_private_note", + "next_url", "limit", "show_private_note", "show_trusted_note", "query_list_path", "show_database", "facets": [{title, items: [{label, count, href, active}]}], "filters": {q, is_write, is_private, source, owner_id}}`. diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index ba837204..229dccd4 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -209,7 +209,13 @@ number. > values (previously silently clamped), and the `/-/allowed` and > `/-/rules` debug endpoints renamed `page`/`page_size` to > `_page`/`_size` with the same validation (400 instead of silent -> capping at 200). The `has_more`/`total` differences remain open. +> capping at 200). `has_more` has been **removed** from the query-list +> JSON — `next: null` is the single end-of-results signal everywhere, +> keeping default response keys minimal (`total` remains a debug-endpoint +> nicety). Fixing this also uncovered and fixed a bug where the query +> list's JSON `next_url` pointed at the HTML page (it dropped the `.json` +> extension) and was relative where the table `next_url` is absolute. +> §3 is now fully resolved. | Endpoint | Mechanism | Token | Extras | |---|---|---|---| diff --git a/tests/test_queries.py b/tests/test_queries.py index 699743c5..c25ec358 100644 --- a/tests/test_queries.py +++ b/tests/test_queries.py @@ -879,7 +879,7 @@ async def test_query_list_html_defaults_to_twenty_and_shows_pagination(): assert response.text.count('aria-label="Query pagination"') == 1 assert "Demo query 20" in response.text assert "Demo query 21" not in response.text - assert 'href="/data/-/queries?_next=' in response.text + assert 'href="http://localhost/data/-/queries?_next=' in response.text assert len(json_response.json()["queries"]) == 25 @@ -3827,3 +3827,27 @@ async def test_stored_query_json_uses_parameters_not_params(): query = [q for q in listing["queries"] if q["name"] == "with_params"][0] assert query["parameters"] == ["name", "age"] assert "params" not in query + + +@pytest.mark.asyncio +async def test_query_list_json_signals_pagination_via_next_only(): + ds = Datasette(memory=True) + ds.add_memory_database("query_list_next_only", name="data") + await ds.invoke_startup() + for i in range(3): + await ds.add_query( + "data", + name="q{}".format(i), + sql="select {}".format(i), + ) + first = (await ds.client.get("/data/-/queries.json?_size=2")).json() + assert "has_more" not in first + assert first["next"] is not None + assert first["next_url"] is not None + # The internal test client cannot follow absolute URLs + next_path = first["next_url"].replace("http://localhost", "") + assert next_path.startswith("/data/-/queries.json?") + last = (await ds.client.get(next_path)).json() + assert "has_more" not in last + assert last["next"] is None + assert last["next_url"] is None From 6e17c513619a139c1157d2c207324a8eddbec8d2 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 23:08:05 +0000 Subject: [PATCH 036/131] Document why upsert returns 200 where insert returns 201 An upsert may update existing rows without creating anything, so it deliberately does not claim resource creation with a 201. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- docs/json_api.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/json_api.rst b/docs/json_api.rst index 32d28733..2105d6e0 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -1944,7 +1944,7 @@ The above example will: Similar to ``/-/insert``, a ``row`` key with an object can be used instead of a ``rows`` array to upsert a single row. -If successful, this will return a ``200`` status code and a ``{"ok": true}`` response body. +If successful, this will return a ``200`` status code and a ``{"ok": true}`` response body. This is deliberately different from the ``201`` returned by :ref:`insert `: an upsert may update existing rows without creating anything, so it does not claim resource creation. Add ``"return": true`` to the request body to return full copies of the affected rows after they have been inserted or updated: From 3322e1f528f4d607eb0693a4cee2df97400eb215 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 23:08:28 +0000 Subject: [PATCH 037/131] Document the boolean query string argument grammar Boolean arguments parsed by value_as_boolean() accept on/true/1 and off/false/0 - state this once in the JSON API docs rather than leaving each argument to imply its own grammar. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- docs/json_api.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/json_api.rst b/docs/json_api.rst index 2105d6e0..c31ba756 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -276,6 +276,10 @@ Here is an example Python function built using `requests Date: Mon, 6 Jul 2026 23:09:10 +0000 Subject: [PATCH 038/131] Advise plugin authors on naming secret configuration keys /-/config redacts values for keys whose names contain secret, key, password, token, hash or dsn. Plugins that follow that naming get automatic redaction; plugins that don't will leak their secrets on that endpoint. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- docs/plugins.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/plugins.rst b/docs/plugins.rst index d2b5c20a..d32a9fe6 100644 --- a/docs/plugins.rst +++ b/docs/plugins.rst @@ -459,6 +459,8 @@ Secret configuration values Some plugins may need configuration that should stay secret - API keys for example. There are two ways in which you can store secret configuration values. +The :ref:`/-/config ` introspection endpoint redacts the values of any configuration keys whose names contain one of these substrings: ``secret``, ``key``, ``password``, ``token``, ``hash`` or ``dsn``. Name your plugin's secret configuration keys accordingly - for example ``api_key`` or ``client_secret`` - so they are automatically redacted there. + **As environment variables**. If your secret lives in an environment variable that is available to the Datasette process, you can indicate that the configuration value should be read from that environment variable like so: .. [[[cog From 022eb6d3a0a53961cff6b3221c2c54f64b4729a1 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 23:09:27 +0000 Subject: [PATCH 039/131] Mark documented items in recommendations Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- stable-api-recommendations.md | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index 229dccd4..9e8fa356 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -243,10 +243,11 @@ Concerns: ## 4. HTTP semantics (P2) -- **201 vs 200:** insert → 201, upsert → 200 (views/table.py:1194), create +- ~~**201 vs 200:** insert → 201, upsert → 200 (views/table.py:1194), create table → 201, store query → 201. Insert-201/upsert-200 is defensible (upsert may not create) but it is undocumented subtlety; state it, or - return 200 for both with an explicit `created` count. + return 200 for both with an explicit `created` count.~~ ✅ **Done** — + documented as deliberate in the upsert docs. - **Destructive-action confirmation is asymmetric:** table drop requires `{"confirm": true}` and has a preview response (views/table.py:1346-1365); row delete executes immediately and ignores the body; query delete @@ -284,10 +285,13 @@ Concerns: accepted input alias for API creation and `datasette.yaml` config. - **Three names for the same concept across error/message payloads:** `error`, `errors`, `message`. See §1. -- **Boolean query parameters have at least three grammars:** `_nl=on`, +- ~~**Boolean query parameters have at least three grammars:** `_nl=on`, `_labels=on/off`, `?all=1`, `is_write=1|0|true|false|t|f|yes|no|on|off`, `_nocount=1`. Adopt one accepted set (the query-list parser at - query_helpers.py:81-94 is a good candidate) and apply it everywhere. + query_helpers.py:81-94 is a good candidate) and apply it everywhere.~~ + ✅ **Documented** — the JSON API docs state the canonical grammar + (`on/true/1`, `off/false/0`), which `value_as_boolean` already accepts + everywhere it is used. - ~~**`.jsono`** survives on the homepage route (identical output to `.json`) and as a row-view redirect. Remove it at 1.0; it is pure legacy.~~ ✅ Removed: the homepage routes only accept `.json` and the row-view @@ -322,9 +326,12 @@ Concerns: task reprs including file paths) behind only `view-instance`. Consider `permissions-debug`, alongside `/-/actions` which already requires it.~~ ✅ **Done** — `/-/threads` now requires `permissions-debug`. -- **(P3) `/-/config` redaction is substring-based** on six key names +- ~~**(P3) `/-/config` redaction is substring-based** on six key names (app.py:2502-2505); plugins storing secrets under other names leak. Worth - a note in plugin authoring docs plus a `redact_keys` plugin hook. + a note in plugin authoring docs plus a `redact_keys` plugin hook.~~ + ✅ **Documented** — the plugin secrets docs now advise naming keys to + match the redaction substrings (a `redact_keys` hook remains a possible + future addition). - **(P3) Database-level checks on `/-/create`** (insert-row/update-row checked against `DatabaseResource`, not the about-to-exist table — table_create_alter.py:819-856) vs table-level checks on `/-/insert`. From 87cd695ca3b9293937228762a75f94e8a4931469 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 23:20:47 +0000 Subject: [PATCH 040/131] Write endpoints parse the body as JSON regardless of Content-Type The insert, upsert, alter and set-column-type endpoints previously required Content-Type: application/json while /-/create parsed the body blind - and insert returned a 500 AttributeError when the header was missing entirely. The lenient rule is now uniform: the body is always parsed as JSON and invalid JSON is a 400. This makes curl -d and requests data=json.dumps(...) invocations work without remembering the header. Cross-site request forgery remains prevented by the Origin and Sec-Fetch-Site checks in CrossOriginProtectionMiddleware, which is the defense the strict content-type requirement was historically standing in for. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/views/table.py | 8 +--- datasette/views/table_create_alter.py | 4 -- docs/json_api.rst | 2 + existing-api.md | 6 +-- stable-api-recommendations.md | 8 +++- tests/test_api_write.py | 13 +------ tests/test_column_types.py | 12 +----- tests/test_error_shape.py | 55 +++++++++++++++++++++++++++ 8 files changed, 69 insertions(+), 39 deletions(-) diff --git a/datasette/views/table.py b/datasette/views/table.py index 0a9c88d4..80ad1aab 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -951,9 +951,7 @@ class TableInsertView(BaseView): def _errors(errors): return None, errors, {} - if not request.headers.get("content-type").startswith("application/json"): - # TODO: handle form-encoded data - return _errors(["Invalid content-type, must be application/json"]) + # The body is parsed as JSON regardless of the Content-Type header try: data = await request.json() except json.JSONDecodeError as e: @@ -1260,10 +1258,6 @@ class TableSetColumnTypeView(BaseView): ): return _error(["Permission denied"], 403) - content_type = request.headers.get("content-type") or "" - if not content_type.startswith("application/json"): - return _error(["Invalid content-type, must be application/json"], 400) - try: data = await request.json() except json.JSONDecodeError as e: diff --git a/datasette/views/table_create_alter.py b/datasette/views/table_create_alter.py index d0384184..c3e06c29 100644 --- a/datasette/views/table_create_alter.py +++ b/datasette/views/table_create_alter.py @@ -1166,10 +1166,6 @@ class TableAlterView(BaseView): if not db.is_mutable: return _error(["Database is immutable"], 403) - content_type = request.headers.get("content-type") or "" - if not content_type.startswith("application/json"): - return _error(["Invalid content-type, must be application/json"], 400) - try: data = await request.json() except json.JSONDecodeError as e: diff --git a/docs/json_api.rst b/docs/json_api.rst index c31ba756..893af634 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -1668,6 +1668,8 @@ The JSON write API Datasette provides a write API for JSON data. This is a POST-only API that requires an authenticated API token, see :ref:`CreateTokenView`. The token will need to have the specified :ref:`authentication_permissions`. +The request body is always parsed as JSON, regardless of the request's ``Content-Type`` header - a body that is not valid JSON returns a ``400`` error. Cross-site request forgery is prevented by Datasette's ``Origin`` and ``Sec-Fetch-Site`` header checks rather than by content type requirements. + The row-based write APIs can write :ref:`binary values in JSON ` using Datasette's Base64 representation for BLOB data. .. _ExecuteWriteView: diff --git a/existing-api.md b/existing-api.md index 55e9d6a2..c58f8f41 100644 --- a/existing-api.md +++ b/existing-api.md @@ -853,8 +853,8 @@ shape) and check permissions with additionally required for `alter: true` (403 `Permission denied for alter-table`). Immutable database → 403 `Database is immutable`. -- **Request** — requires `Content-Type: application/json` (else 400 - `"Invalid content-type, must be application/json"`). Body: +- **Request** — the body is parsed as JSON regardless of the request + `Content-Type` header (invalid JSON → 400). Body: | Field | Rules | |---|---| @@ -937,7 +937,7 @@ shape) and check permissions with does not change the SQLite schema. - **Permission:** `set-column-type` (403 `Permission denied`). -- **Request** (JSON content type required): `{"column": "name", +- **Request**: `{"column": "name", "column_type": {"type": "url", "config": {...}?} | null}`. Unknown keys/invalid structure → detailed 400 errors; unknown type → 400 `"Unknown column type: x"`. Default registered types (via the diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index 9e8fa356..30dd0429 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -254,11 +254,15 @@ Concerns: executes immediately. Decide the 1.0 rule (suggestion: confirmation only for schema-destroying operations, i.e. keep as is — but document it as a deliberate contract). -- **Content-type enforcement is inconsistent:** `/-/insert`, `/-/upsert`, +- ~~**Content-type enforcement is inconsistent:** `/-/insert`, `/-/upsert`, `/-/alter`, `/-/set-column-type` demand `Content-Type: application/json` (400 otherwise); `/-/create` parses the body as JSON regardless of content type; execute-write and the query CRUD endpoints accept both JSON - and form encodings. Pick one rule for JSON-only endpoints. + and form encodings. Pick one rule for JSON-only endpoints.~~ ✅ **Done** + — the lenient rule won: JSON-only write endpoints parse the body as JSON + regardless of `Content-Type` (CSRF protection comes from the + cross-origin header checks, not content types). This also fixed a 500 on + insert when the header was absent entirely. - **JSON-vs-HTML negotiation on POST differs per endpoint:** execute-write and canned queries key off `Accept: application/json` / a `_json` body field; the write API keys off nothing (always JSON); query store keys off diff --git a/tests/test_api_write.py b/tests/test_api_write.py index 14dedb73..840bd05b 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -394,13 +394,6 @@ async def test_insert_rows_post_body_too_large(tmp_path_factory): "Invalid JSON: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)" ], ), - ( - "/data/docs/-/insert", - {}, - "invalid_content_type", - 400, - ["Invalid content-type, must be application/json"], - ), ( "/data/docs/-/insert", [], @@ -582,11 +575,7 @@ async def test_insert_or_upsert_row_errors( json=input, headers={ "Authorization": "Bearer {}".format(token), - "Content-Type": ( - "text/plain" - if special_case == "invalid_content_type" - else "application/json" - ), + "Content-Type": "application/json", }, ) diff --git a/tests/test_column_types.py b/tests/test_column_types.py index 4e553771..cd308ec9 100644 --- a/tests/test_column_types.py +++ b/tests/test_column_types.py @@ -322,12 +322,6 @@ async def test_clear_column_type_api(ds_ct): "Invalid JSON: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)" ], ), - ( - {"column": "title", "column_type": {"type": "email"}}, - "invalid_content_type", - 400, - ["Invalid content-type, must be application/json"], - ), ( [], None, @@ -413,11 +407,7 @@ async def test_set_column_type_api_errors( kwargs = { "headers": { "Authorization": f"Bearer {token}", - "Content-Type": ( - "text/plain" - if special_case == "invalid_content_type" - else "application/json" - ), + "Content-Type": "application/json", } } if special_case == "invalid_json": diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index b5f16b30..040e645a 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -662,3 +662,58 @@ async def test_debug_endpoints_use_size_and_page_parameters(ds_error_shape, endp bad_page = await ds_error_shape.client.get(base + "&_page=0", actor={"id": "root"}) data = assert_canonical_error(bad_page, 400) assert data["errors"] == ["_page must be a positive integer"] + + +# Write endpoints parse the body as JSON regardless of Content-Type + + +@pytest.mark.asyncio +async def test_insert_works_without_content_type_header(ds_error_shape): + # Previously a 500 AttributeError + response = await ds_error_shape.client.post( + "/data/docs/-/insert", + content='{"row": {"id": 1, "title": "One"}}', + actor={"id": "root"}, + ) + assert response.status_code == 201 + assert response.json()["rows"][0]["title"] == "One" + + +@pytest.mark.asyncio +async def test_insert_works_with_form_content_type(ds_error_shape): + # Previously 400 "Invalid content-type, must be application/json" + response = await ds_error_shape.client.post( + "/data/docs/-/insert", + content='{"row": {"id": 2, "title": "Two"}}', + headers={"Content-Type": "application/x-www-form-urlencoded"}, + actor={"id": "root"}, + ) + assert response.status_code == 201 + + +@pytest.mark.asyncio +async def test_insert_form_encoded_body_is_invalid_json(ds_error_shape): + response = await ds_error_shape.client.post( + "/data/docs/-/insert", + content="title=Three", + headers={"Content-Type": "application/x-www-form-urlencoded"}, + actor={"id": "root"}, + ) + data = assert_canonical_error(response, 400) + assert data["errors"][0].startswith("Invalid JSON:") + + +@pytest.mark.asyncio +async def test_alter_and_set_column_type_ignore_content_type(ds_error_shape): + alter = await ds_error_shape.client.post( + "/data/docs/-/alter", + content='{"operations": [{"op": "add_column", "args": {"name": "extra"}}]}', + actor={"id": "root"}, + ) + assert alter.status_code == 200, alter.text + sct = await ds_error_shape.client.post( + "/data/docs/-/set-column-type", + content='{"column": "title", "column_type": {"type": "textarea"}}', + actor={"id": "root"}, + ) + assert sct.status_code == 200, sct.text From 60bac9439d5674f2a8e716568804c29ff43cd43a Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 23:21:42 +0000 Subject: [PATCH 041/131] Mark shape=object item done in recommendations Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- stable-api-recommendations.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index 30dd0429..4da23934 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -390,7 +390,8 @@ Concerns: required"`. The JSON behavior masks caller bugs; return 400 on both.~~ ✅ **Done** — all data formats now return 400; the HTML SQL editor page is unchanged. -3. **`_shape=object` HTTP 200 error** (§1b) — almost certainly unintended. +3. ~~**`_shape=object` HTTP 200 error** (§1b) — almost certainly unintended.~~ + ✅ Done — now 400 (fixed with §1b). 4. ~~**Row delete 500** (§1c) — inconsistent with every sibling endpoint.~~ ✅ Done — now 400. 5. **The "SQL Interrupted" error embeds an HTML fragment in the JSON `error` From 0d962deb05a047b3cecd680983505dcf645a3fac Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 23:34:17 +0000 Subject: [PATCH 042/131] Plain text SQL Interrupted errors in JSON responses The SQL time limit error embedded an HTML fragment (paragraph, textarea and script tags) as the error string in JSON responses. DatasetteError now accepts a plain_message which the exception handler prefers for JSON error bodies; the HTML error page keeps the rich message with the SQL textarea. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/handle_exception.py | 4 +++- datasette/views/base.py | 3 +++ datasette/views/database.py | 4 ++++ datasette/views/row.py | 4 ++++ stable-api-recommendations.md | 6 ++++-- tests/test_api.py | 12 +++--------- tests/test_error_shape.py | 22 ++++++++++++++++++++++ 7 files changed, 43 insertions(+), 12 deletions(-) diff --git a/datasette/handle_exception.py b/datasette/handle_exception.py index fc290dbc..e255ddf2 100644 --- a/datasette/handle_exception.py +++ b/datasette/handle_exception.py @@ -28,6 +28,7 @@ def handle_exception(datasette, request, exception): rich.get_console().print_exception(show_locals=True) title = None + plain_message = None if isinstance(exception, Base400): status = exception.status info = {} @@ -36,6 +37,7 @@ def handle_exception(datasette, request, exception): status = exception.status info = exception.error_dict message = exception.message + plain_message = exception.plain_message if exception.message_is_html: message = Markup(message) title = exception.title @@ -50,7 +52,7 @@ def handle_exception(datasette, request, exception): add_cors_headers(headers) if request.path.split("?")[0].endswith(".json"): body = dict(info) - body.update(error_body(message, status)) + body.update(error_body(plain_message or message, status)) return Response.json(body, status=status, headers=headers) info.update( { diff --git a/datasette/views/base.py b/datasette/views/base.py index a12ae050..88d16753 100644 --- a/datasette/views/base.py +++ b/datasette/views/base.py @@ -29,12 +29,15 @@ class DatasetteError(Exception): status=500, template=None, message_is_html=False, + plain_message=None, ): self.message = message self.title = title self.error_dict = error_dict or {} self.status = status self.message_is_html = message_is_html + # Plain text used for JSON error responses when message is HTML + self.plain_message = plain_message class View: diff --git a/datasette/views/database.py b/datasette/views/database.py index c9dcfa89..10dc66ae 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -819,6 +819,10 @@ class QueryView(View): title="SQL Interrupted", status=400, message_is_html=True, + plain_message=( + "SQL query took too long. The time limit is" + " controlled by the sql_time_limit_ms setting." + ), ) except sqlite3.DatabaseError as ex: query_error = str(ex) diff --git a/datasette/views/row.py b/datasette/views/row.py index 2a103180..d9a3deeb 100644 --- a/datasette/views/row.py +++ b/datasette/views/row.py @@ -200,6 +200,10 @@ class RowView(BaseView): title="SQL Interrupted", status=400, message_is_html=True, + plain_message=( + "SQL query took too long. The time limit is" + " controlled by the sql_time_limit_ms setting." + ), ) except (sqlite3.OperationalError, InvalidSql) as e: raise DatasetteError(str(e), title="Invalid SQL", status=400) diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index 4da23934..0d817612 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -394,9 +394,11 @@ Concerns: ✅ Done — now 400 (fixed with §1b). 4. ~~**Row delete 500** (§1c) — inconsistent with every sibling endpoint.~~ ✅ Done — now 400. -5. **The "SQL Interrupted" error embeds an HTML fragment in the JSON `error` +5. ~~**The "SQL Interrupted" error embeds an HTML fragment in the JSON `error` value** (views/database.py:805-820). Error strings in the JSON API should - be plain text. + be plain text.~~ ✅ **Done** — `DatasetteError` gained a `plain_message` + used for JSON responses; the HTML error page keeps the rich version with + the SQL textarea. §8 is now fully resolved. --- diff --git a/tests/test_api.py b/tests/test_api.py index da0ebc07..9a96f14f 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -329,14 +329,8 @@ def test_sql_time_limit(app_client_shorter_time_limit): ) assert 400 == response.status expected_message = ( - "

SQL query took too long. The time limit is controlled by the\n" - 'sql_time_limit_ms\n' - "configuration option.

\n" - '\n' - "" + "SQL query took too long. The time limit is" + " controlled by the sql_time_limit_ms setting." ) assert response.json == { "ok": False, @@ -356,7 +350,7 @@ async def test_custom_sql_time_limit(ds_client): "/fixtures/-/query.json?sql=select+sleep(0.01)&_timelimit=5", ) assert response.status_code == 400 - assert response.json()["error"].startswith("

SQL query took too long.") + assert response.json()["error"].startswith("SQL query took too long.") @pytest.mark.asyncio diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index 040e645a..44856b36 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -717,3 +717,25 @@ async def test_alter_and_set_column_type_ignore_content_type(ds_error_shape): actor={"id": "root"}, ) assert sct.status_code == 200, sct.text + + +# SQL Interrupted errors carry plain text in JSON, not an HTML fragment + + +@pytest.mark.asyncio +async def test_sql_interrupted_json_error_is_plain_text(ds_client): + response = await ds_client.get( + "/fixtures/-/query.json?sql=select+sleep(0.01)&_timelimit=5" + ) + data = assert_canonical_error(response, 400) + assert "<" not in data["error"] + assert data["error"].startswith("SQL query took too long.") + + +@pytest.mark.asyncio +async def test_sql_interrupted_html_page_keeps_rich_error(ds_client): + response = await ds_client.get( + "/fixtures/-/query?sql=select+sleep(0.01)&_timelimit=5" + ) + assert response.status_code == 400 + assert " Date: Mon, 6 Jul 2026 23:34:17 +0000 Subject: [PATCH 043/131] Remove working analysis documents existing-api.md and stable-api-recommendations.md were working documents for the 1.0 API consistency review. Their content remains available in this branch history; they are not intended to merge to main. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- existing-api.md | 1244 --------------------------------- stable-api-recommendations.md | 469 ------------- 2 files changed, 1713 deletions(-) delete mode 100644 existing-api.md delete mode 100644 stable-api-recommendations.md diff --git a/existing-api.md b/existing-api.md deleted file mode 100644 index c58f8f41..00000000 --- a/existing-api.md +++ /dev/null @@ -1,1244 +0,0 @@ -# Datasette JSON API — As Implemented - -This document describes the JSON API of this Datasette codebase (version `1.0a35`) as -derived directly from the source code. It intentionally ignores the existing `docs/` -directory: every claim below is based on the route table in `datasette/app.py` -(`Datasette._routes()`, app.py:2507-2767) and the view implementations in -`datasette/views/`. - -## Contents - -- [Cross-cutting behavior](#cross-cutting-behavior) -- [Instance endpoints](#instance-endpoints) -- [Database endpoints](#database-endpoints) -- [Table and row read endpoints](#table-and-row-read-endpoints) -- [The write API](#the-write-api) -- [Stored (canned) queries API](#stored-canned-queries-api) -- [Authentication and tokens](#authentication-and-tokens) -- [Appendix: registered actions (permissions)](#appendix-registered-actions-permissions) - ---- - -## Cross-cutting behavior - -### URL formats and content negotiation - -- Most read endpoints are registered with an optional format suffix: - `/(...)(\.(?Pjson))?$`. The bare path returns HTML; the `.json` - extension returns JSON. -- Table, row and query routes accept any `\w+` format extension; formats other - than the built-in `html`, `json`, `csv`, `blob` must be provided by a plugin - via `register_output_renderer`, otherwise the request 404s. -- HTML responses include a `Link: <...>; rel="alternate"; - type="application/json+datasette"` header pointing at the `.json` variant - (views/base.py:141-159), unless the view opts out with - `has_json_alternate = False`. -- Database, table, row and query names in paths are **tilde-encoded** - (a percent-encoding variant using `~` as the escape character; - utils/__init__.py `_TILDE_ENCODING_SAFE`). Multi-column primary keys in row - URLs are comma-separated. -- JSON responses are always compact `json.dumps` output serialized by - `CustomJSONEncoder`; there is no pretty-printing query parameter. Binary - values are serialized as `{"$base64": true, "encoded": "..."}`. -- Success content type: `application/json; charset=utf-8` - (`_shape=array&_nl=on` responses use `text/plain`). - -### Success envelope and stability marker - -Every JSON endpoint that returns an object includes `"ok": true` on -success. `JsonDataView` injects it automatically for dict responses -(views/special.py); the homepage, jump, schema, permission-debug and -autocomplete views add it explicitly. The former top-level-array endpoints -(`/-/plugins`, `/-/databases`, `/-/actions`) now return objects wrapping -their arrays (`{"ok": true, "plugins": [...]}` etc.). - -JSON endpoints that are **not part of the documented API** include a -marker key (`UNSTABLE_API_MESSAGE` in utils/__init__.py): - -```json -"unstable": "This API is not part of Datasette's stable interface and may change at any time" -``` - -Currently: the homepage (`/.json`, `/-/.json`), `/db/-/queries/analyze`, -`POST /db/-/queries/store`, `/db//-/definition`, -`/db/-/query/parameters`, `/db/-/execute-write/analyze` and the -`POST /-/permissions` playground response. - -### Error shape (canonical) - -Every JSON error response uses one canonical shape, built by `error_body()` -(utils/__init__.py): - -```json -{ - "ok": false, - "error": "all messages joined with '; '", - "errors": ["message", "..."], - "status": 404 -} -``` - -- `errors` is a list of one or more message strings (multi-message - validation errors, e.g. per-row insert errors, list them all). -- `error` is the messages joined with `"; "`. -- `status` always matches the HTTP status code. - -The shape is produced by four code paths, all delegating to `error_body()`: - -1. **Exception handler** (handle_exception.py) — `NotFound`, - `DatasetteError`, `BadRequest` etc. on `.json` paths. `DatasetteError` - `error_dict` context keys are merged in; the legacy `title` key is no - longer emitted in JSON (it survives in the HTML error template context). -2. **The `_error()` helper** (views/base.py:183-184) — the write API, - stored-query API, execute-write and permission-denied paths. -3. **JSON renderer errors** (renderer.py) — SQL errors on table/query - endpoints return HTTP 400 with the canonical keys **plus** the context - keys of the response it could not produce: - - ```json - {"ok": false, "error": "no such table: x", "errors": ["no such table: x"], - "status": 400, "rows": [], "truncated": false} - ``` - - Invalid `_shape=` values and `_shape=object` misuse (on queries or - pk-less tables) also return canonical 400 errors. -4. **Permission debug endpoints** (`/-/allowed`, `/-/rules`, `/-/check`, - POST `/-/permissions`) — canonical shape (previously bare - `{"error": ...}` objects). - -Method-not-allowed responses return HTTP 405 with the canonical shape when -the path ends in `.json` or the request content type is `application/json`; -plain text otherwise (views/base.py). - -**`Forbidden` handling:** when a view raises `Forbidden` (e.g. via -`ensure_permission`), the default `forbidden()` plugin hook returns the -canonical JSON error with status 403 when the path ends in `.json` or the -request carries an `Accept: application/json` / `Content-Type: -application/json` header; other requests get an HTML error page -(forbidden.py). Endpoints that check permissions themselves return -`_error(..., 403)` JSON directly. - -### CORS - -When Datasette is started with `--cors`, responses gain -(utils/__init__.py:1297-1302): - -``` -Access-Control-Allow-Origin: * -Access-Control-Allow-Headers: Authorization, Content-Type -Access-Control-Expose-Headers: Link -Access-Control-Allow-Methods: GET, POST, HEAD, OPTIONS -Access-Control-Max-Age: 3600 -``` - -### CSRF / cross-origin protection - -Datasette uses header-based cross-origin protection -(`CrossOriginProtectionMiddleware`, csrf.py:67-178) rather than CSRF tokens -for API calls. For non-GET/HEAD/OPTIONS requests: - -1. Requests carrying `Authorization: Bearer ...` **and no `Cookie` header** - bypass the check entirely (csrf.py:98-110). -2. Otherwise `Sec-Fetch-Site` must be `same-origin` or `none`; other values → 403. -3. If neither `Sec-Fetch-Site` nor `Origin` is present (curl, API clients), - the request passes. -4. Fallback: `Origin` must exactly match the request scheme/host/port → else 403. - -Plain JSON API clients (no cookies, no browser headers) are never blocked; -`Content-Type: application/json` itself plays no role in the CSRF decision. - -### Settings that govern the API - -From `SETTINGS` (app.py:197-287): `default_page_size` (100), -`max_returned_rows` (1000), `max_insert_rows` (100), `sql_time_limit_ms` -(1000), `default_facet_size` (30), `facet_time_limit_ms` (200), -`allow_facet` (true), `allow_download` (true), `allow_signed_tokens` (true), -`default_allow_sql` (true), `max_signed_tokens_ttl` (0), `default_cache_ttl` -(5), `allow_csv_stream` (true), `max_csv_mb` (100), `force_https_urls` -(false), `trace_debug` (false), `base_url` ("/"). - -### The JSON renderer: `_shape`, `_nl`, `_json`, `_json_infinity` - -`json_renderer` (renderer.py:31-126) processes `.json` output for table, row -and query views (but **not** for the instance/database/debug endpoints, which -build JSON directly): - -- **`_shape`** (default `objects`): - - `objects` — `{"ok": true, "rows": [{col: val}, ...], "truncated": false, ...}` - - `arrays` — same envelope, each row a list of values - - `array` — response body is a bare JSON array of row objects - - `arrayfirst` — bare JSON array of the first column's values - - `object` — table views only: an object keyed by primary-key string. - On queries or tables without primary keys: a canonical 400 error - (`_shape=object is only available on tables` / - `_shape=object not available for tables with no primary keys`). - - anything else — canonical HTTP 400 error `Invalid _shape: x` -- **`_nl=on`** — with `_shape=array` only: newline-delimited JSON, `text/plain`. -- **`_json=COLUMN`** (repeatable) — parse that column's string values with - `json.loads` so they nest as JSON; parse failures leave the value unchanged. -- **`_json_infinity=1`** — preserve `Infinity`/`-Infinity`; by default they - are replaced with `null`. -- `columns` is stripped from dict-shaped output unless `?_extra=columns` was - requested (renderer.py:110-113). -- If a SQL error occurred, `_shape` is ignored, HTTP status is 400 and the - envelope carries the canonical error keys alongside `rows`/`truncated`. - -### The `?_extra=` system - -Table, row and query JSON responses support `?_extra=` (repeatable and/or -comma-separated, extras.py:9-14) to add keys to the response. Extras are -scope-registered (`ExtraScope.TABLE` / `ROW` / `QUERY`) and only **public** -extras are available over JSON (extras.py:73-92). Unknown extra names (and -internal HTML-only names) on data formats return 400 -`Unknown _extra: `; HTML pages ignore them. The available names per -scope are listed with the relevant endpoints below. - ---- - -## Instance endpoints - -Most of these are implemented with `JsonDataView` (views/special.py:30-79): -GET-only; bare path renders an HTML page (`show_json.html`), `.json` returns -the data; permission defaults to `view-instance` and denial raises -`Forbidden` → **HTML** 403 page. - -### GET / - -Routes: `/(\.(?Pjsono?))?$` and `/-/(\.(?Pjsono?))?$` -(app.py:2517-2518); `/-` permanently redirects to `/-/`. `IndexView` -(views/index.py:22-189). `GET /.json`, `/.jsono` and `/-/.json` return JSON. - -- **Permission:** `view-instance` (denied → 403). Databases and tables are - further filtered by `view-database` / `view-table` for the actor. -- **Parameters:** `_sort=relationships` sorts each database's truncated table - list by foreign-key relationship count. -- **JSON response** (index.py:147-161) — includes `ok: true` plus: - - `databases` — a **list** of database objects (undocumented API, subject - to change). Each item: `name`, `hash` (or null), `color`, `path`, - `tables_and_views_truncated` (up to 5 items: `name`, `columns`, - `primary_keys`, `count` (int or null), `hidden`, `fts_table`, - `num_relationships_for_sorting`, `private`; view items are just - `{"name", "private"}`), `tables_and_views_more` (bool), `tables_count`, - `table_rows_sum`, `show_table_row_counts`, `hidden_table_rows_sum`, - `hidden_tables_count`, `views_count`, `private`. - - `metadata` — instance metadata object. - -### GET /-/versions(.json) - -`JsonDataView` over `Datasette._versions` (app.py:2548-2551, 2171-2245). -Permission `view-instance`. No parameters. - -Response keys: `python` (`{version, full}`), `datasette` (`{version}` plus -optional `note`), `asgi` (`"3.0"`), `uvicorn` (string or null), `sqlite` -(`{version, fts_versions, extensions, compile_options}`; `extensions` -includes `json1` and optionally `spatialite`), `pysqlite3` (only when -running under pysqlite3). - -### GET /-/plugins(.json) - -app.py:2552-2557, `Datasette._plugins` (app.py:2247-2266). Permission -`view-instance`. - -- **Parameters:** `?all=1` — include Datasette's built-in default plugins - (filtered out by default). -- **Response:** `{"ok": true, "plugins": [...]}` — each plugin is - `{"name", "static", "templates", "version", "hooks"}`, sorted by name. - -### GET /-/settings(.json) - -app.py:2558-2561. Permission `view-instance`. No parameters. Returns a flat -object mapping every setting name (see [Settings](#settings-that-govern-the-api)) -to its effective value. - -### GET /-/config(.json) - -app.py:2562-2565. Permission `view-instance`. No parameters. Returns the full -`datasette.yaml` configuration dict passed through -`redact_keys(config, ("secret", "key", "password", "token", "hash", "dsn"))` -(app.py:2502-2505) — any dict key containing one of those substrings has its -value replaced by `"***"` (utils/__init__.py:1532-1556). - -### GET /-/threads(.json) - -app.py:2566-2569, `Datasette._threads` (app.py:2268-2285). Permission -**`permissions-debug`** (exposes runtime internals). No parameters. - -Response: `num_threads`, `threads` (list of `{name, ident, daemon}`), -`num_tasks`, `tasks` (asyncio task repr strings). When the -`num_sql_threads` setting is 0 the response is exactly -`{"num_threads": 0, "threads": []}`. - -### GET /-/databases(.json) - -app.py:2570-2573, `Datasette._connected_databases` (app.py:2157-2169). -Permission `view-instance`. No parameters. - -Response: `{"ok": true, "databases": [...]}` — each database is -`{"name", "route", "path", "size", "is_mutable", "is_memory", "hash"}`. -Only databases the actor is allowed to `view-database` are listed. - -### GET /-/actor(.json) - -app.py:2574-2579, registered with `permission=None` — **accessible to any -request including anonymous**. No parameters. - -Response: `{"ok": true, "actor": {...}}` or `{"ok": true, "actor": null}` (app.py:2287-2288). - -### GET /-/actions(.json) - -app.py:2580-2589. Permission **`permissions-debug`**. No parameters. - -Response: `{"ok": true, "actions": [...]}` — each action is -`{"name", "abbr", "description", "takes_parent", "takes_child", -"resource_class", "also_requires"}`, sorted by name (app.py:2290-2304). - -### GET /-/auth-token - -`AuthTokenView` (app.py:2590-2593, views/special.py:198-217). GET only, no -`.json` variant, HTML/redirect only. - -- **Parameter:** `token` — the one-time secret printed by `datasette --root`. -- Match → invalidates the token, sets the signed `ds_actor` cookie to - `{"id": "root"}` and 302-redirects to the homepage. Mismatch or reuse → - `Forbidden` → 403 HTML. - -### GET/POST /-/create-token - -`CreateTokenView` (app.py:2594-2597, views/special.py:727-856). **HTML form -endpoint only — there is no JSON request/response mode in this codebase** -(`has_json_alternate = False`; the POST body must be form-encoded, a JSON -content type raises `BadRequest` → 400). - -- **Gates** (each failure → `Forbidden` → 403): `allow_signed_tokens` must be - on; request must have an actor with an `id`; the actor must not itself be - token-derived. -- **POST fields:** `expire_type` (`""`/`minutes`/`hours`/`days`), - `expire_duration` (positive int), plus restriction checkboxes named - `all:`, `database::`, - `resource::

{{ action.name }} {% if action.abbr %}{{ action.abbr }}{% endif %}
:`. -- **Response:** HTML page containing the new `dstok_` token. -- Programmatic alternatives: `datasette create-token` CLI or - `datasette.create_token()`. - -### GET /-/api - -`ApiExplorerView` (app.py:2598-2601, views/special.py:859-1020). HTML API -explorer, GET only. Permission `view-instance` (403 on denial). - -### GET /-/jump(.json) - -`JumpView` (app.py:2602-2605, views/special.py:1023-1201). The route allows -an optional `.json` suffix but the view **always returns JSON**. - -- **Permission:** none checked directly; results are filtered via - `allowed_resources_sql` for the current actor (default items come from the - `jump_items_sql` plugin hook). -- **Parameter:** `q` — whitespace-split terms matched as a case-insensitive - `%term1%term2%` LIKE pattern. -- **Response:** `{"ok": true, "matches": [...], "truncated": bool}`; each match: - `name`, `url`, `type` (`database`/`table`/`view`/`query`/plugin-defined), - `description`, optional `display_name`. Capped at 100 matches. - -### GET /-/schema(.json|.md) - -`InstanceSchemaView` (app.py:2610-2613, views/special.py:1257-1293). - -- **Permission:** no explicit check; only databases the actor can - `view-database` are included (others silently omitted). -- **Formats:** no extension → HTML; `.json` → - `{"ok": true, "schemas": [{"database": name, "schema": "..."}]}`; `.md` → - `text/markdown` rendering. - -### GET/POST /-/logout - -`LogoutView` (app.py:2614-2617, views/special.py:220-238). HTML endpoint. -GET renders a confirmation page (or redirects if anonymous); POST deletes the -`ds_actor` cookie and 302-redirects to `/`. - -### GET/POST /-/permissions - -`PermissionsDebugView` (app.py:2618-2621, views/special.py:241-295). No -`.json` route. Both methods require `view-instance` **and** -`permissions-debug` (403 on denial). - -- **GET** — HTML permission-check log; `?filter=all|exclude-yours|only-yours`. -- **POST** — form-encoded `actor` (JSON string), `permission`, optional - `resource_1`, `resource_2`; returns **JSON** - `{"action", "allowed", "resource": {"parent", "child", "path"}}` plus - `actor_id` when present. Errors: unknown action → 404; child without - parent → 400 (both canonical error shape). - -### GET /-/allowed(.json) - -`AllowedResourcesView` (app.py:2622-2625, views/special.py:298-460). Bare -path always renders the HTML form; `.json` returns JSON. - -- **Permission:** none — reports the **current actor's own** allowed - resources. Items gain a `reason` field if the actor also holds - `permissions-debug`. -- **Parameters:** `action` (required; missing → 400 canonical error, unknown - → 404), `parent`, `child` (requires `parent`), `_page` (default 1), - `_size` (default 50, maximum 200, accepts `max`; out-of-range → 400). -- **Response:** `{"action", "actor_id", "page", "page_size", "total", - "items": [{"parent", "child", "resource"}]}` with optional `next_url` / - `previous_url`. - -### GET /-/rules(.json) - -`PermissionRulesView` (app.py:2626-2629, views/special.py:463-584). -Permission `view-instance` **and** `permissions-debug`. Parameters and error -shapes as `/-/allowed`. Response items: -`{"parent", "child", "resource", "allow" (1|0), "reason", "source_plugin"}`. - -### GET /-/check(.json) - -`PermissionCheckView` (app.py:2630-2633, views/special.py:633-662). -Permission `permissions-debug`. Parameters `action` (required), `parent`, -`child`. Checks the **current request's actor**; response -`{"action", "allowed", "resource": {...}}` plus `actor_id`. - -### GET/POST /-/messages - -`MessagesDebugView` (app.py:2634-2637, views/special.py:703-724). HTML debug -tool for flash messages; permission `view-instance`; POST is form-encoded -(`message`, `message_type` = INFO/WARNING/ERROR/all) and 302-redirects. - -### GET /-/allow-debug - -`AllowDebugView` (app.py:2638-2641, views/special.py:665-700). GET only, HTML -only, **no permission required**. Parameters `actor` and `allow` (JSON -strings); renders the result of `actor_matches_allow()` in the page. - -### GET /-/patterns - -Pattern portfolio page (app.py:2642-2645). HTML only; not part of the JSON API. - -### GET /-/debug/autocomplete - -`AutocompleteDebugView` (app.py:2646-2649, views/special.py:94-195). HTML -debug page for the table autocomplete API; permission `view-instance` plus -`view-table` when `?database=&table=` are supplied. - ---- - -## Database endpoints - -### GET /\.db - -Downloads the raw SQLite file. Route → `database_download` -(app.py:2650-2653; views/database.py:533-570). - -- **Permission:** `view-database-download` (denied → `Forbidden` → 403 HTML). -- **Other gates:** unknown database → 404 `"Invalid database"`; in-memory - database → 404; `allow_download` off **or** mutable database → - `Forbidden("Database download is forbidden")`; no file path → 404. -- **Response:** streamed `application/octet-stream` with a - `content-disposition` attachment; immutable databases with a known hash set - `Etag` and honor `If-None-Match` → 304. - -### GET /\(.json) - -`DatabaseView` (app.py:2654-2657; views/database.py:71-277). Only `html` and -`json` formats are accepted; any other extension → 404 `"Invalid format: ..."`. - -- **Permission:** `view-database` via `check_visibility` (denied → - `Forbidden` → 403 HTML). Table/view listings are filtered by `view-table`; - stored queries by `view-query`. -- **Parameters:** - - `?sql=` — non-blank value 302-redirects to `//-/query?...` - preserving the query string and format. - - No `?_extra=` and no `_shape` support — the JSON is built directly and - returned via `Response.json`, bypassing the JSON renderer - (views/database.py:189-212). -- **JSON response** (all keys always present): - - `ok` — always `true` - - `database` — name; `private` — bool; `path` — URL path; `size` — bytes - - `tables` — list (includes hidden tables), each: - `name`, `columns` (names), `primary_keys`, `count` (int or null, - time-boxed), `count_truncated` (bool — count is a capped lower bound), - `hidden`, `fts_table`, `foreign_keys` (`{incoming: [...], outgoing: [...]}` - of `{other_table, column, other_column}`), `private` - - `hidden_count` — number of hidden tables - - `views` — list of `{name, private}` - - `queries` — **up to 5** stored queries (canonical stored-query objects, - see the stored-queries section); `queries_more` (bool); - `queries_count` (total visible) - - `allow_execute_sql` — bool for this actor - - `table_columns` — `{table: [columns]}`, empty `{}` unless - `allow_execute_sql` (views map to `[]`) - - `metadata` — database metadata dict - -### GET /\/-/query(.json) — arbitrary SQL - -`QueryView` (app.py:2691-2694; views/database.py:573-1130). The same class -also executes stored queries dispatched from the table route (see stored -queries section). - -- **Permission:** `execute-sql` on the database via `check_visibility` - (denied → `Forbidden` → 403 HTML). -- **Parameters:** - - `sql` — SQL to run. Must pass `validate_sql_select` - (utils/__init__.py:345-354): after stripping `--` comment lines it must - start with `select`, `with` or an `explain` variant, and must not contain - `pragma` (except allowlisted `pragma_*()` table-valued functions). - Failure → 400 `DatasetteError` titled `"Invalid SQL"` → JSON - `{"ok": false, "error": "Statement must be a SELECT", "status": 400, - "title": "Invalid SQL"}`. - - Any other `name=value` pair supplies the `:name` named parameter; missing - parameters default to `""`. Names starting with `_` are excluded. - - `_timelimit` — per-request SQL time limit in ms. - - `_shape`, `_nl`, `_json`, `_json_infinity` — see the JSON renderer section. - - `_extra` — QUERY-scope extras: `columns`, `debug`, `request`, - `render_cell`, `query` (`{"sql", "params"}`), `metadata`, `database`, - `database_color`, `private`, `extras`. -- **Response** (default shape): - `{"ok": true, "rows": [{col: val}, ...], "truncated": false}` plus any - requested extras. `truncated: true` when the result hit `max_returned_rows`. -- **Errors:** - - SQLite errors (e.g. `no such table`) are **not** raised — they surface as - HTTP 400 `{"ok": false, "error": "", "rows": [], "truncated": false}`. - - Time limit → 400 titled `"SQL Interrupted"` (the `error` value contains - an HTML fragment). - - `?sql=` omitted or blank → 400 `"?sql= is required"` for all data - formats (`.json`, `.csv`, plugin formats). The HTML page remains the - SQL editor. -- `.csv` streams CSV; unknown extensions → 404. - -### GET /\/-/query/parameters - -`QueryParametersView` (app.py:2687-2690; views/stored_queries.py:26-51). - -- **Permission:** `execute-sql` → 403 JSON - `{"ok": false, "errors": ["Permission denied: need execute-sql"]}`. -- **Parameters:** only `sql` (default `""`); any other key → 400 - `"Invalid keys: ..."`. -- **Response:** 200 `{"ok": true, "parameters": ["name1", ...]}`. SQL with a - parameter beginning `_` → 400 `"Magic parameters are not allowed"`. -- Responses carry `Content-Security-Policy: frame-ancestors 'none'` and - `X-Frame-Options: DENY`. - -### POST /\/-/create - -`TableCreateView` (app.py:2658; views/table_create_alter.py:785-962). -GET → 405. Body is parsed as JSON regardless of content type; invalid JSON → -400 `{"ok": false, "errors": ["Invalid JSON: ..."]}`. - -- **Permissions** (all denials → 403 canonical error JSON, - all checked at the **database** level): - - `create-table` — always required (`["Permission denied"]`) - - `insert-row` — if `rows`/`row` provided (`need insert-row`) - - `update-row` — if `replace: true` (`need update-row`) - - `alter-table` — if `alter: true` on an **existing** table - (`need alter-table`); when the table does not exist yet and rows are - supplied, alter is enabled automatically. -- **Request schema** (pydantic `CreateTableRequest`, extra keys forbidden → - 400 `"Invalid keys: a, b"`): - - `table` (required) — must match `^(?!sqlite_)[^\n]+$` - - `rows` (list of objects) / `row` (single object) — mutually exclusive - - `columns` — list of `{name, type, fk_table, fk_column, not_null, - default, default_expr}`; mutually exclusive with `rows`/`row`; `type` one - of `text`/`integer`/`float`/`blob` (default `text`); `default` and - `default_expr` mutually exclusive; `default_expr` one of - `current_timestamp`, `current_date`, `current_time`, `current_unixtime`, - `current_unixtime_ms`. At least one of `columns`/`rows`/`row` required. - - `pk` (string) / `pks` (list) — mutually exclusive. For an existing table - a differing pk → 400 `"pk cannot be changed for existing table"`. - - `ignore` / `replace` (bools) — mutually exclusive; require `row`/`rows` - and `pk`/`pks`. - - `alter` (bool) — add missing columns when inserting into an existing table. -- **Success** — **201**: - ```json - {"ok": true, "database": "...", "table": "...", - "table_url": "https://.../db/table", "table_api_url": "https://.../db/table.json", - "schema": "CREATE TABLE ...", "row_count": 2} - ``` - `row_count` only when rows were inserted. Write failures → 400 - `{"ok": false, "errors": [""]}`. Emits `create-table` / - `insert-rows` / `alter-table` events. - -### POST /\/-/execute-write - -`ExecuteWriteView` (app.py:2679-2682; views/execute_write.py:236-476). GET on -the same path renders an HTML form (requires `execute-write-sql`). - -- **Permission (POST):** `execute-write-sql` → 403 - `{"ok": false, "errors": ["Permission denied: need execute-write-sql"]}`; - immutable database → 403 `["Database is immutable"]`. -- **Per-statement permissions:** the SQL is analyzed - (`decision_for_write_sql_operation`, write_sql.py:63-189) and each - operation must pass: - - | Operation | Requirement | - |---|---| - | `select` / internal ops / function calls | ignored | - | read of a table | `view-table` on that table | - | `insert` or `update` | **all of** `insert-row`, `update-row`, `delete-row` on the table | - | `delete` | `delete-row` | - | `create table` | `create-table` on the database | - | `alter table`, `create index`, `drop index` | `alter-table` on the table | - | `drop table` | `drop-table` | - | `VACUUM`, virtual-table writes, shadow-table writes | rejected outright (403) | - | statements touching attached databases | rejected (403) | - -- **Body:** JSON (`{"sql": ..., "params": {...}}` — only those two keys) or - form-encoded (`sql` plus one field per parameter, `_sql_param_` prefix - stripped). Validation errors (400): `"SQL is required"`, - `"params must be a dictionary"`, `"Unknown parameters: a, b"`, - `"Magic parameters are not allowed"`, `"Could not analyze query: ..."`, - `"Use /-/query for read-only SQL; this endpoint only executes writes"`. -- **JSON is returned when** the body was JSON, `Accept: application/json`, or - a truthy `_json` field is present; otherwise HTML. -- **Success** — 200: - ```json - {"ok": true, "message": "Query executed, 1 row affected", "rowcount": 1, - "rows": [], "truncated": false, - "analysis": [{"operation": "insert", "database": "db", "table": "t", - "required_permission": "insert-row, update-row, delete-row", - "source": null}]} - ``` - `rows` is populated by `RETURNING` clauses. SQLite errors → 400 - `{"ok": false, "errors": [""]}`. Anti-framing headers on all - responses. - -### GET /\/-/execute-write/analyze - -`ExecuteWriteAnalyzeView` (app.py:2675-2678; views/execute_write.py:479-507). - -- **Permission:** `execute-write-sql` → 403 `errors` JSON. -- **Parameters:** only `sql` allowed (else 400 `"Invalid keys: ..."`). -- **Response** — 200 even when analysis fails (`ok: false` in body): - `{"ok", "parameters", "analysis_error", "analysis_rows": - [{operation, database, table, required_permission, source, allowed}], - "execute_disabled", "execute_disabled_reason"}`. `allowed` is a per-actor - permission check result (true/false/null). - -### GET /\/-/foreign-key-targets - -`DatabaseForeignKeyTargetsView` (app.py:2659-2662; -views/table_create_alter.py:965-1005). - -- **Parameter:** `table` (optional) — only used for the permission check. -- **Permission:** `create-table` on the database, **or** `alter-table` on - `?table=` when it names an existing table. Neither → 403 - `{"ok": false, "errors": ["Permission denied: need create-table"]}`. -- **Response:** 200 `{"ok": true, "database": "...", "targets": - [{"fk_table", "fk_column", "type"}]}` — every non-hidden table with exactly - one primary-key column; `type` is the pk's SQLite type affinity. - -### GET /\/-/schema(.json|.md) - -`DatabaseSchemaView` (app.py:2683-2686; views/special.py:1296-1329). - -- **Permission:** `view-database` (denied → `Forbidden` → 403 HTML). -- **Unknown database** → 404; for `.json`: - `{"ok": false, "error": "Database not found"}`. The permission check runs - first, so unauthorized actors cannot probe for database existence. -- **Responses:** `.json` → 200 `{"ok": true, "database": "", "schema": ""}` - (concatenated `sqlite_master.sql` joined with `;\n`); `.md` → - `text/markdown`; no extension → HTML. - ---- - -## Table and row read endpoints - -### GET /\/\.json - -Route `r"/(?P[^\/\.]+)/(?P
[^\/\.]+)(\.(?P\w+))?$"` → -`table_view` (app.py:2711-2714; views/table.py:1670). Serves both tables and -SQL views. GET/HEAD only — POST returns a plain-text 405. If the name is -neither a table nor a view but matches a stored query, the request is -dispatched to `QueryView` (views/table.py:1703-1712). - -**Permission:** `view-table` via `check_visibility`; denial raises -`Forbidden` → **HTML** 403 page even for `.json`. Unknown table → -`TableNotFound` → 404 (JSON error shape for `.json` paths). - -**Default JSON keys** (views/table.py:2308-2332 + renderer): - -| Key | Meaning | -|---|---| -| `ok` | `true` when data was retrieved without error | -| `next` | pagination token string, or `null` on the last page | -| `next_url` | absolute URL of the next page, or `null` on the last page | -| `rows` | list of row objects `{column: value}` (default `_shape=objects`) | -| `truncated` | always present; `false` for table pages | - -`columns` is computed but removed unless `?_extra=columns` was requested. -When there is a next page the response carries a -`Link: ; rel="next"` header (views/table.py:1911-1912). - -**`?_extra=` options** (TABLE scope; registry -views/table_extras.py:1197-1235; unknown names silently ignored): - -| `_extra=` | Returns | -|---|---| -| `count` | total matching-row count, computed with a `limit 10001` subquery so it caps at 10001; `null` with `_nocount` or on count timeout. Requesting `count` implicitly includes `count_truncated` | -| `count_truncated` | `true` when `count` hit the counting limit (the real count is at least the reported value) | -| `count_sql` | the SQL used for the count | -| `facet_results` | `{"results": {name: facet}, "timed_out": [...]}`; each facet: `{name, type, hideable, toggle_url, results: [{value, label, count, toggle_url, selected}], truncated}` | -| `facets_timed_out` | facet names that exceeded `facet_time_limit_ms` | -| `suggested_facets` | `[{name, toggle_url, (type)}]`; empty when suggestion is disabled or paginating | -| `human_description_en` | English description of filters + sort | -| `next_url` | absolute URL of the next page or `null` | -| `columns` | column names of the returned rows | -| `all_columns` | all table columns regardless of `_col`/`_nocol` | -| `primary_keys` | pk column names (empty for rowid tables and views) | -| `display_columns` | HTML-oriented column metadata | -| `render_cell` | per-row plugin-rendered HTML strings | -| `debug` | `{url_vars, resolved, nofacet, nosuggest}` — explicitly unstable | -| `request` | `{url, path, full_path, host, args}` | -| `query` | `{sql, params}` of the main query | -| `column_types` | `{column: {type, config}}` assigned column types | -| `set_column_type_ui` | UI helper, `null` unless actor has `set-column-type` | -| `metadata` | table metadata dict including column descriptions | -| `extras` | self-describing list of all available extras | -| `database`, `table`, `database_color` | identity/display values | -| `renderers` | `{format_name: url}` of formats that can render this data | -| `custom_table_templates` | template lookup list | -| `sorted_facet_results` | facets as a display-ordered list | -| `table_definition` | `CREATE TABLE` SQL | -| `view_definition` | `CREATE VIEW` SQL, `null` for tables | -| `is_view` | boolean | -| `private` | `true` if visible to this actor but not anonymously | -| `expandable_columns` | `[[foreign_key, label_column_or_null], ...]` | -| `form_hidden_args` | pairs of `_`-prefixed args for HTML forms | - -Non-public extras (`actions`, `filters`, `display_rows`) are HTML-only and -never appear in JSON. `_extra=_html` expands to the full HTML bundle -(views/table_extras.py:1162-1194). Any `_facet*` argument implicitly adds -`facet_results`; `_shape=object` implicitly adds `primary_keys` -(views/table.py:2252-2256). There is **no** `filtered_table_rows_count` -extra — it was replaced by `count`. - -**Column filters `?__=`** (filters.py:260-427). Any -querystring key not starting with `_` is a filter; bare `?column=value` means -`exact`. Columns whose names start with `_` can be filtered as -`?_col__exact=`. Operators: - -| op | SQL | -|---|---| -| `exact` | `"col" = :p` (default) | -| `not` | `"col" != :p` | -| `contains` / `notcontains` | `like '%v%'` / `not like '%v%'` | -| `endswith` / `startswith` | `like '%v'` / `like 'v%'` | -| `gt` / `gte` / `lt` / `lte` | `>` `>=` `<` `<=` (numeric strings cast to int) | -| `like` / `notlike` | raw `like` / `not like` pattern | -| `glob` | `glob` | -| `in` / `notin` | comma-separated list, or JSON array if the value starts with `[` | -| `arraycontains` / `arraynotcontains` | `[not] in (select value from json_each("col"))` (requires JSON1) | -| `date` | `date("col") = :p` | -| `isnull` / `notnull` | `is null` / `is not null` (no value) | -| `isblank` / `notblank` | `(is null or = '')` / opposite (no value) | - -**Special (underscore) parameters:** - -| Param | Behavior | -|---|---| -| `_where=SQL` | extra raw where clause (repeatable); requires `execute-sql` else 403 `"_where= is not allowed"` | -| `_search=q` | FTS against the table's FTS table | -| `_search_=q` | FTS restricted to one column; 400 if invalid | -| `_searchmode=raw` | pass the query straight to `match` | -| `_fts_table=` / `_fts_pk=` | override the FTS table / pk used for joins | -| `_through={"table","column","value"}` | filter via an incoming foreign key (repeatable, JSON value) | -| `_sort=col` / `_sort_desc=col` | sort; 400 if both given or column not sortable | -| `_next=token` | pagination token | -| `_size=N\|max` | page size; default `default_page_size` (100); `max` = `max_returned_rows` (1000); 400 on invalid | -| `_col=name` (repeatable) | return only pks + these columns; 400 on invalid | -| `_nocol=name` (repeatable) | exclude columns; 400 if invalid or a pk | -| `_labels=on` | expand every FK column into `{"value", "label"}` | -| `_label=col` (repeatable) | expand only the named FK column(s) | -| `_facet=col` | request a facet; 400 `"_facet= is not allowed"` when `allow_facet` off | -| `_facet_array=col` / `_facet_date=col` | typed facets | -| `_facet_size=N\|max` | facet bucket count, default 30, capped at `max_returned_rows` | -| `_nocount=1` | skip count (`count` extra → null) | -| `_nofacet=1` | skip facets and suggestions | -| `_nosuggest=1` | skip facet suggestions only | -| `_shape=` | see renderer section; `array`/`object` also force `_nocount` and `_nofacet` | -| `_nl=on` | NDJSON with `_shape=array` | -| `_json=col` / `_json_infinity=1` | renderer options | -| `_timelimit=ms` | custom SQL time limit | -| `_ttl=seconds` | `Cache-Control: max-age=N` (`0` → `no-cache`); default `default_cache_ttl` (5) | -| `_trace=1` | append `_trace` key (requires `trace_debug` setting) | -| `_extra=` | see above | - -**Pagination** is keyset-based for tables: `page_size + 1` rows are fetched; -`next` is built from the last row of the page — comma-joined tilde-encoded -primary-key values, prefixed by the sort value when sorted (`$null` for null -sort values) (views/table.py:2041-2111, 2421-2482). `next_url` is the -absolute URL with `_next` replaced. - -### GET /\/\.json (SQL views) - -Same code path with `is_view=True`. Differences: - -- No primary keys: `primary_keys` → `[]`; `_shape=object` fails; base query - has no `order by`. -- **Pagination is offset-based**: `_next` is an integer offset applied as - `limit N offset M` (views/table.py:2047-2049, 2438-2439) — unlike the - keyset tokens used for tables. -- `view_definition` returns the `CREATE VIEW` SQL; `table_definition` is null. - -### GET /\/\/\.json - -`RowView` (app.py:2715-2718; views/row.py:137). `` is comma-separated -tilde-encoded primary key values (rowid for rowid tables). - -- **Permission:** `view-table` (denied → `Forbidden` → 403 HTML). Missing row - → 404 `"Record not found: [...]"`. -- **Default JSON keys:** `ok`, `database`, `table`, `rows` (single-element - list), `primary_keys`, `primary_key_values`, `query_ms`, - `truncated: false`; `columns` only with `?_extra=columns`. -- **`?_extra=` (ROW scope):** `columns`, `primary_keys`, `render_cell`, - `debug`, `request`, `query`, `column_types`, `metadata`, `extras`, - `database`, `table`, `database_color`, `private`, `foreign_key_tables` - (incoming FKs with `count` and `link`; single-pk rows only). -- **Foreign-key label expansion does not apply to row JSON** — `_labels` has - no effect here; expansion happens only in the HTML path - (views/row.py:445-475). -- `_shape`, `_json`, `_nl`, `_json_infinity`, `_ttl` apply. - -### The .blob format - -`//
/.blob?_blob_column=col` (also on query pages) — -fetches raw binary bytes (blob_renderer.py:10-61). `_blob_column` required -(400 if missing/invalid); optional `_blob_hash` must equal the value's -SHA-256 (else 400 `"Link has expired..."`). Returns `application/binary` as a -download attachment. In JSON output, binary cells appear as -`{"$base64": true, "encoded": "..."}`. - -### GET /\/\/-/schema(.json|.md) - -`TableSchemaView` (app.py:2751-2754; views/special.py:1332-1378). - -- **Permission:** `view-table` via `ensure_permission` (denied → 403 HTML). -- **Responses:** `.json` → 200 `{"ok": true, "database", "table", "schema"}`; - `.md` → `text/markdown`; no extension → HTML. Missing table → 404 - `{"ok": false, "error": "Table not found"}` for `.json`. - -### GET /\/\/-/fragment - -`TableFragmentView` (app.py:2739-2742; views/table.py:1385-1418). -**HTML-only** — returns the `_table.html` partial; no JSON variant. Accepts -table querystring parameters plus `_row=` to render a single row. - -### GET /\/\/-/autocomplete - -`TableAutocompleteView` (app.py:2743-2746; views/table.py:1492-1595). Tables -only — views get 400 `"Autocomplete is only available for tables"`. - -- **Permission:** `view-table` (denied → `Forbidden` → 403). -- **Parameters:** `q` (matched with escaped `LIKE %q%` against pk columns and - the label column) and `_initial` (truthy: with empty `q`, return the 10 - most recent rows). Neither → `{"ok": true, "rows": []}`. -- **Response:** `{"rows": [{"pks": {pk_name: value}, "label": "..."}]}` — max - 10 items; 500 ms query budget with fallbacks, timing out to - `{"ok": true, "rows": []}`. - ---- - -## The write API - -All write endpoints return errors via `_error()` (the canonical error -shape) and check permissions with -`datasette.allowed()` directly, so their 403s are JSON (unlike the -`Forbidden`-raising read endpoints). Routes: app.py:2719-2762. - -### POST /\/\/-/insert - -`TableInsertView` (views/table.py:907-1194). - -- **Permissions:** `insert-row` on the table (denied → 403 - `["Permission denied"]`); `update-row` additionally required for - `replace: true` (403 `need update-row to use "replace"`); `alter-table` - additionally required for `alter: true` (403 - `Permission denied for alter-table`). Immutable database → 403 - `Database is immutable`. -- **Request** — the body is parsed as JSON regardless of the request - `Content-Type` header (invalid JSON → 400). Body: - - | Field | Rules | - |---|---| - | `row` | single object; mutually exclusive with `rows`; forces `return: true` | - | `rows` | list of objects; max `max_insert_rows` (default 100), else 400 `"Too many rows, maximum allowed is 100"` | - | `ignore` | skip rows whose pk already exists; mutually exclusive with `replace` | - | `replace` | replace rows with matching pks (needs `update-row`) | - | `alter` | add missing columns (needs `alter-table`) | - | `return` | include inserted rows in the response | - - One of `row`/`rows` required. Unknown keys → 400 `"Invalid parameter: ..."`. - Unless `alter`, row keys must be existing columns → per-row 400 - `"Row 0 has invalid columns: x, y"`. Values are validated against assigned - column types. -- **Response** — **201** `{"ok": true}`; with `return: true` also `rows` - (the rows as stored, re-fetched by rowid). SQLite errors during the write → - 400 with the message. Emits `insert-rows` (and possibly `alter-table`) - events. - -### POST /\/\/-/upsert - -`TableUpsertView` — subclasses insert (views/table.py:1197-1201). - -- **Permissions:** **both** `insert-row` and `update-row` (403 - `need both insert-row and update-row`); `alter: true` needs `alter-table`. -- **Request:** same as insert, except `ignore`/`replace` are rejected (400 - `"Upsert does not support ignore or replace"`) and **every row must contain - the table's primary key(s)** (per-row 400 - `Row 0 is missing primary key column(s): "id"` / `has null primary key`). -- **Response** — **200** (note: insert returns 201) `{"ok": true}`; with - `return: true`, `rows` re-fetched by pk. Emits `upsert-rows`. - -### POST /\/\/-/alter - -`TableAlterView` (views/table_create_alter.py:1130-1353). - -- **Permission:** `alter-table` (403 `need alter-table`); immutable → 403. -- **Request:** `{"operations": [{"op": ..., "args": {...}}, ...]}` — a - non-empty list, validated by pydantic (extra keys forbidden anywhere; - errors → 400 `location: message`): - - | `op` | `args` | - |---|---| - | `add_column` | `name` (required), `type` (`text`/`integer`/`float`/`blob`, default `text`), `not_null`, `default` xor `default_expr`; `not_null: true` requires a default | - | `rename_column` | `name`, `to` | - | `rename_table` | `to` (must not start `sqlite_`) | - | `alter_column` | `name` + at least one of `type`, `not_null`, `default`, `default_expr` | - | `drop_column` | `name` | - | `set_primary_key` | `columns` (non-empty list) | - | `reorder_columns` | `columns` (non-empty list) | - | `add_foreign_key` | `column`, `fk_table`, optional `fk_column` | - | `drop_foreign_key` | `column` | - | `set_foreign_keys` | `foreign_keys`: list of `{column, fk_table, fk_column?}` | - - `default_expr` must be one of the five `current_*` keywords. Operations are - applied in a single write transaction; any failure → 400. -- **Response** — 200: - ```json - {"ok": true, "database": "...", "table": "", - "table_url": "...", "table_api_url": "...", - "altered": true, "schema": "...", "before_schema": "...", - "operations_applied": 2} - ``` - -### POST /\/\/-/drop - -`TableDropView` (views/table.py:1320-1382). - -- **Permission:** `drop-table` (403 `Permission denied`); immutable → 403. -- **Confirmation flow:** without `{"confirm": true}` in the body, nothing is - dropped and a 200 preview is returned: - `{"ok": true, "database", "table", "row_count", - "message": "Pass \"confirm\": true to confirm"}`. With `confirm: true` → - 200 `{"ok": true}`. Emits `drop-table`. - -### POST /\/\/-/set-column-type - -`TableSetColumnTypeView` (views/table.py:1204-1317). Assigns a Datasette -*column type* (metadata stored in the internal `column_types` table) — it -does not change the SQLite schema. - -- **Permission:** `set-column-type` (403 `Permission denied`). -- **Request**: `{"column": "name", - "column_type": {"type": "url", "config": {...}?} | null}`. Unknown - keys/invalid structure → detailed 400 errors; unknown type → 400 - `"Unknown column type: x"`. Default registered types (via the - `register_column_types` hook): `url`, `email`, `json`, `textarea`. -- **Response** — 200 `{"ok": true, "database", "table", "column", - "column_type": {...} | null}`. - -### GET /\/\/-/foreign-key-suggestions - -`TableForeignKeySuggestionsView` (views/table_create_alter.py:1008-1127). -**GET only** (read-only despite living beside the write endpoints). - -- **Permission:** `alter-table` (403 `need alter-table`); views → 400 - `"Cannot suggest foreign keys for a view"`. -- **Response** — 200: `{"ok": true, "database", "table", - "row_check": {attempted, status, row_limit, sampled_rows, checked_options}, - "columns": [{column, type, affinity, current, - "suggestions": [{fk_table, fk_column, confidence, sampled_values, reasons}], - "options": [...]}]}`. Samples up to 500 rows within 50 ms/200 ms budgets. - -### POST /\/\/\/-/update - -`RowUpdateView` (views/row.py:781-870). - -- **Permissions:** `update-row` (403 `Permission denied`); `alter: true` - additionally requires `alter-table` (403 - `Permission denied for alter-table`). -- **404s:** `Database not found: x` / `Table not found: x` / - `Record not found: [pks]`. -- **Request:** `{"update": {column: value, ...}, "return"?: true, - "alter"?: true}`. Missing/non-dict `update` → 400 - `"JSON must contain an update dictionary"`; unknown keys → 400 - `"Invalid keys: ..."`; write failures (bad column, constraint violation) → - 400 with the message. -- **Response** — 200 `{"ok": true}`; with `return: true`, - `{"ok": true, "rows": [{...}]}` — a single-item list, matching - insert/upsert. Emits `update-row`. - -### POST /\/\/\/-/delete - -`RowDeleteView` (views/row.py:738-778). - -- **Permission:** `delete-row` (403 `Permission denied`). 404s as update. -- **Request:** no body required (any body is ignored — there is no - confirmation step, unlike table drop). -- **Response** — 200 `{"ok": true}`; with `?_redirect_to_table` a `redirect` - key is added. A failure during the write returns 400 with the message, - matching update. Emits `delete-row`. - ---- - -## Stored (canned) queries API - -Stored queries live in the internal database's `queries` table -(utils/internal_db.py:116-133). Queries defined in `datasette.yaml` are -synced in at startup with `source="config"` and `is_trusted` defaulting to -true; queries created via the API get `source="user"`, `is_trusted=false`, -`owner_id` = actor id. - -**Canonical stored-query JSON object** (`stored_query_to_dict`, -stored_queries.py:55-80): - -```json -{ - "database": "...", "name": "...", "sql": "...", - "title": null, "description": null, "description_html": null, - "hide_sql": false, "fragment": null, - "parameters": ["p"], - "is_write": false, "is_private": true, "is_trusted": false, - "source": "user", "owner_id": "...", - "on_success_message": null, "on_success_message_sql": null, - "on_success_redirect": null, - "on_error_message": null, "on_error_redirect": null, - "private": true -} -``` - -`private` appears only in list responses. On input (create/update and -`datasette.yaml`), `params` is accepted as an alias for `parameters`. - -**Default permission rules for queries** (default_permissions/defaults.py): -`view-query` is default-allow, but private queries are visible only to their -owner; the owner may `update-query`/`delete-query` their `source='user'` -queries. - -### GET /-/queries(.json) and GET /\/-/queries(.json) - -`GlobalQueryListView` / `QueryListView` (app.py:2606-2609, 2663-2666; -views/stored_queries.py:69-238). The global variant lists queries across all -databases (`database`/`database_color` are null, `show_database` true). - -- **Permissions:** no single gate; results filtered per query by - `view-query` (private queries appear only for their owner). -- **Parameters:** `_size` (default 20 HTML / **50 JSON**; accepts `max`; - values over `max_returned_rows` or non-integers → 400, matching table - `_size` semantics), `_next` (cursor), `q` (substring search over - name/title/description/sql), `is_write` / `is_private` (booleans; invalid → - 400 `"is_write must be 0 or 1"`), `source`, `owner_id`. -- **Response** — 200: - `{"ok": true, "database", "database_color", "queries": [...], "next", - "next_url", "limit", "show_private_note", - "show_trusted_note", "query_list_path", "show_database", - "facets": [{title, items: [{label, count, href, active}]}], - "filters": {q, is_write, is_private, source, owner_id}}`. - -### GET /\/-/queries/analyze - -`QueryCreateAnalyzeView` (app.py:2667-2670; views/stored_queries.py:290-322). -**GET only** despite being an "analyze" action — POST → 405. - -- **Permissions:** `execute-sql` then `store-query` (each denial → 403 - `errors` JSON). -- **Parameters:** only `sql` (others → 400 `"Invalid keys: ..."`). -- **Response** — 200: `{"ok", "parameters", "analysis_error", - "analysis_rows": [{operation, database, table, required_permission, - source, allowed}], "has_sql", "analysis_is_write", "save_disabled"}`. - -### POST /\/-/queries/store - -`QueryStoreView` (app.py:2671-2674; views/stored_queries.py:325-388). GET on -the same path renders the HTML create form. - -- **Permissions:** `execute-sql` + `store-query` (403 `errors` JSON). -- **Request:** JSON bodies must wrap the fields: - `{"query": {...fields...}}`; form bodies pass fields flat. Fields: - `name` (required; `^[^/\.\n]+$`; conflicts with tables/views or existing - queries → 400), `sql` (required; read SQL must pass `validate_sql_select`; - write SQL must pass per-operation permission checks), `title`, - `description`, `hide_sql`, `fragment`, `parameters`/`params` (must exactly - match the SQL's named parameters; magic parameters rejected), - `is_private` (**default true**), and — only for write SQL — - `on_success_message`, `on_success_redirect`, `on_error_message`, - `on_error_redirect`. `is_write` is derived from SQL analysis; - `is_trusted`, `description_html` and `on_success_message_sql` cannot be - set through this API. -- **Response:** JSON request → **201** `{"ok": true, "query": {...}}`; form - request → 302 redirect. - -### GET /\/\/-/definition - -`QueryDefinitionView` (app.py:2695-2698; views/stored_queries.py:391-408). - -- **Permission:** `view-query` (403 `["Permission denied"]`). -- **Response:** 200 `{"ok": true, "query": {...}}`; 404 - `["Query not found: x"]`. - -### GET/POST /\/\/-/edit - -`QueryEditView` (app.py:2699-2702) — **HTML form endpoint** -(`has_json_alternate = False`), not part of the JSON API. Programmatic -updates use `/-/update`. - -### POST /\/\/-/update - -`QueryUpdateView` (app.py:2703-2706; views/stored_queries.py:411-465). - -- **Permissions:** `update-query` (403 `need update-query`); trusted queries - → 403 `"Trusted queries cannot be updated using the API"`; changing `sql` - additionally requires `execute-sql`. -- **Request:** `{"update": {...partial fields...}, "return"?: true}` — other - top-level keys → 400. Updatable fields: `sql`, `title`, `description`, - `hide_sql`, `fragment`, `parameters`/`params`, `is_private`, `on_*` - fields (write SQL only). New SQL is re-analyzed and `is_write` recomputed. -- **Response:** 200 `{"ok": true}` (plus `query` with `return: true`); 404 - `"Query not found: x"`. - -### POST /\/\/-/delete - -`QueryDeleteView` (app.py:2707-2710; views/stored_queries.py:594-644). GET -renders an HTML confirmation page. - -- **Permission:** `delete-query` (403 `need delete-query`). Trusted - queries → 403 `"Trusted queries cannot be deleted using the API"`, - matching update. -- **Response:** JSON request → 200 `{"ok": true}`; form → 302; 404 - `"Query not found: x"`. No `confirm` field required (unlike table drop). - -### GET/POST /\/\(.json) — executing a stored query - -No dedicated route: the table route resolves the name, and on `TableNotFound` -the request is dispatched to `QueryView` when a stored query matches -(views/table.py:1698-1712). Covers both config-defined and API-stored -queries. - -**GET (read queries)** — `QueryView.get` (views/database.py:695-1130): - -- **Permissions:** `view-query` (denied → `Forbidden` → 403 HTML). Read - queries then require `execute-sql` unless `is_trusted`. Write queries are - **not executed** on GET — JSON returns empty `rows`; HTML shows a POST form. -- **Parameters:** each named `:param` is read from the query string (missing - → `""`); `_timelimit`; renderer options (`_shape`, `_nl`, `_json`, - `_json_infinity`); `_extra` (QUERY scope). -- **Response:** `{"ok": true, "rows": [...], "truncated": false}` + extras. - SQL errors → 400 with `error` in the envelope. - -**POST (write queries)** — `QueryView.post` (views/database.py:574-693): - -- **Permissions:** `view-query`; then, unless `is_trusted`: - `execute-write-sql` on the database **plus** per-operation write - permissions (same table as `/-/execute-write`). Rejection → 403 - `{"ok": false, "message": "...", "redirect": null}` for JSON clients. - Immutable database → 403. -- **Body:** form-encoded or JSON `param=value` pairs (values coerced to - strings). -- **JSON is returned when** `Accept: application/json`, `?_json=1`, or a - `_json` body field is present; otherwise 302 + flash message. -- **Magic parameters** (`:__`, resolved server-side; registered - via `register_magic_parameters`, default_magic_parameters.py): - `_now_epoch`, `_now_date_utc`, `_now_datetime_utc`, `_actor_`, - `_random_chars_`, `_cookie_`, `_header_` (underscores → - hyphens). User-stored queries cannot contain magic parameters — they are a - feature of config/trusted queries. -- **Response:** success → 200 - `{"ok": true, "message": "...", "redirect": "..."|null}` — `message` - honors `on_success_message_sql` / `on_success_message`, falling back to - `"Query executed"` or `"Query executed, N rows affected"`. SQL failure → - **400** canonical error (message honors `on_error_message`) plus a - `redirect` context key from `on_error_redirect`. Operation rejection - (`QueryWriteRejected`, e.g. VACUUM) → 403 canonical error plus - `redirect: null`. - ---- - -## Authentication and tokens - -### Bearer tokens (`dstok_`) - -Signed API tokens are sent as `Authorization: Bearer dstok_...`. The -`actor_from_signed_api_token` hook (default_permissions/tokens.py:25-40) -passes the token to `datasette.verify_token()`, which tries every handler -registered via `register_token_handler`; the default is -`SignedTokenHandler` (tokens.py:117-193). - -- **Format:** `dstok_` + itsdangerous-signed payload (namespace `token`) - containing `a` (actor id), `t` (creation Unix time), optional `d` - (duration seconds), optional `_r` (restrictions). -- **Verification:** a `dstok_`-prefixed token that fails verification — - `allow_signed_tokens` off, invalid signature, missing/non-integer `t`, - malformed `d`, or expired — raises `TokenInvalid`, and the request fails - with **401**, the canonical error body and a - `WWW-Authenticate: Bearer error="invalid_token"` header (even if a valid - `ds_actor` cookie is also present). Tokens with prefixes no registered - handler recognizes are ignored (they may belong to an auth plugin). The - effective duration is `d` capped by `max_signed_tokens_ttl` (default 0 = - no cap; a non-zero setting also imposes a TTL on tokens without `d`). -- **Resulting actor:** `{"id": , "token": "dstok"}` plus `"_r"` and - `"token_expires"` when applicable. - -**Restrictions (`_r`)** (default_permissions/restrictions.py): - -- `"a"`: list of actions allowed on any resource -- `"d"`: `{database_name: [actions]}` -- `"r"`: `{database_name: {table_name: [actions]}}` - -Actions are stored as abbreviations when available (see appendix); checks -accept either the full name or the abbreviation. Restrictions are an -allowlist filter layered on top of normal permission resolution — a -restricted token can never do more than its allowlist, and never more than -the underlying actor could do anyway. - -### Token creation - -- **`/-/create-token`** is an HTML form endpoint only (see the instance - section) — there is no JSON API to mint tokens in this codebase. -- Programmatic alternatives: the `datasette create-token` CLI command and - the `datasette.create_token()` Python API. -- `/-/auth-token` is the one-time `--root` login mechanism, unrelated to API - tokens. - -### Cookie authentication - -Browser sessions use the signed `ds_actor` cookie (set by `/-/auth-token`, -plugins, or login flows; cleared by `/-/logout`). API POSTs from browsers are -subject to the cross-origin checks described in -[CSRF](#csrf--cross-origin-protection). - ---- - -## Appendix: registered actions (permissions) - -From `datasette/default_actions.py` (registered via the `register_actions` -hook). Token restrictions store the abbreviation when available. - -| Action | Abbr | Resource level | Notes | -|---|---|---|---| -| `view-instance` | `vi` | global | | -| `permissions-debug` | `pd` | global | gates the debug endpoints | -| `debug-menu` | `dm` | global | UI only | -| `view-database` | `vd` | database | | -| `view-database-download` | `vdd` | database | `also_requires="view-database"` | -| `execute-sql` | `es` | database | `also_requires="view-database"`; denied when the `default_allow_sql` setting is off | -| `execute-write-sql` | `ews` | database | `also_requires="view-database"` | -| `create-table` | `ct` | database | | -| `store-query` | `sq` | database | `also_requires="execute-sql"` | -| `view-table` | `vt` | table | | -| `insert-row` | `ir` | table | | -| `delete-row` | `dr` | table | | -| `update-row` | `ur` | table | | -| `alter-table` | `at` | table | | -| `set-column-type` | `sct` | table | | -| `drop-table` | `dt` | table | | -| `view-query` | `vq` | query | default-allow; private queries restricted to their owner | -| `update-query` | `uq` | query | query owner allowed by default (source=`user` only) | -| `delete-query` | `dq` | query | query owner allowed by default (source=`user` only) | diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md deleted file mode 100644 index 0d817612..00000000 --- a/stable-api-recommendations.md +++ /dev/null @@ -1,469 +0,0 @@ -# Datasette 1.0 Stable API — Consistency and Completeness Review - -This review is based on `existing-api.md`, which documents the JSON API as -actually implemented in this codebase (`1.0a35`), derived from source. The -goal here is to identify everything that should be made consistent, fixed, or -explicitly scoped out **before** the 1.0 stability promise takes effect — -because after 1.0, every inconsistency below becomes a compatibility -commitment. - -Findings are grouped by theme. Each carries a priority: - -- **P1 — should block 1.0**: breaking to fix later, or a correctness/security - concern. -- **P2 — strongly recommended**: fixable later only via awkward additive - changes. -- **P3 — nice to have / documentation decision**: can be resolved by - documenting the behavior as intentional. - ---- - -## 1. Error responses: four shapes is three too many (P1) — ✅ IMPLEMENTED - -> **Status:** implemented. All four shapes now delegate to a shared -> `error_body()` helper (`datasette/utils/__init__.py`) producing -> `{"ok": false, "error": "", "errors": [...], "status": }`. -> The `title` key is no longer emitted in JSON; the bare `{"error": ...}` -> debug-endpoint shape is gone; `_shape=object` misuse now returns HTTP 400 -> (part of §1b). Covered by `tests/test_error_shape.py` and documented in -> the "Error responses" section of `docs/json_api.rst`. §1a (`Forbidden` → -> JSON) and §1b (write canned-query 200) are now also implemented. Still -> open from this section's sub-items: the §1c status outliers. - -The API currently produces four distinct JSON error shapes depending on which -internal layer generates the error: - -| Shape | Producer | Example endpoints | -|---|---|---| -| `{"ok": false, "error", "status", "title"}` | exception handler (handle_exception.py:50-53) | 404s and `DatasetteError`s on any `.json` path | -| `{"ok": false, "errors": [...]}` | `_error()` helper (views/base.py:183-184) | all write endpoints, stored-query endpoints, execute-write | -| `{"ok": false, "error", "rows": [], "truncated": false}` | JSON renderer (renderer.py:52-56) | SQL errors on table/query reads | -| `{"error": "..."}` (no `ok`) | permission debug views (views/special.py) | `/-/allowed`, `/-/rules`, `/-/check`, POST `/-/permissions` | - -Additionally, write canned queries report failure via a **fifth** vocabulary: -`{"ok": false, "message": ..., "redirect": ...}` with HTTP **200** -(views/database.py:678-690). - -A 1.0 client cannot write a single error handler today. **Recommendation:** -pick one canonical error object — the singular/plural tension is easiest to -resolve as: - -```json -{"ok": false, "error": "human-readable summary", "errors": ["detail", "..."], "status": 400} -``` - -where `errors` is optional and `error` is always present — and route every -error path through it (including the `forbidden` and `handle_exception` -defaults). At minimum, eliminate the bare `{"error": ...}` shape and the -`status`/`title` keys nobody else emits (`title` is a template-rendering -concern that leaked into the API). - -### 1a. `Forbidden` returns an HTML 403 to JSON clients (P1) — ✅ IMPLEMENTED - -> **Status:** implemented — the default `forbidden()` hook now returns the -> canonical JSON error for requests whose path ends in `.json` or that send -> `Accept: application/json` / `Content-Type: application/json`. - -Read endpoints that deny access via `ensure_permission`/`check_visibility` -raise `Forbidden`, and the default `forbidden()` hook renders an **HTML error -page even for `.json` requests** (forbidden.py:4-19, app.py:2895-2904). So: - -- `GET /db/table.json` without `view-table` → 403 **HTML** -- `POST /db/table/-/insert` without `insert-row` → 403 **JSON** - -A JSON client gets unparseable output precisely when it most needs a -machine-readable answer. **Recommendation:** the default forbidden handler -must return the canonical JSON error when the path ends in `.json` or the -request prefers JSON, mirroring `handle_exception`. - -### 1b. Errors that return HTTP 200 (P1) — ✅ IMPLEMENTED - -> **Status:** implemented. `_shape=object` misuse returns 400 (done with -> §1), and write canned-query SQL failures now return **400** with the -> canonical error shape (plus the `redirect` context key); the -> `QueryWriteRejected` 403 branch also uses the canonical shape. - -- `_shape=object` on a query or pk-less table → `{"ok": false, "error": - "_shape=object is only available on tables"}` with **200** - (renderer.py:73-90), while an unknown `_shape` value returns **400** - (renderer.py:101-108). Same class of error, different status. -- Write canned-query SQL failure → **200** `{"ok": false, "message": ...}` - (views/database.py:683-690), while the equivalent failure on - `/-/execute-write` returns **400**. - -**Recommendation:** all `ok: false` responses should carry a 4xx/5xx status. -(`/-/execute-write/analyze` returning `ok: false` with 200 for "analysis -completed, SQL is invalid" is defensible but should then not reuse the `ok` -key — see §2.) - -### 1c. Wrong-status outliers (P2) — ✅ IMPLEMENTED - -- ~~Row **delete** write failures return **500** (views/row.py:757) while row - **update** write failures return **400** (views/row.py:832-835). Same - failure class, different status; pick 400 (or 409 for constraint - violations) for both.~~ ✅ **Done** — delete now returns 400, matching - update and the rest of the write API. -- ~~Invalid or expired bearer tokens silently degrade the request to anonymous, - so clients see a 403 permission error (or worse, anonymous-permitted data) - rather than a 401 (tokens.py:147-193). For 1.0, a malformed/expired - `Authorization: Bearer dstok_...` header should produce **401** with a - distinguishable error, so clients can tell "renew your token" apart from - "you lack permission".~~ ✅ **Done** — token handlers can raise - `TokenInvalid`; Datasette responds 401 with the canonical body and a - `WWW-Authenticate: Bearer error="invalid_token"` header. Unrecognized - token prefixes still fall through to anonymous so auth plugins keep - working. - ---- - -## 2. Success envelope: `ok` is not universal, arrays are not extensible (P1/P2) — ✅ IMPLEMENTED (§2a-2c open) - -> **Status:** recommendations 1-3 are implemented. Every JSON-object -> success response now includes `"ok": true` (`JsonDataView` injects it for -> dict responses; homepage, jump, schema, permission-debug and autocomplete -> views set it explicitly), and the three top-level-array endpoints now -> return objects: `/-/plugins` → `{"ok": true, "plugins": [...]}`, -> `/-/databases` → `{"ok": true, "databases": [...]}`, `/-/actions` → -> `{"ok": true, "actions": [...]}`. Covered by -> `tests/test_success_envelope.py`. The sub-findings §2a (collection -> representations), §2b (`_extra`/`_shape` coverage) and §2c (count -> truncation) remain open. - -Endpoints disagree about the success envelope: - -- **Have `ok: true`:** table/row/query reads, database view, all write - endpoints, stored-query endpoints, `/-/allowed`-style debug data. -- **No `ok` key:** `/-/versions`, `/-/settings`, `/-/config`, `/-/threads`, - `/-/actor`, `/-/jump`, `/-/schema` variants (`{"database", "schema"}`, - `{"schemas": [...]}`), table `/-/schema.json`, `/-/autocomplete` - (`{"rows": []}`), homepage `/.json`. -- **Top-level JSON arrays:** `/-/plugins`, `/-/databases`, `/-/actions` - (app.py:2247-2304). A top-level array can never grow a sibling key - (pagination, warnings, `ok`) without a breaking change. - -**Recommendations:** - -1. (P1) Wrap the three array endpoints in objects before 1.0: - `{"ok": true, "plugins": [...]}` etc. This is the single cheapest - future-proofing fix in this list. -2. (P2) Add `ok: true` to every JSON-object success response, or explicitly - document that `ok` only exists on data endpoints. Half-consistency is the - worst outcome. -3. (P2) `/db/-/schema.json` (`{"database", "schema"}`) and - `/db/table/-/schema.json` should match the envelope style of their sibling - endpoints (they are also the only data endpoints whose 404 uses the - exception shape but whose success has no `ok`). - -### 2a. Collection representations disagree (P2) - -- ~~Homepage `/.json` returns `databases` as an **object keyed by name** - (index.py:147-161); `/-/databases.json` returns an **array**; the database - page returns `tables` as an array. Choose arrays-of-objects everywhere - (objects-keyed-by-name break when names need ordering or pagination).~~ - ✅ **Done** — the homepage returns a list, matching `/-/databases.json`. - The homepage JSON remains deliberately undocumented. -- ~~Insert/upsert with `return: true` respond with `rows` (plural, list); row - update with `return: true` responds with `row` (singular, object) - (views/row.py:837-844). Pick one (`rows` everywhere, even for one row, - matches the read API).~~ ✅ **Done** — row update now returns - `rows: [{...}]`. - -### 2b. `_extra`/`_shape` support is uneven (P2) — partially implemented - -> **Status:** unknown `_extra` names on data formats now return 400 -> `Unknown _extra: ` (HTML pages still ignore them). Extending -> extras/shaping to database/instance scope remains open. - -The extras system (`?_extra=`, scope-registered) is the 1.0 mechanism for -response shaping — but it only exists on table, row and query endpoints. The -database view builds JSON by hand and supports **neither `_extra` nor -`_shape`** (views/database.py:189-212); the homepage likewise. Either extend -extras to database/instance scope before 1.0 or document clearly that shaping -is a table/row/query feature. Also decide the contract for **unknown -`_extra` names, which are currently silently ignored** (extras.py:116-122) — -silent ignoring means typos return the default payload with no signal; -recommend a 400 or a `warnings` key. - -### 2c. Count truncation is invisible in JSON (P2) — ✅ IMPLEMENTED - -> **Status:** implemented — a public `count_truncated` extra now exists and -> is implicitly included whenever `count` is requested. - -The `count` extra is computed with a `limit 10001` subquery, so `count: -10001` actually means "at least 10001" — the `count_truncated` flag exists -but only in the HTML template context, never in JSON (views/table.py: -2334-2337). Expose it (e.g. make `count` be `null` + add `count_estimate`, -or add `count_truncated` to the JSON) before clients start trusting the -number. - ---- - -## 3. Pagination: three mechanisms, two contracts (P2) — partially implemented - -> **Status:** `next_url` now accompanies `next` in the default table JSON -> keys (previously it required `?_extra=next_url`), so every response with -> a `next` token also carries the ready-to-follow URL. Pagination tokens -> are deliberately left undocumented as to their internal structure. -> `_size` is now the single page-size parameter with uniform table-style -> semantics everywhere: query lists accept `max` and 400 on out-of-range -> values (previously silently clamped), and the `/-/allowed` and -> `/-/rules` debug endpoints renamed `page`/`page_size` to -> `_page`/`_size` with the same validation (400 instead of silent -> capping at 200). `has_more` has been **removed** from the query-list -> JSON — `next: null` is the single end-of-results signal everywhere, -> keeping default response keys minimal (`total` remains a debug-endpoint -> nicety). Fixing this also uncovered and fixed a bug where the query -> list's JSON `next_url` pointed at the HTML page (it dropped the `.json` -> extension) and was relative where the table `next_url` is absolute. -> §3 is now fully resolved. - -| Endpoint | Mechanism | Token | Extras | -|---|---|---|---| -| Table `.json` | keyset | tilde-encoded pk/sort values in `_next` | `next` always in body, `next_url` via `_extra`, `Link: rel=next` header | -| SQL view `.json` | **offset** | integer in the same `_next` parameter | same envelope | -| `/-/queries` lists | keyset | cursor in `_next` | `next`, `next_url`, **`has_more`** in body | -| `/-/allowed`, `/-/rules` | **page numbers** | `page`/`page_size` | `total`, `next_url`, `previous_url` | - -Concerns: - -1. The same `_next` parameter means "start after key" on tables but "row - offset" on views. Offset pagination over views is also O(n) and skews - under concurrent writes. If unifiable, unify; if not, document loudly. -2. `has_more` exists on query lists but not table pages; `total` exists on - debug endpoints but not elsewhere. Standardize the pagination block - (suggest: `next`, `next_url` — nullable — everywhere; treat `has_more` as - `next != null`). -3. Page-size parameters: `_size` (default 100, `max` keyword allowed) on - tables; `_size` (default 50 JSON, clamped 1–1000, no `max` keyword) on - query lists; `page_size` (default 50, silently capped at 200) on debug - endpoints. Align names, defaults and the cap behavior (silent capping vs - 400) as far as practical. - ---- - -## 4. HTTP semantics (P2) - -- ~~**201 vs 200:** insert → 201, upsert → 200 (views/table.py:1194), create - table → 201, store query → 201. Insert-201/upsert-200 is defensible - (upsert may not create) but it is undocumented subtlety; state it, or - return 200 for both with an explicit `created` count.~~ ✅ **Done** — - documented as deliberate in the upsert docs. -- **Destructive-action confirmation is asymmetric:** table drop requires - `{"confirm": true}` and has a preview response (views/table.py:1346-1365); - row delete executes immediately and ignores the body; query delete - executes immediately. Decide the 1.0 rule (suggestion: confirmation only - for schema-destroying operations, i.e. keep as is — but document it as a - deliberate contract). -- ~~**Content-type enforcement is inconsistent:** `/-/insert`, `/-/upsert`, - `/-/alter`, `/-/set-column-type` demand `Content-Type: application/json` - (400 otherwise); `/-/create` parses the body as JSON regardless of - content type; execute-write and the query CRUD endpoints accept both JSON - and form encodings. Pick one rule for JSON-only endpoints.~~ ✅ **Done** - — the lenient rule won: JSON-only write endpoints parse the body as JSON - regardless of `Content-Type` (CSRF protection comes from the - cross-origin header checks, not content types). This also fixed a 500 on - insert when the header was absent entirely. -- **JSON-vs-HTML negotiation on POST differs per endpoint:** execute-write - and canned queries key off `Accept: application/json` / a `_json` body - field; the write API keys off nothing (always JSON); query store keys off - request content type. A single documented rule ("responses are JSON if the - request body was JSON or `Accept: application/json`") would cover all of - them. -- **Endpoints named like actions but served over GET:** - `/-/queries/analyze`, `/-/execute-write/analyze`, - `/-/foreign-key-suggestions`, `/-/query/parameters` are all GET (correct, - they are reads) — fine, but `analyze` under a POST-shaped path invites - wrong calls; make sure 405 responses for POST on these return the JSON 405 - shape (they do only when the path ends `.json` or content type is JSON — - a JSON POST to `/-/queries/analyze` gets JSON, a form POST gets text). - ---- - -## 5. Naming and parameter conventions (P2/P3) - -- ~~**`params` and `parameters` are duplicate keys** in every stored-query - object (stored_queries.py:55-80). Delete one before 1.0 (suggest keeping - `parameters`; the write side already accepts both on input).~~ - ✅ **Done** — output objects carry only `parameters` (matching - `/-/query/parameters` and the analyze endpoints); `params` remains an - accepted input alias for API creation and `datasette.yaml` config. -- **Three names for the same concept across error/message payloads:** - `error`, `errors`, `message`. See §1. -- ~~**Boolean query parameters have at least three grammars:** `_nl=on`, - `_labels=on/off`, `?all=1`, `is_write=1|0|true|false|t|f|yes|no|on|off`, - `_nocount=1`. Adopt one accepted set (the query-list parser at - query_helpers.py:81-94 is a good candidate) and apply it everywhere.~~ - ✅ **Documented** — the JSON API docs state the canonical grammar - (`on/true/1`, `off/false/0`), which `value_as_boolean` already accepts - everywhere it is used. -- ~~**`.jsono`** survives on the homepage route (identical output to `.json`) - and as a row-view redirect. Remove it at 1.0; it is pure legacy.~~ - ✅ Removed: the homepage routes only accept `.json` and the row-view - redirect is gone. -- **`_json` is overloaded:** on GET it is a renderer option naming a column - to parse as JSON (repeatable); on canned-query POST a `_json` body field - forces a JSON response. Two unrelated meanings for one name. -- The reserved `/-/` namespace is applied consistently across routes — this - is in good shape. The one gap: table names matching `^-$`-adjacent shapes - are protected by tilde-encoding; keep a test asserting `/-/` can never be - shadowed by user data. - ---- - -## 6. Permissions and security consistency (P1/P2) - -- ~~**(P1) `/-/databases.json` ignores per-database permissions** — it lists - every attached database (name, path on disk, size) to any actor holding - `view-instance` (app.py:2157-2169), while the homepage and every other - endpoint filter by `view-database`. On a public instance with private - databases this leaks filesystem paths and database names. Filter it, or - gate it behind `permissions-debug`.~~ ✅ **Done** — the endpoint now - filters through `allowed_resources("view-database", actor)`. -- ~~**(P2) `/db/-/schema` checks existence before permission** - (views/special.py:1308-1317): an actor without `view-database` can - distinguish "database exists" (403) from "does not exist" (404). - Standardize on permission-check-first (as the table view does) so - unauthorized actors get a uniform response.~~ ✅ **Done** — permission is - checked first; the table schema view also now 404s (instead of a 500 - KeyError) for an unknown database. -- ~~**(P2) `/-/threads` exposes runtime internals** (thread idents, asyncio - task reprs including file paths) behind only `view-instance`. Consider - `permissions-debug`, alongside `/-/actions` which already requires it.~~ - ✅ **Done** — `/-/threads` now requires `permissions-debug`. -- ~~**(P3) `/-/config` redaction is substring-based** on six key names - (app.py:2502-2505); plugins storing secrets under other names leak. Worth - a note in plugin authoring docs plus a `redact_keys` plugin hook.~~ - ✅ **Documented** — the plugin secrets docs now advise naming keys to - match the redaction substrings (a `redact_keys` hook remains a possible - future addition). -- **(P3) Database-level checks on `/-/create`** (insert-row/update-row - checked against `DatabaseResource`, not the about-to-exist table — - table_create_alter.py:819-856) vs table-level checks on `/-/insert`. - Correct by necessity, but document that a token restricted to - table-level `ir` cannot use `/-/create` with rows. - ---- - -## 7. Completeness gaps for a 1.0 JSON API (P2/P3) - -1. **(P2) No JSON API to create tokens.** `/-/create-token` is an HTML form - only (`has_json_alternate = False`, form-encoded POST). Any automation - that wants to mint scoped tokens must shell out to `datasette - create-token`. An intentional JSON mode (actor-authenticated, same - restriction vocabulary) rounds out the write API story — or explicitly - document token minting as CLI/Python-only. -2. **(P2) Row JSON cannot expand foreign-key labels.** `_labels` works on - table JSON but is silently ignored on row JSON (views/row.py:445-475 - expands only for HTML). Either support it or return 400 for unsupported - parameters; silent ignoring is the worst option (see also §2b on unknown - `_extra` values). -3. **(P2) No machine-readable "which write features does this instance/table - support" endpoint.** Clients must probe (`/-/insert` on an immutable - database → 403). The API explorer computes exactly this data for HTML - (views/special.py:863-990); exposing it as JSON would let clients degrade - gracefully. (`/-/allowed.json` covers the permission half already.) -4. **(P3) Table list pagination.** `/db.json` inlines all tables (with - counts) and the homepage truncates to 5 per database; a 10,000-table - database has no paginated table listing. Acceptable for 1.0 if - documented; the internal catalog tables would support a real endpoint - later. -5. **(P3) `Link: rel=next` header** exists on table JSON only. Harmless, but - either add it to the other paginated endpoints or drop it from the - contract (`Access-Control-Expose-Headers: Link` suggests it is meant to - be part of the API). - ---- - -## 8. Behavior that looks like a bug and should be resolved before freezing - -1. ~~**Trusted queries: update is blocked, delete is not.** - `QueryUpdateView` rejects `is_trusted` queries with 403 - (stored_queries.py:426-427) but `QueryDeleteView.post` never checks - `is_trusted` — an actor with `delete-query` can delete a config-defined - trusted query via the API (it will resync on restart, making the - behavior confusing rather than catastrophic). Align delete with update.~~ - ✅ **Done** — both the POST endpoint and the HTML confirmation page now - return 403 `"Trusted queries cannot be deleted using the API"`; - `datasette.remove_query()` remains available for internal use. -2. ~~**GET `/db/-/query` with no `?sql=` returns 200 `{"ok": true, "rows": - []}`** while `.csv` on the same request returns 400 `"?sql= is - required"`. The JSON behavior masks caller bugs; return 400 on both.~~ - ✅ **Done** — all data formats now return 400; the HTML SQL editor page - is unchanged. -3. ~~**`_shape=object` HTTP 200 error** (§1b) — almost certainly unintended.~~ - ✅ Done — now 400 (fixed with §1b). -4. ~~**Row delete 500** (§1c) — inconsistent with every sibling endpoint.~~ - ✅ Done — now 400. -5. ~~**The "SQL Interrupted" error embeds an HTML fragment in the JSON `error` - value** (views/database.py:805-820). Error strings in the JSON API should - be plain text.~~ ✅ **Done** — `DatasetteError` gained a `plain_message` - used for JSON responses; the HTML error page keeps the rich version with - the SQL textarea. §8 is now fully resolved. - ---- - -## 9. Define stability tiers explicitly (P1 — documentation, not code) — ✅ IMPLEMENTED - -> **Status:** implemented. Undocumented JSON endpoints self-describe with -> an `"unstable"` marker key, and `docs/json_api.rst` now opens with an -> "API stability" section (`json_api_stability`) declaring the 1.x -> promise: documented endpoints/keys are stable with additive-only -> changes, pagination tokens are opaque, the error format and token -> restriction semantics are stable, and the exempt tiers (marker-key -> endpoints, debug/support endpoints, explicitly-unstable keys) are -> listed. Cross-referenced from the introspection and permission-debug -> docs. - -Not everything under `/-/` can or should carry a 1.0 guarantee. Recommend -shipping 1.0 with an explicit three-tier contract, per endpoint: - -- **Stable (semver-protected):** table/row/query reads (`.json`, `_shape`, - `_extra` public names, filters, pagination tokens as opaque strings), the - write API (`/-/insert`, `/-/upsert`, `/-/alter`, `/-/drop`, - `/-/set-column-type`, row `/-/update`, `/-/delete`, `/-/create`, - `/-/execute-write`), stored-query CRUD + execution, `/-/versions`, - `/-/plugins`, `/-/settings`, `/-/actor`, `/-/databases`, schema endpoints, - token format & restriction semantics (`_r` abbreviations are wire format - now — they are stored inside issued tokens and cannot change silently). -- **Unstable/debug (documented as exempt):** `/-/threads`, `/-/actions`, - `/-/permissions`, `/-/allowed`, `/-/rules`, `/-/check`, `/-/messages`, - `/-/allow-debug`, `/-/patterns`, `/-/debug/autocomplete`, the `debug` and - `request` extras (the `debug` extra already self-describes as unstable), - `/-/api` and `/-/jump` (UI support endpoints), `/-/autocomplete` and - `/-/fragment` (UI support), `/-/foreign-key-suggestions` and - `/-/foreign-key-targets` (heuristic outputs). -- **Internal:** anything HTML-only (`/-/edit`, `/-/create-token`, - `/-/logout`, `/-/auth-token`). - -Two details make tiering urgent rather than optional: - -- **Extras are enumerable by clients** (`?_extra=extras` self-describes the - registry), so every public extra name is de-facto API. Mark each extra - stable or unstable in its class definition and surface that in the - `extras` output. -- **Pagination tokens leak implementation** (tilde-encoded pk values for - tables, plain integers for views). Declare them opaque now so the view - token can become keyset later without a "breaking" change. - ---- - -## 10. Summary of P1 items (the pre-1.0 checklist) - -1. ~~One canonical JSON error shape; retire the other three (§1).~~ ✅ Done. -2. ~~`Forbidden` → JSON 403 for JSON requests (§1a).~~ ✅ Done. -3. ~~No `ok: false` with HTTP 200 (§1b: `_shape=object`, write canned-query - SQL errors).~~ ✅ Done. -4. ~~Wrap `/-/plugins`, `/-/databases`, `/-/actions` top-level arrays in - objects (§2).~~ ✅ Done. -5. ~~Filter `/-/databases.json` by `view-database` or gate it behind - `permissions-debug` (§6).~~ ✅ Done. -6. ~~401 (not silent-anonymous) for invalid/expired bearer tokens (§1c).~~ - ✅ Done. -7. ~~Publish explicit stability tiers, including extras and pagination-token - opacity (§9).~~ ✅ Done. -8. Resolve the looks-like-a-bug list (§8), especially ~~trusted-query delete - and row-delete 500~~ (both done). - -Everything in P2 is worth doing now because each item is breaking-to-fix -later; each P3 can be resolved by a sentence of documentation declaring the -current behavior intentional. From 8b159144a58992f8fe1f5a0489a3566f57e8d20c Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 23:48:49 +0000 Subject: [PATCH 044/131] Add Response.error() for JSON errors in the standard format Response.error(messages, status=400) builds a JSON error response in Datasette's standard error format, alongside Response.json/html/text. messages can be a single string or a list. All internal error response construction now uses it - the private views.base._error() helper is gone and the verbose Response.json(error_body(...), status=...) sites are converted. error_body() remains for the cases that merge the error keys into a larger payload (the JSON renderer, handle_exception and the permission debug payload builders). Since Response is public plugin API, plugins that build JSON endpoints now have an obvious way to return errors in the canonical shape. Documented in the internals documentation, including the guidance to raise Forbidden/NotFound/BadRequest/DatasetteError instead when the error should content-negotiate. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/app.py | 5 +- datasette/forbidden.py | 4 +- datasette/utils/asgi.py | 14 ++++- datasette/views/base.py | 9 +--- datasette/views/execute_write.py | 16 +++--- datasette/views/row.py | 32 ++++++------ datasette/views/special.py | 18 ++----- datasette/views/stored_queries.py | 70 ++++++++++++++----------- datasette/views/table.py | 74 ++++++++++++++------------- datasette/views/table_create_alter.py | 50 +++++++++--------- docs/internals.rst | 6 ++- tests/test_internals_response.py | 24 +++++++++ 12 files changed, 182 insertions(+), 140 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 463c9be2..9c7e768b 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -113,7 +113,6 @@ from .utils import ( detect_json1, add_cors_headers, display_actor, - error_body, escape_css_string, escape_sqlite, find_spatialite, @@ -2958,9 +2957,7 @@ class DatasetteRouter: headers = {"www-authenticate": 'Bearer error="invalid_token"'} if self.ds.cors: add_cors_headers(headers) - response = Response.json( - error_body([str(exception)], 401), status=401, headers=headers - ) + response = Response.error([str(exception)], 401, headers=headers) await response.asgi_send(send) async def handle_404(self, request, send, exception=None): diff --git a/datasette/forbidden.py b/datasette/forbidden.py index 3a81ac4f..91b1ff96 100644 --- a/datasette/forbidden.py +++ b/datasette/forbidden.py @@ -1,5 +1,5 @@ from datasette import hookimpl, Response -from .utils import add_cors_headers, error_body +from .utils import add_cors_headers @hookimpl(trylast=True) @@ -13,7 +13,7 @@ def forbidden(datasette, request, message): headers = {} if datasette.cors: add_cors_headers(headers) - return Response.json(error_body(message, 403), status=403, headers=headers) + return Response.error(message, 403, headers=headers) return Response.html( await datasette.render_template( "error.html", diff --git a/datasette/utils/asgi.py b/datasette/utils/asgi.py index 7777308f..610b86f2 100644 --- a/datasette/utils/asgi.py +++ b/datasette/utils/asgi.py @@ -1,6 +1,6 @@ import json from typing import Optional -from datasette.utils import MultiParams, calculate_etag, sha256_file +from datasette.utils import MultiParams, calculate_etag, error_body, sha256_file from datasette.utils.multipart import ( parse_form_data, MultipartParseError, @@ -575,6 +575,18 @@ class Response: content_type="application/json; charset=utf-8", ) + @classmethod + def error(cls, messages, status=400, headers=None): + """ + A JSON error response using Datasette's standard error format. + + messages can be a single string or a list of strings. For errors + that should content-negotiate between JSON and HTML, raise + Forbidden, NotFound, BadRequest or DatasetteError instead and let + Datasette's error handling hooks build the response. + """ + return cls.json(error_body(messages, status), status=status, headers=headers) + @classmethod def redirect(cls, path, status=302, headers=None): headers = headers or {} diff --git a/datasette/views/base.py b/datasette/views/base.py index 88d16753..66e14a6d 100644 --- a/datasette/views/base.py +++ b/datasette/views/base.py @@ -5,7 +5,6 @@ import sys from datasette.utils.asgi import Request from datasette.utils import ( add_cors_headers, - error_body, EscapeHtmlWriter, InvalidSql, LimitedWriter, @@ -53,7 +52,7 @@ class View: request.path.endswith(".json") or request.headers.get("content-type") == "application/json" ): - response = Response.json(error_body("Method not allowed", 405), status=405) + response = Response.error("Method not allowed", 405) else: response = Response.text("Method not allowed", status=405) return response @@ -92,7 +91,7 @@ class BaseView: request.path.endswith(".json") or request.headers.get("content-type") == "application/json" ): - response = Response.json(error_body("Method not allowed", 405), status=405) + response = Response.error("Method not allowed", 405) else: response = Response.text("Method not allowed", status=405) return response @@ -180,10 +179,6 @@ class BaseView: return view -def _error(messages, status=400): - return Response.json(error_body(messages, status), status=status) - - async def stream_csv(datasette, fetch_data, request, database): kwargs = {} stream = request.args.get("_stream") diff --git a/datasette/views/execute_write.py b/datasette/views/execute_write.py index 6806e69d..dd35b127 100644 --- a/datasette/views/execute_write.py +++ b/datasette/views/execute_write.py @@ -5,7 +5,7 @@ from datasette.resources import DatabaseResource from datasette.utils import UNSTABLE_API_MESSAGE, sqlite3 from datasette.utils.asgi import Response -from .base import BaseView, _error +from .base import BaseView from .database import display_rows as display_query_rows from .query_helpers import ( QueryValidationError, @@ -348,7 +348,7 @@ class ExecuteWriteView(BaseView): ) if not db.is_mutable: return _block_framing( - _error( + Response.error( ["Cannot execute write SQL because this database is immutable."], 403, ) @@ -367,10 +367,10 @@ class ExecuteWriteView(BaseView): actor=request.actor, ): return _block_framing( - _error(["Permission denied: need execute-write-sql"], 403) + Response.error(["Permission denied: need execute-write-sql"], 403) ) if not db.is_mutable: - return _block_framing(_error(["Database is immutable"], 403)) + return _block_framing(Response.error(["Database is immutable"], 403)) data = {} is_json = request.headers.get("content-type", "").startswith("application/json") @@ -384,7 +384,7 @@ class ExecuteWriteView(BaseView): ) except QueryValidationError as ex: if _wants_json(request, is_json, data): - return _block_framing(_error([ex.message], ex.status)) + return _block_framing(Response.error([ex.message], ex.status)) if ex.flash: self.ds.add_message(request, ex.message, self.ds.ERROR) return await self._render_form( @@ -405,7 +405,7 @@ class ExecuteWriteView(BaseView): except sqlite3.DatabaseError as ex: message = str(ex) if wants_json: - return _block_framing(_error([message], 400)) + return _block_framing(Response.error([message], 400)) return await self._render_form( request, db, @@ -488,13 +488,13 @@ class ExecuteWriteAnalyzeView(BaseView): actor=request.actor, ): return _block_framing( - _error(["Permission denied: need execute-write-sql"], 403) + Response.error(["Permission denied: need execute-write-sql"], 403) ) invalid_keys = set(request.args) - {"sql"} if invalid_keys: return _block_framing( - _error( + Response.error( ["Invalid keys: {}".format(", ".join(sorted(invalid_keys)))], 400, ) diff --git a/datasette/views/row.py b/datasette/views/row.py index d9a3deeb..c90a3bbe 100644 --- a/datasette/views/row.py +++ b/datasette/views/row.py @@ -12,7 +12,7 @@ from datasette.utils.asgi import NotFound, Forbidden, PayloadTooLarge, Response from datasette.database import QueryInterrupted from datasette.events import UpdateRowEvent, DeleteRowEvent from datasette.resources import TableResource -from .base import BaseView, DatasetteError, _error, stream_csv +from .base import BaseView, DatasetteError, stream_csv from datasette.utils import ( add_cors_headers, await_me_maybe, @@ -715,11 +715,13 @@ async def _resolve_row_and_check_permission(datasette, request, permission): try: resolved = await datasette.resolve_row(request) except DatabaseNotFound as e: - return False, _error(["Database not found: {}".format(e.database_name)], 404) + return False, Response.error( + ["Database not found: {}".format(e.database_name)], 404 + ) except TableNotFound as e: - return False, _error(["Table not found: {}".format(e.table)], 404) + return False, Response.error(["Table not found: {}".format(e.table)], 404) except RowNotFound as e: - return False, _error(["Record not found: {}".format(e.pk_values)], 404) + return False, Response.error(["Record not found: {}".format(e.pk_values)], 404) # Ensure user has permission to delete this row if not await datasette.allowed( @@ -727,7 +729,7 @@ async def _resolve_row_and_check_permission(datasette, request, permission): resource=TableResource(database=resolved.db.name, table=resolved.table), actor=request.actor, ): - return False, _error(["Permission denied"], 403) + return False, Response.error(["Permission denied"], 403) return True, resolved @@ -752,7 +754,7 @@ class RowDeleteView(BaseView): try: await resolved.db.execute_write_fn(delete_row, request=request) except Exception as e: - return _error([str(e)], 400) + return Response.error([str(e)], 400) await self.ds.track_event( DeleteRowEvent( @@ -791,24 +793,24 @@ class RowUpdateView(BaseView): try: data = await request.json() except json.JSONDecodeError as e: - return _error(["Invalid JSON: {}".format(e)]) + return Response.error(["Invalid JSON: {}".format(e)]) except PayloadTooLarge as e: - return _error([str(e)], 413) + return Response.error([str(e)], 413) if not isinstance(data, dict): - return _error(["JSON must be a dictionary"]) + return Response.error(["JSON must be a dictionary"]) if "update" not in data or not isinstance(data["update"], dict): - return _error(["JSON must contain an update dictionary"]) + return Response.error(["JSON must contain an update dictionary"]) invalid_keys = set(data.keys()) - {"update", "return", "alter"} if invalid_keys: - return _error(["Invalid keys: {}".format(", ".join(invalid_keys))]) + return Response.error(["Invalid keys: {}".format(", ".join(invalid_keys))]) update = data["update"] try: update = decode_write_json_row(update) except WriteJsonValueError as e: - return _error([str(e)], 400) + return Response.error([str(e)], 400) # Validate column types from datasette.views.table import _validate_column_types @@ -817,7 +819,7 @@ class RowUpdateView(BaseView): self.ds, resolved.db.name, resolved.table, [update] ) if ct_errors: - return _error(ct_errors, 400) + return Response.error(ct_errors, 400) alter = data.get("alter") if alter and not await self.ds.allowed( @@ -825,7 +827,7 @@ class RowUpdateView(BaseView): resource=TableResource(database=resolved.db.name, table=resolved.table), actor=request.actor, ): - return _error(["Permission denied for alter-table"], 403) + return Response.error(["Permission denied for alter-table"], 403) def update_row(conn): sqlite_utils.Database(conn)[resolved.table].update( @@ -835,7 +837,7 @@ class RowUpdateView(BaseView): try: await resolved.db.execute_write_fn(update_row, request=request) except Exception as e: - return _error([str(e)], 400) + return Response.error([str(e)], 400) result = {"ok": True} returned_row = None diff --git a/datasette/views/special.py b/datasette/views/special.py index 9386440c..c13191a1 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -501,13 +501,9 @@ class PermissionRulesView(BaseView): # JSON API - action parameter is required action = request.args.get("action") if not action: - return Response.json( - error_body("action parameter is required", 400), status=400 - ) + return Response.error("action parameter is required", 400) if action not in self.ds.actions: - return Response.json( - error_body(f"Unknown action: {action}", 404), status=404 - ) + return Response.error(f"Unknown action: {action}", 404) actor = request.actor if isinstance(request.actor, dict) else None @@ -516,15 +512,13 @@ class PermissionRulesView(BaseView): if page < 1: raise ValueError except ValueError: - return Response.json( - error_body("_page must be a positive integer", 400), status=400 - ) + return Response.error("_page must be a positive integer", 400) try: page_size = parse_size_limit( request.args.get("_size"), default=50, maximum=200 ) except ValueError as ex: - return Response.json(error_body(str(ex), 400), status=400) + return Response.error(str(ex), 400) offset = (page - 1) * page_size from datasette.utils.actions_sql import build_permission_rules_sql @@ -673,9 +667,7 @@ class PermissionCheckView(BaseView): # JSON API - action parameter is required action = request.args.get("action") if not action: - return Response.json( - error_body("action parameter is required", 400), status=400 - ) + return Response.error("action parameter is required", 400) parent = request.args.get("parent") child = request.args.get("child") diff --git a/datasette/views/stored_queries.py b/datasette/views/stored_queries.py index 3273d13c..d64f37d2 100644 --- a/datasette/views/stored_queries.py +++ b/datasette/views/stored_queries.py @@ -5,7 +5,7 @@ from datasette.stored_queries import stored_query_to_dict from datasette.utils import UNSTABLE_API_MESSAGE, sqlite3, tilde_decode from datasette.utils.asgi import Response -from .base import BaseView, _error +from .base import BaseView from .query_helpers import ( QueryValidationError, _as_bool, @@ -34,12 +34,14 @@ class QueryParametersView(BaseView): resource=DatabaseResource(db.name), actor=request.actor, ): - return _block_framing(_error(["Permission denied: need execute-sql"], 403)) + return _block_framing( + Response.error(["Permission denied: need execute-sql"], 403) + ) invalid_keys = set(request.args) - {"sql"} if invalid_keys: return _block_framing( - _error( + Response.error( ["Invalid keys: {}".format(", ".join(sorted(invalid_keys)))], 400, ) @@ -47,7 +49,7 @@ class QueryParametersView(BaseView): try: parameters = _derived_query_parameters(request.args.get("sql") or "") except QueryValidationError as ex: - return _block_framing(_error([ex.message], ex.status)) + return _block_framing(Response.error([ex.message], ex.status)) return _block_framing( Response.json( { @@ -95,7 +97,7 @@ class QueryListView(BaseView): is_write = _as_optional_bool(request.args.get("is_write"), "is_write") is_private = _as_optional_bool(request.args.get("is_private"), "is_private") except QueryValidationError as ex: - return _error([ex.message], ex.status) + return Response.error([ex.message], ex.status) page = await self.ds.list_queries( database, @@ -306,18 +308,22 @@ class QueryCreateAnalyzeView(BaseView): resource=DatabaseResource(db.name), actor=request.actor, ): - return _block_framing(_error(["Permission denied: need execute-sql"], 403)) + return _block_framing( + Response.error(["Permission denied: need execute-sql"], 403) + ) if not await self.ds.allowed( action="store-query", resource=DatabaseResource(db.name), actor=request.actor, ): - return _block_framing(_error(["Permission denied: need store-query"], 403)) + return _block_framing( + Response.error(["Permission denied: need store-query"], 403) + ) invalid_keys = set(request.args) - {"sql"} if invalid_keys: return _block_framing( - _error( + Response.error( ["Invalid keys: {}".format(", ".join(sorted(invalid_keys)))], 400, ) @@ -352,13 +358,13 @@ class QueryStoreView(QueryCreateView): resource=DatabaseResource(db.name), actor=request.actor, ): - return _error(["Permission denied: need execute-sql"], 403) + return Response.error(["Permission denied: need execute-sql"], 403) if not await self.ds.allowed( action="store-query", resource=DatabaseResource(db.name), actor=request.actor, ): - return _error(["Permission denied: need store-query"], 403) + return Response.error(["Permission denied: need store-query"], 403) is_json = False query_data = {} @@ -375,7 +381,7 @@ class QueryStoreView(QueryCreateView): return await self._error_response( request, db, query_data, ex.message, ex.status ) - return _error([ex.message], ex.status) + return Response.error([ex.message], ex.status) prepared.pop("analysis") name = prepared.pop("name") @@ -384,7 +390,7 @@ class QueryStoreView(QueryCreateView): except sqlite3.IntegrityError as ex: if not is_json and isinstance(query_data, dict): return await self._error_response(request, db, query_data, str(ex), 400) - return _error([str(ex)], 400) + return Response.error([str(ex)], 400) query = await self.ds.get_query(db.name, name) assert query is not None @@ -409,13 +415,13 @@ class QueryDefinitionView(BaseView): query_name = tilde_decode(request.url_vars["query"]) query = await self.ds.get_query(db.name, query_name) if query is None: - return _error(["Query not found: {}".format(query_name)], 404) + return Response.error(["Query not found: {}".format(query_name)], 404) if not await self.ds.allowed( action="view-query", resource=QueryResource(db.name, query_name), actor=request.actor, ): - return _error(["Permission denied"], 403) + return Response.error(["Permission denied"], 403) return Response.json( { "ok": True, @@ -433,15 +439,17 @@ class QueryUpdateView(BaseView): query_name = tilde_decode(request.url_vars["query"]) existing = await self.ds.get_query(db.name, query_name) if existing is None: - return _error(["Query not found: {}".format(query_name)], 404) + return Response.error(["Query not found: {}".format(query_name)], 404) if not await self.ds.allowed( action="update-query", resource=QueryResource(db.name, query_name), actor=request.actor, ): - return _error(["Permission denied: need update-query"], 403) + return Response.error(["Permission denied: need update-query"], 403) if existing.is_trusted: - return _error(["Trusted queries cannot be updated using the API"], 403) + return Response.error( + ["Trusted queries cannot be updated using the API"], 403 + ) try: data, _ = await _json_or_form_payload(request) @@ -467,7 +475,7 @@ class QueryUpdateView(BaseView): self.ds, request, db, existing, update ) except QueryValidationError as ex: - return _error([ex.message], ex.status) + return Response.error([ex.message], ex.status) await self.ds.update_query(db.name, query_name, **update_kwargs) if data.get("return"): @@ -524,32 +532,32 @@ class QueryEditView(BaseView): async def get(self, request): db, query_name, existing = await self._load(request) if existing is None: - return _error(["Query not found: {}".format(query_name)], 404) + return Response.error(["Query not found: {}".format(query_name)], 404) await self.ds.ensure_permission( action="update-query", resource=QueryResource(db.name, query_name), actor=request.actor, ) if existing.is_trusted: - return _error(["Trusted queries cannot be edited"], 403) + return Response.error(["Trusted queries cannot be edited"], 403) return await self._render_form(request, db, existing) async def post(self, request): db, query_name, existing = await self._load(request) if existing is None: - return _error(["Query not found: {}".format(query_name)], 404) + return Response.error(["Query not found: {}".format(query_name)], 404) if not await self.ds.allowed( action="update-query", resource=QueryResource(db.name, query_name), actor=request.actor, ): - return _error(["Permission denied: need update-query"], 403) + return Response.error(["Permission denied: need update-query"], 403) if existing.is_trusted: - return _error(["Trusted queries cannot be edited"], 403) + return Response.error(["Trusted queries cannot be edited"], 403) data, _ = await _json_or_form_payload(request) if not isinstance(data, dict): - return _error(["Invalid form submission"], 400) + return Response.error(["Invalid form submission"], 400) sql = data.get("sql") sql = existing.sql if sql is None else sql.strip() title = data.get("title") or "" @@ -621,14 +629,16 @@ class QueryDeleteView(BaseView): async def get(self, request): db, query_name, existing = await self._load(request) if existing is None: - return _error(["Query not found: {}".format(query_name)], 404) + return Response.error(["Query not found: {}".format(query_name)], 404) await self.ds.ensure_permission( action="delete-query", resource=QueryResource(db.name, query_name), actor=request.actor, ) if existing.is_trusted: - return _error(["Trusted queries cannot be deleted using the API"], 403) + return Response.error( + ["Trusted queries cannot be deleted using the API"], 403 + ) return await self.render( ["query_delete.html"], request, @@ -643,15 +653,17 @@ class QueryDeleteView(BaseView): async def post(self, request): db, query_name, existing = await self._load(request) if existing is None: - return _error(["Query not found: {}".format(query_name)], 404) + return Response.error(["Query not found: {}".format(query_name)], 404) if not await self.ds.allowed( action="delete-query", resource=QueryResource(db.name, query_name), actor=request.actor, ): - return _error(["Permission denied: need delete-query"], 403) + return Response.error(["Permission denied: need delete-query"], 403) if existing.is_trusted: - return _error(["Trusted queries cannot be deleted using the API"], 403) + return Response.error( + ["Trusted queries cannot be deleted using the API"], 403 + ) data, is_json = await _json_or_form_payload(request) await self.ds.remove_query(db.name, query_name) diff --git a/datasette/views/table.py b/datasette/views/table.py index 80ad1aab..8ce4b711 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -60,7 +60,7 @@ from dataclasses import dataclass, field from datasette.extras import ExtraScope from . import Context, from_extra -from .base import BaseView, DatasetteError, _error, stream_csv +from .base import BaseView, DatasetteError, stream_csv from .database import QueryView from .table_create_alter import ( ALTER_TABLE_COLUMN_TYPES, @@ -1035,7 +1035,7 @@ class TableInsertView(BaseView): try: resolved = await self.ds.resolve_table(request) except NotFound as e: - return _error([e.args[0]], 404) + return Response.error([e.args[0]], 404) db = resolved.db database_name = db.name table_name = resolved.table @@ -1043,7 +1043,7 @@ class TableInsertView(BaseView): # Table must exist (may handle table creation in the future) db = self.ds.get_database(database_name) if not await db.table_exists(table_name): - return _error(["Table not found: {}".format(table_name)], 404) + return Response.error(["Table not found: {}".format(table_name)], 404) if upsert: # Must have insert-row AND upsert-row permissions @@ -1059,7 +1059,7 @@ class TableInsertView(BaseView): actor=request.actor, ) ): - return _error( + return Response.error( ["Permission denied: need both insert-row and update-row"], 403 ) else: @@ -1069,10 +1069,10 @@ class TableInsertView(BaseView): resource=TableResource(database=database_name, table=table_name), actor=request.actor, ): - return _error(["Permission denied"], 403) + return Response.error(["Permission denied"], 403) if not db.is_mutable: - return _error(["Database is immutable"], 403) + return Response.error(["Database is immutable"], 403) pks = await db.primary_keys(table_name) @@ -1081,20 +1081,20 @@ class TableInsertView(BaseView): request, db, table_name, pks, upsert ) except PayloadTooLarge as e: - return _error([str(e)], 413) + return Response.error([str(e)], 413) if errors: - return _error(errors, 400) + return Response.error(errors, 400) try: rows = decode_write_json_rows(rows) except WriteJsonValueError as e: - return _error([str(e)], 400) + return Response.error([str(e)], 400) # Validate column types ct_errors = await _validate_column_types( self.ds, database_name, table_name, rows ) if ct_errors: - return _error(ct_errors, 400) + return Response.error(ct_errors, 400) num_rows = len(rows) @@ -1108,14 +1108,16 @@ class TableInsertView(BaseView): alter = extras.get("alter") if upsert and (ignore or replace): - return _error(["Upsert does not support ignore or replace"], 400) + return Response.error(["Upsert does not support ignore or replace"], 400) if replace and not await self.ds.allowed( action="update-row", resource=TableResource(database=database_name, table=table_name), actor=request.actor, ): - return _error(['Permission denied: need update-row to use "replace"'], 403) + return Response.error( + ['Permission denied: need update-row to use "replace"'], 403 + ) initial_schema = None if alter: @@ -1125,7 +1127,7 @@ class TableInsertView(BaseView): resource=TableResource(database=database_name, table=table_name), actor=request.actor, ): - return _error(["Permission denied for alter-table"], 403) + return Response.error(["Permission denied for alter-table"], 403) # Track initial schema to check if it changed later initial_schema = await db.execute_fn( lambda conn: sqlite_utils.Database(conn)[table_name].schema @@ -1165,7 +1167,7 @@ class TableInsertView(BaseView): try: rows = await db.execute_write_fn(insert_or_upsert_rows, request=request) except Exception as e: - return _error([str(e)]) + return Response.error([str(e)]) result = {"ok": True} if should_return: if upsert: @@ -1246,7 +1248,7 @@ class TableSetColumnTypeView(BaseView): try: resolved = await self.ds.resolve_table(request) except NotFound as e: - return _error([e.args[0]], 404) + return Response.error([e.args[0]], 404) database_name = resolved.db.name table_name = resolved.table @@ -1256,39 +1258,39 @@ class TableSetColumnTypeView(BaseView): resource=TableResource(database=database_name, table=table_name), actor=request.actor, ): - return _error(["Permission denied"], 403) + return Response.error(["Permission denied"], 403) try: data = await request.json() except json.JSONDecodeError as e: - return _error(["Invalid JSON: {}".format(e)], 400) + return Response.error(["Invalid JSON: {}".format(e)], 400) except PayloadTooLarge as e: - return _error([str(e)], 413) + return Response.error([str(e)], 413) if not isinstance(data, dict): - return _error(["JSON must be a dictionary"], 400) + return Response.error(["JSON must be a dictionary"], 400) invalid_keys = set(data.keys()) - {"column", "column_type"} if invalid_keys: - return _error( + return Response.error( ['Invalid parameter: "{}"'.format('", "'.join(sorted(invalid_keys)))], 400, ) if "column" not in data: - return _error(['"column" is required'], 400) + return Response.error(['"column" is required'], 400) column = data["column"] if not isinstance(column, str): - return _error(['"column" must be a string'], 400) + return Response.error(['"column" must be a string'], 400) if "column_type" not in data: - return _error(['"column_type" is required'], 400) + return Response.error(['"column_type" is required'], 400) column_details = await self.ds._get_resource_column_details( database_name, table_name ) if column not in column_details: - return _error(["Column not found: {}".format(column)], 400) + return Response.error(["Column not found: {}".format(column)], 400) column_type_data = data["column_type"] if column_type_data is None: @@ -1305,11 +1307,11 @@ class TableSetColumnTypeView(BaseView): ) if not isinstance(column_type_data, dict): - return _error(['"column_type" must be an object or null'], 400) + return Response.error(['"column_type" must be an object or null'], 400) invalid_column_type_keys = set(column_type_data.keys()) - {"type", "config"} if invalid_column_type_keys: - return _error( + return Response.error( [ 'Invalid column_type parameter: "{}"'.format( '", "'.join(sorted(invalid_column_type_keys)) @@ -1319,24 +1321,24 @@ class TableSetColumnTypeView(BaseView): ) if "type" not in column_type_data: - return _error(['"column_type.type" is required'], 400) + return Response.error(['"column_type.type" is required'], 400) column_type = column_type_data["type"] if not isinstance(column_type, str): - return _error(['"column_type.type" must be a string'], 400) + return Response.error(['"column_type.type" must be a string'], 400) config = column_type_data.get("config") if config is not None and not isinstance(config, dict): - return _error(['"column_type.config" must be a dictionary'], 400) + return Response.error(['"column_type.config" must be a dictionary'], 400) if column_type not in self.ds._column_types: - return _error(["Unknown column type: {}".format(column_type)], 400) + return Response.error(["Unknown column type: {}".format(column_type)], 400) try: await self.ds.set_column_type( database_name, table_name, column, column_type, config ) except ValueError as e: - return _error([str(e)], 400) + return Response.error([str(e)], 400) return Response.json( { @@ -1360,22 +1362,22 @@ class TableDropView(BaseView): try: resolved = await self.ds.resolve_table(request) except NotFound as e: - return _error([e.args[0]], 404) + return Response.error([e.args[0]], 404) db = resolved.db database_name = db.name table_name = resolved.table # Table must exist db = self.ds.get_database(database_name) if not await db.table_exists(table_name): - return _error(["Table not found: {}".format(table_name)], 404) + return Response.error(["Table not found: {}".format(table_name)], 404) if not await self.ds.allowed( action="drop-table", resource=TableResource(database=database_name, table=table_name), actor=request.actor, ): - return _error(["Permission denied"], 403) + return Response.error(["Permission denied"], 403) if not db.is_mutable: - return _error(["Database is immutable"], 403) + return Response.error(["Database is immutable"], 403) confirm = False try: data = await request.json() @@ -1383,7 +1385,7 @@ class TableDropView(BaseView): except json.JSONDecodeError: pass except PayloadTooLarge as e: - return _error([str(e)], 413) + return Response.error([str(e)], 413) if not confirm: return Response.json( diff --git a/datasette/views/table_create_alter.py b/datasette/views/table_create_alter.py index c3e06c29..4deeafcc 100644 --- a/datasette/views/table_create_alter.py +++ b/datasette/views/table_create_alter.py @@ -29,7 +29,7 @@ from datasette.utils import ( from datasette.utils.asgi import NotFound, PayloadTooLarge, Response from datasette.utils.sqlite import sqlite_hidden_table_names -from .base import BaseView, _error +from .base import BaseView CREATE_TABLE_COLUMN_TYPES = ["text", "integer", "float", "blob"] CREATE_TABLE_SQLITE_TYPES = { @@ -805,22 +805,22 @@ class TableCreateView(BaseView): resource=DatabaseResource(database=database_name), actor=request.actor, ): - return _error(["Permission denied"], 403) + return Response.error(["Permission denied"], 403) try: data = await request.json() except json.JSONDecodeError as e: - return _error(["Invalid JSON: {}".format(e)]) + return Response.error(["Invalid JSON: {}".format(e)]) except PayloadTooLarge as e: - return _error([str(e)], 413) + return Response.error([str(e)], 413) if not isinstance(data, dict): - return _error(["JSON must be an object"]) + return Response.error(["JSON must be an object"]) try: create_request = CreateTableRequest.model_validate(data) except ValidationError as e: - return _error(_create_table_pydantic_errors(e)) + return Response.error(_create_table_pydantic_errors(e)) ignore = create_request.ignore replace = create_request.replace @@ -832,7 +832,7 @@ class TableCreateView(BaseView): resource=DatabaseResource(database=database_name), actor=request.actor, ): - return _error(["Permission denied: need update-row"], 403) + return Response.error(["Permission denied: need update-row"], 403) table_name = create_request.table table_exists = await db.table_exists(table_name) @@ -846,11 +846,11 @@ class TableCreateView(BaseView): resource=DatabaseResource(database=database_name), actor=request.actor, ): - return _error(["Permission denied: need insert-row"], 403) + return Response.error(["Permission denied: need insert-row"], 403) try: rows = decode_write_json_rows(rows) except WriteJsonValueError as e: - return _error([str(e)], 400) + return Response.error([str(e)], 400) alter = False if rows: @@ -865,7 +865,9 @@ class TableCreateView(BaseView): resource=DatabaseResource(database=database_name), actor=request.actor, ): - return _error(["Permission denied: need alter-table"], 403) + return Response.error( + ["Permission denied: need alter-table"], 403 + ) alter = True pk = create_request.pk @@ -881,7 +883,7 @@ class TableCreateView(BaseView): elif len(actual_pks) > 1 and pks and set(pks) != set(actual_pks): bad_pks = True if bad_pks: - return _error(["pk cannot be changed for existing table"]) + return Response.error(["pk cannot be changed for existing table"]) pks = actual_pks initial_schema = None @@ -924,7 +926,7 @@ class TableCreateView(BaseView): try: schema = await db.execute_write_fn(create_table, request=request) except Exception as e: - return _error([str(e)]) + return Response.error([str(e)]) if initial_schema is not None and initial_schema != schema: await self.ds.track_event( @@ -999,7 +1001,7 @@ class DatabaseForeignKeyTargetsView(BaseView): actor=request.actor, ) if not (can_create_table or can_alter_table): - return _error(["Permission denied: need create-table"], 403) + return Response.error(["Permission denied: need create-table"], 403) hidden_tables = await db.execute_fn( lambda conn: set(sqlite_hidden_table_names(conn)) @@ -1028,21 +1030,21 @@ class TableForeignKeySuggestionsView(BaseView): try: resolved = await self.ds.resolve_table(request) except NotFound as e: - return _error([e.args[0]], 404) + return Response.error([e.args[0]], 404) db = resolved.db database_name = db.name table_name = resolved.table if resolved.is_view: - return _error(["Cannot suggest foreign keys for a view"], 400) + return Response.error(["Cannot suggest foreign keys for a view"], 400) if not await self.ds.allowed( action="alter-table", resource=TableResource(database=database_name, table=table_name), actor=request.actor, ): - return _error(["Permission denied: need alter-table"], 403) + return Response.error(["Permission denied: need alter-table"], 403) source_columns, targets, current_by_column = await db.execute_fn( lambda conn: _foreign_key_suggestion_metadata(conn, table_name) @@ -1150,7 +1152,7 @@ class TableAlterView(BaseView): try: resolved = await self.ds.resolve_table(request) except NotFound as e: - return _error([e.args[0]], 404) + return Response.error([e.args[0]], 404) db = resolved.db database_name = db.name @@ -1161,25 +1163,25 @@ class TableAlterView(BaseView): resource=TableResource(database=database_name, table=table_name), actor=request.actor, ): - return _error(["Permission denied: need alter-table"], 403) + return Response.error(["Permission denied: need alter-table"], 403) if not db.is_mutable: - return _error(["Database is immutable"], 403) + return Response.error(["Database is immutable"], 403) try: data = await request.json() except json.JSONDecodeError as e: - return _error(["Invalid JSON: {}".format(e)], 400) + return Response.error(["Invalid JSON: {}".format(e)], 400) except PayloadTooLarge as e: - return _error([str(e)], 413) + return Response.error([str(e)], 413) if not isinstance(data, dict): - return _error(["JSON must be a dictionary"], 400) + return Response.error(["JSON must be a dictionary"], 400) try: alter_request = AlterTableRequest.model_validate(data) except ValidationError as e: - return _error(_pydantic_errors(e), 400) + return Response.error(_pydantic_errors(e), 400) def alter_table(conn): before_schema = _table_schema_from_conn(conn, table_name) @@ -1328,7 +1330,7 @@ class TableAlterView(BaseView): alter_table, request=request ) except Exception as e: - return _error([str(e)], 400) + return Response.error([str(e)], 400) altered = before_schema != after_schema if altered: diff --git a/docs/internals.rst b/docs/internals.rst index e5e7aca5..7258e6f3 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -284,7 +284,7 @@ For example: content_type="application/xml; charset=utf-8", ) -The quickest way to create responses is using the ``Response.text(...)``, ``Response.html(...)``, ``Response.json(...)`` or ``Response.redirect(...)`` helper methods: +The quickest way to create responses is using the ``Response.text(...)``, ``Response.html(...)``, ``Response.json(...)``, ``Response.error(...)`` or ``Response.redirect(...)`` helper methods: .. code-block:: python @@ -295,6 +295,8 @@ The quickest way to create responses is using the ``Response.text(...)``, ``Resp text_response = Response.text( "This will become utf-8 encoded text" ) + # A JSON error in Datasette's standard error format: + error_response = Response.error("Cannot do that", 400) # Redirects are served as 302, unless you pass status=301: redirect_response = Response.redirect( "https://latest.datasette.io/" @@ -304,6 +306,8 @@ Each of these responses will use the correct corresponding content-type - ``text Each of the helper methods take optional ``status=`` and ``headers=`` arguments, documented above. +``Response.error(messages, status=400)`` returns a JSON error in the :ref:`standard Datasette error format `. ``messages`` can be a single string or a list of strings. Use this for JSON-only endpoints; if your error should content-negotiate between JSON and HTML, raise ``Forbidden``, ``NotFound``, ``BadRequest`` or ``DatasetteError`` instead and Datasette's error handling will build the appropriate response. + .. _internals_response_asgi_send: Returning a response with .asgi_send(send) diff --git a/tests/test_internals_response.py b/tests/test_internals_response.py index 820b20b2..2366dcde 100644 --- a/tests/test_internals_response.py +++ b/tests/test_internals_response.py @@ -1,4 +1,5 @@ from datasette.utils.asgi import Response +import json import pytest @@ -52,3 +53,26 @@ async def test_response_set_cookie(): }, {"type": "http.response.body", "body": b""}, ] == events + + +def test_response_error_single_message(): + response = Response.error("Method not allowed", 405) + assert response.status == 405 + assert response.content_type == "application/json; charset=utf-8" + assert json.loads(response.body) == { + "ok": False, + "error": "Method not allowed", + "errors": ["Method not allowed"], + "status": 405, + } + + +def test_response_error_message_list_and_default_status(): + response = Response.error(["First problem", "Second problem"]) + assert response.status == 400 + assert json.loads(response.body) == { + "ok": False, + "error": "First problem; Second problem", + "errors": ["First problem", "Second problem"], + "status": 400, + } From 4874c292860c7ad67da2a730d6e58dde2718adda Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 23:59:44 +0000 Subject: [PATCH 045/131] Remove the next_url extra - the key is always present next_url became a default table JSON key alongside next, making the extra a no-op. Requesting ?_extra=next_url now returns the standard unknown-extra 400 error. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/views/table.py | 6 +++++- datasette/views/table_extras.py | 17 ----------------- docs/json_api.rst | 13 +------------ docs/template_context.rst | 2 +- tests/test_api.py | 2 +- tests/test_error_shape.py | 8 ++++++++ tests/test_table_api.py | 18 +++++++----------- 7 files changed, 23 insertions(+), 43 deletions(-) diff --git a/datasette/views/table.py b/datasette/views/table.py index 8ce4b711..f7edd744 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -107,7 +107,6 @@ class TableContext(Context): human_description_en: str = from_extra() is_view: bool = from_extra() metadata: dict = from_extra() - next_url: str = from_extra() primary_keys: list = from_extra() private: bool = from_extra() query: dict = from_extra() @@ -124,6 +123,11 @@ class TableContext(Context): metadata={"help": "True if the data for this page was retrieved without errors"} ) next: str = field(metadata={"help": "Pagination token for the next page, or None"}) + next_url: str = field( + metadata={ + "help": "Full URL for the next page of results, or None if there are no more pages. See :ref:`json_api_pagination`." + } + ) count_truncated: bool = field( metadata={ "help": "True if ``count`` is a capped lower bound rather than an exact total, because Datasette stopped counting after its configured row-count limit." diff --git a/datasette/views/table_extras.py b/datasette/views/table_extras.py index e36e8161..50002d92 100644 --- a/datasette/views/table_extras.py +++ b/datasette/views/table_extras.py @@ -321,21 +321,6 @@ class HumanDescriptionEnExtra(Extra): return human_description_en -class NextUrlExtra(Extra): - description = "Full URL for the next page of results" - example = ExtraExample( - "/fixtures/facetable.json?_size=1&_extra=next_url", - note=( - "``null`` if there are no more pages of results. " - "See :ref:`json_api_pagination`." - ), - ) - scopes = {ExtraScope.TABLE} - - async def resolve(self, context): - return context.next_url - - class ColumnsExtra(Extra): description = "List of column names returned by this table, row or query." example = ExtraExample("/fixtures/facetable.json?_extra=columns") @@ -1250,7 +1235,6 @@ TABLE_EXTRA_BUNDLES = { "count", "count_sql", "human_description_en", - "next_url", "metadata", "query", "columns", @@ -1286,7 +1270,6 @@ TABLE_EXTRA_CLASSES = [ SuggestedFacetsExtra, FacetInstancesProvider, HumanDescriptionEnExtra, - NextUrlExtra, ColumnsExtra, AllColumnsExtra, PrimaryKeysExtra, diff --git a/docs/json_api.rst b/docs/json_api.rst index 893af634..ec60a3a9 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -333,7 +333,7 @@ These can be repeated or comma-separated: :: - ?_extra=columns&_extra=count,next_url + ?_extra=columns&_extra=count,count_sql Requesting an ``_extra`` name that does not exist returns a ``400`` error in the :ref:`standard error format `, for example ``{"ok": false, "error": "Unknown _extra: nope", ...}``. @@ -437,17 +437,6 @@ The available table extras are listed below. "where state = \"CA\" sorted by pk" -``next_url`` - Full URL for the next page of results - - ``GET /fixtures/facetable.json?_size=1&_extra=next_url`` - - ``null`` if there are no more pages of results. See :ref:`json_api_pagination`. - - .. code-block:: json - - "http://localhost/fixtures/facetable.json?_size=1&_extra=next_url&_next=1" - ``columns`` List of column names returned by this table, row or query. diff --git a/docs/template_context.rst b/docs/template_context.rst index e9447058..e445b335 100644 --- a/docs/template_context.rst +++ b/docs/template_context.rst @@ -329,7 +329,7 @@ Many of these keys are shared with the :ref:`JSON API ` for this page. Pagination token for the next page, or None ``next_url`` - ``str`` - Full URL for the next page of results + Full URL for the next page of results, or None if there are no more pages. See :ref:`json_api_pagination`. ``ok`` - ``bool`` True if the data for this page was retrieved without errors diff --git a/tests/test_api.py b/tests/test_api.py index 9a96f14f..a15a507c 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -722,7 +722,7 @@ def test_config_cache_size(app_client_larger_cache_size): def test_config_force_https_urls(): with make_app_client(settings={"force_https_urls": True}) as client: response = client.get( - "/fixtures/facetable.json?_size=3&_facet=state&_extra=next_url,suggested_facets" + "/fixtures/facetable.json?_size=3&_facet=state&_extra=suggested_facets" ) assert response.json["next_url"].startswith("https://") assert response.json["facet_results"]["results"]["state"]["results"][0][ diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index 44856b36..768814fd 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -739,3 +739,11 @@ async def test_sql_interrupted_html_page_keeps_rich_error(ds_client): ) assert response.status_code == 400 assert " Date: Tue, 7 Jul 2026 00:25:09 +0000 Subject: [PATCH 046/131] /-/jump is a stable documented endpoint, not a debug exemption Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- docs/introspection.rst | 2 +- docs/json_api.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/introspection.rst b/docs/introspection.rst index ea94b70d..b78e4860 100644 --- a/docs/introspection.rst +++ b/docs/introspection.rst @@ -9,7 +9,7 @@ Each of these pages can be viewed in your browser. Add ``.json`` to the URL to g JSON responses that return an object include an ``"ok": true`` key, consistent with the rest of the :ref:`JSON API `. -The introspection endpoints documented on this page are covered by the :ref:`JSON API stability promise `, with the exception of the debug endpoints ``/-/threads``, ``/-/actions`` and ``/-/jump``, whose shapes may change in future releases. +The introspection endpoints documented on this page are covered by the :ref:`JSON API stability promise `, with the exception of the debug endpoints ``/-/threads`` and ``/-/actions``, whose shapes may change in future releases. .. _JsonDataView_metadata: diff --git a/docs/json_api.rst b/docs/json_api.rst index ec60a3a9..a96fd73d 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -48,7 +48,7 @@ Some JSON endpoints are **exempt** from this promise: debug playground. - Debug and support endpoints are documented so you can use them, but their JSON shapes are not frozen: :ref:`/-/threads `, - :ref:`/-/actions `, :ref:`/-/jump `, + :ref:`/-/actions `, the :ref:`permission debug endpoints ` (``/-/allowed``, ``/-/rules``, ``/-/check``) and the :ref:`table autocomplete endpoint `. From b23fc4ec48cbd68a072379a03aec3af7245084e3 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 00:32:29 +0000 Subject: [PATCH 047/131] Unreleased release notes for the JSON API consistency review Documents the canonical error format, the ok/envelope changes, the array-to-object endpoint conversions, 401s for invalid tokens, the pagination and page-size unification, removed legacy keys and formats, and the new Response.error(), TokenInvalid, count_truncated and unstable-marker APIs. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- docs/changelog.rst | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/docs/changelog.rst b/docs/changelog.rst index 4541981f..3230575d 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -17,6 +17,42 @@ Unreleased - The table and row JSON APIs now support ``?_extra=column_details`` for returning SQLite schema details for columns, including declared type, SQLite affinity, primary key, ``NOT NULL``, default and hidden-column metadata. - POST bodies that Datasette reads fully into memory - such as JSON submitted to the write API - are now capped by the new :ref:`setting_max_post_body_bytes` setting, defaulting to 2MB. Oversized requests are rejected with an HTTP 413 error as soon as the limit is exceeded, protecting smaller servers from memory exhaustion. File uploads are unaffected - ``request.form()`` streams those to disk and has its own separate limits. +This release also includes the results of a detailed consistency review of Datasette's JSON API in preparation for the 1.0 stable release. Several of these changes are backwards-incompatible with previous 1.0 alphas. The new :ref:`API stability documentation ` describes exactly which parts of the JSON API are covered by the 1.0 stability promise. + +JSON API: breaking changes +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- JSON error responses now use a single canonical format across every endpoint: ``{"ok": false, "error": "...", "errors": [...], "status": 400}``. The ``error`` key joins all error messages together, ``errors`` is the full list of messages and ``status`` always matches the HTTP status code. The legacy ``title`` key is no longer included in JSON errors (it remains available to the HTML error template), and endpoints that previously returned bare ``{"error": ...}`` objects have been updated. See :ref:`json_api_errors`. +- Every JSON object success response now includes ``"ok": true``, including introspection endpoints such as ``/-/versions`` and ``/-/settings``. +- ``/-/plugins.json``, ``/-/databases.json`` and ``/-/actions.json`` now return objects - ``{"ok": true, "plugins": [...]}`` and equivalents - instead of top-level JSON arrays, so these responses can gain additional keys in the future without a breaking change. The ``datasette plugins`` CLI command still outputs a plain array. +- ``/-/databases`` now only lists databases the current actor is allowed to view. It previously listed every attached database, including their filesystem paths, to any actor with ``view-instance``. +- Requests with an invalid or expired ``Authorization: Bearer`` token now receive a ``401`` status with the standard error body and a ``WWW-Authenticate: Bearer error="invalid_token"`` header, instead of being silently treated as unauthenticated. Bearer tokens that no registered token handler recognizes are still ignored, so authentication plugins with their own token formats keep working. Plugin :ref:`token handlers ` can raise the new ``datasette.TokenInvalid`` exception to trigger the same behavior. +- Permission errors for JSON requests now return the standard JSON error format with a ``403`` status. The default forbidden handling previously rendered an HTML error page even for ``.json`` requests. +- ``POST`` to a write canned query now returns a ``400`` error when the SQL fails to execute, instead of a ``200`` status with ``"ok": false`` in the body. The error response includes the standard error keys plus a ``"redirect"`` key. +- The :ref:`row update API ` with ``"return": true`` now responds with a ``"rows"`` list, matching insert and upsert, instead of a singular ``"row"`` object. +- Row delete write failures - such as a constraint violation raised by a trigger - now return ``400`` instead of ``500``, matching the other write endpoints. +- ``//-/query.json`` with a missing or blank ``?sql=`` parameter now returns a ``400`` error, as the CSV format already did, instead of a ``200`` with empty rows. +- Unknown ``?_extra=`` names now return a ``400`` error for JSON and other data formats, instead of being silently ignored. HTML pages continue to ignore unknown names. +- Table JSON responses now include ``next_url`` alongside ``next`` by default - both are ``null`` on the final page. The now-redundant ``?_extra=next_url`` parameter has been removed. +- The stored query list JSON no longer includes ``has_more`` - ``"next": null`` is the end-of-results signal across the whole API. This change also uncovered and fixed a bug where the query list ``next_url`` pointed at the HTML page and was a relative path; it is now an absolute URL that preserves the requested format. +- Stored query JSON objects no longer duplicate the list of parameter names as both ``params`` and ``parameters`` - only ``parameters`` remains in the output. ``params`` is still accepted as an input alias when creating or updating queries. +- Page size parameters are now consistent across the API: the stored query lists accept ``?_size=max`` and return a ``400`` error for values over the maximum instead of silently clamping them, and the ``/-/allowed`` and ``/-/rules`` permission debug endpoints renamed their ``page`` and ``page_size`` parameters to ``_page`` and ``_size``, matching the underscore grammar used by every other Datasette system parameter. +- ``/-/threads`` now requires the ``permissions-debug`` permission, since it exposes runtime internals such as file paths. It previously only required ``view-instance``. +- Trusted stored queries - those defined in configuration - can no longer be deleted through the JSON API or web interface, matching the existing restriction on editing them. +- The ``//-/schema`` endpoints now check the ``view-database`` permission before checking whether the database exists, so unauthorized actors can no longer probe for the existence of databases. +- SQL time limit errors in JSON responses are now a plain text message. The error string previously embedded an HTML fragment. +- The undocumented homepage JSON at ``/.json`` now returns ``databases`` as a list of objects rather than an object keyed by database name, matching every other collection in the API. +- The legacy ``.jsono`` format extension, long since superseded by ``?_shape=``, has been removed. + +JSON API: other improvements +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- The :ref:`write API ` endpoints now parse the request body as JSON regardless of the ``Content-Type`` header, so ``curl -d`` invocations work without remembering to set it. Invalid JSON is a ``400`` error. Cross-site request forgery remains prevented by Datasette's ``Origin`` and ``Sec-Fetch-Site`` checks. This also fixes a ``500`` error from the insert API when the ``Content-Type`` header was missing entirely. +- New ``Response.error(messages, status=400)`` helper for plugins that need to return a JSON error in Datasette's standard format. See :ref:`internals_response`. +- New ``count_truncated`` extra for table JSON, included automatically whenever ``count`` is requested. ``true`` means the count reached Datasette's counting limit and the real number of rows may be higher. See :ref:`json_api_extra`. +- JSON endpoints that are not part of the documented stable API now declare themselves with an ``"unstable"`` key in their responses, making the stability tier machine-readable. +- New documentation covering the grammar for :ref:`boolean query string arguments `, the reason :ref:`upsert ` returns ``200`` where insert returns ``201``, and advice for plugin authors on :ref:`naming secret configuration keys ` so that ``/-/config`` redacts them automatically. + .. _v1_0_a35: 1.0a35 (2026-06-23) From b83b12dd7abc5b0fce8353792d796bf3943e4a1a Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 01:13:33 +0000 Subject: [PATCH 048/131] Remove params input alias from the query create and update APIs The alias existed so API payloads could mirror the params key used by queries defined in datasette.yaml, but it was undocumented and untested, and the create endpoint is not part of the stable API. The API now only accepts parameters - sending params is a 400 Invalid keys error. The documented params key for queries in configuration is unchanged. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/views/query_helpers.py | 7 +++--- docs/changelog.rst | 2 +- tests/test_queries.py | 41 ++++++++++++++++++++++++++++++++ 3 files changed, 45 insertions(+), 5 deletions(-) diff --git a/datasette/views/query_helpers.py b/datasette/views/query_helpers.py index e9f85b6d..588891d4 100644 --- a/datasette/views/query_helpers.py +++ b/datasette/views/query_helpers.py @@ -33,7 +33,6 @@ _query_fields = { "hide_sql", "fragment", "parameters", - "params", "is_private", "on_success_message", "on_success_redirect", @@ -540,7 +539,7 @@ async def _prepare_query_create(datasette, request, db, data): raise QueryValidationError("Writable query fields require writable SQL") parameters = _coerce_query_parameters( - data.get("parameters", data.get("params")), + data.get("parameters"), derived, ) return { @@ -585,9 +584,9 @@ async def _prepare_query_update(datasette, request, db, existing: StoredQuery, u actor=request.actor, ) - if "parameters" in update or "params" in update: + if "parameters" in update: parameters = _coerce_query_parameters( - update.get("parameters", update.get("params")), + update.get("parameters"), derived, ) elif "sql" in update: diff --git a/docs/changelog.rst b/docs/changelog.rst index 3230575d..0079aaf2 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -35,7 +35,7 @@ JSON API: breaking changes - Unknown ``?_extra=`` names now return a ``400`` error for JSON and other data formats, instead of being silently ignored. HTML pages continue to ignore unknown names. - Table JSON responses now include ``next_url`` alongside ``next`` by default - both are ``null`` on the final page. The now-redundant ``?_extra=next_url`` parameter has been removed. - The stored query list JSON no longer includes ``has_more`` - ``"next": null`` is the end-of-results signal across the whole API. This change also uncovered and fixed a bug where the query list ``next_url`` pointed at the HTML page and was a relative path; it is now an absolute URL that preserves the requested format. -- Stored query JSON objects no longer duplicate the list of parameter names as both ``params`` and ``parameters`` - only ``parameters`` remains in the output. ``params`` is still accepted as an input alias when creating or updating queries. +- Stored query JSON objects no longer duplicate the list of parameter names as both ``params`` and ``parameters`` - only ``parameters`` remains. The query create and update APIs no longer accept ``params`` as an input alias either; ``params`` is still the documented key for :ref:`queries defined in configuration `. - Page size parameters are now consistent across the API: the stored query lists accept ``?_size=max`` and return a ``400`` error for values over the maximum instead of silently clamping them, and the ``/-/allowed`` and ``/-/rules`` permission debug endpoints renamed their ``page`` and ``page_size`` parameters to ``_page`` and ``_size``, matching the underscore grammar used by every other Datasette system parameter. - ``/-/threads`` now requires the ``permissions-debug`` permission, since it exposes runtime internals such as file paths. It previously only required ``view-instance``. - Trusted stored queries - those defined in configuration - can no longer be deleted through the JSON API or web interface, matching the existing restriction on editing them. diff --git a/tests/test_queries.py b/tests/test_queries.py index c25ec358..1cda740e 100644 --- a/tests/test_queries.py +++ b/tests/test_queries.py @@ -1125,6 +1125,47 @@ async def test_query_update_api_rejects_config_only_fields(): assert query.on_success_message_sql is None +@pytest.mark.asyncio +async def test_query_api_rejects_params_alias(): + # "params" is a datasette.yaml configuration key, not an API input - + # the API only accepts "parameters" + ds = Datasette(memory=True, default_deny=True) + ds.root_enabled = True + db = ds.add_memory_database("query_params_alias", name="data") + await db.execute_write("create table dogs (id integer primary key, name text)") + await ds.invoke_startup() + + store_response = await ds.client.post( + "/data/-/queries/store", + actor={"id": "root"}, + json={ + "query": { + "name": "by_name", + "sql": "select * from dogs where name = :name", + "params": ["name"], + } + }, + ) + assert store_response.status_code == 400 + assert store_response.json()["errors"] == ["Invalid keys: params"] + assert await ds.get_query("data", "by_name") is None + + await ds.add_query( + "data", + "editable", + "select * from dogs where name = :name", + source="user", + owner_id="root", + ) + update_response = await ds.client.post( + "/data/editable/-/update", + actor={"id": "root"}, + json={"update": {"params": ["name"]}}, + ) + assert update_response.status_code == 400 + assert update_response.json()["errors"] == ["Invalid keys: params"] + + @pytest.mark.asyncio async def test_query_update_api_rejects_trusted_queries_but_internal_update_allowed(): ds = Datasette( From 57ce1a059fce43b32cc54ca4afeeb645e7a6f5af Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 01:13:33 +0000 Subject: [PATCH 049/131] Tighten unstable marker release note Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- docs/changelog.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index 0079aaf2..60918ddf 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -50,7 +50,7 @@ JSON API: other improvements - The :ref:`write API ` endpoints now parse the request body as JSON regardless of the ``Content-Type`` header, so ``curl -d`` invocations work without remembering to set it. Invalid JSON is a ``400`` error. Cross-site request forgery remains prevented by Datasette's ``Origin`` and ``Sec-Fetch-Site`` checks. This also fixes a ``500`` error from the insert API when the ``Content-Type`` header was missing entirely. - New ``Response.error(messages, status=400)`` helper for plugins that need to return a JSON error in Datasette's standard format. See :ref:`internals_response`. - New ``count_truncated`` extra for table JSON, included automatically whenever ``count`` is requested. ``true`` means the count reached Datasette's counting limit and the real number of rows may be higher. See :ref:`json_api_extra`. -- JSON endpoints that are not part of the documented stable API now declare themselves with an ``"unstable"`` key in their responses, making the stability tier machine-readable. +- JSON endpoints that are not part of the documented stable API now declare themselves with an ``"unstable"`` key in their responses. - New documentation covering the grammar for :ref:`boolean query string arguments `, the reason :ref:`upsert ` returns ``200`` where insert returns ``201``, and advice for plugin authors on :ref:`naming secret configuration keys ` so that ``/-/config`` redacts them automatically. .. _v1_0_a35: From 4a853cb10ca6fd0c1aaecf0af7dd26665424c540 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 05:18:54 +0000 Subject: [PATCH 050/131] Use UNSTABLE_API_MESSAGE constant in tests Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- tests/test_api.py | 6 ++---- tests/test_permissions.py | 6 ++---- tests/test_queries.py | 6 ++---- tests/test_success_envelope.py | 15 +++++---------- 4 files changed, 11 insertions(+), 22 deletions(-) diff --git a/tests/test_api.py b/tests/test_api.py index a15a507c..235d394b 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -1,5 +1,6 @@ from datasette.app import Datasette from datasette.plugins import DEFAULT_PLUGINS +from datasette.utils import UNSTABLE_API_MESSAGE from datasette.utils.sqlite import sqlite_version from datasette.version import __version__ from .fixtures import make_app_client, EXPECTED_PLUGINS @@ -251,10 +252,7 @@ def test_no_files_uses_memory_database(app_client_no_files): assert response.status == 200 assert { "ok": True, - "unstable": ( - "This API is not part of Datasette's stable interface" - " and may change at any time" - ), + "unstable": UNSTABLE_API_MESSAGE, "databases": [ { "name": "_memory", diff --git a/tests/test_permissions.py b/tests/test_permissions.py index f8d2c808..88fe577f 100644 --- a/tests/test_permissions.py +++ b/tests/test_permissions.py @@ -3,6 +3,7 @@ from asgiref.sync import async_to_sync from datasette.app import Datasette from datasette.cli import cli from datasette.default_permissions import restrictions_allow_action +from datasette.utils import UNSTABLE_API_MESSAGE from .fixtures import assert_permissions_checked, make_app_client from click.testing import CliRunner from bs4 import BeautifulSoup as Soup @@ -740,10 +741,7 @@ async def test_actor_restricted_permissions( } expected = { "ok": True, - "unstable": ( - "This API is not part of Datasette's stable interface" - " and may change at any time" - ), + "unstable": UNSTABLE_API_MESSAGE, "action": permission, "allowed": expected_result, "resource": expected_resource, diff --git a/tests/test_queries.py b/tests/test_queries.py index 1cda740e..a7e492eb 100644 --- a/tests/test_queries.py +++ b/tests/test_queries.py @@ -8,6 +8,7 @@ from bs4 import BeautifulSoup as Soup from datasette.app import Datasette from datasette.resources import DatabaseResource, QueryResource from datasette.stored_queries import StoredQuery, StoredQueryPage +from datasette.utils import UNSTABLE_API_MESSAGE from datasette.utils.asgi import Forbidden from datasette.utils.sqlite import sqlite3, supports_returning @@ -2183,10 +2184,7 @@ async def test_query_parameters_endpoint_uses_get_sql_only(): assert response.status_code == 200 assert response.json() == { "ok": True, - "unstable": "{}".format( - "This API is not part of Datasette's stable interface" - " and may change at any time" - ), + "unstable": UNSTABLE_API_MESSAGE, "parameters": ["name", "id"], } assert permission_denied_response.status_code == 403 diff --git a/tests/test_success_envelope.py b/tests/test_success_envelope.py index 68b042e5..3c413d73 100644 --- a/tests/test_success_envelope.py +++ b/tests/test_success_envelope.py @@ -8,7 +8,7 @@ objects separately - see /-/plugins, /-/databases, /-/actions.) import pytest from datasette.app import Datasette -from datasette.utils import sqlite3 +from datasette.utils import sqlite3, UNSTABLE_API_MESSAGE @pytest.fixture @@ -114,11 +114,6 @@ async def test_actions_json_is_object(ds_envelope): assert "view-instance" in {action["name"] for action in data["actions"]} -UNSTABLE_MESSAGE = ( - "This API is not part of Datasette's stable interface and may change at any time" -) - - @pytest.mark.asyncio @pytest.mark.parametrize( "path", @@ -137,7 +132,7 @@ async def test_undocumented_endpoints_report_unstable(ds_client, path): finally: ds_client.ds.root_enabled = False assert response.status_code == 200 - assert response.json()["unstable"] == UNSTABLE_MESSAGE + assert response.json()["unstable"] == UNSTABLE_API_MESSAGE @pytest.mark.asyncio @@ -148,12 +143,12 @@ async def test_query_store_and_definition_report_unstable(ds_envelope): actor={"id": "root"}, ) assert store.status_code == 201 - assert store.json()["unstable"] == UNSTABLE_MESSAGE + assert store.json()["unstable"] == UNSTABLE_API_MESSAGE definition = await ds_envelope.client.get( "/data/unstable_check/-/definition", actor={"id": "root"} ) assert definition.status_code == 200 - assert definition.json()["unstable"] == UNSTABLE_MESSAGE + assert definition.json()["unstable"] == UNSTABLE_API_MESSAGE @pytest.mark.asyncio @@ -164,7 +159,7 @@ async def test_permissions_post_reports_unstable(ds_envelope): actor={"id": "root"}, ) assert response.status_code == 200 - assert response.json()["unstable"] == UNSTABLE_MESSAGE + assert response.json()["unstable"] == UNSTABLE_API_MESSAGE @pytest.mark.asyncio From be25d6e3e41117a2217a986f11ca9467eac6f558 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 05:18:54 +0000 Subject: [PATCH 051/131] Remove test_query_list_json_signals_pagination_via_next_only Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- tests/test_queries.py | 24 ------------------------ 1 file changed, 24 deletions(-) diff --git a/tests/test_queries.py b/tests/test_queries.py index a7e492eb..ffa948a9 100644 --- a/tests/test_queries.py +++ b/tests/test_queries.py @@ -3866,27 +3866,3 @@ async def test_stored_query_json_uses_parameters_not_params(): query = [q for q in listing["queries"] if q["name"] == "with_params"][0] assert query["parameters"] == ["name", "age"] assert "params" not in query - - -@pytest.mark.asyncio -async def test_query_list_json_signals_pagination_via_next_only(): - ds = Datasette(memory=True) - ds.add_memory_database("query_list_next_only", name="data") - await ds.invoke_startup() - for i in range(3): - await ds.add_query( - "data", - name="q{}".format(i), - sql="select {}".format(i), - ) - first = (await ds.client.get("/data/-/queries.json?_size=2")).json() - assert "has_more" not in first - assert first["next"] is not None - assert first["next_url"] is not None - # The internal test client cannot follow absolute URLs - next_path = first["next_url"].replace("http://localhost", "") - assert next_path.startswith("/data/-/queries.json?") - last = (await ds.client.get(next_path)).json() - assert "has_more" not in last - assert last["next"] is None - assert last["next_url"] is None From b7bbde04bed8efad028d646d636a4364c31cf618 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 05:18:54 +0000 Subject: [PATCH 052/131] Link the consistency review release note to PR #2824 Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- docs/changelog.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index 60918ddf..d48ab152 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -17,7 +17,7 @@ Unreleased - The table and row JSON APIs now support ``?_extra=column_details`` for returning SQLite schema details for columns, including declared type, SQLite affinity, primary key, ``NOT NULL``, default and hidden-column metadata. - POST bodies that Datasette reads fully into memory - such as JSON submitted to the write API - are now capped by the new :ref:`setting_max_post_body_bytes` setting, defaulting to 2MB. Oversized requests are rejected with an HTTP 413 error as soon as the limit is exceeded, protecting smaller servers from memory exhaustion. File uploads are unaffected - ``request.form()`` streams those to disk and has its own separate limits. -This release also includes the results of a detailed consistency review of Datasette's JSON API in preparation for the 1.0 stable release. Several of these changes are backwards-incompatible with previous 1.0 alphas. The new :ref:`API stability documentation ` describes exactly which parts of the JSON API are covered by the 1.0 stability promise. +This release also includes the results of a `detailed consistency review `__ of Datasette's JSON API in preparation for the 1.0 stable release. Several of these changes are backwards-incompatible with previous 1.0 alphas. The new :ref:`API stability documentation ` describes exactly which parts of the JSON API are covered by the 1.0 stability promise. JSON API: breaking changes ~~~~~~~~~~~~~~~~~~~~~~~~~~ From ebd013c6effa52ef74334a4f86e391ed0b622464 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 6 Jul 2026 22:38:40 -0700 Subject: [PATCH 053/131] Bump GitHub Actions versions --- .github/workflows/publish.yml | 8 ++++---- .github/workflows/test.yml | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 87300593..21ed4c12 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -14,7 +14,7 @@ jobs: matrix: python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"] steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v6 with: @@ -35,7 +35,7 @@ jobs: permissions: id-token: write steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - name: Set up Python uses: actions/setup-python@v6 with: @@ -56,7 +56,7 @@ jobs: needs: [deploy] if: "!github.event.release.prerelease" steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - name: Set up Python uses: actions/setup-python@v6 with: @@ -92,7 +92,7 @@ jobs: needs: [deploy] if: "!github.event.release.prerelease" steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - name: Build and push to Docker Hub env: DOCKER_USER: ${{ secrets.DOCKER_USER }} diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index acc2d6b6..9fceca24 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -13,7 +13,7 @@ jobs: matrix: python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"] steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v6 with: @@ -54,7 +54,7 @@ jobs: test-sqlite-utils-4: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - name: Set up Python uses: actions/setup-python@v6 with: From d2695a0c2f2303a71fbd8959f3a3a3451fd50fca Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 6 Jul 2026 22:43:04 -0700 Subject: [PATCH 054/131] Test Datasette against sqlite-utils>=4.0rc4 Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900497417 --- .github/workflows/test.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 9fceca24..b8337fc0 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -67,7 +67,7 @@ jobs: - name: Install dependencies run: | pip install . --group dev - pip install --pre 'sqlite-utils>=4.0rc3' + pip install --pre 'sqlite-utils>=4.0rc4' pip freeze - name: Run tests run: | From 6f27aa112aba805f8bc82cfeeb29ec31f3e84fb0 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 7 Jul 2026 12:03:28 -0700 Subject: [PATCH 055/131] Test against sqlite-utils>=4.0 https://github.com/simonw/sqlite-utils/issues/769 --- .github/workflows/test.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index b8337fc0..d2597f01 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -58,7 +58,7 @@ jobs: - name: Set up Python uses: actions/setup-python@v6 with: - python-version: "3.13" + python-version: "3.14" cache: pip cache-dependency-path: pyproject.toml - name: Build extension for --load-extension test @@ -67,7 +67,7 @@ jobs: - name: Install dependencies run: | pip install . --group dev - pip install --pre 'sqlite-utils>=4.0rc4' + pip install --pre 'sqlite-utils>=4.0' pip freeze - name: Run tests run: | From 96e8b8552314e177165496a6fac0ebb17117507c Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 7 Jul 2026 13:51:29 -0700 Subject: [PATCH 056/131] Upgrade to sqlite-utils 4.0 --- .github/workflows/test.yml | 22 ------------ pyproject.toml | 2 +- tests/test_api_write.py | 70 ++++++++++++++++---------------------- 3 files changed, 31 insertions(+), 63 deletions(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index d2597f01..751eedfd 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -51,25 +51,3 @@ jobs: run: | pip install datasette-init datasette-json-html tests/test-datasette-load-plugins.sh - test-sqlite-utils-4: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v7 - - name: Set up Python - uses: actions/setup-python@v6 - with: - python-version: "3.14" - cache: pip - cache-dependency-path: pyproject.toml - - name: Build extension for --load-extension test - run: |- - (cd tests && gcc ext.c -fPIC -shared -o ext.so) - - name: Install dependencies - run: | - pip install . --group dev - pip install --pre 'sqlite-utils>=4.0' - pip freeze - - name: Run tests - run: | - pytest -n auto -m "not serial" - pytest -m "serial" diff --git a/pyproject.toml b/pyproject.toml index 38776b2c..70496cdb 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -35,7 +35,7 @@ dependencies = [ "PyYAML>=5.3", "mergedeep>=1.1.1", "itsdangerous>=1.1", - "sqlite-utils>=3.30", + "sqlite-utils>=4.0", "asyncinject>=0.7", "setuptools", "pip", diff --git a/tests/test_api_write.py b/tests/test_api_write.py index 840bd05b..a803fbbc 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -3,28 +3,18 @@ from datasette.events import RenameTableEvent from datasette.utils import error_body, escape_sqlite, sqlite3 from .utils import last_event import pytest -import re import time -def schema_variants(schema): - # sqlite-utils < 4 quotes identifiers [like_this] and uses FLOAT; - # sqlite-utils >= 4 quotes them "like_this" and uses REAL. Given a - # schema fragment in the old format, return both variants so tests - # can pass against either version. - converted = re.sub(r"\[([^\]]+)\]", r'"\1"', schema).replace("FLOAT", "REAL") - return (schema, converted) - - def assert_schema_contains(fragment, schema): - assert any( - variant in schema for variant in schema_variants(fragment) - ), "Expected schema to contain {!r}, got {!r}".format(fragment, schema) + assert fragment in schema, "Expected schema to contain {!r}, got {!r}".format( + fragment, schema + ) def assert_schema_not_contains(fragment, schema): - assert not any( - variant in schema for variant in schema_variants(fragment) + assert ( + fragment not in schema ), "Expected schema not to contain {!r}, got {!r}".format(fragment, schema) @@ -94,8 +84,8 @@ async def test_base64_write_api_create_table_infers_blob_and_raw_escapes(ds_writ headers=_headers(token), ) assert response.status_code == 201 - assert_schema_contains("[data] BLOB", response.json()["schema"]) - assert_schema_contains("[literal] TEXT", response.json()["schema"]) + assert_schema_contains('"data" BLOB', response.json()["schema"]) + assert_schema_contains('"literal" TEXT', response.json()["schema"]) rows = (await ds_write.get_database("data").execute(""" select @@ -1217,7 +1207,7 @@ async def test_alter_table_foreign_key_operations(ds_write): data = response.json() assert data["operations_applied"] == 2 assert_schema_contains( - "[owner_id] INTEGER REFERENCES [owners]([id])", data["schema"] + '"owner_id" INTEGER REFERENCES "owners"("id")', data["schema"] ) response = await ds_write.client.post( @@ -1229,7 +1219,7 @@ async def test_alter_table_foreign_key_operations(ds_write): ) assert response.status_code == 200, response.text data = response.json() - assert_schema_not_contains("[owner_id] INTEGER REFERENCES", data["schema"]) + assert_schema_not_contains('"owner_id" INTEGER REFERENCES', data["schema"]) response = await ds_write.client.post( "/data/docs/-/alter", @@ -1254,7 +1244,7 @@ async def test_alter_table_foreign_key_operations(ds_write): assert response.status_code == 200, response.text data = response.json() assert_schema_contains( - "[owner_id] INTEGER REFERENCES [categories]([id])", data["schema"] + '"owner_id" INTEGER REFERENCES "categories"("id")', data["schema"] ) response = await ds_write.client.post( @@ -1264,7 +1254,7 @@ async def test_alter_table_foreign_key_operations(ds_write): ) assert response.status_code == 200, response.text data = response.json() - assert_schema_not_contains("[owner_id] INTEGER REFERENCES", data["schema"]) + assert_schema_not_contains('"owner_id" INTEGER REFERENCES', data["schema"]) @pytest.mark.asyncio @@ -1791,12 +1781,12 @@ async def test_drop_table(ds_write, scenario): "table_url": "http://localhost/data/one", "table_api_url": "http://localhost/data/one.json", "schema": ( - "CREATE TABLE [one] (\n" - " [id] INTEGER PRIMARY KEY,\n" - " [title] TEXT,\n" - " [score] INTEGER,\n" - " [weight] FLOAT,\n" - " [thumbnail] BLOB\n" + 'CREATE TABLE "one" (\n' + ' "id" INTEGER PRIMARY KEY,\n' + ' "title" TEXT,\n' + ' "score" INTEGER,\n' + ' "weight" REAL,\n' + ' "thumbnail" BLOB\n' ")" ), }, @@ -1828,10 +1818,10 @@ async def test_drop_table(ds_write, scenario): "table_url": "http://localhost/data/two", "table_api_url": "http://localhost/data/two.json", "schema": ( - "CREATE TABLE [two] (\n" - " [id] INTEGER PRIMARY KEY,\n" - " [title] TEXT,\n" - " [score] FLOAT\n" + 'CREATE TABLE "two" (\n' + ' "id" INTEGER PRIMARY KEY,\n' + ' "title" TEXT,\n' + ' "score" REAL\n' ")" ), "row_count": 2, @@ -1857,10 +1847,10 @@ async def test_drop_table(ds_write, scenario): "table_url": "http://localhost/data/three", "table_api_url": "http://localhost/data/three.json", "schema": ( - "CREATE TABLE [three] (\n" - " [id] INTEGER PRIMARY KEY,\n" - " [title] TEXT,\n" - " [score] FLOAT\n" + 'CREATE TABLE "three" (\n' + ' "id" INTEGER PRIMARY KEY,\n' + ' "title" TEXT,\n' + ' "score" REAL\n' ")" ), "row_count": 1, @@ -1882,7 +1872,7 @@ async def test_drop_table(ds_write, scenario): "table": "four", "table_url": "http://localhost/data/four", "table_api_url": "http://localhost/data/four.json", - "schema": ("CREATE TABLE [four] (\n" " [name] TEXT\n" ")"), + "schema": ('CREATE TABLE "four" (\n' ' "name" TEXT\n' ")"), "row_count": 1, }, ["create-table", "insert-rows"], @@ -1902,8 +1892,8 @@ async def test_drop_table(ds_write, scenario): "table_url": "http://localhost/data/five", "table_api_url": "http://localhost/data/five.json", "schema": ( - "CREATE TABLE [five] (\n [type] TEXT,\n [key] INTEGER,\n" - " [title] TEXT,\n PRIMARY KEY ([type], [key])\n)" + 'CREATE TABLE "five" (\n "type" TEXT,\n "key" INTEGER,\n' + ' "title" TEXT,\n PRIMARY KEY ("type", "key")\n)' ), "row_count": 1, }, @@ -2193,7 +2183,7 @@ async def test_create_table( # Error expectations list their messages; derive the canonical envelope expected_response = error_body(expected_response["errors"], expected_status) if isinstance(expected_response, dict) and "schema" in expected_response: - assert data.get("schema") in schema_variants(expected_response["schema"]) + assert data.get("schema") == expected_response["schema"] expected_response = dict(expected_response, schema=data.get("schema")) assert data == expected_response # Should have tracked the expected events @@ -2238,7 +2228,7 @@ async def test_create_table_with_foreign_key(ds_write): assert response.status_code == 201 data = response.json() assert_schema_contains( - "[owner_id] INTEGER REFERENCES [owners]([id])", data["schema"] + '"owner_id" INTEGER REFERENCES "owners"("id")', data["schema"] ) From db7ba1d30c0fb9e9e6eb2e39403a062fd93a973d Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 7 Jul 2026 13:58:48 -0700 Subject: [PATCH 057/131] Switch to sqlite-utils migrations for internal.db, closes #2827 --- datasette/utils/internal_db.py | 165 +++++++++++++++++++++------------ tests/test_internal_db.py | 49 ++++++++++ 2 files changed, 153 insertions(+), 61 deletions(-) diff --git a/datasette/utils/internal_db.py b/datasette/utils/internal_db.py index bf172667..10ca32a5 100644 --- a/datasette/utils/internal_db.py +++ b/datasette/utils/internal_db.py @@ -1,9 +1,30 @@ import textwrap + +from sqlite_utils import Database as SQLiteUtilsDatabase +from sqlite_utils import Migrations + from datasette.utils import table_column_details +INTERNAL_DB_SCHEMA_TABLES = { + "catalog_databases", + "catalog_tables", + "catalog_views", + "catalog_columns", + "catalog_indexes", + "catalog_foreign_keys", + "metadata_instance", + "metadata_databases", + "metadata_resources", + "metadata_columns", + "column_types", + "queries", +} -async def init_internal_db(db): - create_tables_sql = textwrap.dedent(""" +INTERNAL_DB_SCHEMA_INDEXES = { + "queries_owner_idx", +} + +INTERNAL_DB_SCHEMA_SQL = textwrap.dedent(""" CREATE TABLE IF NOT EXISTS catalog_databases ( database_name TEXT PRIMARY KEY, path TEXT, @@ -67,74 +88,96 @@ async def init_internal_db(db): FOREIGN KEY (database_name) REFERENCES catalog_databases(database_name), FOREIGN KEY (database_name, table_name) REFERENCES catalog_tables(database_name, table_name) ); + + CREATE TABLE IF NOT EXISTS metadata_instance ( + key text, + value text, + unique(key) + ); + + CREATE TABLE IF NOT EXISTS metadata_databases ( + database_name text, + key text, + value text, + unique(database_name, key) + ); + + CREATE TABLE IF NOT EXISTS metadata_resources ( + database_name text, + resource_name text, + key text, + value text, + unique(database_name, resource_name, key) + ); + + CREATE TABLE IF NOT EXISTS metadata_columns ( + database_name text, + resource_name text, + column_name text, + key text, + value text, + unique(database_name, resource_name, column_name, key) + ); + + CREATE TABLE IF NOT EXISTS column_types ( + database_name TEXT NOT NULL, + resource_name TEXT NOT NULL, + column_name TEXT NOT NULL, + column_type TEXT NOT NULL, + config TEXT, + PRIMARY KEY (database_name, resource_name, column_name) + ); + + CREATE TABLE IF NOT EXISTS queries ( + database_name TEXT NOT NULL, + name TEXT NOT NULL, + sql TEXT NOT NULL, + title TEXT, + description TEXT, + description_html TEXT, + options TEXT NOT NULL DEFAULT '{}', + parameters TEXT NOT NULL DEFAULT '[]', + is_write INTEGER NOT NULL DEFAULT 0 CHECK (is_write IN (0, 1)), + is_private INTEGER NOT NULL DEFAULT 0 CHECK (is_private IN (0, 1)), + is_trusted INTEGER NOT NULL DEFAULT 0 CHECK (is_trusted IN (0, 1)), + source TEXT NOT NULL DEFAULT 'user', + owner_id TEXT, + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (database_name, name) + ); + + CREATE INDEX IF NOT EXISTS queries_owner_idx + ON queries(owner_id); """).strip() - await db.execute_write_script(create_tables_sql) - await initialize_metadata_tables(db) -async def initialize_metadata_tables(db): - await db.execute_write_script(textwrap.dedent(""" - CREATE TABLE IF NOT EXISTS metadata_instance ( - key text, - value text, - unique(key) - ); +internal_migrations = Migrations("datasette_internal") - CREATE TABLE IF NOT EXISTS metadata_databases ( - database_name text, - key text, - value text, - unique(database_name, key) - ); - CREATE TABLE IF NOT EXISTS metadata_resources ( - database_name text, - resource_name text, - key text, - value text, - unique(database_name, resource_name, key) - ); +def _internal_schema_exists(db): + table_names = set(db.table_names()) + if not INTERNAL_DB_SCHEMA_TABLES.issubset(table_names): + return False + index_names = { + row[0] + for row in db.execute("select name from sqlite_master where type = 'index'") + } + return INTERNAL_DB_SCHEMA_INDEXES.issubset(index_names) - CREATE TABLE IF NOT EXISTS metadata_columns ( - database_name text, - resource_name text, - column_name text, - key text, - value text, - unique(database_name, resource_name, column_name, key) - ); - CREATE TABLE IF NOT EXISTS column_types ( - database_name TEXT NOT NULL, - resource_name TEXT NOT NULL, - column_name TEXT NOT NULL, - column_type TEXT NOT NULL, - config TEXT, - PRIMARY KEY (database_name, resource_name, column_name) - ); +@internal_migrations(name="0001_initial") +def initial_internal_schema(db): + if _internal_schema_exists(db): + return + db.executescript(INTERNAL_DB_SCHEMA_SQL) - CREATE TABLE IF NOT EXISTS queries ( - database_name TEXT NOT NULL, - name TEXT NOT NULL, - sql TEXT NOT NULL, - title TEXT, - description TEXT, - description_html TEXT, - options TEXT NOT NULL DEFAULT '{}', - parameters TEXT NOT NULL DEFAULT '[]', - is_write INTEGER NOT NULL DEFAULT 0 CHECK (is_write IN (0, 1)), - is_private INTEGER NOT NULL DEFAULT 0 CHECK (is_private IN (0, 1)), - is_trusted INTEGER NOT NULL DEFAULT 0 CHECK (is_trusted IN (0, 1)), - source TEXT NOT NULL DEFAULT 'user', - owner_id TEXT, - created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, - updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, - PRIMARY KEY (database_name, name) - ); - CREATE INDEX IF NOT EXISTS queries_owner_idx - ON queries(owner_id); - """)) +async def init_internal_db(db): + def apply_migrations(conn): + internal_migrations.apply(SQLiteUtilsDatabase(conn, execute_plugins=False)) + + await db.execute_write_fn(apply_migrations, transaction=False) async def populate_schema_tables(internal_db, db): diff --git a/tests/test_internal_db.py b/tests/test_internal_db.py index 340c4813..e1ab51bb 100644 --- a/tests/test_internal_db.py +++ b/tests/test_internal_db.py @@ -1,6 +1,8 @@ import pytest +import sqlite3 from datasette.utils import escape_sqlite +from datasette.utils.internal_db import INTERNAL_DB_SCHEMA_SQL # ensure refresh_schemas() gets called before interacting with internal_db @@ -17,6 +19,53 @@ async def test_internal_databases(ds_client): assert databases.rows[0]["database_name"] == "fixtures" +@pytest.mark.asyncio +async def test_internal_migrations_recorded(ds_client): + internal_db = await ensure_internal(ds_client) + migrations = await internal_db.execute(""" + select migration_set, name + from _sqlite_migrations + order by id + """) + assert [tuple(row) for row in migrations.rows] == [ + ("datasette_internal", "0001_initial") + ] + + +@pytest.mark.asyncio +async def test_internal_migrations_adopt_existing_internal_db(tmp_path): + from datasette.app import Datasette + + internal_db_path = str(tmp_path / "internal.db") + conn = sqlite3.connect(internal_db_path) + conn.executescript(INTERNAL_DB_SCHEMA_SQL) + conn.execute( + "insert into metadata_instance (key, value) values (?, ?)", + ("legacy", "preserved"), + ) + conn.commit() + conn.close() + + ds = Datasette(internal=internal_db_path) + await ds.invoke_startup() + internal_db = ds.get_internal_database() + + metadata = await internal_db.execute( + "select key, value from metadata_instance where key = 'legacy'" + ) + assert [tuple(row) for row in metadata.rows] == [("legacy", "preserved")] + migrations = await internal_db.execute(""" + select migration_set, name + from _sqlite_migrations + order by id + """) + assert [tuple(row) for row in migrations.rows] == [ + ("datasette_internal", "0001_initial") + ] + + ds.close() + + @pytest.mark.asyncio async def test_internal_tables(ds_client): internal_db = await ensure_internal(ds_client) From a926ab392e6ecf0264bb8d90c0b0411800c498c7 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 7 Jul 2026 14:02:02 -0700 Subject: [PATCH 058/131] Updated internals.rst schema using cog, refs #2827 --- docs/internals.rst | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/internals.rst b/docs/internals.rst index 7258e6f3..2048c7e4 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -2363,6 +2363,14 @@ The internal database schema is as follows: .. code-block:: sql + CREATE TABLE "_sqlite_migrations" ( + "id" INTEGER PRIMARY KEY, + "migration_set" TEXT, + "name" TEXT, + "applied_at" TEXT + ); + CREATE UNIQUE INDEX "idx__sqlite_migrations_migration_set_name" + ON "_sqlite_migrations" ("migration_set", "name"); CREATE TABLE catalog_databases ( database_name TEXT PRIMARY KEY, path TEXT, From 7f37205e7696e8876bfd8fb0d14b543b66a748bf Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 7 Jul 2026 14:04:46 -0700 Subject: [PATCH 059/131] Remove Datasette Desktop from installation guide Until I have time to fix it up and bring it back. --- docs/installation.rst | 7 ------- 1 file changed, 7 deletions(-) diff --git a/docs/installation.rst b/docs/installation.rst index 33d3d6a1..ceec7f23 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -17,13 +17,6 @@ If you want to start making contributions to the Datasette project by installing Basic installation ================== -.. _installation_datasette_desktop: - -Datasette Desktop for Mac -------------------------- - -`Datasette Desktop `__ is a packaged Mac application which bundles Datasette together with Python and allows you to install and run Datasette directly on your laptop. This is the best option for local installation if you are not comfortable using the command line. - .. _installation_homebrew: Using Homebrew From 617acedd387ff80769c6b40c7b673d2c489fc346 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 7 Jul 2026 14:18:10 -0700 Subject: [PATCH 060/131] Remove readthedocs/actions/preview Closes #2828 --- .github/workflows/documentation-links.yml | 16 ---------------- docs/changelog.rst | 10 +++++++--- 2 files changed, 7 insertions(+), 19 deletions(-) delete mode 100644 .github/workflows/documentation-links.yml diff --git a/.github/workflows/documentation-links.yml b/.github/workflows/documentation-links.yml deleted file mode 100644 index b8fb8aaa..00000000 --- a/.github/workflows/documentation-links.yml +++ /dev/null @@ -1,16 +0,0 @@ -name: Read the Docs Pull Request Preview -on: - pull_request: - types: - - opened - -permissions: - pull-requests: write - -jobs: - documentation-links: - runs-on: ubuntu-latest - steps: - - uses: readthedocs/actions/preview@v1 - with: - project-slug: "datasette" diff --git a/docs/changelog.rst b/docs/changelog.rst index d48ab152..de2ca85b 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -12,10 +12,14 @@ Unreleased - Table pages now offer an "Insert multiple rows" mode in the row insertion dialog. This accepts pasted TSV, CSV or JSON, previews the parsed rows before inserting them, validates unknown columns as data is pasted and displays omitted auto integer primary keys as ``auto`` in the preview. (:pr:`2813`) - The bulk insert UI can skip rows with existing primary keys, or update existing rows and insert new rows using the existing ``//
/-/upsert`` API when the actor has both :ref:`insert-row ` and :ref:`update-row ` permissions. (:pr:`2813`) - The "Create table" dialog now includes a "Create table from data" mode. Paste TSV, CSV or JSON rows to preview inferred columns and types, choose the table name and primary key, then create the table and insert those rows in one step. (:pr:`2813`) -- Datasette's JSON APIs now consistently encode every ``BLOB`` value using the documented :ref:`binary value JSON format `, even when the bytes could be decoded as UTF-8 text. -- The insert and edit row dialogs now provide a dedicated control for ``BLOB`` values. Existing binary values are shown by byte size, image values under 10MB are previewed as thumbnails, and replacements can be attached, dropped or pasted into the control. +- Datasette's JSON APIs now consistently encode every ``BLOB`` value using the documented :ref:`binary value JSON format `, even when the bytes could be decoded as UTF-8 text. (:issue:`2806`, :pr:`2822`) +- The insert and edit row dialogs now provide a dedicated control for ``BLOB`` values. Existing binary values are shown by byte size, image values under 10MB are previewed as thumbnails, and replacements can be attached, dropped or pasted into the control. (:issue:`2806`, :pr:`2822`) - The table and row JSON APIs now support ``?_extra=column_details`` for returning SQLite schema details for columns, including declared type, SQLite affinity, primary key, ``NOT NULL``, default and hidden-column metadata. -- POST bodies that Datasette reads fully into memory - such as JSON submitted to the write API - are now capped by the new :ref:`setting_max_post_body_bytes` setting, defaulting to 2MB. Oversized requests are rejected with an HTTP 413 error as soon as the limit is exceeded, protecting smaller servers from memory exhaustion. File uploads are unaffected - ``request.form()`` streams those to disk and has its own separate limits. +- POST bodies that Datasette reads fully into memory - such as JSON submitted to the write API - are now capped by the new :ref:`setting_max_post_body_bytes` setting, defaulting to 2MB. Oversized requests are rejected with an HTTP 413 error as soon as the limit is exceeded, protecting smaller servers from memory exhaustion. File uploads are unaffected - ``request.form()`` streams those to disk and has its own separate limits. (:issue:`2823`) +- The :ref:`execute-write-sql ` interface now supports ``CREATE VIEW`` and ``DROP VIEW`` statements, gated by the new :ref:`create-view ` and :ref:`drop-view ` permissions. (:issue:`2819`, :pr:`2818`) +- Saved-query SQL analysis now handles recursive CTEs, fixing a bug where storing a valid read-only recursive query could be disabled by SQLite's internal ``SQLITE_RECURSIVE`` authorizer callback. (:issue:`2809`, :pr:`2812`) +- Datasette's internal database schema is now managed using `sqlite-utils migrations `__, using the new dependency on ``sqlite-utils>=4.0``. (:issue:`2827`) +- ``datasette.utils.CustomJSONEncoder`` is now documented as a public API for plugins that need to serialize Datasette values to JSON. Thanks, `Chris Amico `__. (:issue:`1983`, :pr:`1996`) This release also includes the results of a `detailed consistency review `__ of Datasette's JSON API in preparation for the 1.0 stable release. Several of these changes are backwards-incompatible with previous 1.0 alphas. The new :ref:`API stability documentation ` describes exactly which parts of the JSON API are covered by the 1.0 stability promise. From 211e70d4e1f1f7d99e06b378993f26e8cbd8937a Mon Sep 17 00:00:00 2001 From: Zain Dana Harper Date: Tue, 7 Jul 2026 14:19:08 -0700 Subject: [PATCH 061/131] Return 400 not 500 for wrong-arity composite-PK row URLs (#2815) Fixes #2811 Co-authored-by: Zain Dana Harper Co-authored-by: Claude Opus 4.8 --- datasette/app.py | 5 +++++ tests/test_api.py | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+) diff --git a/datasette/app.py b/datasette/app.py index 9c7e768b..4ba5d20f 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -134,6 +134,7 @@ from .utils import ( from .tokens import TokenInvalid from .utils.asgi import ( AsgiLifespan, + BadRequest, Forbidden, NotFound, DatabaseNotFound, @@ -2820,6 +2821,10 @@ class Datasette: db, table_name, _ = await self.resolve_table(request) pk_values = urlsafe_components(request.url_vars["pks"]) sql, params, pks = await row_sql_params_pks(db, table_name, pk_values) + if len(pk_values) != len(pks): + raise BadRequest( + "URL row identifier does not match the primary key for this table" + ) results = await db.execute(sql, params, truncate=True) row = results.first() if row is None: diff --git a/tests/test_api.py b/tests/test_api.py index 235d394b..5ed14283 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -369,6 +369,40 @@ async def test_row(ds_client): assert response.json()["rows"] == [{"id": 1, "content": "hello"}] +@pytest.mark.asyncio +@pytest.mark.parametrize("suffix", ("", ".json")) +@pytest.mark.parametrize( + "row_path", + ( + "a", # too few components for a two-column primary key + "a,b,c", # too many components for a two-column primary key + ), +) +async def test_row_pk_arity_mismatch_returns_400(ds_client, row_path, suffix): + # A row URL with the wrong number of comma-separated primary key + # components used to raise an uncaught sqlite3.ProgrammingError (HTTP 500) + # because the SQL had one bind placeholder per PK column but params were + # only bound for the supplied components. It should be a 400 instead, + # mirroring the existing guard in datasette/views/table.py. + response = await ds_client.get( + "/fixtures/compound_primary_key/{}{}".format(row_path, suffix) + ) + assert response.status_code == 400 + if suffix == ".json": + assert response.json()["ok"] is False + assert response.json()["status"] == 400 + + +@pytest.mark.asyncio +async def test_row_compound_pk_correct_arity(ds_client): + # The valid two-component URL still resolves the row. + response = await ds_client.get( + "/fixtures/compound_primary_key/a,b.json?_shape=objects" + ) + assert response.status_code == 200 + assert response.json()["rows"] == [{"pk1": "a", "pk2": "b", "content": "c"}] + + @pytest.mark.asyncio async def test_row_strange_table_name(ds_client): response = await ds_client.get( From bf3e277c9819b45ae34810ff5aafd794fd7ff4b9 Mon Sep 17 00:00:00 2001 From: JSap0914 <116227558+JSap0914@users.noreply.github.com> Date: Wed, 8 Jul 2026 06:23:31 +0900 Subject: [PATCH 062/131] Fix named_parameters when string literals contain comment markers (#2783) named_parameters stripped SQL comments before string literals in separate passes. A string literal such as '-- TODO' would be treated as the start of a line comment, swallowing the rest of the line and hiding any named parameters that followed it. For example: select * from t where note = '-- TODO' and id = :id returned [] instead of ['id'], so the query parameter input form would be missing the :id field. Match comments and string literals in a single left-to-right pass so that whichever construct starts first wins, matching how SQL is actually tokenized. Co-authored-by: JSap0914 --- datasette/utils/__init__.py | 23 +++++++++++++++-------- tests/test_utils.py | 10 ++++++++++ 2 files changed, 25 insertions(+), 8 deletions(-) diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index 64c63dde..7b9a90b9 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -1288,10 +1288,18 @@ class StartupError(Exception): pass -_single_line_comment_re = re.compile(r"--.*") -_multi_line_comment_re = re.compile(r"/\*.*?\*/", re.DOTALL) -_single_quote_re = re.compile(r"'(?:''|[^'])*'") -_double_quote_re = re.compile(r'"(?:\"\"|[^"])*"') +# Comments and string literals, matched in a single pass so that whichever +# construct starts first "wins" - this ensures a comment marker inside a string +# literal (or a quote inside a comment) does not confuse the parameter scan. +_comments_and_strings_re = re.compile( + r""" + --[^\n]* # single line comment + | /\*.*?\*/ # multi line comment + | '(?:''|[^'])*' # single quoted string ('' escapes a quote) + | "(?:""|[^"])*" # double quoted identifier ("" escapes a quote) + """, + re.DOTALL | re.VERBOSE, +) _named_param_re = re.compile(r":(\w+)") @@ -1302,10 +1310,9 @@ def named_parameters(sql: str) -> List[str]: e.g. for ``select * from foo where id=:id`` this would return ``["id"]`` """ - sql = _single_line_comment_re.sub("", sql) - sql = _multi_line_comment_re.sub("", sql) - sql = _single_quote_re.sub("", sql) - sql = _double_quote_re.sub("", sql) + # Strip comments and string literals first so that any ":name" sequences + # inside them are not mistaken for named parameters + sql = _comments_and_strings_re.sub("", sql) # Extract parameters from what is left return _named_param_re.findall(sql) diff --git a/tests/test_utils.py b/tests/test_utils.py index 46dcd89d..0b2fcd2b 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -756,6 +756,16 @@ def test_parse_metadata(content, expected): ("select 1 + :one + :two", ["one", "two"]), ("select 'bob' || '0:00' || :cat", ["cat"]), ("select this is invalid :one, :two, :three", ["one", "two", "three"]), + # A string literal containing a comment marker should not hide + # parameters that come after it + ("select * from t where note = '-- TODO' and id = :id", ["id"]), + ("select '--' || :y", ["y"]), + ("select * from t where note = '/* x */' and id = :id", ["id"]), + # Parameters that live inside a comment should be ignored + ("select :x -- and :ignored", ["x"]), + ("select :x /* and :ignored */ from t", ["x"]), + # Parameters inside a string literal should be ignored + ("select ':ignored' || :real", ["real"]), ), ) @pytest.mark.parametrize("use_async_version", (False, True)) From 54597f22fafb1744718b98f3018da595374f8e7e Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 7 Jul 2026 14:26:34 -0700 Subject: [PATCH 063/131] A few more SQLite string fixes, refs #2783 --- datasette/utils/__init__.py | 4 +++- tests/test_utils.py | 5 +++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index 7b9a90b9..42574d3b 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -1294,9 +1294,11 @@ class StartupError(Exception): _comments_and_strings_re = re.compile( r""" --[^\n]* # single line comment - | /\*.*?\*/ # multi line comment + | /\*.*?(?:\*/|\Z) # multi line comment, possibly to end-of-input | '(?:''|[^'])*' # single quoted string ('' escapes a quote) | "(?:""|[^"])*" # double quoted identifier ("" escapes a quote) + | \[(?:[^\]])*\] # square-bracket quoted identifier + | `(?:``|[^`])*` # backtick quoted identifier """, re.DOTALL | re.VERBOSE, ) diff --git a/tests/test_utils.py b/tests/test_utils.py index 0b2fcd2b..a535ca93 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -764,6 +764,11 @@ def test_parse_metadata(content, expected): # Parameters that live inside a comment should be ignored ("select :x -- and :ignored", ["x"]), ("select :x /* and :ignored */ from t", ["x"]), + ("select :x /* and :ignored", ["x"]), + # Parameters inside quoted identifiers should be ignored + ("select [a:b] from t where id = :id", ["id"]), + ("select `a:b` from t where id = :id", ["id"]), + ("select `a``:b` from t where id = :id", ["id"]), # Parameters inside a string literal should be ignored ("select ':ignored' || :real", ["real"]), ), From a31673c90b56ba72b50d037ff60829070d32e85c Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 7 Jul 2026 14:26:48 -0700 Subject: [PATCH 064/131] Changelog for #2811, #2815, #2783 --- docs/changelog.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/changelog.rst b/docs/changelog.rst index de2ca85b..c217192b 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -16,8 +16,10 @@ Unreleased - The insert and edit row dialogs now provide a dedicated control for ``BLOB`` values. Existing binary values are shown by byte size, image values under 10MB are previewed as thumbnails, and replacements can be attached, dropped or pasted into the control. (:issue:`2806`, :pr:`2822`) - The table and row JSON APIs now support ``?_extra=column_details`` for returning SQLite schema details for columns, including declared type, SQLite affinity, primary key, ``NOT NULL``, default and hidden-column metadata. - POST bodies that Datasette reads fully into memory - such as JSON submitted to the write API - are now capped by the new :ref:`setting_max_post_body_bytes` setting, defaulting to 2MB. Oversized requests are rejected with an HTTP 413 error as soon as the limit is exceeded, protecting smaller servers from memory exhaustion. File uploads are unaffected - ``request.form()`` streams those to disk and has its own separate limits. (:issue:`2823`) +- Row pages for tables with compound primary keys now return a ``400`` error instead of a ``500`` error when the URL row identifier does not contain the correct number of primary key values. Thanks, `Zain Dana Harper `__. (:issue:`2811`, :pr:`2815`) - The :ref:`execute-write-sql ` interface now supports ``CREATE VIEW`` and ``DROP VIEW`` statements, gated by the new :ref:`create-view ` and :ref:`drop-view ` permissions. (:issue:`2819`, :pr:`2818`) - Saved-query SQL analysis now handles recursive CTEs, fixing a bug where storing a valid read-only recursive query could be disabled by SQLite's internal ``SQLITE_RECURSIVE`` authorizer callback. (:issue:`2809`, :pr:`2812`) +- ``named_parameters()`` now correctly ignores SQLite comment markers that appear inside string literals, so query forms no longer drop later ``:named`` parameters from SQL such as ``select '--' || :name``. Thanks, `JSap0914 `__. (:pr:`2783`) - Datasette's internal database schema is now managed using `sqlite-utils migrations `__, using the new dependency on ``sqlite-utils>=4.0``. (:issue:`2827`) - ``datasette.utils.CustomJSONEncoder`` is now documented as a public API for plugins that need to serialize Datasette values to JSON. Thanks, `Chris Amico `__. (:issue:`1983`, :pr:`1996`) From 52ae7d1b6d2ed738904677a80edc0e61ef85fac2 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 7 Jul 2026 14:32:25 -0700 Subject: [PATCH 065/131] Release 1.0a36 Refs #1983, #1996, #2783, #2806, #2809, #2811, #2812, #2813, #2815, #2818, #2819, #2822, #2823, #2827 --- datasette/version.py | 2 +- docs/changelog.rst | 8 +++++--- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/datasette/version.py b/datasette/version.py index 49d270e4..387144e9 100644 --- a/datasette/version.py +++ b/datasette/version.py @@ -1,2 +1,2 @@ -__version__ = "1.0a35" +__version__ = "1.0a36" __version_info__ = tuple(__version__.split(".")) diff --git a/docs/changelog.rst b/docs/changelog.rst index c217192b..e3718543 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -4,10 +4,12 @@ Changelog ========= -.. _unreleased: +.. _v1_0_a36: -Unreleased ----------- +1.0a36 (2026-07-07) +------------------- + +The signature features of this alpha are new UIs for **inserting multiple rows at once** (from TSV, CSV or JSON) and for **creating a table from rows**, plus a large number of small **JSON API consistency fixes** in preparation for a 1.0 stable release. - Table pages now offer an "Insert multiple rows" mode in the row insertion dialog. This accepts pasted TSV, CSV or JSON, previews the parsed rows before inserting them, validates unknown columns as data is pasted and displays omitted auto integer primary keys as ``auto`` in the preview. (:pr:`2813`) - The bulk insert UI can skip rows with existing primary keys, or update existing rows and insert new rows using the existing ``//
/-/upsert`` API when the actor has both :ref:`insert-row ` and :ref:`update-row ` permissions. (:pr:`2813`) From db82123108dec69ffc8541c815ae57a46ae45727 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 7 Jul 2026 14:40:33 -0700 Subject: [PATCH 066/131] Bump a whole lot of GitHub Actions versions --- .github/workflows/deploy-latest.yml | 2 +- .github/workflows/playwright.yml | 6 +++--- .github/workflows/prettier.yml | 4 ++-- .github/workflows/push_docker_tag.yml | 2 +- .github/workflows/spellcheck.yml | 2 +- .github/workflows/stable-docs.yml | 2 +- .github/workflows/test-coverage.yml | 2 +- .github/workflows/test-pyodide.yml | 4 ++-- .github/workflows/test-sqlite-support.yml | 2 +- .github/workflows/tmate-mac.yml | 2 +- .github/workflows/tmate.yml | 2 +- 11 files changed, 15 insertions(+), 15 deletions(-) diff --git a/.github/workflows/deploy-latest.yml b/.github/workflows/deploy-latest.yml index b0640ae8..3fc83438 100644 --- a/.github/workflows/deploy-latest.yml +++ b/.github/workflows/deploy-latest.yml @@ -15,7 +15,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Check out datasette - uses: actions/checkout@v6 + uses: actions/checkout@v7 - name: Set up Python uses: actions/setup-python@v6 with: diff --git a/.github/workflows/playwright.yml b/.github/workflows/playwright.yml index 5275ddef..f5b8dbf6 100644 --- a/.github/workflows/playwright.yml +++ b/.github/workflows/playwright.yml @@ -16,7 +16,7 @@ jobs: matrix: browser: [chromium, firefox, webkit] steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - name: Set up Python 3.14 uses: actions/setup-python@v6 with: @@ -25,14 +25,14 @@ jobs: cache: pip cache-dependency-path: pyproject.toml - name: Cache uv - uses: actions/cache@v5 + uses: actions/cache@v6 with: path: ~/.cache/uv key: ${{ runner.os }}-py3.14-uv-${{ hashFiles('pyproject.toml') }} restore-keys: | ${{ runner.os }}-py3.14-uv- - name: Cache Playwright browsers - uses: actions/cache@v5 + uses: actions/cache@v6 with: path: ~/.cache/ms-playwright/ key: ${{ runner.os }}-playwright-${{ matrix.browser }}-${{ hashFiles('pyproject.toml') }} diff --git a/.github/workflows/prettier.yml b/.github/workflows/prettier.yml index 735e14e9..d92ab82b 100644 --- a/.github/workflows/prettier.yml +++ b/.github/workflows/prettier.yml @@ -10,8 +10,8 @@ jobs: runs-on: ubuntu-latest steps: - name: Check out repo - uses: actions/checkout@v6 - - uses: actions/cache@v5 + uses: actions/checkout@v7 + - uses: actions/cache@v6 name: Configure npm caching with: path: ~/.npm diff --git a/.github/workflows/push_docker_tag.yml b/.github/workflows/push_docker_tag.yml index e622ef4c..c5a4f0db 100644 --- a/.github/workflows/push_docker_tag.yml +++ b/.github/workflows/push_docker_tag.yml @@ -13,7 +13,7 @@ jobs: deploy_docker: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - name: Build and push to Docker Hub env: DOCKER_USER: ${{ secrets.DOCKER_USER }} diff --git a/.github/workflows/spellcheck.yml b/.github/workflows/spellcheck.yml index 9a808194..58635025 100644 --- a/.github/workflows/spellcheck.yml +++ b/.github/workflows/spellcheck.yml @@ -9,7 +9,7 @@ jobs: spellcheck: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - name: Set up Python uses: actions/setup-python@v6 with: diff --git a/.github/workflows/stable-docs.yml b/.github/workflows/stable-docs.yml index 59b5fbc0..ecde5940 100644 --- a/.github/workflows/stable-docs.yml +++ b/.github/workflows/stable-docs.yml @@ -15,7 +15,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout repository - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 0 # We need all commits to find docs/ changes - name: Set up Git user diff --git a/.github/workflows/test-coverage.yml b/.github/workflows/test-coverage.yml index c514048e..e9bd4bab 100644 --- a/.github/workflows/test-coverage.yml +++ b/.github/workflows/test-coverage.yml @@ -15,7 +15,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Check out datasette - uses: actions/checkout@v6 + uses: actions/checkout@v7 - name: Set up Python uses: actions/setup-python@v6 with: diff --git a/.github/workflows/test-pyodide.yml b/.github/workflows/test-pyodide.yml index 5162c47a..5e81ed82 100644 --- a/.github/workflows/test-pyodide.yml +++ b/.github/workflows/test-pyodide.yml @@ -12,7 +12,7 @@ jobs: test: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - name: Set up Python 3.10 uses: actions/setup-python@v6 with: @@ -20,7 +20,7 @@ jobs: cache: 'pip' cache-dependency-path: '**/pyproject.toml' - name: Cache Playwright browsers - uses: actions/cache@v5 + uses: actions/cache@v6 with: path: ~/.cache/ms-playwright/ key: ${{ runner.os }}-browsers diff --git a/.github/workflows/test-sqlite-support.yml b/.github/workflows/test-sqlite-support.yml index d86000bf..2fdb3a40 100644 --- a/.github/workflows/test-sqlite-support.yml +++ b/.github/workflows/test-sqlite-support.yml @@ -25,7 +25,7 @@ jobs: #"3.23.1" # 2018-04-10, before UPSERT ] steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v6 with: diff --git a/.github/workflows/tmate-mac.yml b/.github/workflows/tmate-mac.yml index a033cd92..f2c074a6 100644 --- a/.github/workflows/tmate-mac.yml +++ b/.github/workflows/tmate-mac.yml @@ -10,6 +10,6 @@ jobs: build: runs-on: macos-latest steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - name: Setup tmate session uses: mxschmitt/action-tmate@v3 diff --git a/.github/workflows/tmate.yml b/.github/workflows/tmate.yml index 72af1eec..5b8818c3 100644 --- a/.github/workflows/tmate.yml +++ b/.github/workflows/tmate.yml @@ -11,7 +11,7 @@ jobs: build: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - name: Setup tmate session uses: mxschmitt/action-tmate@v3 env: From ccace40e5a14ec51ae8ace8c65d27ac6bada4fac Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 13 Jul 2026 21:19:04 -0700 Subject: [PATCH 067/131] /-/plugins.json is now an array of objects again (#2843) Reverts the object envelope introduced in 1.0a36 for this endpoint - it once again returns a top-level JSON array of plugin objects. Closes #2842 Claude-Session: https://claude.ai/code/session_012TYc1NTBK4zEjabB3u2zqu Co-authored-by: Claude --- datasette/app.py | 2 +- docs/introspection.rst | 21 +++++++++------------ tests/test_api.py | 4 ++-- tests/test_config_dir.py | 2 +- tests/test_plugins.py | 2 +- tests/test_success_envelope.py | 15 +++++++-------- 6 files changed, 21 insertions(+), 25 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 4ba5d20f..c44f9095 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -2575,7 +2575,7 @@ class Datasette: JsonDataView.as_view( self, "plugins.json", - lambda request: {"plugins": self._plugins(request)}, + self._plugins, needs_request=True, ), r"/-/plugins(\.(?Pjson))?$", diff --git a/docs/introspection.rst b/docs/introspection.rst index b78e4860..14b6249f 100644 --- a/docs/introspection.rst +++ b/docs/introspection.rst @@ -80,18 +80,15 @@ Shows a list of currently installed plugins and their versions. `Plugins example .. code-block:: json - { - "ok": true, - "plugins": [ - { - "name": "datasette_cluster_map", - "static": true, - "templates": false, - "version": "0.10", - "hooks": ["extra_css_urls", "extra_js_urls", "extra_body_script"] - } - ] - } + [ + { + "name": "datasette_cluster_map", + "static": true, + "templates": false, + "version": "0.10", + "hooks": ["extra_css_urls", "extra_js_urls", "extra_body_script"] + } + ] Add ``?all=1`` to include details of the default plugins baked into Datasette. diff --git a/tests/test_api.py b/tests/test_api.py index 5ed14283..d5f519b9 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -614,13 +614,13 @@ async def test_plugins_json(ds_client): response = await ds_client.get("/-/plugins.json") # Filter out TrackEventPlugin actual_plugins = sorted( - [p for p in response.json()["plugins"] if p["name"] != "TrackEventPlugin"], + [p for p in response.json() if p["name"] != "TrackEventPlugin"], key=lambda p: p["name"], ) assert EXPECTED_PLUGINS == actual_plugins # Try with ?all=1 response = await ds_client.get("/-/plugins.json?all=1") - names = {p["name"] for p in response.json()["plugins"]} + names = {p["name"] for p in response.json()} assert names.issuperset(p["name"] for p in EXPECTED_PLUGINS) assert names.issuperset(DEFAULT_PLUGINS) diff --git a/tests/test_config_dir.py b/tests/test_config_dir.py index 42c6ae60..636b17eb 100644 --- a/tests/test_config_dir.py +++ b/tests/test_config_dir.py @@ -109,7 +109,7 @@ def test_settings(config_dir_client): def test_plugins(config_dir_client): response = config_dir_client.get("/-/plugins.json") assert 200 == response.status - plugins = response.json["plugins"] + plugins = response.json assert "hooray.py" in {p["name"] for p in plugins} assert "non_py_file.txt" not in {p["name"] for p in plugins} assert "mypy_cache" not in {p["name"] for p in plugins} diff --git a/tests/test_plugins.py b/tests/test_plugins.py index 5c4034db..59b1c0bf 100644 --- a/tests/test_plugins.py +++ b/tests/test_plugins.py @@ -1482,7 +1482,7 @@ async def test_plugin_is_installed(): datasette.pm.register(DummyPlugin(), name="DummyPlugin") response = await datasette.client.get("/-/plugins.json") assert response.status_code == 200 - installed_plugins = {p["name"] for p in response.json()["plugins"]} + installed_plugins = {p["name"] for p in response.json()} assert "DummyPlugin" in installed_plugins finally: diff --git a/tests/test_success_envelope.py b/tests/test_success_envelope.py index 3c413d73..46a78ef0 100644 --- a/tests/test_success_envelope.py +++ b/tests/test_success_envelope.py @@ -2,8 +2,8 @@ Tests for the canonical JSON success envelope. Every JSON object returned by a Datasette endpoint on success should include -"ok": true. (Endpoints that return a top-level array are being converted to -objects separately - see /-/plugins, /-/databases, /-/actions.) +"ok": true. /-/plugins intentionally returns a top-level array instead, while +/-/databases and /-/actions use the object envelope. """ import pytest @@ -79,17 +79,16 @@ async def test_permissions_post_success_has_ok_true(ds_envelope): @pytest.mark.asyncio -async def test_plugins_json_is_object(ds_client): +async def test_plugins_json_is_array(ds_client): response = await ds_client.get("/-/plugins.json") assert response.status_code == 200 data = response.json() - assert set(data.keys()) == {"ok", "plugins"} - assert data["ok"] is True - assert isinstance(data["plugins"], list) + assert isinstance(data, list) + assert all(isinstance(plugin, dict) for plugin in data) # ?all=1 should include Datasette's default plugins in the same shape response_all = await ds_client.get("/-/plugins.json?all=1") - all_plugins = response_all.json()["plugins"] - assert len(all_plugins) > len(data["plugins"]) + all_plugins = response_all.json() + assert len(all_plugins) > len(data) @pytest.mark.asyncio From 10088dfa1dd7ab0075f97e380121dff4f6a5222c Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 13 Jul 2026 22:42:44 -0700 Subject: [PATCH 068/131] execute_write(transaction=False) parameter, plus fix for errors inside tasks Ensure a write inside a failing Datasette task never becomes visible. Refs #2831 --- datasette/database.py | 7 ++++++- datasette/views/database.py | 9 +++++++- docs/internals.rst | 8 ++++--- tests/test_internals_database.py | 36 ++++++++++++++++++++++++++++++++ 4 files changed, 55 insertions(+), 5 deletions(-) diff --git a/datasette/database.py b/datasette/database.py index e7fe1ed9..eb402b0c 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -246,6 +246,7 @@ class Database: request=None, return_all=False, returning_limit=EXECUTE_WRITE_RETURNING_LIMIT, + transaction=True, ): self._check_not_closed() if returning_limit < 0: @@ -258,7 +259,9 @@ class Database: ) with trace("sql", database=self.name, sql=sql.strip(), params=params): - results = await self.execute_write_fn(_inner, block=block, request=request) + results = await self.execute_write_fn( + _inner, block=block, request=request, transaction=transaction + ) return results async def execute_write_script(self, sql, block=True, request=None): @@ -348,6 +351,7 @@ class Database: self.ds._prepare_connection(self._write_connection, self.name) if transaction: with self._write_connection: + self._write_connection.execute("BEGIN IMMEDIATE") result = fn(self._write_connection) else: result = fn(self._write_connection) @@ -477,6 +481,7 @@ class Database: try: if task.transaction: with conn: + conn.execute("BEGIN IMMEDIATE") result = task.fn(conn) else: result = task.fn(conn) diff --git a/datasette/views/database.py b/datasette/views/database.py index 10dc66ae..11646f45 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -643,8 +643,15 @@ class QueryView(View): ok = None redirect_url = None try: + execute_write_kwargs = {"request": request} + if stored_query.is_trusted: + analysis = await db.analyze_sql(stored_query.sql, params_for_query) + if any( + operation.operation == "vacuum" for operation in analysis.operations + ): + execute_write_kwargs["transaction"] = False cursor = await db.execute_write( - stored_query.sql, params_for_query, request=request + stored_query.sql, params_for_query, **execute_write_kwargs ) # success message can come from on_success_message or on_success_message_sql message = None diff --git a/docs/internals.rst b/docs/internals.rst index 2048c7e4..d2bd46ef 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -2023,8 +2023,8 @@ Example usage: .. _database_execute_write: -await db.execute_write(sql, params=None, block=True, request=None, return_all=False, returning_limit=10) --------------------------------------------------------------------------------------------------------- +await db.execute_write(sql, params=None, block=True, request=None, return_all=False, returning_limit=10, transaction=True) +-------------------------------------------------------------------------------------------------------------------------- SQLite only allows one database connection to write at a time. Datasette handles this for you by maintaining a queue of writes to be executed against a given database. Plugins can submit write operations to this queue and they will be executed in the order in which they are received. @@ -2059,7 +2059,9 @@ If you need to retrieve every row returned by a statement, pass ``return_all=Tru If you pass ``block=False`` this behavior changes to "fire and forget" - queries will be added to the write queue and executed in a separate thread while your code can continue to do other things. The method will return a UUID representing the queued task. -Each call to ``execute_write()`` will be executed inside a transaction. +Each call to ``execute_write()`` will be executed inside a transaction. Pass +``transaction=False`` for statements such as ``VACUUM`` that cannot run inside +a transaction. .. _database_execute_write_script: diff --git a/tests/test_internals_database.py b/tests/test_internals_database.py index bad4e8ca..b1a212d9 100644 --- a/tests/test_internals_database.py +++ b/tests/test_internals_database.py @@ -11,6 +11,7 @@ from datasette.database import _deliver_write_result from datasette.utils.sqlite import sqlite3, supports_returning from datasette.utils import Column import pytest +import sqlite_utils import time import uuid @@ -718,6 +719,41 @@ async def test_execute_write_fn_exception(db): await db.execute_write_fn(write_fn) +@pytest.mark.asyncio +@pytest.mark.parametrize("num_sql_threads", (0, 1)) +async def test_execute_write_fn_sqlite_utils_transaction(tmp_path, num_sql_threads): + # A write inside a failing Datasette task must never become visible or + # survive the rollback. Exercise both the synchronous and writer-thread + # paths against a file-backed database so a second connection can observe + # committed state independently. + db_path = tmp_path / "test.db" + sqlite3.connect(db_path).close() + ds = Datasette([str(db_path)], settings={"num_sql_threads": num_sql_threads}) + db = ds.get_database("test") + await db.execute_write("create table items (id integer primary key)") + # This reader is used inside the write callback, which may run on another + # thread, but it is never accessed concurrently. + reader = sqlite3.connect(db_path, check_same_thread=False) + + def insert_then_fail(conn): + # Datasette must open the outer transaction before sqlite-utils writes. + assert conn.in_transaction + sqlite_utils.Database(conn)["items"].insert({"id": 1}) + # If sqlite-utils committed its own transaction, this would return 1. + assert reader.execute("select count(*) from items").fetchone()[0] == 0 + # Simulate a later step failing after the sqlite-utils write succeeded. + raise ValueError("deliberate") + + try: + with pytest.raises(ValueError, match="deliberate"): + await db.execute_write_fn(insert_then_fail) + # The outer transaction must roll back the sqlite-utils write as well. + assert reader.execute("select count(*) from items").fetchone()[0] == 0 + finally: + reader.close() + db.close() + + @pytest.mark.asyncio @pytest.mark.parametrize("param_name", ["conn", "connection", "db", "c"]) async def test_execute_write_fn_accepts_any_single_param_name(db, param_name): From 7f0a8b38aee771828fe7b3c7e54850249016d753 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 14 Jul 2026 08:40:07 -0700 Subject: [PATCH 069/131] Better permission debug tools and documentation Closes #2841 --- datasette/default_permissions/config.py | 4 + .../templates/_permission_ui_styles.html | 66 +++- .../templates/_permissions_debug_tabs.html | 8 +- datasette/templates/allow_debug.html | 56 ++- datasette/templates/debug_allowed.html | 2 +- datasette/templates/debug_check.html | 344 ++++++++++-------- .../debug_permissions_playground.html | 73 ++-- datasette/templates/debug_rules.html | 2 +- datasette/utils/actions_sql.py | 236 ++++++++++++ datasette/views/special.py | 42 ++- docs/authentication.rst | 89 ++++- tests/test_permissions.py | 228 +++++++++++- 12 files changed, 890 insertions(+), 260 deletions(-) diff --git a/datasette/default_permissions/config.py b/datasette/default_permissions/config.py index aab87c1c..8edc976e 100644 --- a/datasette/default_permissions/config.py +++ b/datasette/default_permissions/config.py @@ -96,6 +96,10 @@ class ConfigPermissionProcessor: """Evaluate an allow block against the current actor.""" if allow_block is None: return None + # Values passed using ``-s permissions.* 1`` or ``0`` are parsed as + # integers, but should retain the CLI's boolean 1/0 behavior. + if isinstance(allow_block, int) and allow_block in (0, 1): + return bool(allow_block) return actor_matches_allow(self.actor, allow_block) def is_in_restriction_allowlist( diff --git a/datasette/templates/_permission_ui_styles.html b/datasette/templates/_permission_ui_styles.html index 53a824f1..21a2ea8f 100644 --- a/datasette/templates/_permission_ui_styles.html +++ b/datasette/templates/_permission_ui_styles.html @@ -6,8 +6,20 @@ padding: 1.5em; margin-bottom: 2em; } +.permission-form form { + max-width: 60rem; +} +.permission-form-grid { + display: grid; + gap: 1.5rem; + grid-template-columns: repeat(2, minmax(0, 1fr)); +} +.permission-form-result { + margin-top: 1rem; + max-width: 60rem; +} .form-section { - margin-bottom: 1em; + margin-bottom: 1.25em; } .form-section label { display: block; @@ -15,22 +27,51 @@ font-weight: bold; } .form-section input[type="text"], -.form-section select { - width: 100%; - max-width: 500px; - padding: 0.5em; +.form-section input[type="number"], +.form-section select, +.permission-textarea { + background-color: #fff; + border: 1px solid #aaa; + border-radius: 4px; box-sizing: border-box; - border: 1px solid #ccc; - border-radius: 3px; + box-shadow: inset 0 1px 2px rgba(0, 0, 0, 0.08); + color: #222; + font-family: inherit; + font-size: 1rem; + line-height: 1.4; + max-width: none; + width: 100%; +} +.form-section input[type="text"] { + height: 3rem; + padding: 0.6rem 0.75rem; +} +.form-section input[type="number"] { + height: 3rem; + max-width: 7rem; + padding: 0.6rem 0.75rem; +} +.form-section select { + height: 3rem; + padding: 0.6rem 0.75rem; +} +.permission-textarea { + font-family: monospace; + min-height: 12rem; + padding: 0.75rem; + resize: vertical; } .form-section input[type="text"]:focus, -.form-section select:focus { - outline: 2px solid #0066cc; +.form-section input[type="number"]:focus, +.form-section select:focus, +.permission-textarea:focus { border-color: #0066cc; + box-shadow: 0 0 0 3px rgba(0, 102, 204, 0.18); + outline: none; } .form-section small { display: block; - margin-top: 0.3em; + margin-top: 0.45em; color: #666; } .form-actions { @@ -142,4 +183,9 @@ text-align: center; color: #666; } +@media only screen and (max-width: 576px) { + .permission-form-grid { + grid-template-columns: minmax(0, 1fr); + } +} diff --git a/datasette/templates/_permissions_debug_tabs.html b/datasette/templates/_permissions_debug_tabs.html index d7203c1e..8e0f486e 100644 --- a/datasette/templates/_permissions_debug_tabs.html +++ b/datasette/templates/_permissions_debug_tabs.html @@ -44,10 +44,10 @@ diff --git a/datasette/templates/allow_debug.html b/datasette/templates/allow_debug.html index 1ecc92df..fda4032c 100644 --- a/datasette/templates/allow_debug.html +++ b/datasette/templates/allow_debug.html @@ -3,29 +3,11 @@ {% block title %}Debug allow rules{% endblock %} {% block extra_head %} +{% include "_permission_ui_styles.html" %} {% endblock %} @@ -38,24 +20,28 @@ p.message-warning {

Use this tool to try out different actor and allow combinations. See Defining permissions with "allow" blocks for documentation.

- -
-

- -
-
-

- -
-
- -
- +
+
+
+
+ + +
+
+ + +
+
+
+ +
+ -{% if error %}

{{ error }}

{% endif %} + {% if error %}

{{ error }}

{% endif %} -{% if result == "True" %}

Result: allow

{% endif %} + {% if result == "True" %}

Result: allow

{% endif %} -{% if result == "False" %}

Result: deny

{% endif %} + {% if result == "False" %}

Result: deny

{% endif %} +
{% endblock %} diff --git a/datasette/templates/debug_allowed.html b/datasette/templates/debug_allowed.html index 83cc1ae6..80249d9c 100644 --- a/datasette/templates/debug_allowed.html +++ b/datasette/templates/debug_allowed.html @@ -49,7 +49,7 @@
- + Number of results per page (max 200)
diff --git a/datasette/templates/debug_check.html b/datasette/templates/debug_check.html index 3b229a25..b9fc636a 100644 --- a/datasette/templates/debug_check.html +++ b/datasette/templates/debug_check.html @@ -1,6 +1,6 @@ {% extends "base.html" %} -{% block title %}Permission Check{% endblock %} +{% block title %}Explain a permission decision{% endblock %} {% block extra_head %} @@ -13,29 +13,35 @@ border-radius: 5px; } #output.allowed { - background-color: #e8f5e9; + background-color: #f3fbf4; border: 2px solid #4caf50; } #output.denied { - background-color: #ffebee; + background-color: #fff7f7; border: 2px solid #f44336; } #output h2 { margin-top: 0; } -#output .result-badge { +#output h3 { + margin-bottom: 0.5em; +} +#output .result-badge, +.effect-badge, +.rule-status { display: inline-block; - padding: 0.3em 0.8em; + padding: 0.2em 0.5em; border-radius: 3px; font-weight: bold; - font-size: 1.1em; } -#output .allowed-badge { - background-color: #4caf50; +#output .allowed-badge, +.effect-allow { + background-color: #2e7d32; color: white; } -#output .denied-badge { - background-color: #f44336; +#output .denied-badge, +.effect-deny { + background-color: #c62828; color: white; } .details-section { @@ -48,70 +54,130 @@ .details-section dd { margin-left: 1em; } +.explanation-section { + background: rgba(255, 255, 255, 0.75); + border: 1px solid #ddd; + border-radius: 4px; + margin-top: 1em; + padding: 0 1em 1em; +} +.rules-table { + border-collapse: collapse; + width: 100%; +} +.rules-table th, +.rules-table td { + border-bottom: 1px solid #ddd; + padding: 0.5em; + text-align: left; + vertical-align: top; +} +.rule-status { + background: #e8f5e9; + color: #1b5e20; +} +.rule-ignored { + background: #eee; + color: #555; + font-weight: normal; +} +.requirement-allowed { + color: #1b5e20; +} +.requirement-denied { + color: #b71c1c; +} +@media only screen and (max-width: 576px) { + .rules-table, + .rules-table tbody, + .rules-table tr, + .rules-table td { + display: block; + } + .rules-table thead { + display: none; + } + .rules-table td::before { + content: attr(data-label) ": "; + font-weight: bold; + } +} {% endblock %} {% block content %} -

Permission check

+

Explain a permission decision

{% set current_tab = "check" %} {% include "_permissions_debug_tabs.html" %} -

Use this tool to test permission checks for the current actor. It queries the /-/check.json API endpoint.

- -{% if request.actor %} -

Current actor: {{ request.actor.get("id", "anonymous") }}

-{% else %} -

Current actor: anonymous (not logged in)

-{% endif %} +

Test an actor, action and resource. The result explains which rules matched, which specificity level won, and whether actor restrictions or required actions changed the verdict.

-
+
- + + + Use null for an anonymous actor. This actor is simulated; it does not change who you are signed in as. +
+ +
+ - The permission action to check + The operation to evaluate
-
- +
+ - For database-level permissions, specify the database name + The database or other parent resource
-
- - - For table-level permissions, specify the table name (requires parent) +
+ + + The table, query or other child resource
- +
+actionSelect.addEventListener('change', updateResourceFields); +(function initializeFromUrl() { + const params = populateFormFromURL(); + updateResourceFields(); + if (params.get('action')) { + performCheck(); + } +})(); + {% endblock %} diff --git a/datasette/templates/debug_permissions_playground.html b/datasette/templates/debug_permissions_playground.html index 4410a677..8b0cbbcf 100644 --- a/datasette/templates/debug_permissions_playground.html +++ b/datasette/templates/debug_permissions_playground.html @@ -1,6 +1,6 @@ {% extends "base.html" %} -{% block title %}Debug permissions{% endblock %} +{% block title %}Permission activity{% endblock %} {% block extra_head %} {% include "_permission_ui_styles.html" %} @@ -20,60 +20,45 @@ .check-action, .check-when, .check-result { font-size: 1.3em; } -textarea { - height: 10em; - width: 95%; - box-sizing: border-box; - padding: 0.5em; - border: 2px dotted black; -} -.two-col { - display: inline-block; - width: 48%; -} -.two-col label { - width: 48%; -} -@media only screen and (max-width: 576px) { - .two-col { - width: 100%; - } -} {% endblock %} {% block content %} -

Permission playground

+

Permission activity

{% set current_tab = "permissions" %} {% include "_permissions_debug_tabs.html" %} -

This tool lets you simulate an actor and a permission check for that actor.

+

Raw simulator

+ +

This form runs a hypothetical permission check and returns its raw explanation JSON. Use the Explain tool for a visual explanation of the same decision.

-
-
- - +
+
+
+ + +
-
-
-
- - -
-
- - -
-
- - +
+
+ + +
+
+ + +
+
+ + +
@@ -125,7 +110,7 @@ debugPost.addEventListener('submit', function(ev) { }); -

Recent permissions checks

+

Recent permission checks

{% if filter != "all" %}All{% else %}All{% endif %}, diff --git a/datasette/templates/debug_rules.html b/datasette/templates/debug_rules.html index d00ba9cc..233c0e94 100644 --- a/datasette/templates/debug_rules.html +++ b/datasette/templates/debug_rules.html @@ -37,7 +37,7 @@

- + Number of results per page (max 200)
diff --git a/datasette/utils/actions_sql.py b/datasette/utils/actions_sql.py index c7137e6b..67d3ce73 100644 --- a/datasette/utils/actions_sql.py +++ b/datasette/utils/actions_sql.py @@ -673,3 +673,239 @@ async def check_permission_for_resource( child=child, ) return results[action] + + +async def explain_permission_for_resource( + *, + datasette: "Datasette", + actor: dict | None, + action: str, + parent: str | None, + child: str | None, +) -> dict: + """Explain a permission decision for one action and resource. + + This is intended for Datasette's permission debugging tools. It uses the + same ``permission_resources_sql`` hook results and the same resolution + rules as :func:`check_permissions_for_actions`, but also returns the + matching rules, actor restriction results and ``also_requires`` chain. + + The returned dictionary is part of Datasette's unstable debugging API. + """ + + action_obj = datasette.actions.get(action) + if action_obj is None: + raise ValueError(f"Unknown action: {action}") + + explanation = await _explain_single_action( + datasette=datasette, + actor=actor, + action=action, + parent=parent, + child=child, + ) + + required_actions = [] + if action_obj.also_requires: + required = await explain_permission_for_resource( + datasette=datasette, + actor=actor, + action=action_obj.also_requires, + parent=parent, + child=child, + ) + required_actions.append(required) + + explanation["required_actions"] = required_actions + explanation["allowed"] = bool( + explanation["rule_allowed"] + and explanation["restriction_allowed"] + and all(required["allowed"] for required in required_actions) + ) + explanation["summary"] = _permission_explanation_summary(explanation) + return explanation + + +async def _explain_single_action( + *, + datasette: "Datasette", + actor: dict | None, + action: str, + parent: str | None, + child: str | None, +) -> dict: + """Return matching rules and restrictions for a single action.""" + from datasette.utils.permissions import SKIP_PERMISSION_CHECKS + + permission_sqls = await gather_permission_sql_from_hooks( + datasette=datasette, + actor=actor, + action=action, + ) + + if permission_sqls is SKIP_PERMISSION_CHECKS: + return { + "action": action, + "rule_allowed": True, + "restriction_allowed": True, + "winning_scope": "global", + "matched_rules": [ + { + "scope": "global", + "effect": "allow", + "source": "skip_permission_checks", + "reason": "Permission checks were explicitly skipped", + "decisive": True, + "ignored_because": None, + } + ], + "restrictions": [], + } + + db = datasette.get_internal_database() + matched_rules = [] + restrictions = [] + + for permission_sql in permission_sqls: + params = dict(permission_sql.params or {}) + parent_param = _unused_parameter_name(params, "_explain_parent") + params[parent_param] = parent + child_param = _unused_parameter_name(params, "_explain_child") + params[child_param] = child + + if permission_sql.sql: + rows = await db.execute( + f""" + SELECT parent, child, allow, reason + FROM ({permission_sql.sql}) AS permission_rules + WHERE (parent IS NULL OR parent = :{parent_param}) + AND (child IS NULL OR child = :{child_param}) + """, + params, + ) + for row in rows: + specificity = ( + 2 + if row["child"] is not None + else 1 if row["parent"] is not None else 0 + ) + matched_rules.append( + { + "scope": ("resource", "parent", "global")[2 - specificity], + "effect": "allow" if row["allow"] else "deny", + "source": permission_sql.source, + "reason": row["reason"], + "_specificity": specificity, + } + ) + + if permission_sql.restriction_sql: + restriction_row = ( + await db.execute( + f""" + SELECT EXISTS( + SELECT 1 FROM ({permission_sql.restriction_sql}) AS restriction_rules + WHERE (parent IS NULL OR parent = :{parent_param}) + AND (child IS NULL OR child = :{child_param}) + ) AS resource_is_in_allowlist + """, + params, + ) + ).first() + restriction_allowed = bool(restriction_row[0]) + restrictions.append( + { + "source": permission_sql.source, + "allowed": restriction_allowed, + "reason": params.get("deny") + or ( + "Resource is included in this restriction allowlist" + if restriction_allowed + else "Resource is not included in this restriction allowlist" + ), + } + ) + + matched_rules.sort( + key=lambda rule: ( + -rule["_specificity"], + 0 if rule["effect"] == "deny" else 1, + rule["source"] or "", + rule["reason"] or "", + ) + ) + + if matched_rules: + winning_specificity = matched_rules[0]["_specificity"] + winning_rules = [ + rule + for rule in matched_rules + if rule["_specificity"] == winning_specificity + ] + rule_allowed = not any(rule["effect"] == "deny" for rule in winning_rules) + winning_scope = winning_rules[0]["scope"] + else: + winning_specificity = None + rule_allowed = False + winning_scope = None + + for rule in matched_rules: + specificity = rule.pop("_specificity") + if specificity != winning_specificity: + rule["decisive"] = False + rule["ignored_because"] = "A more specific rule matched" + elif not rule_allowed and rule["effect"] == "allow": + rule["decisive"] = False + rule["ignored_because"] = "A deny rule matched at the same scope" + else: + rule["decisive"] = True + rule["ignored_because"] = None + + return { + "action": action, + "rule_allowed": rule_allowed, + "restriction_allowed": all( + restriction["allowed"] for restriction in restrictions + ), + "winning_scope": winning_scope, + "matched_rules": matched_rules, + "restrictions": restrictions, + } + + +def _unused_parameter_name(params: dict, preferred: str) -> str: + """Return a SQL parameter name that is not already in ``params``.""" + candidate = preferred + suffix = 2 + while candidate in params: + candidate = f"{preferred}_{suffix}" + suffix += 1 + return candidate + + +def _permission_explanation_summary(explanation: dict) -> str: + denied_requirement = next( + ( + required + for required in explanation["required_actions"] + if not required["allowed"] + ), + None, + ) + if denied_requirement: + return ( + f"Denied because {explanation['action']} also requires " + f"{denied_requirement['action']}, which was denied." + ) + if not explanation["matched_rules"]: + return "Denied because no permission rule matched this actor and resource." + if not explanation["rule_allowed"]: + return ( + f"Denied by a {explanation['winning_scope']}-level rule. " + "Deny rules take precedence over allow rules at the same scope." + ) + if not explanation["restriction_allowed"]: + return ( + "Denied because the resource is not included in the actor's restrictions." + ) + return f"Allowed by the matching {explanation['winning_scope']}-level rule." diff --git a/datasette/views/special.py b/datasette/views/special.py index c13191a1..28d34208 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -600,7 +600,7 @@ class PermissionRulesView(BaseView): async def _check_permission_for_actor(ds, action, parent, child, actor): - """Shared logic for checking permissions. Returns a dict with check results.""" + """Shared logic for checking and explaining a permission decision.""" if action not in ds.actions: return error_body(f"Unknown action: {action}", 404), 404 @@ -629,15 +629,28 @@ async def _check_permission_for_actor(ds, action, parent, child, actor): allowed = await ds.allowed(action=action, resource=resource_obj, actor=actor) + from datasette.utils.actions_sql import explain_permission_for_resource + + explanation = await explain_permission_for_resource( + datasette=ds, + actor=actor, + action=action, + parent=parent, + child=child, + ) + response = { "ok": True, + "unstable": UNSTABLE_API_MESSAGE, "action": action, "allowed": bool(allowed), + "actor": actor, "resource": { "parent": parent, "child": child, "path": _resource_path(parent, child), }, + "explanation": explanation, } if actor and "id" in actor: @@ -655,11 +668,25 @@ class PermissionCheckView(BaseView): as_format = request.url_vars.get("format") if not as_format: + actions = [ + { + "name": action.name, + "description": action.description, + "takes_parent": action.takes_parent, + "takes_child": action.takes_child, + "also_requires": action.also_requires, + } + for action in sorted( + self.ds.actions.values(), key=lambda action: action.name + ) + ] return await self.render( ["debug_check.html"], request, { - "sorted_actions": sorted(self.ds.actions.keys()), + "actions": actions, + "actor_json": request.args.get("actor") + or json.dumps(request.actor, indent=2), "has_debug_permission": True, }, ) @@ -671,9 +698,18 @@ class PermissionCheckView(BaseView): parent = request.args.get("parent") child = request.args.get("child") + actor = request.actor + actor_json = request.args.get("actor") + if actor_json is not None: + try: + actor = json.loads(actor_json) + except json.JSONDecodeError as ex: + return Response.error(f"Invalid actor JSON: {ex}", 400) + if actor is not None and not isinstance(actor, dict): + return Response.error("actor must be a JSON object or null", 400) response, status = await _check_permission_for_actor( - self.ds, action, parent, child, request.actor + self.ds, action, parent, child, actor ) return Response.json(response, status=status) diff --git a/docs/authentication.rst b/docs/authentication.rst index 51fa07d5..d720c4db 100644 --- a/docs/authentication.rst +++ b/docs/authentication.rst @@ -45,7 +45,7 @@ Using the "root" actor Datasette currently leaves almost all forms of authentication to plugins - `datasette-auth-github `__ for example. -The one exception is the "root" account, which you can sign into while using Datasette on your local machine. The root user has **all permissions** - they can perform any action regardless of other permission rules. +The one exception is the "root" account, which you can sign into while using Datasette on your local machine. The root user starts with **all permissions**: Datasette contributes a global allow rule for every action. More specific deny rules can still override that global rule. The ``--root`` flag is designed for local development and testing. When you start Datasette with ``--root``, the root user automatically receives every permission, including: @@ -84,12 +84,12 @@ Click on that link and then visit ``http://127.0.0.1:8001/-/actor`` to confirm t Permissions =========== -Datasette's permissions system is built around SQL queries. Datasette and its plugins construct SQL queries to resolve the list of resources that an actor cas access. - The key question the permissions system answers is this: Is this **actor** allowed to perform this **action**, optionally against this particular **resource**? +Every permission decision can be understood in terms of those three values. Datasette implements the decisions using SQL, but you do not need to understand the generated SQL to configure or debug permissions. + **Actors** are :ref:`described above `. An **action** is a string describing the action the actor would like to perform. A full list is :ref:`provided below ` - examples include ``view-table`` and ``execute-sql``. @@ -138,7 +138,51 @@ This configuration will deny access to everyone except the user with ``id`` of ` How permissions are resolved ---------------------------- -Datasette performs permission checks using the internal :ref:`datasette_allowed`, method which accepts keyword arguments for ``action``, ``resource`` and an optional ``actor``. +Permission rules describe an effect (``allow`` or ``deny``) at one of three levels: + +``resource`` + A specific child resource, such as the ``analytics/sales`` table. + +``parent`` + A parent resource, such as the ``analytics`` database. A parent rule also applies to its child resources. + +``global`` + Every resource for that action. + +Datasette resolves matching rules from most specific to least specific: + +#. Resource rules take precedence over parent and global rules. +#. Parent rules take precedence over global rules. +#. If both allow and deny rules match at the same level, deny takes precedence. +#. If no rule matches, access is denied. + +This means a resource-level allow can provide an exception to a parent-level deny. It also means that two plugins which disagree at the same level resolve to deny. + +.. list-table:: Permission rule examples + :header-rows: 1 + + * - Matching rules + - Result + - Explanation + * - Global allow + - Allow + - The global rule is the most specific matching rule. + * - Global allow, parent deny + - Deny + - The parent rule is more specific. + * - Parent deny, resource allow + - Allow + - The resource rule is more specific. + * - Resource allow and resource deny + - Deny + - Deny takes precedence at the same level. + * - No matching rules + - Deny + - Permissions default to deny when no rule applies. + +The built-in public defaults are global allow rules for actions such as ``view-instance``, ``view-database`` and ``view-table``. They follow the same precedence rules as configuration and plugin rules. The ``--default-deny`` option prevents Datasette from contributing those default allow rules. + +Datasette performs checks using :ref:`datasette_allowed`, which accepts keyword arguments for ``action``, ``resource`` and an optional ``actor``. ``resource`` should be an instance of the appropriate ``Resource`` subclass from :mod:`datasette.resources`—for example ``InstanceResource()``, ``DatabaseResource(database="...``)`` or ``TableResource(database="...", table="...")``. This defaults to ``InstanceResource()`` if not specified. @@ -149,12 +193,12 @@ resources were allowed or denied. The combined sources are: * ``allow`` blocks configured in :ref:`datasette.yaml `. * :ref:`Actor restrictions ` encoded into the actor dictionary or API token. -* The "root" user shortcut when ``--root`` (or :attr:`Datasette.root_enabled `) is active, replying ``True`` to all permission chucks unless configuration rules deny them at a more specific level. +* The "root" user rule when ``--root`` (or :attr:`Datasette.root_enabled `) is active. This is a global allow rule, so a more specific configuration deny can override it. * Any additional SQL provided by plugins implementing :ref:`plugin_hook_permission_resources_sql`. -Datasette evaluates the SQL to determine if the requested ``resource`` is -included. Explicit deny rules returned by configuration or plugins will block -access even if other rules allowed it. +Actor restrictions are applied after the allow/deny rules. They act as an additional allowlist: a restriction can remove access but cannot grant access that the actor did not already have. See :ref:`authentication_cli_create_token_restrict`. + +Some actions have dependencies on other actions. These are evaluated as an ``AND`` condition. For example, ``execute-sql`` also requires ``view-database``: both decisions must be allowed for the final result to be allowed. .. _authentication_permissions_allow: @@ -1145,11 +1189,21 @@ The debug tool at ``/-/permissions`` is available to any actor with the ``permis datasette -s permissions.permissions-debug true data.db -The page shows the permission checks that have been carried out by the Datasette instance. +The permission debug tools answer four different questions: -It also provides an interface for running hypothetical permission checks against a hypothetical actor. This is a useful way of confirming that your configured permissions work in the way you expect. +Why was this decision allowed or denied? + Use :ref:`PermissionCheckView`. It shows every matching rule, identifies the winning specificity level, applies actor restrictions and evaluates any required actions. -This is designed to help administrators and plugin authors understand exactly how permission checks are being carried out, in order to effectively configure Datasette's permission system. +Which resources can the current actor access? + Use :ref:`AllowedResourcesView` to view an access map for a selected action. + +Which raw rules did Datasette and its plugins contribute? + Use :ref:`PermissionRulesView` to inspect the rules before they are resolved into decisions. + +Which checks has this Datasette instance performed recently? + Use ``/-/permissions`` to view recent permission activity. + +These tools are designed to help administrators and plugin authors understand and confirm the effective permissions configuration. These debug endpoints are exempt from the :ref:`JSON API stability promise ` - their JSON shapes may change in future releases. @@ -1184,11 +1238,20 @@ This endpoint requires the ``permissions-debug`` permission. Permission check view --------------------- -The ``/-/check`` endpoint evaluates a single action/resource pair and returns information indicating whether the access was allowed along with diagnostic information. +The ``/-/check`` endpoint evaluates and explains a single actor, action and resource decision. The explanation includes: + +* Every matching allow and deny rule, with its source and reason. +* The winning resource, parent or global scope. +* Rules ignored because a more specific rule matched, or because a deny won at the same scope. +* Actor restriction allowlists that included or excluded the resource. +* Additional actions required by the requested action. +* An explicit default-deny explanation when no rule matched. This endpoint provides an interactive HTML form interface. Add ``.json`` to the URL path (e.g. ``/-/check.json?action=view-instance``) to get the raw JSON response instead. -Pass ``?action=`` to specify the action to check, and optional ``?parent=`` and ``?child=`` parameters to specify the resource. +Pass ``?action=`` to specify the action to check, and optional ``?parent=`` and ``?child=`` parameters to specify the resource. The interactive form also accepts actor JSON, allowing a hypothetical actor to be tested without signing in as that actor. The JSON endpoint accepts the same value using the ``actor`` query string parameter. Use ``actor=null`` to represent an anonymous actor. + +This endpoint requires the ``permissions-debug`` permission. The hypothetical actor is used only for the decision being explained; access to the debug tool is checked against the actor who is actually signed in. .. _authentication_ds_actor: diff --git a/tests/test_permissions.py b/tests/test_permissions.py index 88fe577f..cd1050d0 100644 --- a/tests/test_permissions.py +++ b/tests/test_permissions.py @@ -457,6 +457,20 @@ async def test_permissions_debug(ds_client, filter_): assert checks == expected_checks +@pytest.mark.asyncio +@pytest.mark.parametrize( + "permissions_debug,expected_status", + ( + (1, 200), + (0, 403), + ), +) +async def test_permissions_debug_numeric_boolean(permissions_debug, expected_status): + ds = Datasette(config={"permissions": {"permissions-debug": permissions_debug}}) + response = await ds.client.get("/-/permissions") + assert response.status_code == expected_status + + @pytest.mark.asyncio @pytest.mark.parametrize( "actor,allow,expected_fragment", @@ -748,7 +762,12 @@ async def test_actor_restricted_permissions( } if actor.get("id"): expected["actor_id"] = actor["id"] - assert response.json() == expected + data = response.json() + for key, value in expected.items(): + assert data[key] == value + assert data["actor"] == actor + assert data["explanation"]["allowed"] is expected_result + assert data["explanation"]["summary"] PermConfigTestCase = collections.namedtuple( @@ -1734,6 +1753,8 @@ async def test_permission_check_view_requires_debug_permission(): data = response.json() assert data["action"] == "view-instance" assert data["allowed"] is True + assert data["explanation"]["allowed"] is True + assert data["explanation"]["summary"] @pytest.mark.asyncio @@ -1759,6 +1780,211 @@ async def test_permission_check_view_query_actions(action): } +@pytest.mark.asyncio +async def test_permission_check_explains_specificity_for_hypothetical_actor(): + ds = Datasette( + config={ + "permissions": {"view-table": {"id": "alice"}}, + "databases": { + "analytics": { + "permissions": {"view-table": False}, + "tables": { + "public": {"permissions": {"view-table": {"id": "alice"}}} + }, + } + }, + } + ) + ds.root_enabled = True + await ds.invoke_startup() + + def path_for(child): + return "/-/check.json?" + urllib.parse.urlencode( + { + "action": "view-table", + "parent": "analytics", + "child": child, + "actor": json.dumps({"id": "alice"}), + } + ) + + public_response = await ds.client.get(path_for("public"), actor={"id": "root"}) + assert public_response.status_code == 200 + public = public_response.json() + assert public["actor"] == {"id": "alice"} + assert public["allowed"] is True + assert public["explanation"]["allowed"] is True + assert public["explanation"]["winning_scope"] == "resource" + public_rules = public["explanation"]["matched_rules"] + assert any( + rule["scope"] == "resource" and rule["effect"] == "allow" and rule["decisive"] + for rule in public_rules + ) + assert any( + rule["scope"] == "parent" + and rule["effect"] == "deny" + and rule["ignored_because"] == "A more specific rule matched" + for rule in public_rules + ) + + private_response = await ds.client.get(path_for("private"), actor={"id": "root"}) + assert private_response.status_code == 200 + private = private_response.json() + assert private["allowed"] is False + assert private["explanation"]["allowed"] is False + assert private["explanation"]["winning_scope"] == "parent" + assert private["explanation"]["summary"].startswith("Denied by a parent-level rule") + + +@pytest.mark.asyncio +async def test_permission_check_explains_deny_wins_at_same_scope(): + ds = Datasette(config={"permissions": {"view-table": {"id": "someone-else"}}}) + ds.root_enabled = True + await ds.invoke_startup() + path = "/-/check.json?" + urllib.parse.urlencode( + { + "action": "view-table", + "parent": "analytics", + "child": "users", + "actor": json.dumps({"id": "alice"}), + } + ) + response = await ds.client.get(path, actor={"id": "root"}) + assert response.status_code == 200 + data = response.json() + assert data["allowed"] is False + assert data["explanation"]["winning_scope"] == "global" + rules = data["explanation"]["matched_rules"] + assert any(rule["effect"] == "deny" and rule["decisive"] for rule in rules) + assert any( + rule["effect"] == "allow" + and rule["ignored_because"] == "A deny rule matched at the same scope" + for rule in rules + ) + + +@pytest.mark.asyncio +async def test_permission_check_explains_default_deny(): + ds = Datasette() + ds.root_enabled = True + await ds.invoke_startup() + path = "/-/check.json?" + urllib.parse.urlencode( + { + "action": "insert-row", + "parent": "analytics", + "child": "users", + "actor": json.dumps({"id": "alice"}), + } + ) + response = await ds.client.get(path, actor={"id": "root"}) + assert response.status_code == 200 + data = response.json() + assert data["allowed"] is False + explanation = data["explanation"] + assert explanation["allowed"] is False + assert explanation["matched_rules"] == [] + assert explanation["winning_scope"] is None + assert explanation["summary"] == ( + "Denied because no permission rule matched this actor and resource." + ) + + +@pytest.mark.asyncio +async def test_permission_check_explains_actor_restrictions(): + ds = Datasette() + ds.root_enabled = True + await ds.invoke_startup() + restricted_actor = { + "id": "alice", + "_r": {"r": {"analytics": {"public": ["vt"]}}}, + } + path = "/-/check.json?" + urllib.parse.urlencode( + { + "action": "view-table", + "parent": "analytics", + "child": "private", + "actor": json.dumps(restricted_actor), + } + ) + response = await ds.client.get(path, actor={"id": "root"}) + assert response.status_code == 200 + data = response.json() + assert data["allowed"] is False + explanation = data["explanation"] + assert explanation["rule_allowed"] is True + assert explanation["restriction_allowed"] is False + assert explanation["allowed"] is False + assert explanation["restrictions"] + assert any( + restriction["allowed"] is False for restriction in explanation["restrictions"] + ) + assert "actor's restrictions" in explanation["summary"] + + +@pytest.mark.asyncio +async def test_permission_check_explains_required_actions(): + from datasette import hookimpl + from datasette.permissions import PermissionSQL + + class StoreQueryPermissions: + @hookimpl + def permission_resources_sql(self, actor, action): + if not actor or actor.get("id") != "alice": + return None + if action == "store-query": + return PermissionSQL( + sql="SELECT 'analytics' AS parent, NULL AS child, 1 AS allow, 'alice can store queries' AS reason" + ) + if action == "execute-sql": + return PermissionSQL( + sql="SELECT 'analytics' AS parent, NULL AS child, 0 AS allow, 'alice cannot execute SQL' AS reason" + ) + + ds = Datasette() + ds.root_enabled = True + await ds.invoke_startup() + ds.pm.register(StoreQueryPermissions(), name="store-query-test") + path = "/-/check.json?" + urllib.parse.urlencode( + { + "action": "store-query", + "parent": "analytics", + "actor": json.dumps({"id": "alice"}), + } + ) + response = await ds.client.get(path, actor={"id": "root"}) + assert response.status_code == 200 + data = response.json() + assert data["allowed"] is False + explanation = data["explanation"] + assert explanation["rule_allowed"] is True + assert explanation["required_actions"][0]["action"] == "execute-sql" + assert explanation["required_actions"][0]["allowed"] is False + assert explanation["summary"] == ( + "Denied because store-query also requires execute-sql, which was denied." + ) + + +@pytest.mark.asyncio +async def test_permission_check_hypothetical_actor_validation(): + ds = Datasette() + ds.root_enabled = True + await ds.invoke_startup() + + response = await ds.client.get( + "/-/check.json?action=view-instance&actor=not-json", + actor={"id": "root"}, + ) + assert response.status_code == 400 + assert response.json()["error"].startswith("Invalid actor JSON:") + + response = await ds.client.get( + "/-/check.json?action=view-instance&actor=%5B%5D", + actor={"id": "root"}, + ) + assert response.status_code == 400 + assert response.json()["error"] == "actor must be a JSON object or null" + + @pytest.mark.asyncio async def test_root_allow_block_with_table_restricted_actor(): """ From 9cfc252394eb21d05f45818dac9c374e2d141a35 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 14 Jul 2026 08:41:27 -0700 Subject: [PATCH 070/131] Make internal catalog refresh atomic Refs #2831 --- datasette/app.py | 14 +--- datasette/utils/internal_db.py | 133 ++++++++++++++++++--------------- 2 files changed, 72 insertions(+), 75 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index c44f9095..0e31273d 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -753,19 +753,7 @@ class Datasette: # Compare schema versions to see if we should skip it if schema_version == current_schema_versions.get(database_name): continue - placeholders = "(?, ?, ?, ?)" - values = [database_name, str(db.path), db.is_memory, schema_version] - if db.path is None: - placeholders = "(?, null, ?, ?)" - values = [database_name, db.is_memory, schema_version] - await internal_db.execute_write( - """ - INSERT OR REPLACE INTO catalog_databases (database_name, path, is_memory, schema_version) - VALUES {} - """.format(placeholders), - values, - ) - await populate_schema_tables(internal_db, db) + await populate_schema_tables(internal_db, db, schema_version) @property def urls(self): diff --git a/datasette/utils/internal_db.py b/datasette/utils/internal_db.py index 10ca32a5..e061d882 100644 --- a/datasette/utils/internal_db.py +++ b/datasette/utils/internal_db.py @@ -180,29 +180,9 @@ async def init_internal_db(db): await db.execute_write_fn(apply_migrations, transaction=False) -async def populate_schema_tables(internal_db, db): +async def populate_schema_tables(internal_db, db, schema_version): database_name = db.name - def delete_everything(conn): - conn.execute( - "DELETE FROM catalog_tables WHERE database_name = ?", [database_name] - ) - conn.execute( - "DELETE FROM catalog_views WHERE database_name = ?", [database_name] - ) - conn.execute( - "DELETE FROM catalog_columns WHERE database_name = ?", [database_name] - ) - conn.execute( - "DELETE FROM catalog_foreign_keys WHERE database_name = ?", - [database_name], - ) - conn.execute( - "DELETE FROM catalog_indexes WHERE database_name = ?", [database_name] - ) - - await internal_db.execute_write_fn(delete_everything) - tables = (await db.execute("select * from sqlite_master WHERE type = 'table'")).rows views = (await db.execute("select * from sqlite_master WHERE type = 'view'")).rows @@ -266,47 +246,76 @@ async def populate_schema_tables(internal_db, db): indexes_to_insert, ) = await db.execute_fn(collect_info) - await internal_db.execute_write_many( - """ - INSERT INTO catalog_tables (database_name, table_name, rootpage, sql) - values (?, ?, ?, ?) - """, - tables_to_insert, - ) - await internal_db.execute_write_many( - """ - INSERT INTO catalog_views (database_name, view_name, rootpage, sql) - values (?, ?, ?, ?) - """, - views_to_insert, - ) - await internal_db.execute_write_many( - """ - INSERT INTO catalog_columns ( - database_name, table_name, cid, name, type, "notnull", default_value, is_pk, hidden - ) VALUES ( - :database_name, :table_name, :cid, :name, :type, :notnull, :default_value, :is_pk, :hidden + def replace_catalog(conn): + # Delete child rows before their catalog_tables parents so this also + # works if a prepare_connection plugin enables foreign key enforcement. + for table in ( + "catalog_columns", + "catalog_foreign_keys", + "catalog_indexes", + "catalog_views", + "catalog_tables", + ): + conn.execute( + "DELETE FROM {} WHERE database_name = ?".format(table), + [database_name], + ) + conn.execute( + """ + INSERT OR REPLACE INTO catalog_databases ( + database_name, path, is_memory, schema_version + ) VALUES (?, ?, ?, ?) + """, + [ + database_name, + str(db.path) if db.path is not None else None, + db.is_memory, + schema_version, + ], ) - """, - columns_to_insert, - ) - await internal_db.execute_write_many( - """ - INSERT INTO catalog_foreign_keys ( - database_name, table_name, "id", seq, "table", "from", "to", on_update, on_delete, match - ) VALUES ( - :database_name, :table_name, :id, :seq, :table, :from, :to, :on_update, :on_delete, :match + conn.executemany( + """ + INSERT INTO catalog_tables (database_name, table_name, rootpage, sql) + values (?, ?, ?, ?) + """, + tables_to_insert, ) - """, - foreign_keys_to_insert, - ) - await internal_db.execute_write_many( - """ - INSERT INTO catalog_indexes ( - database_name, table_name, seq, name, "unique", origin, partial - ) VALUES ( - :database_name, :table_name, :seq, :name, :unique, :origin, :partial + conn.executemany( + """ + INSERT INTO catalog_views (database_name, view_name, rootpage, sql) + values (?, ?, ?, ?) + """, + views_to_insert, ) - """, - indexes_to_insert, - ) + conn.executemany( + """ + INSERT INTO catalog_columns ( + database_name, table_name, cid, name, type, "notnull", default_value, is_pk, hidden + ) VALUES ( + :database_name, :table_name, :cid, :name, :type, :notnull, :default_value, :is_pk, :hidden + ) + """, + columns_to_insert, + ) + conn.executemany( + """ + INSERT INTO catalog_foreign_keys ( + database_name, table_name, "id", seq, "table", "from", "to", on_update, on_delete, match + ) VALUES ( + :database_name, :table_name, :id, :seq, :table, :from, :to, :on_update, :on_delete, :match + ) + """, + foreign_keys_to_insert, + ) + conn.executemany( + """ + INSERT INTO catalog_indexes ( + database_name, table_name, seq, name, "unique", origin, partial + ) VALUES ( + :database_name, :table_name, :seq, :name, :unique, :origin, :partial + ) + """, + indexes_to_insert, + ) + + await internal_db.execute_write_fn(replace_catalog) From 591b909a4d216ed76d3c775484df52b54e89dc74 Mon Sep 17 00:00:00 2001 From: TowyTowy <85077986+TowyTowy@users.noreply.github.com> Date: Tue, 14 Jul 2026 17:53:45 +0200 Subject: [PATCH 071/131] Escape table names with [square] brackets, refs #2431 (#2846) Several internal helpers quoted table names using SQLite [bracket] identifiers built with an f-string, e.g. PRAGMA foreign_key_list([{table}]). Bracket quoting cannot escape a "]" character, so any table whose name contains "]" (for example "[foo]" or "foo]") produced "sqlite3.OperationalError: unrecognized token" - crashing schema introspection at startup and 500-ing the table page. Switch these call sites to the existing escape_sqlite() helper, which uses "double quote" quoting with correct "" escaping (the same approach already used elsewhere in the codebase and in the test suite): - utils/internal_db.py: PRAGMA foreign_key_list / index_list - utils/__init__.py: get_outbound_foreign_keys - database.py: table_counts count query - facets.py: default "select * from" SQL Added a regression test covering table names with "]" characters. Co-authored-by: Claude --- datasette/database.py | 3 ++- datasette/facets.py | 2 +- datasette/utils/__init__.py | 2 +- datasette/utils/internal_db.py | 8 +++++--- tests/test_api.py | 33 ++++++++++++++++++++++++++++++++- 5 files changed, 41 insertions(+), 7 deletions(-) diff --git a/datasette/database.py b/datasette/database.py index eb402b0c..bab3a378 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -17,6 +17,7 @@ from .utils import ( detect_fts, detect_primary_keys, detect_spatialite, + escape_sqlite, get_all_foreign_keys, get_outbound_foreign_keys, md5_not_usedforsecurity, @@ -608,7 +609,7 @@ class Database: try: table_count = ( await self.execute( - f"select count(*) from (select * from [{table}] limit {self.count_limit + 1})", + f"select count(*) from (select * from {escape_sqlite(table)} limit {self.count_limit + 1})", custom_time_limit=limit, ) ).rows[0][0] diff --git a/datasette/facets.py b/datasette/facets.py index abe0605e..5f757df3 100644 --- a/datasette/facets.py +++ b/datasette/facets.py @@ -85,7 +85,7 @@ class Facet: self.database = database # For foreign key expansion. Can be None for e.g. stored SQL queries: self.table = table - self.sql = sql or f"select * from [{table}]" + self.sql = sql or f"select * from {escape_sqlite(table)}" self.params = params or [] self.table_config = table_config # row_count can be None, in which case we calculate it ourselves: diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index 42574d3b..18d3ba52 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -636,7 +636,7 @@ def detect_primary_keys(conn, table): def get_outbound_foreign_keys(conn, table): - infos = conn.execute(f"PRAGMA foreign_key_list([{table}])").fetchall() + infos = conn.execute(f"PRAGMA foreign_key_list({escape_sqlite(table)})").fetchall() fks = [] for info in infos: if info is not None: diff --git a/datasette/utils/internal_db.py b/datasette/utils/internal_db.py index e061d882..702b53d8 100644 --- a/datasette/utils/internal_db.py +++ b/datasette/utils/internal_db.py @@ -3,7 +3,7 @@ import textwrap from sqlite_utils import Database as SQLiteUtilsDatabase from sqlite_utils import Migrations -from datasette.utils import table_column_details +from datasette.utils import escape_sqlite, table_column_details INTERNAL_DB_SCHEMA_TABLES = { "catalog_databases", @@ -213,7 +213,7 @@ async def populate_schema_tables(internal_db, db, schema_version): for column in columns ) foreign_keys = conn.execute( - f"PRAGMA foreign_key_list([{table_name}])" + f"PRAGMA foreign_key_list({escape_sqlite(table_name)})" ).fetchall() foreign_keys_to_insert.extend( { @@ -222,7 +222,9 @@ async def populate_schema_tables(internal_db, db, schema_version): } for foreign_key in foreign_keys ) - indexes = conn.execute(f"PRAGMA index_list([{table_name}])").fetchall() + indexes = conn.execute( + f"PRAGMA index_list({escape_sqlite(table_name)})" + ).fetchall() indexes_to_insert.extend( { **{"database_name": database_name, "table_name": table_name}, diff --git a/tests/test_api.py b/tests/test_api.py index d5f519b9..191d064a 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -1,6 +1,6 @@ from datasette.app import Datasette from datasette.plugins import DEFAULT_PLUGINS -from datasette.utils import UNSTABLE_API_MESSAGE +from datasette.utils import UNSTABLE_API_MESSAGE, escape_sqlite, tilde_encode from datasette.utils.sqlite import sqlite_version from datasette.version import __version__ from .fixtures import make_app_client, EXPECTED_PLUGINS @@ -930,6 +930,37 @@ async def test_tilde_encoded_database_names(db_name): assert response2.status_code == 200 +@pytest.mark.asyncio +@pytest.mark.parametrize("table_name", ("[foo]", "foo]", "[foo]/bar")) +async def test_table_with_reserved_characters_in_name(table_name): + # Table names containing characters such as "]" that cannot be escaped + # using SQLite [bracket] quoting used to break schema introspection and + # the table page - https://github.com/simonw/datasette/issues/2431 + ds = Datasette() + db = ds.add_memory_database("test_reserved_table_names") + await db.execute_write( + "create table {} (id integer primary key, name text)".format( + escape_sqlite(table_name) + ) + ) + await db.execute_write( + "insert into {} (id, name) values (1, 'one')".format(escape_sqlite(table_name)) + ) + # Schema introspection (populate_schema_tables) must not crash: + db_response = await ds.client.get("/test_reserved_table_names.json") + assert db_response.status_code == 200 + tables = {t["name"]: t for t in db_response.json()["tables"]} + assert tables[table_name]["count"] == 1 + # And the table page itself must load and return the row: + table_response = await ds.client.get( + "/test_reserved_table_names/{}.json?_shape=array".format( + tilde_encode(table_name) + ) + ) + assert table_response.status_code == 200 + assert table_response.json() == [{"id": 1, "name": "one"}] + + @pytest.mark.asyncio @pytest.mark.parametrize( "config,expected", From 8b7c942d5e5ada887aa89c4d58567af39d5a3e07 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 14 Jul 2026 09:18:51 -0700 Subject: [PATCH 072/131] Major performance boost for SQL permissions, closes #2832 --- datasette/utils/actions_sql.py | 217 +++++++++++++++++---------------- 1 file changed, 111 insertions(+), 106 deletions(-) diff --git a/datasette/utils/actions_sql.py b/datasette/utils/actions_sql.py index 67d3ce73..d767e391 100644 --- a/datasette/utils/actions_sql.py +++ b/datasette/utils/actions_sql.py @@ -252,88 +252,62 @@ async def _build_single_action_sql( ] ) - # Continue with the cascading logic - query_parts.extend( - [ - "child_lvl AS (", - " SELECT b.parent, b.child,", - " MAX(CASE WHEN ar.allow = 0 THEN 1 ELSE 0 END) AS any_deny,", - " MAX(CASE WHEN ar.allow = 1 THEN 1 ELSE 0 END) AS any_allow,", - " json_group_array(CASE WHEN ar.allow = 0 THEN ar.source_plugin || ': ' || ar.reason END) AS deny_reasons,", - " json_group_array(CASE WHEN ar.allow = 1 THEN ar.source_plugin || ': ' || ar.reason END) AS allow_reasons", - " FROM base b", - " LEFT JOIN all_rules ar ON ar.parent = b.parent AND ar.child = b.child", - " GROUP BY b.parent, b.child", - "),", - "parent_lvl AS (", - " SELECT b.parent, b.child,", - " MAX(CASE WHEN ar.allow = 0 THEN 1 ELSE 0 END) AS any_deny,", - " MAX(CASE WHEN ar.allow = 1 THEN 1 ELSE 0 END) AS any_allow,", - " json_group_array(CASE WHEN ar.allow = 0 THEN ar.source_plugin || ': ' || ar.reason END) AS deny_reasons,", - " json_group_array(CASE WHEN ar.allow = 1 THEN ar.source_plugin || ': ' || ar.reason END) AS allow_reasons", - " FROM base b", - " LEFT JOIN all_rules ar ON ar.parent = b.parent AND ar.child IS NULL", - " GROUP BY b.parent, b.child", - "),", - "global_lvl AS (", - " SELECT b.parent, b.child,", - " MAX(CASE WHEN ar.allow = 0 THEN 1 ELSE 0 END) AS any_deny,", - " MAX(CASE WHEN ar.allow = 1 THEN 1 ELSE 0 END) AS any_allow,", - " json_group_array(CASE WHEN ar.allow = 0 THEN ar.source_plugin || ': ' || ar.reason END) AS deny_reasons,", - " json_group_array(CASE WHEN ar.allow = 1 THEN ar.source_plugin || ': ' || ar.reason END) AS allow_reasons", - " FROM base b", - " LEFT JOIN all_rules ar ON ar.parent IS NULL AND ar.child IS NULL", - " GROUP BY b.parent, b.child", - "),", + # Continue with the cascading logic. + # Aggregate the RULES by cascade level (small), rather than grouping + # base x rules (which scales with the number of resources). + def _agg(select_key, where, group_by): + parts = [ + f" SELECT {select_key}", + " MAX(CASE WHEN allow = 0 THEN 1 ELSE 0 END) AS any_deny,", + " MAX(CASE WHEN allow = 1 THEN 1 ELSE 0 END) AS any_allow,", + " json_group_array(CASE WHEN allow = 0 THEN source_plugin || ': ' || reason END) AS deny_reasons,", + " json_group_array(CASE WHEN allow = 1 THEN source_plugin || ': ' || reason END) AS allow_reasons", + f" FROM all_rules WHERE {where}", ] + if group_by: + parts.append(f" GROUP BY {group_by}") + return parts + + query_parts.extend( + ["child_agg AS ("] + + _agg( + "parent, child,", + "parent IS NOT NULL AND child IS NOT NULL", + "parent, child", + ) + + ["),", "parent_agg AS ("] + + _agg("parent,", "parent IS NOT NULL AND child IS NULL", "parent") + + ["),", "global_agg AS ("] + + _agg("", "parent IS NULL AND child IS NULL", None) + + ["),"] ) # Add anonymous decision logic if needed if include_is_private: - query_parts.extend( - [ - "anon_child_lvl AS (", - " SELECT b.parent, b.child,", - " MAX(CASE WHEN ar.allow = 0 THEN 1 ELSE 0 END) AS any_deny,", - " MAX(CASE WHEN ar.allow = 1 THEN 1 ELSE 0 END) AS any_allow", - " FROM base b", - " LEFT JOIN anon_rules ar ON ar.parent = b.parent AND ar.child = b.child", - " GROUP BY b.parent, b.child", - "),", - "anon_parent_lvl AS (", - " SELECT b.parent, b.child,", - " MAX(CASE WHEN ar.allow = 0 THEN 1 ELSE 0 END) AS any_deny,", - " MAX(CASE WHEN ar.allow = 1 THEN 1 ELSE 0 END) AS any_allow", - " FROM base b", - " LEFT JOIN anon_rules ar ON ar.parent = b.parent AND ar.child IS NULL", - " GROUP BY b.parent, b.child", - "),", - "anon_global_lvl AS (", - " SELECT b.parent, b.child,", - " MAX(CASE WHEN ar.allow = 0 THEN 1 ELSE 0 END) AS any_deny,", - " MAX(CASE WHEN ar.allow = 1 THEN 1 ELSE 0 END) AS any_allow", - " FROM base b", - " LEFT JOIN anon_rules ar ON ar.parent IS NULL AND ar.child IS NULL", - " GROUP BY b.parent, b.child", - "),", - "anon_decisions AS (", - " SELECT", - " b.parent, b.child,", - " CASE", - " WHEN acl.any_deny = 1 THEN 0", - " WHEN acl.any_allow = 1 THEN 1", - " WHEN apl.any_deny = 1 THEN 0", - " WHEN apl.any_allow = 1 THEN 1", - " WHEN agl.any_deny = 1 THEN 0", - " WHEN agl.any_allow = 1 THEN 1", - " ELSE 0", - " END AS anon_is_allowed", - " FROM base b", - " JOIN anon_child_lvl acl ON b.parent = acl.parent AND (b.child = acl.child OR (b.child IS NULL AND acl.child IS NULL))", - " JOIN anon_parent_lvl apl ON b.parent = apl.parent AND (b.child = apl.child OR (b.child IS NULL AND apl.child IS NULL))", - " JOIN anon_global_lvl agl ON b.parent = agl.parent AND (b.child = agl.child OR (b.child IS NULL AND agl.child IS NULL))", - "),", + + def _anon_agg(select_key, where, group_by): + parts = [ + f" SELECT {select_key}", + " MAX(CASE WHEN allow = 0 THEN 1 ELSE 0 END) AS any_deny,", + " MAX(CASE WHEN allow = 1 THEN 1 ELSE 0 END) AS any_allow", + f" FROM anon_rules WHERE {where}", ] + if group_by: + parts.append(f" GROUP BY {group_by}") + return parts + + query_parts.extend( + ["anon_child_agg AS ("] + + _anon_agg( + "parent, child,", + "parent IS NOT NULL AND child IS NOT NULL", + "parent, child", + ) + + ["),", "anon_parent_agg AS ("] + + _anon_agg("parent,", "parent IS NOT NULL AND child IS NULL", "parent") + + ["),", "anon_global_agg AS ("] + + _anon_agg("", "parent IS NULL AND child IS NULL", None) + + ["),"] ) # Final decisions @@ -342,31 +316,28 @@ async def _build_single_action_sql( "decisions AS (", " SELECT", " b.parent, b.child,", - " -- Cascading permission logic: child → parent → global, DENY beats ALLOW at each level", + " -- Cascading permission logic: child -> parent -> global, DENY beats ALLOW at each level", " -- Priority order:", - " -- 1. Child-level deny (most specific, blocks access)", - " -- 2. Child-level allow (most specific, grants access)", - " -- 3. Parent-level deny (intermediate, blocks access)", - " -- 4. Parent-level allow (intermediate, grants access)", - " -- 5. Global-level deny (least specific, blocks access)", - " -- 6. Global-level allow (least specific, grants access)", + " -- 1. Child-level deny 2. Child-level allow", + " -- 3. Parent-level deny 4. Parent-level allow", + " -- 5. Global-level deny 6. Global-level allow", " -- 7. Default deny (no rules match)", " CASE", - " WHEN cl.any_deny = 1 THEN 0", - " WHEN cl.any_allow = 1 THEN 1", - " WHEN pl.any_deny = 1 THEN 0", - " WHEN pl.any_allow = 1 THEN 1", - " WHEN gl.any_deny = 1 THEN 0", - " WHEN gl.any_allow = 1 THEN 1", + " WHEN ca.any_deny = 1 THEN 0", + " WHEN ca.any_allow = 1 THEN 1", + " WHEN pa.any_deny = 1 THEN 0", + " WHEN pa.any_allow = 1 THEN 1", + " WHEN ga.any_deny = 1 THEN 0", + " WHEN ga.any_allow = 1 THEN 1", " ELSE 0", " END AS is_allowed,", " CASE", - " WHEN cl.any_deny = 1 THEN cl.deny_reasons", - " WHEN cl.any_allow = 1 THEN cl.allow_reasons", - " WHEN pl.any_deny = 1 THEN pl.deny_reasons", - " WHEN pl.any_allow = 1 THEN pl.allow_reasons", - " WHEN gl.any_deny = 1 THEN gl.deny_reasons", - " WHEN gl.any_allow = 1 THEN gl.allow_reasons", + " WHEN ca.any_deny = 1 THEN ca.deny_reasons", + " WHEN ca.any_allow = 1 THEN ca.allow_reasons", + " WHEN pa.any_deny = 1 THEN pa.deny_reasons", + " WHEN pa.any_allow = 1 THEN pa.allow_reasons", + " WHEN ga.any_deny = 1 THEN ga.deny_reasons", + " WHEN ga.any_allow = 1 THEN ga.allow_reasons", " ELSE '[]'", " END AS reason", ] @@ -374,21 +345,34 @@ async def _build_single_action_sql( if include_is_private: query_parts.append( - " , CASE WHEN ad.anon_is_allowed = 0 THEN 1 ELSE 0 END AS is_private" + " , CASE WHEN (" + "CASE" + " WHEN aca.any_deny = 1 THEN 0" + " WHEN aca.any_allow = 1 THEN 1" + " WHEN apa.any_deny = 1 THEN 0" + " WHEN apa.any_allow = 1 THEN 1" + " WHEN aga.any_deny = 1 THEN 0" + " WHEN aga.any_allow = 1 THEN 1" + " ELSE 0 END" + ") = 0 THEN 1 ELSE 0 END AS is_private" ) query_parts.extend( [ " FROM base b", - " JOIN child_lvl cl ON b.parent = cl.parent AND (b.child = cl.child OR (b.child IS NULL AND cl.child IS NULL))", - " JOIN parent_lvl pl ON b.parent = pl.parent AND (b.child = pl.child OR (b.child IS NULL AND pl.child IS NULL))", - " JOIN global_lvl gl ON b.parent = gl.parent AND (b.child = gl.child OR (b.child IS NULL AND gl.child IS NULL))", + " LEFT JOIN child_agg ca ON ca.parent = b.parent AND ca.child = b.child", + " LEFT JOIN parent_agg pa ON pa.parent = b.parent", + " CROSS JOIN global_agg ga", ] ) if include_is_private: - query_parts.append( - " JOIN anon_decisions ad ON b.parent = ad.parent AND (b.child = ad.child OR (b.child IS NULL AND ad.child IS NULL))" + query_parts.extend( + [ + " LEFT JOIN anon_child_agg aca ON aca.parent = b.parent AND aca.child = b.child", + " LEFT JOIN anon_parent_agg apa ON apa.parent = b.parent", + " CROSS JOIN anon_global_agg aga", + ] ) query_parts.append(")") @@ -400,8 +384,28 @@ async def _build_single_action_sql( restriction_intersect = "\nINTERSECT\n".join( f"SELECT * FROM ({sql})" for sql in restriction_sqls ) + # Decompose by NULL-pattern so the final filter can use pure-equality + # EXISTS lookups (satisfiable via automatic indexes) instead of a + # correlated OR-scan over the whole list. query_parts.extend( - [",", "restriction_list AS (", f" {restriction_intersect}", ")"] + [ + ",", + "restriction_list AS (", + f" {restriction_intersect}", + "),", + "restriction_exact AS (", + " SELECT parent, child FROM restriction_list WHERE parent IS NOT NULL AND child IS NOT NULL", + "),", + "restriction_parent_any AS (", + " SELECT DISTINCT parent FROM restriction_list WHERE parent IS NOT NULL AND child IS NULL", + "),", + "restriction_child_any AS (", + " SELECT DISTINCT child FROM restriction_list WHERE parent IS NULL AND child IS NOT NULL", + "),", + "restriction_all AS (", + " SELECT 1 AS matched FROM restriction_list WHERE parent IS NULL AND child IS NULL LIMIT 1", + ")", + ] ) # Final SELECT @@ -416,10 +420,11 @@ async def _build_single_action_sql( # Add restriction filter if there are restrictions if restriction_sqls: query_parts.append(""" - AND EXISTS ( - SELECT 1 FROM restriction_list r - WHERE (r.parent = decisions.parent OR r.parent IS NULL) - AND (r.child = decisions.child OR r.child IS NULL) + AND ( + EXISTS (SELECT 1 FROM restriction_all) + OR EXISTS (SELECT 1 FROM restriction_parent_any r WHERE r.parent = decisions.parent) + OR EXISTS (SELECT 1 FROM restriction_child_any r WHERE r.child = decisions.child) + OR EXISTS (SELECT 1 FROM restriction_exact r WHERE r.parent = decisions.parent AND r.child = decisions.child) )""") # Add parent filter if specified From 2ffd8a860e84ff58922e633c8e85e9a8e088ca93 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 14 Jul 2026 09:28:29 -0700 Subject: [PATCH 073/131] Release 1.0a37 Refs #2831, #2832, #2841, #2842, #2843, #2846 --- datasette/version.py | 2 +- docs/changelog.rst | 14 ++++++++++++++ 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/datasette/version.py b/datasette/version.py index 387144e9..8e238ab5 100644 --- a/datasette/version.py +++ b/datasette/version.py @@ -1,2 +1,2 @@ -__version__ = "1.0a36" +__version__ = "1.0a37" __version_info__ = tuple(__version__.split(".")) diff --git a/docs/changelog.rst b/docs/changelog.rst index e3718543..1327baa6 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -4,6 +4,20 @@ Changelog ========= +.. _v1_0_a37: + +1.0a37 (2026-07-14) +------------------- + +Performance improvement for SQL-backed permission checks, plus an improved permission debugging interface. + +- SQL used to resolve permission checks now aggregates permission rules before joining them to resources, improving performance on instances with large schemas. (:issue:`2832`) +- The :ref:`PermissionCheckView` permission debugger now explains why a decision was allowed or denied, including the matching rules. The interactive form can also test a hypothetical actor supplied as JSON, and the :ref:`permissions documentation ` now describes resolution rules in more detail. (:issue:`2841`) +- :ref:`database_execute_write` has a new ``transaction=`` parameter, which can be set to ``False`` for statements such as ``VACUUM`` that cannot run inside a transaction. Write tasks now start their transactions using ``BEGIN IMMEDIATE``, which also ensures that writes are rolled back if the task fails. (:issue:`2831`) +- Refreshing a database's schema in Datasette's internal catalog is now performed as a single atomic operation. (:issue:`2831`) +- Fixed schema introspection, table pages, facets and table counts for tables with names containing a ``]`` character. Thanks, `TowyTowy `__. (:issue:`2431`, :pr:`2846`) +- ``/-/plugins.json`` once again returns a top-level JSON array of plugin objects, reverting the object envelope introduced in 1.0a36. This should fix a large number of trivial test failures in existing plugins. (:issue:`2842`, :pr:`2843`) + .. _v1_0_a36: 1.0a36 (2026-07-07) From 481df7ff6d78a8ccf919984d27f27201b081bd53 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 14 Jul 2026 09:31:28 -0700 Subject: [PATCH 074/131] Shorten link text in changelog --- docs/changelog.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index 1327baa6..670166bb 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -13,7 +13,7 @@ Performance improvement for SQL-backed permission checks, plus an improved permi - SQL used to resolve permission checks now aggregates permission rules before joining them to resources, improving performance on instances with large schemas. (:issue:`2832`) - The :ref:`PermissionCheckView` permission debugger now explains why a decision was allowed or denied, including the matching rules. The interactive form can also test a hypothetical actor supplied as JSON, and the :ref:`permissions documentation ` now describes resolution rules in more detail. (:issue:`2841`) -- :ref:`database_execute_write` has a new ``transaction=`` parameter, which can be set to ``False`` for statements such as ``VACUUM`` that cannot run inside a transaction. Write tasks now start their transactions using ``BEGIN IMMEDIATE``, which also ensures that writes are rolled back if the task fails. (:issue:`2831`) +- :ref:`db.execute_write(sql, ..., transaction=True) ` has a new ``transaction=`` parameter, which can be set to ``False`` for statements such as ``VACUUM`` that cannot run inside a transaction. Write tasks now start their transactions using ``BEGIN IMMEDIATE``, which also ensures that writes are rolled back if the task fails. (:issue:`2831`) - Refreshing a database's schema in Datasette's internal catalog is now performed as a single atomic operation. (:issue:`2831`) - Fixed schema introspection, table pages, facets and table counts for tables with names containing a ``]`` character. Thanks, `TowyTowy `__. (:issue:`2431`, :pr:`2846`) - ``/-/plugins.json`` once again returns a top-level JSON array of plugin objects, reverting the object envelope introduced in 1.0a36. This should fix a large number of trivial test failures in existing plugins. (:issue:`2842`, :pr:`2843`) From e889403d3bbe143854262682161c98a57bdb6594 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sat, 25 Jul 2026 15:47:08 -0700 Subject: [PATCH 075/131] Upgrade to ruff>=0.16.0 (#2857) * ruff>=0.16.0 See https://astral.sh/blog/ruff-v0.16.0 * uv run ruff check . --fix --unsafe-fixes * Ruff fixes by Claude Code Opus 5 --- datasette/_pytest_plugin.py | 3 +- datasette/actor_auth_cookie.py | 8 +- datasette/app.py | 354 +++++++++--------- datasette/blob_renderer.py | 7 +- datasette/cli.py | 99 +++-- datasette/column_types.py | 4 +- datasette/csrf.py | 17 +- datasette/database.py | 68 ++-- datasette/default_actions.py | 2 +- datasette/default_magic_parameters.py | 3 +- datasette/default_permissions/__init__.py | 31 +- datasette/default_permissions/config.py | 38 +- datasette/default_permissions/defaults.py | 25 +- datasette/default_permissions/helpers.py | 20 +- datasette/default_permissions/restrictions.py | 24 +- datasette/default_permissions/root.py | 8 +- datasette/default_permissions/tokens.py | 8 +- datasette/default_table_actions.py | 2 +- datasette/events.py | 3 +- datasette/facets.py | 46 +-- datasette/filters.py | 23 +- datasette/fixtures.py | 9 +- datasette/forbidden.py | 3 +- datasette/handle_exception.py | 17 +- datasette/hookspecs.py | 3 +- datasette/inspect.py | 8 +- datasette/jump.py | 2 +- datasette/permissions.py | 7 +- datasette/plugins.py | 20 +- datasette/publish/cloudrun.py | 28 +- datasette/publish/common.py | 10 +- datasette/publish/heroku.py | 14 +- datasette/renderer.py | 7 +- datasette/stored_queries.py | 17 +- datasette/tokens.py | 31 +- datasette/tracer.py | 19 +- datasette/url_builder.py | 11 +- datasette/utils/__init__.py | 219 ++++++++--- datasette/utils/asgi.py | 45 +-- datasette/utils/baseconv.py | 2 +- datasette/utils/check_callable.py | 6 +- datasette/utils/internal_db.py | 11 +- datasette/utils/multipart.py | 82 ++-- datasette/utils/permissions.py | 31 +- datasette/utils/shutil_backport.py | 2 +- datasette/utils/sql_analysis.py | 8 +- datasette/utils/sqlite.py | 8 +- datasette/utils/testing.py | 5 +- datasette/views/__init__.py | 12 +- datasette/views/base.py | 32 +- datasette/views/database.py | 75 ++-- datasette/views/execute_write.py | 50 +-- datasette/views/index.py | 15 +- datasette/views/query_helpers.py | 37 +- datasette/views/row.py | 71 ++-- datasette/views/special.py | 37 +- datasette/views/stored_queries.py | 18 +- datasette/views/table.py | 204 ++++------ datasette/views/table_create_alter.py | 95 +++-- datasette/views/table_extras.py | 153 +++++--- docs/conf.py | 2 - docs/json_api_doc.py | 12 +- docs/metadata_doc.py | 11 +- docs/template_context_doc.py | 14 +- pyproject.toml | 6 +- ruff.toml | 7 +- tests/conftest.py | 26 +- tests/fixtures.py | 18 +- tests/plugins/my_plugin.py | 24 +- tests/plugins/my_plugin_2.py | 20 +- tests/plugins/register_output_renderer.py | 7 +- tests/plugins/sleep_sql_function.py | 3 +- tests/test_actions_sql.py | 5 +- tests/test_actor_restriction_bug.py | 1 + tests/test_allowed_many.py | 7 +- tests/test_allowed_resources.py | 3 +- tests/test_api.py | 31 +- tests/test_api_write.py | 77 ++-- tests/test_auth.py | 23 +- tests/test_base_view.py | 8 +- tests/test_cli.py | 39 +- tests/test_cli_serve_get.py | 12 +- tests/test_cli_serve_server.py | 3 +- tests/test_column_types.py | 12 +- tests/test_config_dir.py | 4 +- tests/test_crossdb.py | 10 +- tests/test_csrf_middleware.py | 2 +- tests/test_csv.py | 8 +- tests/test_custom_pages.py | 2 + tests/test_default_deny.py | 1 + tests/test_docs.py | 12 +- tests/test_docs_plugins.py | 5 +- tests/test_error_shape.py | 14 +- tests/test_extras.py | 9 +- tests/test_facets.py | 13 +- tests/test_filters.py | 5 +- tests/test_html.py | 46 +-- tests/test_internal_db.py | 32 +- tests/test_internals_database.py | 39 +- tests/test_internals_datasette.py | 22 +- tests/test_internals_datasette_client.py | 3 +- tests/test_internals_request.py | 4 +- tests/test_internals_response.py | 4 +- tests/test_internals_urls.py | 3 +- tests/test_label_column_for_table.py | 3 +- tests/test_load_extensions.py | 6 +- tests/test_messages.py | 3 +- tests/test_multipart.py | 4 +- tests/test_package.py | 8 +- tests/test_permission_endpoints.py | 3 +- tests/test_permissions.py | 29 +- tests/test_plugins.py | 62 +-- tests/test_publish_cloudrun.py | 24 +- tests/test_publish_heroku.py | 8 +- tests/test_pytest_autoclose_plugin.py | 1 + tests/test_queries.py | 48 +-- tests/test_restriction_sql.py | 1 + tests/test_routes.py | 5 +- tests/test_schema_endpoints.py | 1 + tests/test_search_tables.py | 1 + tests/test_spatialite.py | 8 +- tests/test_stored_queries.py | 8 +- tests/test_success_envelope.py | 3 +- tests/test_table_api.py | 11 +- tests/test_table_html.py | 50 +-- tests/test_template_context.py | 41 +- tests/test_token_handler.py | 5 +- tests/test_tracer.py | 8 +- tests/test_utils.py | 35 +- tests/test_utils_check_callable.py | 3 +- tests/test_utils_permissions.py | 11 +- tests/test_utils_sql_analysis.py | 2 +- tests/test_write_wrapper.py | 16 +- 133 files changed, 1656 insertions(+), 1578 deletions(-) diff --git a/datasette/_pytest_plugin.py b/datasette/_pytest_plugin.py index 103c616d..587380ed 100644 --- a/datasette/_pytest_plugin.py +++ b/datasette/_pytest_plugin.py @@ -89,7 +89,8 @@ def pytest_runtest_protocol(item, nextitem): continue try: ds.close() - except Exception as e: + except Exception as e: # noqa: BLE001 + # Surfaced as a pytest warning; teardown must not fail the run item.warn( pytest.PytestUnraisableExceptionWarning( f"Error closing Datasette instance: {e!r}" diff --git a/datasette/actor_auth_cookie.py b/datasette/actor_auth_cookie.py index 368213af..7503f1d5 100644 --- a/datasette/actor_auth_cookie.py +++ b/datasette/actor_auth_cookie.py @@ -1,8 +1,10 @@ -from datasette import hookimpl -from itsdangerous import BadSignature -from datasette.utils import baseconv import time +from itsdangerous import BadSignature + +from datasette import hookimpl +from datasette.utils import baseconv + @hookimpl def actor_from_request(datasette, request): diff --git a/datasette/app.py b/datasette/app.py index 0e31273d..c82ea075 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -2,7 +2,8 @@ from __future__ import annotations import asyncio import contextvars -from typing import TYPE_CHECKING, Any, Dict, Iterable, List, Sequence +from collections.abc import Iterable, Sequence +from typing import TYPE_CHECKING, Any if TYPE_CHECKING: from datasette.permissions import Resource @@ -12,11 +13,10 @@ import dataclasses import datetime import functools import glob -import httpx import importlib.metadata import inspect -from itsdangerous import BadSignature import json +import logging import os import re import secrets @@ -28,90 +28,41 @@ import urllib.parse from concurrent import futures from pathlib import Path -from markupsafe import Markup, escape -from itsdangerous import URLSafeSerializer +import httpx +from itsdangerous import BadSignature, URLSafeSerializer from jinja2 import ( ChoiceLoader, Environment, FileSystemLoader, - pass_context, PrefixLoader, + pass_context, ) from jinja2.environment import Template from jinja2.exceptions import TemplateNotFound +from markupsafe import Markup, escape -from .events import Event -from .column_types import SQLiteType from . import stored_queries, write_sql -from .views import Context -from .views.database import ( - database_download, - DatabaseView, - QueryView, -) -from .views.table_create_alter import ( - DatabaseForeignKeyTargetsView, - TableAlterView, - TableCreateView, - TableForeignKeySuggestionsView, -) -from .views.execute_write import ExecuteWriteAnalyzeView, ExecuteWriteView -from .views.stored_queries import ( - QueryCreateAnalyzeView, - QueryDeleteView, - QueryDefinitionView, - QueryEditView, - GlobalQueryListView, - QueryListView, - QueryParametersView, - QueryStoreView, - QueryUpdateView, -) -from .views.index import IndexView -from .views.special import ( - JsonDataView, - PatternPortfolioView, - AutocompleteDebugView, - AuthTokenView, - ApiExplorerView, - CreateTokenView, - LogoutView, - AllowDebugView, - PermissionsDebugView, - MessagesDebugView, - AllowedResourcesView, - PermissionRulesView, - PermissionCheckView, - JumpView, - InstanceSchemaView, - DatabaseSchemaView, - TableSchemaView, -) -from .views.table import ( - TableAutocompleteView, - TableInsertView, - TableUpsertView, - TableSetColumnTypeView, - TableDropView, - TableFragmentView, - table_view, -) -from .views.row import RowView, RowDeleteView, RowUpdateView -from .renderer import json_renderer -from .url_builder import Urls +from .column_types import SQLiteType +from .csrf import CrossOriginProtectionMiddleware from .database import Database, QueryInterrupted - +from .events import Event +from .plugins import DEFAULT_PLUGINS, get_plugins, pm +from .renderer import json_renderer +from .resources import DatabaseResource, TableResource +from .tokens import TokenInvalid +from .tracer import AsgiTracer +from .url_builder import Urls from .utils import ( + SPATIALITE_FUNCTIONS, PaginatedResources, PrefixedUrlString, - SPATIALITE_FUNCTIONS, StartupError, + add_cors_headers, async_call_with_supported_arguments, await_me_maybe, baseconv, call_with_supported_arguments, detect_json1, - add_cors_headers, display_actor, escape_css_string, escape_sqlite, @@ -121,47 +72,97 @@ from .utils import ( move_plugins_and_allow, move_table_config, parse_metadata, + redact_keys, resolve_env_secrets, resolve_routes, + row_sql_params_pks, sha256_file, tilde_decode, tilde_encode, to_css_class, urlsafe_components, - redact_keys, - row_sql_params_pks, ) -from .tokens import TokenInvalid from .utils.asgi import ( AsgiLifespan, + AsgiRunOnFirstRequest, BadRequest, + DatabaseNotFound, Forbidden, NotFound, - DatabaseNotFound, - TableNotFound, - RowNotFound, Request, Response, - AsgiRunOnFirstRequest, - asgi_static, + RowNotFound, + TableNotFound, asgi_send, asgi_send_file, asgi_send_redirect, + asgi_static, ) -from .csrf import CrossOriginProtectionMiddleware from .utils.internal_db import init_internal_db, populate_schema_tables from .utils.sqlite import ( sqlite3, using_pysqlite3, ) -from .tracer import AsgiTracer -from .plugins import pm, DEFAULT_PLUGINS, get_plugins from .version import __version__ - -from .resources import DatabaseResource, TableResource +from .views import Context +from .views.database import ( + DatabaseView, + QueryView, + database_download, +) +from .views.execute_write import ExecuteWriteAnalyzeView, ExecuteWriteView +from .views.index import IndexView +from .views.row import RowDeleteView, RowUpdateView, RowView +from .views.special import ( + AllowDebugView, + AllowedResourcesView, + ApiExplorerView, + AuthTokenView, + AutocompleteDebugView, + CreateTokenView, + DatabaseSchemaView, + InstanceSchemaView, + JsonDataView, + JumpView, + LogoutView, + MessagesDebugView, + PatternPortfolioView, + PermissionCheckView, + PermissionRulesView, + PermissionsDebugView, + TableSchemaView, +) +from .views.stored_queries import ( + GlobalQueryListView, + QueryCreateAnalyzeView, + QueryDefinitionView, + QueryDeleteView, + QueryEditView, + QueryListView, + QueryParametersView, + QueryStoreView, + QueryUpdateView, +) +from .views.table import ( + TableAutocompleteView, + TableDropView, + TableFragmentView, + TableInsertView, + TableSetColumnTypeView, + TableUpsertView, + table_view, +) +from .views.table_create_alter import ( + DatabaseForeignKeyTargetsView, + TableAlterView, + TableCreateView, + TableForeignKeySuggestionsView, +) app_root = Path(__file__).parent.parent +logger = logging.getLogger(__name__) + # Context variable to track when code is executing within a datasette.client request _in_datasette_client = contextvars.ContextVar("in_datasette_client", default=False) @@ -184,7 +185,7 @@ class PermissionCheck: """Represents a logged permission check for debugging purposes.""" when: str - actor: Dict[str, Any] | None + actor: dict[str, Any] | None action: str parent: str | None child: str | None @@ -434,7 +435,7 @@ class Datasette: if config_dir: db_files = [] for ext in ("db", "sqlite", "sqlite3"): - db_files.extend(config_dir.glob("*.{}".format(ext))) + db_files.extend(config_dir.glob(f"*.{ext}")) self.files += tuple(str(f) for f in db_files) if ( config_dir @@ -675,10 +676,10 @@ class Datasette: def get_jinja_environment(self, request: Request = None) -> Environment: environment = self._jinja_env if request: - for environment in pm.hook.jinja2_environment_from_request( + for hook_environment in pm.hook.jinja2_environment_from_request( datasette=self, request=request, env=environment ): - pass + environment = hook_environment return environment def get_action(self, name_or_abbr: str): @@ -732,7 +733,7 @@ class Datasette: catalog_database_names.update( row["database_name"] for row in await internal_db.execute( - "select distinct database_name from {}".format(table) + f"select distinct database_name from {table}" ) if row["database_name"] is not None ) @@ -743,7 +744,7 @@ class Datasette: for stale_db_name in stale_databases: for table in catalog_table_names: conn.execute( - "DELETE FROM {} WHERE database_name = ?".format(table), + f"DELETE FROM {table} WHERE database_name = ?", [stale_db_name], ) @@ -792,17 +793,13 @@ class Datasette: action.name in action_names and action != action_names[action.name] ): - raise StartupError( - "Duplicate action name: {}".format(action.name) - ) + raise StartupError(f"Duplicate action name: {action.name}") if ( action.abbr and action.abbr in action_abbrs and action != action_abbrs[action.abbr] ): - raise StartupError( - "Duplicate action abbr: {}".format(action.abbr) - ) + raise StartupError(f"Duplicate action abbr: {action.abbr}") action_names[action.name] = action if action.abbr: action_abbrs[action.abbr] = action @@ -861,7 +858,7 @@ class Datasette: actor_id: str, *, expires_after: int | None = None, - restrictions: "TokenRestrictions | None" = None, + restrictions: TokenRestrictions | None = None, handler: str | None = None, ) -> str: """ @@ -918,7 +915,7 @@ class Datasette: raise KeyError return matches[0] if name is None: - name = [key for key in self.databases.keys()][0] + name = next(iter(self.databases.keys())) return self.databases[name] def add_database(self, db, name=None, route=None): @@ -931,7 +928,7 @@ class Datasette: suggestion = name i = 2 while name in self.databases: - name = "{}_{}".format(suggestion, i) + name = f"{suggestion}_{i}" i += 1 db.name = name db.route = route or name @@ -966,13 +963,14 @@ class Datasette: for db in dbs: try: db.close() - except Exception as e: + except Exception as e: # noqa: BLE001 + # Collect the first failure and re-raise after every close() has run if first_exception is None: first_exception = e if self.executor is not None: try: self.executor.shutdown(wait=True, cancel_futures=True) - except Exception as e: + except Exception as e: # noqa: BLE001 if first_exception is None: first_exception = e if first_exception is not None: @@ -1321,24 +1319,15 @@ class Datasette: actual = ( actual_sqlite_type.value if actual_sqlite_type is not None - else "unrecognized {!r}".format(column_detail.type) + else f"unrecognized {column_detail.type!r}" ) raise ValueError( - "Column type {!r} is only applicable to SQLite types {} but {}.{}.{} " - "has SQLite type {}".format( - ct_cls.name, - allowed, - database, - resource, - column, - actual, - ) + f"Column type {ct_cls.name!r} is only applicable to SQLite types {allowed} but {database}.{resource}.{column} " + f"has SQLite type {actual}" ) async def _apply_column_types_config(self): """Load column_types from datasette.json config into the internal DB.""" - import logging - for db_name, db_conf in (self.config or {}).get("databases", {}).items(): for table_name, table_conf in db_conf.get("tables", {}).items(): for col_name, ct in table_conf.get("column_types", {}).items(): @@ -1348,7 +1337,7 @@ class Datasette: col_type = ct["type"] config = ct.get("config") if col_type not in self._column_types: - logging.warning( + logger.warning( "column_types config references unknown type %r " "for %s.%s.%s", col_type, @@ -1361,7 +1350,7 @@ class Datasette: db_name, table_name, col_name, col_type, config ) except ValueError as ex: - logging.warning(str(ex)) + logger.warning(str(ex)) async def get_column_type(self, database: str, resource: str, column: str): """ @@ -1414,7 +1403,7 @@ class Datasette: resource: str, column: str, column_type: str, - config: dict = None, + config: dict | None = None, ) -> None: """Assign a column type. Overwrites any existing assignment.""" ct_cls = self._column_types.get(column_type) @@ -1498,9 +1487,7 @@ class Datasette: possible_names = {plugin["name"], plugin["name"].replace("-", "_")} if plugin_name in possible_names: return _resolve_static_asset_path(plugin["static_path"], path) - raise FileNotFoundError( - "No static assets found for plugin {}".format(plugin_name) - ) + raise FileNotFoundError(f"No static assets found for plugin {plugin_name}") def _static_mounted_asset(self, mount_name, path): mount_name = mount_name.strip("/") @@ -1510,7 +1497,7 @@ class Datasette: _resolve_static_asset_path(dirname, path), self.urls.path("/{}/{}".format(mount_name, path.lstrip("/"))), ) - raise FileNotFoundError("No static mount found for {}".format(mount_name)) + raise FileNotFoundError(f"No static mount found for {mount_name}") def _static_asset_hash(self, filepath): filepath = Path(filepath) @@ -1601,18 +1588,17 @@ class Datasette: if await self.allowed(action="view-instance", actor=actor): crumbs.append({"href": self.urls.instance(), "label": "home"}) # Database link - if database: - if await self.allowed( - action="view-database", - resource=DatabaseResource(database=database), - actor=actor, - ): - crumbs.append( - { - "href": self.urls.database(database), - "label": database, - } - ) + if database and await self.allowed( + action="view-database", + resource=DatabaseResource(database=database), + actor=actor, + ): + crumbs.append( + { + "href": self.urls.database(database), + "label": database, + } + ) # Table link if table: assert database, "table= requires database=" @@ -1631,7 +1617,7 @@ class Datasette: async def actors_from_ids( self, actor_ids: Iterable[str | int] - ) -> Dict[int | str, Dict]: + ) -> dict[int | str, dict]: result = pm.hook.actors_from_ids(datasette=self, actor_ids=actor_ids) if result is None: # Do the default thing @@ -1640,9 +1626,9 @@ class Datasette: return result async def track_event(self, event: Event): - assert isinstance(event, self.event_classes), "Invalid event type: {}".format( - type(event) - ) + assert isinstance( + event, self.event_classes + ), f"Invalid event type: {type(event)}" for hook in pm.hook.track_event(datasette=self, event=event): await await_me_maybe(hook) @@ -1679,7 +1665,7 @@ class Datasette: self, actor: dict, action: str, - resource: "Resource" | None = None, + resource: Resource | None = None, ): """ Check if actor can see a resource and if it's private. @@ -1878,10 +1864,7 @@ class Datasette: if truncated and resources: last_resource = resources[-1] # Use tilde-encoding like table pagination - next_token = "{},{}".format( - tilde_encode(str(last_resource.parent)), - tilde_encode(str(last_resource.child)), - ) + next_token = f"{tilde_encode(str(last_resource.parent))},{tilde_encode(str(last_resource.child))}" return PaginatedResources( resources=resources, @@ -1899,7 +1882,7 @@ class Datasette: self, *, action: str, - resource: "Resource" = None, + resource: Resource = None, actor: dict | None = None, ) -> bool: """ @@ -1930,7 +1913,7 @@ class Datasette: self, *, actions: Sequence[str], - resource: "Resource" = None, + resource: Resource = None, actor: dict | None = None, ) -> dict[str, bool]: """ @@ -1951,11 +1934,11 @@ class Datasette: ) # {"edit-schema": True, "drop-table": True, "insert-row": False} """ - from datasette.utils.actions_sql import check_permissions_for_actions from datasette.permissions import ( _permission_check_cache, _skip_permission_checks, ) + from datasette.utils.actions_sql import check_permissions_for_actions # For global actions, resource is None parent = resource.parent if resource else None @@ -2044,7 +2027,7 @@ class Datasette: self, *, action: str, - resource: "Resource" = None, + resource: Resource = None, actor: dict | None = None, ): """ @@ -2098,13 +2081,15 @@ class Datasette: db = self.databases[database] foreign_keys = await db.foreign_keys_for_table(table) # Find the foreign_key for this column - try: - fk = [ + fk = next( + ( foreign_key for foreign_key in foreign_keys if foreign_key["column"] == column - ][0] - except IndexError: + ), + None, + ) + if fk is None: return {} # Ensure user has permission to view the referenced table from datasette.resources import TableResource @@ -2192,16 +2177,17 @@ class Datasette: sqlite_extensions[extension] = result.fetchone()[0] else: sqlite_extensions[extension] = None - except Exception: + except Exception: # noqa: BLE001, S110 + # Probing for optional SQLite extensions - absence is the normal case pass # More details on SpatiaLite if "spatialite" in sqlite_extensions: spatialite_details = {} for fn in SPATIALITE_FUNCTIONS: try: - result = conn.execute("select {}()".format(fn)) + result = conn.execute(f"select {fn}()") spatialite_details[fn] = result.fetchone()[0] - except Exception as e: + except sqlite3.Error as e: spatialite_details[fn] = {"error": str(e)} sqlite_extensions["spatialite"] = spatialite_details @@ -2209,9 +2195,7 @@ class Datasette: fts_versions = [] for fts in ("FTS5", "FTS4", "FTS3"): try: - conn.execute( - "CREATE VIRTUAL TABLE v{fts} USING {fts} (data)".format(fts=fts) - ) + conn.execute(f"CREATE VIRTUAL TABLE v{fts} USING {fts} (data)") fts_versions.append(fts) except sqlite3.OperationalError: continue @@ -2270,7 +2254,7 @@ class Datasette: "static": p["static_path"] is not None, "templates": p["templates_path"] is not None, "version": p.get("version"), - "hooks": list(sorted(set(p["hooks"]))), + "hooks": sorted(set(p["hooks"])), } for p in ps ] @@ -2346,13 +2330,15 @@ class Datasette: async def render_template( self, - templates: List[str] | str | Template, - context: Dict[str, Any] | Context | None = None, + templates: list[str] | str | Template, + context: dict[str, Any] | Context | None = None, request: Request | None = None, view_name: str | None = None, ): if not self._startup_invoked: - raise Exception("render_template() called before await ds.invoke_startup()") + raise RuntimeError( + "render_template() called before await ds.invoke_startup()" + ) context = context or {} if isinstance(templates, Template): template = templates @@ -2398,9 +2384,9 @@ class Datasette: datasette=self, ): extra_vars = await await_me_maybe(extra_vars) - assert isinstance(extra_vars, dict), "extra_vars is of type {}".format( - type(extra_vars) - ) + assert isinstance( + extra_vars, dict + ), f"extra_vars is of type {type(extra_vars)}" extra_template_vars.update(extra_vars) async def menu_links(): @@ -2419,29 +2405,27 @@ class Datasette: # the contract tests fail otherwise template_context = { **context, - **{ - "request": request, - "crumb_items": self._crumb_items, - "urls": self.urls, - "actor": request.actor if request else None, - "menu_links": menu_links, - "display_actor": display_actor, - "show_logout": request is not None - and "ds_actor" in request.cookies - and request.actor, - "zip": zip, - "body_scripts": body_scripts, - "format_bytes": format_bytes, - "show_messages": lambda: self._show_messages(request), - "extra_css_urls": await self._asset_urls( - "extra_css_urls", template, context, request, view_name - ), - "extra_js_urls": await self._asset_urls( - "extra_js_urls", template, context, request, view_name - ), - "base_url": self.setting("base_url"), - "datasette_version": __version__, - }, + "request": request, + "crumb_items": self._crumb_items, + "urls": self.urls, + "actor": request.actor if request else None, + "menu_links": menu_links, + "display_actor": display_actor, + "show_logout": request is not None + and "ds_actor" in request.cookies + and request.actor, + "zip": zip, + "body_scripts": body_scripts, + "format_bytes": format_bytes, + "show_messages": lambda: self._show_messages(request), + "extra_css_urls": await self._asset_urls( + "extra_css_urls", template, context, request, view_name + ), + "extra_js_urls": await self._asset_urls( + "extra_js_urls", template, context, request, view_name + ), + "base_url": self.setting("base_url"), + "datasette_version": __version__, **extra_template_vars, } if request and request.args.get("_context") and self.setting("template_debug"): @@ -2941,7 +2925,8 @@ class DatasetteRouter: custom_response ), "Default forbidden() hook should have been called" return await custom_response.asgi_send(send) - except Exception as exception: + except Exception as exception: # noqa: BLE001 + # This IS the top-level error handler - it must catch everything return await self.handle_exception(request, send, exception) async def handle_401(self, request, send, exception): @@ -2963,7 +2948,7 @@ class DatasetteRouter: request.path.replace("~", "~7E").replace("%", "~").replace(".", "~2E") ) if request.query_string: - new_path += "?{}".format(request.query_string) + new_path += f"?{request.query_string}" await asgi_send_redirect(send, new_path) return # If URL has a trailing slash, redirect to URL without it @@ -3173,8 +3158,7 @@ _curly_re = re.compile(r"({.*?})") def route_pattern_from_filepath(filepath): # Drop the ".html" suffix - if filepath.endswith(".html"): - filepath = filepath[: -len(".html")] + filepath = filepath.removesuffix(".html") re_bits = ["/"] for bit in _curly_re.split(filepath): if _curly_re.match(bit): diff --git a/datasette/blob_renderer.py b/datasette/blob_renderer.py index 4d8c6bea..b6c8b77f 100644 --- a/datasette/blob_renderer.py +++ b/datasette/blob_renderer.py @@ -1,8 +1,9 @@ -from datasette import hookimpl -from datasette.utils.asgi import Response, BadRequest -from datasette.utils import to_css_class import hashlib +from datasette import hookimpl +from datasette.utils import to_css_class +from datasette.utils.asgi import BadRequest, Response + _BLOB_COLUMN = "_blob_column" _BLOB_HASH = "_blob_hash" diff --git a/datasette/cli.py b/datasette/cli.py index 90a33e80..57db83b6 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -1,43 +1,45 @@ import asyncio -import uvicorn -import click -from click import formatting -from click.types import CompositeParamType -from click_default_group import DefaultGroup import functools import json import os import pathlib -from runpy import run_module import shutil -from subprocess import call import sys import textwrap import webbrowser +from runpy import run_module +from subprocess import call + +import click +import uvicorn +from click import formatting +from click.types import CompositeParamType +from click_default_group import DefaultGroup + from .app import ( - Datasette, DEFAULT_SETTINGS, SETTINGS, SQLITE_LIMIT_ATTACHED, + Datasette, pm, ) from .inspect import inspect_tables from .utils import ( + ConnectionProblem, LoadExtension, + SpatialiteConnectionProblem, + SpatialiteNotFound, StartupError, + StaticMount, + ValueAsBooleanError, check_connection, deep_dict_update, find_spatialite, - parse_metadata, - ConnectionProblem, - SpatialiteConnectionProblem, initial_path_for_datasette, pairs_to_nested_config, + parse_metadata, temporary_docker_directory, value_as_boolean, - SpatialiteNotFound, - StaticMount, - ValueAsBooleanError, ) from .utils.sqlite import sqlite3 from .utils.testing import TestClient @@ -75,7 +77,7 @@ class Setting(CompositeParamType): # Datasette 1.0, we turn bare setting names into setting.name # Type checking for those older settings default = DEFAULT_SETTINGS[name] - name = "settings.{}".format(name) + name = f"settings.{name}" if isinstance(default, bool): try: return name, "true" if value_as_boolean(value) else "false" @@ -171,7 +173,6 @@ async def inspect_(files, sqlite_extensions): @cli.group() def publish(): """Publish specified SQLite database files to the internet along with a Datasette-powered interface and API""" - pass # Register publish plugins @@ -578,27 +579,27 @@ def serve( # https://github.com/simonw/datasette/issues/2389 deep_dict_update(config_data, settings_updates) - kwargs = dict( - immutables=immutable, - cache_headers=not reload, - cors=cors, - inspect_data=inspect_data, - config=config_data, - metadata=metadata_data, - sqlite_extensions=sqlite_extensions, - template_dir=template_dir, - plugins_dir=plugins_dir, - static_mounts=static, - settings=None, # These are passed in config= now - memory=memory, - secret=secret, - version_note=version_note, - pdb=pdb, - crossdb=crossdb, - nolock=nolock, - internal=internal, - default_deny=default_deny, - ) + kwargs = { + "immutables": immutable, + "cache_headers": not reload, + "cors": cors, + "inspect_data": inspect_data, + "config": config_data, + "metadata": metadata_data, + "sqlite_extensions": sqlite_extensions, + "template_dir": template_dir, + "plugins_dir": plugins_dir, + "static_mounts": static, + "settings": None, # These are passed in config= now + "memory": memory, + "secret": secret, + "version_note": version_note, + "pdb": pdb, + "crossdb": crossdb, + "nolock": nolock, + "internal": internal, + "default_deny": default_deny, + } # Separate directories from files directories = [f for f in files if os.path.isdir(f)] @@ -621,9 +622,7 @@ def serve( conn.close() else: raise click.ClickException( - "Invalid value for '[FILES]...': Path '{}' does not exist.".format( - file - ) + f"Invalid value for '[FILES]...': Path '{file}' does not exist." ) # Check for duplicate files by resolving all paths to their absolute forms @@ -684,7 +683,7 @@ def serve( client = TestClient(ds) request_headers = {} if token: - request_headers["Authorization"] = "Bearer {}".format(token) + request_headers["Authorization"] = f"Bearer {token}" cookies = {} if actor: cookies["ds_actor"] = client.actor_cookie(json.loads(actor)) @@ -719,9 +718,13 @@ def serve( path = run_sync(lambda: initial_path_for_datasette(ds)) url = f"http://{host}:{port}{path}" webbrowser.open(url) - uvicorn_kwargs = dict( - host=host, port=port, log_level="info", lifespan="on", workers=1 - ) + uvicorn_kwargs = { + "host": host, + "port": port, + "log_level": "info", + "lifespan": "on", + "workers": 1, + } if uds: uvicorn_kwargs["uds"] = uds if ssl_keyfile: @@ -885,7 +888,7 @@ async def check_databases(ds): ) except ConnectionProblem as e: raise click.UsageError( - f"Connection to {database.path} failed check: {str(e.args[0])}" + f"Connection to {database.path} failed check: {e.args[0]!s}" ) # If --crossdb and more than SQLITE_LIMIT_ATTACHED show warning if ( @@ -893,9 +896,5 @@ async def check_databases(ds): and len([db for db in ds.databases.values() if not db.is_memory]) > SQLITE_LIMIT_ATTACHED ): - msg = ( - "Warning: --crossdb only works with the first {} attached databases".format( - SQLITE_LIMIT_ATTACHED - ) - ) + msg = f"Warning: --crossdb only works with the first {SQLITE_LIMIT_ATTACHED} attached databases" click.echo(click.style(msg, bold=True, fg="yellow"), err=True) diff --git a/datasette/column_types.py b/datasette/column_types.py index 11a14ec0..92fdd969 100644 --- a/datasette/column_types.py +++ b/datasette/column_types.py @@ -64,14 +64,14 @@ class ColumnType: Return an HTML string to render this cell value, or None to fall through to the default render_cell plugin hook chain. """ - return None + return async def validate(self, value, datasette): """ Validate a value before it is written. Return None if valid, or a string error message if invalid. """ - return None + return async def transform_value(self, value, datasette): """ diff --git a/datasette/csrf.py b/datasette/csrf.py index df239aee..a62f9473 100644 --- a/datasette/csrf.py +++ b/datasette/csrf.py @@ -40,12 +40,12 @@ def _origin_tuple(value): scheme = (parsed.scheme or "").lower() host = (parsed.hostname or "").lower() if not scheme or not host: - raise ValueError("missing scheme or host in {!r}".format(value)) + raise ValueError(f"missing scheme or host in {value!r}") port = parsed.port # may raise ValueError on bad ports if port is None: port = DEFAULT_PORTS.get(scheme) if port is None: - raise ValueError("unknown default port for scheme {!r}".format(scheme)) + raise ValueError(f"unknown default port for scheme {scheme!r}") return scheme, host, port @@ -125,9 +125,7 @@ class CrossOriginProtectionMiddleware: return await self._forbid( send, - "Sec-Fetch-Site was {!r}, expected 'same-origin' or 'none'".format( - sec_fetch_site - ), + f"Sec-Fetch-Site was {sec_fetch_site!r}, expected 'same-origin' or 'none'", ) return @@ -141,11 +139,11 @@ class CrossOriginProtectionMiddleware: request_scheme = self._request_scheme(scope) try: origin_tuple = _origin_tuple(origin) - expected_tuple = _origin_tuple("{}://{}".format(request_scheme, host)) + expected_tuple = _origin_tuple(f"{request_scheme}://{host}") except ValueError: await self._forbid( send, - "Malformed Origin {!r} or Host {!r}".format(origin, host), + f"Malformed Origin {origin!r} or Host {host!r}", ) return @@ -155,7 +153,7 @@ class CrossOriginProtectionMiddleware: await self._forbid( send, - "Origin {!r} does not match Host {!r}".format(origin, host), + f"Origin {origin!r} does not match Host {host!r}", ) def _request_scheme(self, scope): @@ -163,7 +161,8 @@ class CrossOriginProtectionMiddleware: try: if self.datasette.setting("force_https_urls"): return "https" - except Exception: + except Exception: # noqa: BLE001, S110 + # Settings may not be readable this early; fall back to the ASGI scheme pass return scope.get("scheme") or "http" diff --git a/datasette/database.py b/datasette/database.py index bab3a378..e162d34e 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -1,16 +1,18 @@ import asyncio import atexit -from collections import namedtuple import inspect import os -from pathlib import Path import queue -import sqlite_utils import sys import tempfile import threading import uuid +from collections import namedtuple +from pathlib import Path +import sqlite_utils + +from .inspect import inspect_hash from .tracer import trace from .utils import ( call_with_supported_arguments, @@ -21,14 +23,13 @@ from .utils import ( get_all_foreign_keys, get_outbound_foreign_keys, md5_not_usedforsecurity, - sqlite_timelimit, sqlite3, - table_columns, + sqlite_timelimit, table_column_details, + table_columns, ) from .utils.sql_analysis import SQLAnalysis, analyze_sql_tables from .utils.sqlite import sqlite_hidden_table_names -from .inspect import inspect_hash connections = threading.local() @@ -99,9 +100,7 @@ class Database: def _check_not_closed(self): if self._closed: - raise DatasetteClosedError( - "Database {!r} has been closed".format(self.name) - ) + raise DatasetteClosedError(f"Database {self.name!r} has been closed") def _remove_pending_execute_future(self, future): with self._pending_execute_futures_lock: @@ -140,7 +139,7 @@ class Database: if write: extra_kwargs["isolation_level"] = "IMMEDIATE" if self.memory_name: - uri = "file:{}?mode=memory&cache=shared".format(self.memory_name) + uri = f"file:{self.memory_name}?mode=memory&cache=shared" conn = sqlite3.connect( uri, uri=True, check_same_thread=False, **extra_kwargs ) @@ -193,21 +192,20 @@ class Database: write_thread.join(timeout=10) if write_thread.is_alive(): sys.stderr.write( - "Datasette: write thread for {!r} did not exit within 10s\n".format( - self.name - ) + f"Datasette: write thread for {self.name!r} did not exit within 10s\n" ) sys.stderr.flush() for future in pending_execute_futures: try: future.result() - except Exception: + except Exception: # noqa: BLE001, S110 + # Shutdown teardown - a failed pending write must not block close() pass # Close anything still tracked in _all_file_connections for connection in self._all_file_connections: try: connection.close() - except Exception: + except Exception: # noqa: BLE001, S110 pass self._all_file_connections = [] # Drop per-thread cached read connections we can reach @@ -219,13 +217,13 @@ class Database: if self._read_connection is not None: try: self._read_connection.close() - except Exception: + except Exception: # noqa: BLE001, S110 pass self._read_connection = None if self._write_connection is not None: try: self._write_connection.close() - except Exception: + except Exception: # noqa: BLE001, S110 pass self._write_connection = None if self.is_temp_disk: @@ -371,7 +369,8 @@ class Database: async def _dispatch_events_after_write(): try: await reply_future - except Exception: + except Exception: # noqa: BLE001 + # The write failed; skip success events regardless of why # if the write failed, don't emit success events return for event in pending_events: @@ -424,9 +423,7 @@ class Database: self._write_thread = threading.Thread( target=self._execute_writes, daemon=True ) - self._write_thread.name = "_execute_writes for database {}".format( - self.name - ) + self._write_thread.name = f"_execute_writes for database {self.name}" self._write_thread.start() task_id = uuid.uuid5(uuid.NAMESPACE_DNS, "datasette.io") loop = asyncio.get_running_loop() @@ -447,7 +444,8 @@ class Database: try: conn = self.connect(write=True) self.ds._prepare_connection(conn, self.name) - except Exception as e: + except Exception as e: # noqa: BLE001 + # Stored and re-raised to whoever queues the next write conn_exception = e while True: task = self._write_queue.get() @@ -455,7 +453,8 @@ class Database: if conn is not None: try: conn.close() - except Exception: + except Exception: # noqa: BLE001, S110 + # Best-effort close as the write thread exits pass return exception = None @@ -474,8 +473,9 @@ class Database: except ValueError: # Was probably a memory connection pass - except Exception as e: - sys.stderr.write("{}\n".format(e)) + except Exception as e: # noqa: BLE001 + # Write thread must survive any task failure or the database wedges + sys.stderr.write(f"{e}\n") sys.stderr.flush() exception = e else: @@ -486,8 +486,8 @@ class Database: result = task.fn(conn) else: result = task.fn(conn) - except Exception as e: - sys.stderr.write("{}\n".format(e)) + except Exception as e: # noqa: BLE001 + sys.stderr.write(f"{e}\n") sys.stderr.flush() exception = e _deliver_write_result(task, result, exception) @@ -554,9 +554,7 @@ class Database: raise QueryInterrupted(e, sql, params) if log_sql_errors: sys.stderr.write( - "ERROR: conn={}, sql = {}, params = {}: {}\n".format( - conn, repr(sql), params, e - ) + f"ERROR: conn={conn}, sql = {sql!r}, params = {params}: {e}\n" ) sys.stderr.flush() raise @@ -713,9 +711,9 @@ class Database: column_names and len(column_names) == 2 and ("id" in column_names or "pk" in column_names) - and not set(column_names) == {"id", "pk"} + and set(column_names) != {"id", "pk"} ): - return [c for c in column_names if c not in ("id", "pk")][0] + return next(c for c in column_names if c not in ("id", "pk")) # Couldn't find a label: return None @@ -857,10 +855,10 @@ def _apply_write_wrapper(fn, wrapper_factory, track_event): class WriteTask: __slots__ = ( "fn", - "task_id", + "isolated_connection", "loop", "reply_future", - "isolated_connection", + "task_id", "transaction", ) @@ -901,7 +899,7 @@ class QueryInterrupted(Exception): self.params = params def __str__(self): - return "QueryInterrupted: {}".format(self.e) + return f"QueryInterrupted: {self.e}" class MultipleValues(Exception): diff --git a/datasette/default_actions.py b/datasette/default_actions.py index 602e0df4..ee165ae5 100644 --- a/datasette/default_actions.py +++ b/datasette/default_actions.py @@ -2,8 +2,8 @@ from datasette import hookimpl from datasette.permissions import Action from datasette.resources import ( DatabaseResource, - TableResource, QueryResource, + TableResource, ) diff --git a/datasette/default_magic_parameters.py b/datasette/default_magic_parameters.py index 91c1c5aa..bff5f0a7 100644 --- a/datasette/default_magic_parameters.py +++ b/datasette/default_magic_parameters.py @@ -1,8 +1,9 @@ -from datasette import hookimpl import datetime import os import time +from datasette import hookimpl + def header(key, request): key = key.replace("_", "-").encode("utf-8") diff --git a/datasette/default_permissions/__init__.py b/datasette/default_permissions/__init__.py index 6cd46f04..dee5df42 100644 --- a/datasette/default_permissions/__init__.py +++ b/datasette/default_permissions/__init__.py @@ -17,18 +17,29 @@ UNION/INTERSECT operations. The order of evaluation is: from __future__ import annotations -# Re-export all hooks and public utilities -from .restrictions import ( - actor_restrictions_sql as actor_restrictions_sql, - restrictions_allow_action as restrictions_allow_action, - ActorRestrictions as ActorRestrictions, -) -from .root import root_user_permissions_sql as root_user_permissions_sql from .config import config_permissions_sql as config_permissions_sql +from .defaults import ( + DEFAULT_ALLOW_ACTIONS as DEFAULT_ALLOW_ACTIONS, +) +from .defaults import ( + default_action_permissions_sql as default_action_permissions_sql, +) from .defaults import ( # Avoid "datasette.default_permissions" does not explicitly export attribute default_allow_sql_check as default_allow_sql_check, - default_action_permissions_sql as default_action_permissions_sql, - default_query_permissions_sql as default_query_permissions_sql, - DEFAULT_ALLOW_ACTIONS as DEFAULT_ALLOW_ACTIONS, ) +from .defaults import ( + default_query_permissions_sql as default_query_permissions_sql, +) +from .restrictions import ( + ActorRestrictions as ActorRestrictions, +) + +# Re-export all hooks and public utilities +from .restrictions import ( + actor_restrictions_sql as actor_restrictions_sql, +) +from .restrictions import ( + restrictions_allow_action as restrictions_allow_action, +) +from .root import root_user_permissions_sql as root_user_permissions_sql diff --git a/datasette/default_permissions/config.py b/datasette/default_permissions/config.py index 8edc976e..4494f07f 100644 --- a/datasette/default_permissions/config.py +++ b/datasette/default_permissions/config.py @@ -6,7 +6,7 @@ Applies permission rules from datasette.yaml configuration. from __future__ import annotations -from typing import TYPE_CHECKING, Any, List, Optional, Set, Tuple +from typing import TYPE_CHECKING, Any if TYPE_CHECKING: from datasette.app import Datasette @@ -55,8 +55,8 @@ class ConfigPermissionProcessor: def __init__( self, - datasette: "Datasette", - actor: Optional[dict], + datasette: Datasette, + actor: dict | None, action: str, ): self.datasette = datasette @@ -74,8 +74,8 @@ class ConfigPermissionProcessor: self.restrictions = actor.get("_r", {}) if actor else {} # Pre-compute restriction info for efficiency - self.restricted_databases: Set[str] = set() - self.restricted_tables: Set[Tuple[str, str]] = set() + self.restricted_databases: set[str] = set() + self.restricted_tables: set[tuple[str, str]] = set() if self.has_restrictions: self.restricted_databases = { @@ -92,7 +92,7 @@ class ConfigPermissionProcessor: # Tables implicitly reference their parent databases self.restricted_databases.update(db for db, _ in self.restricted_tables) - def evaluate_allow_block(self, allow_block: Any) -> Optional[bool]: + def evaluate_allow_block(self, allow_block: Any) -> bool | None: """Evaluate an allow block against the current actor.""" if allow_block is None: return None @@ -104,8 +104,8 @@ class ConfigPermissionProcessor: def is_in_restriction_allowlist( self, - parent: Optional[str], - child: Optional[str], + parent: str | None, + child: str | None, ) -> bool: """Check if resource is allowed by actor restrictions.""" if not self.has_restrictions: @@ -147,9 +147,9 @@ class ConfigPermissionProcessor: def add_permissions_rule( self, - parent: Optional[str], - child: Optional[str], - permissions_block: Optional[dict], + parent: str | None, + child: str | None, + permissions_block: dict | None, scope_desc: str, ) -> None: """Add a rule from a permissions:{action} block.""" @@ -169,8 +169,8 @@ class ConfigPermissionProcessor: def add_allow_block_rule( self, - parent: Optional[str], - child: Optional[str], + parent: str | None, + child: str | None, allow_block: Any, scope_desc: str, ) -> None: @@ -202,8 +202,8 @@ class ConfigPermissionProcessor: def _add_restriction_gate_denies( self, - parent: Optional[str], - child: Optional[str], + parent: str | None, + child: str | None, is_allowed: bool, scope_desc: str, ) -> None: @@ -235,7 +235,7 @@ class ConfigPermissionProcessor: if db_name == parent: self.collector.add(db_name, table_name, False, reason) - def process(self) -> Optional[PermissionSQL]: + def process(self) -> PermissionSQL | None: """Process all config rules and return combined PermissionSQL.""" self._process_root_permissions() self._process_databases() @@ -425,10 +425,10 @@ class ConfigPermissionProcessor: @hookimpl(specname="permission_resources_sql") async def config_permissions_sql( - datasette: "Datasette", - actor: Optional[dict], + datasette: Datasette, + actor: dict | None, action: str, -) -> Optional[List[PermissionSQL]]: +) -> list[PermissionSQL] | None: """ Apply permission rules from datasette.yaml configuration. diff --git a/datasette/default_permissions/defaults.py b/datasette/default_permissions/defaults.py index 5bc74425..6f97812b 100644 --- a/datasette/default_permissions/defaults.py +++ b/datasette/default_permissions/defaults.py @@ -6,7 +6,7 @@ Provides default allow rules for standard view/execute actions. from __future__ import annotations -from typing import TYPE_CHECKING, Optional +from typing import TYPE_CHECKING if TYPE_CHECKING: from datasette.app import Datasette @@ -29,29 +29,28 @@ DEFAULT_ALLOW_ACTIONS = frozenset( @hookimpl(specname="permission_resources_sql") async def default_allow_sql_check( - datasette: "Datasette", - actor: Optional[dict], + datasette: Datasette, + actor: dict | None, action: str, -) -> Optional[PermissionSQL]: +) -> PermissionSQL | None: """ Enforce the default_allow_sql setting. When default_allow_sql is false (the default), execute-sql is denied unless explicitly allowed by config or other rules. """ - if action == "execute-sql": - if not datasette.setting("default_allow_sql"): - return PermissionSQL.deny(reason="default_allow_sql is false") + if action == "execute-sql" and not datasette.setting("default_allow_sql"): + return PermissionSQL.deny(reason="default_allow_sql is false") return None @hookimpl(specname="permission_resources_sql") async def default_action_permissions_sql( - datasette: "Datasette", - actor: Optional[dict], + datasette: Datasette, + actor: dict | None, action: str, -) -> Optional[PermissionSQL]: +) -> PermissionSQL | None: """ Provide default allow rules for standard view/execute actions. @@ -71,10 +70,10 @@ async def default_action_permissions_sql( @hookimpl(specname="permission_resources_sql") async def default_query_permissions_sql( - datasette: "Datasette", - actor: Optional[dict], + datasette: Datasette, + actor: dict | None, action: str, -) -> Optional[PermissionSQL]: +) -> PermissionSQL | None: actor_id = actor.get("id") if isinstance(actor, dict) else None if action not in {"view-query", "update-query", "delete-query"}: diff --git a/datasette/default_permissions/helpers.py b/datasette/default_permissions/helpers.py index 47e03569..5e59b7b4 100644 --- a/datasette/default_permissions/helpers.py +++ b/datasette/default_permissions/helpers.py @@ -5,7 +5,7 @@ Shared helper utilities for default permission implementations. from __future__ import annotations from dataclasses import dataclass -from typing import TYPE_CHECKING, List, Optional, Set +from typing import TYPE_CHECKING if TYPE_CHECKING: from datasette.app import Datasette @@ -13,7 +13,7 @@ if TYPE_CHECKING: from datasette.permissions import PermissionSQL -def get_action_name_variants(datasette: "Datasette", action: str) -> Set[str]: +def get_action_name_variants(datasette: Datasette, action: str) -> set[str]: """ Get all name variants for an action (full name and abbreviation). @@ -27,7 +27,7 @@ def get_action_name_variants(datasette: "Datasette", action: str) -> Set[str]: return variants -def action_in_list(datasette: "Datasette", action: str, action_list: list) -> bool: +def action_in_list(datasette: Datasette, action: str, action_list: list) -> bool: """Check if an action (or its abbreviation) is in a list.""" return bool(get_action_name_variants(datasette, action).intersection(action_list)) @@ -36,8 +36,8 @@ def action_in_list(datasette: "Datasette", action: str, action_list: list) -> bo class PermissionRow: """A single permission rule row.""" - parent: Optional[str] - child: Optional[str] + parent: str | None + child: str | None allow: bool reason: str @@ -46,14 +46,14 @@ class PermissionRowCollector: """Collects permission rows and converts them to PermissionSQL.""" def __init__(self, prefix: str = "row"): - self.rows: List[PermissionRow] = [] + self.rows: list[PermissionRow] = [] self.prefix = prefix def add( self, - parent: Optional[str], - child: Optional[str], - allow: Optional[bool], + parent: str | None, + child: str | None, + allow: bool | None, reason: str, if_not_none: bool = False, ) -> None: @@ -62,7 +62,7 @@ class PermissionRowCollector: return self.rows.append(PermissionRow(parent, child, allow, reason)) - def to_permission_sql(self) -> Optional[PermissionSQL]: + def to_permission_sql(self) -> PermissionSQL | None: """Convert collected rows to a PermissionSQL object.""" if not self.rows: return None diff --git a/datasette/default_permissions/restrictions.py b/datasette/default_permissions/restrictions.py index a22cd7e5..88e1d274 100644 --- a/datasette/default_permissions/restrictions.py +++ b/datasette/default_permissions/restrictions.py @@ -8,7 +8,7 @@ contains allowlists of resources the actor can access. from __future__ import annotations from dataclasses import dataclass -from typing import TYPE_CHECKING, List, Optional, Set, Tuple +from typing import TYPE_CHECKING if TYPE_CHECKING: from datasette.app import Datasette @@ -23,12 +23,12 @@ from .helpers import action_in_list, get_action_name_variants class ActorRestrictions: """Parsed actor restrictions from the _r key.""" - global_actions: List[str] # _r.a - globally allowed actions + global_actions: list[str] # _r.a - globally allowed actions database_actions: dict # _r.d - {db_name: [actions]} table_actions: dict # _r.r - {db_name: {table: [actions]}} @classmethod - def from_actor(cls, actor: Optional[dict]) -> Optional["ActorRestrictions"]: + def from_actor(cls, actor: dict | None) -> ActorRestrictions | None: """Parse restrictions from actor dict. Returns None if no restrictions.""" if not actor: return None @@ -44,11 +44,11 @@ class ActorRestrictions: table_actions=restrictions.get("r", {}), ) - def is_action_globally_allowed(self, datasette: "Datasette", action: str) -> bool: + def is_action_globally_allowed(self, datasette: Datasette, action: str) -> bool: """Check if action is in the global allowlist.""" return action_in_list(datasette, action, self.global_actions) - def get_allowed_databases(self, datasette: "Datasette", action: str) -> Set[str]: + def get_allowed_databases(self, datasette: Datasette, action: str) -> set[str]: """Get database names where this action is allowed.""" allowed = set() for db_name, db_actions in self.database_actions.items(): @@ -57,8 +57,8 @@ class ActorRestrictions: return allowed def get_allowed_tables( - self, datasette: "Datasette", action: str - ) -> Set[Tuple[str, str]]: + self, datasette: Datasette, action: str + ) -> set[tuple[str, str]]: """Get (database, table) pairs where this action is allowed.""" allowed = set() for db_name, tables in self.table_actions.items(): @@ -70,10 +70,10 @@ class ActorRestrictions: @hookimpl(specname="permission_resources_sql") async def actor_restrictions_sql( - datasette: "Datasette", - actor: Optional[dict], + datasette: Datasette, + actor: dict | None, action: str, -) -> Optional[List[PermissionSQL]]: +) -> list[PermissionSQL] | None: """ Handle actor restriction-based permission rules. @@ -140,10 +140,10 @@ async def actor_restrictions_sql( def restrictions_allow_action( - datasette: "Datasette", + datasette: Datasette, restrictions: dict, action: str, - resource: Optional[str | Tuple[str, str]], + resource: str | tuple[str, str] | None, ) -> bool: """ Check if restrictions allow the requested action on the requested resource. diff --git a/datasette/default_permissions/root.py b/datasette/default_permissions/root.py index 4931f7ff..22d13f65 100644 --- a/datasette/default_permissions/root.py +++ b/datasette/default_permissions/root.py @@ -6,7 +6,7 @@ Grants full permissions to the root user when --root flag is used. from __future__ import annotations -from typing import TYPE_CHECKING, Optional +from typing import TYPE_CHECKING if TYPE_CHECKING: from datasette.app import Datasette @@ -17,9 +17,9 @@ from datasette.permissions import PermissionSQL @hookimpl(specname="permission_resources_sql") async def root_user_permissions_sql( - datasette: "Datasette", - actor: Optional[dict], -) -> Optional[PermissionSQL]: + datasette: Datasette, + actor: dict | None, +) -> PermissionSQL | None: """ Grant root user full permissions when --root flag is used. """ diff --git a/datasette/default_permissions/tokens.py b/datasette/default_permissions/tokens.py index 7a359dc6..52daf8a2 100644 --- a/datasette/default_permissions/tokens.py +++ b/datasette/default_permissions/tokens.py @@ -7,7 +7,7 @@ to datasette.verify_token() so all registered handlers are tried. from __future__ import annotations -from typing import TYPE_CHECKING, Optional +from typing import TYPE_CHECKING if TYPE_CHECKING: from datasette.app import Datasette @@ -17,15 +17,13 @@ from datasette.tokens import SignedTokenHandler @hookimpl -def register_token_handler(datasette: "Datasette"): +def register_token_handler(datasette: Datasette): """Register the default signed token handler.""" return SignedTokenHandler() @hookimpl(specname="actor_from_request") -async def actor_from_signed_api_token( - datasette: "Datasette", request -) -> Optional[dict]: +async def actor_from_signed_api_token(datasette: Datasette, request) -> dict | None: """ Authenticate requests using API tokens by delegating to all registered token handlers via datasette.verify_token(). diff --git a/datasette/default_table_actions.py b/datasette/default_table_actions.py index e41434ef..0f2f32ef 100644 --- a/datasette/default_table_actions.py +++ b/datasette/default_table_actions.py @@ -20,7 +20,7 @@ def table_actions(datasette, actor, database, table, request): "label": "Alter table", "description": "Change columns and primary key for this table.", "attrs": { - "aria-label": "Alter table {}".format(table), + "aria-label": f"Alter table {table}", "data-table-action": "alter-table", }, } diff --git a/datasette/events.py b/datasette/events.py index e8786da9..5f3fd06e 100644 --- a/datasette/events.py +++ b/datasette/events.py @@ -1,8 +1,9 @@ from abc import ABC, abstractproperty from dataclasses import asdict, dataclass, field -from datasette.hookspecs import hookimpl from datetime import datetime, timezone +from datasette.hookspecs import hookimpl + @dataclass class Event(ABC): diff --git a/datasette/facets.py b/datasette/facets.py index 5f757df3..8c09e1dc 100644 --- a/datasette/facets.py +++ b/datasette/facets.py @@ -1,12 +1,13 @@ import json import urllib + from datasette import hookimpl from datasette.database import QueryInterrupted from datasette.utils import ( + detect_json1, escape_sqlite, path_with_added_args, path_with_removed_args, - detect_json1, sqlite3, ) @@ -30,7 +31,7 @@ def load_facet_configs(request, table_config): assert ( len(facet_config.values()) == 1 ), "Metadata config dicts should be {type: config}" - type, facet_config = list(facet_config.items())[0] + type, facet_config = next(iter(facet_config.items())) if isinstance(facet_config, str): facet_config = {"simple": facet_config} facet_configs.setdefault(type, []).append( @@ -160,18 +161,13 @@ class ColumnFacet(Facet): for column in columns: if column in already_enabled: continue - suggested_facet_sql = """ - with limited as (select * from ({sql}) limit {suggest_consider}) - select {column} as value, count(*) as n from limited + suggested_facet_sql = f""" + with limited as (select * from ({self.sql}) limit {self.suggest_consider}) + select {escape_sqlite(column)} as value, count(*) as n from limited where value is not null group by value - limit {limit} - """.format( - column=escape_sqlite(column), - sql=self.sql, - limit=facet_size + 1, - suggest_consider=self.suggest_consider, - ) + limit {facet_size + 1} + """ distinct_values = None try: distinct_values = await self.ds.execute( @@ -267,7 +263,7 @@ class ColumnFacet(Facet): for row in facet_rows: column_qs = column if column.startswith("_"): - column_qs = "{}__exact".format(column) + column_qs = f"{column}__exact" selected = (column_qs, str(row["value"])) in qs_pairs if selected: toggle_path = path_with_removed_args( @@ -342,12 +338,12 @@ class ArrayFacet(Facet): for v in await self.ds.execute( self.database, ( - "select {column} from ({sql}) " - "where {column} is not null " - "and {column} != '' " - "and json_array_length({column}) > 0 " + f"select {escape_sqlite(column)} from ({self.sql}) " + f"where {escape_sqlite(column)} is not null " + f"and {escape_sqlite(column)} != '' " + f"and json_array_length({escape_sqlite(column)}) > 0 " "limit 100" - ).format(column=escape_sqlite(column), sql=self.sql), + ), self.params, truncate=False, custom_time_limit=self.ds.setting( @@ -388,14 +384,14 @@ class ArrayFacet(Facet): source = source_and_config["source"] column = config.get("column") or config["simple"] # https://github.com/simonw/datasette/issues/448 - facet_sql = """ - with inner as ({sql}), + facet_sql = f""" + with inner as ({self.sql}), deduped_array_items as ( select distinct j.value, inner.* from - json_each([inner].{col}) j + json_each([inner].{escape_sqlite(column)}) j join inner ) select @@ -406,12 +402,8 @@ class ArrayFacet(Facet): group by value order by - count(*) desc, value limit {limit} - """.format( - col=escape_sqlite(column), - sql=self.sql, - limit=facet_size + 1, - ) + count(*) desc, value limit {facet_size + 1} + """ try: facet_rows_results = await self.ds.execute( self.database, diff --git a/datasette/filters.py b/datasette/filters.py index 95cc5f37..1d4e32c2 100644 --- a/datasette/filters.py +++ b/datasette/filters.py @@ -1,8 +1,11 @@ +import json +from typing import ClassVar + from datasette import hookimpl from datasette.resources import DatabaseResource -from datasette.views.base import DatasetteError from datasette.utils.asgi import BadRequest -import json +from datasette.views.base import DatasetteError + from .utils import detect_json1, escape_sqlite, path_with_removed_args @@ -99,9 +102,9 @@ def search_filters(request, database, table, datasette): fts_table=escape_sqlite(fts_table), search_col=escape_sqlite(search_col), match_clause=( - ":search_{}".format(i) + f":search_{i}" if search_mode_raw - else "escape_fts(:search_{})".format(i) + else f"escape_fts(:search_{i})" ), ) ) @@ -134,11 +137,11 @@ def through_filters(request, database, table, datasette): value = through_data["value"] db = datasette.get_database(database) outgoing_foreign_keys = await db.foreign_keys_for_table(through_table) - try: - fk_to_us = [ - fk for fk in outgoing_foreign_keys if fk["other_table"] == table - ][0] - except IndexError: + fk_to_us = next( + (fk for fk in outgoing_foreign_keys if fk["other_table"] == table), + None, + ) + if fk_to_us is None: raise DatasetteError( "Invalid _through - could not find corresponding foreign key" ) @@ -365,7 +368,7 @@ class Filters: ), ] ) - _filters_by_key = {f.key: f for f in _filters} + _filters_by_key: ClassVar[dict[str, Filter]] = {f.key: f for f in _filters} def __init__(self, pairs): self.pairs = pairs diff --git a/datasette/fixtures.py b/datasette/fixtures.py index 7c85e16a..049e35ed 100644 --- a/datasette/fixtures.py +++ b/datasette/fixtures.py @@ -1,9 +1,10 @@ -from datasette.utils.sqlite import sqlite3 -from datasette.utils import documented import itertools import random import string +from datasette.utils import documented +from datasette.utils.sqlite import sqlite3 + __all__ = [ "EXTRA_DATABASE_SQL", "TABLES", @@ -346,9 +347,7 @@ CREATE VIEW searchable_view_configured_by_metadata AS + '\nINSERT INTO no_primary_key VALUES ("RENDER_CELL_DEMO", "a202", "b202", "c202");\n' + "\n".join( [ - 'INSERT INTO compound_three_primary_keys VALUES ("{a}", "{b}", "{c}", "{content}");'.format( - a=a, b=b, c=c, content=content - ) + f'INSERT INTO compound_three_primary_keys VALUES ("{a}", "{b}", "{c}", "{content}");' for a, b, c, content in generate_compound_rows(1001) ] ) diff --git a/datasette/forbidden.py b/datasette/forbidden.py index 91b1ff96..67bf0d8b 100644 --- a/datasette/forbidden.py +++ b/datasette/forbidden.py @@ -1,4 +1,5 @@ -from datasette import hookimpl, Response +from datasette import Response, hookimpl + from .utils import add_cors_headers diff --git a/datasette/handle_exception.py b/datasette/handle_exception.py index e255ddf2..c36d5dbe 100644 --- a/datasette/handle_exception.py +++ b/datasette/handle_exception.py @@ -1,16 +1,21 @@ -from datasette import hookimpl, Response +import traceback + +from markupsafe import Markup + +from datasette import Response, hookimpl + from .utils import add_cors_headers, error_body from .utils.asgi import ( Base400, ) from .views.base import DatasetteError -from markupsafe import Markup -import traceback +# Debugger imports are deliberate - they back the "pdb" setting, which drops +# into a debugger on unhandled exceptions try: - import ipdb as pdb + import ipdb as pdb # noqa: T100 except ImportError: - import pdb + import pdb # noqa: T100 try: import rich @@ -69,7 +74,7 @@ def handle_exception(datasette, request, exception): dict( info, urls=datasette.urls, - menu_links=lambda: [], + menu_links=list, ) ), status=status, diff --git a/datasette/hookspecs.py b/datasette/hookspecs.py index 7c56f882..f89f2f36 100644 --- a/datasette/hookspecs.py +++ b/datasette/hookspecs.py @@ -1,5 +1,4 @@ -from pluggy import HookimplMarker -from pluggy import HookspecMarker +from pluggy import HookimplMarker, HookspecMarker hookspec = HookspecMarker("datasette") hookimpl = HookimplMarker("datasette") diff --git a/datasette/inspect.py b/datasette/inspect.py index 5e681e03..b126ce5c 100644 --- a/datasette/inspect.py +++ b/datasette/inspect.py @@ -1,13 +1,13 @@ import hashlib from .utils import ( - detect_spatialite, detect_fts, detect_primary_keys, + detect_spatialite, escape_sqlite, get_all_foreign_keys, - table_columns, sqlite3, + table_columns, ) HASH_BLOCK_SIZE = 1024 * 1024 @@ -95,10 +95,10 @@ def inspect_tables(conn, database_metadata): """) ] - for t in tables.keys(): + for t, table_info in tables.items(): for hidden_table in hidden_tables: if t == hidden_table or t.startswith(hidden_table): - tables[t]["hidden"] = True + table_info["hidden"] = True continue return tables diff --git a/datasette/jump.py b/datasette/jump.py index d138e827..d70d33df 100644 --- a/datasette/jump.py +++ b/datasette/jump.py @@ -21,7 +21,7 @@ class JumpSQL: search_text: str | None = None, display_name: str | None = None, item_type: str = "menu", - ) -> "JumpSQL": + ) -> JumpSQL: if search_text is None: search_text = " ".join( text for text in (label, display_name, description) if text is not None diff --git a/datasette/permissions.py b/datasette/permissions.py index 786dc026..e03b065c 100644 --- a/datasette/permissions.py +++ b/datasette/permissions.py @@ -1,7 +1,7 @@ +import contextvars from abc import ABC, abstractmethod from dataclasses import dataclass from typing import Any, NamedTuple -import contextvars # Context variable to track when permission checks should be skipped _skip_permission_checks = contextvars.ContextVar( @@ -72,8 +72,8 @@ class Resource(ABC): ) def __repr__(self) -> str: - return "{}(parent={!r}, child={!r})".format( - self.__class__.__name__, self.parent, self.child + return ( + f"{self.__class__.__name__}(parent={self.parent!r}, child={self.child!r})" ) @property @@ -129,7 +129,6 @@ class Resource(ABC): Must return two columns: parent, child """ - pass class AllowedResource(NamedTuple): diff --git a/datasette/plugins.py b/datasette/plugins.py index ae2cb17d..9cf94079 100644 --- a/datasette/plugins.py +++ b/datasette/plugins.py @@ -1,20 +1,14 @@ import importlib +import importlib.metadata as importlib_metadata +import importlib.resources as importlib_resources import os -import pluggy -from pprint import pprint import sys +from pprint import pprint + +import pluggy + from . import hookspecs -if sys.version_info >= (3, 9): - import importlib.resources as importlib_resources -else: - import importlib_resources -if sys.version_info >= (3, 10): - import importlib.metadata as importlib_metadata -else: - import importlib_metadata - - DEFAULT_PLUGINS = ( "datasette.publish.heroku", "datasette.publish.cloudrun", @@ -85,7 +79,7 @@ if DATASETTE_LOAD_PLUGINS is not None: # Ensure name can be found in plugin_to_distinfo later: pm._plugin_distinfo.append((mod, distribution)) except importlib_metadata.PackageNotFoundError: - sys.stderr.write("Plugin {} could not be found\n".format(package_name)) + sys.stderr.write(f"Plugin {package_name} could not be found\n") # Load default plugins diff --git a/datasette/publish/cloudrun.py b/datasette/publish/cloudrun.py index 63d22fe8..9ace865b 100644 --- a/datasette/publish/cloudrun.py +++ b/datasette/publish/cloudrun.py @@ -1,15 +1,17 @@ -from datasette import hookimpl -import click import json import os import re from subprocess import CalledProcessError, check_call, check_output +import click + +from datasette import hookimpl + +from ..utils import temporary_docker_directory from .common import ( add_common_publish_arguments_and_options, fail_if_publish_binary_not_installed, ) -from ..utils import temporary_docker_directory @hookimpl @@ -219,7 +221,7 @@ def publish_subcommand(publish): check_call( "gcloud builds submit --tag {}{}".format( - image_id, " --timeout {}".format(timeout) if timeout else "" + image_id, f" --timeout {timeout}" if timeout else "" ), shell=True, ) @@ -231,7 +233,7 @@ def publish_subcommand(publish): ("--min-instances", min_instances), ): if value is not None: - extra_deploy_options.append("{} {}".format(option, value)) + extra_deploy_options.append(f"{option} {value}") check_call( "gcloud run deploy --allow-unauthenticated --platform=managed --image {} {}{}".format( image_id, @@ -258,24 +260,16 @@ def _ensure_artifact_registry(artifact_project, artifact_region, artifact_reposi ) from exc describe_cmd = ( - "gcloud artifacts repositories describe {repo} --project {project} " - "--location {location} --quiet" - ).format( - repo=artifact_repository, - project=artifact_project, - location=artifact_region, + f"gcloud artifacts repositories describe {artifact_repository} --project {artifact_project} " + f"--location {artifact_region} --quiet" ) try: check_call(describe_cmd, shell=True) return except CalledProcessError: create_cmd = ( - "gcloud artifacts repositories create {repo} --repository-format=docker " - '--location {location} --project {project} --description "Datasette Cloud Run images" --quiet' - ).format( - repo=artifact_repository, - location=artifact_region, - project=artifact_project, + f"gcloud artifacts repositories create {artifact_repository} --repository-format=docker " + f'--location {artifact_region} --project {artifact_project} --description "Datasette Cloud Run images" --quiet' ) try: check_call(create_cmd, shell=True) diff --git a/datasette/publish/common.py b/datasette/publish/common.py index 29665eb3..27dfd4bf 100644 --- a/datasette/publish/common.py +++ b/datasette/publish/common.py @@ -1,9 +1,11 @@ -from ..utils import StaticMount -import click import os import shutil import sys +import click + +from ..utils import StaticMount + def add_common_publish_arguments_and_options(subcommand): for decorator in reversed( @@ -76,9 +78,7 @@ def fail_if_publish_binary_not_installed(binary, publish_target, install_link): """Exit (with error message) if ``binary` isn't installed""" if not shutil.which(binary): click.secho( - "Publishing to {publish_target} requires {binary} to be installed and configured".format( - publish_target=publish_target, binary=binary - ), + f"Publishing to {publish_target} requires {binary} to be installed and configured", bg="red", fg="white", bold=True, diff --git a/datasette/publish/heroku.py b/datasette/publish/heroku.py index f576a346..b0290833 100644 --- a/datasette/publish/heroku.py +++ b/datasette/publish/heroku.py @@ -1,19 +1,21 @@ -from contextlib import contextmanager -from datasette import hookimpl -import click import json import os import pathlib import shlex import shutil -from subprocess import call, check_output import tempfile +from contextlib import contextmanager +from subprocess import call, check_output + +import click + +from datasette import hookimpl +from datasette.utils import link_or_copy, link_or_copy_directory, parse_metadata from .common import ( add_common_publish_arguments_and_options, fail_if_publish_binary_not_installed, ) -from datasette.utils import link_or_copy, link_or_copy_directory, parse_metadata @hookimpl @@ -234,7 +236,7 @@ def temporary_heroku_directory( extras.extend(["--static", f"{mount_point}:{mount_point}"]) quoted_files = " ".join( - ["-i {}".format(shlex.quote(file_name)) for file_name in file_names] + [f"-i {shlex.quote(file_name)}" for file_name in file_names] ) procfile_cmd = "web: datasette serve --host 0.0.0.0 {quoted_files} --cors --port $PORT --inspect-file inspect-data.json {extras}".format( quoted_files=quoted_files, extras=" ".join(extras) diff --git a/datasette/renderer.py b/datasette/renderer.py index 7c94f6ee..0e01f52f 100644 --- a/datasette/renderer.py +++ b/datasette/renderer.py @@ -1,12 +1,13 @@ import json + from datasette.extras import extra_names_from_request from datasette.utils import ( - error_body, - value_as_boolean, - remove_infinites, CustomJSONEncoder, + error_body, path_from_row_pks, + remove_infinites, sqlite3, + value_as_boolean, ) from datasette.utils.asgi import Response diff --git a/datasette/stored_queries.py b/datasette/stored_queries.py index f5f977d9..db3c6548 100644 --- a/datasette/stored_queries.py +++ b/datasette/stored_queries.py @@ -1,8 +1,9 @@ from __future__ import annotations -from dataclasses import dataclass import json -from typing import Any, Iterable +from collections.abc import Iterable +from dataclasses import dataclass +from typing import Any from .utils import tilde_encode, urlsafe_components @@ -386,7 +387,7 @@ async def count_queries( OR q.sql LIKE :query_search ) """) - params["query_search"] = "%{}%".format(q) + params["query_search"] = f"%{q}%" if is_write is not None: where_clauses.append("q.is_write = :query_is_write") params["query_is_write"] = int(bool(is_write)) @@ -462,7 +463,7 @@ async def list_queries( except ValueError: components = [] if database is None and len(components) == 3: - where_clauses.append(""" + where_clauses.append(f""" ( q.database_name > :cursor_database OR ( @@ -476,12 +477,12 @@ async def list_queries( ) ) ) - """.format(sort_key_sql=sort_key_sql)) + """) params["cursor_database"] = components[0] params["cursor_sort_key"] = components[1] params["cursor_name"] = components[2] elif database is not None and len(components) == 2: - where_clauses.append(""" + where_clauses.append(f""" ( {sort_key_sql} > :cursor_sort_key OR ( @@ -489,7 +490,7 @@ async def list_queries( AND q.name > :cursor_name ) ) - """.format(sort_key_sql=sort_key_sql)) + """) params["cursor_sort_key"] = components[0] params["cursor_name"] = components[1] @@ -502,7 +503,7 @@ async def list_queries( OR q.sql LIKE :query_search ) """) - params["query_search"] = "%{}%".format(q) + params["query_search"] = f"%{q}%" if is_write is not None: where_clauses.append("q.is_write = :query_is_write") params["query_is_write"] = int(bool(is_write)) diff --git a/datasette/tokens.py b/datasette/tokens.py index 4f905339..79f840d2 100644 --- a/datasette/tokens.py +++ b/datasette/tokens.py @@ -10,7 +10,7 @@ from __future__ import annotations import dataclasses import time -from typing import TYPE_CHECKING, Optional +from typing import TYPE_CHECKING import itsdangerous @@ -50,24 +50,24 @@ class TokenRestrictions: database: dict[str, list[str]] = dataclasses.field(default_factory=dict) resource: dict[str, dict[str, list[str]]] = dataclasses.field(default_factory=dict) - def allow_all(self, action: str) -> "TokenRestrictions": + def allow_all(self, action: str) -> TokenRestrictions: """Allow an action across all databases and resources.""" self.all.append(action) return self - def allow_database(self, database: str, action: str) -> "TokenRestrictions": + def allow_database(self, database: str, action: str) -> TokenRestrictions: """Allow an action on a specific database.""" self.database.setdefault(database, []).append(action) return self def allow_resource( self, database: str, resource: str, action: str - ) -> "TokenRestrictions": + ) -> TokenRestrictions: """Allow an action on a specific resource within a database.""" self.resource.setdefault(database, {}).setdefault(resource, []).append(action) return self - def abbreviated(self, datasette: "Datasette") -> Optional[dict]: + def abbreviated(self, datasette: Datasette) -> dict | None: """ Return the abbreviated ``_r`` dictionary shape for this set of restrictions, using action abbreviations registered with ``datasette``. @@ -112,16 +112,16 @@ class TokenHandler: async def create_token( self, - datasette: "Datasette", + datasette: Datasette, actor_id: str, *, - expires_after: Optional[int] = None, - restrictions: Optional[TokenRestrictions] = None, + expires_after: int | None = None, + restrictions: TokenRestrictions | None = None, ) -> str: """Create and return a token string for the given actor.""" raise NotImplementedError - async def verify_token(self, datasette: "Datasette", token: str) -> Optional[dict]: + async def verify_token(self, datasette: Datasette, token: str) -> dict | None: """ Verify a token and return an actor dict. @@ -142,11 +142,11 @@ class SignedTokenHandler(TokenHandler): async def create_token( self, - datasette: "Datasette", + datasette: Datasette, actor_id: str, *, - expires_after: Optional[int] = None, - restrictions: Optional[TokenRestrictions] = None, + expires_after: int | None = None, + restrictions: TokenRestrictions | None = None, ) -> str: if not datasette.setting("allow_signed_tokens"): raise ValueError( @@ -163,7 +163,7 @@ class SignedTokenHandler(TokenHandler): token["_r"] = abbreviated return "dstok_{}".format(datasette.sign(token, namespace="token")) - async def verify_token(self, datasette: "Datasette", token: str) -> Optional[dict]: + async def verify_token(self, datasette: Datasette, token: str) -> dict | None: prefix = "dstok_" if not token.startswith(prefix): @@ -200,9 +200,8 @@ class SignedTokenHandler(TokenHandler): ): duration = max_signed_tokens_ttl - if duration: - if time.time() - created > duration: - raise TokenInvalid("Token has expired") + if duration and time.time() - created > duration: + raise TokenInvalid("Token has expired") actor = {"id": decoded["a"], "token": "dstok"} diff --git a/datasette/tracer.py b/datasette/tracer.py index 28f3cc09..1fbda6f9 100644 --- a/datasette/tracer.py +++ b/datasette/tracer.py @@ -1,10 +1,11 @@ import asyncio +import json +import time +import traceback from contextlib import contextmanager from contextvars import ContextVar + from markupsafe import escape -import time -import json -import traceback tracers = {} @@ -132,17 +133,17 @@ class AsgiTracer: "num_traces": len(traces), "traces": traces, } - try: - content_type = [ + content_type = next( + ( v.decode("utf8") for k, v in response_headers if k.lower() == b"content-type" - ][0] - except IndexError: - content_type = "" + ), + "", + ) if "text/html" in content_type and b"" in accumulated_body: extra = escape(json.dumps(trace_info, indent=2)) - extra_html = f"
{extra}
".encode("utf8") + extra_html = f"
{extra}
".encode() accumulated_body = accumulated_body.replace(b"", extra_html) elif "json" in content_type and accumulated_body.startswith(b"{"): data = json.loads(accumulated_body.decode("utf8")) diff --git a/datasette/url_builder.py b/datasette/url_builder.py index 16b3d42b..f8da20f3 100644 --- a/datasette/url_builder.py +++ b/datasette/url_builder.py @@ -1,6 +1,7 @@ -from .utils import tilde_encode, path_with_format, PrefixedUrlString import urllib +from .utils import PrefixedUrlString, path_with_format, tilde_encode + class Urls: def __init__(self, ds): @@ -8,8 +9,7 @@ class Urls: def path(self, path, format=None): if not isinstance(path, PrefixedUrlString): - if path.startswith("/"): - path = path[1:] + path = path.removeprefix("/") path = self.ds.setting("base_url") + path if format is not None: path = path_with_format(path=path, format=format) @@ -56,6 +56,7 @@ class Urls: return PrefixedUrlString(path) def row_blob(self, database, table, row_path, column): - return self.table(database, table) + "/{}.blob?_blob_column={}".format( - row_path, urllib.parse.quote_plus(column) + return ( + self.table(database, table) + + f"/{row_path}.blob?_blob_column={urllib.parse.quote_plus(column)}" ) diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index 18d3ba52..eb46d549 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -1,29 +1,31 @@ import asyncio +import base64 import binascii -from contextlib import contextmanager -import aiofiles -import click -from collections import OrderedDict, namedtuple, Counter import copy import dataclasses -import base64 import hashlib import inspect import json -import markupsafe -import mergedeep import os import re +import secrets import shlex +import shutil import tempfile -import typing import time import types -import secrets -import shutil -from typing import Iterable, List, Tuple +import typing import urllib +from collections import Counter, OrderedDict, namedtuple +from collections.abc import Iterable +from contextlib import contextmanager + +import aiofiles +import click +import markupsafe +import mergedeep import yaml + from .shutil_backport import copytree from .sqlite import sqlite3, supports_table_xinfo @@ -36,7 +38,7 @@ if typing.TYPE_CHECKING: class PaginatedResources: """Paginated results from allowed_resources query.""" - resources: List["Resource"] + resources: list["Resource"] next: str | None # Keyset token for next page (None if no more results) _datasette: typing.Any = dataclasses.field(default=None, repr=False) _action: str = dataclasses.field(default=None, repr=False) @@ -83,22 +85,132 @@ class PaginatedResources: # From https://www.sqlite.org/lang_keywords.html -reserved_words = set( - ( - "abort action add after all alter analyze and as asc attach autoincrement " - "before begin between by cascade case cast check collate column commit " - "conflict constraint create cross current_date current_time " - "current_timestamp database default deferrable deferred delete desc detach " - "distinct drop each else end escape except exclusive exists explain fail " - "for foreign from full glob group having if ignore immediate in index " - "indexed initially inner insert instead intersect into is isnull join key " - "left like limit match natural no not notnull null of offset on or order " - "outer plan pragma primary query raise recursive references regexp reindex " - "release rename replace restrict right rollback row savepoint select set " - "table temp temporary then to transaction trigger union unique update using " - "vacuum values view virtual when where with without" - ).split() -) +reserved_words = { + "abort", + "action", + "add", + "after", + "all", + "alter", + "analyze", + "and", + "as", + "asc", + "attach", + "autoincrement", + "before", + "begin", + "between", + "by", + "cascade", + "case", + "cast", + "check", + "collate", + "column", + "commit", + "conflict", + "constraint", + "create", + "cross", + "current_date", + "current_time", + "current_timestamp", + "database", + "default", + "deferrable", + "deferred", + "delete", + "desc", + "detach", + "distinct", + "drop", + "each", + "else", + "end", + "escape", + "except", + "exclusive", + "exists", + "explain", + "fail", + "for", + "foreign", + "from", + "full", + "glob", + "group", + "having", + "if", + "ignore", + "immediate", + "in", + "index", + "indexed", + "initially", + "inner", + "insert", + "instead", + "intersect", + "into", + "is", + "isnull", + "join", + "key", + "left", + "like", + "limit", + "match", + "natural", + "no", + "not", + "notnull", + "null", + "of", + "offset", + "on", + "or", + "order", + "outer", + "plan", + "pragma", + "primary", + "query", + "raise", + "recursive", + "references", + "regexp", + "reindex", + "release", + "rename", + "replace", + "restrict", + "right", + "rollback", + "row", + "savepoint", + "select", + "set", + "table", + "temp", + "temporary", + "then", + "to", + "transaction", + "trigger", + "union", + "unique", + "update", + "using", + "vacuum", + "values", + "view", + "virtual", + "when", + "where", + "with", + "without", +} APT_GET_DOCKERFILE_EXTRAS = r""" RUN apt-get update && \ @@ -158,7 +270,7 @@ functions_marked_as_documented = [] def documented(fn=None, *, label=None): def decorate(fn): - fn._datasette_docs_label = label or "internals_utils_{}".format(fn.__name__) + fn._datasette_docs_label = label or f"internals_utils_{fn.__name__}" functions_marked_as_documented.append(fn) return fn @@ -360,7 +472,7 @@ disallawed_sql_res = [ ( re.compile(f"pragma(?!_({'|'.join(allowed_pragmas)}))"), "Statement contained a disallowed PRAGMA. Allowed pragma functions are {}".format( - ", ".join("pragma_{}()".format(pragma) for pragma in allowed_pragmas) + ", ".join(f"pragma_{pragma}()" for pragma in allowed_pragmas) ), ) ] @@ -534,10 +646,7 @@ CMD {cmd}""".format( else "" ), environment_variables="\n".join( - [ - "ENV {} '{}'".format(key, value) - for key, value in environment_variables.items() - ] + [f"ENV {key} '{value}'" for key, value in environment_variables.items()] ), install_from=" ".join(install), files=" ".join(files), @@ -640,7 +749,7 @@ def get_outbound_foreign_keys(conn, table): fks = [] for info in infos: if info is not None: - id, seq, table_name, from_, to_, on_update, on_delete, match = info + id, seq, table_name, from_, to_, _on_update, _on_delete, _match = info fks.append( { "column": from_, @@ -741,7 +850,7 @@ def detect_json1(conn=None): try: conn.execute("SELECT json('{}')") return True - except Exception: + except sqlite3.Error: return False finally: if close_conn: @@ -821,9 +930,7 @@ def is_url(value): if not value.startswith("http://") and not value.startswith("https://"): return False # Any whitespace at all is invalid - if whitespace_re.search(value): - return False - return True + return not whitespace_re.search(value) css_class_re = re.compile(r"^[a-zA-Z]+[_a-zA-Z0-9-]*$") @@ -876,7 +983,9 @@ def module_from_path(path, name): mod.__file__ = path with open(path, "r") as file: code = compile(file.read(), path, "exec", dont_inherit=True) - exec(code, mod.__dict__) + # Executing the file is the whole point - this is how --plugins-dir loads + # plugins and how metadata/config .py files are evaluated + exec(code, mod.__dict__) # noqa: S102 return mod @@ -1033,9 +1142,7 @@ def escape_fts(query): query += '"' bits = _escape_fts_re.split(query) bits = [b for b in bits if b and b != '""'] - return " ".join( - '"{}"'.format(bit) if not bit.startswith('"') else bit for bit in bits - ) + return " ".join(f'"{bit}"' if not bit.startswith('"') else bit for bit in bits) class MultiParams: @@ -1047,7 +1154,7 @@ class MultiParams: data[key], (list, tuple) ), "dictionary data should be a dictionary of key => [list]" self._data = data - elif isinstance(data, list) or isinstance(data, tuple): + elif isinstance(data, (list, tuple)): new_data = {} for item in data: assert ( @@ -1137,9 +1244,7 @@ def _gather_arguments(fn, kwargs): for parameter in parameters: if parameter not in kwargs: raise TypeError( - "{} requires parameters {}, missing: {}".format( - fn, tuple(parameters), set(parameters) - set(kwargs.keys()) - ) + f"{fn} requires parameters {tuple(parameters)}, missing: {set(parameters) - set(kwargs.keys())}" ) call_with.append(kwargs[parameter]) return call_with @@ -1208,9 +1313,9 @@ def resolve_env_secrets(config, environ): """Create copy that recursively replaces {"$env": "NAME"} with values from environ""" if isinstance(config, dict): if list(config.keys()) == ["$env"]: - return environ.get(list(config.values())[0]) + return environ.get(next(iter(config.values()))) elif list(config.keys()) == ["$file"]: - with open(list(config.values())[0]) as fp: + with open(next(iter(config.values()))) as fp: return fp.read() else: return { @@ -1306,7 +1411,7 @@ _named_param_re = re.compile(r":(\w+)") @documented -def named_parameters(sql: str) -> List[str]: +def named_parameters(sql: str) -> list[str]: """ Given a SQL statement, return a list of named parameters that are used in the statement @@ -1319,7 +1424,7 @@ def named_parameters(sql: str) -> List[str]: return _named_param_re.findall(sql) -async def derive_named_parameters(db: "Database", sql: str) -> List[str]: +async def derive_named_parameters(db: "Database", sql: str) -> list[str]: """ This undocumented but stable method exists for backwards compatibility with plugins that were using it before it switched to named_parameters() @@ -1343,9 +1448,9 @@ def parse_size_limit(value, default, maximum, name="_size"): if size < 0: raise ValueError except ValueError: - raise ValueError("{} must be a positive integer".format(name)) + raise ValueError(f"{name} must be a positive integer") if size > maximum: - raise ValueError("{} must be <= {}".format(name, maximum)) + raise ValueError(f"{name} must be <= {maximum}") return size @@ -1403,7 +1508,7 @@ class TildeEncoder(dict): elif b == _space: res = "+" else: - res = "~{:02X}".format(b) + res = f"~{b:02X}" self[b] = res return res @@ -1498,7 +1603,7 @@ def _combine(base: dict, update: dict) -> dict: return base -def pairs_to_nested_config(pairs: typing.List[typing.Tuple[str, typing.Any]]) -> dict: +def pairs_to_nested_config(pairs: list[tuple[str, typing.Any]]) -> dict: """ Parse a list of key-value pairs into a nested dictionary. """ @@ -1513,7 +1618,7 @@ def make_slot_function(name, datasette, request, **kwargs): from datasette.plugins import pm method = getattr(pm.hook, name, None) - assert method is not None, "No hook found for {}".format(name) + assert method is not None, f"No hook found for {name}" async def inner(): html_bits = [] @@ -1537,7 +1642,7 @@ def prune_empty_dicts(d: dict): d.pop(key, None) -def move_plugins_and_allow(source: dict, destination: dict) -> Tuple[dict, dict]: +def move_plugins_and_allow(source: dict, destination: dict) -> tuple[dict, dict]: """ Move 'plugins' and 'allow' keys from source to destination dictionary. Creates hierarchy in destination if needed. After moving, recursively remove any keys diff --git a/datasette/utils/asgi.py b/datasette/utils/asgi.py index 610b86f2..812194fd 100644 --- a/datasette/utils/asgi.py +++ b/datasette/utils/asgi.py @@ -1,28 +1,29 @@ import json -from typing import Optional +import re +from http.cookies import Morsel, SimpleCookie +from mimetypes import guess_type +from pathlib import Path +from urllib.parse import parse_qs, parse_qsl, urlunparse + +import aiofiles +import aiofiles.os + from datasette.utils import MultiParams, calculate_etag, error_body, sha256_file from datasette.utils.multipart import ( - parse_form_data, - MultipartParseError, - FormData, - DEFAULT_MAX_FILE_SIZE, - DEFAULT_MAX_REQUEST_SIZE, - DEFAULT_MAX_FIELDS, - DEFAULT_MAX_FILES, - DEFAULT_MAX_PARTS, DEFAULT_MAX_FIELD_SIZE, + DEFAULT_MAX_FIELDS, + DEFAULT_MAX_FILE_SIZE, + DEFAULT_MAX_FILES, DEFAULT_MAX_MEMORY_FILE_SIZE, DEFAULT_MAX_PART_HEADER_BYTES, DEFAULT_MAX_PART_HEADER_LINES, + DEFAULT_MAX_PARTS, + DEFAULT_MAX_REQUEST_SIZE, DEFAULT_MIN_FREE_DISK_BYTES, + FormData, + MultipartParseError, + parse_form_data, ) -from mimetypes import guess_type -from urllib.parse import parse_qs, urlunparse, parse_qsl -from pathlib import Path -from http.cookies import SimpleCookie, Morsel -import aiofiles -import aiofiles.os -import re # Workaround for adding samesite support to pre 3.8 python Morsel._reserved["samesite"] = "SameSite" @@ -88,7 +89,7 @@ class Request: self.max_post_body_bytes = max_post_body_bytes def __repr__(self): - return ''.format(self.method, self.url) + return f'' @property def method(self): @@ -167,7 +168,7 @@ class Request: if max_bytes is None: max_bytes = self.max_post_body_bytes too_large = PayloadTooLarge( - "Request body exceeded maximum size of {} bytes".format(max_bytes) + f"Request body exceeded maximum size of {max_bytes} bytes" ) if max_bytes: # Reject early if the client declares an oversized body @@ -206,7 +207,7 @@ class Request: max_request_size: int = DEFAULT_MAX_REQUEST_SIZE, max_fields: int = DEFAULT_MAX_FIELDS, max_files: int = DEFAULT_MAX_FILES, - max_parts: Optional[int] = DEFAULT_MAX_PARTS, + max_parts: int | None = DEFAULT_MAX_PARTS, max_field_size: int = DEFAULT_MAX_FIELD_SIZE, max_memory_file_size: int = DEFAULT_MAX_MEMORY_FILE_SIZE, max_part_header_bytes: int = DEFAULT_MAX_PART_HEADER_BYTES, @@ -529,9 +530,9 @@ class Response: httponly=False, samesite="lax", ): - assert samesite in SAMESITE_VALUES, "samesite should be one of {}".format( - SAMESITE_VALUES - ) + assert ( + samesite in SAMESITE_VALUES + ), f"samesite should be one of {SAMESITE_VALUES}" cookie = SimpleCookie() cookie[key] = value for prop_name, prop_value in ( diff --git a/datasette/utils/baseconv.py b/datasette/utils/baseconv.py index c4b64908..0469d7a8 100644 --- a/datasette/utils/baseconv.py +++ b/datasette/utils/baseconv.py @@ -13,7 +13,7 @@ Originally shared here: https://www.djangosnippets.org/snippets/1431/ """ -class BaseConverter(object): +class BaseConverter: decimal_digits = "0123456789" def __init__(self, digits): diff --git a/datasette/utils/check_callable.py b/datasette/utils/check_callable.py index a0997d20..e21a769b 100644 --- a/datasette/utils/check_callable.py +++ b/datasette/utils/check_callable.py @@ -1,6 +1,6 @@ import inspect import types -from typing import NamedTuple, Any +from typing import Any, NamedTuple class CallableStatus(NamedTuple): @@ -19,7 +19,7 @@ def check_callable(obj: Any) -> CallableStatus: if isinstance(obj, types.FunctionType): return CallableStatus(True, inspect.iscoroutinefunction(obj)) - if hasattr(obj, "__call__"): + if callable(obj): return CallableStatus(True, inspect.iscoroutinefunction(obj.__call__)) - assert False, "obj {} is somehow callable with no __call__ method".format(repr(obj)) + assert False, f"obj {obj!r} is somehow callable with no __call__ method" diff --git a/datasette/utils/internal_db.py b/datasette/utils/internal_db.py index 702b53d8..0ddeb847 100644 --- a/datasette/utils/internal_db.py +++ b/datasette/utils/internal_db.py @@ -207,7 +207,8 @@ async def populate_schema_tables(internal_db, db, schema_version): columns = table_column_details(conn, table_name) columns_to_insert.extend( { - **{"database_name": database_name, "table_name": table_name}, + "database_name": database_name, + "table_name": table_name, **column._asdict(), } for column in columns @@ -217,7 +218,8 @@ async def populate_schema_tables(internal_db, db, schema_version): ).fetchall() foreign_keys_to_insert.extend( { - **{"database_name": database_name, "table_name": table_name}, + "database_name": database_name, + "table_name": table_name, **dict(foreign_key), } for foreign_key in foreign_keys @@ -227,7 +229,8 @@ async def populate_schema_tables(internal_db, db, schema_version): ).fetchall() indexes_to_insert.extend( { - **{"database_name": database_name, "table_name": table_name}, + "database_name": database_name, + "table_name": table_name, **dict(index), } for index in indexes @@ -259,7 +262,7 @@ async def populate_schema_tables(internal_db, db, schema_version): "catalog_tables", ): conn.execute( - "DELETE FROM {} WHERE database_name = ?".format(table), + f"DELETE FROM {table} WHERE database_name = ?", [database_name], ) conn.execute( diff --git a/datasette/utils/multipart.py b/datasette/utils/multipart.py index cfa77486..13289b1c 100644 --- a/datasette/utils/multipart.py +++ b/datasette/utils/multipart.py @@ -11,15 +11,10 @@ Supports: import asyncio import shutil import tempfile +from collections.abc import Callable from dataclasses import dataclass, field from typing import ( Any, - Callable, - Dict, - List, - Optional, - Tuple, - Union, ) from urllib.parse import parse_qsl @@ -29,7 +24,7 @@ DEFAULT_MAX_REQUEST_SIZE = 100 * 1024 * 1024 # 100MB DEFAULT_MAX_FIELDS = 1000 DEFAULT_MAX_FILES = 100 # If max_parts is not specified, it defaults to max_fields + max_files -DEFAULT_MAX_PARTS: Optional[int] = None +DEFAULT_MAX_PARTS: int | None = None DEFAULT_MAX_FIELD_SIZE = 100 * 1024 # 100KB DEFAULT_MAX_MEMORY_FILE_SIZE = 1024 * 1024 # 1MB DEFAULT_MAX_PART_HEADER_BYTES = 16 * 1024 # 16KB @@ -40,8 +35,6 @@ DEFAULT_MIN_FREE_DISK_BYTES = 50 * 1024 * 1024 # 50MB class MultipartParseError(Exception): """Raised when multipart parsing fails.""" - pass - @dataclass class UploadedFile: @@ -57,7 +50,7 @@ class UploadedFile: name: str filename: str - content_type: Optional[str] + content_type: str | None size: int _file: tempfile.SpooledTemporaryFile = field(repr=False) @@ -86,7 +79,8 @@ class UploadedFile: def __del__(self): try: self._file.close() - except Exception: + except Exception: # noqa: BLE001, S110 + # __del__ must never raise pass @@ -98,27 +92,27 @@ class FormData: """ def __init__(self): - self._data: List[Tuple[str, Union[str, UploadedFile]]] = [] + self._data: list[tuple[str, str | UploadedFile]] = [] - def append(self, key: str, value: Union[str, UploadedFile]) -> None: + def append(self, key: str, value: str | UploadedFile) -> None: """Add a key-value pair.""" self._data.append((key, value)) - def __getitem__(self, key: str) -> Union[str, UploadedFile]: + def __getitem__(self, key: str) -> str | UploadedFile: """Get the first value for a key.""" for k, v in self._data: if k == key: return v raise KeyError(key) - def get(self, key: str, default: Any = None) -> Optional[Union[str, UploadedFile]]: + def get(self, key: str, default: Any = None) -> str | UploadedFile | None: """Get the first value for a key, or default if not found.""" try: return self[key] except KeyError: return default - def getlist(self, key: str) -> List[Union[str, UploadedFile]]: + def getlist(self, key: str) -> list[str | UploadedFile]: """Get all values for a key.""" return [v for k, v in self._data if k == key] @@ -142,15 +136,15 @@ class FormData: """Return unique keys.""" return list(self) - def items(self) -> List[Tuple[str, Union[str, UploadedFile]]]: + def items(self) -> list[tuple[str, str | UploadedFile]]: """Return all key-value pairs.""" return list(self._data) - def values(self) -> List[Union[str, UploadedFile]]: + def values(self) -> list[str | UploadedFile]: """Return all values.""" return [v for _, v in self._data] - def _uploaded_files(self) -> List[UploadedFile]: + def _uploaded_files(self) -> list[UploadedFile]: """Return UploadedFile instances contained in this form.""" return [v for _, v in self._data if isinstance(v, UploadedFile)] @@ -163,7 +157,7 @@ class FormData: for uploaded in self._uploaded_files(): try: uploaded.close_sync() - except Exception: + except Exception: # noqa: BLE001, S110 # Best-effort cleanup; ignore close errors pass @@ -172,7 +166,7 @@ class FormData: for uploaded in self._uploaded_files(): try: await uploaded.close() - except Exception: + except Exception: # noqa: BLE001, S110 # Best-effort cleanup; ignore close errors pass @@ -189,13 +183,13 @@ class FormData: await self.aclose() -def parse_content_disposition(header: str) -> Dict[str, Optional[str]]: +def parse_content_disposition(header: str) -> dict[str, str | None]: """ Parse Content-Disposition header value. Returns dict with 'name', 'filename' keys (filename may be None). """ - result: Dict[str, Optional[str]] = {"name": None, "filename": None} + result: dict[str, str | None] = {"name": None, "filename": None} # Split on semicolons, handling quoted strings parts = [] @@ -238,7 +232,8 @@ def parse_content_disposition(header: str) -> Dict[str, Optional[str]]: from urllib.parse import unquote result["filename"] = unquote(encoded, encoding="utf-8") - except Exception: + except Exception: # noqa: BLE001, S110 + # Malformed RFC 5987 filename* - fall back to the plain filename pass continue @@ -250,20 +245,19 @@ def parse_content_disposition(header: str) -> Dict[str, Optional[str]]: if key == "name": result["name"] = value - elif key == "filename": - # Only set if filename* hasn't already set it - if result["filename"] is None: - # Strip path components (security) - # Handle both Unix and Windows paths - value = value.replace("\\", "/") - if "/" in value: - value = value.rsplit("/", 1)[-1] - result["filename"] = value + # Only set filename if filename* hasn't already set it + elif key == "filename" and result["filename"] is None: + # Strip path components (security) + # Handle both Unix and Windows paths + value = value.replace("\\", "/") + if "/" in value: + value = value.rsplit("/", 1)[-1] + result["filename"] = value return result -def parse_content_type(header: str) -> Tuple[str, Dict[str, str]]: +def parse_content_type(header: str) -> tuple[str, dict[str, str]]: """ Parse Content-Type header value. @@ -307,7 +301,7 @@ class MultipartParser: max_request_size: int = DEFAULT_MAX_REQUEST_SIZE, max_fields: int = DEFAULT_MAX_FIELDS, max_files: int = DEFAULT_MAX_FILES, - max_parts: Optional[int] = DEFAULT_MAX_PARTS, + max_parts: int | None = DEFAULT_MAX_PARTS, max_field_size: int = DEFAULT_MAX_FIELD_SIZE, max_memory_file_size: int = DEFAULT_MAX_MEMORY_FILE_SIZE, max_part_header_bytes: int = DEFAULT_MAX_PART_HEADER_BYTES, @@ -348,12 +342,12 @@ class MultipartParser: self._tempdir = tempfile.gettempdir() # Current part state - self.current_headers: Dict[str, str] = {} - self.current_file: Optional[tempfile.SpooledTemporaryFile] = None + self.current_headers: dict[str, str] = {} + self.current_file: tempfile.SpooledTemporaryFile | None = None self.current_body = bytearray() - self.current_name: Optional[str] = None - self.current_filename: Optional[str] = None - self.current_content_type: Optional[str] = None + self.current_name: str | None = None + self.current_filename: str | None = None + self.current_content_type: str | None = None def feed(self, chunk: bytes) -> None: """Feed a chunk of data to the parser.""" @@ -454,7 +448,7 @@ class MultipartParser: # Parse header try: line_str = line.decode("utf-8", errors="replace") - except Exception: + except UnicodeDecodeError: line_str = line.decode("latin-1") if ":" in line_str: @@ -481,7 +475,9 @@ class MultipartParser: if self.file_count > self.max_files: raise MultipartParseError("Too many files") if self.handle_files: - self.current_file = tempfile.SpooledTemporaryFile( + # Outlives this method - it is filled in across parser callbacks + # and then handed to the UploadedFile the caller consumes + self.current_file = tempfile.SpooledTemporaryFile( # noqa: SIM115 max_size=self.max_memory_file_size ) else: @@ -644,7 +640,7 @@ async def parse_form_data( max_request_size: int = DEFAULT_MAX_REQUEST_SIZE, max_fields: int = DEFAULT_MAX_FIELDS, max_files: int = DEFAULT_MAX_FILES, - max_parts: Optional[int] = DEFAULT_MAX_PARTS, + max_parts: int | None = DEFAULT_MAX_PARTS, max_field_size: int = DEFAULT_MAX_FIELD_SIZE, max_memory_file_size: int = DEFAULT_MAX_MEMORY_FILE_SIZE, max_part_header_bytes: int = DEFAULT_MAX_PART_HEADER_BYTES, diff --git a/datasette/utils/permissions.py b/datasette/utils/permissions.py index fd1e41a1..5a8ee8e2 100644 --- a/datasette/utils/permissions.py +++ b/datasette/utils/permissions.py @@ -2,8 +2,9 @@ from __future__ import annotations import json -from typing import Any, Dict, Iterable, List, Sequence, Tuple import sqlite3 +from collections.abc import Iterable, Sequence +from typing import Any from datasette.permissions import PermissionSQL from datasette.plugins import pm @@ -15,7 +16,7 @@ SKIP_PERMISSION_CHECKS = object() async def gather_permission_sql_from_hooks( *, datasette, actor: dict | None, action: str -) -> List[PermissionSQL] | object: +) -> list[PermissionSQL] | object: """Collect PermissionSQL objects from the permission_resources_sql hook. Ensures that each returned PermissionSQL has a populated ``source``. @@ -34,7 +35,7 @@ async def gather_permission_sql_from_hooks( hookimpls = hook_caller.get_hookimpls() hook_results = list(hook_caller(datasette=datasette, actor=actor, action=action)) - collected: List[PermissionSQL] = [] + collected: list[PermissionSQL] = [] actor_json = json.dumps(actor) if actor is not None else None actor_id = actor.get("id") if isinstance(actor, dict) else None @@ -71,7 +72,7 @@ def _iter_permission_sql_from_result( if isinstance(result, PermissionSQL): return [result] if isinstance(result, (list, tuple)): - collected: List[PermissionSQL] = [] + collected: list[PermissionSQL] = [] for item in result: collected.extend(_iter_permission_sql_from_result(item, action=action)) return collected @@ -90,7 +91,7 @@ def _iter_permission_sql_from_result( def build_rules_union( actor: dict | None, plugins: Sequence[PermissionSQL] -) -> Tuple[str, Dict[str, Any]]: +) -> tuple[str, dict[str, Any]]: """ Compose plugin SQL into a UNION ALL. @@ -102,10 +103,10 @@ def build_rules_union( The system reserves these parameter names: :actor, :actor_id, :action, :filter_parent Plugin parameters should be prefixed with a unique identifier (e.g., source name). """ - parts: List[str] = [] + parts: list[str] = [] actor_json = json.dumps(actor) if actor else None actor_id = actor.get("id") if actor else None - params: Dict[str, Any] = {"actor": actor_json, "actor_id": actor_id} + params: dict[str, Any] = {"actor": actor_json, "actor_id": actor_id} for p in plugins: # No namespacing - just use plugin params as-is @@ -141,10 +142,10 @@ async def resolve_permissions_from_catalog( plugins: Sequence[Any], action: str, candidate_sql: str, - candidate_params: Dict[str, Any] | None = None, + candidate_params: dict[str, Any] | None = None, *, implicit_deny: bool = True, -) -> List[Dict[str, Any]]: +) -> list[dict[str, Any]]: """ Resolve permissions by embedding the provided *candidate_sql* in a CTE. @@ -168,8 +169,8 @@ async def resolve_permissions_from_catalog( - parent, child, allow, reason, source_plugin, depth - resource (rendered "/parent/child" or "/parent" or "/") """ - resolved_plugins: List[PermissionSQL] = [] - restriction_sqls: List[str] = [] + resolved_plugins: list[PermissionSQL] = [] + restriction_sqls: list[str] = [] for plugin in plugins: if callable(plugin) and not isinstance(plugin, PermissionSQL): @@ -398,11 +399,11 @@ async def resolve_permissions_with_candidates( db, actor: dict | None, plugins: Sequence[Any], - candidates: List[Tuple[str, str | None]], + candidates: list[tuple[str, str | None]], action: str, *, implicit_deny: bool = True, -) -> List[Dict[str, Any]]: +) -> list[dict[str, Any]]: """ Resolve permissions without any external candidate table by embedding the candidates as a UNION of parameterized SELECTs in a CTE. @@ -411,8 +412,8 @@ async def resolve_permissions_with_candidates( actor: actor dict (or None), made available as :actor (JSON), :actor_id, and :action """ # Build a small CTE for candidates. - cand_rows_sql: List[str] = [] - cand_params: Dict[str, Any] = {} + cand_rows_sql: list[str] = [] + cand_params: dict[str, Any] = {} for i, (parent, child) in enumerate(candidates): pkey = f"cand_p_{i}" ckey = f"cand_c_{i}" diff --git a/datasette/utils/shutil_backport.py b/datasette/utils/shutil_backport.py index d1fd1bd7..d323f5d6 100644 --- a/datasette/utils/shutil_backport.py +++ b/datasette/utils/shutil_backport.py @@ -6,7 +6,7 @@ https://github.com/python/cpython/blob/v3.8.3/LICENSE """ import os -from shutil import copy, copy2, copystat, Error +from shutil import Error, copy, copy2, copystat def _copytree( diff --git a/datasette/utils/sql_analysis.py b/datasette/utils/sql_analysis.py index 1be28982..334545bd 100644 --- a/datasette/utils/sql_analysis.py +++ b/datasette/utils/sql_analysis.py @@ -413,12 +413,12 @@ def analyze_sql_tables( database=None, table=None, sqlite_schema=sqlite_schema, - target="{} {}".format(arg1, arg2) if arg2 is not None else arg1, + target=f"{arg1} {arg2}" if arg2 is not None else arg1, source=source, ) return sqlite3.SQLITE_OK - action_name = _AUTHORIZER_ACTION_NAMES.get(action, "SQLITE_{}".format(action)) + action_name = _AUTHORIZER_ACTION_NAMES.get(action, f"SQLITE_{action}") record( "unknown", "unknown", @@ -521,9 +521,7 @@ def analyze_sql_tables( and key.target in _SQLITE_INTERNAL_SCHEMA_FUNCTIONS ): return True - if key_is_drop_table_delete(key): - return True - return False + return bool(key_is_drop_table_delete(key)) def table_kind_for(key: OperationKey) -> SQLiteTableType | None: if ( diff --git a/datasette/utils/sqlite.py b/datasette/utils/sqlite.py index 4743ae4c..d3926f6f 100644 --- a/datasette/utils/sqlite.py +++ b/datasette/utils/sqlite.py @@ -100,7 +100,7 @@ def sqlite_hidden_table_names(conn, *, schema: str | None = "main") -> list[str] schema_table = _sqlite_schema_table(schema) try: rows = conn.execute( - "select name, sql from {} where type = 'table'".format(schema_table) + f"select name, sql from {schema_table} where type = 'table'" ).fetchall() except sqlite3.DatabaseError: return [] @@ -127,7 +127,7 @@ def _sqlite_table_type_from_schema( schema_table = _sqlite_schema_table(schema) try: row = conn.execute( - "select type, sql from {} where name = ?".format(schema_table), + f"select type, sql from {schema_table} where name = ?", (table,), ).fetchone() except sqlite3.DatabaseError: @@ -155,7 +155,7 @@ def _is_known_shadow_table( schema_table = _sqlite_schema_table(schema) try: rows = conn.execute( - "select name, sql from {} where type = 'table'".format(schema_table) + f"select name, sql from {schema_table} where type = 'table'" ).fetchall() except sqlite3.DatabaseError: return False @@ -174,7 +174,7 @@ def _sqlite_schema_table(schema: str | None) -> str: return "sqlite_master" if schema == "temp": return "sqlite_temp_master" - return "{}.sqlite_master".format(_quote_identifier(schema)) + return f"{_quote_identifier(schema)}.sqlite_master" def _quote_identifier(value: str) -> str: diff --git a/datasette/utils/testing.py b/datasette/utils/testing.py index de7e94af..a8be47bf 100644 --- a/datasette/utils/testing.py +++ b/datasette/utils/testing.py @@ -1,6 +1,7 @@ -from asgiref.sync import async_to_sync -from urllib.parse import urlencode import json +from urllib.parse import urlencode + +from asgiref.sync import async_to_sync # These wrapper classes pre-date the introduction of # datasette.client and httpx to Datasette. They could diff --git a/datasette/views/__init__.py b/datasette/views/__init__.py index ed7e175f..bac3b39e 100644 --- a/datasette/views/__init__.py +++ b/datasette/views/__init__.py @@ -1,7 +1,7 @@ -from dataclasses import dataclass import dataclasses import types import typing +from dataclasses import dataclass @dataclass(frozen=True) @@ -74,16 +74,14 @@ class Context: extra_class = table_extra_registry.classes_by_name[name] except KeyError: raise KeyError( - "{}.{} is declared with from_extra() but there is no " - "registered extra of that name".format(cls.__name__, name) + f"{cls.__name__}.{name} is declared with from_extra() but there is no " + "registered extra of that name" ) if cls.extras_scope is not None and not extra_class.available_for( cls.extras_scope ): raise ValueError( - "{}.{} is declared with from_extra() but the {} extra is " - "not available for scope {}".format( - cls.__name__, name, name, cls.extras_scope - ) + f"{cls.__name__}.{name} is declared with from_extra() but the {name} extra is " + f"not available for scope {cls.extras_scope}" ) return extra_class.description or "" diff --git a/datasette/views/base.py b/datasette/views/base.py index 66e14a6d..48108cec 100644 --- a/datasette/views/base.py +++ b/datasette/views/base.py @@ -2,20 +2,20 @@ import csv import hashlib import sys -from datasette.utils.asgi import Request from datasette.utils import ( - add_cors_headers, EscapeHtmlWriter, InvalidSql, LimitedWriter, + add_cors_headers, path_from_row_pks, path_with_format, sqlite3, ) from datasette.utils.asgi import ( AsgiStream, - Response, BadRequest, + Request, + Response, ) @@ -129,12 +129,10 @@ class BaseView: template = environment.select_template(templates) template_context = { **context, - **{ - "select_templates": [ - f"{'*' if template_name == template.name else ''}{template_name}" - for template_name in templates - ], - }, + "select_templates": [ + f"{'*' if template_name == template.name else ''}{template_name}" + for template_name in templates + ], } headers = {} if self.has_json_alternate: @@ -151,9 +149,7 @@ class BaseView: template_context["alternate_url_json"] = alternate_url_json headers.update( { - "Link": '<{}>; rel="alternate"; type="application/json+datasette"'.format( - alternate_url_json - ) + "Link": f'<{alternate_url_json}>; rel="alternate"; type="application/json+datasette"' } ) return Response.html( @@ -184,9 +180,7 @@ async def stream_csv(datasette, fetch_data, request, database): stream = request.args.get("_stream") # Do not calculate facets or counts: extra_parameters = [ - "{}=1".format(key) - for key in ("_nofacet", "_nocount") - if not request.args.get(key) + f"{key}=1" for key in ("_nofacet", "_nocount") if not request.args.get(key) ] if extra_parameters: # Replace request object with a new one with modified scope @@ -216,9 +210,6 @@ async def stream_csv(datasette, fetch_data, request, database): except (sqlite3.OperationalError, InvalidSql) as e: raise DatasetteError(str(e), title="Invalid SQL", status=400) - except sqlite3.OperationalError as e: - raise DatasetteError(str(e)) - except DatasetteError: raise @@ -325,8 +316,9 @@ async def stream_csv(datasette, fetch_data, request, database): else: new_row.append(cell) await writer.writerow(new_row) - except Exception as ex: - sys.stderr.write("Caught this error: {}\n".format(ex)) + except Exception as ex: # noqa: BLE001 + # Streaming CSV: report the error into the response body and stop + sys.stderr.write(f"Caught this error: {ex}\n") sys.stderr.flush() await r.write(str(ex)) return diff --git a/datasette/views/database.py b/datasette/views/database.py index 11646f45..f54ffd38 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -1,49 +1,52 @@ -from dataclasses import asdict, dataclass, field -from urllib.parse import parse_qsl, urlencode import asyncio import hashlib import itertools import json -import markupsafe import os import textwrap +from dataclasses import asdict, dataclass, field +from urllib.parse import parse_qsl, urlencode + +import markupsafe -from datasette.extras import extra_names_from_request, ExtraScope from datasette.database import QueryInterrupted +from datasette.extras import ExtraScope, extra_names_from_request +from datasette.plugins import pm from datasette.resources import DatabaseResource, QueryResource from datasette.stored_queries import StoredQuery, stored_query_to_dict -from datasette.write_sql import QueryWriteRejected from datasette.utils import ( + InvalidSql, add_cors_headers, await_me_maybe, - error_body, call_with_supported_arguments, - named_parameters as derive_named_parameters, + error_body, format_bytes, - make_slot_function, - tilde_decode, - to_css_class, - validate_sql_select, is_url, + make_slot_function, path_with_added_args, path_with_format, path_with_removed_args, sqlite3, + tilde_decode, + to_css_class, truncate_url, - InvalidSql, + validate_sql_select, ) -from datasette.utils.asgi import AsgiFileDownload, NotFound, Response, Forbidden -from datasette.plugins import pm +from datasette.utils import ( + named_parameters as derive_named_parameters, +) +from datasette.utils.asgi import AsgiFileDownload, Forbidden, NotFound, Response +from datasette.write_sql import QueryWriteRejected +from . import Context from .base import DatasetteError, View, stream_csv from .query_helpers import _ensure_stored_query_execution_permissions, _table_columns +from .table_create_alter import _create_table_ui_context from .table_extras import ( QueryExtraContext, resolve_query_extras, table_extra_registry, ) -from .table_create_alter import _create_table_ui_context -from . import Context @dataclass @@ -100,7 +103,7 @@ class DatabaseView(View): return response if format_ not in ("html", "json"): - raise NotFound("Invalid format: {}".format(format_)) + raise NotFound(f"Invalid format: {format_}") metadata = await datasette.get_database_metadata(database) @@ -164,7 +167,7 @@ class DatabaseView(View): "label": "Create table", "description": "Create a new table in this database.", "attrs": { - "aria-label": "Create table in {}".format(database), + "aria-label": f"Create table in {database}", "data-database-action": "create-table", }, } @@ -271,9 +274,7 @@ class DatabaseView(View): view_name="database", ), headers={ - "Link": '<{}>; rel="alternate"; type="application/json+datasette"'.format( - alternate_url_json - ) + "Link": f'<{alternate_url_json}>; rel="alternate"; type="application/json+datasette"' }, ) @@ -556,7 +557,7 @@ async def database_download(request, datasette): if datasette.cors: add_cors_headers(headers) if db.hash: - etag = '"{}"'.format(db.hash) + etag = f'"{db.hash}"' headers["Etag"] = etag # Has user seen this already? if_none_match = request.headers.get("if-none-match") @@ -664,8 +665,9 @@ class QueryView(View): ).first() if message_result: message = message_result[0] - except Exception as ex: - message = "Error running on_success_message_sql: {}".format(ex) + except Exception as ex: # noqa: BLE001 + # Stored-query on_success_message_sql is user-authored + message = f"Error running on_success_message_sql: {ex}" message_type = datasette.ERROR if not message: if stored_query.on_success_message: @@ -679,7 +681,8 @@ class QueryView(View): redirect_url = stored_query.on_success_redirect ok = True - except Exception as ex: + except Exception as ex: # noqa: BLE001 + # Stored-query execution is user-authored SQL message = stored_query.on_error_message or str(ex) message_type = datasette.ERROR redirect_url = stored_query.on_error_redirect @@ -813,16 +816,16 @@ class QueryView(View): rows = results.rows except QueryInterrupted as ex: raise DatasetteError( - textwrap.dedent(""" + textwrap.dedent(f"""

SQL query took too long. The time limit is controlled by the sql_time_limit_ms configuration option.

- + - """.format(markupsafe.escape(ex.sql))).strip(), + """).strip(), title="SQL Interrupted", status=400, message_is_html=True, @@ -838,8 +841,6 @@ class QueryView(View): columns = [] except (sqlite3.OperationalError, InvalidSql) as ex: raise DatasetteError(str(ex), title="Invalid SQL", status=400) - except sqlite3.OperationalError as ex: - raise DatasetteError(str(ex)) except DatasetteError: raise @@ -861,7 +862,7 @@ class QueryView(View): return data, None, None return await stream_csv(datasette, fetch_data_for_csv, request, db.name) - elif format_ in datasette.renderers.keys(): + elif format_ in datasette.renderers: if not sql: raise DatasetteError("?sql= is required", status=400) data = {"ok": True, "rows": rows, "columns": columns} @@ -953,9 +954,7 @@ class QueryView(View): } headers.update( { - "Link": '<{}>; rel="alternate"; type="application/json+datasette"'.format( - alternate_url_json - ) + "Link": f'<{alternate_url_json}>; rel="alternate"; type="application/json+datasette"' } ) metadata = await query_metadata() @@ -1036,9 +1035,7 @@ class QueryView(View): + "?" + urlencode( { - **{ - "sql": sql, - }, + "sql": sql, **named_parameter_values, } ) @@ -1140,7 +1137,7 @@ class QueryView(View): headers=headers, ) else: - assert False, "Invalid format: {}".format(format_) + assert False, f"Invalid format: {format_}" if datasette.cors: add_cors_headers(r.headers) return r @@ -1241,7 +1238,7 @@ async def display_rows(datasette, database, request, rows, columns): '<Binary: {:,} byte{}>'.format( blob_url, ( - ' title="{}"'.format(formatted) + f' title="{formatted}"' if "bytes" not in formatted else "" ), diff --git a/datasette/views/execute_write.py b/datasette/views/execute_write.py index dd35b127..3dfb810c 100644 --- a/datasette/views/execute_write.py +++ b/datasette/views/execute_write.py @@ -8,8 +8,8 @@ from datasette.utils.asgi import Response from .base import BaseView from .database import display_rows as display_query_rows from .query_helpers import ( - QueryValidationError, SQL_PARAMETER_FORM_PREFIX, + QueryValidationError, _analysis_is_write, _analysis_rows, _analysis_rows_with_permissions, @@ -31,15 +31,7 @@ WRITE_TEMPLATE_LABELS = { "delete": "Delete rows", } WRITE_TEMPLATE_OPERATIONS = tuple(WRITE_TEMPLATE_LABELS) -CREATE_TABLE_TEMPLATE_SQL = "\n".join( - ( - "create table new_table (", - " id integer primary key,", - " name text", - " -- created text default (datetime('now'))", - ")", - ) -) +CREATE_TABLE_TEMPLATE_SQL = "create table new_table (\n id integer primary key,\n name text\n -- created text default (datetime('now'))\n)" def _parameter_names(columns): @@ -49,11 +41,11 @@ def _parameter_names(columns): base = re.sub(r"[^a-z0-9_]+", "_", column.lower()) base = base.strip("_") or "value" if base[0].isdigit(): - base = "p_{}".format(base) + base = f"p_{base}" name = base index = 2 while name in seen: - name = "{}_{}".format(base, index) + name = f"{base}_{index}" index += 1 seen.add(name) names[column] = name @@ -65,7 +57,7 @@ def _quote_identifier(identifier): def _preferred_where_column(table, columns): - lower_table_id = "{}_id".format(table.lower()) + lower_table_id = f"{table.lower()}_id" return ( next((column for column in columns if column.lower() == "id"), None) or next( @@ -90,17 +82,15 @@ def _insert_template_sql(table, columns): auto_pk = _auto_incrementing_primary_key(columns) insert_columns = [column for column in column_names if column != auto_pk] if not insert_columns: - return "insert into {}\ndefault values".format(_quote_identifier(table)) + return f"insert into {_quote_identifier(table)}\ndefault values" names = _parameter_names(insert_columns) return "\n".join( ( - "insert into {} (".format(_quote_identifier(table)), - ",\n".join( - " {}".format(_quote_identifier(column)) for column in insert_columns - ), + f"insert into {_quote_identifier(table)} (", + ",\n".join(f" {_quote_identifier(column)}" for column in insert_columns), ")", "values (", - ",\n".join(" :{}".format(names[column]) for column in insert_columns), + ",\n".join(f" :{names[column]}" for column in insert_columns), ")", ) ) @@ -114,18 +104,14 @@ def _update_template_sql(table, columns): if not set_columns: return "\n".join( ( - "update {}".format(_quote_identifier(table)), - "set {} = :new_{}".format( - _quote_identifier(where_column), names[where_column] - ), - "where {} = :{}".format( - _quote_identifier(where_column), names[where_column] - ), + f"update {_quote_identifier(table)}", + f"set {_quote_identifier(where_column)} = :new_{names[where_column]}", + f"where {_quote_identifier(where_column)} = :{names[where_column]}", ) ) return "\n".join( ( - "update {}".format(_quote_identifier(table)), + f"update {_quote_identifier(table)}", "set " + ",\n".join( "{}{} = :{}".format( @@ -135,9 +121,7 @@ def _update_template_sql(table, columns): ) for index, column in enumerate(set_columns) ), - "where {} = :{}".format( - _quote_identifier(where_column), names[where_column] - ), + f"where {_quote_identifier(where_column)} = :{names[where_column]}", ) ) @@ -148,10 +132,8 @@ def _delete_template_sql(table, columns): where_column = _preferred_where_column(table, column_names) return "\n".join( ( - "delete from {}".format(_quote_identifier(table)), - "where {} = :{}".format( - _quote_identifier(where_column), names[where_column] - ), + f"delete from {_quote_identifier(table)}", + f"where {_quote_identifier(where_column)} = :{names[where_column]}", ) ) diff --git a/datasette/views/index.py b/datasette/views/index.py index 67296cd1..f73ee38a 100644 --- a/datasette/views/index.py +++ b/datasette/views/index.py @@ -2,11 +2,11 @@ import json from datasette.plugins import pm from datasette.utils import ( + UNSTABLE_API_MESSAGE, + CustomJSONEncoder, add_cors_headers, await_me_maybe, make_slot_function, - CustomJSONEncoder, - UNSTABLE_API_MESSAGE, ) from datasette.utils.asgi import Response from datasette.version import __version__ @@ -46,15 +46,15 @@ class IndexView(BaseView): databases = [] # Iterate over allowed databases instead of all databases - for name in allowed_db_dict.keys(): + for name, allowed_db in allowed_db_dict.items(): db = self.ds.databases[name] - database_private = allowed_db_dict[name].private + database_private = allowed_db.private # Get allowed tables/views for this database allowed_for_db = tables_by_db.get(name, {}) # Get table names from allowed set instead of db.table_names() - table_names = [child_name for child_name in allowed_for_db.keys()] + table_names = [child_name for child_name in allowed_for_db] hidden_table_names = set(await db.hidden_table_names()) @@ -99,7 +99,7 @@ class IndexView(BaseView): # We will be sorting by number of relationships, so populate that field all_foreign_keys = await db.get_all_foreign_keys() for table, foreign_keys in all_foreign_keys.items(): - if table in tables.keys(): + if table in tables: count = len(foreign_keys["incoming"] + foreign_keys["outgoing"]) tables[table]["num_relationships_for_sorting"] = count @@ -121,8 +121,7 @@ class IndexView(BaseView): # Only add views if this is less than TRUNCATE_AT if len(tables_and_views_truncated) < TRUNCATE_AT: num_views_to_add = TRUNCATE_AT - len(tables_and_views_truncated) - for view in views[:num_views_to_add]: - tables_and_views_truncated.append(view) + tables_and_views_truncated.extend(views[:num_views_to_add]) databases.append( { diff --git a/datasette/views/query_helpers.py b/datasette/views/query_helpers.py index 588891d4..725d9cdb 100644 --- a/datasette/views/query_helpers.py +++ b/datasette/views/query_helpers.py @@ -5,6 +5,19 @@ from datasette.resources import DatabaseResource from datasette.stored_queries import ( StoredQuery, ) +from datasette.utils import ( + InvalidSql, + escape_sqlite, + parse_size_limit, + path_from_row_pks, + sqlite3, + validate_sql_select, +) +from datasette.utils import ( + named_parameters as derive_named_parameters, +) +from datasette.utils.asgi import Forbidden +from datasette.utils.sql_analysis import Operation, SQLAnalysis from datasette.write_sql import ( IgnoreWriteSqlOperation, QueryWriteRejected, @@ -12,17 +25,6 @@ from datasette.write_sql import ( decision_for_write_sql_operation, operation_is_write, ) -from datasette.utils import ( - parse_size_limit, - named_parameters as derive_named_parameters, - escape_sqlite, - path_from_row_pks, - sqlite3, - validate_sql_select, - InvalidSql, -) -from datasette.utils.asgi import Forbidden -from datasette.utils.sql_analysis import Operation, SQLAnalysis _query_name_re = re.compile(r"^[^/\.\n]+$") @@ -91,7 +93,7 @@ def _as_optional_bool(value, name): return True if lowered in {"0", "false", "f", "no", "off"}: return False - raise QueryValidationError("{} must be 0 or 1".format(name)) + raise QueryValidationError(f"{name} must be 0 or 1") def _query_list_limit(value, default, maximum): @@ -171,7 +173,7 @@ async def _json_or_form_payload(request): try: return json.loads(body or b"{}"), True except json.JSONDecodeError as e: - raise QueryValidationError("Invalid JSON: {}".format(e)) + raise QueryValidationError(f"Invalid JSON: {e}") return await request.post_vars(), False @@ -192,7 +194,7 @@ async def _analyze_user_query(datasette, db, sql, *, actor): try: analysis = await db.analyze_sql(sql, params) except sqlite3.DatabaseError as ex: - raise QueryValidationError("Could not analyze query: {}".format(ex)) from ex + raise QueryValidationError(f"Could not analyze query: {ex}") from ex is_write = _analysis_is_write(analysis) if is_write: @@ -293,8 +295,7 @@ def _coerce_execute_write_payload(data, is_json): for key, value in data.items(): if key in {"sql", "csrftoken", "_json"}: continue - if key.startswith(SQL_PARAMETER_FORM_PREFIX): - key = key[len(SQL_PARAMETER_FORM_PREFIX) :] + key = key.removeprefix(SQL_PARAMETER_FORM_PREFIX) params[key] = value if not isinstance(params, dict): raise QueryValidationError("params must be a dictionary") @@ -314,7 +315,7 @@ async def _prepare_execute_write(datasette, db, sql, params, actor): try: analysis = await db.analyze_sql(sql, params) except sqlite3.DatabaseError as ex: - raise QueryValidationError("Could not analyze query: {}".format(ex)) from ex + raise QueryValidationError(f"Could not analyze query: {ex}") from ex if not _analysis_is_write(analysis): raise QueryValidationError( "Use /-/query for read-only SQL; this endpoint only executes writes" @@ -496,7 +497,7 @@ async def _inserted_row_url(datasette, db, analysis, cursor): ) try: result = await db.execute( - "select {} from {} where rowid = ?".format(select, escape_sqlite(table)), + f"select {select} from {escape_sqlite(table)} where rowid = ?", [lastrowid], ) except sqlite3.DatabaseError: diff --git a/datasette/views/row.py b/datasette/views/row.py index c90a3bbe..b1388299 100644 --- a/datasette/views/row.py +++ b/datasette/views/row.py @@ -8,34 +8,35 @@ from dataclasses import dataclass, field import markupsafe import sqlite_utils -from datasette.utils.asgi import NotFound, Forbidden, PayloadTooLarge, Response from datasette.database import QueryInterrupted -from datasette.events import UpdateRowEvent, DeleteRowEvent +from datasette.events import DeleteRowEvent, UpdateRowEvent +from datasette.extras import ExtraScope, extra_names_from_request +from datasette.plugins import pm from datasette.resources import TableResource -from .base import BaseView, DatasetteError, stream_csv from datasette.utils import ( + CustomJSONEncoder, + CustomRow, + InvalidSql, + WriteJsonValueError, add_cors_headers, await_me_maybe, call_with_supported_arguments, - CustomJSONEncoder, - CustomRow, decode_write_json_row, - InvalidSql, + escape_sqlite, make_slot_function, path_from_row_pks, path_with_format, path_with_removed_args, - to_css_class, - escape_sqlite, sqlite3, - WriteJsonValueError, + to_css_class, ) -from datasette.plugins import pm -from datasette.extras import extra_names_from_request, ExtraScope +from datasette.utils.asgi import Forbidden, NotFound, PayloadTooLarge, Response + from . import Context, from_extra +from .base import BaseView, DatasetteError, stream_csv from .table import ( - display_columns_and_rows, _table_page_data, + display_columns_and_rows, row_label_from_label_column, ) from .table_extras import RowExtraContext, resolve_row_extras, table_extra_registry @@ -187,16 +188,16 @@ class RowView(BaseView): data, extra_template_data, templates = response_or_template_contexts except QueryInterrupted as ex: raise DatasetteError( - textwrap.dedent(""" + textwrap.dedent(f"""

SQL query took too long. The time limit is controlled by the sql_time_limit_ms configuration option.

- + - """.format(markupsafe.escape(ex.sql))).strip(), + """).strip(), title="SQL Interrupted", status=400, message_is_html=True, @@ -207,15 +208,13 @@ class RowView(BaseView): ) except (sqlite3.OperationalError, InvalidSql) as e: raise DatasetteError(str(e), title="Invalid SQL", status=400) - except sqlite3.OperationalError as e: - raise DatasetteError(str(e)) except DatasetteError: raise end = time.perf_counter() data["query_ms"] = (end - start) * 1000 - if format_ in self.ds.renderers.keys(): + if format_ in self.ds.renderers: # Dispatch request to the correct output format renderer # (CSV is not handled here due to streaming) result = call_with_supported_arguments( @@ -258,7 +257,7 @@ class RowView(BaseView): if status_code is not None: response.status = status_code else: - raise NotFound("Invalid format: {}".format(format_)) + raise NotFound(f"Invalid format: {format_}") ttl = request.args.get("_ttl", None) if ttl is None or not ttl.isdigit(): @@ -373,9 +372,7 @@ class RowView(BaseView): view_name=self.name, ), headers={ - "Link": '<{}>; rel="alternate"; type="application/json+datasette"'.format( - alternate_url_json - ) + "Link": f'<{alternate_url_json}>; rel="alternate"; type="application/json+datasette"' }, ) @@ -500,7 +497,7 @@ class RowView(BaseView): row_action_label = pk_path if row_label and row_label != pk_path: - row_action_label = "{} {}".format(pk_path, row_label) + row_action_label = f"{pk_path} {row_label}" row_action_permissions = {} if is_table and db.is_mutable: @@ -513,7 +510,7 @@ class RowView(BaseView): row_actions = [] if row_action_permissions.get("update-row"): attrs = { - "aria-label": "Edit row {}".format(row_action_label), + "aria-label": f"Edit row {row_action_label}", "data-row": row_path, "data-row-action": "edit", } @@ -529,7 +526,7 @@ class RowView(BaseView): ) if row_action_permissions.get("delete-row"): attrs = { - "aria-label": "Delete row {}".format(row_action_label), + "aria-label": f"Delete row {row_action_label}", "data-row": row_path, "data-row-action": "delete", } @@ -679,7 +676,7 @@ class RowView(BaseView): key, ",".join(pk_values), ) - foreign_key_tables.append({**fk, **{"count": count, "link": link}}) + foreign_key_tables.append({**fk, "count": count, "link": link}) return foreign_key_tables @@ -705,23 +702,21 @@ async def _row_flash_message(db, action, resolved, row=None): if label: label = _truncated_row_flash_label(label) if label and label != pk_label: - return "{} row {} ({})".format(action, pk_label, label) - return "{} row {}".format(action, pk_label) + return f"{action} row {pk_label} ({label})" + return f"{action} row {pk_label}" async def _resolve_row_and_check_permission(datasette, request, permission): - from datasette.app import DatabaseNotFound, TableNotFound, RowNotFound + from datasette.app import DatabaseNotFound, RowNotFound, TableNotFound try: resolved = await datasette.resolve_row(request) except DatabaseNotFound as e: - return False, Response.error( - ["Database not found: {}".format(e.database_name)], 404 - ) + return False, Response.error([f"Database not found: {e.database_name}"], 404) except TableNotFound as e: - return False, Response.error(["Table not found: {}".format(e.table)], 404) + return False, Response.error([f"Table not found: {e.table}"], 404) except RowNotFound as e: - return False, Response.error(["Record not found: {}".format(e.pk_values)], 404) + return False, Response.error([f"Record not found: {e.pk_values}"], 404) # Ensure user has permission to delete this row if not await datasette.allowed( @@ -753,7 +748,8 @@ class RowDeleteView(BaseView): try: await resolved.db.execute_write_fn(delete_row, request=request) - except Exception as e: + except Exception as e: # noqa: BLE001 + # TODO: narrow to expected write errors so Datasette bugs surface as 500s return Response.error([str(e)], 400) await self.ds.track_event( @@ -793,7 +789,7 @@ class RowUpdateView(BaseView): try: data = await request.json() except json.JSONDecodeError as e: - return Response.error(["Invalid JSON: {}".format(e)]) + return Response.error([f"Invalid JSON: {e}"]) except PayloadTooLarge as e: return Response.error([str(e)], 413) @@ -836,7 +832,8 @@ class RowUpdateView(BaseView): try: await resolved.db.execute_write_fn(update_row, request=request) - except Exception as e: + except Exception as e: # noqa: BLE001 + # TODO: narrow to expected write errors so Datasette bugs surface as 500s return Response.error([str(e)], 400) result = {"ok": True} diff --git a/datasette/views/special.py b/datasette/views/special.py index 28d34208..a77f221f 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -1,23 +1,25 @@ import json import logging +import secrets +import urllib + +from datasette.events import CreateTokenEvent, LoginEvent, LogoutEvent from datasette.jump import JumpSQL, namespace_sql_params from datasette.plugins import pm -from datasette.events import LogoutEvent, LoginEvent, CreateTokenEvent from datasette.resources import DatabaseResource, TableResource -from datasette.utils.asgi import Response, Forbidden from datasette.utils import ( UNSTABLE_API_MESSAGE, actor_matches_allow, - parse_size_limit, add_cors_headers, await_me_maybe, error_body, - tilde_encode, + parse_size_limit, tilde_decode, + tilde_encode, ) +from datasette.utils.asgi import Forbidden, Response + from .base import BaseView, View -import secrets -import urllib logger = logging.getLogger(__name__) @@ -179,9 +181,7 @@ class AutocompleteDebugView(BaseView): ) context.update( { - "autocomplete_url": "{}/-/autocomplete".format( - self.ds.urls.table(database_name, table_name) - ), + "autocomplete_url": f"{self.ds.urls.table(database_name, table_name)}/-/autocomplete", "label_column": await db.label_column_for_table(table_name), } ) @@ -420,8 +420,11 @@ class AllowedResourcesView(BaseView): row["reason"] = resource.reasons allowed_rows.append(row) - except Exception: - # If catalog tables don't exist yet, return empty results + except Exception: # noqa: BLE001 + # Returns empty results if the catalog tables don't exist yet, but + # also swallows the AttributeError raised for instance-level actions + # such as view-instance, which have no resource_class. + # TODO: handle that case explicitly and narrow this to sqlite3.Error return ( { "ok": True, @@ -523,7 +526,7 @@ class PermissionRulesView(BaseView): from datasette.utils.actions_sql import build_permission_rules_sql - union_sql, union_params, restriction_sqls = await build_permission_rules_sql( + union_sql, union_params, _restriction_sqls = await build_permission_rules_sql( self.ds, actor, action ) await self.ds.refresh_schemas() @@ -936,7 +939,7 @@ class ApiExplorerView(BaseView): tables.append({"name": table, "links": table_links}) table_links.append( { - "label": "Get rows for {}".format(table), + "label": f"Get rows for {table}", "method": "GET", "path": self.ds.urls.table(name, table, format="json"), } @@ -956,7 +959,7 @@ class ApiExplorerView(BaseView): { "path": self.ds.urls.table(name, table) + "/-/insert", "method": "POST", - "label": "Insert rows into {}".format(table), + "label": f"Insert rows into {table}", "json": { "rows": [ { @@ -970,7 +973,7 @@ class ApiExplorerView(BaseView): { "path": self.ds.urls.table(name, table) + "/-/upsert", "method": "POST", - "label": "Upsert rows into {}".format(table), + "label": f"Upsert rows into {table}", "json": { "rows": [ { @@ -1000,7 +1003,7 @@ class ApiExplorerView(BaseView): table_links.append( { "path": self.ds.urls.table(name, table) + "/-/drop", - "label": "Drop table {}".format(table), + "label": f"Drop table {table}", "json": {"confirm": False}, "method": "POST", } @@ -1017,7 +1020,7 @@ class ApiExplorerView(BaseView): database_links.append( { "path": self.ds.urls.database(name) + "/-/create", - "label": "Create table in {}".format(name), + "label": f"Create table in {name}", "json": { "table": "new_table", "columns": [ diff --git a/datasette/views/stored_queries.py b/datasette/views/stored_queries.py index d64f37d2..0bbe9f38 100644 --- a/datasette/views/stored_queries.py +++ b/datasette/views/stored_queries.py @@ -124,7 +124,7 @@ class QueryListView(BaseView): pairs.append(("_next", page.next)) next_url = self.ds.absolute_url( request, - "{}?{}".format(request.path, urlencode(pairs)), + f"{request.path}?{urlencode(pairs)}", ) current_filters = { @@ -415,7 +415,7 @@ class QueryDefinitionView(BaseView): query_name = tilde_decode(request.url_vars["query"]) query = await self.ds.get_query(db.name, query_name) if query is None: - return Response.error(["Query not found: {}".format(query_name)], 404) + return Response.error([f"Query not found: {query_name}"], 404) if not await self.ds.allowed( action="view-query", resource=QueryResource(db.name, query_name), @@ -439,7 +439,7 @@ class QueryUpdateView(BaseView): query_name = tilde_decode(request.url_vars["query"]) existing = await self.ds.get_query(db.name, query_name) if existing is None: - return Response.error(["Query not found: {}".format(query_name)], 404) + return Response.error([f"Query not found: {query_name}"], 404) if not await self.ds.allowed( action="update-query", resource=QueryResource(db.name, query_name), @@ -532,7 +532,7 @@ class QueryEditView(BaseView): async def get(self, request): db, query_name, existing = await self._load(request) if existing is None: - return Response.error(["Query not found: {}".format(query_name)], 404) + return Response.error([f"Query not found: {query_name}"], 404) await self.ds.ensure_permission( action="update-query", resource=QueryResource(db.name, query_name), @@ -545,7 +545,7 @@ class QueryEditView(BaseView): async def post(self, request): db, query_name, existing = await self._load(request) if existing is None: - return Response.error(["Query not found: {}".format(query_name)], 404) + return Response.error([f"Query not found: {query_name}"], 404) if not await self.ds.allowed( action="update-query", resource=QueryResource(db.name, query_name), @@ -629,7 +629,7 @@ class QueryDeleteView(BaseView): async def get(self, request): db, query_name, existing = await self._load(request) if existing is None: - return Response.error(["Query not found: {}".format(query_name)], 404) + return Response.error([f"Query not found: {query_name}"], 404) await self.ds.ensure_permission( action="delete-query", resource=QueryResource(db.name, query_name), @@ -653,7 +653,7 @@ class QueryDeleteView(BaseView): async def post(self, request): db, query_name, existing = await self._load(request) if existing is None: - return Response.error(["Query not found: {}".format(query_name)], 404) + return Response.error([f"Query not found: {query_name}"], 404) if not await self.ds.allowed( action="delete-query", resource=QueryResource(db.name, query_name), @@ -665,13 +665,13 @@ class QueryDeleteView(BaseView): ["Trusted queries cannot be deleted using the API"], 403 ) - data, is_json = await _json_or_form_payload(request) + _data, is_json = await _json_or_form_payload(request) await self.ds.remove_query(db.name, query_name) if is_json: return Response.json({"ok": True}) self.ds.add_message( request, - "Query “{}” deleted".format(existing.title or query_name), + f"Query “{existing.title or query_name}” deleted", self.ds.INFO, ) return Response.redirect(self.ds.urls.path(self.ds.urls.database(db.name))) diff --git a/datasette/views/table.py b/datasette/views/table.py index f7edd744..3eb80854 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -3,48 +3,51 @@ import itertools import json import urllib import urllib.parse +from dataclasses import dataclass, field import markupsafe +import sqlite_utils +from datasette import tracer from datasette.column_types import SQLiteType -from datasette.extras import extra_names_from_request -from datasette.plugins import pm +from datasette.database import QueryInterrupted from datasette.events import ( AlterTableEvent, DropTableEvent, InsertRowsEvent, UpsertRowsEvent, ) -from datasette.database import QueryInterrupted -from datasette import tracer +from datasette.extras import ExtraScope, extra_names_from_request +from datasette.filters import Filters +from datasette.plugins import pm from datasette.resources import DatabaseResource, TableResource from datasette.utils import ( - add_cors_headers, - await_me_maybe, - call_with_supported_arguments, CustomJSONEncoder, CustomRow, + InvalidSql, + WriteJsonValueError, + add_cors_headers, append_querystring, + await_me_maybe, + call_with_supported_arguments, compound_keys_after_sql, decode_write_json_rows, - format_bytes, - make_slot_function, - tilde_encode, escape_sqlite, filters_should_redirect, + format_bytes, is_url, + make_slot_function, path_from_row_pks, path_with_added_args, path_with_format, path_with_removed_args, path_with_replaced_args, + sqlite3, + tilde_encode, to_css_class, truncate_url, urlsafe_components, value_as_boolean, - InvalidSql, - WriteJsonValueError, - sqlite3, ) from datasette.utils.asgi import ( BadRequest, @@ -54,11 +57,7 @@ from datasette.utils.asgi import ( Request, Response, ) -from datasette.filters import Filters -import sqlite_utils -from dataclasses import dataclass, field -from datasette.extras import ExtraScope from . import Context, from_extra from .base import BaseView, DatasetteError, stream_csv from .database import QueryView @@ -536,7 +535,7 @@ async def _table_insert_ui( columns.append(column_data) data = { - "path": "{}/-/insert".format(datasette.urls.table(database_name, table_name)), + "path": f"{datasette.urls.table(database_name, table_name)}/-/insert", "tableName": table_name, "columns": columns, "bulkColumns": bulk_columns, @@ -544,8 +543,8 @@ async def _table_insert_ui( "maxInsertRows": datasette.setting("max_insert_rows"), } if can_update: - data["upsertPath"] = "{}/-/upsert".format( - datasette.urls.table(database_name, table_name) + data["upsertPath"] = ( + f"{datasette.urls.table(database_name, table_name)}/-/upsert" ) return data @@ -604,7 +603,7 @@ async def _table_alter_ui( columns.append(column_data) data = { - "path": "{}/-/alter".format(datasette.urls.table(database_name, table_name)), + "path": f"{datasette.urls.table(database_name, table_name)}/-/alter", "tableName": table_name, "columns": columns, "primaryKeys": pks, @@ -630,9 +629,7 @@ async def _table_alter_ui( actor=request.actor, ) if can_drop_table: - data["dropPath"] = "{}/-/drop".format( - datasette.urls.table(database_name, table_name) - ) + data["dropPath"] = f"{datasette.urls.table(database_name, table_name)}/-/drop" return data @@ -728,12 +725,10 @@ async def display_columns_and_rows( row_label = row_label_from_label_column(row, label_column) row_action_label = pk_path if row_label and row_label != pk_path: - row_action_label = "{} {}".format(pk_path, row_label) + row_action_label = f"{pk_path} {row_label}" table_path = datasette.urls.table(database_name, table_name) - row_link = '{flat_pks}'.format( - table_path=table_path, - flat_pks=str(markupsafe.escape(pk_path)), - flat_pks_quoted=row_path, + row_link = ( + f'{markupsafe.escape(pk_path)!s}' ) edit_icon = ( '

{}

'.format(error) in response2.text + assert f'

{error}

' in response2.text else: # Check create-token event event = last_event(app_client.ds) @@ -228,7 +231,7 @@ def test_auth_create_token( # And test that token response3 = app_client.get( "/-/actor.json", - headers={"Authorization": "Bearer {}".format("dstok_{}".format(token))}, + headers={"Authorization": "Bearer {}".format(f"dstok_{token}")}, ) assert response3.status == 200 assert response3.json["actor"]["id"] == "test" @@ -241,7 +244,7 @@ async def test_auth_create_token_not_allowed_for_tokens(ds_client): ) response = await ds_client.get( "/-/create-token", - headers={"Authorization": "Bearer dstok_{}".format(ds_tok)}, + headers={"Authorization": f"Bearer dstok_{ds_tok}"}, ) assert response.status_code == 403 @@ -286,12 +289,12 @@ async def test_auth_with_dstok_token(ds_client, scenario, should_work): elif scenario == "invalid_token": token = "invalid" if token: - token = "dstok_{}".format(token) + token = f"dstok_{token}" if scenario == "allow_signed_tokens_off": ds_client.ds._settings["allow_signed_tokens"] = False headers = {} if token: - headers["Authorization"] = "Bearer {}".format(token) + headers["Authorization"] = f"Bearer {token}" response = await ds_client.get("/-/actor.json", headers=headers) try: if should_work: @@ -338,7 +341,7 @@ def test_cli_create_token(app_client, expires): assert details.keys() == expected_keys assert details["a"] == "test" response = app_client.get( - "/-/actor.json", headers={"Authorization": "Bearer {}".format(token)} + "/-/actor.json", headers={"Authorization": f"Bearer {token}"} ) if expires is None or expires > 0: expected_actor = { diff --git a/tests/test_base_view.py b/tests/test_base_view.py index c1b0cf20..b46f7ce1 100644 --- a/tests/test_base_view.py +++ b/tests/test_base_view.py @@ -1,8 +1,10 @@ -from datasette.views.base import View +import json + +import pytest + from datasette import Request, Response from datasette.app import Datasette -import json -import pytest +from datasette.views.base import View class GetView(View): diff --git a/tests/test_cli.py b/tests/test_cli.py index cbd8edad..fbd4a8a9 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -1,23 +1,28 @@ -from .fixtures import ( - make_app_client, - TestClient as _TestClient, - EXPECTED_PLUGINS, -) -from datasette.app import SETTINGS -from datasette.plugins import DEFAULT_PLUGINS, pm -from datasette.cli import cli, serve -from datasette.version import __version__ -from datasette.utils import tilde_encode -from datasette.utils.sqlite import sqlite3 -from click.testing import CliRunner import io import json import pathlib -import pytest import sys import textwrap from unittest import mock +import pytest +from click.testing import CliRunner + +from datasette.app import SETTINGS +from datasette.cli import cli, serve +from datasette.plugins import DEFAULT_PLUGINS, pm +from datasette.utils import tilde_encode +from datasette.utils.sqlite import sqlite3 +from datasette.version import __version__ + +from .fixtures import ( + EXPECTED_PLUGINS, + make_app_client, +) +from .fixtures import ( + TestClient as _TestClient, +) + def test_inspect_cli(app_client): runner = CliRunner() @@ -460,7 +465,7 @@ def test_serve_create(tmpdir): @pytest.mark.parametrize("argument", ("-c", "--config")) @pytest.mark.parametrize("format_", ("json", "yaml")) def test_serve_config(tmpdir, argument, format_): - config_path = tmpdir / "datasette.{}".format(format_) + config_path = tmpdir / f"datasette.{format_}" config_path.write_text( ( "settings:\n default_page_size: 5\n" @@ -513,13 +518,13 @@ def test_weird_database_names(tmpdir, filename): result1 = runner.invoke(cli, [db_path, "--get", "/"]) assert result1.exit_code == 0, result1.output filename_no_stem = filename.rsplit(".", 1)[0] - expected_link = '{}'.format( - tilde_encode(filename_no_stem), filename_no_stem + expected_link = ( + f'{filename_no_stem}' ) assert expected_link in result1.output # Now try hitting that database page result2 = runner.invoke( - cli, [db_path, "--get", "/{}".format(tilde_encode(filename_no_stem))] + cli, [db_path, "--get", f"/{tilde_encode(filename_no_stem)}"] ) assert result2.exit_code == 0, result2.output diff --git a/tests/test_cli_serve_get.py b/tests/test_cli_serve_get.py index fe9416d6..01b84f59 100644 --- a/tests/test_cli_serve_get.py +++ b/tests/test_cli_serve_get.py @@ -1,8 +1,10 @@ +import json +import textwrap + +from click.testing import CliRunner + from datasette.cli import cli from datasette.plugins import pm -from click.testing import CliRunner -import textwrap -import json def test_serve_with_get(tmp_path_factory): @@ -44,9 +46,9 @@ def test_serve_with_get(tmp_path_factory): # Annoyingly that new test plugin stays resident - we need # to manually unregister it to avoid conflict with other tests - to_unregister = [ + to_unregister = next( p for p in pm.get_plugins() if p.__name__ == "init_for_serve_with_get.py" - ][0] + ) pm.unregister(to_unregister) diff --git a/tests/test_cli_serve_server.py b/tests/test_cli_serve_server.py index 47f23c08..b7604bb8 100644 --- a/tests/test_cli_serve_server.py +++ b/tests/test_cli_serve_server.py @@ -1,6 +1,7 @@ +import socket + import httpx import pytest -import socket @pytest.mark.serial diff --git a/tests/test_column_types.py b/tests/test_column_types.py index cd308ec9..50c6daed 100644 --- a/tests/test_column_types.py +++ b/tests/test_column_types.py @@ -1,7 +1,11 @@ import json import logging +import time +import markupsafe +import pytest from bs4 import BeautifulSoup as Soup + from datasette.app import Datasette from datasette.column_types import ( ColumnType, @@ -9,11 +13,7 @@ from datasette.column_types import ( ) from datasette.hookspecs import hookimpl from datasette.plugins import pm -from datasette.utils import error_body, sqlite3 -from datasette.utils import StartupError -import markupsafe -import pytest -import time +from datasette.utils import StartupError, error_body, sqlite3 @pytest.fixture @@ -104,7 +104,7 @@ def write_token(ds, actor_id="root", permissions=None): def _headers(token): return { - "Authorization": "Bearer {}".format(token), + "Authorization": f"Bearer {token}", "Content-Type": "application/json", } diff --git a/tests/test_config_dir.py b/tests/test_config_dir.py index 636b17eb..00540464 100644 --- a/tests/test_config_dir.py +++ b/tests/test_config_dir.py @@ -1,10 +1,12 @@ import json import pathlib + import pytest from datasette.app import Datasette -from datasette.utils.sqlite import sqlite3 from datasette.utils import StartupError +from datasette.utils.sqlite import sqlite3 + from .fixtures import TestClient as _TestClient PLUGIN = """ diff --git a/tests/test_crossdb.py b/tests/test_crossdb.py index 11e53224..ffd0870c 100644 --- a/tests/test_crossdb.py +++ b/tests/test_crossdb.py @@ -1,7 +1,9 @@ -from datasette.cli import cli -from click.testing import CliRunner -import urllib import sqlite3 +import urllib + +from click.testing import CliRunner + +from datasette.cli import cli def test_crossdb_join(app_client_two_attached_databases_crossdb_enabled): @@ -40,7 +42,7 @@ def test_crossdb_warning_if_too_many_databases(tmp_path_factory): db_dir = tmp_path_factory.mktemp("dbs") dbs = [] for i in range(11): - path = str(db_dir / "db_{}.db".format(i)) + path = str(db_dir / f"db_{i}.db") conn = sqlite3.connect(path) conn.execute("vacuum") conn.close() diff --git a/tests/test_csrf_middleware.py b/tests/test_csrf_middleware.py index 2fcfb216..6c78f69d 100644 --- a/tests/test_csrf_middleware.py +++ b/tests/test_csrf_middleware.py @@ -44,7 +44,7 @@ async def _run_middleware(scope): await mw(scope, None, send) if inner_called: return ("allowed",) - start = [m for m in sent if m["type"] == "http.response.start"][0] + start = next(m for m in sent if m["type"] == "http.response.start") return ("blocked", start["status"]) diff --git a/tests/test_csv.py b/tests/test_csv.py index a2f03776..7758a3c0 100644 --- a/tests/test_csv.py +++ b/tests/test_csv.py @@ -1,8 +1,10 @@ -from datasette.app import Datasette -from bs4 import BeautifulSoup as Soup -import pytest import urllib.parse +import pytest +from bs4 import BeautifulSoup as Soup + +from datasette.app import Datasette + EXPECTED_TABLE_CSV = """id,content 1,hello 2,world diff --git a/tests/test_custom_pages.py b/tests/test_custom_pages.py index 86cdcc6b..32cfc43d 100644 --- a/tests/test_custom_pages.py +++ b/tests/test_custom_pages.py @@ -1,5 +1,7 @@ import pathlib + import pytest + from .fixtures import make_app_client TEST_TEMPLATE_DIRS = str(pathlib.Path(__file__).parent / "test_templates") diff --git a/tests/test_default_deny.py b/tests/test_default_deny.py index f1e43064..f456a17f 100644 --- a/tests/test_default_deny.py +++ b/tests/test_default_deny.py @@ -1,4 +1,5 @@ import pytest + from datasette.app import Datasette from datasette.resources import DatabaseResource, TableResource diff --git a/tests/test_docs.py b/tests/test_docs.py index 0bcb5e62..16df6a46 100644 --- a/tests/test_docs.py +++ b/tests/test_docs.py @@ -2,20 +2,22 @@ Tests to ensure certain things are documented. """ -from datasette import app, utils +import re +from pathlib import Path + +import pytest + import datasette.fixtures # noqa: F401 +from datasette import app, utils from datasette.app import Datasette from datasette.filters import Filters -from pathlib import Path -import pytest -import re docs_path = Path(__file__).parent.parent / "docs" label_re = re.compile(r"\.\. _([^\s:]+):") def get_headings(content, underline="-"): - heading_re = re.compile(r"(\w+)(\([^)]*\))?\n\{}+\n".format(underline)) + heading_re = re.compile(rf"(\w+)(\([^)]*\))?\n\{underline}+\n") return {h[0] for h in heading_re.findall(content)} diff --git a/tests/test_docs_plugins.py b/tests/test_docs_plugins.py index 613160ac..4a0014b4 100644 --- a/tests/test_docs_plugins.py +++ b/tests/test_docs_plugins.py @@ -1,10 +1,11 @@ # fmt: off # -- start datasette_with_plugin_fixture -- -from datasette import hookimpl -from datasette.app import Datasette import pytest import pytest_asyncio +from datasette import hookimpl +from datasette.app import Datasette + @pytest_asyncio.fixture async def datasette_with_plugin(): diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index 768814fd..94c9a7c9 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -17,8 +17,10 @@ present and the legacy "title" key must not be. https://github.com/simonw/datasette/issues - 1.0 API consistency """ -import pytest import time + +import pytest + from datasette.app import Datasette from datasette.utils import sqlite3 @@ -86,7 +88,7 @@ async def test_write_api_validation_error_shape(ds_error_shape): "/data/docs/-/insert", json={"rows": [{"nope": 1}, {"also_nope": 2}]}, headers={ - "Authorization": "Bearer {}".format(token), + "Authorization": f"Bearer {token}", "Content-Type": "application/json", }, ) @@ -410,7 +412,7 @@ async def test_expired_token_returns_401(ds_error_shape): ) ) response = await ds_error_shape.client.get( - "/-/actor.json", headers={"Authorization": "Bearer {}".format(token)} + "/-/actor.json", headers={"Authorization": f"Bearer {token}"} ) data = assert_canonical_error(response, 401) assert "expired" in data["error"].lower() @@ -446,7 +448,7 @@ async def test_valid_token_still_authenticates(ds_error_shape): ) ) response = await ds_error_shape.client.get( - "/-/actor.json", headers={"Authorization": "Bearer {}".format(token)} + "/-/actor.json", headers={"Authorization": f"Bearer {token}"} ) assert response.status_code == 200 assert response.json()["actor"]["id"] == "root" @@ -477,7 +479,7 @@ async def test_token_when_signed_tokens_disabled_returns_401(tmp_path_factory): ds.sign({"a": "root", "t": int(time.time())}, namespace="token") ) response = await ds.client.get( - "/-/actor.json", headers={"Authorization": "Bearer {}".format(token)} + "/-/actor.json", headers={"Authorization": f"Bearer {token}"} ) data = assert_canonical_error(response, 401) assert "not enabled" in data["error"] @@ -642,7 +644,7 @@ async def test_query_list_size_rejects_non_integer(ds_client): @pytest.mark.asyncio @pytest.mark.parametrize("endpoint", ("allowed", "rules")) async def test_debug_endpoints_use_size_and_page_parameters(ds_error_shape, endpoint): - base = "/-/{}.json?action=view-instance".format(endpoint) + base = f"/-/{endpoint}.json?action=view-instance" ok = await ds_error_shape.client.get( base + "&_size=1&_page=1", actor={"id": "root"} ) diff --git a/tests/test_extras.py b/tests/test_extras.py index 73b4965e..4e008926 100644 --- a/tests/test_extras.py +++ b/tests/test_extras.py @@ -1,4 +1,5 @@ import asyncio +from typing import ClassVar import pytest @@ -7,7 +8,7 @@ from datasette.extras import Extra, ExtraRegistry, ExtraScope class SlowValueExtra(Extra): description = "Returns context['value'], optionally slowly" - scopes = {ExtraScope.TABLE} + scopes: ClassVar[set[ExtraScope]] = {ExtraScope.TABLE} async def resolve(self, context): if context["slow"]: @@ -17,7 +18,7 @@ class SlowValueExtra(Extra): class DependentExtra(Extra): description = "Depends on slow_value" - scopes = {ExtraScope.TABLE} + scopes: ClassVar[set[ExtraScope]] = {ExtraScope.TABLE} async def resolve(self, context, slow_value): return slow_value + 1 @@ -25,7 +26,7 @@ class DependentExtra(Extra): class InternalOnlyExtra(Extra): description = "Internal extra for HTML templates only" - scopes = {ExtraScope.TABLE} + scopes: ClassVar[set[ExtraScope]] = {ExtraScope.TABLE} public = False async def resolve(self, context): @@ -52,7 +53,7 @@ def _registered_extra_classes(): @pytest.mark.parametrize("cls", _registered_extra_classes(), ids=lambda cls: cls.key()) def test_registered_extras_have_descriptions(cls): # Every registered extra is part of the documented template/JSON contract - assert cls.description, "{} is missing a description".format(cls.__name__) + assert cls.description, f"{cls.__name__} is missing a description" def test_registry_is_built_once_per_scope(): diff --git a/tests/test_facets.py b/tests/test_facets.py index 8c22ffce..b8eb6e61 100644 --- a/tests/test_facets.py +++ b/tests/test_facets.py @@ -1,11 +1,14 @@ +import json + +import pytest + from datasette.app import Datasette from datasette.database import Database -from datasette.facets import Facet, ColumnFacet, ArrayFacet, DateFacet -from datasette.utils.asgi import Request +from datasette.facets import ArrayFacet, ColumnFacet, DateFacet, Facet from datasette.utils import detect_json1 +from datasette.utils.asgi import Request + from .fixtures import make_app_client -import json -import pytest @pytest.mark.asyncio @@ -537,7 +540,7 @@ async def test_facet_size(): for j in range(1, 4): await db.execute_write( "insert into neighbourhoods (city, neighbourhood) values (?, ?)", - ["City {}".format(i), "Neighbourhood {}".format(j)], + [f"City {i}", f"Neighbourhood {j}"], ) response = await ds.client.get( "/test_facet_size/neighbourhoods.json?_extra=suggested_facets" diff --git a/tests/test_filters.py b/tests/test_filters.py index eda9e9a1..8d0f3512 100644 --- a/tests/test_filters.py +++ b/tests/test_filters.py @@ -1,7 +1,8 @@ -from datasette.filters import Filters, through_filters, where_filters, search_filters -from datasette.utils.asgi import Request import pytest +from datasette.filters import Filters, search_filters, through_filters, where_filters +from datasette.utils.asgi import Request + @pytest.mark.parametrize( "args,expected_where,expected_params", diff --git a/tests/test_html.py b/tests/test_html.py index b4c47d80..6a7b4907 100644 --- a/tests/test_html.py +++ b/tests/test_html.py @@ -1,16 +1,19 @@ -from bs4 import BeautifulSoup as Soup -from datasette.app import Datasette -from datasette.utils import allowed_pragmas -from .fixtures import make_app_client -from .utils import assert_footer_links, inner_html import copy import hashlib import json import pathlib -import pytest import re import urllib.parse +import pytest +from bs4 import BeautifulSoup as Soup + +from datasette.app import Datasette +from datasette.utils import allowed_pragmas + +from .fixtures import make_app_client +from .utils import assert_footer_links, inner_html + def test_homepage(app_client_two_attached_databases): response = app_client_two_attached_databases.get("/") @@ -142,9 +145,7 @@ def test_static_mounts_hash_cache_control(): ) incorrect_hash = hashlib.sha256(b"incorrect").hexdigest()[:12] - response = client.get( - "/custom-static/test_html.py?_hash={}".format(incorrect_hash) - ) + response = client.get(f"/custom-static/test_html.py?_hash={incorrect_hash}") assert response.status_code == 200 assert "cache-control" not in response.headers @@ -219,11 +220,9 @@ async def test_disallowed_custom_sql_pragma(ds_client): "/fixtures/-/query?sql=SELECT+*+FROM+pragma_not_on_allow_list('idx52')" ) assert response.status_code == 400 - pragmas = ", ".join("pragma_{}()".format(pragma) for pragma in allowed_pragmas) + pragmas = ", ".join(f"pragma_{pragma}()" for pragma in allowed_pragmas) assert ( - "Statement contained a disallowed PRAGMA. Allowed pragma functions are {}".format( - pragmas - ) + f"Statement contained a disallowed PRAGMA. Allowed pragma functions are {pragmas}" in response.text ) @@ -778,8 +777,8 @@ def test_stored_query_show_hide_metadata_option( }, memory=True, ) as client: - expected_show_hide_fragment = '({})'.format( - expected_show_hide_link, expected_show_hide_text + expected_show_hide_fragment = ( + f'({expected_show_hide_text})' ) response = client.get("/_memory/one" + querystring) html = response.text @@ -788,10 +787,7 @@ def test_stored_query_show_hide_metadata_option( )[0] assert show_hide_fragment == expected_show_hide_fragment if expected_hidden: - assert ( - ''.format(expected_hidden) - in html - ) + assert f'' in html else: assert '; rel="alternate"; type="application/json+datasette"'.format( - expected - ) + assert link == f'<{expected}>; rel="alternate"; type="application/json+datasette"' assert ( - ''.format( - expected - ) + f'' in response.text ) @@ -1292,8 +1284,8 @@ async def test_database_color(ds_client): expected_color = ds_client.ds.get_database("fixtures").color # Should be something like #9403e5 expected_fragments = ( - "10px solid #{}".format(expected_color), - "border-color: #{}".format(expected_color), + f"10px solid #{expected_color}", + f"border-color: #{expected_color}", ) assert len(expected_color) == 6 for path in ( diff --git a/tests/test_internal_db.py b/tests/test_internal_db.py index e1ab51bb..b4bb964d 100644 --- a/tests/test_internal_db.py +++ b/tests/test_internal_db.py @@ -1,6 +1,7 @@ -import pytest import sqlite3 +import pytest + from datasette.utils import escape_sqlite from datasette.utils.internal_db import INTERNAL_DB_SCHEMA_SQL @@ -137,7 +138,7 @@ async def test_internal_foreign_key_references(ds_client): return { row[1] for row in conn.execute( - "PRAGMA table_info({})".format(escape_sqlite(table_name)) + f"PRAGMA table_info({escape_sqlite(table_name)})" ).fetchall() } @@ -147,7 +148,7 @@ async def test_internal_foreign_key_references(ds_client): for _, name in sorted( (row[5], row[1]) for row in conn.execute( - "PRAGMA table_info({})".format(escape_sqlite(table_name)) + f"PRAGMA table_info({escape_sqlite(table_name)})" ).fetchall() if row[5] ) @@ -159,7 +160,7 @@ async def test_internal_foreign_key_references(ds_client): for table_name in table_names: foreign_key_rows = conn.execute( - "PRAGMA foreign_key_list({})".format(escape_sqlite(table_name)) + f"PRAGMA foreign_key_list({escape_sqlite(table_name)})" ).fetchall() foreign_keys_by_id = {} for foreign_key in foreign_key_rows: @@ -169,25 +170,16 @@ async def test_internal_foreign_key_references(ds_client): foreign_key_rows.sort(key=lambda row: row[1]) other_table = foreign_key_rows[0][2] other_columns = [row[4] for row in foreign_key_rows] - message = 'Column "{}.{}" references other table "{}" which does not exist'.format( - table_name, foreign_key_rows[0][3], other_table - ) + message = f'Column "{table_name}.{foreign_key_rows[0][3]}" references other table "{other_table}" which does not exist' assert other_table in table_names, message + " (bad table)" if all(other_column is None for other_column in other_columns): other_columns = primary_keys_for_table(other_table) - length_message = 'Foreign key from "{}" to "{}" has {} columns but references {} columns'.format( - table_name, - other_table, - len(foreign_key_rows), - len(other_columns), - ) + length_message = f'Foreign key from "{table_name}" to "{other_table}" has {len(foreign_key_rows)} columns but references {len(other_columns)} columns' assert len(other_columns) == len(foreign_key_rows), length_message for foreign_key, other_column in zip(foreign_key_rows, other_columns): column = foreign_key[3] - message = 'Column "{}.{}" references other column "{}.{}" which does not exist'.format( - table_name, column, other_table, other_column - ) + message = f'Column "{table_name}.{column}" references other column "{other_table}.{other_column}" which does not exist' assert other_column in columns_by_table[other_table], ( message + " (bad column)" ) @@ -245,10 +237,10 @@ async def test_stale_catalog_entry_database_fix(tmp_path): @pytest.mark.asyncio async def test_stale_catalog_child_entries_removed_for_missing_database(tmp_path): - from datasette.app import Datasette - import sqlite3 + from datasette.app import Datasette + internal_db_path = str(tmp_path / "internal.db") alpha_db_path = str(tmp_path / "alpha.db") bravo_db_path = str(tmp_path / "bravo.db") @@ -293,10 +285,10 @@ async def test_stale_catalog_child_entries_removed_for_missing_database(tmp_path @pytest.mark.asyncio async def test_orphan_stale_catalog_child_entries_removed(tmp_path): - from datasette.app import Datasette - import sqlite3 + from datasette.app import Datasette + internal_db_path = str(tmp_path / "internal.db") alpha_db_path = str(tmp_path / "alpha.db") diff --git a/tests/test_internals_database.py b/tests/test_internals_database.py index b1a212d9..b1093b1c 100644 --- a/tests/test_internals_database.py +++ b/tests/test_internals_database.py @@ -3,17 +3,23 @@ Tests for the datasette.database.Database class """ import asyncio +import uuid from types import SimpleNamespace -from datasette.app import Datasette -from datasette.database import Database, ExecuteWriteResult, Results, MultipleValues -from datasette.database import DatasetteClosedError -from datasette.database import _deliver_write_result -from datasette.utils.sqlite import sqlite3, supports_returning -from datasette.utils import Column + import pytest import sqlite_utils -import time -import uuid + +from datasette.app import Datasette +from datasette.database import ( + Database, + DatasetteClosedError, + ExecuteWriteResult, + MultipleValues, + Results, + _deliver_write_result, +) +from datasette.utils import Column +from datasette.utils.sqlite import sqlite3, supports_returning requires_sqlite_returning = pytest.mark.skipif( not supports_returning(), reason="SQLite does not support RETURNING" @@ -44,7 +50,7 @@ async def test_results_first(db): @pytest.mark.parametrize("expected", (True, False)) async def test_results_bool(db, expected): where = "" if expected else "where pk = 0" - results = await db.execute("select * from facetable {}".format(where)) + results = await db.execute(f"select * from facetable {where}") assert bool(results) is expected @@ -616,7 +622,7 @@ async def test_execute_write_block_false(db): "update roadside_attractions set name = ? where pk = ?", ["Mystery!", 1], ) - time.sleep(0.1) + await asyncio.sleep(0.1) rows = await db.execute("select name from roadside_attractions where pk = 1") assert "Mystery!" == rows.rows[0][0] @@ -634,7 +640,7 @@ async def test_execute_write_with_returning_block_false(db): ) assert isinstance(task_id, uuid.UUID) - time.sleep(0.1) + await asyncio.sleep(0.1) assert ( await db.execute("select name from write_returning_block_false") ).single_value() == "Cleo" @@ -760,9 +766,10 @@ async def test_execute_write_fn_accepts_any_single_param_name(db, param_name): # Plugins historically relied on the fact that the callback was invoked # positionally, so any parameter name worked. Preserve that contract. scope = {} - exec( - "def write_fn({0}):\n" - " return {0}.execute('select 1 + 1').fetchone()[0]".format(param_name), + # exec() is how we build a function with a parameterized argument name + exec( # noqa: S102 + f"def write_fn({param_name}):\n" + f" return {param_name}.execute('select 1 + 1').fetchone()[0]", scope, ) write_fn = scope["write_fn"] @@ -786,7 +793,9 @@ async def test_execute_write_fn_with_track_event(db): @pytest.mark.asyncio -@pytest.mark.timeout(1) +# func_only so the budget covers the write-thread call under test, not the +# one-off app_client fixture setup this test may be first to trigger +@pytest.mark.timeout(1, func_only=True) async def test_execute_write_fn_connection_exception(tmpdir, app_client): path = str(tmpdir / "immutable.db") conn = sqlite3.connect(path) diff --git a/tests/test_internals_datasette.py b/tests/test_internals_datasette.py index 85598c05..ed2aeaf0 100644 --- a/tests/test_internals_datasette.py +++ b/tests/test_internals_datasette.py @@ -9,13 +9,15 @@ import importlib import os import sqlite3 import time + +import pytest +from itsdangerous import BadSignature + from datasette import Context -from datasette.app import Datasette, Database, ResourcesSQL +from datasette.app import Database, Datasette, ResourcesSQL from datasette.database import DatasetteClosedError from datasette.resources import DatabaseResource from datasette.utils import PrefixedUrlString -from itsdangerous import BadSignature -import pytest @pytest.fixture @@ -77,9 +79,7 @@ async def test_static_template_function_hashes_core_asset(tmp_path, monkeypatch) template = ds.get_jinja_environment().from_string("{{ static('demo.js') }}") expected_hash = hashlib.sha256(b"const demo = true;").hexdigest()[:12] - assert await template.render_async() == "/-/static/demo.js?_hash={}".format( - expected_hash - ) + assert await template.render_async() == f"/-/static/demo.js?_hash={expected_hash}" assert isinstance(ds.static("demo.js"), PrefixedUrlString) @@ -101,7 +101,7 @@ def test_static_hash_recalculated_when_cache_headers_disabled(tmp_path, monkeypa asset_path.write_bytes(b"let a = 2;") expected_hash = hashlib.sha256(b"let a = 2;").hexdigest()[:12] - assert ds.static("demo.js") == "/-/static/demo.js?_hash={}".format(expected_hash) + assert ds.static("demo.js") == f"/-/static/demo.js?_hash={expected_hash}" assert ds.static("demo.js") != first_url @@ -114,12 +114,12 @@ def test_static_hashes_mounted_static_file(tmp_path): expected_hash = hashlib.sha256(b"body { color: black; }").hexdigest()[:12] assert ds.static("styles.css", mount="assets") == ( - "/assets/styles.css?_hash={}".format(expected_hash) + f"/assets/styles.css?_hash={expected_hash}" ) ds._settings["base_url"] = "/prefix/" assert ds.static("styles.css", mount="assets") == ( - "/prefix/assets/styles.css?_hash={}".format(expected_hash) + f"/prefix/assets/styles.css?_hash={expected_hash}" ) @@ -144,9 +144,7 @@ def test_static_hashes_plugin_static_file(tmp_path, monkeypatch): expected_hash = hashlib.sha256(b"console.log('plugin');").hexdigest()[:12] assert ds.static("plugin.js", plugin="datasette_cluster_map") == ( - "/-/static-plugins/datasette_cluster_map/plugin.js?_hash={}".format( - expected_hash - ) + f"/-/static-plugins/datasette_cluster_map/plugin.js?_hash={expected_hash}" ) diff --git a/tests/test_internals_datasette_client.py b/tests/test_internals_datasette_client.py index e9aaaae8..51b38f8d 100644 --- a/tests/test_internals_datasette_client.py +++ b/tests/test_internals_datasette_client.py @@ -1,6 +1,7 @@ import httpx import pytest import pytest_asyncio + from datasette.app import Datasette @@ -238,7 +239,7 @@ async def test_in_client_returns_false_outside_request(datasette): @pytest.mark.asyncio async def test_in_client_returns_true_inside_request(): """Test that datasette.in_client() returns True inside a client request""" - from datasette import hookimpl, Response + from datasette import Response, hookimpl class TestPlugin: __name__ = "test_in_client_plugin" diff --git a/tests/test_internals_request.py b/tests/test_internals_request.py index 6d2dc70a..e982628b 100644 --- a/tests/test_internals_request.py +++ b/tests/test_internals_request.py @@ -1,7 +1,9 @@ -from datasette.utils.asgi import PayloadTooLarge, Request import json + import pytest +from datasette.utils.asgi import PayloadTooLarge, Request + def _post_scope(headers=None): return { diff --git a/tests/test_internals_response.py b/tests/test_internals_response.py index 2366dcde..aa3e1ae2 100644 --- a/tests/test_internals_response.py +++ b/tests/test_internals_response.py @@ -1,7 +1,9 @@ -from datasette.utils.asgi import Response import json + import pytest +from datasette.utils.asgi import Response + def test_response_html(): response = Response.html("Hello from HTML") diff --git a/tests/test_internals_urls.py b/tests/test_internals_urls.py index 24fa745d..50c61995 100644 --- a/tests/test_internals_urls.py +++ b/tests/test_internals_urls.py @@ -1,6 +1,7 @@ +import pytest + from datasette.app import Datasette from datasette.utils import PrefixedUrlString -import pytest @pytest.fixture(scope="module") diff --git a/tests/test_label_column_for_table.py b/tests/test_label_column_for_table.py index 7667b595..b67b8882 100644 --- a/tests/test_label_column_for_table.py +++ b/tests/test_label_column_for_table.py @@ -1,6 +1,7 @@ import pytest -from datasette.database import Database + from datasette.app import Datasette +from datasette.database import Database @pytest.mark.asyncio diff --git a/tests/test_load_extensions.py b/tests/test_load_extensions.py index cdadb091..61cdb3e0 100644 --- a/tests/test_load_extensions.py +++ b/tests/test_load_extensions.py @@ -1,7 +1,9 @@ -from datasette.app import Datasette -import pytest from pathlib import Path +import pytest + +from datasette.app import Datasette + # not necessarily a full path - the full compiled path looks like "ext.dylib" # or another suffix, but sqlite will, under the hood, decide which file # extension to use based on the operating system (apple=dylib, windows=dll etc) diff --git a/tests/test_messages.py b/tests/test_messages.py index 62d9f647..60eb7938 100644 --- a/tests/test_messages.py +++ b/tests/test_messages.py @@ -1,6 +1,7 @@ -from .utils import cookie_was_deleted import pytest +from .utils import cookie_was_deleted + @pytest.mark.asyncio @pytest.mark.parametrize( diff --git a/tests/test_multipart.py b/tests/test_multipart.py index 0dc3ecd7..ab38bfb7 100644 --- a/tests/test_multipart.py +++ b/tests/test_multipart.py @@ -6,12 +6,12 @@ Uses TDD approach - these tests are written first, then implementation follows. import base64 import json -import pytest from collections import namedtuple +import pytest from multipart_form_data_conformance import get_tests_dir -from datasette.utils.asgi import Request, BadRequest +from datasette.utils.asgi import BadRequest, Request def make_receive(body: bytes): diff --git a/tests/test_package.py b/tests/test_package.py index f05f3ece..43b20589 100644 --- a/tests/test_package.py +++ b/tests/test_package.py @@ -1,9 +1,11 @@ -from click.testing import CliRunner -from datasette import cli -from unittest import mock import os import pathlib +from unittest import mock + import pytest +from click.testing import CliRunner + +from datasette import cli class CaptureDockerfile: diff --git a/tests/test_permission_endpoints.py b/tests/test_permission_endpoints.py index 8726ab62..c54bfbfd 100644 --- a/tests/test_permission_endpoints.py +++ b/tests/test_permission_endpoints.py @@ -6,6 +6,7 @@ Tests for permission endpoints: import pytest import pytest_asyncio + from datasette.app import Datasette @@ -432,8 +433,8 @@ async def test_execute_sql_requires_view_database(): A user who has execute-sql permission but not view-database permission should not be able to execute SQL on that database. """ - from datasette.permissions import PermissionSQL from datasette import hookimpl + from datasette.permissions import PermissionSQL class TestPermissionPlugin: __name__ = "TestPermissionPlugin" diff --git a/tests/test_permissions.py b/tests/test_permissions.py index cd1050d0..73c44682 100644 --- a/tests/test_permissions.py +++ b/tests/test_permissions.py @@ -1,20 +1,23 @@ import collections +import copy +import json +import re +import time +import urllib +from pprint import pprint + +import pytest +import pytest_asyncio from asgiref.sync import async_to_sync +from bs4 import BeautifulSoup as Soup +from click.testing import CliRunner + from datasette.app import Datasette from datasette.cli import cli from datasette.default_permissions import restrictions_allow_action from datasette.utils import UNSTABLE_API_MESSAGE + from .fixtures import assert_permissions_checked, make_app_client -from click.testing import CliRunner -from bs4 import BeautifulSoup as Soup -import copy -import json -from pprint import pprint -import pytest_asyncio -import pytest -import re -import time -import urllib @pytest.fixture(scope="module") @@ -602,9 +605,7 @@ def test_permissions_cascade(cascade_app_client, path, permissions, expected_sta ) assert ( response.status == expected_status - ), "path: {}, permissions: {}, expected_status: {}, status: {}".format( - path, permissions, expected_status, response.status - ) + ), f"path: {path}, permissions: {permissions}, expected_status: {expected_status}, status: {response.status}" finally: cascade_app_client.ds.config = previous_config @@ -2039,7 +2040,7 @@ async def test_databases_json_respects_view_database(tmp_path_factory): paths = [] for name in ("public", "private"): - path = str(db_directory / "{}.db".format(name)) + path = str(db_directory / f"{name}.db") conn = _sqlite3.connect(path) conn.execute("vacuum") conn.close() diff --git a/tests/test_plugins.py b/tests/test_plugins.py index 59b1c0bf..734f0fc2 100644 --- a/tests/test_plugins.py +++ b/tests/test_plugins.py @@ -1,21 +1,3 @@ -from bs4 import BeautifulSoup as Soup -from .fixtures import ( - make_app_client, - TEMP_PLUGIN_SECRET_FILE, - PLUGINS_DIR, - TestClient as _TestClient, -) # noqa -from click.testing import CliRunner -from datasette.app import Datasette -from datasette import cli, hookimpl -from datasette.fixtures import TABLES -from datasette.filters import FilterArguments -from datasette.plugins import get_plugins, DEFAULT_PLUGINS, pm -from datasette.permissions import PermissionSQL, Action -from datasette.resources import DatabaseResource -from datasette.utils.sqlite import sqlite3 -from datasette.utils import StartupError, await_me_maybe -from jinja2 import ChoiceLoader, FileSystemLoader import base64 import datetime import importlib @@ -24,9 +6,32 @@ import os import pathlib import re import textwrap -import pytest import urllib +import pytest +from bs4 import BeautifulSoup as Soup +from click.testing import CliRunner +from jinja2 import ChoiceLoader, FileSystemLoader + +from datasette import cli, hookimpl +from datasette.app import Datasette +from datasette.filters import FilterArguments +from datasette.fixtures import TABLES +from datasette.permissions import Action, PermissionSQL +from datasette.plugins import DEFAULT_PLUGINS, get_plugins, pm +from datasette.resources import DatabaseResource +from datasette.utils import StartupError, await_me_maybe +from datasette.utils.sqlite import sqlite3 + +from .fixtures import ( + PLUGINS_DIR, + TEMP_PLUGIN_SECRET_FILE, + make_app_client, +) +from .fixtures import ( + TestClient as _TestClient, +) + at_memory_re = re.compile(r" at 0x\w+") @@ -35,7 +40,7 @@ at_memory_re = re.compile(r" at 0x\w+") ) def test_plugin_hooks_have_tests(plugin_hook): """Every plugin hook should be referenced in this test module""" - tests_in_this_module = [t for t in globals().keys() if t.startswith("test_hook_")] + tests_in_this_module = [t for t in globals() if t.startswith("test_hook_")] ok = False for test in tests_in_this_module: if plugin_hook in test: @@ -125,11 +130,11 @@ async def test_hook_extra_css_urls(ds_client, path, expected_decoded_object): response = await ds_client.get(path) assert response.status_code == 200 links = Soup(response.text, "html.parser").find_all("link") - special_href = [ + special_href = next( link for link in links if link.attrs["href"].endswith("/extra-css-urls-demo.css") - ][0]["href"] + )["href"] # This link has a base64-encoded JSON blob in it encoded = special_href.split("/")[3] actual_decoded_object = json.loads(base64.b64decode(encoded).decode("utf8")) @@ -152,7 +157,7 @@ async def test_hook_extra_js_urls(ds_client): "type": "module", }, ]: - assert any(s == attrs for s in script_attrs), "Expected: {}".format(attrs) + assert any(s == attrs for s in script_attrs), f"Expected: {attrs}" @pytest.mark.asyncio @@ -315,7 +320,8 @@ async def test_plugin_config_env_from_list(ds_client): @pytest.mark.asyncio async def test_plugin_config_file(ds_client): - with open(TEMP_PLUGIN_SECRET_FILE, "w") as fp: + # Blocking write is fine here - it is tiny test setup, not request handling + with open(TEMP_PLUGIN_SECRET_FILE, "w") as fp: # noqa: ASYNC230 fp.write("FROM_FILE") assert {"foo": "FROM_FILE"} == ds_client.ds.plugin_config("file-plugin") os.remove(TEMP_PLUGIN_SECRET_FILE) @@ -823,7 +829,7 @@ def test_hook_register_routes_with_datasette(configured_path): assert response.status_code == 200 assert configured_path.upper() == response.text # Other one should 404 - other_path = [p for p in ("path1", "path2") if configured_path != p][0] + other_path = next(p for p in ("path1", "path2") if configured_path != p) assert client.get(f"/{other_path}/", follow_redirects=True).status_code == 404 @@ -928,7 +934,7 @@ async def test_plugin_startup_can_add_queries(): await datasette.add_query( "data", "from_startup", - "select {}".format(result.first()[0]), + f"select {result.first()[0]}", source="plugin", ) @@ -1040,7 +1046,7 @@ async def test_hook_handle_exception(ds_client): @pytest.mark.asyncio @pytest.mark.parametrize("param", ("_custom_error", "_custom_error_async")) async def test_hook_handle_exception_custom_response(ds_client, param): - response = await ds_client.get("/trigger-error?{}=1".format(param)) + response = await ds_client.get(f"/trigger-error?{param}=1") assert response.text == param @@ -1374,7 +1380,7 @@ async def test_hook_register_actions_no_duplicates(duplicate): # This should error: with pytest.raises(StartupError) as ex: await ds.invoke_startup() - assert "Duplicate action {}".format(duplicate) in str(ex.value) + assert f"Duplicate action {duplicate}" in str(ex.value) @pytest.mark.asyncio diff --git a/tests/test_publish_cloudrun.py b/tests/test_publish_cloudrun.py index 6617bc77..aebcaa33 100644 --- a/tests/test_publish_cloudrun.py +++ b/tests/test_publish_cloudrun.py @@ -1,10 +1,12 @@ -from click.testing import CliRunner -from datasette import cli -from unittest import mock import json import os -import pytest import textwrap +from unittest import mock + +import pytest +from click.testing import CliRunner + +from datasette import cli @pytest.mark.serial @@ -70,9 +72,7 @@ def test_publish_cloudrun_prompts_for_service( ), mock.call(f"gcloud builds submit --tag {tag}", shell=True), mock.call( - "gcloud run deploy --allow-unauthenticated --platform=managed --image {} input-service --max-instances 1".format( - tag - ), + f"gcloud run deploy --allow-unauthenticated --platform=managed --image {tag} input-service --max-instances 1", shell=True, ), ] @@ -107,9 +107,7 @@ def test_publish_cloudrun(mock_call, mock_output, mock_which, tmp_path_factory): ), mock.call(f"gcloud builds submit --tag {tag}", shell=True), mock.call( - "gcloud run deploy --allow-unauthenticated --platform=managed --image {} test --max-instances 1".format( - tag - ), + f"gcloud run deploy --allow-unauthenticated --platform=managed --image {tag} test --max-instances 1", shell=True, ), ] @@ -186,13 +184,13 @@ def test_publish_cloudrun_memory_cpu( tag = f"us-docker.pkg.dev/{mock_output.return_value}/datasette/datasette-test" expected_call = ( "gcloud run deploy --allow-unauthenticated --platform=managed" - " --image {} test".format(tag) + f" --image {tag} test" ) expected_build_call = f"gcloud builds submit --tag {tag}" if memory: - expected_call += " --memory {}".format(memory) + expected_call += f" --memory {memory}" if cpu: - expected_call += " --cpu {}".format(cpu) + expected_call += f" --cpu {cpu}" if timeout: expected_build_call += f" --timeout {timeout}" # max_instances defaults to 1 diff --git a/tests/test_publish_heroku.py b/tests/test_publish_heroku.py index cab83654..4302ed94 100644 --- a/tests/test_publish_heroku.py +++ b/tests/test_publish_heroku.py @@ -1,9 +1,11 @@ -from click.testing import CliRunner -from datasette import cli -from unittest import mock import os import pathlib +from unittest import mock + import pytest +from click.testing import CliRunner + +from datasette import cli @pytest.mark.serial diff --git a/tests/test_pytest_autoclose_plugin.py b/tests/test_pytest_autoclose_plugin.py index 3af1aace..9b17d24b 100644 --- a/tests/test_pytest_autoclose_plugin.py +++ b/tests/test_pytest_autoclose_plugin.py @@ -20,6 +20,7 @@ def _run_pytest(tmp_path: Path) -> subprocess.CompletedProcess: cwd=str(tmp_path), capture_output=True, text=True, + check=False, ) diff --git a/tests/test_queries.py b/tests/test_queries.py index ffa948a9..15b7ad0f 100644 --- a/tests/test_queries.py +++ b/tests/test_queries.py @@ -15,37 +15,29 @@ from datasette.utils.sqlite import sqlite3, supports_returning requires_sqlite_returning = pytest.mark.skipif( not supports_returning(), reason="SQLite does not support RETURNING" ) -EXPECTED_CREATE_TABLE_TEMPLATE_SQL = "\n".join( - ( - "create table new_table (", - " id integer primary key,", - " name text", - " -- created text default (datetime('now'))", - ")", - ) -) +EXPECTED_CREATE_TABLE_TEMPLATE_SQL = "create table new_table (\n id integer primary key,\n name text\n -- created text default (datetime('now'))\n)" def _template_option_attributes(html, table): - match = re.search(r'
'.format( - i, i - ), + f'', f'', f'', f'', @@ -1667,7 +1668,7 @@ async def test_row_update_sets_message(): assert response.status_code == 200 assert response.json()["rows"][0]["name"] == long_name assert ds.unsign(response.cookies["ds_messages"], "messages") == [ - ["Updated row 1 ({})".format(truncated_name), ds.INFO] + [f"Updated row 1 ({truncated_name})", ds.INFO] ] finally: ds.close() @@ -1680,9 +1681,9 @@ def test_table_data_uses_base_url(app_client_base_url_prefix): import re soup = Soup(response.text, "html.parser") - table_script = [ + table_script = next( s for s in soup.find_all("script") if "_datasetteTableData" in (s.string or "") - ][0] + ) match = re.search( r"window\._datasetteTableData\s*=\s*({.*?});", table_script.string, @@ -1710,16 +1711,17 @@ def test_table_fragment_custom_table_include(): @pytest.mark.asyncio async def test_table_fragment_uses_render_cell_hook(): - from datasette import hookimpl from markupsafe import Markup + from datasette import hookimpl + class TestRenderCellPlugin: __name__ = "TestRenderCellPlugin" @hookimpl def render_cell(self, value, column, table, database): if database == "data" and table == "items" and column == "name": - return Markup("{}".format(value)) + return Markup(f"{value}") return None ds = Datasette(memory=True) @@ -2258,18 +2260,16 @@ def test_allow_facet_off(allow_facet): ) async def test_format_of_binary_links(size, title, length_bytes): ds = Datasette() - db_name = "binary-links-{}".format(size) + db_name = f"binary-links-{size}" db = ds.add_memory_database(db_name) - sql = "select zeroblob({}) as blob".format(size) - await db.execute_write("create table blobs as {}".format(sql)) - response = await ds.client.get("/{}/blobs".format(db_name)) + sql = f"select zeroblob({size}) as blob" + await db.execute_write(f"create table blobs as {sql}") + response = await ds.client.get(f"/{db_name}/blobs") assert response.status_code == 200 - expected = "{}><Binary: {} bytes>".format(title, length_bytes) + expected = f"{title}><Binary: {length_bytes} bytes>" assert expected in response.text # And test with arbitrary SQL query too - sql_response = await ds.client.get( - "{}/-/query".format(db_name), params={"sql": sql} - ) + sql_response = await ds.client.get(f"{db_name}/-/query", params={"sql": sql}) assert sql_response.status_code == 200 assert expected in sql_response.text diff --git a/tests/test_template_context.py b/tests/test_template_context.py index 7923d0e7..691c2d64 100644 --- a/tests/test_template_context.py +++ b/tests/test_template_context.py @@ -10,7 +10,7 @@ from dataclasses import dataclass, field import pytest -from datasette.app import Datasette, TEMPLATE_BASE_CONTEXT +from datasette.app import TEMPLATE_BASE_CONTEXT, Datasette from datasette.extras import ExtraScope from datasette.fixtures import write_fixture_database from datasette.template_contexts import PAGES, documented_context_keys @@ -40,17 +40,17 @@ def test_documented_fields(): @pytest.mark.parametrize("klass", PAGES.values(), ids=lambda klass: klass.__name__) def test_context_class_fields_all_have_help(klass): for context_field in klass.documented_fields(): - assert context_field.help, "{}.{} is missing documentation".format( - klass.__name__, context_field.name - ) + assert ( + context_field.help + ), f"{klass.__name__}.{context_field.name} is missing documentation" @pytest.mark.parametrize("klass", PAGES.values(), ids=lambda klass: klass.__name__) def test_context_class_has_docstring_and_documented_template(klass): - assert klass.__doc__, "{} is missing a docstring".format(klass.__name__) - assert klass.documented_template, "{} is missing a documented_template".format( - klass.__name__ - ) + assert klass.__doc__, f"{klass.__name__} is missing a docstring" + assert ( + klass.documented_template + ), f"{klass.__name__} is missing a documented_template" def test_from_extra_documentation_comes_from_the_extra_class(): @@ -105,7 +105,7 @@ def isolate_extra_template_vars_plugins(): # for the rest of the process. The contract documents plugin-free # Datasette core, so unregister any non-default plugin that adds # template variables via the extra_template_vars hook - from datasette.plugins import pm, DEFAULT_PLUGINS + from datasette.plugins import DEFAULT_PLUGINS, pm hook_plugins = {impl.plugin for impl in pm.hook.extra_template_vars.get_hookimpls()} removed = [] @@ -182,18 +182,18 @@ async def test_template_context_matches_documented_contract( undocumented = actual - documented no_longer_present = documented - actual assert not undocumented, ( - "Undocumented keys in {} template context: {} - add them to the " - "page's Context class".format(page_name, sorted(undocumented)) + f"Undocumented keys in {page_name} template context: {sorted(undocumented)} - add them to the " + "page's Context class" ) assert not no_longer_present, ( - "Documented keys missing from {} template context: {} - this would " - "break custom templates".format(page_name, sorted(no_longer_present)) + f"Documented keys missing from {page_name} template context: {sorted(no_longer_present)} - this would " + "break custom templates" ) def test_base_context_keys_all_have_docs(): for name, doc in TEMPLATE_BASE_CONTEXT.items(): - assert doc, "Base context key {} is missing docs".format(name) + assert doc, f"Base context key {name} is missing docs" def test_template_context_docs_cover_every_documented_key(): @@ -201,15 +201,14 @@ def test_template_context_docs_cover_every_documented_key(): assert docs_path.exists(), "docs/template_context.rst is missing" docs = docs_path.read_text() for name in TEMPLATE_BASE_CONTEXT: - assert "``{}``".format(name) in docs, name + assert f"``{name}``" in docs, name for page_name, klass in PAGES.items(): title = "{} page".format(klass.__name__.removesuffix("Context")) assert title in docs, title for context_field in klass.documented_fields(): - assert "``{}``".format(context_field.name) in docs, "{} ({} page)".format( - context_field.name, page_name - ) assert ( - "``{}`` - ``{}``".format(context_field.name, context_field.type_name) - in docs - ), "{} type ({} page)".format(context_field.name, page_name) + f"``{context_field.name}``" in docs + ), f"{context_field.name} ({page_name} page)" + assert ( + f"``{context_field.name}`` - ``{context_field.type_name}``" in docs + ), f"{context_field.name} type ({page_name} page)" diff --git a/tests/test_token_handler.py b/tests/test_token_handler.py index f5bbfead..10021ddf 100644 --- a/tests/test_token_handler.py +++ b/tests/test_token_handler.py @@ -2,16 +2,17 @@ Tests for the register_token_handler plugin hook. """ +import pytest + from datasette.app import Datasette from datasette.hookspecs import hookimpl from datasette.plugins import pm from datasette.tokens import ( + SignedTokenHandler, TokenHandler, TokenInvalid, TokenRestrictions, - SignedTokenHandler, ) -import pytest @pytest.fixture diff --git a/tests/test_tracer.py b/tests/test_tracer.py index 9db211d3..21cfa952 100644 --- a/tests/test_tracer.py +++ b/tests/test_tracer.py @@ -1,4 +1,5 @@ import pytest + from .fixtures import make_app_client @@ -75,10 +76,9 @@ async def test_trace_child_tasks_resets_contextvar_on_exception(): from datasette import tracer before = tracer.trace_task_id.get() - with pytest.raises(ValueError): - with tracer.trace_child_tasks(): - assert tracer.trace_task_id.get() is not None - raise ValueError("simulated error") + with pytest.raises(ValueError), tracer.trace_child_tasks(): + assert tracer.trace_task_id.get() is not None + raise ValueError("simulated error") # The contextvar must be reset even though the block raised assert tracer.trace_task_id.get() == before diff --git a/tests/test_utils.py b/tests/test_utils.py index a535ca93..1808b3cf 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -2,8 +2,17 @@ Tests for various datasette helper functions. """ -from datasette.app import Datasette +import hashlib +import json +import os +import pathlib +import tempfile +from unittest.mock import patch + +import pytest + from datasette import utils +from datasette.app import Datasette from datasette.utils.asgi import Request from datasette.utils.sqlite import ( sqlite3, @@ -11,13 +20,6 @@ from datasette.utils.sqlite import ( sqlite_table_type, supports_returning, ) -import hashlib -import json -import os -import pathlib -import pytest -import tempfile -from unittest.mock import patch @pytest.mark.parametrize( @@ -194,7 +196,7 @@ def test_validate_sql_select_good(good_sql): @pytest.mark.parametrize("open_quote,close_quote", [('"', '"'), ("[", "]")]) def test_detect_fts(open_quote, close_quote): - sql = """ + sql = f""" CREATE TABLE "Dumb_Table" ( "TreeID" INTEGER, "qSpecies" TEXT @@ -209,9 +211,9 @@ def test_detect_fts(open_quote, close_quote): "qCaretaker" TEXT ); CREATE VIEW Test_View AS SELECT * FROM Dumb_Table; - CREATE VIRTUAL TABLE {open}Street_Tree_List_fts{close} USING FTS4 ("qAddress", "qCaretaker", "qSpecies", content={open}Street_Tree_List{close}); + CREATE VIRTUAL TABLE {open_quote}Street_Tree_List_fts{close_quote} USING FTS4 ("qAddress", "qCaretaker", "qSpecies", content={open_quote}Street_Tree_List{close_quote}); CREATE VIRTUAL TABLE r USING rtree(a, b, c); - """.format(open=open_quote, close=close_quote) + """ conn = utils.sqlite3.connect(":memory:") conn.executescript(sql) assert None is utils.detect_fts(conn, "Dumb_Table") @@ -262,8 +264,8 @@ def test_escape_sqlite_prevents_injection(): conn.execute("CREATE TABLE users (id INTEGER, password TEXT)") conn.execute("INSERT INTO users VALUES (1, 'super_secret_password')") malicious = "users] UNION SELECT password FROM users--" - conn.execute('CREATE TABLE "{}" (id INTEGER)'.format(malicious)) - sql = "select count(*) from {}".format(utils.escape_sqlite(malicious)) + conn.execute(f'CREATE TABLE "{malicious}" (id INTEGER)') + sql = f"select count(*) from {utils.escape_sqlite(malicious)}" results = conn.execute(sql).fetchall() conn.close() # The injected UNION must not execute - only the empty malicious table @@ -273,16 +275,16 @@ def test_escape_sqlite_prevents_injection(): @pytest.mark.parametrize("table", ("regular", "has'single quote")) def test_detect_fts_different_table_names(table): - sql = """ + sql = f""" CREATE TABLE [{table}] ( "TreeID" INTEGER, "qSpecies" TEXT ); CREATE VIRTUAL TABLE [{table}_fts] USING FTS4 ("qSpecies", content="{table}"); - """.format(table=table) + """ conn = utils.sqlite3.connect(":memory:") conn.executescript(sql) - assert "{table}_fts".format(table=table) == utils.detect_fts(conn, table) + assert f"{table}_fts" == utils.detect_fts(conn, table) conn.close() @@ -690,7 +692,6 @@ def test_resolve_env_secrets(config, expected): [ ({"id": "blah"}, "blah"), ({"id": "blah", "login": "l"}, "l"), - ({"id": "blah", "login": "l"}, "l"), ({"id": "blah", "login": "l", "username": "u"}, "u"), ({"login": "l", "name": "n"}, "n"), ( diff --git a/tests/test_utils_check_callable.py b/tests/test_utils_check_callable.py index 4f72f9ff..857b73cd 100644 --- a/tests/test_utils_check_callable.py +++ b/tests/test_utils_check_callable.py @@ -1,6 +1,7 @@ -from datasette.utils.check_callable import check_callable import pytest +from datasette.utils.check_callable import check_callable + class AsyncClass: async def __call__(self): diff --git a/tests/test_utils_permissions.py b/tests/test_utils_permissions.py index bc3599c2..918dab95 100644 --- a/tests/test_utils_permissions.py +++ b/tests/test_utils_permissions.py @@ -1,14 +1,17 @@ +from collections.abc import Callable + import pytest + from datasette.app import Datasette from datasette.permissions import PermissionSQL from datasette.utils.permissions import resolve_permissions_from_catalog -from typing import Callable, List @pytest.fixture def db(): ds = Datasette() import tempfile + from datasette.database import Database path = tempfile.mktemp(suffix="demo.db") @@ -127,7 +130,7 @@ def plugin_root_deny_for_all() -> Callable[[str], PermissionSQL]: def plugin_conflicting_same_child_rules( user: str, parent: str, child: str -) -> List[Callable[[str], PermissionSQL]]: +) -> list[Callable[[str], PermissionSQL]]: def allow_provider(action: str) -> PermissionSQL: return PermissionSQL( """ @@ -277,9 +280,7 @@ async def test_alice_global_allow_with_specific_denies_catalog(db): # Alice can see everything except accounting/sales and hr/* assert "/accounting/sales" in res_denied(rows) for r in rows: - if r["parent"] == "hr": - assert r["allow"] == 0 - elif r["resource"] == "/accounting/sales": + if r["parent"] == "hr" or r["resource"] == "/accounting/sales": assert r["allow"] == 0 else: assert r["allow"] == 1 diff --git a/tests/test_utils_sql_analysis.py b/tests/test_utils_sql_analysis.py index 979ff9e1..a6f95e5b 100644 --- a/tests/test_utils_sql_analysis.py +++ b/tests/test_utils_sql_analysis.py @@ -1,7 +1,7 @@ import pytest -from datasette.utils.sqlite import sqlite3 from datasette.utils.sql_analysis import analyze_sql_tables +from datasette.utils.sqlite import sqlite3 @pytest.fixture diff --git a/tests/test_write_wrapper.py b/tests/test_write_wrapper.py index 88ce5520..66599c54 100644 --- a/tests/test_write_wrapper.py +++ b/tests/test_write_wrapper.py @@ -3,14 +3,16 @@ Tests for the write_wrapper plugin hook. """ import asyncio +import sqlite3 +import time from dataclasses import dataclass + +import pytest + from datasette.app import Datasette from datasette.events import Event from datasette.hookspecs import hookimpl from datasette.plugins import pm -import pytest -import sqlite3 -import time @dataclass @@ -113,7 +115,8 @@ async def test_write_wrapper_exception_thrown_into_generator(datasette): def wrapper(conn): try: yield - except Exception as e: + except Exception as e: # noqa: BLE001 + # Test helper deliberately captures whatever the wrapped write raised caught["error"] = e return wrapper @@ -232,7 +235,6 @@ async def test_write_wrapper_return_none_skips(datasette): @hookimpl def write_wrapper(datasette, database, request, transaction): log.append("hook-called") - return None pm.register(Plugin(), name="test_skip") try: @@ -339,7 +341,7 @@ async def test_write_wrapper_via_api(tmp_path): "/test/api_test/-/insert", json={"row": {"name": "test"}, "return": True}, headers={ - "Authorization": "Bearer {}".format(token), + "Authorization": f"Bearer {token}", "Content-Type": "application/json", }, ) @@ -466,7 +468,7 @@ async def test_write_wrapper_set_authorizer(datasette, actor, table, should_deny try: request = FakeRequest(actor) if should_deny: - with pytest.raises(Exception): + with pytest.raises(sqlite3.DatabaseError, match="not authorized"): await db.execute_write_fn( lambda conn: conn.execute( f"insert into {table} (value) values ('test')" From eb6c2b96b9e3e96d119997c9ec8316dba168ea75 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 6 Aug 2026 10:22:35 -0700 Subject: [PATCH 076/131] Fix for SQL injection issue in table filters, refs #2868 --- datasette/filters.py | 58 ++++++++++++++++++++++++----------------- tests/test_filters.py | 33 +++++++++++++++++++++-- tests/test_table_api.py | 30 ++++++++++++++++++++- 3 files changed, 94 insertions(+), 27 deletions(-) diff --git a/datasette/filters.py b/datasette/filters.py index 1d4e32c2..3cfb36e5 100644 --- a/datasette/filters.py +++ b/datasette/filters.py @@ -209,10 +209,14 @@ class TemplatedFilter(Filter): if self.numeric and converted.isdigit(): converted = int(converted) if self.no_argument: - kwargs = {"c": column} + kwargs = {"c": _quote_sqlite_identifier(column)} converted = None else: - kwargs = {"c": column, "p": f"p{param_counter}", "t": table} + kwargs = { + "c": _quote_sqlite_identifier(column), + "p": f"p{param_counter}", + "t": _quote_sqlite_identifier(table), + } return self.sql_template.format(**kwargs), converted def human_clause(self, column, value): @@ -226,6 +230,14 @@ class TemplatedFilter(Filter): return template.format(c=column, v=value) +def _quote_sqlite_identifier(identifier): + # Preserve the historic always-quoted SQL generated by TemplatedFilter. + escaped = escape_sqlite(identifier) + if escaped == identifier: + return f'"{identifier}"' + return escaped + + class InFilter(Filter): key = "in" display = "in" @@ -267,56 +279,56 @@ class Filters: TemplatedFilter( "exact", "=", - '"{c}" = :{p}', + "{c} = :{p}", lambda c, v: "{c} = {v}" if v.isdigit() else '{c} = "{v}"', ), TemplatedFilter( "not", "!=", - '"{c}" != :{p}', + "{c} != :{p}", lambda c, v: "{c} != {v}" if v.isdigit() else '{c} != "{v}"', ), TemplatedFilter( "contains", "contains", - '"{c}" like :{p}', + "{c} like :{p}", '{c} contains "{v}"', format="%{}%", ), TemplatedFilter( "notcontains", "does not contain", - '"{c}" not like :{p}', + "{c} not like :{p}", '{c} does not contain "{v}"', format="%{}%", ), TemplatedFilter( "endswith", "ends with", - '"{c}" like :{p}', + "{c} like :{p}", '{c} ends with "{v}"', format="%{}", ), TemplatedFilter( "startswith", "starts with", - '"{c}" like :{p}', + "{c} like :{p}", '{c} starts with "{v}"', format="{}%", ), - TemplatedFilter("gt", ">", '"{c}" > :{p}', "{c} > {v}", numeric=True), + TemplatedFilter("gt", ">", "{c} > :{p}", "{c} > {v}", numeric=True), TemplatedFilter( - "gte", "\u2265", '"{c}" >= :{p}', "{c} \u2265 {v}", numeric=True + "gte", "\u2265", "{c} >= :{p}", "{c} \u2265 {v}", numeric=True ), - TemplatedFilter("lt", "<", '"{c}" < :{p}', "{c} < {v}", numeric=True), + TemplatedFilter("lt", "<", "{c} < :{p}", "{c} < {v}", numeric=True), TemplatedFilter( - "lte", "\u2264", '"{c}" <= :{p}', "{c} \u2264 {v}", numeric=True + "lte", "\u2264", "{c} <= :{p}", "{c} \u2264 {v}", numeric=True ), - TemplatedFilter("like", "like", '"{c}" like :{p}', '{c} like "{v}"'), + TemplatedFilter("like", "like", "{c} like :{p}", '{c} like "{v}"'), TemplatedFilter( - "notlike", "not like", '"{c}" not like :{p}', '{c} not like "{v}"' + "notlike", "not like", "{c} not like :{p}", '{c} not like "{v}"' ), - TemplatedFilter("glob", "glob", '"{c}" glob :{p}', '{c} glob "{v}"'), + TemplatedFilter("glob", "glob", "{c} glob :{p}", '{c} glob "{v}"'), InFilter(), NotInFilter(), ] @@ -325,13 +337,13 @@ class Filters: TemplatedFilter( "arraycontains", "array contains", - """:{p} in (select value from json_each([{t}].[{c}]))""", + """:{p} in (select value from json_each({t}.{c}))""", '{c} contains "{v}"', ), TemplatedFilter( "arraynotcontains", "array does not contain", - """:{p} not in (select value from json_each([{t}].[{c}]))""", + """:{p} not in (select value from json_each({t}.{c}))""", '{c} does not contain "{v}"', ), ] @@ -339,30 +351,28 @@ class Filters: else [] ) + [ + TemplatedFilter("date", "date", "date({c}) = :{p}", '"{c}" is on date {v}'), TemplatedFilter( - "date", "date", 'date("{c}") = :{p}', '"{c}" is on date {v}' - ), - TemplatedFilter( - "isnull", "is null", '"{c}" is null', "{c} is null", no_argument=True + "isnull", "is null", "{c} is null", "{c} is null", no_argument=True ), TemplatedFilter( "notnull", "is not null", - '"{c}" is not null', + "{c} is not null", "{c} is not null", no_argument=True, ), TemplatedFilter( "isblank", "is blank", - '("{c}" is null or "{c}" = "")', + "({c} is null or {c} = '')", "{c} is blank", no_argument=True, ), TemplatedFilter( "notblank", "is not blank", - '("{c}" is not null and "{c}" != "")', + "({c} is not null and {c} != '')", "{c} is not blank", no_argument=True, ), diff --git a/tests/test_filters.py b/tests/test_filters.py index 8d0f3512..9f201fdf 100644 --- a/tests/test_filters.py +++ b/tests/test_filters.py @@ -66,12 +66,12 @@ from datasette.utils.asgi import Request # JSON arraycontains, arraynotcontains ( (("Availability+Info__arraycontains", "yes"),), - [":p0 in (select value from json_each([table].[Availability+Info]))"], + [':p0 in (select value from json_each("table"."Availability+Info"))'], ["yes"], ), ( (("Availability+Info__arraynotcontains", "yes"),), - [":p0 not in (select value from json_each([table].[Availability+Info]))"], + [':p0 not in (select value from json_each("table"."Availability+Info"))'], ["yes"], ), ], @@ -83,6 +83,35 @@ def test_build_where(args, expected_where, expected_params): assert {f"p{i}": param for i, param in enumerate(expected_params)} == actual_params +@pytest.mark.parametrize( + "key,expected_where", + ( + ( + 'has"quote__exact', + '"has""quote" = :p0', + ), + ( + 'has"quote__isnull', + '"has""quote" is null', + ), + ( + "has]bracket__arraycontains", + ':p0 in (select value from json_each("table"."has]bracket"))', + ), + ), +) +def test_build_where_escapes_column_names(key, expected_where): + filters = Filters(((key, "value"),)) + sql_bits, _ = filters.build_where_clauses("table") + assert sql_bits == [expected_where] + + +def test_build_where_escapes_table_name(): + filters = Filters((("tags__arraycontains", "value"),)) + sql_bits, _ = filters.build_where_clauses("items]bracket") + assert sql_bits == [':p0 in (select value from json_each("items]bracket"."tags"))'] + + @pytest.mark.asyncio async def test_through_filters_from_request(ds_client): request = Request.fake( diff --git a/tests/test_table_api.py b/tests/test_table_api.py index 32dd37f2..6c0c021b 100644 --- a/tests/test_table_api.py +++ b/tests/test_table_api.py @@ -4,7 +4,7 @@ import urllib import pytest from datasette.fixtures import generate_compound_rows, generate_sortable_rows -from datasette.utils import detect_json1 +from datasette.utils import detect_json1, tilde_encode from datasette.utils.sqlite import sqlite_version from .fixtures import make_app_client @@ -689,6 +689,34 @@ async def test_table_filter_queries_multiple_of_same_type(ds_client): ] == response.json()["rows"] +@pytest.mark.skipif(not detect_json1(), reason="Requires the SQLite json1 module") +def test_table_filters_quote_identifiers(): + with make_app_client( + extra_databases={"demo.db": """ + create table "items]bracket" ( + id integer primary key, + "name""quote" text, + "tags]bracket" text + ); + insert into "items]bracket" values (1, 'Alice', '["red"]'); + """}, + ) as client: + table_path = tilde_encode("items]bracket") + exact_query = urllib.parse.urlencode( + {'name"quote__exact': "Alice", "_shape": "arrays"} + ) + exact_response = client.get(f"/demo/{table_path}.json?{exact_query}") + assert exact_response.status == 200 + assert exact_response.json["rows"] == [[1, "Alice", '["red"]']] + + array_query = urllib.parse.urlencode( + {"tags]bracket__arraycontains": "red", "_shape": "arrays"} + ) + array_response = client.get(f"/demo/{table_path}.json?{array_query}") + assert array_response.status == 200 + assert array_response.json["rows"] == [[1, "Alice", '["red"]']] + + @pytest.mark.skipif(not detect_json1(), reason="Requires the SQLite json1 module") @pytest.mark.asyncio async def test_table_filter_json_arraycontains(ds_client): From 12b25affb55c124da21d0b1ee58bba2f00b9bcb3 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 6 Aug 2026 11:20:25 -0700 Subject: [PATCH 077/131] Release 1.0a38 Refs #2868 --- datasette/version.py | 2 +- docs/changelog.rst | 11 +++++++++++ 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/datasette/version.py b/datasette/version.py index 8e238ab5..2ec12fd2 100644 --- a/datasette/version.py +++ b/datasette/version.py @@ -1,2 +1,2 @@ -__version__ = "1.0a37" +__version__ = "1.0a38" __version_info__ = tuple(__version__.split(".")) diff --git a/docs/changelog.rst b/docs/changelog.rst index 670166bb..66a7caab 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -4,6 +4,17 @@ Changelog ========= +.. _v1_0_a38: + +1.0a38 (2026-08-06) +------------------- + +This release fixes a **SQL injection** security issue that affects Datasette instances that serve a **mixture of public and private tables** in the same database, with access configured using the :ref:`Datasette permissions system `. + +Site administrators who serve private tables in this way are advised to disable the :ref:`execute-sql permission ` on that database to prevent users from accessing private tables using raw SQL queries. The bug that has been fixed would have allowed users with access to any public table to execute SQL injection attacks despite that restriction, giving them read-only access to data in private tables in the same database. + +This fix is also available in Datasette 0.65.3. + .. _v1_0_a37: 1.0a37 (2026-07-14) From 0337fba234bf574629d56be631468ea060495fa0 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 10 Aug 2026 15:03:29 -0700 Subject: [PATCH 078/131] disable_fts() before dropping table Closes #2874 --- datasette/views/table.py | 4 +++- tests/test_api_write.py | 37 +++++++++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+), 1 deletion(-) diff --git a/datasette/views/table.py b/datasette/views/table.py index 3eb80854..7c814b27 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1391,7 +1391,9 @@ class TableDropView(BaseView): # Drop table def drop_table(conn): - sqlite_utils.Database(conn)[table_name].drop() + table = sqlite_utils.Database(conn)[table_name] + table.disable_fts() + table.drop() await db.execute_write_fn(drop_table, request=request) await self.ds.track_event( diff --git a/tests/test_api_write.py b/tests/test_api_write.py index 7801d0a3..11ef30de 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -1,6 +1,7 @@ import time import pytest +import sqlite_utils from datasette.app import Datasette from datasette.events import RenameTableEvent @@ -1725,6 +1726,42 @@ async def test_drop_table(ds_write, scenario): assert (await ds_write.client.get("/data/docs")).status_code == 404 +@pytest.mark.asyncio +async def test_drop_table_cleans_up_fts(ds_write): + db = ds_write.get_database("data") + + def enable_fts(conn): + sqlite_utils.Database(conn)["docs"].enable_fts(["title"], create_triggers=True) + + await db.execute_write_fn(enable_fts) + assert { + row[0] + for row in await db.execute( + "select name from sqlite_master where type = 'table' and name like 'docs_fts%'" + ) + } == { + "docs_fts", + "docs_fts_config", + "docs_fts_data", + "docs_fts_docsize", + "docs_fts_idx", + } + + response = await ds_write.client.post( + "/data/docs/-/drop", + json={"confirm": True}, + headers=_headers(write_token(ds_write)), + ) + + assert response.json() == {"ok": True} + assert [ + row[0] + for row in await db.execute( + "select name from sqlite_master where type = 'table' and name like 'docs_fts%'" + ) + ] == [] + + @pytest.mark.asyncio @pytest.mark.parametrize( "input,expected_status,expected_response,expected_events", From e78b8a2e6ac69310c06fdacc6ca0a6ab309ffe0b Mon Sep 17 00:00:00 2001 From: Alex Garcia Date: Tue, 1 Sep 2026 09:32:37 -0700 Subject: [PATCH 079/131] Run datasette serve startup and uvicorn on a single event loop (#2886) * Run datasette serve startup and uvicorn on a single event loop * Move the serve-subprocess test plumbing into a conftest fixture * Fix datasette-litestream URL and trim marker-task test comments * Explain why serve_with_plugins needs a subprocess and plugin files * Apply ruff 0.16 and black fixes * Tweaked some comments --- datasette/cli.py | 91 ++++++++++++++----------- pyproject.toml | 2 +- tests/conftest.py | 84 ++++++++++++++++++++++- tests/test_cli_serve_server.py | 117 +++++++++++++++++++++++++++++++++ 4 files changed, 253 insertions(+), 41 deletions(-) diff --git a/datasette/cli.py b/datasette/cli.py index 57db83b6..12024a14 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -663,16 +663,6 @@ def serve( # Private utility mechanism for writing unit tests return ds - # Run async soundness checks before startup hooks, since invoke_startup - # now populates internal tables which requires querying each database - run_sync(lambda: check_databases(ds)) - - # Run the "startup" plugin hooks - try: - run_sync(ds.invoke_startup) - except StartupError as e: - raise click.ClickException(e.args[0]) - if headers and not get: raise click.ClickException("--headers can only be used with --get") @@ -680,6 +670,14 @@ def serve( raise click.ClickException("--token can only be used with --get") if get: + # --get means we don't run Uvicorn at all + run_sync(lambda: check_databases(ds)) + + try: + run_sync(ds.invoke_startup) + except StartupError as e: + raise click.ClickException(e.args[0]) + client = TestClient(ds) request_headers = {} if token: @@ -704,34 +702,51 @@ def serve( sys.exit(exit_code) return - # Start the server - url = None - if root: - ds.root_enabled = True - url = "http://{}:{}{}?token={}".format( - host, port, ds.urls.path("-/auth-token"), ds._root_token - ) - click.echo(url) - if open_browser: - if url is None: - # Figure out most convenient URL - to table, database or homepage - path = run_sync(lambda: initial_path_for_datasette(ds)) - url = f"http://{host}:{port}{path}" - webbrowser.open(url) - uvicorn_kwargs = { - "host": host, - "port": port, - "log_level": "info", - "lifespan": "on", - "workers": 1, - } - if uds: - uvicorn_kwargs["uds"] = uds - if ssl_keyfile: - uvicorn_kwargs["ssl_keyfile"] = ssl_keyfile - if ssl_certfile: - uvicorn_kwargs["ssl_certfile"] = ssl_certfile - uvicorn.run(ds.app(), **uvicorn_kwargs) + # check_databases, invoke_startup() and the uvicorn server all run on a + # single event loop, so that anything a plugin's "startup" hook schedules + # on the loop (asyncio.create_task, Lock/Queue/Event objects, ...) is + # still alive when the server starts handling requests. + async def _serve_async(): + # Populate internal catalog tables before invoke_startup + await check_databases(ds) + + # Run the "startup" plugin hooks + try: + await ds.invoke_startup() + except StartupError as e: + raise click.ClickException(e.args[0]) + + # Start the server + url = None + if root: + ds.root_enabled = True + url = "http://{}:{}{}?token={}".format( + host, port, ds.urls.path("-/auth-token"), ds._root_token + ) + click.echo(url) + if open_browser: + if url is None: + # Figure out most convenient URL - to table, database or homepage + path = await initial_path_for_datasette(ds) + url = f"http://{host}:{port}{path}" + webbrowser.open(url) + uvicorn_kwargs = { + "host": host, + "port": port, + "log_level": "info", + "lifespan": "on", + "workers": 1, + } + if uds: + uvicorn_kwargs["uds"] = uds + if ssl_keyfile: + uvicorn_kwargs["ssl_keyfile"] = ssl_keyfile + if ssl_certfile: + uvicorn_kwargs["ssl_certfile"] = ssl_certfile + server = uvicorn.Server(uvicorn.Config(ds.app(), **uvicorn_kwargs)) + await server.serve() + + asyncio.run(_serve_async()) @cli.command() diff --git a/pyproject.toml b/pyproject.toml index cf5db905..e658955f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -30,7 +30,7 @@ dependencies = [ "hupper>=1.9", "httpx>=0.20,<1.0", "pluggy>=1.0", - "uvicorn>=0.11", + "uvicorn>=0.29", "aiofiles>=0.4", "PyYAML>=5.3", "mergedeep>=1.1.1", diff --git a/tests/conftest.py b/tests/conftest.py index a2e6aba2..12dce417 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -2,6 +2,7 @@ import importlib.metadata import os import pathlib import re +import socket import subprocess import sys import tempfile @@ -32,17 +33,31 @@ UNDOCUMENTED_PERMISSIONS = { } -def wait_until_responds(url, timeout=5.0, client=httpx, **kwargs): +def wait_until_responds(url, timeout=5.0, client=httpx, process=None, **kwargs): start = time.time() while time.time() - start < timeout: + # If the server died there is no point waiting out the timeout - fail + # now, with its output, instead of after `timeout` seconds of silence + if process is not None and process.poll() is not None: + raise AssertionError( + "Server exited early with returncode {}\n{}".format( + process.returncode, process.stdout.read().decode("utf-8") + ) + ) try: client.get(url, **kwargs) return - except httpx.ConnectError: + except httpx.TransportError: time.sleep(0.1) raise AssertionError(f"Timed out waiting for {url} to respond") +def find_free_port(): + with socket.socket() as sock: + sock.bind(("127.0.0.1", 0)) + return sock.getsockname()[1] + + @pytest.fixture def bare_ds(): """ @@ -301,6 +316,71 @@ def ds_unix_domain_socket_server(tmp_path_factory): pass +@pytest.fixture +def serve_with_plugins(tmp_path): + """Factory fixture for starting ``datasette serve`` in a subprocess with + plugins written to a temporary ``--plugins-dir``. + + For tests that need the real serve path: event-loop wiring, exit codes, + signals. The usual in-process ``pm.register`` plugin pattern can't reach + a subprocess, so plugin source is written out as importable files instead. + + Unlike ``ds_localhost_http_server`` this is function-scoped and takes a + fresh port each time, because each test needs its own plugins. Call it as:: + + proc, port = serve_with_plugins({"my_plugin": PLUGIN_SOURCE}) + + ``plugins`` maps module name to Python source. Pass + ``wait_for_startup=False`` when the server is expected to fail during + startup rather than begin serving. Extra CLI arguments are passed through. + Every process started is terminated when the test ends. + """ + processes = [] + + def start(plugins, *extra_args, wait_for_startup=True): + plugins_dir = tmp_path / "plugins" + plugins_dir.mkdir(exist_ok=True) + for module_name, source in plugins.items(): + (plugins_dir / f"{module_name}.py").write_text(source, "utf-8") + port = find_free_port() + proc = subprocess.Popen( + [ + sys.executable, + "-m", + "datasette", + "--memory", + "--plugins-dir", + str(plugins_dir), + "-h", + "127.0.0.1", + "-p", + str(port), + *extra_args, + ], + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + # Avoid FileNotFoundError: [Errno 2] No such file or directory: + cwd=tempfile.gettempdir(), + ) + processes.append(proc) + if wait_for_startup: + wait_until_responds( + f"http://127.0.0.1:{port}/-/versions.json", process=proc + ) + return proc, port + + yield start + + for proc in processes: + if proc.poll() is None: + proc.terminate() + try: + proc.wait(timeout=5) + except subprocess.TimeoutExpired: + proc.kill() + proc.wait() + + # Import fixtures from fixtures.py to make them available from .fixtures import ( # noqa: F401 TEMP_PLUGIN_SECRET_FILE, diff --git a/tests/test_cli_serve_server.py b/tests/test_cli_serve_server.py index b7604bb8..b76180fd 100644 --- a/tests/test_cli_serve_server.py +++ b/tests/test_cli_serve_server.py @@ -1,4 +1,5 @@ import socket +import time import httpx import pytest @@ -28,3 +29,119 @@ def test_serve_unix_domain_socket(ds_unix_domain_socket_server): "path": "/_memory", "tables": [], }.items() <= response.json().items() + + +# Shaped after datasette-litestream's startup hook, which schedules a +# background task with asyncio.get_running_loop().create_task(...): +# https://github.com/datasette/datasette-litestream +MARKER_TASK_PLUGIN = """ +import asyncio +from datasette import hookimpl +from datasette.utils.asgi import Response + + +@hookimpl +def startup(datasette): + datasette._startup_calls = getattr(datasette, "_startup_calls", 0) + 1 + + async def _mark(): + # Must await before setting the flag: a task with no internal + # await point could finish on the throwaway loop before it + # closed, masking the regression this test guards against. + await asyncio.sleep(0.2) + datasette._marker_task_ran = True + + asyncio.get_running_loop().create_task(_mark()) + + +@hookimpl +def register_routes(): + async def marker_status(datasette): + return Response.json( + { + "marker_task_ran": getattr(datasette, "_marker_task_ran", False), + "startup_calls": getattr(datasette, "_startup_calls", 0), + } + ) + + return [(r"^/-/marker-task-ran$", marker_status)] +""" + + +STARTUP_ERROR_PLUGIN = """ +from datasette import hookimpl +from datasette.utils import StartupError + + +@hookimpl +def startup(datasette): + raise StartupError("boom from plugin") +""" + + +@pytest.mark.serial +def test_startup_hook_background_task_runs_on_serving_loop(serve_with_plugins): + """ + Litestream-shaped regression test: a startup hook that does + asyncio.get_running_loop().create_task(...) must have that task + actually execute before/while the server is handling requests. This + only holds if invoke_startup() and uvicorn.Server.serve() share one + event loop. This test fails against unmodified main, where + invoke_startup() runs on a throwaway loop that is closed before + uvicorn opens its own loop to serve. + """ + _, port = serve_with_plugins({"marker_task_plugin": MARKER_TASK_PLUGIN}) + # The fixture has already waited for the server to answer requests. The + # marker task deliberately awaits before setting its flag, so poll for a + # moment rather than assuming it landed before the first request arrived. + deadline = time.time() + 3.0 + payload = {} + while time.time() < deadline: + payload = httpx.get( + f"http://127.0.0.1:{port}/-/marker-task-ran", timeout=1.0 + ).json() + if payload["marker_task_ran"]: + break + time.sleep(0.05) + assert payload.get("marker_task_ran"), ( + "The startup hook's asyncio.create_task(...) never ran - " + "invoke_startup() and the server are not sharing an event loop" + ) + # Polling above means this test would also pass if the startup hook were + # re-run on the serving loop by the first-request fallback - which would + # hide exactly the bug being tested. invoke_startup() is idempotent today + # so that cannot happen; assert it explicitly so that if the idempotency + # guard is ever removed this test fails loudly instead of silently + # becoming a no-op. + assert payload["startup_calls"] == 1, ( + "startup hook ran {} times - the marker may have been set by a " + "re-run on the serving loop rather than by the original task".format( + payload["startup_calls"] + ) + ) + + +@pytest.mark.serial +def test_startup_error_fails_fast_before_port_binds(serve_with_plugins): + """ + A "startup" plugin hook that raises StartupError must fail fast: print + the message, exit non-zero, and never accept a connection on the port - + the failure must happen before uvicorn.Server binds the socket. + """ + proc, port = serve_with_plugins( + {"startup_error_plugin": STARTUP_ERROR_PLUGIN}, wait_for_startup=False + ) + stdout, _ = proc.communicate(timeout=15) + output = stdout.decode("utf-8") + assert proc.returncode not in (0, None), output + assert "boom from plugin" in output, output + + # Nothing is listening on the port now the process has exited. This + # confirms the socket was not left bound; on its own it cannot prove the + # failure preceded the bind, since a port nothing ever touched also + # refuses connections. + with ( + pytest.raises(OSError), + socket.create_connection(("127.0.0.1", port), timeout=0.2), + ): + pass From 3e018bb1b571cef87c67ae718a5472f23d6b3c6f Mon Sep 17 00:00:00 2001 From: Alex Garcia Date: Tue, 1 Sep 2026 09:39:25 -0700 Subject: [PATCH 080/131] Run startup via ASGI lifespan instead of waiting for the first request (#2887) * Run startup via ASGI lifespan instead of waiting for the first request * Ensure immutable table counts still precompute when startup ran first --- datasette/app.py | 47 ++++++-- datasette/cli.py | 7 +- datasette/utils/asgi.py | 42 +++++-- tests/test_lifespan.py | 259 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 337 insertions(+), 18 deletions(-) create mode 100644 tests/test_lifespan.py diff --git a/datasette/app.py b/datasette/app.py index c82ea075..42be7425 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -453,8 +453,10 @@ class Datasette: self.databases = collections.OrderedDict() self.actions = {} # .invoke_startup() will populate this self._column_types = {} # .invoke_startup() will populate this + self._setup_db_done = False try: self._refresh_schemas_lock = asyncio.Lock() + self._startup_lock = asyncio.Lock() except RuntimeError as rex: # Workaround for intermittent test failure, see: # https://github.com/simonw/datasette/issues/1802 @@ -462,6 +464,7 @@ class Datasette: loop = asyncio.new_event_loop() asyncio.set_event_loop(loop) self._refresh_schemas_lock = asyncio.Lock() + self._startup_lock = asyncio.Lock() else: raise self.crossdb = crossdb @@ -2803,24 +2806,52 @@ class Datasette: raise RowNotFound(db.name, table_name, pk_values) return ResolvedRow(db, table_name, sql, params, pks, pk_values, results.first()) + async def _startup_sequence(self): + """Idempotently run the full startup sequence: table counts for + immutable databases, then invoke_startup(). Safe to call more than + once and safe to call concurrently - callers block until whichever + call got there first has finished. + + This is the single entry point used by both AsgiLifespan (so + real deployments finish startup before accepting requests) and + AsgiRunOnFirstRequest (the fallback for hosts that never send + lifespan events, e.g. DatasetteClient's httpx.ASGITransport), and + `datasette serve` (cli.py) calls it too. The fast path below checks + both `_startup_invoked` and `_setup_db_done` - not just the former - + so that a bare `await ds.invoke_startup()` made by a caller ahead of + `_startup_sequence()` (which only sets `_startup_invoked`) can't + make this method skip the immutable-database table-count precompute. + """ + if self._startup_invoked and self._setup_db_done: + return + async with self._startup_lock: + if self._startup_invoked and self._setup_db_done: + return + if not self._setup_db_done: + # First time server starts up, calculate table counts for + # immutable databases + for database in self.databases.values(): + if not database.is_mutable: + await database.table_counts(limit=60 * 60 * 1000) + self._setup_db_done = True + await self.invoke_startup() + def app(self): """Returns an ASGI app function that serves the whole of Datasette""" routes = self._routes() - async def setup_db(): - # First time server starts up, calculate table counts for immutable databases - for database in self.databases.values(): - if not database.is_mutable: - await database.table_counts(limit=60 * 60 * 1000) - async def _close_on_shutdown(): self.close() asgi = CrossOriginProtectionMiddleware(DatasetteRouter(self, routes), self) if self.setting("trace_debug"): asgi = AsgiTracer(asgi) - asgi = AsgiLifespan(asgi, on_shutdown=[_close_on_shutdown]) - asgi = AsgiRunOnFirstRequest(asgi, on_startup=[setup_db, self.invoke_startup]) + asgi = AsgiLifespan( + asgi, + on_startup=[self._startup_sequence], + on_shutdown=[_close_on_shutdown], + ) + asgi = AsgiRunOnFirstRequest(asgi, on_startup=[self._startup_sequence]) for wrapper in pm.hook.asgi_wrapper(datasette=self): asgi = wrapper(asgi) return asgi diff --git a/datasette/cli.py b/datasette/cli.py index 12024a14..2694c1f6 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -710,9 +710,12 @@ def serve( # Populate internal catalog tables before invoke_startup await check_databases(ds) - # Run the "startup" plugin hooks + # Run the full startup sequence (immutable-database table-count + # precompute + the "startup" plugin hooks) via the same entry point + # AsgiLifespan/AsgiRunOnFirstRequest use, so it's not skipped when + # uvicorn's lifespan.startup fires moments later. try: - await ds.invoke_startup() + await ds._startup_sequence() except StartupError as e: raise click.ClickException(e.args[0]) diff --git a/datasette/utils/asgi.py b/datasette/utils/asgi.py index 812194fd..2614ad02 100644 --- a/datasette/utils/asgi.py +++ b/datasette/utils/asgi.py @@ -1,3 +1,4 @@ +import asyncio import json import re from http.cookies import Morsel, SimpleCookie @@ -300,12 +301,24 @@ class AsgiLifespan: while True: message = await receive() if message["type"] == "lifespan.startup": - for fn in self.on_startup: - await fn() + try: + for fn in self.on_startup: + await fn() + except Exception as e: # noqa: BLE001 + await send( + {"type": "lifespan.startup.failed", "message": str(e)} + ) + return await send({"type": "lifespan.startup.complete"}) elif message["type"] == "lifespan.shutdown": - for fn in self.on_shutdown: - await fn() + try: + for fn in self.on_shutdown: + await fn() + except Exception as e: # noqa: BLE001 + await send( + {"type": "lifespan.shutdown.failed", "message": str(e)} + ) + return await send({"type": "lifespan.shutdown.complete"}) return else: @@ -624,10 +637,23 @@ class AsgiRunOnFirstRequest: self.asgi = asgi self.on_startup = on_startup self._started = False + # Guards against concurrent early requests interleaving with startup: + # without this, several requests could all observe `_started is + # False` and proceed before any of them finish running the hooks. + self._lock = asyncio.Lock() async def __call__(self, scope, receive, send): - if not self._started: - self._started = True - for hook in self.on_startup: - await hook() + # Leave "lifespan" scope events alone - this shim only exists as a + # fallback for hosts that never send them. It wraps AsgiLifespan, so + # if it ran on_startup here too, a startup exception would escape + # before AsgiLifespan's own try/except got a chance to turn it into + # a lifespan.startup.failed message. + if scope["type"] != "lifespan" and not self._started: + async with self._lock: + # Re-check: another request may have finished startup while + # we were waiting for the lock. + if not self._started: + for hook in self.on_startup: + await hook() + self._started = True return await self.asgi(scope, receive, send) diff --git a/tests/test_lifespan.py b/tests/test_lifespan.py new file mode 100644 index 00000000..3655285e --- /dev/null +++ b/tests/test_lifespan.py @@ -0,0 +1,259 @@ +""" +Tests for wiring Datasette startup (setup_db table counts + invoke_startup) +into the ASGI lifespan protocol. + +These exercise Datasette._startup_sequence() via three different callers: +- AsgiLifespan, by hand-driving lifespan.startup messages (no HTTP request) +- AsgiRunOnFirstRequest, the fallback for hosts that never send lifespan + events (this is what DatasetteClient / plain httpx.ASGITransport uses) +- Both at once, to prove startup hooks run at most once +""" + +import asyncio +import contextlib +import sqlite3 + +import httpx +import pytest + +from datasette import hookimpl +from datasette.app import Datasette +from datasette.database import Database +from datasette.plugins import pm + + +async def _drive_lifespan_startup(app): + """Send a single lifespan.startup message into app's ASGI lifespan loop + and return the list of messages sent back - without ever sending + lifespan.shutdown. Mirrors what a real server does: after startup + completes it parks waiting for the next event. We cancel that wait + once we've observed the startup response, rather than closing the + Datasette instance down with a shutdown message. + """ + messages_sent = [] + startup_responded = asyncio.Event() + delivered = False + + async def receive(): + nonlocal delivered + if not delivered: + delivered = True + return {"type": "lifespan.startup"} + # No further messages: block until the task is cancelled below, + # same as a real server parked waiting for lifespan.shutdown. + await asyncio.Event().wait() + + async def send(message): + messages_sent.append(message) + startup_responded.set() + + task = asyncio.create_task(app({"type": "lifespan"}, receive, send)) + try: + await asyncio.wait_for(startup_responded.wait(), timeout=5) + finally: + task.cancel() + with contextlib.suppress(asyncio.CancelledError): + await task + return messages_sent + + +@pytest.mark.asyncio +async def test_lifespan_startup_runs_before_any_request(): + ds = Datasette(memory=True) + assert ds._startup_invoked is False + app = ds.app() + + messages = await _drive_lifespan_startup(app) + + assert {"type": "lifespan.startup.complete"} in messages + assert ds._startup_invoked is True + # Internal catalog tables should be populated too, entirely without an + # HTTP request having been made. + internal_db = ds.get_internal_database() + databases = await internal_db.execute("select * from catalog_databases") + assert len(databases.rows) >= 1 + + +@pytest.mark.asyncio +async def test_lifespan_startup_failure_reports_lifespan_startup_failed(): + class RaisingStartupPlugin: + __name__ = "RaisingStartupPlugin" + + @hookimpl + def startup(self, datasette): + async def inner(): + raise RuntimeError("boom from startup hook") + + return inner + + ds = Datasette(memory=True) + pm.register(RaisingStartupPlugin(), name="raising_startup_plugin") + try: + app = ds.app() + messages = await _drive_lifespan_startup(app) + finally: + pm.unregister(name="raising_startup_plugin") + + assert messages == [ + {"type": "lifespan.startup.failed", "message": "boom from startup hook"} + ] + # The exception happened before invoke_startup() got to the end of its + # body, so startup is not considered to have completed. + assert ds._startup_invoked is False + + +@pytest.mark.asyncio +async def test_startup_runs_exactly_once_across_lifespan_and_first_request(): + call_count = {"n": 0} + + class CountingStartupPlugin: + __name__ = "CountingStartupPlugin" + + @hookimpl + def startup(self, datasette): + async def inner(): + call_count["n"] += 1 + + return inner + + ds = Datasette(memory=True) + pm.register(CountingStartupPlugin(), name="counting_startup_plugin") + try: + # Build the ASGI app once, the way a real deployment does - and + # reuse the SAME app instance for both the lifespan drive and the + # HTTP requests below, since a fresh ds.app() call would reset the + # AsgiRunOnFirstRequest fallback's state. + app = ds.app() + + messages = await _drive_lifespan_startup(app) + assert {"type": "lifespan.startup.complete"} in messages + assert call_count["n"] == 1 + + # A first HTTP request (as if the host never sent lifespan events, + # or lifespan already ran) should not run the hook again. + transport = httpx.ASGITransport(app=app) + async with httpx.AsyncClient( + transport=transport, base_url="http://localhost" + ) as client: + response1 = await client.get("/-/versions.json") + assert response1.status_code == 200 + # ... nor should a second, repeat request. + response2 = await client.get("/-/versions.json") + assert response2.status_code == 200 + finally: + pm.unregister(name="counting_startup_plugin") + + assert call_count["n"] == 1 + + +@pytest.mark.asyncio +async def test_no_lifespan_first_request_still_triggers_startup(): + # Pin today's behavior: a client that never drives ASGI lifespan events + # at all (like httpx.ASGITransport, which DatasetteClient uses) still + # gets startup armed by the AsgiRunOnFirstRequest fallback. + ds = Datasette(memory=True) + assert ds._startup_invoked is False + app = ds.app() + transport = httpx.ASGITransport(app=app) + async with httpx.AsyncClient( + transport=transport, base_url="http://localhost" + ) as client: + response = await client.get("/-/versions.json") + assert response.status_code == 200 + + assert ds._startup_invoked is True + internal_db = ds.get_internal_database() + databases = await internal_db.execute("select * from catalog_databases") + assert len(databases.rows) >= 1 + + +@pytest.mark.asyncio +async def test_datasette_client_first_request_triggers_startup(): + # Same as above, but through the real DatasetteClient (ds.client) that + # plugins and tests actually use, to confirm nothing regressed there. + ds = Datasette(memory=True) + assert ds._startup_invoked is False + response = await ds.client.get("/-/versions.json") + assert response.status_code == 200 + assert ds._startup_invoked is True + + +@pytest.mark.asyncio +async def test_concurrent_first_requests_all_wait_for_slow_startup(): + call_count = {"n": 0} + + class SlowStartupPlugin: + __name__ = "SlowStartupPlugin" + + @hookimpl + def startup(self, datasette): + async def inner(): + call_count["n"] += 1 + await asyncio.sleep(0.2) + + return inner + + ds = Datasette(memory=True) + pm.register(SlowStartupPlugin(), name="slow_startup_plugin") + try: + app = ds.app() + transport = httpx.ASGITransport(app=app) + async with httpx.AsyncClient( + transport=transport, base_url="http://localhost" + ) as client: + responses = await asyncio.gather( + *[client.get("/-/versions.json") for _ in range(10)] + ) + finally: + pm.unregister(name="slow_startup_plugin") + + # Every one of the 10 simultaneous first requests must have blocked + # until startup actually finished, not raced ahead of it. + assert all(response.status_code == 200 for response in responses) + assert call_count["n"] == 1 + assert ds._startup_invoked is True + + +@pytest.mark.asyncio +async def test_setup_db_still_runs_when_invoke_startup_ran_first(tmp_path, monkeypatch): + # Regression test: `datasette serve` (cli.py _serve_async) calls + # ds.invoke_startup() directly, before uvicorn ever sends a + # lifespan.startup event that drives _startup_sequence(). If + # _startup_sequence()'s fast path only checked `_startup_invoked`, it + # would see startup already done and skip the immutable-database + # table-count precompute (setup_db) entirely - a silent regression + # versus main, where AsgiRunOnFirstRequest ran setup_db unconditionally + # on request #1. + db_path = tmp_path / "immutable.db" + conn = sqlite3.connect(str(db_path)) + conn.execute("create table t (id integer primary key)") + conn.commit() + conn.close() + + ds = Datasette([], immutables=[str(db_path)]) + + call_count = {"n": 0} + original_table_counts = Database.table_counts + + async def counting_table_counts(self, *args, **kwargs): + call_count["n"] += 1 + return await original_table_counts(self, *args, **kwargs) + + monkeypatch.setattr(Database, "table_counts", counting_table_counts) + + # Simulate the CLI path: invoke_startup() runs directly and completes + # BEFORE _startup_sequence() ever gets a chance to run setup_db. + await ds.invoke_startup() + assert ds._startup_invoked is True + assert call_count["n"] == 0 + + # The lifespan/first-request path (or the CLI itself, per the fix) + # calling the shared entry point afterwards must still precompute + # table counts for immutable databases. + await ds._startup_sequence() + assert call_count["n"] == 1 + assert ds._setup_db_done is True + + # Idempotency: a second call must not recompute. + await ds._startup_sequence() + assert call_count["n"] == 1 From bdc973174096cae350ddaa733a10ed8b3ffd970b Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 1 Sep 2026 13:37:15 -0700 Subject: [PATCH 081/131] check-latest: true, add 3.15 to test matrix, to test RCs (#2895) See https://simonwillison.net/2026/Sep/1/python-315-rc-2/ --- .github/workflows/test.yml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 751eedfd..2a8c0ae4 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -11,16 +11,17 @@ jobs: strategy: fail-fast: false matrix: - python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"] + python-version: ["3.10", "3.11", "3.12", "3.13", "3.14", "3.15"] steps: - uses: actions/checkout@v7 - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v6 + uses: actions/setup-python@v7 with: python-version: ${{ matrix.python-version }} allow-prereleases: true cache: pip cache-dependency-path: pyproject.toml + check-latest: true - name: Build extension for --load-extension test run: |- (cd tests && gcc ext.c -fPIC -shared -o ext.so) From 7403ae68bb0e1c39f2ff1927953d2775b932b9d3 Mon Sep 17 00:00:00 2001 From: Zain Dana Harper <17142659+HarperZ9@users.noreply.github.com> Date: Sun, 26 Jul 2026 14:22:17 -0700 Subject: [PATCH 082/131] Give each non-blocking write a distinct task id, refs #2860, #2859 execute_write_fn(fn, block=False) is documented to return "a UUID representing the queued task". Two things stopped that being true. _send_to_write_thread() derived the id from uuid.uuid5(NAMESPACE_DNS, "datasette.io"), which is deterministic, so every non-blocking write in every database in every process returned 3f143baa-4e3d-5842-a36f-4fa2f683b72f. A constant cannot identify a particular task. Now uuid4(). Refs #2860. With num_sql_threads=0 there is no write thread, so execute_write_fn took the synchronous branch and `result` was the write function's return value, normally None. The block=False path then unpacked it unconditionally and raised TypeError: cannot unpack non-iterable NoneType object. The non-threaded branch now returns the same (task_id, reply_future) shape, with the future already resolved because the write has finished, so both modes share one code path. Refs #2859. test_execute_write_fn_block_false only asserted isinstance(task_id, uuid.UUID), which a constant satisfies. The new test is parametrized over threaded and non-threaded and asserts two calls return different ids, so either regression fails it. --- datasette/database.py | 11 ++++++++++- tests/test_internals_database.py | 27 +++++++++++++++++++++++++++ 2 files changed, 37 insertions(+), 1 deletion(-) diff --git a/datasette/database.py b/datasette/database.py index e162d34e..90c4e429 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -354,6 +354,15 @@ class Database: result = fn(self._write_connection) else: result = fn(self._write_connection) + if not block: + # There is no write thread here, so the write has already + # finished. Hand back the same (task_id, reply_future) shape + # _send_to_write_thread() returns, with the future already + # resolved, so the block=False path below is identical in + # both modes. + reply_future = asyncio.get_running_loop().create_future() + reply_future.set_result(result) + result = (uuid.uuid4(), reply_future) else: result = await self._send_to_write_thread( fn, block=block, transaction=transaction @@ -425,7 +434,7 @@ class Database: ) self._write_thread.name = f"_execute_writes for database {self.name}" self._write_thread.start() - task_id = uuid.uuid5(uuid.NAMESPACE_DNS, "datasette.io") + task_id = uuid.uuid4() loop = asyncio.get_running_loop() reply_future = loop.create_future() self._write_queue.put( diff --git a/tests/test_internals_database.py b/tests/test_internals_database.py index b1093b1c..97513123 100644 --- a/tests/test_internals_database.py +++ b/tests/test_internals_database.py @@ -705,6 +705,33 @@ async def test_execute_write_fn_block_false(db): assert isinstance(task_id, uuid.UUID) +@pytest.mark.asyncio +@pytest.mark.parametrize("disable_threads", (False, True)) +async def test_execute_write_fn_block_false_returns_uuid(tmp_path, disable_threads): + # block=False is documented to return "a UUID representing the queued task". + # With num_sql_threads=0 there is no write thread, so the non-threaded branch + # has to satisfy the same contract as the threaded one. + settings = {"num_sql_threads": 0} if disable_threads else {} + ds = Datasette([], memory=True, settings=settings) + await ds.invoke_startup() + db = ds.add_memory_database("test_block_false") + await db.execute_write( + "create table if not exists t (id integer primary key, v text)" + ) + + def write_fn(conn): + conn.execute("insert into t (v) values ('a')") + # Returns None, like most write functions. + + task_id = await db.execute_write_fn(write_fn, block=False) + + assert isinstance(task_id, uuid.UUID) + # Distinct per call, so a caller can tell two queued tasks apart. + second = await db.execute_write_fn(write_fn, block=False) + assert isinstance(second, uuid.UUID) + assert second != task_id + + @pytest.mark.asyncio async def test_execute_write_fn_block_true(db): def write_fn(conn): From bdaa8cc76cc69b4016747cc04f0ec50b418fbb7b Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 3 Sep 2026 14:45:03 -0700 Subject: [PATCH 083/131] Disable extension loading once --load-extension extensions are loaded Refs GHSA-2mvv-ffvc-q5p6 Co-authored-by: Alex Garcia <15178711+asg017@users.noreply.github.com> --- datasette/app.py | 29 ++++++++++++++++++------- tests/test_load_extensions.py | 41 +++++++++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+), 8 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 42be7425..b89ab30c 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -1532,15 +1532,28 @@ class Datasette: conn.row_factory = sqlite3.Row conn.text_factory = lambda x: str(x, "utf-8", "replace") if self.sqlite_extensions and database != INTERNAL_DB_NAME: + # Extension loading is only enabled for as long as it takes to + # load the configured extensions. Leaving it enabled would let + # anyone who can execute SQL call load_extension() themselves. conn.enable_load_extension(True) - for extension in self.sqlite_extensions: - # "extension" is either a string path to the extension - # or a 2-item tuple that specifies which entrypoint to load. - if isinstance(extension, tuple): - path, entrypoint = extension - conn.execute("SELECT load_extension(?, ?)", [path, entrypoint]) - else: - conn.execute("SELECT load_extension(?)", [extension]) + try: + for extension in self.sqlite_extensions: + # "extension" is either a string path to the extension + # or a 2-item tuple that specifies which entrypoint to load. + if isinstance(extension, tuple): + path, entrypoint = extension + if sys.version_info >= (3, 12): + conn.load_extension(path, entrypoint=entrypoint) + else: + # Connection.load_extension() only gained the + # entrypoint argument in Python 3.12 + conn.execute( + "SELECT load_extension(?, ?)", [path, entrypoint] + ) + else: + conn.load_extension(extension) + finally: + conn.enable_load_extension(False) if self.setting("cache_size_kb"): conn.execute(f"PRAGMA cache_size=-{self.setting('cache_size_kb')}") # pylint: disable=no-member diff --git a/tests/test_load_extensions.py b/tests/test_load_extensions.py index 61cdb3e0..a7c2bc24 100644 --- a/tests/test_load_extensions.py +++ b/tests/test_load_extensions.py @@ -1,4 +1,5 @@ from pathlib import Path +from unittest import mock import pytest @@ -20,6 +21,29 @@ def has_compiled_ext(): return False +@pytest.mark.parametrize("load_fails", (False, True)) +def test_load_extension_is_disabled(load_fails): + ds = Datasette(sqlite_extensions=[COMPILED_EXTENSION_PATH]) + connection = mock.Mock() + if load_fails: + connection.load_extension.side_effect = RuntimeError + + if load_fails: + with pytest.raises(RuntimeError): + ds._prepare_connection(connection, "data") + else: + ds._prepare_connection(connection, "data") + + # Extensions are loaded using the Python API, never via SQL + assert connection.load_extension.mock_calls == [ + mock.call(COMPILED_EXTENSION_PATH), + ] + assert connection.enable_load_extension.mock_calls == [ + mock.call(True), + mock.call(False), + ] + + @pytest.mark.asyncio @pytest.mark.skipif(not has_compiled_ext(), reason="Requires compiled ext.c") async def test_load_extension_default_entrypoint(): @@ -64,3 +88,20 @@ async def test_load_extension_multiple_entrypoints(): response = await ds.client.get("/_memory/-/query.json?_shape=arrays&sql=select+c()") assert response.status_code == 200 assert response.json()["rows"][0][0] == "c" + + +@pytest.mark.asyncio +@pytest.mark.skipif(not has_compiled_ext(), reason="Requires compiled ext.c") +async def test_sql_cannot_load_additional_extension(): + ds = Datasette(sqlite_extensions=[COMPILED_EXTENSION_PATH]) + + response = await ds.client.get( + "/_memory/-/query.json", + params={ + "sql": "select load_extension(:path, :entrypoint)", + "path": COMPILED_EXTENSION_PATH, + "entrypoint": "sqlite3_ext_b_init", + }, + ) + assert response.status_code == 400 + assert response.json()["error"] == "not authorized" From c7944fc454c9c7014719cfd6dc3dbb76f4841a9b Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 3 Sep 2026 14:27:18 -0700 Subject: [PATCH 084/131] Skip deploy if environment variables are missing --- .github/workflows/deploy-latest.yml | 33 ++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-) diff --git a/.github/workflows/deploy-latest.yml b/.github/workflows/deploy-latest.yml index 3fc83438..46f03b01 100644 --- a/.github/workflows/deploy-latest.yml +++ b/.github/workflows/deploy-latest.yml @@ -14,24 +14,46 @@ jobs: deploy: runs-on: ubuntu-latest steps: + - name: Check deployment prerequisites + id: deployment-prerequisites + env: + GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }} + LATEST_DATASETTE_SECRET: ${{ secrets.LATEST_DATASETTE_SECRET }} + run: | + missing=() + for variable in GCP_SA_KEY LATEST_DATASETTE_SECRET; do + if [[ -z "${!variable:-}" ]]; then + missing+=("$variable") + fi + done + if (( ${#missing[@]} )); then + echo "::notice::Skipping deployment because required environment variables are missing: ${missing[*]}" + echo "available=false" >> "$GITHUB_OUTPUT" + else + echo "available=true" >> "$GITHUB_OUTPUT" + fi - name: Check out datasette + if: ${{ steps.deployment-prerequisites.outputs.available == 'true' }} uses: actions/checkout@v7 - name: Set up Python + if: ${{ steps.deployment-prerequisites.outputs.available == 'true' }} uses: actions/setup-python@v6 with: python-version: "3.13" cache: pip - name: Install Python dependencies + if: ${{ steps.deployment-prerequisites.outputs.available == 'true' }} run: | python -m pip install --upgrade pip python -m pip install . --group dev python -m pip install sphinx-to-sqlite==0.1a1 - name: Run tests - if: ${{ github.ref == 'refs/heads/main' }} + if: ${{ steps.deployment-prerequisites.outputs.available == 'true' && github.ref == 'refs/heads/main' }} run: | pytest -n auto -m "not serial" pytest -m "serial" - name: Build fixtures.db and other files needed to deploy the demo + if: ${{ steps.deployment-prerequisites.outputs.available == 'true' }} run: |- python tests/fixtures.py \ fixtures.db \ @@ -40,13 +62,14 @@ jobs: plugins \ --extra-db-filename extra_database.db - name: Build docs.db - if: ${{ github.ref == 'refs/heads/main' }} + if: ${{ steps.deployment-prerequisites.outputs.available == 'true' && github.ref == 'refs/heads/main' }} run: |- cd docs DISABLE_SPHINX_INLINE_TABS=1 sphinx-build -b xml . _build sphinx-to-sqlite ../docs.db _build cd .. - name: Set up the alternate-route demo + if: ${{ steps.deployment-prerequisites.outputs.available == 'true' }} run: | echo ' from datasette import hookimpl @@ -58,6 +81,7 @@ jobs: ' > plugins/alternative_route.py cp fixtures.db fixtures2.db - name: And the counters writable stored query demo + if: ${{ steps.deployment-prerequisites.outputs.available == 'true' }} run: | cat > plugins/counters.py < Date: Thu, 3 Sep 2026 14:35:35 -0700 Subject: [PATCH 085/131] execute-write: Check view-table for every table in a CREATE VIEW Refs GHSA-53fc-rhfg-h7qp Co-authored-by: Alex Garcia <15178711+asg017@users.noreply.github.com> --- datasette/utils/sql_analysis.py | 62 +++++++++++++++++++++++++++++- tests/test_queries.py | 68 --------------------------------- 2 files changed, 60 insertions(+), 70 deletions(-) diff --git a/datasette/utils/sql_analysis.py b/datasette/utils/sql_analysis.py index 334545bd..22bb55c4 100644 --- a/datasette/utils/sql_analysis.py +++ b/datasette/utils/sql_analysis.py @@ -1,6 +1,7 @@ from dataclasses import dataclass from typing import Literal +from datasette.utils import escape_sqlite from datasette.utils.sqlite import SQLiteTableType, sqlite3, sqlite_table_type SQLOperation = Literal[ @@ -208,7 +209,9 @@ def analyze_sql_tables( This function is synchronous and connection-based. It temporarily installs a SQLite authorizer, prepares ``EXPLAIN ``, and returns the operation - callbacks observed while SQLite compiles the statement. + callbacks observed while SQLite compiles the statement. ``CREATE VIEW`` is + additionally executed inside a rolled-back savepoint so its source-table reads + can be discovered by analyzing a query against the temporary view. """ operations: dict[OperationKey, set[str]] = {} @@ -532,7 +535,7 @@ def analyze_sql_tables( return None return table_kind_cache[(key.sqlite_schema, key.table)] - return SQLAnalysis( + analysis = SQLAnalysis( operations=tuple( Operation( operation=key.operation, @@ -549,3 +552,58 @@ def analyze_sql_tables( for key, columns in operations.items() ) ) + + # SQLite does not resolve the SELECT body of a view when preparing CREATE + # VIEW, so its authorizer does not report reads from the view's source + # tables. Temporarily create the view, analyze a query against it (which + # does resolve the body), then roll the schema change back. Database-level + # callers use an isolated writable connection for this analysis. + create_view_operations = tuple( + operation + for operation in analysis.operations + if operation.operation == "create" and operation.target_type == "view" + ) + if not create_view_operations: + return analysis + + savepoint = "datasette_analyze_create_view" + conn.execute(f"SAVEPOINT {savepoint}") + try: + conn.execute(sql, params if params is not None else {}) + dependency_reads = [] + for view_operation in create_view_operations: + if view_operation.sqlite_schema is None or view_operation.table is None: + raise sqlite3.OperationalError( + "Could not determine the created view name" + ) + quoted_schema = escape_sqlite(view_operation.sqlite_schema) + quoted_view = escape_sqlite(view_operation.table) + qualified_view = f"{quoted_schema}.{quoted_view}" + view_analysis = analyze_sql_tables( + conn, + f"SELECT * FROM {qualified_view}", + database_name=database_name, + schema_to_database=schema_to_database, + ) + dependency_reads.extend( + operation + for operation in view_analysis.operations + if operation.operation == "read" + and not ( + operation.sqlite_schema == view_operation.sqlite_schema + and operation.table == view_operation.table + ) + ) + finally: + conn.execute(f"ROLLBACK TO {savepoint}") + conn.execute(f"RELEASE {savepoint}") + + existing_operations = set(analysis.operations) + return SQLAnalysis( + operations=analysis.operations + + tuple( + operation + for operation in dependency_reads + if operation not in existing_operations + ) + ) diff --git a/tests/test_queries.py b/tests/test_queries.py index 15b7ad0f..ebe8b832 100644 --- a/tests/test_queries.py +++ b/tests/test_queries.py @@ -3248,74 +3248,6 @@ async def test_execute_write_create_table_uses_create_table_permission(): assert not await db.table_exists("should_not_exist") -@pytest.mark.asyncio -async def test_execute_write_create_view_uses_create_view_permission(): - ds = Datasette( - memory=True, - default_deny=True, - config={ - "permissions": { - "insert-row": {"id": "row-writer"}, - "update-row": {"id": "row-writer"}, - }, - "databases": { - "data": { - "permissions": { - "view-database": {"id": ["creator", "row-writer"]}, - "execute-write-sql": {"id": ["creator", "row-writer"]}, - "create-view": {"id": "creator"}, - } - } - }, - }, - ) - db = ds.add_memory_database("execute_write_create_view", name="data") - await db.execute_write("create table dogs (id integer primary key, name text)") - await ds.invoke_startup() - - analysis_response = await ds.client.get( - "/data/-/execute-write/analyze", - actor={"id": "creator"}, - params={"sql": "create view dog_names as select id, name from dogs"}, - ) - allowed_response = await ds.client.post( - "/data/-/execute-write", - actor={"id": "creator"}, - json={"sql": "create view dog_names as select id, name from dogs"}, - ) - row_permission_response = await ds.client.post( - "/data/-/execute-write", - actor={"id": "row-writer"}, - json={"sql": "create view should_not_exist as select id from dogs"}, - ) - - assert analysis_response.status_code == 200 - analysis_data = analysis_response.json() - assert analysis_data["ok"] is True - assert analysis_data["execute_disabled"] is False - assert analysis_data["analysis_rows"] == [ - { - "operation": "create", - "database": "data", - "table": "dog_names", - "required_permission": "create-view", - "source": None, - "allowed": True, - } - ] - - assert allowed_response.status_code == 200 - assert allowed_response.json()["ok"] is True - assert allowed_response.json()["message"] == "Query executed" - assert await db.view_exists("dog_names") - - assert row_permission_response.status_code == 403 - assert row_permission_response.json()["errors"] == [ - "Permission denied: need create-view on data" - ] - assert not await db.view_exists("should_not_exist") - - @pytest.mark.parametrize( ( "database_name", From c280c47424e87019376f534fbd349fd1a55d53a3 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 3 Sep 2026 14:35:40 -0700 Subject: [PATCH 086/131] POST /db/-/create checks table-level insert/update/alter permissions Refs GHSA-53fc-rhfg-h7qp Co-authored-by: Alex Garcia <15178711+asg017@users.noreply.github.com> --- datasette/views/table_create_alter.py | 12 +-- tests/test_api_write.py | 116 ++++++++++++++++++++++++++ 2 files changed, 123 insertions(+), 5 deletions(-) diff --git a/datasette/views/table_create_alter.py b/datasette/views/table_create_alter.py index 56b28877..f8f8c31e 100644 --- a/datasette/views/table_create_alter.py +++ b/datasette/views/table_create_alter.py @@ -821,16 +821,18 @@ class TableCreateView(BaseView): ignore = create_request.ignore replace = create_request.replace + table_name = create_request.table + table_exists = await db.table_exists(table_name) + table_resource = TableResource(database=database_name, table=table_name) + # Replacing rows requires update-row permission if replace and not await self.ds.allowed( action="update-row", - resource=DatabaseResource(database=database_name), + resource=table_resource, actor=request.actor, ): return Response.error(["Permission denied: need update-row"], 403) - table_name = create_request.table - table_exists = await db.table_exists(table_name) columns = create_request.columns rows = create_request.rows_list @@ -838,7 +840,7 @@ class TableCreateView(BaseView): # Must have insert-row permission if not await self.ds.allowed( action="insert-row", - resource=DatabaseResource(database=database_name), + resource=table_resource, actor=request.actor, ): return Response.error(["Permission denied: need insert-row"], 403) @@ -857,7 +859,7 @@ class TableCreateView(BaseView): if create_request.alter: if not await self.ds.allowed( action="alter-table", - resource=DatabaseResource(database=database_name), + resource=table_resource, actor=request.actor, ): return Response.error( diff --git a/tests/test_api_write.py b/tests/test_api_write.py index 11ef30de..1c560cf5 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -2745,3 +2745,119 @@ async def test_create_using_alter_against_existing_table( insert_rows_event = ds_write._tracked_events[1] assert insert_rows_event.name == "insert-rows" assert insert_rows_event.num_rows == 1 + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("denied_action", "request_body"), + ( + ( + "insert-row", + { + "table": "salaries", + "rows": [{"id": 9, "note": "INJ-VIA-CREATE"}], + }, + ), + ( + "update-row", + { + "table": "salaries", + "rows": [{"id": 1, "note": "REPLACED"}], + "pk": "id", + "replace": True, + }, + ), + ( + "alter-table", + { + "table": "salaries", + "rows": [{"id": 9, "note": "INSERTED", "extra": "NEW"}], + "alter": True, + }, + ), + ), +) +async def test_create_table_existing_table_respects_table_level_denial( + denied_action, request_body +): + # GHSA-53fc-rhfg-h7qp issue 2: POST /db/-/create against an existing table + # inserts rows into it, so insert-row (and update-row / alter-table) must be + # checked against the TableResource, not just the DatabaseResource. + ds = Datasette( + memory=True, + config={ + "databases": { + # id=editor user has each permission at the database level, but + # the selected action is explicitly denied on the salaries table + "data": { + "permissions": { + "create-table": {"id": "editor"}, + "insert-row": {"id": "editor"}, + "update-row": {"id": "editor"}, + "alter-table": {"id": "editor"}, + }, + "tables": { + "salaries": {"permissions": {denied_action: False}}, + }, + } + } + }, + ) + db = ds.add_memory_database( + f"create_table_existing_table_denied_{denied_action}", name="data" + ) + await db.execute_write("create table salaries (id integer primary key, note text)") + await db.execute_write("insert into salaries values (1, 'TOPSECRET-A')") + await ds.invoke_startup() + + if denied_action == "insert-row": + # Sanity: direct insert into salaries is denied for this actor + direct = await ds.client.post( + "/data/salaries/-/insert", + actor={"id": "editor"}, + json={"row": {"id": 9, "note": "INJ-DIRECT"}}, + ) + assert direct.status_code == 403 + + response = await ds.client.post( + "/data/-/create", + actor={"id": "editor"}, + json=request_body, + ) + assert response.status_code == 403, response.json() + assert response.json()["errors"] == [f"Permission denied: need {denied_action}"] + rows = (await db.execute("select id, note from salaries order by id")).rows + assert [tuple(r) for r in rows] == [(1, "TOPSECRET-A")] + assert await db.table_columns("salaries") == ["id", "note"] + + +@pytest.mark.asyncio +async def test_create_table_respects_predeclared_table_level_denial(): + ds = Datasette( + memory=True, + config={ + "databases": { + "data": { + "permissions": { + "create-table": {"id": "editor"}, + "insert-row": {"id": "editor"}, + }, + "tables": { + "planned_table": {"permissions": {"insert-row": False}}, + }, + } + } + }, + ) + db = ds.add_memory_database("create_table_predeclared_denial", name="data") + await ds.invoke_startup() + + response = await ds.client.post( + "/data/-/create", + actor={"id": "editor"}, + json={"table": "planned_table", "rows": [{"id": 1}]}, + ) + + assert response.status_code == 403, response.json() + assert response.json()["errors"] == ["Permission denied: need insert-row"] + assert not await db.table_exists("planned_table") From 577aeb73f06ec48df630e75af47713bf029fc0c8 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 3 Sep 2026 14:35:45 -0700 Subject: [PATCH 087/131] Disallow ?_through= if user lacks view-table permission Refs GHSA-53fc-rhfg-h7qp Co-authored-by: Alex Garcia <15178711+asg017@users.noreply.github.com> --- datasette/filters.py | 7 ++++++- tests/test_table_api.py | 31 +++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+), 1 deletion(-) diff --git a/datasette/filters.py b/datasette/filters.py index 3cfb36e5..af922eda 100644 --- a/datasette/filters.py +++ b/datasette/filters.py @@ -2,7 +2,7 @@ import json from typing import ClassVar from datasette import hookimpl -from datasette.resources import DatabaseResource +from datasette.resources import DatabaseResource, TableResource from datasette.utils.asgi import BadRequest from datasette.views.base import DatasetteError @@ -135,6 +135,11 @@ def through_filters(request, database, table, datasette): through_table = through_data["table"] other_column = through_data["column"] value = through_data["value"] + await datasette.ensure_permission( + action="view-table", + resource=TableResource(database=database, table=through_table), + actor=request.actor, + ) db = datasette.get_database(database) outgoing_foreign_keys = await db.foreign_keys_for_table(through_table) fk_to_us = next( diff --git a/tests/test_table_api.py b/tests/test_table_api.py index 6c0c021b..ec4a1368 100644 --- a/tests/test_table_api.py +++ b/tests/test_table_api.py @@ -1778,3 +1778,34 @@ async def test_next_url_included_by_default(ds_client): data = response.json() assert data["next"] is None assert data["next_url"] is None + + +@pytest.mark.asyncio +async def test_table_through_requires_view_table_on_through_table(): + # GHSA-53fc-rhfg-h7qp issue 3: ?_through= runs a sub-select against the + # caller-supplied through table, so the actor must be allowed to view it. + # Otherwise it is an equality oracle over any column of a denied table. + from datasette.app import Datasette + + ds = Datasette( + memory=True, + config={"databases": {"data": {"tables": {"salaries": {"allow": False}}}}}, + ) + db = ds.add_memory_database("table_through_denied", name="data") + await db.execute_write("create table people (id integer primary key, name text)") + await db.execute_write( + "create table salaries (id integer primary key, " + "person_id integer references people(id), note text)" + ) + await db.execute_write("insert into people values (1, 'alice'), (2, 'bob')") + await db.execute_write("insert into salaries values (1, 1, 'TOPSECRET-A')") + await ds.invoke_startup() + + # Sanity: anonymous cannot read salaries directly + assert (await ds.client.get("/data/salaries.json")).status_code == 403 + + response = await ds.client.get( + "/data/people.json?_shape=array" + '&_through={"table":"salaries","column":"note","value":"TOPSECRET-A"}' + ) + assert response.status_code == 403, response.text From f8e8e65af7403666f227bb6f0d523bcf2d1e11aa Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 3 Sep 2026 14:35:48 -0700 Subject: [PATCH 088/131] actor cookie respects expire_after Refs GHSA-53fc-rhfg-h7qp Co-authored-by: Alex Garcia <15178711+asg017@users.noreply.github.com> --- datasette/app.py | 2 +- tests/test_auth.py | 22 ++++++++++++++++++++++ 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/datasette/app.py b/datasette/app.py index b89ab30c..6683d4dc 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -2462,7 +2462,7 @@ class Datasette: ): data = {"a": actor} if expire_after: - expires_at = int(time.time()) + (24 * 60 * 60) + expires_at = int(time.time()) + expire_after data["e"] = baseconv.base62.encode(expires_at) response.set_cookie("ds_actor", self.sign(data, "actor")) diff --git a/tests/test_auth.py b/tests/test_auth.py index e7a5402e..6024e3bb 100644 --- a/tests/test_auth.py +++ b/tests/test_auth.py @@ -524,3 +524,25 @@ async def test_root_without_root_enabled_no_special_permissions(ds_client): ) is not True ), "Root without root_enabled should not automatically get set-column-type" + + +@pytest.mark.parametrize("expire_after", (1, 300, 3600, 30 * 24 * 60 * 60)) +def test_set_actor_cookie_honours_expire_after(expire_after): + # GHSA-53fc-rhfg-h7qp issue 4: expire_after is documented as a number of + # seconds, but every value was being replaced with 24 hours. + from datasette.app import Datasette + from datasette.utils.asgi import Response + + ds = Datasette(memory=True) + response = Response.text("") + before = int(time.time()) + ds.set_actor_cookie(response, {"id": "test"}, expire_after=expire_after) + after = int(time.time()) + + (header,) = response._set_cookie_headers + assert header.startswith("ds_actor=") + value = header[len("ds_actor=") :].split(";", 1)[0] + data = ds.unsign(value, "actor") + assert data["a"] == {"id": "test"} + expires_at = baseconv.base62.decode(data["e"]) + assert before + expire_after <= expires_at <= after + expire_after From 435e55ff0a254a77f700a06f5c31bb9f3bf31764 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 3 Sep 2026 14:35:53 -0700 Subject: [PATCH 089/131] Remove JSON syntax highlighting Refs GHSA-hp2x-vx2r-6vxg Co-authored-by: Alex Garcia <15178711+asg017@users.noreply.github.com> --- .../static/json-format-highlight-1.0.1.js | 56 ------------------- datasette/templates/api_explorer.html | 5 +- datasette/templates/debug_allowed.html | 5 +- datasette/templates/debug_check.html | 5 +- datasette/templates/debug_rules.html | 5 +- 5 files changed, 8 insertions(+), 68 deletions(-) delete mode 100644 datasette/static/json-format-highlight-1.0.1.js diff --git a/datasette/static/json-format-highlight-1.0.1.js b/datasette/static/json-format-highlight-1.0.1.js deleted file mode 100644 index 0e6e2c29..00000000 --- a/datasette/static/json-format-highlight-1.0.1.js +++ /dev/null @@ -1,56 +0,0 @@ -/* -https://github.com/luyilin/json-format-highlight -From https://unpkg.com/json-format-highlight@1.0.1/dist/json-format-highlight.js -MIT Licensed -*/ -(function (global, factory) { - typeof exports === "object" && typeof module !== "undefined" - ? (module.exports = factory()) - : typeof define === "function" && define.amd - ? define(factory) - : (global.jsonFormatHighlight = factory()); -})(this, function () { - "use strict"; - - var defaultColors = { - keyColor: "dimgray", - numberColor: "lightskyblue", - stringColor: "lightcoral", - trueColor: "lightseagreen", - falseColor: "#f66578", - nullColor: "cornflowerblue", - }; - - function index(json, colorOptions) { - if (colorOptions === void 0) colorOptions = {}; - - if (!json) { - return; - } - if (typeof json !== "string") { - json = JSON.stringify(json, null, 2); - } - var colors = Object.assign({}, defaultColors, colorOptions); - json = json.replace(/&/g, "&").replace(//g, ">"); - return json.replace( - /("(\\u[a-zA-Z0-9]{4}|\\[^u]|[^\\"])*"(\s*:)?|\b(true|false|null)\b|-?\d+(?:\.\d*)?(?:[eE][+]?\d+)?)/g, - function (match) { - var color = colors.numberColor; - if (/^"/.test(match)) { - color = /:$/.test(match) ? colors.keyColor : colors.stringColor; - } else { - color = /true/.test(match) - ? colors.trueColor - : /false/.test(match) - ? colors.falseColor - : /null/.test(match) - ? colors.nullColor - : color; - } - return '' + match + ""; - }, - ); - } - - return index; -}); diff --git a/datasette/templates/api_explorer.html b/datasette/templates/api_explorer.html index 4927cb8d..32686af1 100644 --- a/datasette/templates/api_explorer.html +++ b/datasette/templates/api_explorer.html @@ -3,7 +3,6 @@ {% block title %}API Explorer{% endblock %} {% block extra_head %} - {% endblock %} {% block content %} @@ -126,7 +125,7 @@ getForm.addEventListener("submit", (ev) => { document.getElementById('response-status').textContent = response.status; return response.json(); }).then((data) => { - output.querySelector('pre').innerHTML = jsonFormatHighlight(data); + output.querySelector('pre').textContent = JSON.stringify(data, null, 2); errorList.style.display = 'none'; }).catch((error) => { alert(error); @@ -174,7 +173,7 @@ postForm.addEventListener("submit", (ev) => { } else { errorList.style.display = 'none'; } - output.querySelector('pre').innerHTML = jsonFormatHighlight(data); + output.querySelector('pre').textContent = JSON.stringify(data, null, 2); output.style.display = 'block'; }).catch(err => { alert("Error: " + err); diff --git a/datasette/templates/debug_allowed.html b/datasette/templates/debug_allowed.html index 80249d9c..c73cdfb7 100644 --- a/datasette/templates/debug_allowed.html +++ b/datasette/templates/debug_allowed.html @@ -3,7 +3,6 @@ {% block title %}Allowed Resources{% endblock %} {% block extra_head %} - {% include "_permission_ui_styles.html" %} {% include "_debug_common_functions.html" %} {% endblock %} @@ -198,7 +197,7 @@ function displayResults(data) { } // Update raw JSON - document.getElementById('raw-json').innerHTML = jsonFormatHighlight(data); + document.getElementById('raw-json').textContent = JSON.stringify(data, null, 2); } function displayError(data) { @@ -208,7 +207,7 @@ function displayError(data) { resultsContent.innerHTML = `
Error: ${escapeHtml(data.error || 'Unknown error')}
`; - document.getElementById('raw-json').innerHTML = jsonFormatHighlight(data); + document.getElementById('raw-json').textContent = JSON.stringify(data, null, 2); } // Disable child input if parent is empty diff --git a/datasette/templates/debug_check.html b/datasette/templates/debug_check.html index b9fc636a..c0081c66 100644 --- a/datasette/templates/debug_check.html +++ b/datasette/templates/debug_check.html @@ -3,7 +3,6 @@ {% block title %}Explain a permission decision{% endblock %} {% block extra_head %} - {% include "_permission_ui_styles.html" %} {% include "_debug_common_functions.html" %}
{}{i}{i}{i}a{i}