From 8985ecf4387164e76147e7188693a08dc1a7dd1e Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 1 Jul 2026 22:32:28 +0000 Subject: [PATCH 01/46] Add JSON API reference and 1.0 stability review documents - existing-api.md: complete reference for the JSON API as implemented, derived from source code (routes, views, renderer, permissions) - stable-api-recommendations.md: consistency and completeness review with prioritized recommendations for the 1.0 stable release Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- existing-api.md | 1211 +++++++++++++++++++++++++++++++++ stable-api-recommendations.md | 367 ++++++++++ 2 files changed, 1578 insertions(+) create mode 100644 existing-api.md create mode 100644 stable-api-recommendations.md diff --git a/existing-api.md b/existing-api.md new file mode 100644 index 00000000..b3eda9d4 --- /dev/null +++ b/existing-api.md @@ -0,0 +1,1211 @@ +# Datasette JSON API — As Implemented + +This document describes the JSON API of this Datasette codebase (version `1.0a35`) as +derived directly from the source code. It intentionally ignores the existing `docs/` +directory: every claim below is based on the route table in `datasette/app.py` +(`Datasette._routes()`, app.py:2507-2767) and the view implementations in +`datasette/views/`. + +## Contents + +- [Cross-cutting behavior](#cross-cutting-behavior) +- [Instance endpoints](#instance-endpoints) +- [Database endpoints](#database-endpoints) +- [Table and row read endpoints](#table-and-row-read-endpoints) +- [The write API](#the-write-api) +- [Stored (canned) queries API](#stored-canned-queries-api) +- [Authentication and tokens](#authentication-and-tokens) +- [Appendix: registered actions (permissions)](#appendix-registered-actions-permissions) + +--- + +## Cross-cutting behavior + +### URL formats and content negotiation + +- Most read endpoints are registered with an optional format suffix: + `/(...)(\.(?Pjson))?$`. The bare path returns HTML; the `.json` + extension returns JSON. The homepage additionally accepts the legacy + `.jsono` extension, which returns identical JSON (app.py:2517-2518). +- Table, row and query routes accept any `\w+` format extension; formats other + than the built-in `html`, `json`, `csv`, `blob` must be provided by a plugin + via `register_output_renderer`, otherwise the request 404s. +- HTML responses include a `Link: <...>; rel="alternate"; + type="application/json+datasette"` header pointing at the `.json` variant + (views/base.py:141-159), unless the view opts out with + `has_json_alternate = False`. +- Database, table, row and query names in paths are **tilde-encoded** + (a percent-encoding variant using `~` as the escape character; + utils/__init__.py `_TILDE_ENCODING_SAFE`). Multi-column primary keys in row + URLs are comma-separated. +- JSON responses are always compact `json.dumps` output serialized by + `CustomJSONEncoder`; there is no pretty-printing query parameter. Binary + values are serialized as `{"$base64": true, "encoded": "..."}`. +- Success content type: `application/json; charset=utf-8` + (`_shape=array&_nl=on` responses use `text/plain`). + +### Error shapes (there are several) + +The codebase produces **four distinct JSON error shapes**, depending on which +layer generates the error: + +1. **Exception handler** (handle_exception.py:21-59) — used when a view raises + `NotFound`, `Forbidden` (JSON paths only — see below), `DatasetteError`, + `BadRequest` etc. and the request path ends in `.json`: + + ```json + {"ok": false, "error": "message", "status": 404, "title": null} + ``` + +2. **The `_error()` helper** (views/base.py:183-184) — used by the write API, + stored-query API, execute-write and several permission-denied paths: + + ```json + {"ok": false, "errors": ["message", "..."]} + ``` + + Note: plural `errors`, a list, and no `status`/`title` keys. + +3. **JSON renderer errors** (renderer.py:52-56) — SQL errors on table/query + endpoints return HTTP 400 with the error embedded in the data envelope: + + ```json + {"ok": false, "error": "no such table: x", "rows": [], "truncated": false} + ``` + + An invalid `_shape=` value produces `{"ok": false, "error": "Invalid _shape: x", + "status": 400, "title": null}` (renderer.py:101-108). + +4. **Ad-hoc `{"error": ...}` objects** — the permission debug endpoints + (`/-/allowed`, `/-/rules`, `/-/check`, POST `/-/permissions`) return e.g. + `{"error": "Unknown action: x"}` with no `ok` key (views/special.py). + +Method-not-allowed responses return HTTP 405 +`{"ok": false, "error": "Method not allowed"}` when the path ends in `.json` +or the request content type is `application/json`; plain text otherwise +(views/base.py:53, 88-98). + +**`Forbidden` is special:** when a view raises `Forbidden` (e.g. via +`ensure_permission`), the default `forbidden()` plugin hook renders an **HTML +error page with status 403 even for `.json` requests** +(forbidden.py:4-19, app.py:2895-2904). Endpoints that check permissions +themselves and return `_error(..., 403)` produce JSON instead. So a JSON +client may receive either an HTML 403 page or a JSON 403 body depending on +the endpoint. + +### CORS + +When Datasette is started with `--cors`, responses gain +(utils/__init__.py:1297-1302): + +``` +Access-Control-Allow-Origin: * +Access-Control-Allow-Headers: Authorization, Content-Type +Access-Control-Expose-Headers: Link +Access-Control-Allow-Methods: GET, POST, HEAD, OPTIONS +Access-Control-Max-Age: 3600 +``` + +### CSRF / cross-origin protection + +Datasette uses header-based cross-origin protection +(`CrossOriginProtectionMiddleware`, csrf.py:67-178) rather than CSRF tokens +for API calls. For non-GET/HEAD/OPTIONS requests: + +1. Requests carrying `Authorization: Bearer ...` **and no `Cookie` header** + bypass the check entirely (csrf.py:98-110). +2. Otherwise `Sec-Fetch-Site` must be `same-origin` or `none`; other values → 403. +3. If neither `Sec-Fetch-Site` nor `Origin` is present (curl, API clients), + the request passes. +4. Fallback: `Origin` must exactly match the request scheme/host/port → else 403. + +Plain JSON API clients (no cookies, no browser headers) are never blocked; +`Content-Type: application/json` itself plays no role in the CSRF decision. + +### Settings that govern the API + +From `SETTINGS` (app.py:197-287): `default_page_size` (100), +`max_returned_rows` (1000), `max_insert_rows` (100), `sql_time_limit_ms` +(1000), `default_facet_size` (30), `facet_time_limit_ms` (200), +`allow_facet` (true), `allow_download` (true), `allow_signed_tokens` (true), +`default_allow_sql` (true), `max_signed_tokens_ttl` (0), `default_cache_ttl` +(5), `allow_csv_stream` (true), `max_csv_mb` (100), `force_https_urls` +(false), `trace_debug` (false), `base_url` ("/"). + +### The JSON renderer: `_shape`, `_nl`, `_json`, `_json_infinity` + +`json_renderer` (renderer.py:31-126) processes `.json` output for table, row +and query views (but **not** for the instance/database/debug endpoints, which +build JSON directly): + +- **`_shape`** (default `objects`): + - `objects` — `{"ok": true, "rows": [{col: val}, ...], "truncated": false, ...}` + - `arrays` — same envelope, each row a list of values + - `array` — response body is a bare JSON array of row objects + - `arrayfirst` — bare JSON array of the first column's values + - `object` — table views only: an object keyed by primary-key string. + On queries: `{"ok": false, "error": "_shape=object is only available on + tables"}` (with HTTP status 200); on tables without primary keys a similar + error. + - anything else — HTTP 400 `{"ok": false, "error": "Invalid _shape: x", + "status": 400, "title": null}` +- **`_nl=on`** — with `_shape=array` only: newline-delimited JSON, `text/plain`. +- **`_json=COLUMN`** (repeatable) — parse that column's string values with + `json.loads` so they nest as JSON; parse failures leave the value unchanged. +- **`_json_infinity=1`** — preserve `Infinity`/`-Infinity`; by default they + are replaced with `null`. +- `columns` is stripped from dict-shaped output unless `?_extra=columns` was + requested (renderer.py:110-113). +- If a SQL error occurred, `_shape` is ignored, HTTP status is 400 and the + envelope carries `"ok": false, "error": ...` (renderer.py:52-56). + +### The `?_extra=` system + +Table, row and query JSON responses support `?_extra=` (repeatable and/or +comma-separated, extras.py:9-14) to add keys to the response. Extras are +scope-registered (`ExtraScope.TABLE` / `ROW` / `QUERY`) and only **public** +extras are available over JSON (extras.py:73-92). Unknown extra names are +silently ignored. The available names per scope are listed with the relevant +endpoints below. + +--- + +## Instance endpoints + +Most of these are implemented with `JsonDataView` (views/special.py:30-79): +GET-only; bare path renders an HTML page (`show_json.html`), `.json` returns +the data; permission defaults to `view-instance` and denial raises +`Forbidden` → **HTML** 403 page. + +### GET / + +Routes: `/(\.(?Pjsono?))?$` and `/-/(\.(?Pjsono?))?$` +(app.py:2517-2518); `/-` permanently redirects to `/-/`. `IndexView` +(views/index.py:22-189). `GET /.json`, `/.jsono` and `/-/.json` return JSON. + +- **Permission:** `view-instance` (denied → 403). Databases and tables are + further filtered by `view-database` / `view-table` for the actor. +- **Parameters:** `_sort=relationships` sorts each database's truncated table + list by foreign-key relationship count. +- **JSON response** (index.py:147-161): + - `databases` — an **object keyed by database name** (not a list). Each + value: `name`, `hash` (or null), `color`, `path`, + `tables_and_views_truncated` (up to 5 items: `name`, `columns`, + `primary_keys`, `count` (int or null), `hidden`, `fts_table`, + `num_relationships_for_sorting`, `private`; view items are just + `{"name", "private"}`), `tables_and_views_more` (bool), `tables_count`, + `table_rows_sum`, `show_table_row_counts`, `hidden_table_rows_sum`, + `hidden_tables_count`, `views_count`, `private`. + - `metadata` — instance metadata object. + +### GET /-/versions(.json) + +`JsonDataView` over `Datasette._versions` (app.py:2548-2551, 2171-2245). +Permission `view-instance`. No parameters. + +Response keys: `python` (`{version, full}`), `datasette` (`{version}` plus +optional `note`), `asgi` (`"3.0"`), `uvicorn` (string or null), `sqlite` +(`{version, fts_versions, extensions, compile_options}`; `extensions` +includes `json1` and optionally `spatialite`), `pysqlite3` (only when +running under pysqlite3). + +### GET /-/plugins(.json) + +app.py:2552-2557, `Datasette._plugins` (app.py:2247-2266). Permission +`view-instance`. + +- **Parameters:** `?all=1` — include Datasette's built-in default plugins + (filtered out by default). +- **Response:** a JSON **array**, sorted by name, of + `{"name", "static", "templates", "version", "hooks"}`. + +### GET /-/settings(.json) + +app.py:2558-2561. Permission `view-instance`. No parameters. Returns a flat +object mapping every setting name (see [Settings](#settings-that-govern-the-api)) +to its effective value. + +### GET /-/config(.json) + +app.py:2562-2565. Permission `view-instance`. No parameters. Returns the full +`datasette.yaml` configuration dict passed through +`redact_keys(config, ("secret", "key", "password", "token", "hash", "dsn"))` +(app.py:2502-2505) — any dict key containing one of those substrings has its +value replaced by `"***"` (utils/__init__.py:1532-1556). + +### GET /-/threads(.json) + +app.py:2566-2569, `Datasette._threads` (app.py:2268-2285). Permission +`view-instance`. No parameters. + +Response: `num_threads`, `threads` (list of `{name, ident, daemon}`), +`num_tasks`, `tasks` (asyncio task repr strings). When the +`num_sql_threads` setting is 0 the response is exactly +`{"num_threads": 0, "threads": []}`. + +### GET /-/databases(.json) + +app.py:2570-2573, `Datasette._connected_databases` (app.py:2157-2169). +Permission `view-instance`. No parameters. + +Response: a JSON array of `{"name", "route", "path", "size", "is_mutable", +"is_memory", "hash"}` — **all attached databases are listed regardless of +per-database `view-database` permissions**. + +### GET /-/actor(.json) + +app.py:2574-2579, registered with `permission=None` — **accessible to any +request including anonymous**. No parameters. + +Response: `{"actor": {...}}` or `{"actor": null}` (app.py:2287-2288). + +### GET /-/actions(.json) + +app.py:2580-2589. Permission **`permissions-debug`**. No parameters. + +Response: a JSON array, sorted by name, of `{"name", "abbr", "description", +"takes_parent", "takes_child", "resource_class", "also_requires"}` +(app.py:2290-2304). + +### GET /-/auth-token + +`AuthTokenView` (app.py:2590-2593, views/special.py:198-217). GET only, no +`.json` variant, HTML/redirect only. + +- **Parameter:** `token` — the one-time secret printed by `datasette --root`. +- Match → invalidates the token, sets the signed `ds_actor` cookie to + `{"id": "root"}` and 302-redirects to the homepage. Mismatch or reuse → + `Forbidden` → 403 HTML. + +### GET/POST /-/create-token + +`CreateTokenView` (app.py:2594-2597, views/special.py:727-856). **HTML form +endpoint only — there is no JSON request/response mode in this codebase** +(`has_json_alternate = False`; the POST body must be form-encoded, a JSON +content type raises `BadRequest` → 400). + +- **Gates** (each failure → `Forbidden` → 403): `allow_signed_tokens` must be + on; request must have an actor with an `id`; the actor must not itself be + token-derived. +- **POST fields:** `expire_type` (`""`/`minutes`/`hours`/`days`), + `expire_duration` (positive int), plus restriction checkboxes named + `all:`, `database::`, + `resource:::`. +- **Response:** HTML page containing the new `dstok_` token. +- Programmatic alternatives: `datasette create-token` CLI or + `datasette.create_token()`. + +### GET /-/api + +`ApiExplorerView` (app.py:2598-2601, views/special.py:859-1020). HTML API +explorer, GET only. Permission `view-instance` (403 on denial). + +### GET /-/jump(.json) + +`JumpView` (app.py:2602-2605, views/special.py:1023-1201). The route allows +an optional `.json` suffix but the view **always returns JSON**. + +- **Permission:** none checked directly; results are filtered via + `allowed_resources_sql` for the current actor (default items come from the + `jump_items_sql` plugin hook). +- **Parameter:** `q` — whitespace-split terms matched as a case-insensitive + `%term1%term2%` LIKE pattern. +- **Response:** `{"matches": [...], "truncated": bool}`; each match: + `name`, `url`, `type` (`database`/`table`/`view`/`query`/plugin-defined), + `description`, optional `display_name`. Capped at 100 matches. + +### GET /-/schema(.json|.md) + +`InstanceSchemaView` (app.py:2610-2613, views/special.py:1257-1293). + +- **Permission:** no explicit check; only databases the actor can + `view-database` are included (others silently omitted). +- **Formats:** no extension → HTML; `.json` → + `{"schemas": [{"database": name, "schema": "..."}]}`; `.md` → + `text/markdown` rendering. + +### GET/POST /-/logout + +`LogoutView` (app.py:2614-2617, views/special.py:220-238). HTML endpoint. +GET renders a confirmation page (or redirects if anonymous); POST deletes the +`ds_actor` cookie and 302-redirects to `/`. + +### GET/POST /-/permissions + +`PermissionsDebugView` (app.py:2618-2621, views/special.py:241-295). No +`.json` route. Both methods require `view-instance` **and** +`permissions-debug` (403 on denial). + +- **GET** — HTML permission-check log; `?filter=all|exclude-yours|only-yours`. +- **POST** — form-encoded `actor` (JSON string), `permission`, optional + `resource_1`, `resource_2`; returns **JSON** + `{"action", "allowed", "resource": {"parent", "child", "path"}}` plus + `actor_id` when present. Errors: unknown action → 404 `{"error": ...}`; + child without parent → 400 `{"error": ...}`. + +### GET /-/allowed(.json) + +`AllowedResourcesView` (app.py:2622-2625, views/special.py:298-460). Bare +path always renders the HTML form; `.json` returns JSON. + +- **Permission:** none — reports the **current actor's own** allowed + resources. Items gain a `reason` field if the actor also holds + `permissions-debug`. +- **Parameters:** `action` (required; missing → 400 `{"error": ...}`, unknown + → 404), `parent`, `child` (requires `parent`), `page` (default 1), + `page_size` (default 50, silently capped at 200). +- **Response:** `{"action", "actor_id", "page", "page_size", "total", + "items": [{"parent", "child", "resource"}]}` with optional `next_url` / + `previous_url`. + +### GET /-/rules(.json) + +`PermissionRulesView` (app.py:2626-2629, views/special.py:463-584). +Permission `view-instance` **and** `permissions-debug`. Parameters and error +shapes as `/-/allowed`. Response items: +`{"parent", "child", "resource", "allow" (1|0), "reason", "source_plugin"}`. + +### GET /-/check(.json) + +`PermissionCheckView` (app.py:2630-2633, views/special.py:633-662). +Permission `permissions-debug`. Parameters `action` (required), `parent`, +`child`. Checks the **current request's actor**; response +`{"action", "allowed", "resource": {...}}` plus `actor_id`. + +### GET/POST /-/messages + +`MessagesDebugView` (app.py:2634-2637, views/special.py:703-724). HTML debug +tool for flash messages; permission `view-instance`; POST is form-encoded +(`message`, `message_type` = INFO/WARNING/ERROR/all) and 302-redirects. + +### GET /-/allow-debug + +`AllowDebugView` (app.py:2638-2641, views/special.py:665-700). GET only, HTML +only, **no permission required**. Parameters `actor` and `allow` (JSON +strings); renders the result of `actor_matches_allow()` in the page. + +### GET /-/patterns + +Pattern portfolio page (app.py:2642-2645). HTML only; not part of the JSON API. + +### GET /-/debug/autocomplete + +`AutocompleteDebugView` (app.py:2646-2649, views/special.py:94-195). HTML +debug page for the table autocomplete API; permission `view-instance` plus +`view-table` when `?database=&table=` are supplied. + +--- + +## Database endpoints + +### GET /\.db + +Downloads the raw SQLite file. Route → `database_download` +(app.py:2650-2653; views/database.py:533-570). + +- **Permission:** `view-database-download` (denied → `Forbidden` → 403 HTML). +- **Other gates:** unknown database → 404 `"Invalid database"`; in-memory + database → 404; `allow_download` off **or** mutable database → + `Forbidden("Database download is forbidden")`; no file path → 404. +- **Response:** streamed `application/octet-stream` with a + `content-disposition` attachment; immutable databases with a known hash set + `Etag` and honor `If-None-Match` → 304. + +### GET /\(.json) + +`DatabaseView` (app.py:2654-2657; views/database.py:71-277). Only `html` and +`json` formats are accepted; any other extension → 404 `"Invalid format: ..."`. + +- **Permission:** `view-database` via `check_visibility` (denied → + `Forbidden` → 403 HTML). Table/view listings are filtered by `view-table`; + stored queries by `view-query`. +- **Parameters:** + - `?sql=` — non-blank value 302-redirects to `//-/query?...` + preserving the query string and format. + - No `?_extra=` and no `_shape` support — the JSON is built directly and + returned via `Response.json`, bypassing the JSON renderer + (views/database.py:189-212). +- **JSON response** (all keys always present): + - `ok` — always `true` + - `database` — name; `private` — bool; `path` — URL path; `size` — bytes + - `tables` — list (includes hidden tables), each: + `name`, `columns` (names), `primary_keys`, `count` (int or null, + time-boxed), `count_truncated` (bool — count is a capped lower bound), + `hidden`, `fts_table`, `foreign_keys` (`{incoming: [...], outgoing: [...]}` + of `{other_table, column, other_column}`), `private` + - `hidden_count` — number of hidden tables + - `views` — list of `{name, private}` + - `queries` — **up to 5** stored queries (canonical stored-query objects, + see the stored-queries section); `queries_more` (bool); + `queries_count` (total visible) + - `allow_execute_sql` — bool for this actor + - `table_columns` — `{table: [columns]}`, empty `{}` unless + `allow_execute_sql` (views map to `[]`) + - `metadata` — database metadata dict + +### GET /\/-/query(.json) — arbitrary SQL + +`QueryView` (app.py:2691-2694; views/database.py:573-1130). The same class +also executes stored queries dispatched from the table route (see stored +queries section). + +- **Permission:** `execute-sql` on the database via `check_visibility` + (denied → `Forbidden` → 403 HTML). +- **Parameters:** + - `sql` — SQL to run. Must pass `validate_sql_select` + (utils/__init__.py:345-354): after stripping `--` comment lines it must + start with `select`, `with` or an `explain` variant, and must not contain + `pragma` (except allowlisted `pragma_*()` table-valued functions). + Failure → 400 `DatasetteError` titled `"Invalid SQL"` → JSON + `{"ok": false, "error": "Statement must be a SELECT", "status": 400, + "title": "Invalid SQL"}`. + - Any other `name=value` pair supplies the `:name` named parameter; missing + parameters default to `""`. Names starting with `_` are excluded. + - `_timelimit` — per-request SQL time limit in ms. + - `_shape`, `_nl`, `_json`, `_json_infinity` — see the JSON renderer section. + - `_extra` — QUERY-scope extras: `columns`, `debug`, `request`, + `render_cell`, `query` (`{"sql", "params"}`), `metadata`, `database`, + `database_color`, `private`, `extras`. +- **Response** (default shape): + `{"ok": true, "rows": [{col: val}, ...], "truncated": false}` plus any + requested extras. `truncated: true` when the result hit `max_returned_rows`. +- **Errors:** + - SQLite errors (e.g. `no such table`) are **not** raised — they surface as + HTTP 400 `{"ok": false, "error": "", "rows": [], "truncated": false}`. + - Time limit → 400 titled `"SQL Interrupted"` (the `error` value contains + an HTML fragment). + - `?sql=` omitted → 200 `{"ok": true, "rows": [], "truncated": false}` + (the CSV format instead errors 400 `"?sql= is required"`). +- `.csv` streams CSV; unknown extensions → 404. + +### GET /\/-/query/parameters + +`QueryParametersView` (app.py:2687-2690; views/stored_queries.py:26-51). + +- **Permission:** `execute-sql` → 403 JSON + `{"ok": false, "errors": ["Permission denied: need execute-sql"]}`. +- **Parameters:** only `sql` (default `""`); any other key → 400 + `"Invalid keys: ..."`. +- **Response:** 200 `{"ok": true, "parameters": ["name1", ...]}`. SQL with a + parameter beginning `_` → 400 `"Magic parameters are not allowed"`. +- Responses carry `Content-Security-Policy: frame-ancestors 'none'` and + `X-Frame-Options: DENY`. + +### POST /\/-/create + +`TableCreateView` (app.py:2658; views/table_create_alter.py:785-962). +GET → 405. Body is parsed as JSON regardless of content type; invalid JSON → +400 `{"ok": false, "errors": ["Invalid JSON: ..."]}`. + +- **Permissions** (all denials → 403 `{"ok": false, "errors": [...]}`, + all checked at the **database** level): + - `create-table` — always required (`["Permission denied"]`) + - `insert-row` — if `rows`/`row` provided (`need insert-row`) + - `update-row` — if `replace: true` (`need update-row`) + - `alter-table` — if `alter: true` on an **existing** table + (`need alter-table`); when the table does not exist yet and rows are + supplied, alter is enabled automatically. +- **Request schema** (pydantic `CreateTableRequest`, extra keys forbidden → + 400 `"Invalid keys: a, b"`): + - `table` (required) — must match `^(?!sqlite_)[^\n]+$` + - `rows` (list of objects) / `row` (single object) — mutually exclusive + - `columns` — list of `{name, type, fk_table, fk_column, not_null, + default, default_expr}`; mutually exclusive with `rows`/`row`; `type` one + of `text`/`integer`/`float`/`blob` (default `text`); `default` and + `default_expr` mutually exclusive; `default_expr` one of + `current_timestamp`, `current_date`, `current_time`, `current_unixtime`, + `current_unixtime_ms`. At least one of `columns`/`rows`/`row` required. + - `pk` (string) / `pks` (list) — mutually exclusive. For an existing table + a differing pk → 400 `"pk cannot be changed for existing table"`. + - `ignore` / `replace` (bools) — mutually exclusive; require `row`/`rows` + and `pk`/`pks`. + - `alter` (bool) — add missing columns when inserting into an existing table. +- **Success** — **201**: + ```json + {"ok": true, "database": "...", "table": "...", + "table_url": "https://.../db/table", "table_api_url": "https://.../db/table.json", + "schema": "CREATE TABLE ...", "row_count": 2} + ``` + `row_count` only when rows were inserted. Write failures → 400 + `{"ok": false, "errors": [""]}`. Emits `create-table` / + `insert-rows` / `alter-table` events. + +### POST /\/-/execute-write + +`ExecuteWriteView` (app.py:2679-2682; views/execute_write.py:236-476). GET on +the same path renders an HTML form (requires `execute-write-sql`). + +- **Permission (POST):** `execute-write-sql` → 403 + `{"ok": false, "errors": ["Permission denied: need execute-write-sql"]}`; + immutable database → 403 `["Database is immutable"]`. +- **Per-statement permissions:** the SQL is analyzed + (`decision_for_write_sql_operation`, write_sql.py:63-189) and each + operation must pass: + + | Operation | Requirement | + |---|---| + | `select` / internal ops / function calls | ignored | + | read of a table | `view-table` on that table | + | `insert` or `update` | **all of** `insert-row`, `update-row`, `delete-row` on the table | + | `delete` | `delete-row` | + | `create table` | `create-table` on the database | + | `alter table`, `create index`, `drop index` | `alter-table` on the table | + | `drop table` | `drop-table` | + | `VACUUM`, virtual-table writes, shadow-table writes | rejected outright (403) | + | statements touching attached databases | rejected (403) | + +- **Body:** JSON (`{"sql": ..., "params": {...}}` — only those two keys) or + form-encoded (`sql` plus one field per parameter, `_sql_param_` prefix + stripped). Validation errors (400): `"SQL is required"`, + `"params must be a dictionary"`, `"Unknown parameters: a, b"`, + `"Magic parameters are not allowed"`, `"Could not analyze query: ..."`, + `"Use /-/query for read-only SQL; this endpoint only executes writes"`. +- **JSON is returned when** the body was JSON, `Accept: application/json`, or + a truthy `_json` field is present; otherwise HTML. +- **Success** — 200: + ```json + {"ok": true, "message": "Query executed, 1 row affected", "rowcount": 1, + "rows": [], "truncated": false, + "analysis": [{"operation": "insert", "database": "db", "table": "t", + "required_permission": "insert-row, update-row, delete-row", + "source": null}]} + ``` + `rows` is populated by `RETURNING` clauses. SQLite errors → 400 + `{"ok": false, "errors": [""]}`. Anti-framing headers on all + responses. + +### GET /\/-/execute-write/analyze + +`ExecuteWriteAnalyzeView` (app.py:2675-2678; views/execute_write.py:479-507). + +- **Permission:** `execute-write-sql` → 403 `errors` JSON. +- **Parameters:** only `sql` allowed (else 400 `"Invalid keys: ..."`). +- **Response** — 200 even when analysis fails (`ok: false` in body): + `{"ok", "parameters", "analysis_error", "analysis_rows": + [{operation, database, table, required_permission, source, allowed}], + "execute_disabled", "execute_disabled_reason"}`. `allowed` is a per-actor + permission check result (true/false/null). + +### GET /\/-/foreign-key-targets + +`DatabaseForeignKeyTargetsView` (app.py:2659-2662; +views/table_create_alter.py:965-1005). + +- **Parameter:** `table` (optional) — only used for the permission check. +- **Permission:** `create-table` on the database, **or** `alter-table` on + `?table=` when it names an existing table. Neither → 403 + `{"ok": false, "errors": ["Permission denied: need create-table"]}`. +- **Response:** 200 `{"ok": true, "database": "...", "targets": + [{"fk_table", "fk_column", "type"}]}` — every non-hidden table with exactly + one primary-key column; `type` is the pk's SQLite type affinity. + +### GET /\/-/schema(.json|.md) + +`DatabaseSchemaView` (app.py:2683-2686; views/special.py:1296-1329). + +- **Permission:** `view-database` (denied → `Forbidden` → 403 HTML). +- **Unknown database** → 404; for `.json`: + `{"ok": false, "error": "Database not found"}`. (The existence check runs + before the permission check.) +- **Responses:** `.json` → 200 `{"database": "", "schema": ""}` + (concatenated `sqlite_master.sql` joined with `;\n`); `.md` → + `text/markdown`; no extension → HTML. Note the JSON has **no `ok` key** on + success. + +--- + +## Table and row read endpoints + +### GET /\/\.json + +Route `r"/(?P[^\/\.]+)/(?P
[^\/\.]+)(\.(?P\w+))?$"` → +`table_view` (app.py:2711-2714; views/table.py:1670). Serves both tables and +SQL views. GET/HEAD only — POST returns a plain-text 405. If the name is +neither a table nor a view but matches a stored query, the request is +dispatched to `QueryView` (views/table.py:1703-1712). + +**Permission:** `view-table` via `check_visibility`; denial raises +`Forbidden` → **HTML** 403 page even for `.json`. Unknown table → +`TableNotFound` → 404 (JSON error shape for `.json` paths). + +**Default JSON keys** (views/table.py:2308-2332 + renderer): + +| Key | Meaning | +|---|---| +| `ok` | `true` when data was retrieved without error | +| `next` | pagination token string, or `null` on the last page | +| `rows` | list of row objects `{column: value}` (default `_shape=objects`) | +| `truncated` | always present; `false` for table pages | + +`columns` is computed but removed unless `?_extra=columns` was requested. +When there is a next page the response carries a +`Link: ; rel="next"` header (views/table.py:1911-1912). + +**`?_extra=` options** (TABLE scope; registry +views/table_extras.py:1197-1235; unknown names silently ignored): + +| `_extra=` | Returns | +|---|---| +| `count` | total matching-row count, computed with a `limit 10001` subquery so it caps at 10001; `null` with `_nocount` or on count timeout | +| `count_sql` | the SQL used for the count | +| `facet_results` | `{"results": {name: facet}, "timed_out": [...]}`; each facet: `{name, type, hideable, toggle_url, results: [{value, label, count, toggle_url, selected}], truncated}` | +| `facets_timed_out` | facet names that exceeded `facet_time_limit_ms` | +| `suggested_facets` | `[{name, toggle_url, (type)}]`; empty when suggestion is disabled or paginating | +| `human_description_en` | English description of filters + sort | +| `next_url` | absolute URL of the next page or `null` | +| `columns` | column names of the returned rows | +| `all_columns` | all table columns regardless of `_col`/`_nocol` | +| `primary_keys` | pk column names (empty for rowid tables and views) | +| `display_columns` | HTML-oriented column metadata | +| `render_cell` | per-row plugin-rendered HTML strings | +| `debug` | `{url_vars, resolved, nofacet, nosuggest}` — explicitly unstable | +| `request` | `{url, path, full_path, host, args}` | +| `query` | `{sql, params}` of the main query | +| `column_types` | `{column: {type, config}}` assigned column types | +| `set_column_type_ui` | UI helper, `null` unless actor has `set-column-type` | +| `metadata` | table metadata dict including column descriptions | +| `extras` | self-describing list of all available extras | +| `database`, `table`, `database_color` | identity/display values | +| `renderers` | `{format_name: url}` of formats that can render this data | +| `custom_table_templates` | template lookup list | +| `sorted_facet_results` | facets as a display-ordered list | +| `table_definition` | `CREATE TABLE` SQL | +| `view_definition` | `CREATE VIEW` SQL, `null` for tables | +| `is_view` | boolean | +| `private` | `true` if visible to this actor but not anonymously | +| `expandable_columns` | `[[foreign_key, label_column_or_null], ...]` | +| `form_hidden_args` | pairs of `_`-prefixed args for HTML forms | + +Non-public extras (`actions`, `filters`, `display_rows`) are HTML-only and +never appear in JSON. `_extra=_html` expands to the full HTML bundle +(views/table_extras.py:1162-1194). Any `_facet*` argument implicitly adds +`facet_results`; `_shape=object` implicitly adds `primary_keys` +(views/table.py:2252-2256). There is **no** `filtered_table_rows_count` +extra — it was replaced by `count`. + +**Column filters `?__=`** (filters.py:260-427). Any +querystring key not starting with `_` is a filter; bare `?column=value` means +`exact`. Columns whose names start with `_` can be filtered as +`?_col__exact=`. Operators: + +| op | SQL | +|---|---| +| `exact` | `"col" = :p` (default) | +| `not` | `"col" != :p` | +| `contains` / `notcontains` | `like '%v%'` / `not like '%v%'` | +| `endswith` / `startswith` | `like '%v'` / `like 'v%'` | +| `gt` / `gte` / `lt` / `lte` | `>` `>=` `<` `<=` (numeric strings cast to int) | +| `like` / `notlike` | raw `like` / `not like` pattern | +| `glob` | `glob` | +| `in` / `notin` | comma-separated list, or JSON array if the value starts with `[` | +| `arraycontains` / `arraynotcontains` | `[not] in (select value from json_each("col"))` (requires JSON1) | +| `date` | `date("col") = :p` | +| `isnull` / `notnull` | `is null` / `is not null` (no value) | +| `isblank` / `notblank` | `(is null or = '')` / opposite (no value) | + +**Special (underscore) parameters:** + +| Param | Behavior | +|---|---| +| `_where=SQL` | extra raw where clause (repeatable); requires `execute-sql` else 403 `"_where= is not allowed"` | +| `_search=q` | FTS against the table's FTS table | +| `_search_=q` | FTS restricted to one column; 400 if invalid | +| `_searchmode=raw` | pass the query straight to `match` | +| `_fts_table=` / `_fts_pk=` | override the FTS table / pk used for joins | +| `_through={"table","column","value"}` | filter via an incoming foreign key (repeatable, JSON value) | +| `_sort=col` / `_sort_desc=col` | sort; 400 if both given or column not sortable | +| `_next=token` | pagination token | +| `_size=N\|max` | page size; default `default_page_size` (100); `max` = `max_returned_rows` (1000); 400 on invalid | +| `_col=name` (repeatable) | return only pks + these columns; 400 on invalid | +| `_nocol=name` (repeatable) | exclude columns; 400 if invalid or a pk | +| `_labels=on` | expand every FK column into `{"value", "label"}` | +| `_label=col` (repeatable) | expand only the named FK column(s) | +| `_facet=col` | request a facet; 400 `"_facet= is not allowed"` when `allow_facet` off | +| `_facet_array=col` / `_facet_date=col` | typed facets | +| `_facet_size=N\|max` | facet bucket count, default 30, capped at `max_returned_rows` | +| `_nocount=1` | skip count (`count` extra → null) | +| `_nofacet=1` | skip facets and suggestions | +| `_nosuggest=1` | skip facet suggestions only | +| `_shape=` | see renderer section; `array`/`object` also force `_nocount` and `_nofacet` | +| `_nl=on` | NDJSON with `_shape=array` | +| `_json=col` / `_json_infinity=1` | renderer options | +| `_timelimit=ms` | custom SQL time limit | +| `_ttl=seconds` | `Cache-Control: max-age=N` (`0` → `no-cache`); default `default_cache_ttl` (5) | +| `_trace=1` | append `_trace` key (requires `trace_debug` setting) | +| `_extra=` | see above | + +**Pagination** is keyset-based for tables: `page_size + 1` rows are fetched; +`next` is built from the last row of the page — comma-joined tilde-encoded +primary-key values, prefixed by the sort value when sorted (`$null` for null +sort values) (views/table.py:2041-2111, 2421-2482). `next_url` is the +absolute URL with `_next` replaced. + +### GET /\/\.json (SQL views) + +Same code path with `is_view=True`. Differences: + +- No primary keys: `primary_keys` → `[]`; `_shape=object` fails; base query + has no `order by`. +- **Pagination is offset-based**: `_next` is an integer offset applied as + `limit N offset M` (views/table.py:2047-2049, 2438-2439) — unlike the + keyset tokens used for tables. +- `view_definition` returns the `CREATE VIEW` SQL; `table_definition` is null. + +### GET /\/\/\.json + +`RowView` (app.py:2715-2718; views/row.py:137). `` is comma-separated +tilde-encoded primary key values (rowid for rowid tables). + +- **Permission:** `view-table` (denied → `Forbidden` → 403 HTML). Missing row + → 404 `"Record not found: [...]"`. +- **Default JSON keys:** `ok`, `database`, `table`, `rows` (single-element + list), `primary_keys`, `primary_key_values`, `query_ms`, + `truncated: false`; `columns` only with `?_extra=columns`. +- **`?_extra=` (ROW scope):** `columns`, `primary_keys`, `render_cell`, + `debug`, `request`, `query`, `column_types`, `metadata`, `extras`, + `database`, `table`, `database_color`, `private`, `foreign_key_tables` + (incoming FKs with `count` and `link`; single-pk rows only). +- **Foreign-key label expansion does not apply to row JSON** — `_labels` has + no effect here; expansion happens only in the HTML path + (views/row.py:445-475). +- `_shape`, `_json`, `_nl`, `_json_infinity`, `_ttl` apply. A `.jsono` + request redirects to `.json?_shape=objects`. + +### The .blob format + +`//
/.blob?_blob_column=col` (also on query pages) — +fetches raw binary bytes (blob_renderer.py:10-61). `_blob_column` required +(400 if missing/invalid); optional `_blob_hash` must equal the value's +SHA-256 (else 400 `"Link has expired..."`). Returns `application/binary` as a +download attachment. In JSON output, binary cells appear as +`{"$base64": true, "encoded": "..."}`. + +### GET /\/\/-/schema(.json|.md) + +`TableSchemaView` (app.py:2751-2754; views/special.py:1332-1378). + +- **Permission:** `view-table` via `ensure_permission` (denied → 403 HTML). +- **Responses:** `.json` → 200 `{"database", "table", "schema"}` (no `ok` + key); `.md` → `text/markdown`; no extension → HTML. Missing table → 404 + `{"ok": false, "error": "Table not found"}` for `.json`. + +### GET /\/\/-/fragment + +`TableFragmentView` (app.py:2739-2742; views/table.py:1385-1418). +**HTML-only** — returns the `_table.html` partial; no JSON variant. Accepts +table querystring parameters plus `_row=` to render a single row. + +### GET /\/\/-/autocomplete + +`TableAutocompleteView` (app.py:2743-2746; views/table.py:1492-1595). Tables +only — views get 400 `"Autocomplete is only available for tables"`. + +- **Permission:** `view-table` (denied → `Forbidden` → 403). +- **Parameters:** `q` (matched with escaped `LIKE %q%` against pk columns and + the label column) and `_initial` (truthy: with empty `q`, return the 10 + most recent rows). Neither → `{"rows": []}`. +- **Response:** `{"rows": [{"pks": {pk_name: value}, "label": "..."}]}` — max + 10 items; 500 ms query budget with fallbacks, timing out to + `{"rows": []}`. + +--- + +## The write API + +All write endpoints return errors via `_error()` +(`{"ok": false, "errors": [...]}`) and check permissions with +`datasette.allowed()` directly, so their 403s are JSON (unlike the +`Forbidden`-raising read endpoints). Routes: app.py:2719-2762. + +### POST /\/\/-/insert + +`TableInsertView` (views/table.py:907-1194). + +- **Permissions:** `insert-row` on the table (denied → 403 + `["Permission denied"]`); `update-row` additionally required for + `replace: true` (403 `need update-row to use "replace"`); `alter-table` + additionally required for `alter: true` (403 + `Permission denied for alter-table`). Immutable database → 403 + `Database is immutable`. +- **Request** — requires `Content-Type: application/json` (else 400 + `"Invalid content-type, must be application/json"`). Body: + + | Field | Rules | + |---|---| + | `row` | single object; mutually exclusive with `rows`; forces `return: true` | + | `rows` | list of objects; max `max_insert_rows` (default 100), else 400 `"Too many rows, maximum allowed is 100"` | + | `ignore` | skip rows whose pk already exists; mutually exclusive with `replace` | + | `replace` | replace rows with matching pks (needs `update-row`) | + | `alter` | add missing columns (needs `alter-table`) | + | `return` | include inserted rows in the response | + + One of `row`/`rows` required. Unknown keys → 400 `"Invalid parameter: ..."`. + Unless `alter`, row keys must be existing columns → per-row 400 + `"Row 0 has invalid columns: x, y"`. Values are validated against assigned + column types. +- **Response** — **201** `{"ok": true}`; with `return: true` also `rows` + (the rows as stored, re-fetched by rowid). SQLite errors during the write → + 400 with the message. Emits `insert-rows` (and possibly `alter-table`) + events. + +### POST /\/\/-/upsert + +`TableUpsertView` — subclasses insert (views/table.py:1197-1201). + +- **Permissions:** **both** `insert-row` and `update-row` (403 + `need both insert-row and update-row`); `alter: true` needs `alter-table`. +- **Request:** same as insert, except `ignore`/`replace` are rejected (400 + `"Upsert does not support ignore or replace"`) and **every row must contain + the table's primary key(s)** (per-row 400 + `Row 0 is missing primary key column(s): "id"` / `has null primary key`). +- **Response** — **200** (note: insert returns 201) `{"ok": true}`; with + `return: true`, `rows` re-fetched by pk. Emits `upsert-rows`. + +### POST /\/\/-/alter + +`TableAlterView` (views/table_create_alter.py:1130-1353). + +- **Permission:** `alter-table` (403 `need alter-table`); immutable → 403. +- **Request:** `{"operations": [{"op": ..., "args": {...}}, ...]}` — a + non-empty list, validated by pydantic (extra keys forbidden anywhere; + errors → 400 `location: message`): + + | `op` | `args` | + |---|---| + | `add_column` | `name` (required), `type` (`text`/`integer`/`float`/`blob`, default `text`), `not_null`, `default` xor `default_expr`; `not_null: true` requires a default | + | `rename_column` | `name`, `to` | + | `rename_table` | `to` (must not start `sqlite_`) | + | `alter_column` | `name` + at least one of `type`, `not_null`, `default`, `default_expr` | + | `drop_column` | `name` | + | `set_primary_key` | `columns` (non-empty list) | + | `reorder_columns` | `columns` (non-empty list) | + | `add_foreign_key` | `column`, `fk_table`, optional `fk_column` | + | `drop_foreign_key` | `column` | + | `set_foreign_keys` | `foreign_keys`: list of `{column, fk_table, fk_column?}` | + + `default_expr` must be one of the five `current_*` keywords. Operations are + applied in a single write transaction; any failure → 400. +- **Response** — 200: + ```json + {"ok": true, "database": "...", "table": "", + "table_url": "...", "table_api_url": "...", + "altered": true, "schema": "...", "before_schema": "...", + "operations_applied": 2} + ``` + +### POST /\/\/-/drop + +`TableDropView` (views/table.py:1320-1382). + +- **Permission:** `drop-table` (403 `Permission denied`); immutable → 403. +- **Confirmation flow:** without `{"confirm": true}` in the body, nothing is + dropped and a 200 preview is returned: + `{"ok": true, "database", "table", "row_count", + "message": "Pass \"confirm\": true to confirm"}`. With `confirm: true` → + 200 `{"ok": true}`. Emits `drop-table`. + +### POST /\/\/-/set-column-type + +`TableSetColumnTypeView` (views/table.py:1204-1317). Assigns a Datasette +*column type* (metadata stored in the internal `column_types` table) — it +does not change the SQLite schema. + +- **Permission:** `set-column-type` (403 `Permission denied`). +- **Request** (JSON content type required): `{"column": "name", + "column_type": {"type": "url", "config": {...}?} | null}`. Unknown + keys/invalid structure → detailed 400 errors; unknown type → 400 + `"Unknown column type: x"`. Default registered types (via the + `register_column_types` hook): `url`, `email`, `json`, `textarea`. +- **Response** — 200 `{"ok": true, "database", "table", "column", + "column_type": {...} | null}`. + +### GET /\/\/-/foreign-key-suggestions + +`TableForeignKeySuggestionsView` (views/table_create_alter.py:1008-1127). +**GET only** (read-only despite living beside the write endpoints). + +- **Permission:** `alter-table` (403 `need alter-table`); views → 400 + `"Cannot suggest foreign keys for a view"`. +- **Response** — 200: `{"ok": true, "database", "table", + "row_check": {attempted, status, row_limit, sampled_rows, checked_options}, + "columns": [{column, type, affinity, current, + "suggestions": [{fk_table, fk_column, confidence, sampled_values, reasons}], + "options": [...]}]}`. Samples up to 500 rows within 50 ms/200 ms budgets. + +### POST /\/\/\/-/update + +`RowUpdateView` (views/row.py:781-870). + +- **Permissions:** `update-row` (403 `Permission denied`); `alter: true` + additionally requires `alter-table` (403 + `Permission denied for alter-table`). +- **404s:** `Database not found: x` / `Table not found: x` / + `Record not found: [pks]`. +- **Request:** `{"update": {column: value, ...}, "return"?: true, + "alter"?: true}`. Missing/non-dict `update` → 400 + `"JSON must contain an update dictionary"`; unknown keys → 400 + `"Invalid keys: ..."`; write failures (bad column, constraint violation) → + 400 with the message. +- **Response** — 200 `{"ok": true}`; with `return: true`, + `{"ok": true, "row": {...}}` (singular `row`, unlike insert/upsert's + `rows`). Emits `update-row`. + +### POST /\/\/\/-/delete + +`RowDeleteView` (views/row.py:738-778). + +- **Permission:** `delete-row` (403 `Permission denied`). 404s as update. +- **Request:** no body required (any body is ignored — there is no + confirmation step, unlike table drop). +- **Response** — 200 `{"ok": true}`; with `?_redirect_to_table` a `redirect` + key is added. A failure during the write returns **500** with the message + (unlike update's 400). Emits `delete-row`. + +--- + +## Stored (canned) queries API + +Stored queries live in the internal database's `queries` table +(utils/internal_db.py:116-133). Queries defined in `datasette.yaml` are +synced in at startup with `source="config"` and `is_trusted` defaulting to +true; queries created via the API get `source="user"`, `is_trusted=false`, +`owner_id` = actor id. + +**Canonical stored-query JSON object** (`stored_query_to_dict`, +stored_queries.py:55-80): + +```json +{ + "database": "...", "name": "...", "sql": "...", + "title": null, "description": null, "description_html": null, + "hide_sql": false, "fragment": null, + "params": ["p"], "parameters": ["p"], + "is_write": false, "is_private": true, "is_trusted": false, + "source": "user", "owner_id": "...", + "on_success_message": null, "on_success_message_sql": null, + "on_success_redirect": null, + "on_error_message": null, "on_error_redirect": null, + "private": true +} +``` + +`params` and `parameters` are identical lists, both always present. +`private` appears only in list responses. + +**Default permission rules for queries** (default_permissions/defaults.py): +`view-query` is default-allow, but private queries are visible only to their +owner; the owner may `update-query`/`delete-query` their `source='user'` +queries. + +### GET /-/queries(.json) and GET /\/-/queries(.json) + +`GlobalQueryListView` / `QueryListView` (app.py:2606-2609, 2663-2666; +views/stored_queries.py:69-238). The global variant lists queries across all +databases (`database`/`database_color` are null, `show_database` true). + +- **Permissions:** no single gate; results filtered per query by + `view-query` (private queries appear only for their owner). +- **Parameters:** `_size` (default 20 HTML / **50 JSON**, clamped 1–1000; + non-integer → 400), `_next` (cursor), `q` (substring search over + name/title/description/sql), `is_write` / `is_private` (booleans; invalid → + 400 `"is_write must be 0 or 1"`), `source`, `owner_id`. +- **Response** — 200: + `{"ok": true, "database", "database_color", "queries": [...], "next", + "next_url", "has_more", "limit", "show_private_note", + "show_trusted_note", "query_list_path", "show_database", + "facets": [{title, items: [{label, count, href, active}]}], + "filters": {q, is_write, is_private, source, owner_id}}`. + +### GET /\/-/queries/analyze + +`QueryCreateAnalyzeView` (app.py:2667-2670; views/stored_queries.py:290-322). +**GET only** despite being an "analyze" action — POST → 405. + +- **Permissions:** `execute-sql` then `store-query` (each denial → 403 + `errors` JSON). +- **Parameters:** only `sql` (others → 400 `"Invalid keys: ..."`). +- **Response** — 200: `{"ok", "parameters", "analysis_error", + "analysis_rows": [{operation, database, table, required_permission, + source, allowed}], "has_sql", "analysis_is_write", "save_disabled"}`. + +### POST /\/-/queries/store + +`QueryStoreView` (app.py:2671-2674; views/stored_queries.py:325-388). GET on +the same path renders the HTML create form. + +- **Permissions:** `execute-sql` + `store-query` (403 `errors` JSON). +- **Request:** JSON bodies must wrap the fields: + `{"query": {...fields...}}`; form bodies pass fields flat. Fields: + `name` (required; `^[^/\.\n]+$`; conflicts with tables/views or existing + queries → 400), `sql` (required; read SQL must pass `validate_sql_select`; + write SQL must pass per-operation permission checks), `title`, + `description`, `hide_sql`, `fragment`, `parameters`/`params` (must exactly + match the SQL's named parameters; magic parameters rejected), + `is_private` (**default true**), and — only for write SQL — + `on_success_message`, `on_success_redirect`, `on_error_message`, + `on_error_redirect`. `is_write` is derived from SQL analysis; + `is_trusted`, `description_html` and `on_success_message_sql` cannot be + set through this API. +- **Response:** JSON request → **201** `{"ok": true, "query": {...}}`; form + request → 302 redirect. + +### GET /\/\/-/definition + +`QueryDefinitionView` (app.py:2695-2698; views/stored_queries.py:391-408). + +- **Permission:** `view-query` (403 `["Permission denied"]`). +- **Response:** 200 `{"ok": true, "query": {...}}`; 404 + `["Query not found: x"]`. + +### GET/POST /\/\/-/edit + +`QueryEditView` (app.py:2699-2702) — **HTML form endpoint** +(`has_json_alternate = False`), not part of the JSON API. Programmatic +updates use `/-/update`. + +### POST /\/\/-/update + +`QueryUpdateView` (app.py:2703-2706; views/stored_queries.py:411-465). + +- **Permissions:** `update-query` (403 `need update-query`); trusted queries + → 403 `"Trusted queries cannot be updated using the API"`; changing `sql` + additionally requires `execute-sql`. +- **Request:** `{"update": {...partial fields...}, "return"?: true}` — other + top-level keys → 400. Updatable fields: `sql`, `title`, `description`, + `hide_sql`, `fragment`, `parameters`/`params`, `is_private`, `on_*` + fields (write SQL only). New SQL is re-analyzed and `is_write` recomputed. +- **Response:** 200 `{"ok": true}` (plus `query` with `return: true`); 404 + `"Query not found: x"`. + +### POST /\/\/-/delete + +`QueryDeleteView` (app.py:2707-2710; views/stored_queries.py:594-644). GET +renders an HTML confirmation page. + +- **Permission:** `delete-query` (403 `need delete-query`). Unlike update, + **trusted queries are not blocked** from API deletion. +- **Response:** JSON request → 200 `{"ok": true}`; form → 302; 404 + `"Query not found: x"`. No `confirm` field required (unlike table drop). + +### GET/POST /\/\(.json) — executing a stored query + +No dedicated route: the table route resolves the name, and on `TableNotFound` +the request is dispatched to `QueryView` when a stored query matches +(views/table.py:1698-1712). Covers both config-defined and API-stored +queries. + +**GET (read queries)** — `QueryView.get` (views/database.py:695-1130): + +- **Permissions:** `view-query` (denied → `Forbidden` → 403 HTML). Read + queries then require `execute-sql` unless `is_trusted`. Write queries are + **not executed** on GET — JSON returns empty `rows`; HTML shows a POST form. +- **Parameters:** each named `:param` is read from the query string (missing + → `""`); `_timelimit`; renderer options (`_shape`, `_nl`, `_json`, + `_json_infinity`); `_extra` (QUERY scope). +- **Response:** `{"ok": true, "rows": [...], "truncated": false}` + extras. + SQL errors → 400 with `error` in the envelope. + +**POST (write queries)** — `QueryView.post` (views/database.py:574-693): + +- **Permissions:** `view-query`; then, unless `is_trusted`: + `execute-write-sql` on the database **plus** per-operation write + permissions (same table as `/-/execute-write`). Rejection → 403 + `{"ok": false, "message": "...", "redirect": null}` for JSON clients. + Immutable database → 403. +- **Body:** form-encoded or JSON `param=value` pairs (values coerced to + strings). +- **JSON is returned when** `Accept: application/json`, `?_json=1`, or a + `_json` body field is present; otherwise 302 + flash message. +- **Magic parameters** (`:__`, resolved server-side; registered + via `register_magic_parameters`, default_magic_parameters.py): + `_now_epoch`, `_now_date_utc`, `_now_datetime_utc`, `_actor_`, + `_random_chars_`, `_cookie_`, `_header_` (underscores → + hyphens). User-stored queries cannot contain magic parameters — they are a + feature of config/trusted queries. +- **Response — 200 for both success and SQL failure** (only permission + rejection is 403): + `{"ok": true|false, "message": "...", "redirect": "..."|null}` — + `message` honors `on_success_message_sql` / `on_success_message` / + `on_error_message`, falling back to `"Query executed"` or + `"Query executed, N rows affected"`. + +--- + +## Authentication and tokens + +### Bearer tokens (`dstok_`) + +Signed API tokens are sent as `Authorization: Bearer dstok_...`. The +`actor_from_signed_api_token` hook (default_permissions/tokens.py:25-40) +passes the token to `datasette.verify_token()`, which tries every handler +registered via `register_token_handler`; the default is +`SignedTokenHandler` (tokens.py:117-193). + +- **Format:** `dstok_` + itsdangerous-signed payload (namespace `token`) + containing `a` (actor id), `t` (creation Unix time), optional `d` + (duration seconds), optional `_r` (restrictions). +- **Verification** returns no actor when: `allow_signed_tokens` is off, the + signature is invalid, `t` is missing/non-integer, or the token is expired. + The effective duration is `d` capped by `max_signed_tokens_ttl` (default 0 + = no cap; a non-zero setting also imposes a TTL on tokens without `d`). +- **Resulting actor:** `{"id": , "token": "dstok"}` plus `"_r"` and + `"token_expires"` when applicable. Invalid/expired tokens silently produce + an anonymous request (no 401) — the failure then surfaces as a 403 from + whatever permission check the request hits. + +**Restrictions (`_r`)** (default_permissions/restrictions.py): + +- `"a"`: list of actions allowed on any resource +- `"d"`: `{database_name: [actions]}` +- `"r"`: `{database_name: {table_name: [actions]}}` + +Actions are stored as abbreviations when available (see appendix); checks +accept either the full name or the abbreviation. Restrictions are an +allowlist filter layered on top of normal permission resolution — a +restricted token can never do more than its allowlist, and never more than +the underlying actor could do anyway. + +### Token creation + +- **`/-/create-token`** is an HTML form endpoint only (see the instance + section) — there is no JSON API to mint tokens in this codebase. +- Programmatic alternatives: the `datasette create-token` CLI command and + the `datasette.create_token()` Python API. +- `/-/auth-token` is the one-time `--root` login mechanism, unrelated to API + tokens. + +### Cookie authentication + +Browser sessions use the signed `ds_actor` cookie (set by `/-/auth-token`, +plugins, or login flows; cleared by `/-/logout`). API POSTs from browsers are +subject to the cross-origin checks described in +[CSRF](#csrf--cross-origin-protection). + +--- + +## Appendix: registered actions (permissions) + +From `datasette/default_actions.py` (registered via the `register_actions` +hook). Token restrictions store the abbreviation when available. + +| Action | Abbr | Resource level | Notes | +|---|---|---|---| +| `view-instance` | `vi` | global | | +| `permissions-debug` | `pd` | global | gates the debug endpoints | +| `debug-menu` | `dm` | global | UI only | +| `view-database` | `vd` | database | | +| `view-database-download` | `vdd` | database | `also_requires="view-database"` | +| `execute-sql` | `es` | database | `also_requires="view-database"`; denied when the `default_allow_sql` setting is off | +| `execute-write-sql` | `ews` | database | `also_requires="view-database"` | +| `create-table` | `ct` | database | | +| `store-query` | `sq` | database | `also_requires="execute-sql"` | +| `view-table` | `vt` | table | | +| `insert-row` | `ir` | table | | +| `delete-row` | `dr` | table | | +| `update-row` | `ur` | table | | +| `alter-table` | `at` | table | | +| `set-column-type` | `sct` | table | | +| `drop-table` | `dt` | table | | +| `view-query` | `vq` | query | default-allow; private queries restricted to their owner | +| `update-query` | `uq` | query | query owner allowed by default (source=`user` only) | +| `delete-query` | `dq` | query | query owner allowed by default (source=`user` only) | diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md new file mode 100644 index 00000000..e630ae4b --- /dev/null +++ b/stable-api-recommendations.md @@ -0,0 +1,367 @@ +# Datasette 1.0 Stable API — Consistency and Completeness Review + +This review is based on `existing-api.md`, which documents the JSON API as +actually implemented in this codebase (`1.0a35`), derived from source. The +goal here is to identify everything that should be made consistent, fixed, or +explicitly scoped out **before** the 1.0 stability promise takes effect — +because after 1.0, every inconsistency below becomes a compatibility +commitment. + +Findings are grouped by theme. Each carries a priority: + +- **P1 — should block 1.0**: breaking to fix later, or a correctness/security + concern. +- **P2 — strongly recommended**: fixable later only via awkward additive + changes. +- **P3 — nice to have / documentation decision**: can be resolved by + documenting the behavior as intentional. + +--- + +## 1. Error responses: four shapes is three too many (P1) + +The API currently produces four distinct JSON error shapes depending on which +internal layer generates the error: + +| Shape | Producer | Example endpoints | +|---|---|---| +| `{"ok": false, "error", "status", "title"}` | exception handler (handle_exception.py:50-53) | 404s and `DatasetteError`s on any `.json` path | +| `{"ok": false, "errors": [...]}` | `_error()` helper (views/base.py:183-184) | all write endpoints, stored-query endpoints, execute-write | +| `{"ok": false, "error", "rows": [], "truncated": false}` | JSON renderer (renderer.py:52-56) | SQL errors on table/query reads | +| `{"error": "..."}` (no `ok`) | permission debug views (views/special.py) | `/-/allowed`, `/-/rules`, `/-/check`, POST `/-/permissions` | + +Additionally, write canned queries report failure via a **fifth** vocabulary: +`{"ok": false, "message": ..., "redirect": ...}` with HTTP **200** +(views/database.py:678-690). + +A 1.0 client cannot write a single error handler today. **Recommendation:** +pick one canonical error object — the singular/plural tension is easiest to +resolve as: + +```json +{"ok": false, "error": "human-readable summary", "errors": ["detail", "..."], "status": 400} +``` + +where `errors` is optional and `error` is always present — and route every +error path through it (including the `forbidden` and `handle_exception` +defaults). At minimum, eliminate the bare `{"error": ...}` shape and the +`status`/`title` keys nobody else emits (`title` is a template-rendering +concern that leaked into the API). + +### 1a. `Forbidden` returns an HTML 403 to JSON clients (P1) + +Read endpoints that deny access via `ensure_permission`/`check_visibility` +raise `Forbidden`, and the default `forbidden()` hook renders an **HTML error +page even for `.json` requests** (forbidden.py:4-19, app.py:2895-2904). So: + +- `GET /db/table.json` without `view-table` → 403 **HTML** +- `POST /db/table/-/insert` without `insert-row` → 403 **JSON** + +A JSON client gets unparseable output precisely when it most needs a +machine-readable answer. **Recommendation:** the default forbidden handler +must return the canonical JSON error when the path ends in `.json` or the +request prefers JSON, mirroring `handle_exception`. + +### 1b. Errors that return HTTP 200 (P1) + +- `_shape=object` on a query or pk-less table → `{"ok": false, "error": + "_shape=object is only available on tables"}` with **200** + (renderer.py:73-90), while an unknown `_shape` value returns **400** + (renderer.py:101-108). Same class of error, different status. +- Write canned-query SQL failure → **200** `{"ok": false, "message": ...}` + (views/database.py:683-690), while the equivalent failure on + `/-/execute-write` returns **400**. + +**Recommendation:** all `ok: false` responses should carry a 4xx/5xx status. +(`/-/execute-write/analyze` returning `ok: false` with 200 for "analysis +completed, SQL is invalid" is defensible but should then not reuse the `ok` +key — see §2.) + +### 1c. Wrong-status outliers (P2) + +- Row **delete** write failures return **500** (views/row.py:757) while row + **update** write failures return **400** (views/row.py:832-835). Same + failure class, different status; pick 400 (or 409 for constraint + violations) for both. +- Invalid or expired bearer tokens silently degrade the request to anonymous, + so clients see a 403 permission error (or worse, anonymous-permitted data) + rather than a 401 (tokens.py:147-193). For 1.0, a malformed/expired + `Authorization: Bearer dstok_...` header should produce **401** with a + distinguishable error, so clients can tell "renew your token" apart from + "you lack permission". + +--- + +## 2. Success envelope: `ok` is not universal, arrays are not extensible (P1/P2) + +Endpoints disagree about the success envelope: + +- **Have `ok: true`:** table/row/query reads, database view, all write + endpoints, stored-query endpoints, `/-/allowed`-style debug data. +- **No `ok` key:** `/-/versions`, `/-/settings`, `/-/config`, `/-/threads`, + `/-/actor`, `/-/jump`, `/-/schema` variants (`{"database", "schema"}`, + `{"schemas": [...]}`), table `/-/schema.json`, `/-/autocomplete` + (`{"rows": []}`), homepage `/.json`. +- **Top-level JSON arrays:** `/-/plugins`, `/-/databases`, `/-/actions` + (app.py:2247-2304). A top-level array can never grow a sibling key + (pagination, warnings, `ok`) without a breaking change. + +**Recommendations:** + +1. (P1) Wrap the three array endpoints in objects before 1.0: + `{"ok": true, "plugins": [...]}` etc. This is the single cheapest + future-proofing fix in this list. +2. (P2) Add `ok: true` to every JSON-object success response, or explicitly + document that `ok` only exists on data endpoints. Half-consistency is the + worst outcome. +3. (P2) `/db/-/schema.json` (`{"database", "schema"}`) and + `/db/table/-/schema.json` should match the envelope style of their sibling + endpoints (they are also the only data endpoints whose 404 uses the + exception shape but whose success has no `ok`). + +### 2a. Collection representations disagree (P2) + +- Homepage `/.json` returns `databases` as an **object keyed by name** + (index.py:147-161); `/-/databases.json` returns an **array**; the database + page returns `tables` as an array. Choose arrays-of-objects everywhere + (objects-keyed-by-name break when names need ordering or pagination). +- Insert/upsert with `return: true` respond with `rows` (plural, list); row + update with `return: true` responds with `row` (singular, object) + (views/row.py:837-844). Pick one (`rows` everywhere, even for one row, + matches the read API). + +### 2b. `_extra`/`_shape` support is uneven (P2) + +The extras system (`?_extra=`, scope-registered) is the 1.0 mechanism for +response shaping — but it only exists on table, row and query endpoints. The +database view builds JSON by hand and supports **neither `_extra` nor +`_shape`** (views/database.py:189-212); the homepage likewise. Either extend +extras to database/instance scope before 1.0 or document clearly that shaping +is a table/row/query feature. Also decide the contract for **unknown +`_extra` names, which are currently silently ignored** (extras.py:116-122) — +silent ignoring means typos return the default payload with no signal; +recommend a 400 or a `warnings` key. + +### 2c. Count truncation is invisible in JSON (P2) + +The `count` extra is computed with a `limit 10001` subquery, so `count: +10001` actually means "at least 10001" — the `count_truncated` flag exists +but only in the HTML template context, never in JSON (views/table.py: +2334-2337). Expose it (e.g. make `count` be `null` + add `count_estimate`, +or add `count_truncated` to the JSON) before clients start trusting the +number. + +--- + +## 3. Pagination: three mechanisms, two contracts (P2) + +| Endpoint | Mechanism | Token | Extras | +|---|---|---|---| +| Table `.json` | keyset | tilde-encoded pk/sort values in `_next` | `next` always in body, `next_url` via `_extra`, `Link: rel=next` header | +| SQL view `.json` | **offset** | integer in the same `_next` parameter | same envelope | +| `/-/queries` lists | keyset | cursor in `_next` | `next`, `next_url`, **`has_more`** in body | +| `/-/allowed`, `/-/rules` | **page numbers** | `page`/`page_size` | `total`, `next_url`, `previous_url` | + +Concerns: + +1. The same `_next` parameter means "start after key" on tables but "row + offset" on views. Offset pagination over views is also O(n) and skews + under concurrent writes. If unifiable, unify; if not, document loudly. +2. `has_more` exists on query lists but not table pages; `total` exists on + debug endpoints but not elsewhere. Standardize the pagination block + (suggest: `next`, `next_url` — nullable — everywhere; treat `has_more` as + `next != null`). +3. Page-size parameters: `_size` (default 100, `max` keyword allowed) on + tables; `_size` (default 50 JSON, clamped 1–1000, no `max` keyword) on + query lists; `page_size` (default 50, silently capped at 200) on debug + endpoints. Align names, defaults and the cap behavior (silent capping vs + 400) as far as practical. + +--- + +## 4. HTTP semantics (P2) + +- **201 vs 200:** insert → 201, upsert → 200 (views/table.py:1194), create + table → 201, store query → 201. Insert-201/upsert-200 is defensible + (upsert may not create) but it is undocumented subtlety; state it, or + return 200 for both with an explicit `created` count. +- **Destructive-action confirmation is asymmetric:** table drop requires + `{"confirm": true}` and has a preview response (views/table.py:1346-1365); + row delete executes immediately and ignores the body; query delete + executes immediately. Decide the 1.0 rule (suggestion: confirmation only + for schema-destroying operations, i.e. keep as is — but document it as a + deliberate contract). +- **Content-type enforcement is inconsistent:** `/-/insert`, `/-/upsert`, + `/-/alter`, `/-/set-column-type` demand `Content-Type: application/json` + (400 otherwise); `/-/create` parses the body as JSON regardless of + content type; execute-write and the query CRUD endpoints accept both JSON + and form encodings. Pick one rule for JSON-only endpoints. +- **JSON-vs-HTML negotiation on POST differs per endpoint:** execute-write + and canned queries key off `Accept: application/json` / a `_json` body + field; the write API keys off nothing (always JSON); query store keys off + request content type. A single documented rule ("responses are JSON if the + request body was JSON or `Accept: application/json`") would cover all of + them. +- **Endpoints named like actions but served over GET:** + `/-/queries/analyze`, `/-/execute-write/analyze`, + `/-/foreign-key-suggestions`, `/-/query/parameters` are all GET (correct, + they are reads) — fine, but `analyze` under a POST-shaped path invites + wrong calls; make sure 405 responses for POST on these return the JSON 405 + shape (they do only when the path ends `.json` or content type is JSON — + a JSON POST to `/-/queries/analyze` gets JSON, a form POST gets text). + +--- + +## 5. Naming and parameter conventions (P2/P3) + +- **`params` and `parameters` are duplicate keys** in every stored-query + object (stored_queries.py:55-80). Delete one before 1.0 (suggest keeping + `parameters`; the write side already accepts both on input). +- **Three names for the same concept across error/message payloads:** + `error`, `errors`, `message`. See §1. +- **Boolean query parameters have at least three grammars:** `_nl=on`, + `_labels=on/off`, `?all=1`, `is_write=1|0|true|false|t|f|yes|no|on|off`, + `_nocount=1`. Adopt one accepted set (the query-list parser at + query_helpers.py:81-94 is a good candidate) and apply it everywhere. +- **`.jsono`** survives on the homepage route (identical output to `.json`) + and as a row-view redirect. Remove it at 1.0; it is pure legacy. +- **`_json` is overloaded:** on GET it is a renderer option naming a column + to parse as JSON (repeatable); on canned-query POST a `_json` body field + forces a JSON response. Two unrelated meanings for one name. +- The reserved `/-/` namespace is applied consistently across routes — this + is in good shape. The one gap: table names matching `^-$`-adjacent shapes + are protected by tilde-encoding; keep a test asserting `/-/` can never be + shadowed by user data. + +--- + +## 6. Permissions and security consistency (P1/P2) + +- **(P1) `/-/databases.json` ignores per-database permissions** — it lists + every attached database (name, path on disk, size) to any actor holding + `view-instance` (app.py:2157-2169), while the homepage and every other + endpoint filter by `view-database`. On a public instance with private + databases this leaks filesystem paths and database names. Filter it, or + gate it behind `permissions-debug`. +- **(P2) `/db/-/schema` checks existence before permission** + (views/special.py:1308-1317): an actor without `view-database` can + distinguish "database exists" (403) from "does not exist" (404). + Standardize on permission-check-first (as the table view does) so + unauthorized actors get a uniform response. +- **(P2) `/-/threads` exposes runtime internals** (thread idents, asyncio + task reprs including file paths) behind only `view-instance`. Consider + `permissions-debug`, alongside `/-/actions` which already requires it. +- **(P3) `/-/config` redaction is substring-based** on six key names + (app.py:2502-2505); plugins storing secrets under other names leak. Worth + a note in plugin authoring docs plus a `redact_keys` plugin hook. +- **(P3) Database-level checks on `/-/create`** (insert-row/update-row + checked against `DatabaseResource`, not the about-to-exist table — + table_create_alter.py:819-856) vs table-level checks on `/-/insert`. + Correct by necessity, but document that a token restricted to + table-level `ir` cannot use `/-/create` with rows. + +--- + +## 7. Completeness gaps for a 1.0 JSON API (P2/P3) + +1. **(P2) No JSON API to create tokens.** `/-/create-token` is an HTML form + only (`has_json_alternate = False`, form-encoded POST). Any automation + that wants to mint scoped tokens must shell out to `datasette + create-token`. An intentional JSON mode (actor-authenticated, same + restriction vocabulary) rounds out the write API story — or explicitly + document token minting as CLI/Python-only. +2. **(P2) Row JSON cannot expand foreign-key labels.** `_labels` works on + table JSON but is silently ignored on row JSON (views/row.py:445-475 + expands only for HTML). Either support it or return 400 for unsupported + parameters; silent ignoring is the worst option (see also §2b on unknown + `_extra` values). +3. **(P2) No machine-readable "which write features does this instance/table + support" endpoint.** Clients must probe (`/-/insert` on an immutable + database → 403). The API explorer computes exactly this data for HTML + (views/special.py:863-990); exposing it as JSON would let clients degrade + gracefully. (`/-/allowed.json` covers the permission half already.) +4. **(P3) Table list pagination.** `/db.json` inlines all tables (with + counts) and the homepage truncates to 5 per database; a 10,000-table + database has no paginated table listing. Acceptable for 1.0 if + documented; the internal catalog tables would support a real endpoint + later. +5. **(P3) `Link: rel=next` header** exists on table JSON only. Harmless, but + either add it to the other paginated endpoints or drop it from the + contract (`Access-Control-Expose-Headers: Link` suggests it is meant to + be part of the API). + +--- + +## 8. Behavior that looks like a bug and should be resolved before freezing + +1. **Trusted queries: update is blocked, delete is not.** + `QueryUpdateView` rejects `is_trusted` queries with 403 + (stored_queries.py:426-427) but `QueryDeleteView.post` never checks + `is_trusted` — an actor with `delete-query` can delete a config-defined + trusted query via the API (it will resync on restart, making the + behavior confusing rather than catastrophic). Align delete with update. +2. **GET `/db/-/query` with no `?sql=` returns 200 `{"ok": true, "rows": + []}`** while `.csv` on the same request returns 400 `"?sql= is + required"`. The JSON behavior masks caller bugs; return 400 on both. +3. **`_shape=object` HTTP 200 error** (§1b) — almost certainly unintended. +4. **Row delete 500** (§1c) — inconsistent with every sibling endpoint. +5. **The "SQL Interrupted" error embeds an HTML fragment in the JSON `error` + value** (views/database.py:805-820). Error strings in the JSON API should + be plain text. + +--- + +## 9. Define stability tiers explicitly (P1 — documentation, not code) + +Not everything under `/-/` can or should carry a 1.0 guarantee. Recommend +shipping 1.0 with an explicit three-tier contract, per endpoint: + +- **Stable (semver-protected):** table/row/query reads (`.json`, `_shape`, + `_extra` public names, filters, pagination tokens as opaque strings), the + write API (`/-/insert`, `/-/upsert`, `/-/alter`, `/-/drop`, + `/-/set-column-type`, row `/-/update`, `/-/delete`, `/-/create`, + `/-/execute-write`), stored-query CRUD + execution, `/-/versions`, + `/-/plugins`, `/-/settings`, `/-/actor`, `/-/databases`, schema endpoints, + token format & restriction semantics (`_r` abbreviations are wire format + now — they are stored inside issued tokens and cannot change silently). +- **Unstable/debug (documented as exempt):** `/-/threads`, `/-/actions`, + `/-/permissions`, `/-/allowed`, `/-/rules`, `/-/check`, `/-/messages`, + `/-/allow-debug`, `/-/patterns`, `/-/debug/autocomplete`, the `debug` and + `request` extras (the `debug` extra already self-describes as unstable), + `/-/api` and `/-/jump` (UI support endpoints), `/-/autocomplete` and + `/-/fragment` (UI support), `/-/foreign-key-suggestions` and + `/-/foreign-key-targets` (heuristic outputs). +- **Internal:** anything HTML-only (`/-/edit`, `/-/create-token`, + `/-/logout`, `/-/auth-token`). + +Two details make tiering urgent rather than optional: + +- **Extras are enumerable by clients** (`?_extra=extras` self-describes the + registry), so every public extra name is de-facto API. Mark each extra + stable or unstable in its class definition and surface that in the + `extras` output. +- **Pagination tokens leak implementation** (tilde-encoded pk values for + tables, plain integers for views). Declare them opaque now so the view + token can become keyset later without a "breaking" change. + +--- + +## 10. Summary of P1 items (the pre-1.0 checklist) + +1. One canonical JSON error shape; retire the other three (§1). +2. `Forbidden` → JSON 403 for JSON requests (§1a). +3. No `ok: false` with HTTP 200 (§1b: `_shape=object`, write canned-query + SQL errors). +4. Wrap `/-/plugins`, `/-/databases`, `/-/actions` top-level arrays in + objects (§2). +5. Filter `/-/databases.json` by `view-database` or gate it behind + `permissions-debug` (§6). +6. 401 (not silent-anonymous) for invalid/expired bearer tokens (§1c). +7. Publish explicit stability tiers, including extras and pagination-token + opacity (§9). +8. Resolve the looks-like-a-bug list (§8), especially trusted-query delete + and row-delete 500. + +Everything in P2 is worth doing now because each item is breaking-to-fix +later; each P3 can be resolved by a sentence of documentation declaring the +current behavior intentional. From 0679e04bd3c293dd3d0efd45aefd6ae57133f68d Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 03:12:15 +0000 Subject: [PATCH 02/46] Unify JSON error responses into one canonical shape All JSON error responses now use a single format built by the new datasette.utils.error_body() helper: {"ok": false, "error": "...", "errors": ["..."], "status": 400} - error is all messages joined with '; ', errors is the full list, status always matches the HTTP status code - The exception handler no longer emits the legacy title key in JSON (it is still available to the HTML error template) - The permission debug endpoints (/-/allowed, /-/rules, /-/check, POST /-/permissions) no longer return bare {"error": ...} objects - JSON renderer SQL errors keep their rows/truncated context keys but now include the canonical keys as well - _shape=object misuse (queries or tables without primary keys) now returns HTTP 400 instead of 200 with an error body - Method-not-allowed 405 responses use the canonical shape Adds tests/test_error_shape.py covering all four previous shape producers, updates affected tests, and documents the format in a new 'Error responses' section of docs/json_api.rst. Implements section 1 of stable-api-recommendations.md. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/handle_exception.py | 41 ++++---- datasette/renderer.py | 14 +-- datasette/utils/__init__.py | 21 ++++ datasette/views/base.py | 11 +- datasette/views/special.py | 44 ++++---- docs/json_api.rst | 59 +++++++++-- existing-api.md | 94 ++++++++--------- stable-api-recommendations.md | 14 ++- tests/test_api.py | 25 ++--- tests/test_api_write.py | 80 +++++++-------- tests/test_base_view.py | 2 + tests/test_column_types.py | 4 +- tests/test_error_shape.py | 185 ++++++++++++++++++++++++++++++++++ tests/test_table_api.py | 8 +- tests/test_table_html.py | 2 +- 15 files changed, 432 insertions(+), 172 deletions(-) create mode 100644 tests/test_error_shape.py diff --git a/datasette/handle_exception.py b/datasette/handle_exception.py index 2b311644..fc290dbc 100644 --- a/datasette/handle_exception.py +++ b/datasette/handle_exception.py @@ -1,5 +1,5 @@ from datasette import hookimpl, Response -from .utils import add_cors_headers +from .utils import add_cors_headers, error_body from .utils.asgi import ( Base400, ) @@ -45,6 +45,13 @@ def handle_exception(datasette, request, exception): message = str(exception) traceback.print_exc() templates = [f"{status}.html", "error.html"] + headers = {} + if datasette.cors: + add_cors_headers(headers) + if request.path.split("?")[0].endswith(".json"): + body = dict(info) + body.update(error_body(message, status)) + return Response.json(body, status=status, headers=headers) info.update( { "ok": False, @@ -53,24 +60,18 @@ def handle_exception(datasette, request, exception): "title": title, } ) - headers = {} - if datasette.cors: - add_cors_headers(headers) - if request.path.split("?")[0].endswith(".json"): - return Response.json(info, status=status, headers=headers) - else: - environment = datasette.get_jinja_environment(request) - template = environment.select_template(templates) - return Response.html( - await template.render_async( - dict( - info, - urls=datasette.urls, - menu_links=lambda: [], - ) - ), - status=status, - headers=headers, - ) + environment = datasette.get_jinja_environment(request) + template = environment.select_template(templates) + return Response.html( + await template.render_async( + dict( + info, + urls=datasette.urls, + menu_links=lambda: [], + ) + ), + status=status, + headers=headers, + ) return inner diff --git a/datasette/renderer.py b/datasette/renderer.py index f40e3dbb..7c94f6ee 100644 --- a/datasette/renderer.py +++ b/datasette/renderer.py @@ -1,6 +1,7 @@ import json from datasette.extras import extra_names_from_request from datasette.utils import ( + error_body, value_as_boolean, remove_infinites, CustomJSONEncoder, @@ -52,8 +53,7 @@ def json_renderer(request, args, data, error, truncated=None): if error: shape = "objects" status_code = 400 - data["error"] = error - data["ok"] = False + data.update(error_body(error, status_code)) if truncated is not None: data["truncated"] = truncated @@ -87,7 +87,8 @@ def json_renderer(request, args, data, error, truncated=None): object_rows[pk_string] = row data = object_rows if shape_error: - data = {"ok": False, "error": shape_error} + status_code = 400 + data = error_body(shape_error, status_code) elif shape == "array": data = data["rows"] @@ -100,12 +101,7 @@ def json_renderer(request, args, data, error, truncated=None): data["rows"] = [list(row.values()) for row in data["rows"]] else: status_code = 400 - data = { - "ok": False, - "error": f"Invalid _shape: {shape}", - "status": 400, - "title": None, - } + data = error_body(f"Invalid _shape: {shape}", status_code) # Don't include "columns" in output # https://github.com/simonw/datasette/issues/2136 diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index 1caff3a8..17c8702f 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -1294,6 +1294,27 @@ async def derive_named_parameters(db: "Database", sql: str) -> List[str]: return named_parameters(sql) +def error_body(messages, status): + """ + The canonical JSON error body used by every Datasette JSON error response: + + {"ok": False, "error": "...", "errors": ["...", ...], "status": 400} + + "error" is all of the messages joined with "; ", "errors" is the full + list, "status" matches the HTTP status code. Callers may add extra + context keys to the returned dictionary but must not remove these four. + """ + if isinstance(messages, str): + messages = [messages] + messages = [str(message) for message in messages] + return { + "ok": False, + "error": "; ".join(messages), + "errors": messages, + "status": status, + } + + def add_cors_headers(headers): headers["Access-Control-Allow-Origin"] = "*" headers["Access-Control-Allow-Headers"] = "Authorization, Content-Type" diff --git a/datasette/views/base.py b/datasette/views/base.py index 30026f4b..a12ae050 100644 --- a/datasette/views/base.py +++ b/datasette/views/base.py @@ -5,6 +5,7 @@ import sys from datasette.utils.asgi import Request from datasette.utils import ( add_cors_headers, + error_body, EscapeHtmlWriter, InvalidSql, LimitedWriter, @@ -49,9 +50,7 @@ class View: request.path.endswith(".json") or request.headers.get("content-type") == "application/json" ): - response = Response.json( - {"ok": False, "error": "Method not allowed"}, status=405 - ) + response = Response.json(error_body("Method not allowed", 405), status=405) else: response = Response.text("Method not allowed", status=405) return response @@ -90,9 +89,7 @@ class BaseView: request.path.endswith(".json") or request.headers.get("content-type") == "application/json" ): - response = Response.json( - {"ok": False, "error": "Method not allowed"}, status=405 - ) + response = Response.json(error_body("Method not allowed", 405), status=405) else: response = Response.text("Method not allowed", status=405) return response @@ -181,7 +178,7 @@ class BaseView: def _error(messages, status=400): - return Response.json({"ok": False, "errors": messages}, status=status) + return Response.json(error_body(messages, status), status=status) async def stream_csv(datasette, fetch_data, request, database): diff --git a/datasette/views/special.py b/datasette/views/special.py index 3245bc13..602ec5ea 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -9,6 +9,7 @@ from datasette.utils import ( actor_matches_allow, add_cors_headers, await_me_maybe, + error_body, tilde_encode, tilde_decode, ) @@ -348,26 +349,29 @@ class AllowedResourcesView(BaseView): async def _allowed_payload(self, request, has_debug_permission): action = request.args.get("action") if not action: - return {"error": "action parameter is required"}, 400 + return error_body("action parameter is required", 400), 400 if action not in self.ds.actions: - return {"error": f"Unknown action: {action}"}, 404 + return error_body(f"Unknown action: {action}", 404), 404 actor = request.actor if isinstance(request.actor, dict) else None actor_id = actor.get("id") if actor else None parent_filter = request.args.get("parent") child_filter = request.args.get("child") if child_filter and not parent_filter: - return {"error": "parent must be provided when child is specified"}, 400 + return ( + error_body("parent must be provided when child is specified", 400), + 400, + ) try: page = int(request.args.get("page", "1")) page_size = int(request.args.get("page_size", "50")) except ValueError: - return {"error": "page and page_size must be integers"}, 400 + return error_body("page and page_size must be integers", 400), 400 if page < 1: - return {"error": "page must be >= 1"}, 400 + return error_body("page must be >= 1", 400), 400 if page_size < 1: - return {"error": "page_size must be >= 1"}, 400 + return error_body("page_size must be >= 1", 400), 400 max_page_size = 200 if page_size > max_page_size: page_size = max_page_size @@ -485,9 +489,13 @@ class PermissionRulesView(BaseView): # JSON API - action parameter is required action = request.args.get("action") if not action: - return Response.json({"error": "action parameter is required"}, status=400) + return Response.json( + error_body("action parameter is required", 400), status=400 + ) if action not in self.ds.actions: - return Response.json({"error": f"Unknown action: {action}"}, status=404) + return Response.json( + error_body(f"Unknown action: {action}", 404), status=404 + ) actor = request.actor if isinstance(request.actor, dict) else None @@ -496,12 +504,12 @@ class PermissionRulesView(BaseView): page_size = int(request.args.get("page_size", "50")) except ValueError: return Response.json( - {"error": "page and page_size must be integers"}, status=400 + error_body("page and page_size must be integers", 400), status=400 ) if page < 1: - return Response.json({"error": "page must be >= 1"}, status=400) + return Response.json(error_body("page must be >= 1", 400), status=400) if page_size < 1: - return Response.json({"error": "page_size must be >= 1"}, status=400) + return Response.json(error_body("page_size must be >= 1", 400), status=400) max_page_size = 200 if page_size > max_page_size: page_size = max_page_size @@ -587,15 +595,15 @@ class PermissionRulesView(BaseView): async def _check_permission_for_actor(ds, action, parent, child, actor): """Shared logic for checking permissions. Returns a dict with check results.""" if action not in ds.actions: - return {"error": f"Unknown action: {action}"}, 404 + return error_body(f"Unknown action: {action}", 404), 404 if child and not parent: - return {"error": "parent is required when child is provided"}, 400 + return error_body("parent is required when child is provided", 400), 400 # Use the action's properties to create the appropriate resource object action_obj = ds.actions.get(action) if not action_obj: - return {"error": f"Unknown action: {action}"}, 400 + return error_body(f"Unknown action: {action}", 400), 400 # Global actions (no resource_class) don't have a resource if action_obj.resource_class is None: @@ -610,7 +618,7 @@ async def _check_permission_for_actor(ds, action, parent, child, actor): resource_obj = action_obj.resource_class(parent) else: # This shouldn't happen given validation in Action.__post_init__ - return {"error": f"Invalid action configuration: {action}"}, 500 + return error_body(f"Invalid action configuration: {action}", 500), 500 allowed = await ds.allowed(action=action, resource=resource_obj, actor=actor) @@ -651,7 +659,9 @@ class PermissionCheckView(BaseView): # JSON API - action parameter is required action = request.args.get("action") if not action: - return Response.json({"error": "action parameter is required"}, status=400) + return Response.json( + error_body("action parameter is required", 400), status=400 + ) parent = request.args.get("parent") child = request.args.get("child") @@ -1229,7 +1239,7 @@ class SchemaBaseView(BaseView): if self.ds.cors: add_cors_headers(headers) return Response.json( - {"ok": False, "error": error_message}, status=status, headers=headers + error_body(error_message, status), status=status, headers=headers ) else: return Response.text(error_message, status=status) diff --git a/docs/json_api.rst b/docs/json_api.rst index eca22fdc..3df66a8e 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -50,6 +50,37 @@ The ``"truncated"`` key lets you know if the query was truncated. This can happe For table pages, an additional key ``"next"`` may be present. This indicates that the next page in the pagination set can be retrieved using ``?_next=VALUE``. +.. _json_api_errors: + +Error responses +--------------- + +Every JSON error response from Datasette uses the same format: + +.. code-block:: json + + { + "ok": false, + "error": "Table not found", + "errors": [ + "Table not found" + ], + "status": 404 + } + +- ``"ok"`` is always ``false`` for an error. +- ``"errors"`` is a list of one or more error message strings. Endpoints that + validate multiple things at once - such as the :ref:`insert API ` - + may return several messages here. +- ``"error"`` is all of those messages joined with ``"; "``, for + convenience when displaying a single string. +- ``"status"`` matches the HTTP status code of the response. + +Some endpoints add extra context keys. For example, a SQL error from a +:ref:`custom query ` also includes the empty +``"rows"`` and ``"truncated"`` keys of the response it was unable to +produce. + .. _json_api_custom_sql: Executing custom SQL @@ -1625,15 +1656,17 @@ the execute-write returning row limit, which defaults to 10: ] } -Errors use the standard Datasette error format: +Errors use the :ref:`standard Datasette error format `: .. code-block:: json { "ok": false, + "error": "Permission denied: need execute-write-sql", "errors": [ "Permission denied: need execute-write-sql" - ] + ], + "status": 403 } .. _TableInsertView: @@ -1727,9 +1760,11 @@ If any of your rows have a primary key that is already in use, you will get an e { "ok": false, + "error": "UNIQUE constraint failed: new_table.id", "errors": [ "UNIQUE constraint failed: new_table.id" - ] + ], + "status": 400 } Pass ``"ignore": true`` to ignore these errors and insert the other rows: @@ -1859,9 +1894,11 @@ When using upsert you must provide the primary key column (or columns if the tab { "ok": false, + "error": "Row 0 is missing primary key column(s): \"id\"", "errors": [ "Row 0 is missing primary key column(s): \"id\"" - ] + ], + "status": 400 } If your table does not have an explicit primary key you should pass the SQLite ``rowid`` key instead. @@ -1921,7 +1958,7 @@ The returned JSON will look like this: } } -Any errors will return ``{"errors": ["... descriptive message ..."], "ok": false}``, and a ``400`` status code for a bad input or a ``403`` status code for an authentication or permission error. +Any errors will use the :ref:`standard error format `, with a ``400`` status code for a bad input or a ``403`` status code for an authentication or permission error. Pass ``"alter: true`` to automatically add any missing columns to the table. This requires the :ref:`actions_alter_table` permission. @@ -1942,7 +1979,7 @@ To delete a row, make a ``POST`` to ``//
//-/delete``. If successful, this will return a ``200`` status code and a ``{"ok": true}`` response body. -Any errors will return ``{"errors": ["... descriptive message ..."], "ok": false}``, and a ``400`` status code for a bad input or a ``403`` status code for an authentication or permission error. +Any errors will use the :ref:`standard error format `, with a ``400`` status code for a bad input or a ``403`` status code for an authentication or permission error. .. _TableCreateView: @@ -2122,9 +2159,11 @@ If you pass a row to the create endpoint with a primary key that already exists { "ok": false, + "error": "UNIQUE constraint failed: creatures.id", "errors": [ "UNIQUE constraint failed: creatures.id" - ] + ], + "status": 400 } You can avoid this error by passing the same ``"ignore": true`` or ``"replace": true`` options to the create endpoint as you can to the :ref:`insert endpoint `. @@ -2360,7 +2399,7 @@ A successful response returns the new schema and the previous schema. If the req "operations_applied": 11 } -Any errors will return ``{"errors": ["... descriptive message ..."], "ok": false}``, and a ``400`` status code for a bad input or a ``403`` status code for an authentication or permission error. +Any errors will use the :ref:`standard error format `, with a ``400`` status code for a bad input or a ``403`` status code for an authentication or permission error. .. _TableSetColumnTypeView: @@ -2424,7 +2463,7 @@ To clear an existing column type assignment, set ``column_type`` to ``null``: This API stores the assignment in Datasette's internal database, so it can be used with immutable databases as well as mutable ones. -Any errors will return ``{"errors": ["... descriptive message ..."], "ok": false}``, and a ``400`` status code for a bad input or a ``403`` status code for an authentication or permission error. +Any errors will use the :ref:`standard error format `, with a ``400`` status code for a bad input or a ``403`` status code for an authentication or permission error. .. _TableDropView: @@ -2461,4 +2500,4 @@ If you pass the following POST body: Then the table will be dropped and a status ``200`` response of ``{"ok": true}`` will be returned. -Any errors will return ``{"errors": ["... descriptive message ..."], "ok": false}``, and a ``400`` status code for a bad input or a ``403`` status code for an authentication or permission error. +Any errors will use the :ref:`standard error format `, with a ``400`` status code for a bad input or a ``403`` status code for an authentication or permission error. diff --git a/existing-api.md b/existing-api.md index b3eda9d4..1933d287 100644 --- a/existing-api.md +++ b/existing-api.md @@ -44,46 +44,51 @@ directory: every claim below is based on the route table in `datasette/app.py` - Success content type: `application/json; charset=utf-8` (`_shape=array&_nl=on` responses use `text/plain`). -### Error shapes (there are several) +### Error shape (canonical) -The codebase produces **four distinct JSON error shapes**, depending on which -layer generates the error: +Every JSON error response uses one canonical shape, built by `error_body()` +(utils/__init__.py): -1. **Exception handler** (handle_exception.py:21-59) — used when a view raises - `NotFound`, `Forbidden` (JSON paths only — see below), `DatasetteError`, - `BadRequest` etc. and the request path ends in `.json`: +```json +{ + "ok": false, + "error": "all messages joined with '; '", + "errors": ["message", "..."], + "status": 404 +} +``` + +- `errors` is a list of one or more message strings (multi-message + validation errors, e.g. per-row insert errors, list them all). +- `error` is the messages joined with `"; "`. +- `status` always matches the HTTP status code. + +The shape is produced by four code paths, all delegating to `error_body()`: + +1. **Exception handler** (handle_exception.py) — `NotFound`, + `DatasetteError`, `BadRequest` etc. on `.json` paths. `DatasetteError` + `error_dict` context keys are merged in; the legacy `title` key is no + longer emitted in JSON (it survives in the HTML error template context). +2. **The `_error()` helper** (views/base.py:183-184) — the write API, + stored-query API, execute-write and permission-denied paths. +3. **JSON renderer errors** (renderer.py) — SQL errors on table/query + endpoints return HTTP 400 with the canonical keys **plus** the context + keys of the response it could not produce: ```json - {"ok": false, "error": "message", "status": 404, "title": null} + {"ok": false, "error": "no such table: x", "errors": ["no such table: x"], + "status": 400, "rows": [], "truncated": false} ``` -2. **The `_error()` helper** (views/base.py:183-184) — used by the write API, - stored-query API, execute-write and several permission-denied paths: + Invalid `_shape=` values and `_shape=object` misuse (on queries or + pk-less tables) also return canonical 400 errors. +4. **Permission debug endpoints** (`/-/allowed`, `/-/rules`, `/-/check`, + POST `/-/permissions`) — canonical shape (previously bare + `{"error": ...}` objects). - ```json - {"ok": false, "errors": ["message", "..."]} - ``` - - Note: plural `errors`, a list, and no `status`/`title` keys. - -3. **JSON renderer errors** (renderer.py:52-56) — SQL errors on table/query - endpoints return HTTP 400 with the error embedded in the data envelope: - - ```json - {"ok": false, "error": "no such table: x", "rows": [], "truncated": false} - ``` - - An invalid `_shape=` value produces `{"ok": false, "error": "Invalid _shape: x", - "status": 400, "title": null}` (renderer.py:101-108). - -4. **Ad-hoc `{"error": ...}` objects** — the permission debug endpoints - (`/-/allowed`, `/-/rules`, `/-/check`, POST `/-/permissions`) return e.g. - `{"error": "Unknown action: x"}` with no `ok` key (views/special.py). - -Method-not-allowed responses return HTTP 405 -`{"ok": false, "error": "Method not allowed"}` when the path ends in `.json` -or the request content type is `application/json`; plain text otherwise -(views/base.py:53, 88-98). +Method-not-allowed responses return HTTP 405 with the canonical shape when +the path ends in `.json` or the request content type is `application/json`; +plain text otherwise (views/base.py). **`Forbidden` is special:** when a view raises `Forbidden` (e.g. via `ensure_permission`), the default `forbidden()` plugin hook renders an **HTML @@ -144,11 +149,10 @@ build JSON directly): - `array` — response body is a bare JSON array of row objects - `arrayfirst` — bare JSON array of the first column's values - `object` — table views only: an object keyed by primary-key string. - On queries: `{"ok": false, "error": "_shape=object is only available on - tables"}` (with HTTP status 200); on tables without primary keys a similar - error. - - anything else — HTTP 400 `{"ok": false, "error": "Invalid _shape: x", - "status": 400, "title": null}` + On queries or tables without primary keys: a canonical 400 error + (`_shape=object is only available on tables` / + `_shape=object not available for tables with no primary keys`). + - anything else — canonical HTTP 400 error `Invalid _shape: x` - **`_nl=on`** — with `_shape=array` only: newline-delimited JSON, `text/plain`. - **`_json=COLUMN`** (repeatable) — parse that column's string values with `json.loads` so they nest as JSON; parse failures leave the value unchanged. @@ -157,7 +161,7 @@ build JSON directly): - `columns` is stripped from dict-shaped output unless `?_extra=columns` was requested (renderer.py:110-113). - If a SQL error occurred, `_shape` is ignored, HTTP status is 400 and the - envelope carries `"ok": false, "error": ...` (renderer.py:52-56). + envelope carries the canonical error keys alongside `rows`/`truncated`. ### The `?_extra=` system @@ -340,8 +344,8 @@ GET renders a confirmation page (or redirects if anonymous); POST deletes the - **POST** — form-encoded `actor` (JSON string), `permission`, optional `resource_1`, `resource_2`; returns **JSON** `{"action", "allowed", "resource": {"parent", "child", "path"}}` plus - `actor_id` when present. Errors: unknown action → 404 `{"error": ...}`; - child without parent → 400 `{"error": ...}`. + `actor_id` when present. Errors: unknown action → 404; child without + parent → 400 (both canonical error shape). ### GET /-/allowed(.json) @@ -351,7 +355,7 @@ path always renders the HTML form; `.json` returns JSON. - **Permission:** none — reports the **current actor's own** allowed resources. Items gain a `reason` field if the actor also holds `permissions-debug`. -- **Parameters:** `action` (required; missing → 400 `{"error": ...}`, unknown +- **Parameters:** `action` (required; missing → 400 canonical error, unknown → 404), `parent`, `child` (requires `parent`), `page` (default 1), `page_size` (default 50, silently capped at 200). - **Response:** `{"action", "actor_id", "page", "page_size", "total", @@ -497,7 +501,7 @@ queries section). GET → 405. Body is parsed as JSON regardless of content type; invalid JSON → 400 `{"ok": false, "errors": ["Invalid JSON: ..."]}`. -- **Permissions** (all denials → 403 `{"ok": false, "errors": [...]}`, +- **Permissions** (all denials → 403 canonical error JSON, all checked at the **database** level): - `create-table` — always required (`["Permission denied"]`) - `insert-row` — if `rows`/`row` provided (`need insert-row`) @@ -812,8 +816,8 @@ only — views get 400 `"Autocomplete is only available for tables"`. ## The write API -All write endpoints return errors via `_error()` -(`{"ok": false, "errors": [...]}`) and check permissions with +All write endpoints return errors via `_error()` (the canonical error +shape) and check permissions with `datasette.allowed()` directly, so their 403s are JSON (unlike the `Forbidden`-raising read endpoints). Routes: app.py:2719-2762. diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index e630ae4b..9eac543f 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -18,7 +18,17 @@ Findings are grouped by theme. Each carries a priority: --- -## 1. Error responses: four shapes is three too many (P1) +## 1. Error responses: four shapes is three too many (P1) — ✅ IMPLEMENTED + +> **Status:** implemented. All four shapes now delegate to a shared +> `error_body()` helper (`datasette/utils/__init__.py`) producing +> `{"ok": false, "error": "", "errors": [...], "status": }`. +> The `title` key is no longer emitted in JSON; the bare `{"error": ...}` +> debug-endpoint shape is gone; `_shape=object` misuse now returns HTTP 400 +> (part of §1b). Covered by `tests/test_error_shape.py` and documented in +> the "Error responses" section of `docs/json_api.rst`. Still open from +> this section's sub-items: §1a (`Forbidden` → HTML), the write +> canned-query 200 (§1b), and the §1c status outliers. The API currently produces four distinct JSON error shapes depending on which internal layer generates the error: @@ -348,7 +358,7 @@ Two details make tiering urgent rather than optional: ## 10. Summary of P1 items (the pre-1.0 checklist) -1. One canonical JSON error shape; retire the other three (§1). +1. ~~One canonical JSON error shape; retire the other three (§1).~~ ✅ Done. 2. `Forbidden` → JSON 403 for JSON requests (§1a). 3. No `ok: false` with HTTP 200 (§1b: `_shape=object`, write canned-query SQL errors). diff --git a/tests/test_api.py b/tests/test_api.py index f57d0206..e5ed1d23 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -323,20 +323,21 @@ def test_sql_time_limit(app_client_shorter_time_limit): "/fixtures/-/query.json?sql=select+sleep(0.5)", ) assert 400 == response.status + expected_message = ( + "

SQL query took too long. The time limit is controlled by the\n" + 'sql_time_limit_ms\n' + "configuration option.

\n" + '\n' + "" + ) assert response.json == { "ok": False, - "error": ( - "

SQL query took too long. The time limit is controlled by the\n" - 'sql_time_limit_ms\n' - "configuration option.

\n" - '\n' - "" - ), + "error": expected_message, + "errors": [expected_message], "status": 400, - "title": "SQL Interrupted", } @@ -350,7 +351,7 @@ async def test_custom_sql_time_limit(ds_client): "/fixtures/-/query.json?sql=select+sleep(0.01)&_timelimit=5", ) assert response.status_code == 400 - assert response.json()["title"] == "SQL Interrupted" + assert response.json()["error"].startswith("

SQL query took too long.") @pytest.mark.asyncio diff --git a/tests/test_api_write.py b/tests/test_api_write.py index 76797742..17542d4b 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -1,6 +1,6 @@ from datasette.app import Datasette from datasette.events import RenameTableEvent -from datasette.utils import escape_sqlite, sqlite3 +from datasette.utils import error_body, escape_sqlite, sqlite3 from .utils import last_event import pytest import time @@ -788,7 +788,12 @@ async def test_update_row_invalid_key(ds_write): headers=_headers(token), ) assert response.status_code == 400 - assert response.json() == {"ok": False, "errors": ["Invalid keys: bad_key"]} + assert response.json() == { + "ok": False, + "error": "Invalid keys: bad_key", + "errors": ["Invalid keys: bad_key"], + "status": 400, + } @pytest.mark.asyncio @@ -1103,10 +1108,9 @@ async def test_alter_table_foreign_key_requires_fk_table_for_fk_column(ds_write) headers=_headers(write_token(ds_write, permissions=["at"])), ) assert response.status_code == 400 - assert response.json() == { - "ok": False, - "errors": ["operations.0.add_foreign_key.args: fk_column requires fk_table"], - } + assert response.json() == error_body( + ["operations.0.add_foreign_key.args: fk_column requires fk_table"], 400 + ) @pytest.mark.asyncio @@ -1130,10 +1134,9 @@ async def test_alter_table_foreign_key_without_fk_column_requires_single_pk(ds_w headers=_headers(token), ) assert response.status_code == 400 - assert response.json() == { - "ok": False, - "errors": ["Could not detect single primary key for table 'accounts'"], - } + assert response.json() == error_body( + ["Could not detect single primary key for table 'accounts'"], 400 + ) @pytest.mark.asyncio @@ -1199,10 +1202,7 @@ async def test_foreign_key_suggestions_permission_denied(ds_write): headers=_headers(token), ) assert response.status_code == 403 - assert response.json() == { - "ok": False, - "errors": ["Permission denied: need alter-table"], - } + assert response.json() == error_body(["Permission denied: need alter-table"], 403) @pytest.mark.asyncio @@ -1313,10 +1313,7 @@ async def test_foreign_key_targets_permission_denied(ds_write): headers=_headers(token), ) assert response.status_code == 403 - assert response.json() == { - "ok": False, - "errors": ["Permission denied: need create-table"], - } + assert response.json() == error_body(["Permission denied: need create-table"], 403) @pytest.mark.asyncio @@ -1339,10 +1336,7 @@ async def test_alter_table_permission_denied(ds_write): headers=_headers(token), ) assert response.status_code == 403 - assert response.json() == { - "ok": False, - "errors": ["Permission denied: need alter-table"], - } + assert response.json() == error_body(["Permission denied: need alter-table"], 403) @pytest.mark.asyncio @@ -2021,6 +2015,9 @@ async def test_create_table( ) assert response.status_code == expected_status data = response.json() + if expected_response.get("ok") is False: + # Error expectations list their messages; derive the canonical envelope + expected_response = error_body(expected_response["errors"], expected_status) assert data == expected_response # Should have tracked the expected events events = ds_write._tracked_events @@ -2218,13 +2215,12 @@ async def test_create_table_column_validation(ds_write, column, expected_error): ) if expected_error: assert response.status_code == 400 - assert response.json() == {"ok": False, "errors": [expected_error]} + assert response.json() == error_body([expected_error], 400) else: assert response.status_code == 400 - assert response.json() == { - "ok": False, - "errors": ["Could not detect single primary key for table 'owners'"], - } + assert response.json() == error_body( + ["Could not detect single primary key for table 'owners'"], 400 + ) @pytest.mark.asyncio @@ -2262,10 +2258,9 @@ async def test_create_table_foreign_key_without_fk_column_requires_single_pk(ds_ headers=_headers(token), ) assert response.status_code == 400 - assert response.json() == { - "ok": False, - "errors": ["Could not detect single primary key for table 'accounts'"], - } + assert response.json() == error_body( + ["Could not detect single primary key for table 'accounts'"], 400 + ) @pytest.mark.asyncio @@ -2415,10 +2410,9 @@ async def test_create_table_error_if_pk_changed(ds_write): headers=_headers(token), ) assert second_response.status_code == 400 - assert second_response.json() == { - "ok": False, - "errors": ["pk cannot be changed for existing table"], - } + assert second_response.json() == error_body( + ["pk cannot be changed for existing table"], 400 + ) @pytest.mark.asyncio @@ -2442,10 +2436,9 @@ async def test_create_table_error_rows_twice_with_duplicates(ds_write): headers=_headers(token), ) assert second_response.status_code == 400 - assert second_response.json() == { - "ok": False, - "errors": ["UNIQUE constraint failed: test_create_twice.id"], - } + assert second_response.json() == error_body( + ["UNIQUE constraint failed: test_create_twice.id"], 400 + ) @pytest.mark.asyncio @@ -2468,6 +2461,8 @@ async def test_method_not_allowed(ds_write, path): assert response.json() == { "ok": False, "error": "Method not allowed", + "errors": ["Method not allowed"], + "status": 405, } @@ -2535,10 +2530,9 @@ async def test_create_using_alter_against_existing_table( ) if not has_alter_permission: assert response2.status_code == 403 - assert response2.json() == { - "ok": False, - "errors": ["Permission denied: need alter-table"], - } + assert response2.json() == error_body( + ["Permission denied: need alter-table"], 403 + ) else: assert response2.status_code == 201 diff --git a/tests/test_base_view.py b/tests/test_base_view.py index 2cd4d601..c1b0cf20 100644 --- a/tests/test_base_view.py +++ b/tests/test_base_view.py @@ -53,6 +53,8 @@ async def test_get_view(): assert json.loads(post_json_response.body) == { "ok": False, "error": "Method not allowed", + "errors": ["Method not allowed"], + "status": 405, } assert post_json_response.status == 405 diff --git a/tests/test_column_types.py b/tests/test_column_types.py index 45a9e7d1..4e553771 100644 --- a/tests/test_column_types.py +++ b/tests/test_column_types.py @@ -9,7 +9,7 @@ from datasette.column_types import ( ) from datasette.hookspecs import hookimpl from datasette.plugins import pm -from datasette.utils import sqlite3 +from datasette.utils import error_body, sqlite3 from datasette.utils import StartupError import markupsafe import pytest @@ -426,7 +426,7 @@ async def test_set_column_type_api_errors( kwargs["json"] = body response = await ds_ct.client.post("/data/posts/-/set-column-type", **kwargs) assert response.status_code == expected_status - assert response.json() == {"ok": False, "errors": expected_errors} + assert response.json() == error_body(expected_errors, expected_status) @pytest.mark.asyncio diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py new file mode 100644 index 00000000..6747e47a --- /dev/null +++ b/tests/test_error_shape.py @@ -0,0 +1,185 @@ +""" +Tests for the canonical JSON error shape. + +Every JSON error response from Datasette should use one shape: + + { + "ok": false, + "error": "", + "errors": ["", ...], + "status": + } + +Additional context keys (for example "rows" and "truncated" on SQL errors) +are permitted, but "ok", "error", "errors" and "status" must always be +present and the legacy "title" key must not be. + +https://github.com/simonw/datasette/issues - 1.0 API consistency +""" + +import pytest +from datasette.app import Datasette +from datasette.utils import sqlite3 + + +def assert_canonical_error(response, expected_status): + assert response.status_code == expected_status + data = response.json() + assert data["ok"] is False + assert isinstance(data["error"], str) + assert data["error"] + assert isinstance(data["errors"], list) + assert data["errors"] + assert all(isinstance(message, str) for message in data["errors"]) + assert data["error"] == "; ".join(data["errors"]) + assert data["status"] == expected_status + assert "title" not in data + return data + + +@pytest.fixture +def ds_error_shape(tmp_path_factory): + db_directory = tmp_path_factory.mktemp("dbs") + db_path = str(db_directory / "data.db") + conn = sqlite3.connect(db_path) + conn.execute("vacuum") + conn.execute("create table docs (id integer primary key, title text)") + conn.close() + ds = Datasette([db_path]) + ds.root_enabled = True + yield ds + ds.close() + + +# Shape 1: the exception handler (handle_exception.py) + + +@pytest.mark.asyncio +async def test_not_found_error_shape(ds_client): + response = await ds_client.get("/fixtures/no_such_table.json") + assert_canonical_error(response, 404) + + +@pytest.mark.asyncio +async def test_datasette_error_with_title_omits_title_key(ds_client): + # DatasetteError(title="Invalid SQL") previously leaked a "title" key + response = await ds_client.get( + "/fixtures/-/query.json?sql=update+facetable+set+state+=+1" + ) + data = assert_canonical_error(response, 400) + assert data["errors"] == ["Statement must be a SELECT"] + + +# Shape 2: the _error() helper (views/base.py) - write API and friends + + +@pytest.mark.asyncio +async def test_write_api_validation_error_shape(ds_error_shape): + token = "dstok_{}".format( + ds_error_shape.sign( + {"a": "root", "token": "dstok", "t": 0}, + namespace="token", + ) + ) + response = await ds_error_shape.client.post( + "/data/docs/-/insert", + json={"rows": [{"nope": 1}, {"also_nope": 2}]}, + headers={ + "Authorization": "Bearer {}".format(token), + "Content-Type": "application/json", + }, + ) + data = assert_canonical_error(response, 400) + # Multiple messages: errors keeps them all, error joins them + assert len(data["errors"]) == 2 + assert data["errors"][0].startswith("Row 0") + assert data["errors"][1].startswith("Row 1") + + +@pytest.mark.asyncio +async def test_write_api_permission_denied_shape(ds_error_shape): + response = await ds_error_shape.client.post( + "/data/docs/-/insert", + json={"rows": [{"title": "hello"}]}, + headers={"Content-Type": "application/json"}, + ) + assert_canonical_error(response, 403) + + +# Shape 3: the JSON renderer (renderer.py) + + +@pytest.mark.asyncio +async def test_sql_error_shape_keeps_context_keys(ds_client): + response = await ds_client.get( + "/fixtures/-/query.json?sql=select+*+from+no_such_table" + ) + data = assert_canonical_error(response, 400) + # Renderer errors keep their context keys + assert data["rows"] == [] + assert "truncated" in data + + +@pytest.mark.asyncio +async def test_invalid_shape_error_shape(ds_client): + response = await ds_client.get("/fixtures/-/query.json?sql=select+1&_shape=bananas") + data = assert_canonical_error(response, 400) + assert data["errors"] == ["Invalid _shape: bananas"] + + +@pytest.mark.asyncio +async def test_shape_object_on_query_is_a_400_error(ds_client): + # Previously returned HTTP 200 with an ok: false body + response = await ds_client.get("/fixtures/-/query.json?sql=select+1&_shape=object") + data = assert_canonical_error(response, 400) + assert data["errors"] == ["_shape=object is only available on tables"] + + +# Shape 4: bare {"error": ...} from the permission debug endpoints + + +@pytest.mark.asyncio +async def test_allowed_missing_action_error_shape(ds_client): + response = await ds_client.get("/-/allowed.json") + data = assert_canonical_error(response, 400) + assert data["errors"] == ["action parameter is required"] + + +@pytest.mark.asyncio +async def test_allowed_unknown_action_error_shape(ds_client): + response = await ds_client.get("/-/allowed.json?action=no_such_action") + assert_canonical_error(response, 404) + + +@pytest.mark.asyncio +async def test_check_unknown_action_error_shape(ds_error_shape): + response = await ds_error_shape.client.get( + "/-/check.json?action=no_such_action", + actor={"id": "root"}, + ) + assert_canonical_error(response, 404) + + +@pytest.mark.asyncio +async def test_rules_missing_action_error_shape(ds_error_shape): + response = await ds_error_shape.client.get( + "/-/rules.json", + actor={"id": "root"}, + ) + data = assert_canonical_error(response, 400) + assert data["errors"] == ["action parameter is required"] + + +# Other stragglers + + +@pytest.mark.asyncio +async def test_method_not_allowed_error_shape(ds_client): + response = await ds_client.post("/fixtures.json") + assert_canonical_error(response, 405) + + +@pytest.mark.asyncio +async def test_schema_unknown_database_error_shape(ds_client): + response = await ds_client.get("/no_such_db/-/schema.json") + assert_canonical_error(response, 404) diff --git a/tests/test_table_api.py b/tests/test_table_api.py index 272e39e3..c8ba31b7 100644 --- a/tests/test_table_api.py +++ b/tests/test_table_api.py @@ -31,8 +31,8 @@ async def test_table_not_exists_json(ds_client): assert (await ds_client.get("/fixtures/blah.json")).json() == { "ok": False, "error": "Table not found", + "errors": ["Table not found"], "status": 404, - "title": None, } @@ -242,8 +242,8 @@ async def test_table_shape_invalid(ds_client): assert response.json() == { "ok": False, "error": "Invalid _shape: invalid", + "errors": ["Invalid _shape: invalid"], "status": 400, - "title": None, } @@ -635,8 +635,8 @@ async def test_searchable_invalid_column(ds_client): assert response.json() == { "ok": False, "error": "Cannot search by that column", + "errors": ["Cannot search by that column"], "status": 400, - "title": None, } @@ -775,7 +775,7 @@ async def test_table_filter_extra_where_invalid(ds_client): "/fixtures/facetable.json?_where=_neighborhood=Dogpatch'" ) assert response.status_code == 400 - assert "Invalid SQL" == response.json()["title"] + assert "unrecognized token" in response.json()["error"] def test_table_filter_extra_where_disabled_if_no_sql_allowed(): diff --git a/tests/test_table_html.py b/tests/test_table_html.py index 3af2bb08..46d43c6c 100644 --- a/tests/test_table_html.py +++ b/tests/test_table_html.py @@ -1979,8 +1979,8 @@ async def test_sort_errors(ds_client, json, params, error): assert response.json() == { "ok": False, "error": error, + "errors": [error], "status": 400, - "title": None, } else: assert error in response.text From bc51c00724e09f71cd452139153f29eb528fd637 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 05:10:26 +0000 Subject: [PATCH 03/46] Remove legacy .jsono format extension The homepage routes now only accept .json, and the row view no longer redirects .jsono to .json?_shape=objects. The .jsono extension was superseded by ?_shape= and returned output identical to .json. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/app.py | 4 ++-- datasette/views/row.py | 13 ------------- existing-api.md | 6 ++---- stable-api-recommendations.md | 6 ++++-- 4 files changed, 8 insertions(+), 21 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 9c9b7de4..e13f0731 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -2514,8 +2514,8 @@ class Datasette: def add_route(view, regex): routes.append((regex, view)) - add_route(IndexView.as_view(self), r"/(\.(?Pjsono?))?$") - add_route(IndexView.as_view(self), r"/-/(\.(?Pjsono?))?$") + add_route(IndexView.as_view(self), r"/(\.(?Pjson))?$") + add_route(IndexView.as_view(self), r"/-/(\.(?Pjson))?$") add_route(permanent_redirect("/-/"), r"/-$") add_route(favicon, "/favicon.ico") diff --git a/datasette/views/row.py b/datasette/views/row.py index 129216b9..c99b75d1 100644 --- a/datasette/views/row.py +++ b/datasette/views/row.py @@ -21,7 +21,6 @@ from datasette.utils import ( InvalidSql, make_slot_function, path_from_row_pks, - path_with_added_args, path_with_format, path_with_removed_args, to_css_class, @@ -209,18 +208,6 @@ class RowView(BaseView): end = time.perf_counter() data["query_ms"] = (end - start) * 1000 - # Special case for .jsono extension - redirect to _shape=objects - if format_ == "jsono": - return self.redirect( - request, - path_with_added_args( - request, - {"_shape": "objects"}, - path=request.path.rsplit(".jsono", 1)[0] + ".json", - ), - forward_querystring=False, - ) - if format_ in self.ds.renderers.keys(): # Dispatch request to the correct output format renderer # (CSV is not handled here due to streaming) diff --git a/existing-api.md b/existing-api.md index 1933d287..d7678613 100644 --- a/existing-api.md +++ b/existing-api.md @@ -25,8 +25,7 @@ directory: every claim below is based on the route table in `datasette/app.py` - Most read endpoints are registered with an optional format suffix: `/(...)(\.(?Pjson))?$`. The bare path returns HTML; the `.json` - extension returns JSON. The homepage additionally accepts the legacy - `.jsono` extension, which returns identical JSON (app.py:2517-2518). + extension returns JSON. - Table, row and query routes accept any `\w+` format extension; formats other than the built-in `html`, `json`, `csv`, `blob` must be provided by a plugin via `register_output_renderer`, otherwise the request 404s. @@ -772,8 +771,7 @@ tilde-encoded primary key values (rowid for rowid tables). - **Foreign-key label expansion does not apply to row JSON** — `_labels` has no effect here; expansion happens only in the HTML path (views/row.py:445-475). -- `_shape`, `_json`, `_nl`, `_json_infinity`, `_ttl` apply. A `.jsono` - request redirects to `.json?_shape=objects`. +- `_shape`, `_json`, `_nl`, `_json_infinity`, `_ttl` apply. ### The .blob format diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index 9eac543f..8d8a87b4 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -233,8 +233,10 @@ Concerns: `_labels=on/off`, `?all=1`, `is_write=1|0|true|false|t|f|yes|no|on|off`, `_nocount=1`. Adopt one accepted set (the query-list parser at query_helpers.py:81-94 is a good candidate) and apply it everywhere. -- **`.jsono`** survives on the homepage route (identical output to `.json`) - and as a row-view redirect. Remove it at 1.0; it is pure legacy. +- ~~**`.jsono`** survives on the homepage route (identical output to `.json`) + and as a row-view redirect. Remove it at 1.0; it is pure legacy.~~ + ✅ Removed: the homepage routes only accept `.json` and the row-view + redirect is gone. - **`_json` is overloaded:** on GET it is a renderer option naming a column to parse as JSON (repeatable); on canned-query POST a `_json` body field forces a JSON response. Two unrelated meanings for one name. From 089e96a43763c5f215218498335d81958c59a414 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 13:40:05 +0000 Subject: [PATCH 04/46] Add "ok": true to every JSON object success response JsonDataView now injects "ok": true into dict responses, covering /-/versions, /-/settings, /-/config, /-/threads and /-/actor. The homepage JSON, /-/jump, the three /-/schema endpoints, /-/allowed, /-/rules, /-/check, POST /-/permissions and the table /-/autocomplete endpoint set it explicitly. The remaining top-level array endpoints (/-/plugins, /-/databases, /-/actions) will be converted to objects in separate commits. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/views/index.py | 1 + datasette/views/special.py | 10 ++- datasette/views/table.py | 9 ++- docs/introspection.rst | 9 +++ docs/json_api.rst | 2 +- docs/pages.rst | 6 +- existing-api.md | 30 +++++---- stable-api-recommendations.md | 10 ++- tests/test_api.py | 8 ++- tests/test_auth.py | 8 +-- tests/test_autocomplete.py | 25 +++++--- tests/test_cli_serve_get.py | 8 ++- tests/test_docs.py | 4 +- tests/test_internals_datasette.py | 2 +- tests/test_internals_datasette_client.py | 6 +- tests/test_permissions.py | 3 +- tests/test_plugins.py | 8 ++- tests/test_success_envelope.py | 78 ++++++++++++++++++++++++ 18 files changed, 180 insertions(+), 47 deletions(-) create mode 100644 tests/test_success_envelope.py diff --git a/datasette/views/index.py b/datasette/views/index.py index 6a9462ac..86f8ae7d 100644 --- a/datasette/views/index.py +++ b/datasette/views/index.py @@ -151,6 +151,7 @@ class IndexView(BaseView): return Response( json.dumps( { + "ok": True, "databases": {db["name"]: db for db in databases}, "metadata": await self.ds.get_instance_metadata(), }, diff --git a/datasette/views/special.py b/datasette/views/special.py index 602ec5ea..c289a240 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -63,6 +63,8 @@ class JsonDataView(BaseView): headers = {} if self.ds.cors: add_cors_headers(headers) + if isinstance(data, dict): + data = {"ok": True, **data} return Response.json(data, headers=headers) else: context = { @@ -414,6 +416,7 @@ class AllowedResourcesView(BaseView): # If catalog tables don't exist yet, return empty results return ( { + "ok": True, "action": action, "actor_id": actor_id, "page": page, @@ -448,6 +451,7 @@ class AllowedResourcesView(BaseView): return f"{request.path}?{query}" response = { + "ok": True, "action": action, "actor_id": actor_id, "page": page, @@ -573,6 +577,7 @@ class PermissionRulesView(BaseView): return f"{request.path}?{query}" response = { + "ok": True, "action": action, "actor_id": (actor or {}).get("id") if actor else None, "page": page, @@ -623,6 +628,7 @@ async def _check_permission_for_actor(ds, action, parent, child, actor): allowed = await ds.allowed(action=action, resource=resource_obj, actor=actor) response = { + "ok": True, "action": action, "allowed": bool(allowed), "resource": { @@ -1208,7 +1214,7 @@ class JumpView(BaseView): match["display_name"] = row["display_name"] matches.append(match) - return Response.json({"matches": matches, "truncated": truncated}) + return Response.json({"ok": True, "matches": matches, "truncated": truncated}) class SchemaBaseView(BaseView): @@ -1230,7 +1236,7 @@ class SchemaBaseView(BaseView): headers = {} if self.ds.cors: add_cors_headers(headers) - return Response.json(data, headers=headers) + return Response.json({"ok": True, **data}, headers=headers) def format_error_response(self, error_message, format_, status=404): """Format error response based on requested format.""" diff --git a/datasette/views/table.py b/datasette/views/table.py index 1fc151e6..ef9831b2 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1525,7 +1525,7 @@ class TableAutocompleteView(BaseView): and value_as_boolean(initial_arg) ) if not q and not initial: - return Response.json({"rows": []}) + return Response.json({"ok": True, "rows": []}) params = { "q": q, "like": "%{}%".format(_escape_like(q)), @@ -1588,10 +1588,13 @@ class TableAutocompleteView(BaseView): custom_time_limit=AUTOCOMPLETE_TIME_LIMIT_MS, ) except QueryInterrupted: - return Response.json({"rows": []}) + return Response.json({"ok": True, "rows": []}) return Response.json( - {"rows": _autocomplete_response_rows(results.rows, pks, label_column)} + { + "ok": True, + "rows": _autocomplete_response_rows(results.rows, pks, label_column), + } ) diff --git a/docs/introspection.rst b/docs/introspection.rst index 4834f441..d0780763 100644 --- a/docs/introspection.rst +++ b/docs/introspection.rst @@ -7,6 +7,8 @@ Datasette includes some pages and JSON API endpoints for introspecting the curre Each of these pages can be viewed in your browser. Add ``.json`` to the URL to get back the contents as JSON. +JSON responses that return an object include an ``"ok": true`` key, consistent with the rest of the :ref:`JSON API `. + .. _JsonDataView_metadata: /-/metadata @@ -37,6 +39,7 @@ Shows the version of Datasette, Python and SQLite. `Versions example ` for this instance of Datasette. T .. code-block:: json { + "ok": true, "settings": { "template_debug": true, "trace_debug": true, @@ -160,6 +165,7 @@ The endpoint supports a ``?q=`` query parameter for filtering items by name. .. code-block:: json { + "ok": true, "matches": [ { "name": "fixtures", @@ -188,6 +194,7 @@ Search example with ``?q=facet`` returns only items matching ``.*facet.*``: .. code-block:: json { + "ok": true, "matches": [ { "name": "fixtures: facetable", @@ -220,6 +227,7 @@ Shows details of threads and ``asyncio`` tasks. `Threads example jsono?))?$` and `/-/(\.(?Pjsono?))?$` further filtered by `view-database` / `view-table` for the actor. - **Parameters:** `_sort=relationships` sorts each database's truncated table list by foreign-key relationship count. -- **JSON response** (index.py:147-161): +- **JSON response** (index.py:147-161) — includes `ok: true` plus: - `databases` — an **object keyed by database name** (not a list). Each value: `name`, `hash` (or null), `color`, `path`, `tables_and_views_truncated` (up to 5 items: `name`, `columns`, @@ -260,7 +269,7 @@ per-database `view-database` permissions**. app.py:2574-2579, registered with `permission=None` — **accessible to any request including anonymous**. No parameters. -Response: `{"actor": {...}}` or `{"actor": null}` (app.py:2287-2288). +Response: `{"ok": true, "actor": {...}}` or `{"ok": true, "actor": null}` (app.py:2287-2288). ### GET /-/actions(.json) @@ -313,7 +322,7 @@ an optional `.json` suffix but the view **always returns JSON**. `jump_items_sql` plugin hook). - **Parameter:** `q` — whitespace-split terms matched as a case-insensitive `%term1%term2%` LIKE pattern. -- **Response:** `{"matches": [...], "truncated": bool}`; each match: +- **Response:** `{"ok": true, "matches": [...], "truncated": bool}`; each match: `name`, `url`, `type` (`database`/`table`/`view`/`query`/plugin-defined), `description`, optional `display_name`. Capped at 100 matches. @@ -324,7 +333,7 @@ an optional `.json` suffix but the view **always returns JSON**. - **Permission:** no explicit check; only databases the actor can `view-database` are included (others silently omitted). - **Formats:** no extension → HTML; `.json` → - `{"schemas": [{"database": name, "schema": "..."}]}`; `.md` → + `{"ok": true, "schemas": [{"database": name, "schema": "..."}]}`; `.md` → `text/markdown` rendering. ### GET/POST /-/logout @@ -610,10 +619,9 @@ views/table_create_alter.py:965-1005). - **Unknown database** → 404; for `.json`: `{"ok": false, "error": "Database not found"}`. (The existence check runs before the permission check.) -- **Responses:** `.json` → 200 `{"database": "", "schema": ""}` +- **Responses:** `.json` → 200 `{"ok": true, "database": "", "schema": ""}` (concatenated `sqlite_master.sql` joined with `;\n`); `.md` → - `text/markdown`; no extension → HTML. Note the JSON has **no `ok` key** on - success. + `text/markdown`; no extension → HTML. --- @@ -787,8 +795,8 @@ download attachment. In JSON output, binary cells appear as `TableSchemaView` (app.py:2751-2754; views/special.py:1332-1378). - **Permission:** `view-table` via `ensure_permission` (denied → 403 HTML). -- **Responses:** `.json` → 200 `{"database", "table", "schema"}` (no `ok` - key); `.md` → `text/markdown`; no extension → HTML. Missing table → 404 +- **Responses:** `.json` → 200 `{"ok": true, "database", "table", "schema"}`; + `.md` → `text/markdown`; no extension → HTML. Missing table → 404 `{"ok": false, "error": "Table not found"}` for `.json`. ### GET /\/\/-/fragment @@ -805,10 +813,10 @@ only — views get 400 `"Autocomplete is only available for tables"`. - **Permission:** `view-table` (denied → `Forbidden` → 403). - **Parameters:** `q` (matched with escaped `LIKE %q%` against pk columns and the label column) and `_initial` (truthy: with empty `q`, return the 10 - most recent rows). Neither → `{"rows": []}`. + most recent rows). Neither → `{"ok": true, "rows": []}`. - **Response:** `{"rows": [{"pks": {pk_name: value}, "label": "..."}]}` — max 10 items; 500 ms query budget with fallbacks, timing out to - `{"rows": []}`. + `{"ok": true, "rows": []}`. --- diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index 8d8a87b4..48a864af 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -102,7 +102,15 @@ key — see §2.) --- -## 2. Success envelope: `ok` is not universal, arrays are not extensible (P1/P2) +## 2. Success envelope: `ok` is not universal, arrays are not extensible (P1/P2) — ✅ PARTIALLY IMPLEMENTED + +> **Status:** recommendation 2 and 3 are implemented — every JSON-object +> success response now includes `"ok": true` (`JsonDataView` injects it for +> dict responses; homepage, jump, schema, permission-debug and autocomplete +> views set it explicitly; covered by `tests/test_success_envelope.py`). +> Recommendation 1 (wrapping the `/-/plugins`, `/-/databases`, `/-/actions` +> top-level arrays in objects) is being landed as separate per-endpoint +> commits. §2a/2b/2c remain open. Endpoints disagree about the success envelope: diff --git a/tests/test_api.py b/tests/test_api.py index e5ed1d23..4635b236 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -250,6 +250,7 @@ def test_no_files_uses_memory_database(app_client_no_files): response = app_client_no_files.get("/.json") assert response.status == 200 assert { + "ok": True, "databases": { "_memory": { "name": "_memory", @@ -525,7 +526,7 @@ def test_databases_json(app_client_two_attached_databases_one_immutable): @pytest.mark.asyncio async def test_threads_json(ds_client): response = await ds_client.get("/-/threads.json") - expected_keys = {"threads", "num_threads"} + expected_keys = {"ok", "threads", "num_threads"} if sys.version_info >= (3, 7, 0): expected_keys.update({"tasks", "num_tasks"}) data = response.json() @@ -610,6 +611,7 @@ async def test_actions_json(ds_client): async def test_settings_json(ds_client): response = await ds_client.get("/-/settings.json") assert response.json() == { + "ok": True, "default_page_size": 50, "default_facet_size": 30, "default_allow_sql": True, @@ -884,7 +886,7 @@ async def test_config_json(config, expected): "/-/config.json should return redacted configuration" ds = Datasette(config=config) response = await ds.client.get("/-/config.json") - assert response.json() == expected + assert response.json() == {"ok": True, **expected} @pytest.mark.asyncio @@ -980,7 +982,7 @@ async def test_config_json(config, expected): async def test_upgrade_metadata(metadata, expected_config, expected_metadata): ds = Datasette(metadata=metadata) response = await ds.client.get("/-/config.json") - assert response.json() == expected_config + assert response.json() == {"ok": True, **expected_config} response2 = await ds.client.get("/-/metadata.json") assert response2.json() == expected_metadata diff --git a/tests/test_auth.py b/tests/test_auth.py index 5868a21c..8e83d397 100644 --- a/tests/test_auth.py +++ b/tests/test_auth.py @@ -294,7 +294,7 @@ async def test_auth_with_dstok_token(ds_client, scenario, should_work): try: if should_work: data = response.json() - assert data.keys() == {"actor"} + assert data.keys() == {"ok", "actor"} actor = data["actor"] expected_keys = {"id", "token"} if scenario != "valid_unlimited_token": @@ -305,7 +305,7 @@ async def test_auth_with_dstok_token(ds_client, scenario, should_work): if scenario != "valid_unlimited_token": assert isinstance(actor["token_expires"], int) else: - assert response.json() == {"actor": None} + assert response.json() == {"ok": True, "actor": None} finally: ds_client.ds._settings["allow_signed_tokens"] = True @@ -337,10 +337,10 @@ def test_cli_create_token(app_client, expires): } if expires and expires > 0: expected_actor["token_expires"] = details["t"] + expires - assert response.json == {"actor": expected_actor} + assert response.json == {"ok": True, "actor": expected_actor} else: expected_actor = None - assert response.json == {"actor": expected_actor} + assert response.json == {"ok": True, "actor": expected_actor} @pytest.mark.asyncio diff --git a/tests/test_autocomplete.py b/tests/test_autocomplete.py index 76b9c902..194fcf01 100644 --- a/tests/test_autocomplete.py +++ b/tests/test_autocomplete.py @@ -25,13 +25,14 @@ async def test_autocomplete_single_pk_exact_match_and_label_order(): assert response.status_code == 200 assert response.json() == { + "ok": True, "rows": [ {"pks": {"id": 2}, "label": "Longer non-label pk match"}, {"pks": {"id": 20}, "label": "2"}, {"pks": {"id": 21}, "label": "22"}, {"pks": {"id": 3}, "label": "A label containing 2"}, {"pks": {"id": 200}, "label": "A"}, - ] + ], } @@ -52,12 +53,12 @@ async def test_autocomplete_blank_q_returns_no_results(): response = await ds.client.get("/autocomplete_blank/people/-/autocomplete?q=") assert response.status_code == 200 - assert response.json() == {"rows": []} + assert response.json() == {"ok": True, "rows": []} response = await ds.client.get("/autocomplete_blank/people/-/autocomplete") assert response.status_code == 200 - assert response.json() == {"rows": []} + assert response.json() == {"ok": True, "rows": []} @pytest.mark.asyncio @@ -81,11 +82,12 @@ async def test_autocomplete_initial_returns_latest_rows(): assert response.status_code == 200 assert response.json() == { + "ok": True, "rows": [ {"pks": {"id": 3}, "label": "Cleo"}, {"pks": {"id": 2}, "label": "Bob"}, {"pks": {"id": 1}, "label": "Alice"}, - ] + ], } response = await ds.client.get( @@ -94,11 +96,12 @@ async def test_autocomplete_initial_returns_latest_rows(): assert response.status_code == 200 assert response.json() == { + "ok": True, "rows": [ {"pks": {"id": 3}, "label": "Cleo"}, {"pks": {"id": 2}, "label": "Bob"}, {"pks": {"id": 1}, "label": "Alice"}, - ] + ], } @@ -121,9 +124,10 @@ async def test_autocomplete_escapes_like_characters(): assert response.status_code == 200 assert response.json() == { + "ok": True, "rows": [ {"pks": {"id": 1}, "label": "100% real"}, - ] + ], } @@ -149,11 +153,12 @@ async def test_autocomplete_compound_pk_searches_all_pk_columns(): assert response.status_code == 200 assert response.json() == { + "ok": True, "rows": [ {"pks": {"country": "mx", "code": "ca"}, "label": "Campeche"}, {"pks": {"country": "us", "code": "ca"}, "label": "California"}, {"pks": {"country": "ca", "code": "bc"}, "label": "British Columbia"}, - ] + ], } @@ -184,9 +189,10 @@ async def test_autocomplete_primary_key_called_label(): assert response.status_code == 200 assert response.json() == { + "ok": True, "rows": [ {"pks": {"label": "abc"}, "label": "Display value"}, - ] + ], } @@ -246,8 +252,9 @@ async def test_autocomplete_timeout_uses_prefix_fallback(monkeypatch): assert timeout_was_simulated data = response.json() assert data == { + "ok": True, "rows": [ {"pks": {"id": f"item-1999{i:02d}"}, "label": f"name 1999{i:02d}"} for i in range(10) - ] + ], } diff --git a/tests/test_cli_serve_get.py b/tests/test_cli_serve_get.py index dc852201..fe9416d6 100644 --- a/tests/test_cli_serve_get.py +++ b/tests/test_cli_serve_get.py @@ -95,7 +95,10 @@ def test_serve_with_get_and_token(): ], ) assert 0 == result2.exit_code, result2.output - assert json.loads(result2.output) == {"actor": {"id": "root", "token": "dstok"}} + assert json.loads(result2.output) == { + "ok": True, + "actor": {"id": "root", "token": "dstok"}, + } def test_serve_with_get_exit_code_for_error(): @@ -130,8 +133,9 @@ def test_serve_get_actor(): ) assert result.exit_code == 0 assert json.loads(result.output) == { + "ok": True, "actor": { "id": "root", "extra": "x", - } + }, } diff --git a/tests/test_docs.py b/tests/test_docs.py index 13b3a549..0bcb5e62 100644 --- a/tests/test_docs.py +++ b/tests/test_docs.py @@ -248,7 +248,7 @@ async def test_homepage(): async def test_actor_is_null(): ds = Datasette(memory=True) response = await ds.client.get("/-/actor.json") - assert response.json() == {"actor": None} + assert response.json() == {"ok": True, "actor": None} # -- end test_actor_is_null -- @@ -258,5 +258,5 @@ async def test_signed_cookie_actor(): ds = Datasette(memory=True) cookies = {"ds_actor": ds.client.actor_cookie({"id": "root"})} response = await ds.client.get("/-/actor.json", cookies=cookies) - assert response.json() == {"actor": {"id": "root"}} + assert response.json() == {"ok": True, "actor": {"id": "root"}} # -- end test_signed_cookie_actor -- diff --git a/tests/test_internals_datasette.py b/tests/test_internals_datasette.py index 2eaee3f9..1de394a5 100644 --- a/tests/test_internals_datasette.py +++ b/tests/test_internals_datasette.py @@ -188,7 +188,7 @@ async def test_num_sql_threads_zero(): await db.execute_write("create table t(id integer primary key)") await db.execute_write("insert into t (id) values (1)") response = await ds.client.get("/-/threads.json") - assert response.json() == {"num_threads": 0, "threads": []} + assert response.json() == {"ok": True, "num_threads": 0, "threads": []} response2 = await ds.client.get("/test_num_sql_threads_zero/t.json?_shape=array") assert response2.json() == [{"id": 1}] diff --git a/tests/test_internals_datasette_client.py b/tests/test_internals_datasette_client.py index 543077a5..e9aaaae8 100644 --- a/tests/test_internals_datasette_client.py +++ b/tests/test_internals_datasette_client.py @@ -318,7 +318,7 @@ async def test_actor_parameter_sets_cookie(datasette): """Passing actor= should sign a ds_actor cookie and authenticate the request.""" response = await datasette.client.get("/-/actor.json", actor={"id": "root"}) assert response.status_code == 200 - assert response.json() == {"actor": {"id": "root"}} + assert response.json() == {"ok": True, "actor": {"id": "root"}} @pytest.mark.asyncio @@ -327,7 +327,7 @@ async def test_actor_parameter_works_with_request_method(datasette): "GET", "/-/actor.json", actor={"id": "root"} ) assert response.status_code == 200 - assert response.json() == {"actor": {"id": "root"}} + assert response.json() == {"ok": True, "actor": {"id": "root"}} @pytest.mark.asyncio @@ -362,7 +362,7 @@ async def test_actor_parameter_merges_with_other_cookies(datasette): cookies={"unrelated": "value"}, ) assert response.status_code == 200 - assert response.json() == {"actor": {"id": "root"}} + assert response.json() == {"ok": True, "actor": {"id": "root"}} @pytest.mark.asyncio diff --git a/tests/test_permissions.py b/tests/test_permissions.py index 8323fe92..7d99213b 100644 --- a/tests/test_permissions.py +++ b/tests/test_permissions.py @@ -739,6 +739,7 @@ async def test_actor_restricted_permissions( "path": expected_path, } expected = { + "ok": True, "action": permission, "allowed": expected_result, "resource": expected_resource, @@ -1115,7 +1116,7 @@ def test_cli_create_token(options, expected): ], ) assert 0 == result2.exit_code, result2.output - assert json.loads(result2.output) == {"actor": expected} + assert json.loads(result2.output) == {"ok": True, "actor": expected} _visible_tables_re = re.compile(r">\/((\w+)\/(\w+))\.json<\/a> - Get rows for") diff --git a/tests/test_plugins.py b/tests/test_plugins.py index da14c714..59b1c0bf 100644 --- a/tests/test_plugins.py +++ b/tests/test_plugins.py @@ -783,9 +783,13 @@ async def test_hook_permission_resources_sql(): @pytest.mark.asyncio async def test_actor_json(ds_client): - assert (await ds_client.get("/-/actor.json")).json() == {"actor": None} + assert (await ds_client.get("/-/actor.json")).json() == { + "ok": True, + "actor": None, + } assert (await ds_client.get("/-/actor.json?_bot2=1")).json() == { - "actor": {"id": "bot2", "1+1": 2} + "ok": True, + "actor": {"id": "bot2", "1+1": 2}, } diff --git a/tests/test_success_envelope.py b/tests/test_success_envelope.py new file mode 100644 index 00000000..48407660 --- /dev/null +++ b/tests/test_success_envelope.py @@ -0,0 +1,78 @@ +""" +Tests for the canonical JSON success envelope. + +Every JSON object returned by a Datasette endpoint on success should include +"ok": true. (Endpoints that return a top-level array are being converted to +objects separately - see /-/plugins, /-/databases, /-/actions.) +""" + +import pytest +from datasette.app import Datasette +from datasette.utils import sqlite3 + + +@pytest.fixture +def ds_envelope(tmp_path_factory): + db_directory = tmp_path_factory.mktemp("dbs") + db_path = str(db_directory / "data.db") + conn = sqlite3.connect(db_path) + conn.execute("vacuum") + conn.execute("create table docs (id integer primary key, title text)") + conn.close() + ds = Datasette([db_path]) + ds.root_enabled = True + yield ds + ds.close() + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "path", + ( + "/.json", + "/-/.json", + "/-/versions.json", + "/-/settings.json", + "/-/config.json", + "/-/threads.json", + "/-/actor.json", + "/-/jump.json", + "/-/schema.json", + "/fixtures/-/schema.json", + "/fixtures/facetable/-/schema.json", + "/-/allowed.json?action=view-instance", + "/fixtures/facet_cities/-/autocomplete?_initial=1", + ), +) +async def test_success_object_has_ok_true(ds_client, path): + response = await ds_client.get(path) + assert response.status_code == 200 + data = response.json() + assert isinstance(data, dict) + assert data["ok"] is True + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "path", + ( + "/-/rules.json?action=view-instance", + "/-/check.json?action=view-instance", + ), +) +async def test_permission_debug_success_has_ok_true(ds_envelope, path): + response = await ds_envelope.client.get(path, actor={"id": "root"}) + assert response.status_code == 200 + data = response.json() + assert data["ok"] is True + + +@pytest.mark.asyncio +async def test_permissions_post_success_has_ok_true(ds_envelope): + response = await ds_envelope.client.post( + "/-/permissions", + data={"actor": '{"id": "root"}', "permission": "view-instance"}, + actor={"id": "root"}, + ) + assert response.status_code == 200 + assert response.json()["ok"] is True From b74a8e5b12b6e69a8717d0b0b7a37f12162f8ffc Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 13:42:24 +0000 Subject: [PATCH 05/46] Convert /-/plugins.json from top-level array to object /-/plugins.json now returns {"ok": true, "plugins": [...]} instead of a bare JSON array, so the response can grow additional keys without a breaking change. The `datasette plugins` CLI command still outputs a plain array. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/app.py | 5 ++++- docs/introspection.rst | 21 ++++++++++++--------- existing-api.md | 4 ++-- tests/test_api.py | 4 ++-- tests/test_config_dir.py | 7 ++++--- tests/test_plugins.py | 2 +- tests/test_success_envelope.py | 14 ++++++++++++++ 7 files changed, 39 insertions(+), 18 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index e13f0731..b554f9bc 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -2551,7 +2551,10 @@ class Datasette: ) add_route( JsonDataView.as_view( - self, "plugins.json", self._plugins, needs_request=True + self, + "plugins.json", + lambda request: {"plugins": self._plugins(request)}, + needs_request=True, ), r"/-/plugins(\.(?Pjson))?$", ) diff --git a/docs/introspection.rst b/docs/introspection.rst index d0780763..99238204 100644 --- a/docs/introspection.rst +++ b/docs/introspection.rst @@ -78,15 +78,18 @@ Shows a list of currently installed plugins and their versions. `Plugins example .. code-block:: json - [ - { - "name": "datasette_cluster_map", - "static": true, - "templates": false, - "version": "0.10", - "hooks": ["extra_css_urls", "extra_js_urls", "extra_body_script"] - } - ] + { + "ok": true, + "plugins": [ + { + "name": "datasette_cluster_map", + "static": true, + "templates": false, + "version": "0.10", + "hooks": ["extra_css_urls", "extra_js_urls", "extra_body_script"] + } + ] + } Add ``?all=1`` to include details of the default plugins baked into Datasette. diff --git a/existing-api.md b/existing-api.md index 0e7ec907..db613d30 100644 --- a/existing-api.md +++ b/existing-api.md @@ -228,8 +228,8 @@ app.py:2552-2557, `Datasette._plugins` (app.py:2247-2266). Permission - **Parameters:** `?all=1` — include Datasette's built-in default plugins (filtered out by default). -- **Response:** a JSON **array**, sorted by name, of - `{"name", "static", "templates", "version", "hooks"}`. +- **Response:** `{"ok": true, "plugins": [...]}` — each plugin is + `{"name", "static", "templates", "version", "hooks"}`, sorted by name. ### GET /-/settings(.json) diff --git a/tests/test_api.py b/tests/test_api.py index 4635b236..3583503c 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -541,13 +541,13 @@ async def test_plugins_json(ds_client): response = await ds_client.get("/-/plugins.json") # Filter out TrackEventPlugin actual_plugins = sorted( - [p for p in response.json() if p["name"] != "TrackEventPlugin"], + [p for p in response.json()["plugins"] if p["name"] != "TrackEventPlugin"], key=lambda p: p["name"], ) assert EXPECTED_PLUGINS == actual_plugins # Try with ?all=1 response = await ds_client.get("/-/plugins.json?all=1") - names = {p["name"] for p in response.json()} + names = {p["name"] for p in response.json()["plugins"]} assert names.issuperset(p["name"] for p in EXPECTED_PLUGINS) assert names.issuperset(DEFAULT_PLUGINS) diff --git a/tests/test_config_dir.py b/tests/test_config_dir.py index 0a9b30d8..74407e20 100644 --- a/tests/test_config_dir.py +++ b/tests/test_config_dir.py @@ -109,9 +109,10 @@ def test_settings(config_dir_client): def test_plugins(config_dir_client): response = config_dir_client.get("/-/plugins.json") assert 200 == response.status - assert "hooray.py" in {p["name"] for p in response.json} - assert "non_py_file.txt" not in {p["name"] for p in response.json} - assert "mypy_cache" not in {p["name"] for p in response.json} + plugins = response.json["plugins"] + assert "hooray.py" in {p["name"] for p in plugins} + assert "non_py_file.txt" not in {p["name"] for p in plugins} + assert "mypy_cache" not in {p["name"] for p in plugins} def test_templates_and_plugin(config_dir_client): diff --git a/tests/test_plugins.py b/tests/test_plugins.py index 59b1c0bf..5c4034db 100644 --- a/tests/test_plugins.py +++ b/tests/test_plugins.py @@ -1482,7 +1482,7 @@ async def test_plugin_is_installed(): datasette.pm.register(DummyPlugin(), name="DummyPlugin") response = await datasette.client.get("/-/plugins.json") assert response.status_code == 200 - installed_plugins = {p["name"] for p in response.json()} + installed_plugins = {p["name"] for p in response.json()["plugins"]} assert "DummyPlugin" in installed_plugins finally: diff --git a/tests/test_success_envelope.py b/tests/test_success_envelope.py index 48407660..cecc8d70 100644 --- a/tests/test_success_envelope.py +++ b/tests/test_success_envelope.py @@ -76,3 +76,17 @@ async def test_permissions_post_success_has_ok_true(ds_envelope): ) assert response.status_code == 200 assert response.json()["ok"] is True + + +@pytest.mark.asyncio +async def test_plugins_json_is_object(ds_client): + response = await ds_client.get("/-/plugins.json") + assert response.status_code == 200 + data = response.json() + assert set(data.keys()) == {"ok", "plugins"} + assert data["ok"] is True + assert isinstance(data["plugins"], list) + # ?all=1 should include Datasette's default plugins in the same shape + response_all = await ds_client.get("/-/plugins.json?all=1") + all_plugins = response_all.json()["plugins"] + assert len(all_plugins) > len(data["plugins"]) From 19e54b10d4d955e22346339a38f34190e3a1b36b Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 13:43:44 +0000 Subject: [PATCH 06/46] Convert /-/databases.json from top-level array to object /-/databases.json now returns {"ok": true, "databases": [...]} instead of a bare JSON array, so the response can grow additional keys without a breaking change. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/app.py | 6 +++++- docs/introspection.rst | 23 +++++++++++++---------- existing-api.md | 7 ++++--- tests/test_api.py | 6 ++++-- tests/test_cli.py | 6 +++--- tests/test_config_dir.py | 2 +- tests/test_internals_datasette.py | 2 +- tests/test_routes.py | 2 +- tests/test_success_envelope.py | 11 +++++++++++ 9 files changed, 43 insertions(+), 22 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index b554f9bc..bc35669f 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -2571,7 +2571,11 @@ class Datasette: r"/-/threads(\.(?Pjson))?$", ) add_route( - JsonDataView.as_view(self, "databases.json", self._connected_databases), + JsonDataView.as_view( + self, + "databases.json", + lambda: {"databases": self._connected_databases()}, + ), r"/-/databases(\.(?Pjson))?$", ) add_route( diff --git a/docs/introspection.rst b/docs/introspection.rst index 99238204..ab47c0a5 100644 --- a/docs/introspection.rst +++ b/docs/introspection.rst @@ -141,16 +141,19 @@ Shows currently attached databases. `Databases example len(data["plugins"]) + + +@pytest.mark.asyncio +async def test_databases_json_is_object(ds_client): + response = await ds_client.get("/-/databases.json") + assert response.status_code == 200 + data = response.json() + assert set(data.keys()) == {"ok", "databases"} + assert data["ok"] is True + assert isinstance(data["databases"], list) + assert "fixtures" in {db["name"] for db in data["databases"]} From 23ccdaeffc1d403563933af5fcfa90be6947f7f4 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 13:46:26 +0000 Subject: [PATCH 07/46] Convert /-/actions.json from top-level array to object /-/actions.json now returns {"ok": true, "actions": [...]} instead of a bare JSON array, so the response can grow additional keys without a breaking change. The debug_actions.html template reads data.actions, and the endpoint is now documented in docs/introspection.rst. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/app.py | 2 +- datasette/templates/debug_actions.html | 4 ++-- docs/introspection.rst | 24 ++++++++++++++++++++++++ existing-api.md | 12 ++++++------ stable-api-recommendations.md | 19 +++++++++++-------- tests/test_api.py | 2 +- tests/test_success_envelope.py | 11 +++++++++++ 7 files changed, 56 insertions(+), 18 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index bc35669f..5afca2a2 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -2588,7 +2588,7 @@ class Datasette: JsonDataView.as_view( self, "actions.json", - self._actions, + lambda: {"actions": self._actions()}, template="debug_actions.html", permission="permissions-debug", ), diff --git a/datasette/templates/debug_actions.html b/datasette/templates/debug_actions.html index 0ef7b329..c9dccaaa 100644 --- a/datasette/templates/debug_actions.html +++ b/datasette/templates/debug_actions.html @@ -9,7 +9,7 @@ {% include "_permissions_debug_tabs.html" %}

- This Datasette instance has registered {{ data|length }} action{{ data|length != 1 and "s" or "" }}. + This Datasette instance has registered {{ data.actions|length }} action{{ data.actions|length != 1 and "s" or "" }}. Actions are used by the permission system to control access to different features.

@@ -26,7 +26,7 @@ - {% for action in data %} + {% for action in data.actions %} diff --git a/docs/introspection.rst b/docs/introspection.rst index ab47c0a5..0010e8b7 100644 --- a/docs/introspection.rst +++ b/docs/introspection.rst @@ -155,6 +155,30 @@ Shows currently attached databases. `Databases example **Status:** recommendation 2 and 3 are implemented — every JSON-object +> **Status:** recommendations 1-3 are implemented. Every JSON-object > success response now includes `"ok": true` (`JsonDataView` injects it for > dict responses; homepage, jump, schema, permission-debug and autocomplete -> views set it explicitly; covered by `tests/test_success_envelope.py`). -> Recommendation 1 (wrapping the `/-/plugins`, `/-/databases`, `/-/actions` -> top-level arrays in objects) is being landed as separate per-endpoint -> commits. §2a/2b/2c remain open. +> views set it explicitly), and the three top-level-array endpoints now +> return objects: `/-/plugins` → `{"ok": true, "plugins": [...]}`, +> `/-/databases` → `{"ok": true, "databases": [...]}`, `/-/actions` → +> `{"ok": true, "actions": [...]}`. Covered by +> `tests/test_success_envelope.py`. The sub-findings §2a (collection +> representations), §2b (`_extra`/`_shape` coverage) and §2c (count +> truncation) remain open. Endpoints disagree about the success envelope: @@ -372,8 +375,8 @@ Two details make tiering urgent rather than optional: 2. `Forbidden` → JSON 403 for JSON requests (§1a). 3. No `ok: false` with HTTP 200 (§1b: `_shape=object`, write canned-query SQL errors). -4. Wrap `/-/plugins`, `/-/databases`, `/-/actions` top-level arrays in - objects (§2). +4. ~~Wrap `/-/plugins`, `/-/databases`, `/-/actions` top-level arrays in + objects (§2).~~ ✅ Done. 5. Filter `/-/databases.json` by `view-database` or gate it behind `permissions-debug` (§6). 6. 401 (not silent-anonymous) for invalid/expired bearer tokens (§1c). diff --git a/tests/test_api.py b/tests/test_api.py index 3263a88c..b035edb9 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -579,7 +579,7 @@ async def test_actions_json(ds_client): try: ds_client.ds.root_enabled = True response = await ds_client.get("/-/actions.json", actor={"id": "root"}) - data = response.json() + data = response.json()["actions"] finally: ds_client.ds.root_enabled = original_root_enabled assert isinstance(data, list) diff --git a/tests/test_success_envelope.py b/tests/test_success_envelope.py index 290a51e0..22ce6580 100644 --- a/tests/test_success_envelope.py +++ b/tests/test_success_envelope.py @@ -101,3 +101,14 @@ async def test_databases_json_is_object(ds_client): assert data["ok"] is True assert isinstance(data["databases"], list) assert "fixtures" in {db["name"] for db in data["databases"]} + + +@pytest.mark.asyncio +async def test_actions_json_is_object(ds_envelope): + response = await ds_envelope.client.get("/-/actions.json", actor={"id": "root"}) + assert response.status_code == 200 + data = response.json() + assert set(data.keys()) == {"ok", "actions"} + assert data["ok"] is True + assert isinstance(data["actions"], list) + assert "view-instance" in {action["name"] for action in data["actions"]} From f091b6dab165ec9f3a1d7689604abe97e102e60f Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 14:00:30 +0000 Subject: [PATCH 08/46] Filter /-/databases by view-database permission /-/databases previously listed every attached database (including filesystem paths and sizes) to any actor with view-instance, while the homepage and every other endpoint filtered by view-database. The endpoint now only lists databases the current actor is allowed to view. JsonDataView data callbacks may now be async. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/app.py | 15 ++++++++++++++- datasette/views/special.py | 4 ++-- docs/introspection.rst | 2 +- existing-api.md | 3 +-- stable-api-recommendations.md | 9 +++++---- tests/test_permissions.py | 32 ++++++++++++++++++++++++++++++++ 6 files changed, 55 insertions(+), 10 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 5afca2a2..dd5d7e8c 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -2168,6 +2168,18 @@ class Datasette: for name, d in self.databases.items() ] + async def _connected_databases_for_actor(self, actor): + page = await self.allowed_resources("view-database", actor) + allowed_names = {resource.parent async for resource in page.all()} + return [ + database + for database in self._connected_databases() + if database["name"] in allowed_names + ] + + async def _databases_data(self, request): + return {"databases": await self._connected_databases_for_actor(request.actor)} + def _versions(self): conn = sqlite3.connect(":memory:") self._prepare_connection(conn, "_memory") @@ -2574,7 +2586,8 @@ class Datasette: JsonDataView.as_view( self, "databases.json", - lambda: {"databases": self._connected_databases()}, + self._databases_data, + needs_request=True, ), r"/-/databases(\.(?Pjson))?$", ) diff --git a/datasette/views/special.py b/datasette/views/special.py index c289a240..82a76e76 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -53,9 +53,9 @@ class JsonDataView(BaseView): if self.permission: await self.ds.ensure_permission(action=self.permission, actor=request.actor) if self.needs_request: - data = self.data_callback(request) + data = await await_me_maybe(self.data_callback(request)) else: - data = self.data_callback() + data = await await_me_maybe(self.data_callback()) # Return JSON or HTML depending on format parameter as_format = request.url_vars.get("format") diff --git a/docs/introspection.rst b/docs/introspection.rst index 0010e8b7..37f258d7 100644 --- a/docs/introspection.rst +++ b/docs/introspection.rst @@ -137,7 +137,7 @@ Any keys that include the one of the following substrings in their names will be /-/databases ------------ -Shows currently attached databases. `Databases example `_: +Shows currently attached databases that the current actor is allowed to view, based on the ``view-database`` permission. `Databases example `_: .. code-block:: json diff --git a/existing-api.md b/existing-api.md index 5168819a..722331cd 100644 --- a/existing-api.md +++ b/existing-api.md @@ -262,8 +262,7 @@ Permission `view-instance`. No parameters. Response: `{"ok": true, "databases": [...]}` — each database is `{"name", "route", "path", "size", "is_mutable", "is_memory", "hash"}`. -**All attached databases are listed regardless of per-database -`view-database` permissions**. +Only databases the actor is allowed to `view-database` are listed. ### GET /-/actor(.json) diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index bee4b414..a8690f51 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -260,12 +260,13 @@ Concerns: ## 6. Permissions and security consistency (P1/P2) -- **(P1) `/-/databases.json` ignores per-database permissions** — it lists +- ~~**(P1) `/-/databases.json` ignores per-database permissions** — it lists every attached database (name, path on disk, size) to any actor holding `view-instance` (app.py:2157-2169), while the homepage and every other endpoint filter by `view-database`. On a public instance with private databases this leaks filesystem paths and database names. Filter it, or - gate it behind `permissions-debug`. + gate it behind `permissions-debug`.~~ ✅ **Done** — the endpoint now + filters through `allowed_resources("view-database", actor)`. - **(P2) `/db/-/schema` checks existence before permission** (views/special.py:1308-1317): an actor without `view-database` can distinguish "database exists" (403) from "does not exist" (404). @@ -377,8 +378,8 @@ Two details make tiering urgent rather than optional: SQL errors). 4. ~~Wrap `/-/plugins`, `/-/databases`, `/-/actions` top-level arrays in objects (§2).~~ ✅ Done. -5. Filter `/-/databases.json` by `view-database` or gate it behind - `permissions-debug` (§6). +5. ~~Filter `/-/databases.json` by `view-database` or gate it behind + `permissions-debug` (§6).~~ ✅ Done. 6. 401 (not silent-anonymous) for invalid/expired bearer tokens (§1c). 7. Publish explicit stability tiers, including extras and pagination-token opacity (§9). diff --git a/tests/test_permissions.py b/tests/test_permissions.py index 7d99213b..32606789 100644 --- a/tests/test_permissions.py +++ b/tests/test_permissions.py @@ -1800,3 +1800,35 @@ async def test_root_allow_block_with_table_restricted_actor(): actor=admin_actor, ) assert result is True + + +@pytest.mark.asyncio +async def test_databases_json_respects_view_database(tmp_path_factory): + # https://github.com/simonw/datasette - /-/databases should not list + # databases the actor is not allowed to view + db_directory = tmp_path_factory.mktemp("dbs") + from datasette.utils import sqlite3 as _sqlite3 + + paths = [] + for name in ("public", "private"): + path = str(db_directory / "{}.db".format(name)) + conn = _sqlite3.connect(path) + conn.execute("vacuum") + conn.close() + paths.append(path) + ds = Datasette( + paths, + config={"databases": {"private": {"allow": {"id": "root"}}}}, + ) + ds.root_enabled = True + await ds.invoke_startup() + try: + anon_response = await ds.client.get("/-/databases.json") + assert anon_response.status_code == 200 + anon_names = {db["name"] for db in anon_response.json()["databases"]} + assert anon_names == {"public"} + root_response = await ds.client.get("/-/databases.json", actor={"id": "root"}) + root_names = {db["name"] for db in root_response.json()["databases"]} + assert root_names == {"public", "private"} + finally: + ds.close() From ae10a99811111ff81413cc8da1210b6e824d54ff Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 14:09:01 +0000 Subject: [PATCH 09/46] Return canonical JSON error for Forbidden on JSON requests The default forbidden() hook previously rendered an HTML error page even for .json requests. It now returns the canonical JSON error shape with status 403 when the request path ends in .json or the request sends an Accept: application/json or Content-Type: application/json header. HTML requests still get the error page. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/forbidden.py | 10 +++++++ docs/json_api.rst | 5 ++++ docs/plugin_hooks.rst | 2 ++ existing-api.md | 14 +++++----- stable-api-recommendations.md | 14 ++++++---- tests/test_cli.py | 9 ++++--- tests/test_error_shape.py | 49 +++++++++++++++++++++++++++++++++++ 7 files changed, 88 insertions(+), 15 deletions(-) diff --git a/datasette/forbidden.py b/datasette/forbidden.py index 41c48396..3a81ac4f 100644 --- a/datasette/forbidden.py +++ b/datasette/forbidden.py @@ -1,9 +1,19 @@ from datasette import hookimpl, Response +from .utils import add_cors_headers, error_body @hookimpl(trylast=True) def forbidden(datasette, request, message): async def inner(): + if ( + request.path.split("?")[0].endswith(".json") + or "application/json" in (request.headers.get("accept") or "") + or request.headers.get("content-type") == "application/json" + ): + headers = {} + if datasette.cors: + add_cors_headers(headers) + return Response.json(error_body(message, 403), status=403, headers=headers) return Response.html( await datasette.render_template( "error.html", diff --git a/docs/json_api.rst b/docs/json_api.rst index 27d2d705..a561aa9c 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -81,6 +81,11 @@ Some endpoints add extra context keys. For example, a SQL error from a ``"rows"`` and ``"truncated"`` keys of the response it was unable to produce. +Permission errors use the same format: a request that fails a permission +check receives a ``403`` with this JSON error body when the URL ends in +``.json`` or the request sends an ``Accept: application/json`` or +``Content-Type: application/json`` header. + .. _json_api_custom_sql: Executing custom SQL diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index 81ef4acd..8614a15c 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -1685,6 +1685,8 @@ forbidden(datasette, request, message) Plugins can use this to customize how Datasette responds when a 403 Forbidden error occurs - usually because a page failed a permission check, see :ref:`authentication_permissions`. +Datasette's default behavior returns the :ref:`standard JSON error format ` with a 403 status when the request path ends in ``.json`` or the request has an ``Accept: application/json`` or ``Content-Type: application/json`` header; other requests get an HTML error page. + If a plugin hook wishes to react to the error, it should return a :ref:`Response object `. This example returns a redirect to a ``/-/login`` page: diff --git a/existing-api.md b/existing-api.md index 722331cd..90a931d1 100644 --- a/existing-api.md +++ b/existing-api.md @@ -98,13 +98,13 @@ Method-not-allowed responses return HTTP 405 with the canonical shape when the path ends in `.json` or the request content type is `application/json`; plain text otherwise (views/base.py). -**`Forbidden` is special:** when a view raises `Forbidden` (e.g. via -`ensure_permission`), the default `forbidden()` plugin hook renders an **HTML -error page with status 403 even for `.json` requests** -(forbidden.py:4-19, app.py:2895-2904). Endpoints that check permissions -themselves and return `_error(..., 403)` produce JSON instead. So a JSON -client may receive either an HTML 403 page or a JSON 403 body depending on -the endpoint. +**`Forbidden` handling:** when a view raises `Forbidden` (e.g. via +`ensure_permission`), the default `forbidden()` plugin hook returns the +canonical JSON error with status 403 when the path ends in `.json` or the +request carries an `Accept: application/json` / `Content-Type: +application/json` header; other requests get an HTML error page +(forbidden.py). Endpoints that check permissions themselves return +`_error(..., 403)` JSON directly. ### CORS diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index a8690f51..d68bd32e 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -26,9 +26,9 @@ Findings are grouped by theme. Each carries a priority: > The `title` key is no longer emitted in JSON; the bare `{"error": ...}` > debug-endpoint shape is gone; `_shape=object` misuse now returns HTTP 400 > (part of §1b). Covered by `tests/test_error_shape.py` and documented in -> the "Error responses" section of `docs/json_api.rst`. Still open from -> this section's sub-items: §1a (`Forbidden` → HTML), the write -> canned-query 200 (§1b), and the §1c status outliers. +> the "Error responses" section of `docs/json_api.rst`. §1a (`Forbidden` → +> JSON) is now also implemented. Still open from this section's sub-items: +> the write canned-query 200 (§1b) and the §1c status outliers. The API currently produces four distinct JSON error shapes depending on which internal layer generates the error: @@ -58,7 +58,11 @@ defaults). At minimum, eliminate the bare `{"error": ...}` shape and the `status`/`title` keys nobody else emits (`title` is a template-rendering concern that leaked into the API). -### 1a. `Forbidden` returns an HTML 403 to JSON clients (P1) +### 1a. `Forbidden` returns an HTML 403 to JSON clients (P1) — ✅ IMPLEMENTED + +> **Status:** implemented — the default `forbidden()` hook now returns the +> canonical JSON error for requests whose path ends in `.json` or that send +> `Accept: application/json` / `Content-Type: application/json`. Read endpoints that deny access via `ensure_permission`/`check_visibility` raise `Forbidden`, and the default `forbidden()` hook renders an **HTML error @@ -373,7 +377,7 @@ Two details make tiering urgent rather than optional: ## 10. Summary of P1 items (the pre-1.0 checklist) 1. ~~One canonical JSON error shape; retire the other three (§1).~~ ✅ Done. -2. `Forbidden` → JSON 403 for JSON requests (§1a). +2. ~~`Forbidden` → JSON 403 for JSON requests (§1a).~~ ✅ Done. 3. No `ok: false` with HTTP 200 (§1b: `_shape=object`, write canned-query SQL errors). 4. ~~Wrap `/-/plugins`, `/-/databases`, `/-/actions` top-level arrays in diff --git a/tests/test_cli.py b/tests/test_cli.py index 7b528a8e..cbd8edad 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -385,7 +385,9 @@ def test_setting_boolean_validation_false_values(value): ) # Should be forbidden (setting is false) assert result.exit_code == 1, result.output - assert "Forbidden" in result.output + error = json.loads(result.output) + assert error["ok"] is False + assert error["status"] == 403 @pytest.mark.parametrize("value", ("on", "true", "1")) @@ -425,8 +427,9 @@ def test_setting_default_allow_sql(default_allow_sql): assert json.loads(result.output)["rows"][0] == {"21": 21} else: assert result.exit_code == 1, result.output - # This isn't JSON at the moment, maybe it should be though - assert "Forbidden" in result.output + error = json.loads(result.output) + assert error["ok"] is False + assert error["status"] == 403 def test_sql_errors_logged_to_stderr(): diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index 6747e47a..7f12abc6 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -183,3 +183,52 @@ async def test_method_not_allowed_error_shape(ds_client): async def test_schema_unknown_database_error_shape(ds_client): response = await ds_client.get("/no_such_db/-/schema.json") assert_canonical_error(response, 404) + + +# Forbidden responses (the default forbidden() hook) + + +@pytest.fixture +def ds_forbidden(tmp_path_factory): + db_directory = tmp_path_factory.mktemp("dbs") + db_path = str(db_directory / "data.db") + conn = sqlite3.connect(db_path) + conn.execute("vacuum") + conn.execute("create table docs (id integer primary key, title text)") + conn.close() + ds = Datasette( + [db_path], + config={"databases": {"data": {"tables": {"docs": {"allow": {"id": "root"}}}}}}, + ) + ds.root_enabled = True + yield ds + ds.close() + + +@pytest.mark.asyncio +async def test_forbidden_json_path_returns_canonical_json(ds_forbidden): + response = await ds_forbidden.client.get("/data/docs.json") + data = assert_canonical_error(response, 403) + assert "permission" in data["error"].lower() + + +@pytest.mark.asyncio +async def test_forbidden_accept_json_returns_canonical_json(ds_forbidden): + response = await ds_forbidden.client.get( + "/data/docs", headers={"Accept": "application/json"} + ) + assert_canonical_error(response, 403) + + +@pytest.mark.asyncio +async def test_forbidden_html_path_still_returns_html(ds_forbidden): + response = await ds_forbidden.client.get("/data/docs") + assert response.status_code == 403 + assert response.headers["content-type"].startswith("text/html") + + +@pytest.mark.asyncio +async def test_forbidden_json_path_allowed_actor_still_works(ds_forbidden): + response = await ds_forbidden.client.get("/data/docs.json", actor={"id": "root"}) + assert response.status_code == 200 + assert response.json()["ok"] is True From e8048e023fa22971647ae38bfdf24b21b1853ffd Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 14:35:54 +0000 Subject: [PATCH 10/46] Return 400 for write canned-query SQL failures POST to a write canned query previously returned HTTP 200 with {"ok": false, "message": ...} when the SQL failed to execute, so JSON clients (and anything that trusts HTTP status) recorded success for failed writes. SQL failures now return 400 with the canonical error shape plus the "redirect" context key from on_error_redirect; the QueryWriteRejected 403 branch uses the canonical shape too. Successful executions and the HTML flash-message flow are unchanged. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/views/database.py | 22 +++--- docs/sql_queries.rst | 18 ++++- existing-api.md | 14 ++-- stable-api-recommendations.md | 15 ++-- tests/test_error_shape.py | 131 ++++++++++++++++++++++++++++++++++ tests/test_queries.py | 4 +- 6 files changed, 179 insertions(+), 25 deletions(-) diff --git a/datasette/views/database.py b/datasette/views/database.py index e02de657..cf7a6db3 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -16,6 +16,7 @@ from datasette.write_sql import QueryWriteRejected from datasette.utils import ( add_cors_headers, await_me_maybe, + error_body, call_with_supported_arguments, named_parameters as derive_named_parameters, format_bytes, @@ -607,11 +608,7 @@ class QueryView(View): "_json" ): return Response.json( - { - "ok": False, - "message": ex.message, - "redirect": None, - }, + dict(error_body([ex.message], 403), redirect=None), status=403, ) datasette.add_message(request, ex.message, datasette.ERROR) @@ -681,12 +678,17 @@ class QueryView(View): redirect_url = stored_query.on_error_redirect ok = False if should_return_json: + if ok: + return Response.json( + { + "ok": True, + "message": message, + "redirect": redirect_url, + } + ) return Response.json( - { - "ok": ok, - "message": message, - "redirect": redirect_url, - } + dict(error_body([message], 400), redirect=redirect_url), + status=400, ) else: datasette.add_message(request, message, message_type) diff --git a/docs/sql_queries.rst b/docs/sql_queries.rst index 371348fb..4c6e4426 100644 --- a/docs/sql_queries.rst +++ b/docs/sql_queries.rst @@ -657,7 +657,7 @@ There are three options for specifying that you would like the response to your - Include ``?_json=1`` in the URL that you POST to - Include ``"_json": 1`` in your JSON body, or ``&_json=1`` in your form encoded body -The JSON response will look like this: +A successful JSON response will look like this: .. code-block:: json @@ -667,7 +667,21 @@ The JSON response will look like this: "redirect": "/data/add_name" } -The ``"message"`` and ``"redirect"`` values here will take into account ``on_success_message``, ``on_success_message_sql``, ``on_success_redirect``, ``on_error_message`` and ``on_error_redirect``, if they have been set. +If the SQL fails to execute - for example a constraint violation - the response uses the :ref:`standard error format ` with a ``400`` status, plus the ``"redirect"`` key from the query configuration: + +.. code-block:: json + + { + "ok": false, + "error": "UNIQUE constraint failed: docs.id", + "errors": [ + "UNIQUE constraint failed: docs.id" + ], + "status": 400, + "redirect": null + } + +The ``"message"``, ``"error"`` and ``"redirect"`` values here take into account ``on_success_message``, ``on_success_message_sql``, ``on_success_redirect``, ``on_error_message`` and ``on_error_redirect``, if they have been set. .. _pagination: diff --git a/existing-api.md b/existing-api.md index 90a931d1..e0842804 100644 --- a/existing-api.md +++ b/existing-api.md @@ -1132,12 +1132,14 @@ queries. `_random_chars_`, `_cookie_`, `_header_` (underscores → hyphens). User-stored queries cannot contain magic parameters — they are a feature of config/trusted queries. -- **Response — 200 for both success and SQL failure** (only permission - rejection is 403): - `{"ok": true|false, "message": "...", "redirect": "..."|null}` — - `message` honors `on_success_message_sql` / `on_success_message` / - `on_error_message`, falling back to `"Query executed"` or - `"Query executed, N rows affected"`. +- **Response:** success → 200 + `{"ok": true, "message": "...", "redirect": "..."|null}` — `message` + honors `on_success_message_sql` / `on_success_message`, falling back to + `"Query executed"` or `"Query executed, N rows affected"`. SQL failure → + **400** canonical error (message honors `on_error_message`) plus a + `redirect` context key from `on_error_redirect`. Operation rejection + (`QueryWriteRejected`, e.g. VACUUM) → 403 canonical error plus + `redirect: null`. --- diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index d68bd32e..c878cb70 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -27,8 +27,8 @@ Findings are grouped by theme. Each carries a priority: > debug-endpoint shape is gone; `_shape=object` misuse now returns HTTP 400 > (part of §1b). Covered by `tests/test_error_shape.py` and documented in > the "Error responses" section of `docs/json_api.rst`. §1a (`Forbidden` → -> JSON) is now also implemented. Still open from this section's sub-items: -> the write canned-query 200 (§1b) and the §1c status outliers. +> JSON) and §1b (write canned-query 200) are now also implemented. Still +> open from this section's sub-items: the §1c status outliers. The API currently produces four distinct JSON error shapes depending on which internal layer generates the error: @@ -76,7 +76,12 @@ machine-readable answer. **Recommendation:** the default forbidden handler must return the canonical JSON error when the path ends in `.json` or the request prefers JSON, mirroring `handle_exception`. -### 1b. Errors that return HTTP 200 (P1) +### 1b. Errors that return HTTP 200 (P1) — ✅ IMPLEMENTED + +> **Status:** implemented. `_shape=object` misuse returns 400 (done with +> §1), and write canned-query SQL failures now return **400** with the +> canonical error shape (plus the `redirect` context key); the +> `QueryWriteRejected` 403 branch also uses the canonical shape. - `_shape=object` on a query or pk-less table → `{"ok": false, "error": "_shape=object is only available on tables"}` with **200** @@ -378,8 +383,8 @@ Two details make tiering urgent rather than optional: 1. ~~One canonical JSON error shape; retire the other three (§1).~~ ✅ Done. 2. ~~`Forbidden` → JSON 403 for JSON requests (§1a).~~ ✅ Done. -3. No `ok: false` with HTTP 200 (§1b: `_shape=object`, write canned-query - SQL errors). +3. ~~No `ok: false` with HTTP 200 (§1b: `_shape=object`, write canned-query + SQL errors).~~ ✅ Done. 4. ~~Wrap `/-/plugins`, `/-/databases`, `/-/actions` top-level arrays in objects (§2).~~ ✅ Done. 5. ~~Filter `/-/databases.json` by `view-database` or gate it behind diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index 7f12abc6..dd1fcfc9 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -232,3 +232,134 @@ async def test_forbidden_json_path_allowed_actor_still_works(ds_forbidden): response = await ds_forbidden.client.get("/data/docs.json", actor={"id": "root"}) assert response.status_code == 200 assert response.json()["ok"] is True + + +# Write canned queries: SQL failures must not return HTTP 200 + + +@pytest.fixture +def ds_write_query(tmp_path_factory): + db_directory = tmp_path_factory.mktemp("dbs") + db_path = str(db_directory / "data.db") + conn = sqlite3.connect(db_path) + conn.execute("vacuum") + conn.execute("create table docs (id integer primary key, title text)") + conn.close() + ds = Datasette( + [db_path], + config={ + "databases": { + "data": { + "queries": { + "add_doc": { + "sql": ( + "insert into docs (id, title)" " values (:id, :title)" + ), + "write": True, + }, + "add_doc_custom_error": { + "sql": ( + "insert into docs (id, title)" " values (:id, :title)" + ), + "write": True, + "on_error_message": "Custom error message", + "on_error_redirect": "/data", + }, + } + } + } + }, + ) + yield ds + ds.close() + + +@pytest.mark.asyncio +async def test_write_query_success_returns_200(ds_write_query): + response = await ds_write_query.client.post( + "/data/add_doc", + json={"id": 1, "title": "One"}, + headers={"Accept": "application/json"}, + ) + assert response.status_code == 200 + data = response.json() + assert data["ok"] is True + assert data["message"] == "Query executed, 1 row affected" + assert data["redirect"] is None + + +@pytest.mark.asyncio +async def test_write_query_sql_failure_returns_400(ds_write_query): + for _ in range(2): + response = await ds_write_query.client.post( + "/data/add_doc", + json={"id": 1, "title": "One"}, + headers={"Accept": "application/json"}, + ) + data = assert_canonical_error(response, 400) + assert "UNIQUE constraint failed" in data["error"] + # The redirect context key from the canned query flow is preserved + assert data["redirect"] is None + + +@pytest.mark.asyncio +async def test_write_query_failure_uses_on_error_message_and_redirect( + ds_write_query, +): + for _ in range(2): + response = await ds_write_query.client.post( + "/data/add_doc_custom_error", + json={"id": 1, "title": "One"}, + headers={"Accept": "application/json"}, + ) + data = assert_canonical_error(response, 400) + assert data["error"] == "Custom error message" + assert data["redirect"] == "/data" + + +@pytest.mark.asyncio +async def test_write_query_forbidden_is_canonical_403(ds_write_query): + # An untrusted write query run by an actor without execute-write-sql + # raises Forbidden, handled by the forbidden() hook + await ds_write_query.invoke_startup() + await ds_write_query.add_query( + "data", + name="untrusted_add", + sql="insert into docs (id, title) values (:id, :title)", + is_write=True, + is_trusted=False, + source="user", + owner_id="someone", + ) + response = await ds_write_query.client.post( + "/data/untrusted_add", + json={"id": 5, "title": "Five"}, + headers={"Accept": "application/json"}, + actor={"id": "someone"}, + ) + assert_canonical_error(response, 403) + + +@pytest.mark.asyncio +async def test_write_query_rejected_operation_is_canonical_403(ds_write_query): + # A rejected operation (VACUUM) raises QueryWriteRejected, handled by + # the dedicated branch in QueryView.post - root has execute-write-sql + ds_write_query.root_enabled = True + await ds_write_query.invoke_startup() + await ds_write_query.add_query( + "data", + name="vacuum_it", + sql="vacuum", + is_write=True, + is_trusted=False, + source="user", + owner_id="root", + ) + response = await ds_write_query.client.post( + "/data/vacuum_it", + json={}, + headers={"Accept": "application/json"}, + actor={"id": "root"}, + ) + data = assert_canonical_error(response, 403) + assert data["redirect"] is None diff --git a/tests/test_queries.py b/tests/test_queries.py index 6dfcc8b7..b79a9af4 100644 --- a/tests/test_queries.py +++ b/tests/test_queries.py @@ -3093,9 +3093,9 @@ async def test_untrusted_stored_write_query_rejects_virtual_table_control_insert ) assert denied_response.status_code == 403 - assert denied_response.json()["message"] == ( + assert denied_response.json()["errors"] == [ "Writes to virtual tables are not allowed in user-supplied SQL" - ) + ] assert ( await db.execute("select count(*) from docs where docs match 'hello'") ).first()[0] == 1 From b2cdc81d3495a497adf8a911d1171cf4609c9da4 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 14:55:41 +0000 Subject: [PATCH 11/46] Return 400 instead of 500 for row delete write failures Row delete previously returned 500 when the write failed (for example a constraint violation raised by a trigger or foreign key), while row update and every other write endpoint report the same failure class as 400. Delete now matches. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/views/row.py | 2 +- existing-api.md | 4 ++-- stable-api-recommendations.md | 8 +++++--- tests/test_error_shape.py | 32 ++++++++++++++++++++++++++++++++ 4 files changed, 40 insertions(+), 6 deletions(-) diff --git a/datasette/views/row.py b/datasette/views/row.py index c99b75d1..bdd78ed4 100644 --- a/datasette/views/row.py +++ b/datasette/views/row.py @@ -742,7 +742,7 @@ class RowDeleteView(BaseView): try: await resolved.db.execute_write_fn(delete_row, request=request) except Exception as e: - return _error([str(e)], 500) + return _error([str(e)], 400) await self.ds.track_event( DeleteRowEvent( diff --git a/existing-api.md b/existing-api.md index e0842804..d600179e 100644 --- a/existing-api.md +++ b/existing-api.md @@ -968,8 +968,8 @@ does not change the SQLite schema. - **Request:** no body required (any body is ignored — there is no confirmation step, unlike table drop). - **Response** — 200 `{"ok": true}`; with `?_redirect_to_table` a `redirect` - key is added. A failure during the write returns **500** with the message - (unlike update's 400). Emits `delete-row`. + key is added. A failure during the write returns 400 with the message, + matching update. Emits `delete-row`. --- diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index c878cb70..4fec5a00 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -98,10 +98,11 @@ key — see §2.) ### 1c. Wrong-status outliers (P2) -- Row **delete** write failures return **500** (views/row.py:757) while row +- ~~Row **delete** write failures return **500** (views/row.py:757) while row **update** write failures return **400** (views/row.py:832-835). Same failure class, different status; pick 400 (or 409 for constraint - violations) for both. + violations) for both.~~ ✅ **Done** — delete now returns 400, matching + update and the rest of the write API. - Invalid or expired bearer tokens silently degrade the request to anonymous, so clients see a 403 permission error (or worse, anonymous-permitted data) rather than a 401 (tokens.py:147-193). For 1.0, a malformed/expired @@ -337,7 +338,8 @@ Concerns: []}`** while `.csv` on the same request returns 400 `"?sql= is required"`. The JSON behavior masks caller bugs; return 400 on both. 3. **`_shape=object` HTTP 200 error** (§1b) — almost certainly unintended. -4. **Row delete 500** (§1c) — inconsistent with every sibling endpoint. +4. ~~**Row delete 500** (§1c) — inconsistent with every sibling endpoint.~~ + ✅ Done — now 400. 5. **The "SQL Interrupted" error embeds an HTML fragment in the JSON `error` value** (views/database.py:805-820). Error strings in the JSON API should be plain text. diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index dd1fcfc9..21d37cc2 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -363,3 +363,35 @@ async def test_write_query_rejected_operation_is_canonical_403(ds_write_query): ) data = assert_canonical_error(response, 403) assert data["redirect"] is None + + +# Row delete write failures must be 400, matching row update + + +@pytest.mark.asyncio +async def test_row_delete_write_failure_is_400(tmp_path_factory): + db_directory = tmp_path_factory.mktemp("dbs") + db_path = str(db_directory / "data.db") + conn = sqlite3.connect(db_path) + conn.execute("vacuum") + conn.execute("create table docs (id integer primary key, title text)") + conn.execute("insert into docs (id, title) values (1, 'One')") + conn.execute( + "create trigger no_delete before delete on docs " + "begin select raise(abort, 'deletes are blocked'); end" + ) + conn.commit() + conn.close() + ds = Datasette([db_path]) + ds.root_enabled = True + try: + response = await ds.client.post( + "/data/docs/1/-/delete", + json={}, + headers={"Content-Type": "application/json"}, + actor={"id": "root"}, + ) + data = assert_canonical_error(response, 400) + assert "deletes are blocked" in data["error"] + finally: + ds.close() From aaaffe45b851c974fde9ce894191192718f601e5 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 15:18:12 +0000 Subject: [PATCH 12/46] Return 401 for invalid or expired bearer tokens Invalid dstok_ tokens - bad signature, malformed payload, expired, or presented while allow_signed_tokens is off - previously degraded the request to anonymous, so clients saw a 403 permission error or worse, a 200 with anonymous-visible data. Token handlers can now raise TokenInvalid for tokens they recognize but reject; Datasette responds with 401, the canonical JSON error body and a WWW-Authenticate: Bearer error="invalid_token" header, even when a valid cookie is also present. Bearer tokens no registered handler recognizes are still ignored, so authentication plugins with their own token formats keep working. TokenInvalid is exported from the datasette package for use by plugin token handlers. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/__init__.py | 2 +- datasette/app.py | 31 +++++++++++- datasette/tokens.py | 44 +++++++++++++----- docs/authentication.rst | 4 +- docs/plugin_hooks.rst | 4 ++ existing-api.md | 17 ++++--- stable-api-recommendations.md | 13 ++++-- tests/test_api_write.py | 17 +++---- tests/test_auth.py | 19 ++++++-- tests/test_error_shape.py | 88 +++++++++++++++++++++++++++++++++++ tests/test_token_handler.py | 20 +++++--- 11 files changed, 214 insertions(+), 45 deletions(-) diff --git a/datasette/__init__.py b/datasette/__init__.py index eb18e59e..de46861c 100644 --- a/datasette/__init__.py +++ b/datasette/__init__.py @@ -1,7 +1,7 @@ from datasette.permissions import Permission # noqa from datasette.version import __version_info__, __version__ # noqa from datasette.events import Event # noqa -from datasette.tokens import TokenHandler, TokenRestrictions # noqa +from datasette.tokens import TokenHandler, TokenInvalid, TokenRestrictions # noqa from datasette.utils.asgi import Forbidden, NotFound, Request, Response # noqa from datasette.utils import actor_matches_allow # noqa from datasette.views import Context # noqa diff --git a/datasette/app.py b/datasette/app.py index dd5d7e8c..9982c58e 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -111,7 +111,9 @@ from .utils import ( baseconv, call_with_supported_arguments, detect_json1, + add_cors_headers, display_actor, + error_body, escape_css_string, escape_sqlite, find_spatialite, @@ -130,6 +132,7 @@ from .utils import ( redact_keys, row_sql_params_pks, ) +from .tokens import TokenInvalid from .utils.asgi import ( AsgiLifespan, Forbidden, @@ -905,7 +908,9 @@ class Datasette: Verify an API token by trying all registered token handlers. Returns an actor dict from the first handler that recognizes the - token, or None if no handler accepts it. + token, or None if no handler accepts it. A handler may raise + TokenInvalid for a token it recognizes but rejects (bad signature, + expired) - Datasette turns that into a 401 response. """ for token_handler in self._token_handlers(): result = await token_handler.verify_token(self, token) @@ -2887,13 +2892,24 @@ class DatasetteRouter: # Handle authentication default_actor = scope.get("actor") or None actor = None + token_error = None results = pm.hook.actor_from_request(datasette=self.ds, request=request) for result in results: - result = await await_me_maybe(result) + try: + result = await await_me_maybe(result) + except TokenInvalid as ex: + # A presented token was recognized but rejected - fail the + # request with a 401 even if another credential is valid, + # but keep awaiting the remaining coroutines first + if token_error is None: + token_error = ex + continue if result and actor is None: actor = result # Don't break — we must await all coroutines to avoid # "coroutine was never awaited" warnings + if token_error is not None: + return await self.handle_401(request, send, token_error) scope_modifications["actor"] = actor or default_actor scope = dict(scope, **scope_modifications) @@ -2925,6 +2941,17 @@ class DatasetteRouter: except Exception as exception: return await self.handle_exception(request, send, exception) + async def handle_401(self, request, send, exception): + # A presented bearer token was recognized by a handler but rejected. + # Bearer tokens are API credentials, so this is always JSON. + headers = {"www-authenticate": 'Bearer error="invalid_token"'} + if self.ds.cors: + add_cors_headers(headers) + response = Response.json( + error_body([str(exception)], 401), status=401, headers=headers + ) + await response.asgi_send(send) + async def handle_404(self, request, send, exception=None): # If path contains % encoding, redirect to tilde encoding if "%" in request.path: diff --git a/datasette/tokens.py b/datasette/tokens.py index 38a55529..4f905339 100644 --- a/datasette/tokens.py +++ b/datasette/tokens.py @@ -18,6 +18,21 @@ if TYPE_CHECKING: from datasette.app import Datasette +class TokenInvalid(Exception): + """ + Raised by a TokenHandler when a token it recognizes is invalid - + for example a bad signature, malformed payload or expired token. + + Datasette responds to this with an HTTP 401 error. Handlers should + return None instead for tokens they do not recognize at all, so that + other registered handlers get a chance to verify them. + """ + + def __init__(self, message="Invalid token"): + self.message = message + super().__init__(message) + + @dataclasses.dataclass class TokenRestrictions: """ @@ -108,8 +123,12 @@ class TokenHandler: async def verify_token(self, datasette: "Datasette", token: str) -> Optional[dict]: """ - Verify a token and return an actor dict, or None if this handler - does not recognize the token. + Verify a token and return an actor dict. + + Return None if this handler does not recognize the token at all, + so other handlers can try it. Raise TokenInvalid if the token is + recognized but invalid (bad signature, malformed, expired) - the + request will fail with a 401 error. """ raise NotImplementedError @@ -147,29 +166,32 @@ class SignedTokenHandler(TokenHandler): async def verify_token(self, datasette: "Datasette", token: str) -> Optional[dict]: prefix = "dstok_" - if not datasette.setting("allow_signed_tokens"): + if not token.startswith(prefix): + # Not one of our tokens - leave it for other handlers return None + if not datasette.setting("allow_signed_tokens"): + raise TokenInvalid( + "Signed tokens are not enabled for this Datasette instance" + ) + max_signed_tokens_ttl = datasette.setting("max_signed_tokens_ttl") - if not token.startswith(prefix): - return None - raw = token[len(prefix) :] try: decoded = datasette.unsign(raw, namespace="token") except itsdangerous.BadSignature: - return None + raise TokenInvalid("Invalid token signature") if "t" not in decoded: - return None + raise TokenInvalid("Invalid token: no timestamp") created = decoded["t"] if not isinstance(created, int): - return None + raise TokenInvalid("Invalid token: invalid timestamp") duration = decoded.get("d") if duration is not None and not isinstance(duration, int): - return None + raise TokenInvalid("Invalid token: invalid duration") if (duration is None and max_signed_tokens_ttl) or ( duration is not None @@ -180,7 +202,7 @@ class SignedTokenHandler(TokenHandler): if duration: if time.time() - created > duration: - return None + raise TokenInvalid("Token has expired") actor = {"id": decoded["a"], "token": "dstok"} diff --git a/docs/authentication.rst b/docs/authentication.rst index 8101699c..72ac5fa6 100644 --- a/docs/authentication.rst +++ b/docs/authentication.rst @@ -991,7 +991,9 @@ The ``/-/create-token`` page cannot be accessed by actors that are authenticated Datasette plugins that implement their own form of API token authentication should follow this convention. -You can disable the signed token feature entirely using the :ref:`allow_signed_tokens ` setting. +If a request presents a token that a token handler recognizes but rejects - an invalid signature, a malformed payload or an expired token - Datasette responds with a ``401`` status, the :ref:`standard JSON error format ` and a ``WWW-Authenticate: Bearer error="invalid_token"`` header. This means API clients can distinguish "your token needs to be renewed" (``401``) from "your token does not grant this permission" (``403``). A ``Bearer`` token that no registered handler recognizes at all is ignored, since it may be intended for an authentication plugin. + +You can disable the signed token feature entirely using the :ref:`allow_signed_tokens ` setting. Requests presenting a ``dstok_`` token while the feature is disabled receive a ``401``. .. _authentication_cli_create_token: diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index 8614a15c..049cb292 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -2546,6 +2546,10 @@ The default ``SignedTokenHandler`` uses itsdangerous signed tokens (``dstok_`` p async def verify_token(self, datasette, token): # Look up token in database, return actor dict or None + # if this handler does not recognize the token. Raise + # datasette.TokenInvalid for a token this handler + # recognizes but rejects (revoked, expired) - Datasette + # will respond with a 401 error. ... diff --git a/existing-api.md b/existing-api.md index d600179e..8c6e1c4e 100644 --- a/existing-api.md +++ b/existing-api.md @@ -1156,14 +1156,17 @@ registered via `register_token_handler`; the default is - **Format:** `dstok_` + itsdangerous-signed payload (namespace `token`) containing `a` (actor id), `t` (creation Unix time), optional `d` (duration seconds), optional `_r` (restrictions). -- **Verification** returns no actor when: `allow_signed_tokens` is off, the - signature is invalid, `t` is missing/non-integer, or the token is expired. - The effective duration is `d` capped by `max_signed_tokens_ttl` (default 0 - = no cap; a non-zero setting also imposes a TTL on tokens without `d`). +- **Verification:** a `dstok_`-prefixed token that fails verification — + `allow_signed_tokens` off, invalid signature, missing/non-integer `t`, + malformed `d`, or expired — raises `TokenInvalid`, and the request fails + with **401**, the canonical error body and a + `WWW-Authenticate: Bearer error="invalid_token"` header (even if a valid + `ds_actor` cookie is also present). Tokens with prefixes no registered + handler recognizes are ignored (they may belong to an auth plugin). The + effective duration is `d` capped by `max_signed_tokens_ttl` (default 0 = + no cap; a non-zero setting also imposes a TTL on tokens without `d`). - **Resulting actor:** `{"id": , "token": "dstok"}` plus `"_r"` and - `"token_expires"` when applicable. Invalid/expired tokens silently produce - an anonymous request (no 401) — the failure then surfaces as a 403 from - whatever permission check the request hits. + `"token_expires"` when applicable. **Restrictions (`_r`)** (default_permissions/restrictions.py): diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index 4fec5a00..273a272c 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -96,19 +96,23 @@ request prefers JSON, mirroring `handle_exception`. completed, SQL is invalid" is defensible but should then not reuse the `ok` key — see §2.) -### 1c. Wrong-status outliers (P2) +### 1c. Wrong-status outliers (P2) — ✅ IMPLEMENTED - ~~Row **delete** write failures return **500** (views/row.py:757) while row **update** write failures return **400** (views/row.py:832-835). Same failure class, different status; pick 400 (or 409 for constraint violations) for both.~~ ✅ **Done** — delete now returns 400, matching update and the rest of the write API. -- Invalid or expired bearer tokens silently degrade the request to anonymous, +- ~~Invalid or expired bearer tokens silently degrade the request to anonymous, so clients see a 403 permission error (or worse, anonymous-permitted data) rather than a 401 (tokens.py:147-193). For 1.0, a malformed/expired `Authorization: Bearer dstok_...` header should produce **401** with a distinguishable error, so clients can tell "renew your token" apart from - "you lack permission". + "you lack permission".~~ ✅ **Done** — token handlers can raise + `TokenInvalid`; Datasette responds 401 with the canonical body and a + `WWW-Authenticate: Bearer error="invalid_token"` header. Unrecognized + token prefixes still fall through to anonymous so auth plugins keep + working. --- @@ -391,7 +395,8 @@ Two details make tiering urgent rather than optional: objects (§2).~~ ✅ Done. 5. ~~Filter `/-/databases.json` by `view-database` or gate it behind `permissions-debug` (§6).~~ ✅ Done. -6. 401 (not silent-anonymous) for invalid/expired bearer tokens (§1c). +6. ~~401 (not silent-anonymous) for invalid/expired bearer tokens (§1c).~~ + ✅ Done. 7. Publish explicit stability tiers, including extras and pagination-token opacity (§9). 8. Resolve the looks-like-a-bug list (§8), especially trusted-query delete diff --git a/tests/test_api_write.py b/tests/test_api_write.py index 17542d4b..a29f5c99 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -202,8 +202,8 @@ async def test_insert_rows(ds_write, return_rows): "/data/docs/-/insert", {"rows": [{"title": "Test"} for i in range(10)]}, "bad_token", - 403, - ["Permission denied"], + 401, + ["Invalid token signature"], ), ( "/data/docs/-/insert", @@ -410,12 +410,13 @@ async def test_insert_or_upsert_row_errors( }, ) - actor_response = ( - await ds_write.client.get("/-/actor.json", headers=kwargs["headers"]) - ).json() - assert set((actor_response["actor"] or {}).get("_r", {}).get("a") or []) == set( - token_permissions - ) + if special_case != "bad_token": + actor_response = ( + await ds_write.client.get("/-/actor.json", headers=kwargs["headers"]) + ).json() + assert set((actor_response["actor"] or {}).get("_r", {}).get("a") or []) == set( + token_permissions + ) if special_case == "invalid_json": del kwargs["json"] diff --git a/tests/test_auth.py b/tests/test_auth.py index 8e83d397..d2913ecc 100644 --- a/tests/test_auth.py +++ b/tests/test_auth.py @@ -236,7 +236,9 @@ def test_auth_create_token( @pytest.mark.asyncio async def test_auth_create_token_not_allowed_for_tokens(ds_client): - ds_tok = ds_client.ds.sign({"a": "test", "token": "dstok"}, "token") + ds_tok = ds_client.ds.sign( + {"a": "test", "token": "dstok", "t": int(time.time())}, "token" + ) response = await ds_client.get( "/-/create-token", headers={"Authorization": "Bearer dstok_{}".format(ds_tok)}, @@ -304,8 +306,16 @@ async def test_auth_with_dstok_token(ds_client, scenario, should_work): assert actor["token"] == "dstok" if scenario != "valid_unlimited_token": assert isinstance(actor["token_expires"], int) - else: + elif scenario == "no_token": + # No credentials presented - request proceeds as anonymous assert response.json() == {"ok": True, "actor": None} + else: + # Invalid credentials presented - hard 401 + assert response.status_code == 401 + data = response.json() + assert data["ok"] is False + assert data["status"] == 401 + assert response.headers["www-authenticate"].startswith("Bearer") finally: ds_client.ds._settings["allow_signed_tokens"] = True @@ -339,8 +349,9 @@ def test_cli_create_token(app_client, expires): expected_actor["token_expires"] = details["t"] + expires assert response.json == {"ok": True, "actor": expected_actor} else: - expected_actor = None - assert response.json == {"ok": True, "actor": expected_actor} + # Expired token - hard 401 + assert response.status == 401 + assert response.json["ok"] is False @pytest.mark.asyncio diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index 21d37cc2..49b2c314 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -18,6 +18,7 @@ https://github.com/simonw/datasette/issues - 1.0 API consistency """ import pytest +import time from datasette.app import Datasette from datasette.utils import sqlite3 @@ -395,3 +396,90 @@ async def test_row_delete_write_failure_is_400(tmp_path_factory): assert "deletes are blocked" in data["error"] finally: ds.close() + + +# Invalid bearer tokens must produce 401, not silent anonymous access + + +@pytest.mark.asyncio +async def test_expired_token_returns_401(ds_error_shape): + token = "dstok_{}".format( + ds_error_shape.sign( + {"a": "root", "t": int(time.time()) - 2000, "d": 1000}, + namespace="token", + ) + ) + response = await ds_error_shape.client.get( + "/-/actor.json", headers={"Authorization": "Bearer {}".format(token)} + ) + data = assert_canonical_error(response, 401) + assert "expired" in data["error"].lower() + assert response.headers["www-authenticate"].startswith("Bearer") + + +@pytest.mark.asyncio +async def test_bad_signature_token_returns_401(ds_error_shape): + response = await ds_error_shape.client.get( + "/-/actor.json", headers={"Authorization": "Bearer dstok_garbage"} + ) + data = assert_canonical_error(response, 401) + assert response.headers["www-authenticate"].startswith("Bearer") + + +@pytest.mark.asyncio +async def test_unrecognized_token_prefix_stays_anonymous(ds_error_shape): + # No registered handler claims this token - it might belong to a + # plugin's actor_from_request hook, so it must not hard-fail + response = await ds_error_shape.client.get( + "/-/actor.json", headers={"Authorization": "Bearer sometoken_abc"} + ) + assert response.status_code == 200 + assert response.json() == {"ok": True, "actor": None} + + +@pytest.mark.asyncio +async def test_valid_token_still_authenticates(ds_error_shape): + token = "dstok_{}".format( + ds_error_shape.sign( + {"a": "root", "t": int(time.time())}, + namespace="token", + ) + ) + response = await ds_error_shape.client.get( + "/-/actor.json", headers={"Authorization": "Bearer {}".format(token)} + ) + assert response.status_code == 200 + assert response.json()["actor"]["id"] == "root" + + +@pytest.mark.asyncio +async def test_bad_token_beats_valid_cookie(ds_error_shape): + # A malformed Authorization header is a hard error even if a valid + # ds_actor cookie is also present + response = await ds_error_shape.client.get( + "/-/actor.json", + headers={"Authorization": "Bearer dstok_garbage"}, + cookies={"ds_actor": ds_error_shape.client.actor_cookie({"id": "root"})}, + ) + assert_canonical_error(response, 401) + + +@pytest.mark.asyncio +async def test_token_when_signed_tokens_disabled_returns_401(tmp_path_factory): + db_directory = tmp_path_factory.mktemp("dbs") + db_path = str(db_directory / "data.db") + conn = sqlite3.connect(db_path) + conn.execute("vacuum") + conn.close() + ds = Datasette([db_path], settings={"allow_signed_tokens": False}) + try: + token = "dstok_{}".format( + ds.sign({"a": "root", "t": int(time.time())}, namespace="token") + ) + response = await ds.client.get( + "/-/actor.json", headers={"Authorization": "Bearer {}".format(token)} + ) + data = assert_canonical_error(response, 401) + assert "not enabled" in data["error"] + finally: + ds.close() diff --git a/tests/test_token_handler.py b/tests/test_token_handler.py index 5c87f577..f5bbfead 100644 --- a/tests/test_token_handler.py +++ b/tests/test_token_handler.py @@ -5,7 +5,12 @@ Tests for the register_token_handler plugin hook. from datasette.app import Datasette from datasette.hookspecs import hookimpl from datasette.plugins import pm -from datasette.tokens import TokenHandler, TokenRestrictions, SignedTokenHandler +from datasette.tokens import ( + TokenHandler, + TokenInvalid, + TokenRestrictions, + SignedTokenHandler, +) import pytest @@ -66,10 +71,10 @@ async def test_verify_token_unknown_returns_none(datasette): @pytest.mark.asyncio -async def test_verify_token_bad_signature_returns_none(datasette): - """verify_token() should return None for tokens with bad signatures.""" - result = await datasette.verify_token("dstok_tampered_data_here") - assert result is None +async def test_verify_token_bad_signature_raises(datasette): + """verify_token() should raise TokenInvalid for tokens with bad signatures.""" + with pytest.raises(TokenInvalid): + await datasette.verify_token("dstok_tampered_data_here") @pytest.mark.asyncio @@ -334,5 +339,6 @@ async def test_signed_tokens_disabled(): ds = Datasette(settings={"allow_signed_tokens": False}) with pytest.raises(ValueError, match="Signed tokens are not enabled"): await ds.create_token("test_actor", handler="signed") - # verify_token should return None rather than raising - assert await ds.verify_token("dstok_anything") is None + # verify_token should raise TokenInvalid for a dstok_ token + with pytest.raises(TokenInvalid, match="not enabled"): + await ds.verify_token("dstok_anything") From ea9c1b1524279c03f0368c714438c5aacebcbec3 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 15:39:11 +0000 Subject: [PATCH 13/46] Return 400 for query data formats when ?sql= is missing GET /db/-/query.json with no (or blank) ?sql= previously returned 200 with empty rows, masking caller bugs, while the .csv format returned 400 "?sql= is required" for the same request. All data formats now return the 400; the HTML SQL editor page is unchanged. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/views/database.py | 2 ++ existing-api.md | 5 +++-- stable-api-recommendations.md | 6 ++++-- tests/test_error_shape.py | 24 ++++++++++++++++++++++++ tests/test_table_api.py | 6 ++++-- 5 files changed, 37 insertions(+), 6 deletions(-) diff --git a/datasette/views/database.py b/datasette/views/database.py index cf7a6db3..99e85d2b 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -851,6 +851,8 @@ class QueryView(View): return await stream_csv(datasette, fetch_data_for_csv, request, db.name) elif format_ in datasette.renderers.keys(): + if not sql: + raise DatasetteError("?sql= is required", status=400) data = {"ok": True, "rows": rows, "columns": columns} extras = extra_names_from_request(request) if extras: diff --git a/existing-api.md b/existing-api.md index 8c6e1c4e..9947edeb 100644 --- a/existing-api.md +++ b/existing-api.md @@ -486,8 +486,9 @@ queries section). HTTP 400 `{"ok": false, "error": "", "rows": [], "truncated": false}`. - Time limit → 400 titled `"SQL Interrupted"` (the `error` value contains an HTML fragment). - - `?sql=` omitted → 200 `{"ok": true, "rows": [], "truncated": false}` - (the CSV format instead errors 400 `"?sql= is required"`). + - `?sql=` omitted or blank → 400 `"?sql= is required"` for all data + formats (`.json`, `.csv`, plugin formats). The HTML page remains the + SQL editor. - `.csv` streams CSV; unknown extensions → 404. ### GET /\/-/query/parameters diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index 273a272c..11f0a75d 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -338,9 +338,11 @@ Concerns: `is_trusted` — an actor with `delete-query` can delete a config-defined trusted query via the API (it will resync on restart, making the behavior confusing rather than catastrophic). Align delete with update. -2. **GET `/db/-/query` with no `?sql=` returns 200 `{"ok": true, "rows": +2. ~~**GET `/db/-/query` with no `?sql=` returns 200 `{"ok": true, "rows": []}`** while `.csv` on the same request returns 400 `"?sql= is - required"`. The JSON behavior masks caller bugs; return 400 on both. + required"`. The JSON behavior masks caller bugs; return 400 on both.~~ + ✅ **Done** — all data formats now return 400; the HTML SQL editor page + is unchanged. 3. **`_shape=object` HTTP 200 error** (§1b) — almost certainly unintended. 4. ~~**Row delete 500** (§1c) — inconsistent with every sibling endpoint.~~ ✅ Done — now 400. diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index 49b2c314..9d4a566c 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -483,3 +483,27 @@ async def test_token_when_signed_tokens_disabled_returns_401(tmp_path_factory): assert "not enabled" in data["error"] finally: ds.close() + + +# GET /db/-/query without SQL: 400 for data formats, HTML editor stays 200 + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "path", + ( + "/fixtures/-/query.json", + "/fixtures/-/query.json?sql=", + ), +) +async def test_query_json_without_sql_is_400(ds_client, path): + response = await ds_client.get(path) + data = assert_canonical_error(response, 400) + assert data["errors"] == ["?sql= is required"] + + +@pytest.mark.asyncio +async def test_query_html_without_sql_is_still_the_editor(ds_client): + response = await ds_client.get("/fixtures/-/query") + assert response.status_code == 200 + assert response.headers["content-type"].startswith("text/html") diff --git a/tests/test_table_api.py b/tests/test_table_api.py index c8ba31b7..41c89f39 100644 --- a/tests/test_table_api.py +++ b/tests/test_table_api.py @@ -180,9 +180,11 @@ def test_query_extra_query_reports_bound_params(): assert response.json["query"]["params"] == {} -def test_query_extra_query_does_not_echo_querystring_without_sql(): +def test_query_extra_query_does_not_echo_querystring(): with make_app_client() as client: - response = client.get("/fixtures/-/query.json?_extra=query&foo=bar") + response = client.get( + "/fixtures/-/query.json?sql=select+1&_extra=query&foo=bar" + ) assert response.status == 200 assert response.json["query"]["params"] == {} From f3f5e891c9f3ad644a0b78c592ddd9d4f511d4b9 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 15:49:49 +0000 Subject: [PATCH 14/46] Block API deletion of trusted stored queries QueryUpdateView already rejected is_trusted queries but QueryDeleteView did not, so an actor with delete-query could delete a config-defined trusted query - which would then silently reappear on restart when the config re-syncs. Both the POST endpoint and the HTML confirmation page now return 403, matching update. datasette.remove_query() is unchanged for internal use. The docs already claimed this behavior ("Trusted stored queries cannot be edited or deleted through the web interface or the JSON API") - the code now matches them. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/views/stored_queries.py | 4 +++ existing-api.md | 5 ++-- stable-api-recommendations.md | 11 ++++--- tests/test_queries.py | 48 +++++++++++++++++++++++++++++++ 4 files changed, 62 insertions(+), 6 deletions(-) diff --git a/datasette/views/stored_queries.py b/datasette/views/stored_queries.py index 2753f876..d1f151dd 100644 --- a/datasette/views/stored_queries.py +++ b/datasette/views/stored_queries.py @@ -610,6 +610,8 @@ class QueryDeleteView(BaseView): resource=QueryResource(db.name, query_name), actor=request.actor, ) + if existing.is_trusted: + return _error(["Trusted queries cannot be deleted using the API"], 403) return await self.render( ["query_delete.html"], request, @@ -631,6 +633,8 @@ class QueryDeleteView(BaseView): actor=request.actor, ): return _error(["Permission denied: need delete-query"], 403) + if existing.is_trusted: + return _error(["Trusted queries cannot be deleted using the API"], 403) data, is_json = await _json_or_form_payload(request) await self.ds.remove_query(db.name, query_name) diff --git a/existing-api.md b/existing-api.md index 9947edeb..e11491fe 100644 --- a/existing-api.md +++ b/existing-api.md @@ -1093,8 +1093,9 @@ updates use `/-/update`. `QueryDeleteView` (app.py:2707-2710; views/stored_queries.py:594-644). GET renders an HTML confirmation page. -- **Permission:** `delete-query` (403 `need delete-query`). Unlike update, - **trusted queries are not blocked** from API deletion. +- **Permission:** `delete-query` (403 `need delete-query`). Trusted + queries → 403 `"Trusted queries cannot be deleted using the API"`, + matching update. - **Response:** JSON request → 200 `{"ok": true}`; form → 302; 404 `"Query not found: x"`. No `confirm` field required (unlike table drop). diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index 11f0a75d..8c3131e9 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -332,12 +332,15 @@ Concerns: ## 8. Behavior that looks like a bug and should be resolved before freezing -1. **Trusted queries: update is blocked, delete is not.** +1. ~~**Trusted queries: update is blocked, delete is not.** `QueryUpdateView` rejects `is_trusted` queries with 403 (stored_queries.py:426-427) but `QueryDeleteView.post` never checks `is_trusted` — an actor with `delete-query` can delete a config-defined trusted query via the API (it will resync on restart, making the - behavior confusing rather than catastrophic). Align delete with update. + behavior confusing rather than catastrophic). Align delete with update.~~ + ✅ **Done** — both the POST endpoint and the HTML confirmation page now + return 403 `"Trusted queries cannot be deleted using the API"`; + `datasette.remove_query()` remains available for internal use. 2. ~~**GET `/db/-/query` with no `?sql=` returns 200 `{"ok": true, "rows": []}`** while `.csv` on the same request returns 400 `"?sql= is required"`. The JSON behavior masks caller bugs; return 400 on both.~~ @@ -401,8 +404,8 @@ Two details make tiering urgent rather than optional: ✅ Done. 7. Publish explicit stability tiers, including extras and pagination-token opacity (§9). -8. Resolve the looks-like-a-bug list (§8), especially trusted-query delete - and row-delete 500. +8. Resolve the looks-like-a-bug list (§8), especially ~~trusted-query delete + and row-delete 500~~ (both done). Everything in P2 is worth doing now because each item is breaking-to-fix later; each P3 can be resolved by a sentence of documentation declaring the diff --git a/tests/test_queries.py b/tests/test_queries.py index b79a9af4..c5c1c3cd 100644 --- a/tests/test_queries.py +++ b/tests/test_queries.py @@ -3654,3 +3654,51 @@ async def test_stored_write_query_with_truncated_returning_message(): assert response.status_code == 200 assert response.json()["ok"] is True assert response.json()["message"] == "Query executed" + + +@pytest.mark.asyncio +async def test_query_delete_api_rejects_trusted_queries(): + ds = Datasette( + memory=True, + default_deny=True, + config={ + "databases": { + "data": { + "permissions": { + "view-query": {"id": "editor"}, + "delete-query": {"id": "editor"}, + }, + "queries": { + "trusted_report": { + "sql": "select 1 as one", + }, + }, + } + } + }, + ) + ds.add_memory_database("query_delete_trusted_api", name="data") + await ds.invoke_startup() + + response = await ds.client.post( + "/data/trusted_report/-/delete", + actor={"id": "editor"}, + json={}, + ) + assert response.status_code == 403 + assert response.json()["errors"] == [ + "Trusted queries cannot be deleted using the API" + ] + # The query must still exist + assert await ds.get_query("data", "trusted_report") is not None + + # The HTML confirmation page refuses too + get_response = await ds.client.get( + "/data/trusted_report/-/delete", + actor={"id": "editor"}, + ) + assert get_response.status_code == 403 + + # datasette.remove_query() remains available for internal use + await ds.remove_query("data", "trusted_report") + assert await ds.get_query("data", "trusted_report") is None From 6488b7a30e792c6b1af0a596ce81f7e1a0f179a3 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 16:00:25 +0000 Subject: [PATCH 15/46] Remove duplicate params key from stored query JSON objects Every stored-query object carried the same list of parameter names twice, as both "params" and "parameters". Output objects now carry only "parameters", consistent with /-/query/parameters and the two analyze endpoints (and distinct from the "params" bound-values dictionary used by the query extra and /-/execute-write). "params" remains an accepted input alias for query creation, update and datasette.yaml config. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/stored_queries.py | 1 - docs/json_api.rst | 1 - existing-api.md | 6 +++--- stable-api-recommendations.md | 7 +++++-- tests/test_queries.py | 30 ++++++++++++++++++++++++++++++ 5 files changed, 38 insertions(+), 7 deletions(-) diff --git a/datasette/stored_queries.py b/datasette/stored_queries.py index a6123daa..d7e1ec99 100644 --- a/datasette/stored_queries.py +++ b/datasette/stored_queries.py @@ -62,7 +62,6 @@ def stored_query_to_dict(query: StoredQuery) -> dict[str, Any]: "description_html": query.description_html, "hide_sql": query.hide_sql, "fragment": query.fragment, - "params": list(query.parameters), "parameters": list(query.parameters), "is_write": query.is_write, "is_private": query.is_private, diff --git a/docs/json_api.rst b/docs/json_api.rst index a561aa9c..54f0b6bd 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -1173,7 +1173,6 @@ The following extras are available for arbitrary SQL query responses and stored, "description_html": null, "hide_sql": false, "fragment": null, - "params": [], "parameters": [], "is_write": false, "is_private": false, diff --git a/existing-api.md b/existing-api.md index e11491fe..abb82e2c 100644 --- a/existing-api.md +++ b/existing-api.md @@ -990,7 +990,7 @@ stored_queries.py:55-80): "database": "...", "name": "...", "sql": "...", "title": null, "description": null, "description_html": null, "hide_sql": false, "fragment": null, - "params": ["p"], "parameters": ["p"], + "parameters": ["p"], "is_write": false, "is_private": true, "is_trusted": false, "source": "user", "owner_id": "...", "on_success_message": null, "on_success_message_sql": null, @@ -1000,8 +1000,8 @@ stored_queries.py:55-80): } ``` -`params` and `parameters` are identical lists, both always present. -`private` appears only in list responses. +`private` appears only in list responses. On input (create/update and +`datasette.yaml`), `params` is accepted as an alias for `parameters`. **Default permission rules for queries** (default_permissions/defaults.py): `view-query` is default-allow, but private queries are visible only to their diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index 8c3131e9..d1388549 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -249,9 +249,12 @@ Concerns: ## 5. Naming and parameter conventions (P2/P3) -- **`params` and `parameters` are duplicate keys** in every stored-query +- ~~**`params` and `parameters` are duplicate keys** in every stored-query object (stored_queries.py:55-80). Delete one before 1.0 (suggest keeping - `parameters`; the write side already accepts both on input). + `parameters`; the write side already accepts both on input).~~ + ✅ **Done** — output objects carry only `parameters` (matching + `/-/query/parameters` and the analyze endpoints); `params` remains an + accepted input alias for API creation and `datasette.yaml` config. - **Three names for the same concept across error/message payloads:** `error`, `errors`, `message`. See §1. - **Boolean query parameters have at least three grammars:** `_nl=on`, diff --git a/tests/test_queries.py b/tests/test_queries.py index c5c1c3cd..11828c4e 100644 --- a/tests/test_queries.py +++ b/tests/test_queries.py @@ -3702,3 +3702,33 @@ async def test_query_delete_api_rejects_trusted_queries(): # datasette.remove_query() remains available for internal use await ds.remove_query("data", "trusted_report") assert await ds.get_query("data", "trusted_report") is None + + +@pytest.mark.asyncio +async def test_stored_query_json_uses_parameters_not_params(): + ds = Datasette( + memory=True, + config={ + "databases": { + "data": { + "queries": { + "with_params": { + "sql": "select :name as name, :age as age", + "params": ["name", "age"], + }, + }, + } + } + }, + ) + ds.add_memory_database("query_parameters_key", name="data") + await ds.invoke_startup() + + definition = (await ds.client.get("/data/with_params/-/definition")).json() + assert definition["query"]["parameters"] == ["name", "age"] + assert "params" not in definition["query"] + + listing = (await ds.client.get("/data/-/queries.json")).json() + query = [q for q in listing["queries"] if q["name"] == "with_params"][0] + assert query["parameters"] == ["name", "age"] + assert "params" not in query From b09dceea889bf076c3d35af406c5b0ae50654731 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 16:03:05 +0000 Subject: [PATCH 16/46] Row update return:true responds with rows list, matching insert/upsert Row update previously returned a singular "row" object where insert and upsert return a "rows" list. All write endpoints now use "rows". Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/views/row.py | 2 +- docs/json_api.rst | 12 +++++++----- existing-api.md | 4 ++-- stable-api-recommendations.md | 5 +++-- tests/test_api_write.py | 4 ++-- tests/test_error_shape.py | 24 ++++++++++++++++++++++++ tests/test_table_html.py | 2 +- 7 files changed, 40 insertions(+), 13 deletions(-) diff --git a/datasette/views/row.py b/datasette/views/row.py index bdd78ed4..71539f34 100644 --- a/datasette/views/row.py +++ b/datasette/views/row.py @@ -828,7 +828,7 @@ class RowUpdateView(BaseView): resolved.sql, resolved.params, truncate=True ) returned_row = results.dicts()[0] - result["row"] = returned_row + result["rows"] = [returned_row] await self.ds.track_event( UpdateRowEvent( diff --git a/docs/json_api.rst b/docs/json_api.rst index 54f0b6bd..b8287a64 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -1955,11 +1955,13 @@ The returned JSON will look like this: { "ok": true, - "row": { - "id": 1, - "title": "New title", - "other_column": "Will be present here too" - } + "rows": [ + { + "id": 1, + "title": "New title", + "other_column": "Will be present here too" + } + ] } Any errors will use the :ref:`standard error format `, with a ``400`` status code for a bad input or a ``403`` status code for an authentication or permission error. diff --git a/existing-api.md b/existing-api.md index abb82e2c..eb7b5c20 100644 --- a/existing-api.md +++ b/existing-api.md @@ -958,8 +958,8 @@ does not change the SQLite schema. `"Invalid keys: ..."`; write failures (bad column, constraint violation) → 400 with the message. - **Response** — 200 `{"ok": true}`; with `return: true`, - `{"ok": true, "row": {...}}` (singular `row`, unlike insert/upsert's - `rows`). Emits `update-row`. + `{"ok": true, "rows": [{...}]}` — a single-item list, matching + insert/upsert. Emits `update-row`. ### POST /\/\/\/-/delete diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index d1388549..a4dbb344 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -160,10 +160,11 @@ Endpoints disagree about the success envelope: (index.py:147-161); `/-/databases.json` returns an **array**; the database page returns `tables` as an array. Choose arrays-of-objects everywhere (objects-keyed-by-name break when names need ordering or pagination). -- Insert/upsert with `return: true` respond with `rows` (plural, list); row +- ~~Insert/upsert with `return: true` respond with `rows` (plural, list); row update with `return: true` responds with `row` (singular, object) (views/row.py:837-844). Pick one (`rows` everywhere, even for one row, - matches the read API). + matches the read API).~~ ✅ **Done** — row update now returns + `rows: [{...}]`. ### 2b. `_extra`/`_shape` support is uneven (P2) diff --git a/tests/test_api_write.py b/tests/test_api_write.py index a29f5c99..e3363db0 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -1481,9 +1481,9 @@ async def test_update_row(ds_write, input, expected_errors, use_return): assert response.json()["ok"] is True if not use_return: - assert "row" not in response.json() + assert "rows" not in response.json() else: - returned_row = response.json()["row"] + returned_row = response.json()["rows"][0] assert returned_row["id"] == pk for k, v in input.items(): assert returned_row[k] == v diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index 9d4a566c..29f4e8fc 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -507,3 +507,27 @@ async def test_query_html_without_sql_is_still_the_editor(ds_client): response = await ds_client.get("/fixtures/-/query") assert response.status_code == 200 assert response.headers["content-type"].startswith("text/html") + + +# Write API return:true responses use "rows" consistently + + +@pytest.mark.asyncio +async def test_row_update_return_uses_rows_list(ds_error_shape): + await ds_error_shape.client.post( + "/data/docs/-/insert", + json={"row": {"id": 1, "title": "One"}}, + headers={"Content-Type": "application/json"}, + actor={"id": "root"}, + ) + response = await ds_error_shape.client.post( + "/data/docs/1/-/update", + json={"update": {"title": "Updated"}, "return": True}, + headers={"Content-Type": "application/json"}, + actor={"id": "root"}, + ) + assert response.status_code == 200 + data = response.json() + assert data["ok"] is True + assert "row" not in data + assert data["rows"] == [{"id": 1, "title": "Updated"}] diff --git a/tests/test_table_html.py b/tests/test_table_html.py index 46d43c6c..86fb4254 100644 --- a/tests/test_table_html.py +++ b/tests/test_table_html.py @@ -1629,7 +1629,7 @@ async def test_row_update_sets_message(): json={"update": {"name": long_name}, "return": True}, ) assert response.status_code == 200 - assert response.json()["row"]["name"] == long_name + assert response.json()["rows"][0]["name"] == long_name assert ds.unsign(response.cookies["ds_messages"], "messages") == [ ["Updated row 1 ({})".format(truncated_name), ds.INFO] ] From b958d03c0f0f322eeb4135e141e7877734211eef Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 16:09:40 +0000 Subject: [PATCH 17/46] Expose count truncation in table JSON via count_truncated extra The count extra is computed with a limit subquery, so a count equal to count_limit + 1 (default 10001) actually means "at least this many" - but only the HTML view knew that. A public count_truncated extra now reports the flag and is implicitly included whenever count is requested, using the same logic the HTML view already used. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/views/table.py | 23 ++++---------------- datasette/views/table_extras.py | 34 ++++++++++++++++++++++++++++++ docs/json_api.rst | 9 ++++++++ existing-api.md | 3 ++- stable-api-recommendations.md | 5 ++++- tests/test_table_api.py | 37 +++++++++++++++++++++++++++++++++ 6 files changed, 90 insertions(+), 21 deletions(-) diff --git a/datasette/views/table.py b/datasette/views/table.py index ef9831b2..44919281 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -62,6 +62,7 @@ from .table_create_alter import ( from .table_extras import ( TABLE_EXTRA_BUNDLES, TableExtraContext, + count_is_truncated, precompute_database_action_permissions, precompute_table_action_permissions, resolve_table_extras, @@ -2257,6 +2258,8 @@ async def table_view_data( extras.add("facet_results") if request.args.get("_shape") == "object": extras.add("primary_keys") + if "count" in extras: + extras.add("count_truncated") if extra_extras: extras.update(extra_extras) @@ -2335,7 +2338,7 @@ async def table_view_data( data["rows"] = transformed_rows if context_for_html_hack: - data["count_truncated"] = _count_truncated_for_table_page( + data["count_truncated"] = count_is_truncated( datasette, db, database_name, table_name, count_sql, data.get("count") ) data.update(extra_context_from_filters) @@ -2403,24 +2406,6 @@ async def table_view_data( return data, rows[:page_size], columns, expanded_columns, sql, next_url -def _count_truncated_for_table_page( - datasette, db, database_name, table_name, count_sql, count -): - if count != db.count_limit + 1: - return False - if ( - not db.is_mutable - and datasette.inspect_data - and count_sql == f"select count(*) from {table_name} " - ): - try: - datasette.inspect_data[database_name]["tables"][table_name]["count"] - return False - except KeyError: - pass - return True - - async def _next_value_and_url( datasette, db, diff --git a/datasette/views/table_extras.py b/datasette/views/table_extras.py index db659c80..f8ece55b 100644 --- a/datasette/views/table_extras.py +++ b/datasette/views/table_extras.py @@ -140,6 +140,39 @@ class CountExtra(Extra): return count +def count_is_truncated(datasette, db, database_name, table_name, count_sql, count): + if count != db.count_limit + 1: + return False + if ( + not db.is_mutable + and datasette.inspect_data + and count_sql == f"select count(*) from {table_name} " + ): + try: + datasette.inspect_data[database_name]["tables"][table_name]["count"] + return False + except KeyError: + pass + return True + + +class CountTruncatedExtra(Extra): + description = "True if the count hit Datasette's counting limit, meaning the real number of matching rows is at least the reported count." + example = ExtraExample("/fixtures/facetable.json?_extra=count,count_truncated") + scopes = {ExtraScope.TABLE} + expensive = True + + async def resolve(self, context, count): + return count_is_truncated( + context.datasette, + context.db, + context.database_name, + context.table_name, + context.count_sql, + count, + ) + + class FacetInstancesProvider(Provider): scopes = {ExtraScope.TABLE} @@ -1196,6 +1229,7 @@ TABLE_EXTRA_BUNDLES = { TABLE_EXTRA_CLASSES = [ CountExtra, + CountTruncatedExtra, CountSqlExtra, FacetResultsExtra, FacetsTimedOutExtra, diff --git a/docs/json_api.rst b/docs/json_api.rst index b8287a64..2a859ba2 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -302,6 +302,15 @@ The available table extras are listed below. 15 +``count_truncated`` + True if the count hit Datasette's counting limit, meaning the real number of matching rows is at least the reported count. (May execute additional queries.) + + ``GET /fixtures/facetable.json?_extra=count,count_truncated`` + + .. code-block:: json + + false + ``count_sql`` SQL query string used to calculate the total count for the current table view, including active filters. diff --git a/existing-api.md b/existing-api.md index eb7b5c20..a8a2237f 100644 --- a/existing-api.md +++ b/existing-api.md @@ -658,7 +658,8 @@ views/table_extras.py:1197-1235; unknown names silently ignored): | `_extra=` | Returns | |---|---| -| `count` | total matching-row count, computed with a `limit 10001` subquery so it caps at 10001; `null` with `_nocount` or on count timeout | +| `count` | total matching-row count, computed with a `limit 10001` subquery so it caps at 10001; `null` with `_nocount` or on count timeout. Requesting `count` implicitly includes `count_truncated` | +| `count_truncated` | `true` when `count` hit the counting limit (the real count is at least the reported value) | | `count_sql` | the SQL used for the count | | `facet_results` | `{"results": {name: facet}, "timed_out": [...]}`; each facet: `{name, type, hideable, toggle_url, results: [{value, label, count, toggle_url, selected}], truncated}` | | `facets_timed_out` | facet names that exceeded `facet_time_limit_ms` | diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index a4dbb344..64b2acbc 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -178,7 +178,10 @@ is a table/row/query feature. Also decide the contract for **unknown silent ignoring means typos return the default payload with no signal; recommend a 400 or a `warnings` key. -### 2c. Count truncation is invisible in JSON (P2) +### 2c. Count truncation is invisible in JSON (P2) — ✅ IMPLEMENTED + +> **Status:** implemented — a public `count_truncated` extra now exists and +> is implicitly included whenever `count` is requested. The `count` extra is computed with a `limit 10001` subquery, so `count: 10001` actually means "at least 10001" — the `count_truncated` flag exists diff --git a/tests/test_table_api.py b/tests/test_table_api.py index 41c89f39..e3737ee6 100644 --- a/tests/test_table_api.py +++ b/tests/test_table_api.py @@ -1504,6 +1504,7 @@ async def test_col_nocol_errors(ds_client, path, expected_error): "rows": [{"id": "1", "content": "hey", "content2": "world"}], "truncated": False, "count": 1, + "count_truncated": False, }, ), ), @@ -1590,3 +1591,39 @@ async def test_extra_render_cell(): finally: ds.pm.unregister(name="TestRenderCellPlugin") + + +@pytest.mark.asyncio +async def test_count_truncated_included_with_count_extra(tmp_path_factory): + from datasette.app import Datasette + from datasette.utils import sqlite3 + + db_directory = tmp_path_factory.mktemp("dbs") + db_path = str(db_directory / "counts.db") + conn = sqlite3.connect(db_path) + conn.execute("vacuum") + conn.execute("create table big (id integer primary key)") + conn.execute("create table small (id integer primary key)") + conn.executemany("insert into big (id) values (?)", [(i,) for i in range(10)]) + conn.executemany("insert into small (id) values (?)", [(i,) for i in range(3)]) + conn.commit() + conn.close() + ds = Datasette([db_path]) + ds.get_database("counts").count_limit = 5 + try: + response = await ds.client.get("/counts/big.json?_extra=count") + data = response.json() + # Count is capped at count_limit + 1 and flagged as truncated + assert data["count"] == 6 + assert data["count_truncated"] is True + + response = await ds.client.get("/counts/small.json?_extra=count") + data = response.json() + assert data["count"] == 3 + assert data["count_truncated"] is False + + # count_truncated can also be requested on its own + response = await ds.client.get("/counts/big.json?_extra=count_truncated") + assert response.json()["count_truncated"] is True + finally: + ds.close() From 9ee95cab3d9f451dd3529a83bd54a3998be436e0 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 16:11:30 +0000 Subject: [PATCH 18/46] Schema endpoints check permission before database existence /db/-/schema previously returned 404 for missing databases before checking view-database, letting unauthorized actors probe for database existence. The permission check now runs first, so actors without view-database get a uniform 403. The table schema endpoint also now returns 404 for an unknown database instead of an unhandled KeyError. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/views/special.py | 13 +++++++----- existing-api.md | 4 ++-- stable-api-recommendations.md | 6 ++++-- tests/test_error_shape.py | 39 +++++++++++++++++++++++++++++++++++ 4 files changed, 53 insertions(+), 9 deletions(-) diff --git a/datasette/views/special.py b/datasette/views/special.py index 82a76e76..6afb6437 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -1321,17 +1321,17 @@ class DatabaseSchemaView(SchemaBaseView): database_name = request.url_vars["database"] format_ = request.url_vars.get("format") or "html" - # Check if database exists - if database_name not in self.ds.databases: - return self.format_error_response("Database not found", format_) - - # Check view-database permission + # Permission check comes first, so actors without view-database + # cannot distinguish existing databases from missing ones await self.ds.ensure_permission( action="view-database", resource=DatabaseResource(database=database_name), actor=request.actor, ) + if database_name not in self.ds.databases: + return self.format_error_response("Database not found", format_) + schema = await self.get_database_schema(database_name) if format_ == "json": @@ -1365,6 +1365,9 @@ class TableSchemaView(SchemaBaseView): actor=request.actor, ) + if database_name not in self.ds.databases: + return self.format_error_response("Database not found", format_) + # Get schema for the table db = self.ds.databases[database_name] result = await db.execute( diff --git a/existing-api.md b/existing-api.md index a8a2237f..6ce29bf5 100644 --- a/existing-api.md +++ b/existing-api.md @@ -618,8 +618,8 @@ views/table_create_alter.py:965-1005). - **Permission:** `view-database` (denied → `Forbidden` → 403 HTML). - **Unknown database** → 404; for `.json`: - `{"ok": false, "error": "Database not found"}`. (The existence check runs - before the permission check.) + `{"ok": false, "error": "Database not found"}`. The permission check runs + first, so unauthorized actors cannot probe for database existence. - **Responses:** `.json` → 200 `{"ok": true, "database": "", "schema": ""}` (concatenated `sqlite_master.sql` joined with `;\n`); `.md` → `text/markdown`; no extension → HTML. diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index 64b2acbc..a92fbfbd 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -288,11 +288,13 @@ Concerns: databases this leaks filesystem paths and database names. Filter it, or gate it behind `permissions-debug`.~~ ✅ **Done** — the endpoint now filters through `allowed_resources("view-database", actor)`. -- **(P2) `/db/-/schema` checks existence before permission** +- ~~**(P2) `/db/-/schema` checks existence before permission** (views/special.py:1308-1317): an actor without `view-database` can distinguish "database exists" (403) from "does not exist" (404). Standardize on permission-check-first (as the table view does) so - unauthorized actors get a uniform response. + unauthorized actors get a uniform response.~~ ✅ **Done** — permission is + checked first; the table schema view also now 404s (instead of a 500 + KeyError) for an unknown database. - **(P2) `/-/threads` exposes runtime internals** (thread idents, asyncio task reprs including file paths) behind only `view-instance`. Consider `permissions-debug`, alongside `/-/actions` which already requires it. diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index 29f4e8fc..55370c18 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -531,3 +531,42 @@ async def test_row_update_return_uses_rows_list(ds_error_shape): assert data["ok"] is True assert "row" not in data assert data["rows"] == [{"id": 1, "title": "Updated"}] + + +# Schema endpoints: no existence oracle, no 500 on unknown database + + +@pytest.mark.asyncio +async def test_schema_endpoints_no_existence_oracle(tmp_path_factory): + db_directory = tmp_path_factory.mktemp("dbs") + db_path = str(db_directory / "data.db") + conn = sqlite3.connect(db_path) + conn.execute("vacuum") + conn.execute("create table docs (id integer primary key)") + conn.close() + ds = Datasette([db_path], default_deny=True) + ds.root_enabled = True + try: + # An actor without view-database cannot distinguish an existing + # database from a missing one + denied_existing = await ds.client.get("/data/-/schema.json") + denied_missing = await ds.client.get("/nope/-/schema.json") + assert denied_existing.status_code == denied_missing.status_code == 403 + + # An authorized actor sees the real thing + root_existing = await ds.client.get( + "/data/-/schema.json", actor={"id": "root"} + ) + assert root_existing.status_code == 200 + root_missing = await ds.client.get( + "/nope/-/schema.json", actor={"id": "root"} + ) + assert root_missing.status_code == 404 + finally: + ds.close() + + +@pytest.mark.asyncio +async def test_table_schema_unknown_database_is_404_not_500(ds_client): + response = await ds_client.get("/no_such_db/some_table/-/schema.json") + assert_canonical_error(response, 404) From e0ba8b3c6a3d7aa2e4be9866a3dce712196f9a73 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 16:16:02 +0000 Subject: [PATCH 19/46] Return 400 for unknown _extra names on data formats Unknown ?_extra= names (including internal HTML-only extras such as display_rows) were silently ignored, so a typo returned the default payload with no signal. Table, row and query data formats now return 400 "Unknown _extra: ". HTML pages continue to ignore unknown names. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/extras.py | 13 +++++++++++++ datasette/views/database.py | 3 ++- datasette/views/row.py | 3 +++ datasette/views/table.py | 4 ++++ docs/json_api.rst | 2 ++ existing-api.md | 7 ++++--- stable-api-recommendations.md | 6 +++++- tests/test_error_shape.py | 34 ++++++++++++++++++++++++++++++++++ tests/test_table_api.py | 4 ++-- 9 files changed, 69 insertions(+), 7 deletions(-) diff --git a/datasette/extras.py b/datasette/extras.py index 36014185..fb8c2e06 100644 --- a/datasette/extras.py +++ b/datasette/extras.py @@ -5,6 +5,8 @@ from typing import ClassVar from asyncinject import Registry +from datasette.utils.asgi import BadRequest + def extra_names_from_request(request): extra_bits = request.args.getlist("_extra") @@ -113,6 +115,17 @@ class ExtraRegistry: self._allowed_names[key] = names return names + def validate_requested(self, requested, scope): + """ + Raise BadRequest if any requested extra name is not a public extra + for this scope. Used by data formats such as .json - HTML pages + silently ignore unknown names instead. + """ + allowed = self._allowed_names_for_scope(scope, include_internal=False) + unknown = sorted(name for name in requested if name not in allowed) + if unknown: + raise BadRequest("Unknown _extra: {}".format(", ".join(unknown))) + async def resolve(self, requested, context, scope, include_internal=False): allowed_names = self._allowed_names_for_scope(scope, include_internal) requested_names = [name for name in requested if name in allowed_names] diff --git a/datasette/views/database.py b/datasette/views/database.py index 99e85d2b..b7131a05 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -8,7 +8,7 @@ import markupsafe import os import textwrap -from datasette.extras import extra_names_from_request +from datasette.extras import extra_names_from_request, ExtraScope from datasette.database import QueryInterrupted from datasette.resources import DatabaseResource, QueryResource from datasette.stored_queries import StoredQuery, stored_query_to_dict @@ -855,6 +855,7 @@ class QueryView(View): raise DatasetteError("?sql= is required", status=400) data = {"ok": True, "rows": rows, "columns": columns} extras = extra_names_from_request(request) + table_extra_registry.validate_requested(extras, ExtraScope.QUERY) if extras: query_extra_context = QueryExtraContext( datasette=datasette, diff --git a/datasette/views/row.py b/datasette/views/row.py index 71539f34..296a0ac1 100644 --- a/datasette/views/row.py +++ b/datasette/views/row.py @@ -596,6 +596,9 @@ class RowView(BaseView): } extras = extra_names_from_request(request) + if request.url_vars.get("format"): + # Data formats reject unknown extras; HTML ignores them + table_extra_registry.validate_requested(extras, ExtraScope.ROW) # Process extras row_extra_context = RowExtraContext( diff --git a/datasette/views/table.py b/datasette/views/table.py index 44919281..ed3b1276 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -2254,6 +2254,10 @@ async def table_view_data( # Resolve extras extras = extra_names_from_request(request) + if not extra_extras: + # Data formats reject unknown extras; the HTML path (which passes + # extra_extras={"_html"}) resolves internal extras of its own + table_extra_registry.validate_requested(extras, ExtraScope.TABLE) if any(k for k in request.args.keys() if k == "_facet" or k.startswith("_facet_")): extras.add("facet_results") if request.args.get("_shape") == "object": diff --git a/docs/json_api.rst b/docs/json_api.rst index 2a859ba2..0069055b 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -283,6 +283,8 @@ These can be repeated or comma-separated: ?_extra=columns&_extra=count,next_url +Requesting an ``_extra`` name that does not exist returns a ``400`` error in the :ref:`standard error format `, for example ``{"ok": false, "error": "Unknown _extra: nope", ...}``. + .. [[[cog from json_api_doc import table_extras table_extras(cog) diff --git a/existing-api.md b/existing-api.md index 6ce29bf5..cf456921 100644 --- a/existing-api.md +++ b/existing-api.md @@ -176,9 +176,10 @@ build JSON directly): Table, row and query JSON responses support `?_extra=` (repeatable and/or comma-separated, extras.py:9-14) to add keys to the response. Extras are scope-registered (`ExtraScope.TABLE` / `ROW` / `QUERY`) and only **public** -extras are available over JSON (extras.py:73-92). Unknown extra names are -silently ignored. The available names per scope are listed with the relevant -endpoints below. +extras are available over JSON (extras.py:73-92). Unknown extra names (and +internal HTML-only names) on data formats return 400 +`Unknown _extra: `; HTML pages ignore them. The available names per +scope are listed with the relevant endpoints below. --- diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index a92fbfbd..1bc2412e 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -166,7 +166,11 @@ Endpoints disagree about the success envelope: matches the read API).~~ ✅ **Done** — row update now returns `rows: [{...}]`. -### 2b. `_extra`/`_shape` support is uneven (P2) +### 2b. `_extra`/`_shape` support is uneven (P2) — partially implemented + +> **Status:** unknown `_extra` names on data formats now return 400 +> `Unknown _extra: ` (HTML pages still ignore them). Extending +> extras/shaping to database/instance scope remains open. The extras system (`?_extra=`, scope-registered) is the 1.0 mechanism for response shaping — but it only exists on table, row and query endpoints. The diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index 55370c18..f0ff0a42 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -570,3 +570,37 @@ async def test_schema_endpoints_no_existence_oracle(tmp_path_factory): async def test_table_schema_unknown_database_is_404_not_500(ds_client): response = await ds_client.get("/no_such_db/some_table/-/schema.json") assert_canonical_error(response, 404) + + +# Unknown _extra names are a 400, not silently ignored + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "path", + ( + "/fixtures/facetable.json?_extra=nope", + "/fixtures/facetable.json?_extra=count,nope", + "/fixtures/simple_primary_key/1.json?_extra=nope", + "/fixtures/-/query.json?sql=select+1&_extra=nope", + ), +) +async def test_unknown_extra_is_400(ds_client, path): + response = await ds_client.get(path) + data = assert_canonical_error(response, 400) + assert data["errors"] == ["Unknown _extra: nope"] + + +@pytest.mark.asyncio +async def test_html_only_extra_via_json_is_400(ds_client): + # display_rows exists for the HTML view but is not part of the JSON API + response = await ds_client.get("/fixtures/facetable.json?_extra=display_rows") + data = assert_canonical_error(response, 400) + assert data["errors"] == ["Unknown _extra: display_rows"] + + +@pytest.mark.asyncio +async def test_unknown_extra_ignored_on_html_pages(ds_client): + response = await ds_client.get("/fixtures/facetable?_extra=nope") + assert response.status_code == 200 + assert response.headers["content-type"].startswith("text/html") diff --git a/tests/test_table_api.py b/tests/test_table_api.py index e3737ee6..ff413be8 100644 --- a/tests/test_table_api.py +++ b/tests/test_table_api.py @@ -123,8 +123,8 @@ async def test_html_only_extras_are_not_available_via_json(ds_client, extra): # These extras exist for the HTML view; their values are not JSON # serializable so they are internal, not part of the JSON API response = await ds_client.get(f"/fixtures/facetable.json?_extra={extra}") - assert response.status_code == 200 - assert extra not in response.json() + assert response.status_code == 400 + assert response.json()["errors"] == [f"Unknown _extra: {extra}"] @pytest.mark.asyncio From 0bf3a54716a18521332aa55dac6a3e420851c82a Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 16:19:03 +0000 Subject: [PATCH 20/46] Include next_url in default table JSON keys Table JSON responses previously only included the next pagination token by default - the ready-to-follow next_url required ?_extra=next_url. Both keys are now always present (null on the final page), which the pagination documentation already claimed. The next_url extra remains valid for backwards compatibility. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/views/table.py | 1 + docs/json_api.rst | 3 ++- existing-api.md | 1 + stable-api-recommendations.md | 9 ++++++++- tests/test_table_api.py | 21 +++++++++++++++++++++ 5 files changed, 33 insertions(+), 2 deletions(-) diff --git a/datasette/views/table.py b/datasette/views/table.py index ed3b1276..69d5a075 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -2318,6 +2318,7 @@ async def table_view_data( data = { "ok": True, "next": next_value and str(next_value) or None, + "next_url": next_url, } data.update( await resolve_table_extras( diff --git a/docs/json_api.rst b/docs/json_api.rst index 0069055b..dc611a39 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -48,7 +48,7 @@ The ``"rows"`` key is a list of objects, each one representing a row. The ``"truncated"`` key lets you know if the query was truncated. This can happen if a SQL query returns more than 1,000 results (or the :ref:`setting_max_returned_rows` setting). -For table pages, an additional key ``"next"`` may be present. This indicates that the next page in the pagination set can be retrieved using ``?_next=VALUE``. +For table pages, two additional keys are present: ``"next"``, an opaque token that can be used to retrieve the next page using ``?_next=TOKEN``, and ``"next_url"``, the full URL of that next page. Both are ``null`` on the final page. See :ref:`json_api_pagination`. .. _json_api_errors: @@ -128,6 +128,7 @@ options: { "ok": true, "next": null, + "next_url": null, "rows": [ [3, "Detroit"], [2, "Los Angeles"], diff --git a/existing-api.md b/existing-api.md index cf456921..f7f161f3 100644 --- a/existing-api.md +++ b/existing-api.md @@ -647,6 +647,7 @@ dispatched to `QueryView` (views/table.py:1703-1712). |---|---| | `ok` | `true` when data was retrieved without error | | `next` | pagination token string, or `null` on the last page | +| `next_url` | absolute URL of the next page, or `null` on the last page | | `rows` | list of row objects `{column: value}` (default `_shape=objects`) | | `truncated` | always present; `false` for table pages | diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index 1bc2412e..d3d5b6a6 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -196,7 +196,14 @@ number. --- -## 3. Pagination: three mechanisms, two contracts (P2) +## 3. Pagination: three mechanisms, two contracts (P2) — partially implemented + +> **Status:** `next_url` now accompanies `next` in the default table JSON +> keys (previously it required `?_extra=next_url`), so every response with +> a `next` token also carries the ready-to-follow URL. Pagination tokens +> are deliberately left undocumented as to their internal structure. The +> `_size`/`page_size` naming and `has_more`/`total` differences remain +> open. | Endpoint | Mechanism | Token | Extras | |---|---|---|---| diff --git a/tests/test_table_api.py b/tests/test_table_api.py index ff413be8..0a593ff8 100644 --- a/tests/test_table_api.py +++ b/tests/test_table_api.py @@ -1491,6 +1491,7 @@ async def test_col_nocol_errors(ds_client, path, expected_error): { "ok": True, "next": None, + "next_url": None, "columns": ["id", "content", "content2"], "rows": [{"id": "1", "content": "hey", "content2": "world"}], "truncated": False, @@ -1501,6 +1502,7 @@ async def test_col_nocol_errors(ds_client, path, expected_error): { "ok": True, "next": None, + "next_url": None, "rows": [{"id": "1", "content": "hey", "content2": "world"}], "truncated": False, "count": 1, @@ -1627,3 +1629,22 @@ async def test_count_truncated_included_with_count_extra(tmp_path_factory): assert response.json()["count_truncated"] is True finally: ds.close() + + +@pytest.mark.asyncio +async def test_next_url_included_by_default(ds_client): + response = await ds_client.get("/fixtures/compound_three_primary_keys.json") + data = response.json() + assert data["next"] is not None + assert data["next_url"].endswith( + "/fixtures/compound_three_primary_keys.json?_next=" + + urllib.parse.quote(data["next"], safe="") + ) + # Follow to the last page - next and next_url are both null there + while data["next"]: + response = await ds_client.get( + "/fixtures/compound_three_primary_keys.json?_next=" + data["next"] + ) + data = response.json() + assert data["next"] is None + assert data["next_url"] is None From e5e9aca871bed96398d4702b7ae25abcfbb96ac8 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 16:22:31 +0000 Subject: [PATCH 21/46] Require permissions-debug for /-/threads /-/threads exposes runtime internals - thread idents and asyncio task reprs including file paths - but only required view-instance. It now requires permissions-debug, like /-/actions. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/app.py | 4 +++- docs/introspection.rst | 2 +- existing-api.md | 2 +- stable-api-recommendations.md | 5 +++-- tests/test_api.py | 6 +++++- tests/test_error_shape.py | 20 ++++++++++++++------ tests/test_internals_datasette.py | 3 ++- tests/test_success_envelope.py | 2 +- 8 files changed, 30 insertions(+), 14 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 9982c58e..daa3848b 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -2584,7 +2584,9 @@ class Datasette: r"/-/config(\.(?Pjson))?$", ) add_route( - JsonDataView.as_view(self, "threads.json", self._threads), + JsonDataView.as_view( + self, "threads.json", self._threads, permission="permissions-debug" + ), r"/-/threads(\.(?Pjson))?$", ) add_route( diff --git a/docs/introspection.rst b/docs/introspection.rst index 37f258d7..2d13f68b 100644 --- a/docs/introspection.rst +++ b/docs/introspection.rst @@ -252,7 +252,7 @@ Without those query string arguments, the page lists up to five tables with dete /-/threads ---------- -Shows details of threads and ``asyncio`` tasks. `Threads example `_: +Shows details of threads and ``asyncio`` tasks. This endpoint requires the ``permissions-debug`` permission, since it exposes runtime internals. `Threads example `_: .. code-block:: json diff --git a/existing-api.md b/existing-api.md index f7f161f3..af7ca3b9 100644 --- a/existing-api.md +++ b/existing-api.md @@ -249,7 +249,7 @@ value replaced by `"***"` (utils/__init__.py:1532-1556). ### GET /-/threads(.json) app.py:2566-2569, `Datasette._threads` (app.py:2268-2285). Permission -`view-instance`. No parameters. +**`permissions-debug`** (exposes runtime internals). No parameters. Response: `num_threads`, `threads` (list of `{name, ident, daemon}`), `num_tasks`, `tasks` (asyncio task repr strings). When the diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index d3d5b6a6..3c216c04 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -306,9 +306,10 @@ Concerns: unauthorized actors get a uniform response.~~ ✅ **Done** — permission is checked first; the table schema view also now 404s (instead of a 500 KeyError) for an unknown database. -- **(P2) `/-/threads` exposes runtime internals** (thread idents, asyncio +- ~~**(P2) `/-/threads` exposes runtime internals** (thread idents, asyncio task reprs including file paths) behind only `view-instance`. Consider - `permissions-debug`, alongside `/-/actions` which already requires it. + `permissions-debug`, alongside `/-/actions` which already requires it.~~ + ✅ **Done** — `/-/threads` now requires `permissions-debug`. - **(P3) `/-/config` redaction is substring-based** on six key names (app.py:2502-2505); plugins storing secrets under other names leak. Worth a note in plugin authoring docs plus a `redact_keys` plugin hook. diff --git a/tests/test_api.py b/tests/test_api.py index b035edb9..7a575144 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -525,7 +525,11 @@ def test_databases_json(app_client_two_attached_databases_one_immutable): @pytest.mark.asyncio async def test_threads_json(ds_client): - response = await ds_client.get("/-/threads.json") + ds_client.ds.root_enabled = True + try: + response = await ds_client.get("/-/threads.json", actor={"id": "root"}) + finally: + ds_client.ds.root_enabled = False expected_keys = {"ok", "threads", "num_threads"} if sys.version_info >= (3, 7, 0): expected_keys.update({"tasks", "num_tasks"}) diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index f0ff0a42..a34fbed7 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -554,13 +554,9 @@ async def test_schema_endpoints_no_existence_oracle(tmp_path_factory): assert denied_existing.status_code == denied_missing.status_code == 403 # An authorized actor sees the real thing - root_existing = await ds.client.get( - "/data/-/schema.json", actor={"id": "root"} - ) + root_existing = await ds.client.get("/data/-/schema.json", actor={"id": "root"}) assert root_existing.status_code == 200 - root_missing = await ds.client.get( - "/nope/-/schema.json", actor={"id": "root"} - ) + root_missing = await ds.client.get("/nope/-/schema.json", actor={"id": "root"}) assert root_missing.status_code == 404 finally: ds.close() @@ -604,3 +600,15 @@ async def test_unknown_extra_ignored_on_html_pages(ds_client): response = await ds_client.get("/fixtures/facetable?_extra=nope") assert response.status_code == 200 assert response.headers["content-type"].startswith("text/html") + + +# /-/threads exposes runtime internals and requires permissions-debug + + +@pytest.mark.asyncio +async def test_threads_requires_permissions_debug(ds_error_shape): + denied = await ds_error_shape.client.get("/-/threads.json") + assert_canonical_error(denied, 403) + allowed = await ds_error_shape.client.get("/-/threads.json", actor={"id": "root"}) + assert allowed.status_code == 200 + assert allowed.json()["ok"] is True diff --git a/tests/test_internals_datasette.py b/tests/test_internals_datasette.py index 2af069c9..85598c05 100644 --- a/tests/test_internals_datasette.py +++ b/tests/test_internals_datasette.py @@ -184,10 +184,11 @@ async def test_datasette_constructor(): @pytest.mark.asyncio async def test_num_sql_threads_zero(): ds = Datasette([], memory=True, settings={"num_sql_threads": 0}) + ds.root_enabled = True db = ds.add_database(Database(ds, memory_name="test_num_sql_threads_zero")) await db.execute_write("create table t(id integer primary key)") await db.execute_write("insert into t (id) values (1)") - response = await ds.client.get("/-/threads.json") + response = await ds.client.get("/-/threads.json", actor={"id": "root"}) assert response.json() == {"ok": True, "num_threads": 0, "threads": []} response2 = await ds.client.get("/test_num_sql_threads_zero/t.json?_shape=array") assert response2.json() == [{"id": 1}] diff --git a/tests/test_success_envelope.py b/tests/test_success_envelope.py index 22ce6580..97cfd72a 100644 --- a/tests/test_success_envelope.py +++ b/tests/test_success_envelope.py @@ -34,7 +34,6 @@ def ds_envelope(tmp_path_factory): "/-/versions.json", "/-/settings.json", "/-/config.json", - "/-/threads.json", "/-/actor.json", "/-/jump.json", "/-/schema.json", @@ -58,6 +57,7 @@ async def test_success_object_has_ok_true(ds_client, path): ( "/-/rules.json?action=view-instance", "/-/check.json?action=view-instance", + "/-/threads.json", ), ) async def test_permission_debug_success_has_ok_true(ds_envelope, path): From 13dc7a08b79c7b62e6eb2c54befa5e6c02915bd3 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 16:30:04 +0000 Subject: [PATCH 22/46] Fix foreign key label test expectations for default next_url key Follow-up to the commit adding next_url to the default table JSON keys. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- tests/test_table_html.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/test_table_html.py b/tests/test_table_html.py index 86fb4254..87c48b6b 100644 --- a/tests/test_table_html.py +++ b/tests/test_table_html.py @@ -2313,6 +2313,7 @@ async def test_foreign_key_labels_obey_permissions(config): assert root_b.json() == { "ok": True, "next": None, + "next_url": None, "rows": [{"id": 1, "name": "world", "a_id": {"value": 1, "label": "hello"}}], "truncated": False, } @@ -2320,6 +2321,7 @@ async def test_foreign_key_labels_obey_permissions(config): assert anon_b.json() == { "ok": True, "next": None, + "next_url": None, "rows": [{"id": 1, "name": "world", "a_id": 1}], "truncated": False, } From 5cb2bc6909be9a9f9cbd23e7f19f142091cbb5af Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 16:45:19 +0000 Subject: [PATCH 23/46] Homepage JSON returns databases as a list /.json previously returned databases as an object keyed by database name, unlike /-/databases.json and every other collection in the API. It now returns a list of database objects. The HTML template already consumed the list. This endpoint remains undocumented. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/views/index.py | 2 +- existing-api.md | 4 ++-- stable-api-recommendations.md | 6 ++++-- tests/test_api.py | 18 ++++++++++-------- 4 files changed, 17 insertions(+), 13 deletions(-) diff --git a/datasette/views/index.py b/datasette/views/index.py index 86f8ae7d..8c5534cb 100644 --- a/datasette/views/index.py +++ b/datasette/views/index.py @@ -152,7 +152,7 @@ class IndexView(BaseView): json.dumps( { "ok": True, - "databases": {db["name"]: db for db in databases}, + "databases": databases, "metadata": await self.ds.get_instance_metadata(), }, cls=CustomJSONEncoder, diff --git a/existing-api.md b/existing-api.md index af7ca3b9..8d390f76 100644 --- a/existing-api.md +++ b/existing-api.md @@ -201,8 +201,8 @@ Routes: `/(\.(?Pjsono?))?$` and `/-/(\.(?Pjsono?))?$` - **Parameters:** `_sort=relationships` sorts each database's truncated table list by foreign-key relationship count. - **JSON response** (index.py:147-161) — includes `ok: true` plus: - - `databases` — an **object keyed by database name** (not a list). Each - value: `name`, `hash` (or null), `color`, `path`, + - `databases` — a **list** of database objects (undocumented API, subject + to change). Each item: `name`, `hash` (or null), `color`, `path`, `tables_and_views_truncated` (up to 5 items: `name`, `columns`, `primary_keys`, `count` (int or null), `hidden`, `fts_table`, `num_relationships_for_sorting`, `private`; view items are just diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index 3c216c04..d64b3fb8 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -156,10 +156,12 @@ Endpoints disagree about the success envelope: ### 2a. Collection representations disagree (P2) -- Homepage `/.json` returns `databases` as an **object keyed by name** +- ~~Homepage `/.json` returns `databases` as an **object keyed by name** (index.py:147-161); `/-/databases.json` returns an **array**; the database page returns `tables` as an array. Choose arrays-of-objects everywhere - (objects-keyed-by-name break when names need ordering or pagination). + (objects-keyed-by-name break when names need ordering or pagination).~~ + ✅ **Done** — the homepage returns a list, matching `/-/databases.json`. + The homepage JSON remains deliberately undocumented. - ~~Insert/upsert with `return: true` respond with `rows` (plural, list); row update with `return: true` responds with `row` (singular, object) (views/row.py:837-844). Pick one (`rows` everywhere, even for one row, diff --git a/tests/test_api.py b/tests/test_api.py index 7a575144..17edc0e5 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -26,8 +26,9 @@ async def test_homepage(ds_client): "title", ] databases = data.get("databases") - assert databases.keys() == {"fixtures": 0}.keys() - d = databases["fixtures"] + assert isinstance(databases, list) + assert [d["name"] for d in databases] == ["fixtures"] + d = databases[0] assert d["name"] == "fixtures" assert isinstance(d["tables_count"], int) assert isinstance(len(d["tables_and_views_truncated"]), int) @@ -43,7 +44,7 @@ async def test_homepage_sort_by_relationships(ds_client): assert response.status_code == 200 tables = [ t["name"] - for t in response.json()["databases"]["fixtures"]["tables_and_views_truncated"] + for t in response.json()["databases"][0]["tables_and_views_truncated"] ] assert tables == [ "simple_primary_key", @@ -251,8 +252,8 @@ def test_no_files_uses_memory_database(app_client_no_files): assert response.status == 200 assert { "ok": True, - "databases": { - "_memory": { + "databases": [ + { "name": "_memory", "hash": None, "color": "a6c7b9", @@ -267,7 +268,7 @@ def test_no_files_uses_memory_database(app_client_no_files): "views_count": 0, "private": False, }, - }, + ], "metadata": {}, } == response.json # Try that SQL query @@ -852,8 +853,9 @@ async def test_tilde_encoded_database_names(db_name): ds = Datasette() ds.add_memory_database(db_name) response = await ds.client.get("/.json") - assert db_name in response.json()["databases"].keys() - path = response.json()["databases"][db_name]["path"] + databases_by_name = {d["name"]: d for d in response.json()["databases"]} + assert db_name in databases_by_name + path = databases_by_name[db_name]["path"] # And the JSON for that database response2 = await ds.client.get(path + ".json") assert response2.status_code == 200 From afa7b1ba0db8e5084889a15261559bcf21435171 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 17:01:53 +0000 Subject: [PATCH 24/46] Add unstable marker to undocumented JSON endpoints JSON endpoints that are not part of the documented API now include "unstable": "This API is not part of Datasette's stable interface and may change at any time" in their responses, making the stability tier machine-readable. Applied to the homepage (/.json and /-/.json), /db/-/queries/analyze, POST /db/-/queries/store, /db//-/definition, /db/-/query/parameters, /db/-/execute-write/analyze and the POST /-/permissions playground response. The message lives in datasette.utils.UNSTABLE_API_MESSAGE. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/utils/__init__.py | 5 +++ datasette/views/execute_write.py | 10 ++--- datasette/views/index.py | 2 + datasette/views/special.py | 7 ++++ datasette/views/stored_queries.py | 35 +++++++++++++----- existing-api.md | 14 ++++++- stable-api-recommendations.md | 8 +++- tests/test_api.py | 7 +++- tests/test_permissions.py | 4 ++ tests/test_queries.py | 9 ++++- tests/test_success_envelope.py | 61 +++++++++++++++++++++++++++++++ 11 files changed, 142 insertions(+), 20 deletions(-) diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index 17c8702f..1d921f02 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -1294,6 +1294,11 @@ async def derive_named_parameters(db: "Database", sql: str) -> List[str]: return named_parameters(sql) +UNSTABLE_API_MESSAGE = ( + "This API is not part of Datasette's stable interface and may change at any time" +) + + def error_body(messages, status): """ The canonical JSON error body used by every Datasette JSON error response: diff --git a/datasette/views/execute_write.py b/datasette/views/execute_write.py index b7e8288e..6806e69d 100644 --- a/datasette/views/execute_write.py +++ b/datasette/views/execute_write.py @@ -2,7 +2,7 @@ import re from urllib.parse import urlencode from datasette.resources import DatabaseResource -from datasette.utils import sqlite3 +from datasette.utils import UNSTABLE_API_MESSAGE, sqlite3 from datasette.utils.asgi import Response from .base import BaseView, _error @@ -500,8 +500,6 @@ class ExecuteWriteAnalyzeView(BaseView): ) ) sql = request.args.get("sql") or "" - return _block_framing( - Response.json( - await _execute_write_analysis_data(self.ds, db, sql, request.actor) - ) - ) + analysis = await _execute_write_analysis_data(self.ds, db, sql, request.actor) + analysis["unstable"] = UNSTABLE_API_MESSAGE + return _block_framing(Response.json(analysis)) diff --git a/datasette/views/index.py b/datasette/views/index.py index 8c5534cb..67296cd1 100644 --- a/datasette/views/index.py +++ b/datasette/views/index.py @@ -6,6 +6,7 @@ from datasette.utils import ( await_me_maybe, make_slot_function, CustomJSONEncoder, + UNSTABLE_API_MESSAGE, ) from datasette.utils.asgi import Response from datasette.version import __version__ @@ -152,6 +153,7 @@ class IndexView(BaseView): json.dumps( { "ok": True, + "unstable": UNSTABLE_API_MESSAGE, "databases": databases, "metadata": await self.ds.get_instance_metadata(), }, diff --git a/datasette/views/special.py b/datasette/views/special.py index 6afb6437..28d4d6a1 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -6,6 +6,7 @@ from datasette.events import LogoutEvent, LoginEvent, CreateTokenEvent from datasette.resources import DatabaseResource, TableResource from datasette.utils.asgi import Response, Forbidden from datasette.utils import ( + UNSTABLE_API_MESSAGE, actor_matches_allow, add_cors_headers, await_me_maybe, @@ -295,6 +296,12 @@ class PermissionsDebugView(BaseView): response, status = await _check_permission_for_actor( self.ds, permission, parent, child, actor ) + if response.get("ok"): + response = { + "ok": True, + "unstable": UNSTABLE_API_MESSAGE, + **response, + } return Response.json(response, status=status) diff --git a/datasette/views/stored_queries.py b/datasette/views/stored_queries.py index d1f151dd..ea49d983 100644 --- a/datasette/views/stored_queries.py +++ b/datasette/views/stored_queries.py @@ -2,7 +2,7 @@ from urllib.parse import parse_qsl, urlencode from datasette.resources import DatabaseResource, QueryResource from datasette.stored_queries import stored_query_to_dict -from datasette.utils import sqlite3, tilde_decode +from datasette.utils import UNSTABLE_API_MESSAGE, sqlite3, tilde_decode from datasette.utils.asgi import Response from .base import BaseView, _error @@ -48,7 +48,15 @@ class QueryParametersView(BaseView): parameters = _derived_query_parameters(request.args.get("sql") or "") except QueryValidationError as ex: return _block_framing(_error([ex.message], ex.status)) - return _block_framing(Response.json({"ok": True, "parameters": parameters})) + return _block_framing( + Response.json( + { + "ok": True, + "unstable": UNSTABLE_API_MESSAGE, + "parameters": parameters, + } + ) + ) def _query_list_url(path, query_string, *, set_args=None, remove_args=None): @@ -315,11 +323,9 @@ class QueryCreateAnalyzeView(BaseView): ) ) sql = request.args.get("sql") or "" - return _block_framing( - Response.json( - await _query_create_analysis_data(self.ds, db, sql, request.actor) - ) - ) + analysis = await _query_create_analysis_data(self.ds, db, sql, request.actor) + analysis["unstable"] = UNSTABLE_API_MESSAGE + return _block_framing(Response.json(analysis)) class QueryStoreView(QueryCreateView): @@ -384,7 +390,12 @@ class QueryStoreView(QueryCreateView): assert query is not None if is_json: return Response.json( - {"ok": True, "query": stored_query_to_dict(query)}, status=201 + { + "ok": True, + "unstable": UNSTABLE_API_MESSAGE, + "query": stored_query_to_dict(query), + }, + status=201, ) self.ds.add_message(request, "Query saved", self.ds.INFO) return Response.redirect(self.ds.urls.path(self.ds.urls.table(db.name, name))) @@ -405,7 +416,13 @@ class QueryDefinitionView(BaseView): actor=request.actor, ): return _error(["Permission denied"], 403) - return Response.json({"ok": True, "query": stored_query_to_dict(query)}) + return Response.json( + { + "ok": True, + "unstable": UNSTABLE_API_MESSAGE, + "query": stored_query_to_dict(query), + } + ) class QueryUpdateView(BaseView): diff --git a/existing-api.md b/existing-api.md index 8d390f76..2f9996dc 100644 --- a/existing-api.md +++ b/existing-api.md @@ -43,7 +43,7 @@ directory: every claim below is based on the route table in `datasette/app.py` - Success content type: `application/json; charset=utf-8` (`_shape=array&_nl=on` responses use `text/plain`). -### Success envelope +### Success envelope and stability marker Every JSON endpoint that returns an object includes `"ok": true` on success. `JsonDataView` injects it automatically for dict responses @@ -52,6 +52,18 @@ autocomplete views add it explicitly. The former top-level-array endpoints (`/-/plugins`, `/-/databases`, `/-/actions`) now return objects wrapping their arrays (`{"ok": true, "plugins": [...]}` etc.). +JSON endpoints that are **not part of the documented API** include a +marker key (`UNSTABLE_API_MESSAGE` in utils/__init__.py): + +```json +"unstable": "This API is not part of Datasette's stable interface and may change at any time" +``` + +Currently: the homepage (`/.json`, `/-/.json`), `/db/-/queries/analyze`, +`POST /db/-/queries/store`, `/db//-/definition`, +`/db/-/query/parameters`, `/db/-/execute-write/analyze` and the +`POST /-/permissions` playground response. + ### Error shape (canonical) Every JSON error response uses one canonical shape, built by `error_body()` diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index d64b3fb8..baac80f7 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -378,7 +378,13 @@ Concerns: --- -## 9. Define stability tiers explicitly (P1 — documentation, not code) +## 9. Define stability tiers explicitly (P1 — documentation, not code) — partially implemented + +> **Status:** undocumented JSON endpoints now self-describe with an +> `"unstable": "This API is not part of Datasette's stable interface and +> may change at any time"` key (homepage, queries analyze/store/definition, +> query parameters, execute-write analyze, permissions playground POST). +> The written tier documentation remains to be done. Not everything under `/-/` can or should carry a 1.0 guarantee. Recommend shipping 1.0 with an explicit three-tier contract, per endpoint: diff --git a/tests/test_api.py b/tests/test_api.py index 17edc0e5..8ce38fad 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -43,8 +43,7 @@ async def test_homepage_sort_by_relationships(ds_client): response = await ds_client.get("/.json?_sort=relationships") assert response.status_code == 200 tables = [ - t["name"] - for t in response.json()["databases"][0]["tables_and_views_truncated"] + t["name"] for t in response.json()["databases"][0]["tables_and_views_truncated"] ] assert tables == [ "simple_primary_key", @@ -252,6 +251,10 @@ def test_no_files_uses_memory_database(app_client_no_files): assert response.status == 200 assert { "ok": True, + "unstable": ( + "This API is not part of Datasette's stable interface" + " and may change at any time" + ), "databases": [ { "name": "_memory", diff --git a/tests/test_permissions.py b/tests/test_permissions.py index 32606789..f8d2c808 100644 --- a/tests/test_permissions.py +++ b/tests/test_permissions.py @@ -740,6 +740,10 @@ async def test_actor_restricted_permissions( } expected = { "ok": True, + "unstable": ( + "This API is not part of Datasette's stable interface" + " and may change at any time" + ), "action": permission, "allowed": expected_result, "resource": expected_resource, diff --git a/tests/test_queries.py b/tests/test_queries.py index 11828c4e..7f29a3fb 100644 --- a/tests/test_queries.py +++ b/tests/test_queries.py @@ -2140,7 +2140,14 @@ async def test_query_parameters_endpoint_uses_get_sql_only(): ) assert response.status_code == 200 - assert response.json() == {"ok": True, "parameters": ["name", "id"]} + assert response.json() == { + "ok": True, + "unstable": "{}".format( + "This API is not part of Datasette's stable interface" + " and may change at any time" + ), + "parameters": ["name", "id"], + } assert permission_denied_response.status_code == 403 assert permission_denied_response.json()["errors"] == [ "Permission denied: need execute-sql" diff --git a/tests/test_success_envelope.py b/tests/test_success_envelope.py index 97cfd72a..68b042e5 100644 --- a/tests/test_success_envelope.py +++ b/tests/test_success_envelope.py @@ -112,3 +112,64 @@ async def test_actions_json_is_object(ds_envelope): assert data["ok"] is True assert isinstance(data["actions"], list) assert "view-instance" in {action["name"] for action in data["actions"]} + + +UNSTABLE_MESSAGE = ( + "This API is not part of Datasette's stable interface and may change at any time" +) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "path", + ( + "/.json", + "/-/.json", + "/fixtures/-/queries/analyze?sql=select+1", + "/fixtures/-/query/parameters?sql=select+:name", + "/fixtures/-/execute-write/analyze?sql=delete+from+facetable", + ), +) +async def test_undocumented_endpoints_report_unstable(ds_client, path): + ds_client.ds.root_enabled = True + try: + response = await ds_client.get(path, actor={"id": "root"}) + finally: + ds_client.ds.root_enabled = False + assert response.status_code == 200 + assert response.json()["unstable"] == UNSTABLE_MESSAGE + + +@pytest.mark.asyncio +async def test_query_store_and_definition_report_unstable(ds_envelope): + store = await ds_envelope.client.post( + "/data/-/queries/store", + json={"query": {"name": "unstable_check", "sql": "select 1"}}, + actor={"id": "root"}, + ) + assert store.status_code == 201 + assert store.json()["unstable"] == UNSTABLE_MESSAGE + definition = await ds_envelope.client.get( + "/data/unstable_check/-/definition", actor={"id": "root"} + ) + assert definition.status_code == 200 + assert definition.json()["unstable"] == UNSTABLE_MESSAGE + + +@pytest.mark.asyncio +async def test_permissions_post_reports_unstable(ds_envelope): + response = await ds_envelope.client.post( + "/-/permissions", + data={"actor": '{"id": "root"}', "permission": "view-instance"}, + actor={"id": "root"}, + ) + assert response.status_code == 200 + assert response.json()["unstable"] == UNSTABLE_MESSAGE + + +@pytest.mark.asyncio +async def test_documented_endpoints_do_not_report_unstable(ds_client): + for path in ("/-/versions.json", "/fixtures.json", "/fixtures/facetable.json"): + response = await ds_client.get(path) + assert response.status_code == 200 + assert "unstable" not in response.json() From 404ee4c3a79c213d47e263e796364f8b471bde05 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 17:07:04 +0000 Subject: [PATCH 25/46] Document the JSON API stability promise docs/json_api.rst now opens with an API stability section declaring what the 1.x promise covers: documented endpoints, parameters and response keys are stable with additive-only changes; pagination tokens are opaque strings; the error format and token restriction semantics are stable. It lists the exempt tiers: endpoints carrying the "unstable" marker key, debug and support endpoints (/-/threads, /-/actions, /-/jump, the permission debug endpoints, table autocomplete), and keys explicitly labeled unstable such as the execute-write analysis block. Cross-referenced from the introspection and permission-debug documentation. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- docs/authentication.rst | 2 ++ docs/introspection.rst | 2 ++ docs/json_api.rst | 49 ++++++++++++++++++++++++++++++++++- stable-api-recommendations.md | 20 ++++++++------ 4 files changed, 64 insertions(+), 9 deletions(-) diff --git a/docs/authentication.rst b/docs/authentication.rst index 72ac5fa6..7d4d019d 100644 --- a/docs/authentication.rst +++ b/docs/authentication.rst @@ -1151,6 +1151,8 @@ It also provides an interface for running hypothetical permission checks against This is designed to help administrators and plugin authors understand exactly how permission checks are being carried out, in order to effectively configure Datasette's permission system. +These debug endpoints are exempt from the :ref:`JSON API stability promise ` - their JSON shapes may change in future releases. + .. _AllowedResourcesView: Allowed resources view diff --git a/docs/introspection.rst b/docs/introspection.rst index 2d13f68b..ea94b70d 100644 --- a/docs/introspection.rst +++ b/docs/introspection.rst @@ -9,6 +9,8 @@ Each of these pages can be viewed in your browser. Add ``.json`` to the URL to g JSON responses that return an object include an ``"ok": true`` key, consistent with the rest of the :ref:`JSON API `. +The introspection endpoints documented on this page are covered by the :ref:`JSON API stability promise `, with the exception of the debug endpoints ``/-/threads``, ``/-/actions`` and ``/-/jump``, whose shapes may change in future releases. + .. _JsonDataView_metadata: /-/metadata diff --git a/docs/json_api.rst b/docs/json_api.rst index dc611a39..4b1f26d2 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -9,6 +9,53 @@ through the Datasette user interface can also be accessed as JSON via the API. To access the API for a page, either click on the ``.json`` link on that page or edit the URL and add a ``.json`` extension to it. +.. _json_api_stability: + +API stability +------------- + +Datasette 1.0 makes a stability promise for its JSON API: the endpoints, +parameters and response keys documented here and on the pages this +documentation links to will not change in backwards-incompatible ways for +the duration of the 1.x release series. + +Stability means: + +- Documented endpoints will keep their URLs, methods, parameters and + permission requirements. +- Documented response keys will keep their names and types. New keys may be + **added** in any release - clients should ignore keys they do not + recognize. +- The documented ``?_extra=`` names, ``?_shape=`` values and + :ref:`column filter operators ` are stable. +- Pagination tokens - the ``"next"`` key and ``?_next=`` parameter - are + **opaque strings**. Pass them back exactly as you received them; their + internal structure is not part of the API and can change at any time. +- The :ref:`standard error format ` and the + :ref:`API token format and restriction semantics ` are + stable, including the action abbreviations stored inside signed tokens. + +Some JSON endpoints are **exempt** from this promise: + +- Endpoints that are not documented include this marker key in their + responses and can change at any time:: + + "unstable": "This API is not part of Datasette's stable interface and may change at any time" + + This currently covers the instance homepage (``/.json``), the stored + query ``analyze``/``store``/``definition`` endpoints, ``/-/query/parameters``, + ``/-/execute-write/analyze`` and the JSON returned by the ``/-/permissions`` + debug playground. +- Debug and support endpoints are documented so you can use them, but their + JSON shapes are not frozen: :ref:`/-/threads `, + :ref:`/-/actions `, :ref:`/-/jump `, + the :ref:`permission debug endpoints ` + (``/-/allowed``, ``/-/rules``, ``/-/check``) and the + :ref:`table autocomplete endpoint `. +- Response keys explicitly labeled as unstable in this documentation, such + as the ``"analysis"`` block returned by :ref:`execute-write ` + and the ``debug`` and ``request`` extras. + .. _json_api_default: Default representation @@ -1612,7 +1659,7 @@ Unsupported SQL operations are rejected by default. ``VACUUM`` is not allowed in A successful response includes a message, the SQLite ``rowcount``, a ``"rows"`` list, a ``"truncated"`` flag and a summary of the operations that were executed: -The shape of the ``"analysis"`` block is not yet considered a stable API and may change in future Datasette releases. +The shape of the ``"analysis"`` block is not part of the :ref:`stable API ` and may change in future Datasette releases. .. code-block:: json diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index baac80f7..b86d8228 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -378,13 +378,17 @@ Concerns: --- -## 9. Define stability tiers explicitly (P1 — documentation, not code) — partially implemented +## 9. Define stability tiers explicitly (P1 — documentation, not code) — ✅ IMPLEMENTED -> **Status:** undocumented JSON endpoints now self-describe with an -> `"unstable": "This API is not part of Datasette's stable interface and -> may change at any time"` key (homepage, queries analyze/store/definition, -> query parameters, execute-write analyze, permissions playground POST). -> The written tier documentation remains to be done. +> **Status:** implemented. Undocumented JSON endpoints self-describe with +> an `"unstable"` marker key, and `docs/json_api.rst` now opens with an +> "API stability" section (`json_api_stability`) declaring the 1.x +> promise: documented endpoints/keys are stable with additive-only +> changes, pagination tokens are opaque, the error format and token +> restriction semantics are stable, and the exempt tiers (marker-key +> endpoints, debug/support endpoints, explicitly-unstable keys) are +> listed. Cross-referenced from the introspection and permission-debug +> docs. Not everything under `/-/` can or should carry a 1.0 guarantee. Recommend shipping 1.0 with an explicit three-tier contract, per endpoint: @@ -431,8 +435,8 @@ Two details make tiering urgent rather than optional: `permissions-debug` (§6).~~ ✅ Done. 6. ~~401 (not silent-anonymous) for invalid/expired bearer tokens (§1c).~~ ✅ Done. -7. Publish explicit stability tiers, including extras and pagination-token - opacity (§9). +7. ~~Publish explicit stability tiers, including extras and pagination-token + opacity (§9).~~ ✅ Done. 8. Resolve the looks-like-a-bug list (§8), especially ~~trusted-query delete and row-delete 500~~ (both done). From 3a0ea585572ce56576b25ba3a843c587b98891c0 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 4 Jul 2026 17:54:03 +0000 Subject: [PATCH 26/46] Unify page-size parameters on _size with table semantics The stored query lists silently clamped out-of-range ?_size= values (a request for 5000 quietly returned 1000) and did not accept the max keyword. They now share the table view semantics via a new parse_size_limit() helper: blank means default, "max" means the maximum (max_returned_rows for query lists), negative or non-integer values are a 400, and values over the maximum are a 400 instead of being silently clamped. The /-/allowed and /-/rules debug endpoints renamed their bare page/page_size parameters to _page/_size, matching the underscore grammar used by every other system parameter, with the same validation (400 instead of silently capping page_size at 200). Their HTML debug pages and next_url/previous_url builders use the new names. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/templates/debug_allowed.html | 10 ++--- datasette/templates/debug_rules.html | 10 ++--- datasette/utils/__init__.py | 22 +++++++++++ datasette/views/query_helpers.py | 9 ++--- datasette/views/special.py | 53 +++++++++++++------------- datasette/views/stored_queries.py | 1 + docs/authentication.rst | 4 +- docs/pages.rst | 2 +- existing-api.md | 9 +++-- stable-api-recommendations.md | 10 +++-- tests/test_error_shape.py | 50 ++++++++++++++++++++++++ tests/test_html.py | 6 +-- tests/test_permission_endpoints.py | 14 +++---- 13 files changed, 137 insertions(+), 63 deletions(-) diff --git a/datasette/templates/debug_allowed.html b/datasette/templates/debug_allowed.html index 4f8106b8..83cc1ae6 100644 --- a/datasette/templates/debug_allowed.html +++ b/datasette/templates/debug_allowed.html @@ -49,7 +49,7 @@
- + Number of results per page (max 200)
@@ -88,7 +88,7 @@ const hasDebugPermission = {{ 'true' if has_debug_permission else 'false' }}; (function() { const params = populateFormFromURL(); const action = params.get('action'); - const page = params.get('page'); + const page = params.get('_page'); if (action) { fetchResults(page ? parseInt(page) : 1); } @@ -102,14 +102,14 @@ async function fetchResults(page = 1) { const params = new URLSearchParams(); for (const [key, value] of formData.entries()) { - if (value && key !== 'page_size') { + if (value && key !== '_size' && key !== '_page') { params.append(key, value); } } const pageSize = document.getElementById('page_size').value || '50'; - params.append('page', page.toString()); - params.append('page_size', pageSize); + params.append('_page', page.toString()); + params.append('_size', pageSize); try { const response = await fetch('{{ urls.path("-/allowed.json") }}?' + params.toString(), { diff --git a/datasette/templates/debug_rules.html b/datasette/templates/debug_rules.html index aafa755d..d00ba9cc 100644 --- a/datasette/templates/debug_rules.html +++ b/datasette/templates/debug_rules.html @@ -37,7 +37,7 @@
- + Number of results per page (max 200)
@@ -75,7 +75,7 @@ const submitBtn = document.getElementById('submit-btn'); (function() { const params = populateFormFromURL(); const action = params.get('action'); - const page = params.get('page'); + const page = params.get('_page'); if (action) { fetchResults(page ? parseInt(page) : 1); } @@ -89,14 +89,14 @@ async function fetchResults(page = 1) { const params = new URLSearchParams(); for (const [key, value] of formData.entries()) { - if (value && key !== 'page_size') { + if (value && key !== '_size' && key !== '_page') { params.append(key, value); } } const pageSize = document.getElementById('page_size').value || '50'; - params.append('page', page.toString()); - params.append('page_size', pageSize); + params.append('_page', page.toString()); + params.append('_size', pageSize); try { const response = await fetch('{{ urls.path("-/rules.json") }}?' + params.toString(), { diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index 1d921f02..19de31ff 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -1294,6 +1294,28 @@ async def derive_named_parameters(db: "Database", sql: str) -> List[str]: return named_parameters(sql) +def parse_size_limit(value, default, maximum, name="_size"): + """ + Parse a page-size parameter using the same semantics as the table + view's ?_size=: blank means default, "max" means maximum, integers + must be 0 or greater and no larger than maximum. Raises ValueError + with a message suitable for a 400 response. + """ + if value in (None, ""): + return default + if value == "max": + return maximum + try: + size = int(value) + if size < 0: + raise ValueError + except ValueError: + raise ValueError("{} must be a positive integer".format(name)) + if size > maximum: + raise ValueError("{} must be <= {}".format(name, maximum)) + return size + + UNSTABLE_API_MESSAGE = ( "This API is not part of Datasette's stable interface and may change at any time" ) diff --git a/datasette/views/query_helpers.py b/datasette/views/query_helpers.py index 026a999f..e9f85b6d 100644 --- a/datasette/views/query_helpers.py +++ b/datasette/views/query_helpers.py @@ -13,6 +13,7 @@ from datasette.write_sql import ( operation_is_write, ) from datasette.utils import ( + parse_size_limit, named_parameters as derive_named_parameters, escape_sqlite, path_from_row_pks, @@ -94,13 +95,11 @@ def _as_optional_bool(value, name): raise QueryValidationError("{} must be 0 or 1".format(name)) -def _query_list_limit(value, default=50): - if value in (None, ""): - return default +def _query_list_limit(value, default, maximum): try: - return min(max(1, int(value)), 1000) + return parse_size_limit(value, default, maximum) except ValueError as ex: - raise QueryValidationError("_size must be an integer") from ex + raise QueryValidationError(str(ex)) from ex def _derived_query_parameters(sql): diff --git a/datasette/views/special.py b/datasette/views/special.py index 28d4d6a1..9386440c 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -8,6 +8,7 @@ from datasette.utils.asgi import Response, Forbidden from datasette.utils import ( UNSTABLE_API_MESSAGE, actor_matches_allow, + parse_size_limit, add_cors_headers, await_me_maybe, error_body, @@ -373,17 +374,17 @@ class AllowedResourcesView(BaseView): ) try: - page = int(request.args.get("page", "1")) - page_size = int(request.args.get("page_size", "50")) + page = int(request.args.get("_page", "1")) + if page < 1: + raise ValueError except ValueError: - return error_body("page and page_size must be integers", 400), 400 - if page < 1: - return error_body("page must be >= 1", 400), 400 - if page_size < 1: - return error_body("page_size must be >= 1", 400), 400 - max_page_size = 200 - if page_size > max_page_size: - page_size = max_page_size + return error_body("_page must be a positive integer", 400), 400 + try: + page_size = parse_size_limit( + request.args.get("_size"), default=50, maximum=200 + ) + except ValueError as ex: + return error_body(str(ex), 400), 400 offset = (page - 1) * page_size # Use the simplified allowed_resources method @@ -448,12 +449,12 @@ class AllowedResourcesView(BaseView): def build_page_url(page_number): pairs = [] for key in request.args: - if key in {"page", "page_size"}: + if key in {"_page", "_size"}: continue for value in request.args.getlist(key): pairs.append((key, value)) - pairs.append(("page", str(page_number))) - pairs.append(("page_size", str(page_size))) + pairs.append(("_page", str(page_number))) + pairs.append(("_size", str(page_size))) query = urllib.parse.urlencode(pairs) return f"{request.path}?{query}" @@ -511,19 +512,19 @@ class PermissionRulesView(BaseView): actor = request.actor if isinstance(request.actor, dict) else None try: - page = int(request.args.get("page", "1")) - page_size = int(request.args.get("page_size", "50")) + page = int(request.args.get("_page", "1")) + if page < 1: + raise ValueError except ValueError: return Response.json( - error_body("page and page_size must be integers", 400), status=400 + error_body("_page must be a positive integer", 400), status=400 ) - if page < 1: - return Response.json(error_body("page must be >= 1", 400), status=400) - if page_size < 1: - return Response.json(error_body("page_size must be >= 1", 400), status=400) - max_page_size = 200 - if page_size > max_page_size: - page_size = max_page_size + try: + page_size = parse_size_limit( + request.args.get("_size"), default=50, maximum=200 + ) + except ValueError as ex: + return Response.json(error_body(str(ex), 400), status=400) offset = (page - 1) * page_size from datasette.utils.actions_sql import build_permission_rules_sql @@ -574,12 +575,12 @@ class PermissionRulesView(BaseView): def build_page_url(page_number): pairs = [] for key in request.args: - if key in {"page", "page_size"}: + if key in {"_page", "_size"}: continue for value in request.args.getlist(key): pairs.append((key, value)) - pairs.append(("page", str(page_number))) - pairs.append(("page_size", str(page_size))) + pairs.append(("_page", str(page_number))) + pairs.append(("_size", str(page_size))) query = urllib.parse.urlencode(pairs) return f"{request.path}?{query}" diff --git a/datasette/views/stored_queries.py b/datasette/views/stored_queries.py index ea49d983..bb23a6bd 100644 --- a/datasette/views/stored_queries.py +++ b/datasette/views/stored_queries.py @@ -90,6 +90,7 @@ class QueryListView(BaseView): limit = _query_list_limit( request.args.get("_size"), default=20 if format_ == "html" else 50, + maximum=self.ds.max_returned_rows, ) is_write = _as_optional_bool(request.args.get("is_write"), "is_write") is_private = _as_optional_bool(request.args.get("is_private"), "is_private") diff --git a/docs/authentication.rst b/docs/authentication.rst index 7d4d019d..0a476c1c 100644 --- a/docs/authentication.rst +++ b/docs/authentication.rst @@ -1162,7 +1162,7 @@ The ``/-/allowed`` endpoint displays resources that the current actor can access This endpoint provides an interactive HTML form interface. Add ``.json`` to the URL path (e.g. ``/-/allowed.json``) to get the raw JSON response instead. -Pass ``?action=view-table`` (or another action) to select the action. Optional ``parent=`` and ``child=`` query parameters can narrow the results to a specific database/table pair. +Pass ``?action=view-table`` (or another action) to select the action. Optional ``parent=`` and ``child=`` query parameters can narrow the results to a specific database/table pair. Results are paginated: ``?_size=`` sets the page size (default 50, maximum 200, ``max`` for the maximum) and ``?_page=`` selects a page. This endpoint is publicly accessible to help users understand their own permissions. The potentially sensitive ``reason`` field is only shown to users with the ``permissions-debug`` permission - it shows the plugins and explanatory reasons that were responsible for each decision. @@ -1175,7 +1175,7 @@ The ``/-/rules`` endpoint displays all permission rules (both allow and deny) fo This endpoint provides an interactive HTML form interface. Add ``.json`` to the URL path (e.g. ``/-/rules.json?action=view-table``) to get the raw JSON response instead. -Pass ``?action=`` as a query parameter to specify which action to check. +Pass ``?action=`` as a query parameter to specify which action to check. The ``?_size=`` and ``?_page=`` pagination parameters work the same as on ``/-/allowed``. This endpoint requires the ``permissions-debug`` permission. diff --git a/docs/pages.rst b/docs/pages.rst index 67fe390b..65c03a49 100644 --- a/docs/pages.rst +++ b/docs/pages.rst @@ -95,7 +95,7 @@ Use the :ref:`ExecuteWriteView` JSON API to execute writable SQL programmaticall Stored query browsers --------------------- -The ``/-/queries`` page lists stored queries across every database visible to the current actor. The ``/database-name/-/queries`` page lists stored queries for a single database. +The ``/-/queries`` page lists stored queries across every database visible to the current actor. The ``/database-name/-/queries`` page lists stored queries for a single database. The JSON versions accept ``?_size=`` (default 50, ``max`` for the :ref:`setting_max_returned_rows` limit) and a ``?_next=`` pagination token. These pages support search, pagination and filters for read-only or writable queries and private or public queries. Adding a ``.json`` extension to either URL returns the same list as JSON. diff --git a/existing-api.md b/existing-api.md index 2f9996dc..c0c8e442 100644 --- a/existing-api.md +++ b/existing-api.md @@ -377,8 +377,8 @@ path always renders the HTML form; `.json` returns JSON. resources. Items gain a `reason` field if the actor also holds `permissions-debug`. - **Parameters:** `action` (required; missing → 400 canonical error, unknown - → 404), `parent`, `child` (requires `parent`), `page` (default 1), - `page_size` (default 50, silently capped at 200). + → 404), `parent`, `child` (requires `parent`), `_page` (default 1), + `_size` (default 50, maximum 200, accepts `max`; out-of-range → 400). - **Response:** `{"action", "actor_id", "page", "page_size", "total", "items": [{"parent", "child", "resource"}]}` with optional `next_url` / `previous_url`. @@ -1031,8 +1031,9 @@ databases (`database`/`database_color` are null, `show_database` true). - **Permissions:** no single gate; results filtered per query by `view-query` (private queries appear only for their owner). -- **Parameters:** `_size` (default 20 HTML / **50 JSON**, clamped 1–1000; - non-integer → 400), `_next` (cursor), `q` (substring search over +- **Parameters:** `_size` (default 20 HTML / **50 JSON**; accepts `max`; + values over `max_returned_rows` or non-integers → 400, matching table + `_size` semantics), `_next` (cursor), `q` (substring search over name/title/description/sql), `is_write` / `is_private` (booleans; invalid → 400 `"is_write must be 0 or 1"`), `source`, `owner_id`. - **Response** — 200: diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index b86d8228..ba837204 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -203,9 +203,13 @@ number. > **Status:** `next_url` now accompanies `next` in the default table JSON > keys (previously it required `?_extra=next_url`), so every response with > a `next` token also carries the ready-to-follow URL. Pagination tokens -> are deliberately left undocumented as to their internal structure. The -> `_size`/`page_size` naming and `has_more`/`total` differences remain -> open. +> are deliberately left undocumented as to their internal structure. +> `_size` is now the single page-size parameter with uniform table-style +> semantics everywhere: query lists accept `max` and 400 on out-of-range +> values (previously silently clamped), and the `/-/allowed` and +> `/-/rules` debug endpoints renamed `page`/`page_size` to +> `_page`/`_size` with the same validation (400 instead of silent +> capping at 200). The `has_more`/`total` differences remain open. | Endpoint | Mechanism | Token | Extras | |---|---|---|---| diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index a34fbed7..2398940b 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -612,3 +612,53 @@ async def test_threads_requires_permissions_debug(ds_error_shape): allowed = await ds_error_shape.client.get("/-/threads.json", actor={"id": "root"}) assert allowed.status_code == 200 assert allowed.json()["ok"] is True + + +# _size is the one page-size parameter, with uniform validation + + +@pytest.mark.asyncio +async def test_query_list_size_supports_max_keyword(ds_client): + response = await ds_client.get("/fixtures/-/queries.json?_size=max") + assert response.status_code == 200 + # ds_client runs with max_returned_rows=100 + assert response.json()["limit"] == 100 + + +@pytest.mark.asyncio +async def test_query_list_size_rejects_out_of_range(ds_client): + response = await ds_client.get("/fixtures/-/queries.json?_size=5000") + data = assert_canonical_error(response, 400) + assert data["errors"] == ["_size must be <= 100"] + + +@pytest.mark.asyncio +async def test_query_list_size_rejects_non_integer(ds_client): + response = await ds_client.get("/fixtures/-/queries.json?_size=bananas") + data = assert_canonical_error(response, 400) + assert data["errors"] == ["_size must be a positive integer"] + + +@pytest.mark.asyncio +@pytest.mark.parametrize("endpoint", ("allowed", "rules")) +async def test_debug_endpoints_use_size_and_page_parameters(ds_error_shape, endpoint): + base = "/-/{}.json?action=view-instance".format(endpoint) + ok = await ds_error_shape.client.get( + base + "&_size=1&_page=1", actor={"id": "root"} + ) + assert ok.status_code == 200 + assert ok.json()["page_size"] == 1 + + max_size = await ds_error_shape.client.get( + base + "&_size=max", actor={"id": "root"} + ) + assert max_size.status_code == 200 + assert max_size.json()["page_size"] == 200 + + too_big = await ds_error_shape.client.get(base + "&_size=500", actor={"id": "root"}) + data = assert_canonical_error(too_big, 400) + assert data["errors"] == ["_size must be <= 200"] + + bad_page = await ds_error_shape.client.get(base + "&_page=0", actor={"id": "root"}) + data = assert_canonical_error(bad_page, 400) + assert data["errors"] == ["_page must be a positive integer"] diff --git a/tests/test_html.py b/tests/test_html.py index 22943300..b4c47d80 100644 --- a/tests/test_html.py +++ b/tests/test_html.py @@ -1363,12 +1363,12 @@ async def test_permission_debug_tabs_with_query_string(ds_client): # Test /-/allowed with query string response = await ds_client.get( - "/-/allowed?action=view-table&page_size=50", actor=actor + "/-/allowed?action=view-table&_size=50", actor=actor ) assert response.status_code == 200 # Check that Rules and Check tabs have the query string - assert 'href="/-/rules?action=view-table&page_size=50"' in response.text - assert 'href="/-/check?action=view-table&page_size=50"' in response.text + assert 'href="/-/rules?action=view-table&_size=50"' in response.text + assert 'href="/-/check?action=view-table&_size=50"' in response.text # Playground and Actions should not have query string assert 'href="/-/permissions"' in response.text assert 'href="/-/actions"' in response.text diff --git a/tests/test_permission_endpoints.py b/tests/test_permission_endpoints.py index e25be23e..8726ab62 100644 --- a/tests/test_permission_endpoints.py +++ b/tests/test_permission_endpoints.py @@ -137,9 +137,7 @@ async def test_allowed_json_pagination(): await ds.refresh_schemas() # Test page 1 - response = await ds.client.get( - "/-/allowed.json?action=view-table&page_size=10&page=1" - ) + response = await ds.client.get("/-/allowed.json?action=view-table&_size=10&_page=1") assert response.status_code == 200 data = response.json() assert data["page"] == 1 @@ -147,9 +145,7 @@ async def test_allowed_json_pagination(): assert len(data["items"]) == 10 # Test page 2 - response = await ds.client.get( - "/-/allowed.json?action=view-table&page_size=10&page=2" - ) + response = await ds.client.get("/-/allowed.json?action=view-table&_size=10&_page=2") assert response.status_code == 200 data = response.json() assert data["page"] == 2 @@ -157,10 +153,10 @@ async def test_allowed_json_pagination(): # Verify items are different between pages response1 = await ds.client.get( - "/-/allowed.json?action=view-table&page_size=10&page=1" + "/-/allowed.json?action=view-table&_size=10&_page=1" ) response2 = await ds.client.get( - "/-/allowed.json?action=view-table&page_size=10&page=2" + "/-/allowed.json?action=view-table&_size=10&_page=2" ) items1 = {(item["parent"], item["child"]) for item in response1.json()["items"]} items2 = {(item["parent"], item["child"]) for item in response2.json()["items"]} @@ -323,7 +319,7 @@ async def test_rules_json_pagination(): # Test basic pagination structure - just verify it returns paginated results response = await ds.client.get( - "/-/rules.json?action=view-table&page_size=2&page=1", + "/-/rules.json?action=view-table&_size=2&_page=1", actor={"id": "root"}, ) assert response.status_code == 200 From f4dfd6e0f73a441ad0aa28b31bcdcf1c73942b29 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 22:19:35 +0000 Subject: [PATCH 27/46] Fix unused variable flagged by ruff Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- tests/test_error_shape.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index 2398940b..b5f16b30 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -422,7 +422,7 @@ async def test_bad_signature_token_returns_401(ds_error_shape): response = await ds_error_shape.client.get( "/-/actor.json", headers={"Authorization": "Bearer dstok_garbage"} ) - data = assert_canonical_error(response, 401) + assert_canonical_error(response, 401) assert response.headers["www-authenticate"].startswith("Bearer") From 5c418efd7f18f9b469ccf14c1c8f86fd69d4427b Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 22:42:12 +0000 Subject: [PATCH 28/46] Update tests from main for canonical error shape and rows key Two tests merged from main were written against the pre-merge response shapes: the max_post_body_bytes 413 error now uses the canonical error envelope, and row update with return:true responds with a rows list rather than a singular row. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- tests/test_api_write.py | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/tests/test_api_write.py b/tests/test_api_write.py index 7145a859..14dedb73 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -169,7 +169,10 @@ async def test_base64_write_api_insert_upsert_update_decode_blobs(ds_write): headers=_headers(token), ) assert update_response.status_code == 200 - assert update_response.json()["row"]["data"] == {"$base64": True, "encoded": "/wAB"} + assert update_response.json()["rows"][0]["data"] == { + "$base64": True, + "encoded": "/wAB", + } rows = (await db.execute(""" select @@ -344,10 +347,9 @@ async def test_insert_rows_post_body_too_large(tmp_path_factory): headers=_headers(token), ) assert response.status_code == 413 - assert response.json() == { - "ok": False, - "errors": ["Request body exceeded maximum size of 100 bytes"], - } + assert response.json() == error_body( + ["Request body exceeded maximum size of 100 bytes"], 413 + ) # A small body should still work response2 = await ds.client.post( "/data/docs/-/insert", From e892c686c20ab21dd9b3d3307a8b2af36149a685 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 23:02:42 +0000 Subject: [PATCH 29/46] Remove has_more from query list JSON - next: null signals the end next: null (with next_url: null) is the single end-of-results signal across the API, keeping default response keys to a minimum. The StoredQueryPage.has_more attribute on the documented Python API is unchanged. Also fixes a bug this uncovered: the query list JSON next_url pointed at the HTML page (it was built from the query list path, dropping the .json extension) and was a relative path where the table view next_url is absolute. It is now built from the request path and absolute, so it preserves the requested format and can be followed directly. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/stored_queries.py | 1 - datasette/views/stored_queries.py | 7 +++---- existing-api.md | 2 +- stable-api-recommendations.md | 8 +++++++- tests/test_queries.py | 26 +++++++++++++++++++++++++- 5 files changed, 36 insertions(+), 8 deletions(-) diff --git a/datasette/stored_queries.py b/datasette/stored_queries.py index d7e1ec99..f5f977d9 100644 --- a/datasette/stored_queries.py +++ b/datasette/stored_queries.py @@ -83,7 +83,6 @@ def stored_query_page_to_dict(page: StoredQueryPage) -> dict[str, Any]: return { "queries": [stored_query_to_dict(query) for query in page.queries], "next": page.next, - "has_more": page.has_more, "limit": page.limit, } diff --git a/datasette/views/stored_queries.py b/datasette/views/stored_queries.py index bb23a6bd..3273d13c 100644 --- a/datasette/views/stored_queries.py +++ b/datasette/views/stored_queries.py @@ -120,9 +120,9 @@ class QueryListView(BaseView): if key != "_next" ] pairs.append(("_next", page.next)) - next_url = "{}?{}".format( - query_list_path, - urlencode(pairs), + next_url = self.ds.absolute_url( + request, + "{}?{}".format(request.path, urlencode(pairs)), ) current_filters = { @@ -208,7 +208,6 @@ class QueryListView(BaseView): "queries": page.queries, "next": page.next, "next_url": next_url, - "has_more": page.has_more, "limit": page.limit, "show_private_note": any(query.is_private for query in page.queries), "show_trusted_note": any(query.is_trusted for query in page.queries), diff --git a/existing-api.md b/existing-api.md index c0c8e442..55e9d6a2 100644 --- a/existing-api.md +++ b/existing-api.md @@ -1038,7 +1038,7 @@ databases (`database`/`database_color` are null, `show_database` true). 400 `"is_write must be 0 or 1"`), `source`, `owner_id`. - **Response** — 200: `{"ok": true, "database", "database_color", "queries": [...], "next", - "next_url", "has_more", "limit", "show_private_note", + "next_url", "limit", "show_private_note", "show_trusted_note", "query_list_path", "show_database", "facets": [{title, items: [{label, count, href, active}]}], "filters": {q, is_write, is_private, source, owner_id}}`. diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index ba837204..229dccd4 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -209,7 +209,13 @@ number. > values (previously silently clamped), and the `/-/allowed` and > `/-/rules` debug endpoints renamed `page`/`page_size` to > `_page`/`_size` with the same validation (400 instead of silent -> capping at 200). The `has_more`/`total` differences remain open. +> capping at 200). `has_more` has been **removed** from the query-list +> JSON — `next: null` is the single end-of-results signal everywhere, +> keeping default response keys minimal (`total` remains a debug-endpoint +> nicety). Fixing this also uncovered and fixed a bug where the query +> list's JSON `next_url` pointed at the HTML page (it dropped the `.json` +> extension) and was relative where the table `next_url` is absolute. +> §3 is now fully resolved. | Endpoint | Mechanism | Token | Extras | |---|---|---|---| diff --git a/tests/test_queries.py b/tests/test_queries.py index 699743c5..c25ec358 100644 --- a/tests/test_queries.py +++ b/tests/test_queries.py @@ -879,7 +879,7 @@ async def test_query_list_html_defaults_to_twenty_and_shows_pagination(): assert response.text.count('aria-label="Query pagination"') == 1 assert "Demo query 20" in response.text assert "Demo query 21" not in response.text - assert 'href="/data/-/queries?_next=' in response.text + assert 'href="http://localhost/data/-/queries?_next=' in response.text assert len(json_response.json()["queries"]) == 25 @@ -3827,3 +3827,27 @@ async def test_stored_query_json_uses_parameters_not_params(): query = [q for q in listing["queries"] if q["name"] == "with_params"][0] assert query["parameters"] == ["name", "age"] assert "params" not in query + + +@pytest.mark.asyncio +async def test_query_list_json_signals_pagination_via_next_only(): + ds = Datasette(memory=True) + ds.add_memory_database("query_list_next_only", name="data") + await ds.invoke_startup() + for i in range(3): + await ds.add_query( + "data", + name="q{}".format(i), + sql="select {}".format(i), + ) + first = (await ds.client.get("/data/-/queries.json?_size=2")).json() + assert "has_more" not in first + assert first["next"] is not None + assert first["next_url"] is not None + # The internal test client cannot follow absolute URLs + next_path = first["next_url"].replace("http://localhost", "") + assert next_path.startswith("/data/-/queries.json?") + last = (await ds.client.get(next_path)).json() + assert "has_more" not in last + assert last["next"] is None + assert last["next_url"] is None From 6e17c513619a139c1157d2c207324a8eddbec8d2 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 23:08:05 +0000 Subject: [PATCH 30/46] Document why upsert returns 200 where insert returns 201 An upsert may update existing rows without creating anything, so it deliberately does not claim resource creation with a 201. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- docs/json_api.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/json_api.rst b/docs/json_api.rst index 32d28733..2105d6e0 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -1944,7 +1944,7 @@ The above example will: Similar to ``/-/insert``, a ``row`` key with an object can be used instead of a ``rows`` array to upsert a single row. -If successful, this will return a ``200`` status code and a ``{"ok": true}`` response body. +If successful, this will return a ``200`` status code and a ``{"ok": true}`` response body. This is deliberately different from the ``201`` returned by :ref:`insert `: an upsert may update existing rows without creating anything, so it does not claim resource creation. Add ``"return": true`` to the request body to return full copies of the affected rows after they have been inserted or updated: From 3322e1f528f4d607eb0693a4cee2df97400eb215 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 23:08:28 +0000 Subject: [PATCH 31/46] Document the boolean query string argument grammar Boolean arguments parsed by value_as_boolean() accept on/true/1 and off/false/0 - state this once in the JSON API docs rather than leaving each argument to imply its own grammar. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- docs/json_api.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/json_api.rst b/docs/json_api.rst index 2105d6e0..c31ba756 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -276,6 +276,10 @@ Here is an example Python function built using `requests Date: Mon, 6 Jul 2026 23:09:10 +0000 Subject: [PATCH 32/46] Advise plugin authors on naming secret configuration keys /-/config redacts values for keys whose names contain secret, key, password, token, hash or dsn. Plugins that follow that naming get automatic redaction; plugins that don't will leak their secrets on that endpoint. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- docs/plugins.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/plugins.rst b/docs/plugins.rst index d2b5c20a..d32a9fe6 100644 --- a/docs/plugins.rst +++ b/docs/plugins.rst @@ -459,6 +459,8 @@ Secret configuration values Some plugins may need configuration that should stay secret - API keys for example. There are two ways in which you can store secret configuration values. +The :ref:`/-/config ` introspection endpoint redacts the values of any configuration keys whose names contain one of these substrings: ``secret``, ``key``, ``password``, ``token``, ``hash`` or ``dsn``. Name your plugin's secret configuration keys accordingly - for example ``api_key`` or ``client_secret`` - so they are automatically redacted there. + **As environment variables**. If your secret lives in an environment variable that is available to the Datasette process, you can indicate that the configuration value should be read from that environment variable like so: .. [[[cog From 022eb6d3a0a53961cff6b3221c2c54f64b4729a1 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 23:09:27 +0000 Subject: [PATCH 33/46] Mark documented items in recommendations Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- stable-api-recommendations.md | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index 229dccd4..9e8fa356 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -243,10 +243,11 @@ Concerns: ## 4. HTTP semantics (P2) -- **201 vs 200:** insert → 201, upsert → 200 (views/table.py:1194), create +- ~~**201 vs 200:** insert → 201, upsert → 200 (views/table.py:1194), create table → 201, store query → 201. Insert-201/upsert-200 is defensible (upsert may not create) but it is undocumented subtlety; state it, or - return 200 for both with an explicit `created` count. + return 200 for both with an explicit `created` count.~~ ✅ **Done** — + documented as deliberate in the upsert docs. - **Destructive-action confirmation is asymmetric:** table drop requires `{"confirm": true}` and has a preview response (views/table.py:1346-1365); row delete executes immediately and ignores the body; query delete @@ -284,10 +285,13 @@ Concerns: accepted input alias for API creation and `datasette.yaml` config. - **Three names for the same concept across error/message payloads:** `error`, `errors`, `message`. See §1. -- **Boolean query parameters have at least three grammars:** `_nl=on`, +- ~~**Boolean query parameters have at least three grammars:** `_nl=on`, `_labels=on/off`, `?all=1`, `is_write=1|0|true|false|t|f|yes|no|on|off`, `_nocount=1`. Adopt one accepted set (the query-list parser at - query_helpers.py:81-94 is a good candidate) and apply it everywhere. + query_helpers.py:81-94 is a good candidate) and apply it everywhere.~~ + ✅ **Documented** — the JSON API docs state the canonical grammar + (`on/true/1`, `off/false/0`), which `value_as_boolean` already accepts + everywhere it is used. - ~~**`.jsono`** survives on the homepage route (identical output to `.json`) and as a row-view redirect. Remove it at 1.0; it is pure legacy.~~ ✅ Removed: the homepage routes only accept `.json` and the row-view @@ -322,9 +326,12 @@ Concerns: task reprs including file paths) behind only `view-instance`. Consider `permissions-debug`, alongside `/-/actions` which already requires it.~~ ✅ **Done** — `/-/threads` now requires `permissions-debug`. -- **(P3) `/-/config` redaction is substring-based** on six key names +- ~~**(P3) `/-/config` redaction is substring-based** on six key names (app.py:2502-2505); plugins storing secrets under other names leak. Worth - a note in plugin authoring docs plus a `redact_keys` plugin hook. + a note in plugin authoring docs plus a `redact_keys` plugin hook.~~ + ✅ **Documented** — the plugin secrets docs now advise naming keys to + match the redaction substrings (a `redact_keys` hook remains a possible + future addition). - **(P3) Database-level checks on `/-/create`** (insert-row/update-row checked against `DatabaseResource`, not the about-to-exist table — table_create_alter.py:819-856) vs table-level checks on `/-/insert`. From 87cd695ca3b9293937228762a75f94e8a4931469 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 23:20:47 +0000 Subject: [PATCH 34/46] Write endpoints parse the body as JSON regardless of Content-Type The insert, upsert, alter and set-column-type endpoints previously required Content-Type: application/json while /-/create parsed the body blind - and insert returned a 500 AttributeError when the header was missing entirely. The lenient rule is now uniform: the body is always parsed as JSON and invalid JSON is a 400. This makes curl -d and requests data=json.dumps(...) invocations work without remembering the header. Cross-site request forgery remains prevented by the Origin and Sec-Fetch-Site checks in CrossOriginProtectionMiddleware, which is the defense the strict content-type requirement was historically standing in for. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/views/table.py | 8 +--- datasette/views/table_create_alter.py | 4 -- docs/json_api.rst | 2 + existing-api.md | 6 +-- stable-api-recommendations.md | 8 +++- tests/test_api_write.py | 13 +------ tests/test_column_types.py | 12 +----- tests/test_error_shape.py | 55 +++++++++++++++++++++++++++ 8 files changed, 69 insertions(+), 39 deletions(-) diff --git a/datasette/views/table.py b/datasette/views/table.py index 0a9c88d4..80ad1aab 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -951,9 +951,7 @@ class TableInsertView(BaseView): def _errors(errors): return None, errors, {} - if not request.headers.get("content-type").startswith("application/json"): - # TODO: handle form-encoded data - return _errors(["Invalid content-type, must be application/json"]) + # The body is parsed as JSON regardless of the Content-Type header try: data = await request.json() except json.JSONDecodeError as e: @@ -1260,10 +1258,6 @@ class TableSetColumnTypeView(BaseView): ): return _error(["Permission denied"], 403) - content_type = request.headers.get("content-type") or "" - if not content_type.startswith("application/json"): - return _error(["Invalid content-type, must be application/json"], 400) - try: data = await request.json() except json.JSONDecodeError as e: diff --git a/datasette/views/table_create_alter.py b/datasette/views/table_create_alter.py index d0384184..c3e06c29 100644 --- a/datasette/views/table_create_alter.py +++ b/datasette/views/table_create_alter.py @@ -1166,10 +1166,6 @@ class TableAlterView(BaseView): if not db.is_mutable: return _error(["Database is immutable"], 403) - content_type = request.headers.get("content-type") or "" - if not content_type.startswith("application/json"): - return _error(["Invalid content-type, must be application/json"], 400) - try: data = await request.json() except json.JSONDecodeError as e: diff --git a/docs/json_api.rst b/docs/json_api.rst index c31ba756..893af634 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -1668,6 +1668,8 @@ The JSON write API Datasette provides a write API for JSON data. This is a POST-only API that requires an authenticated API token, see :ref:`CreateTokenView`. The token will need to have the specified :ref:`authentication_permissions`. +The request body is always parsed as JSON, regardless of the request's ``Content-Type`` header - a body that is not valid JSON returns a ``400`` error. Cross-site request forgery is prevented by Datasette's ``Origin`` and ``Sec-Fetch-Site`` header checks rather than by content type requirements. + The row-based write APIs can write :ref:`binary values in JSON ` using Datasette's Base64 representation for BLOB data. .. _ExecuteWriteView: diff --git a/existing-api.md b/existing-api.md index 55e9d6a2..c58f8f41 100644 --- a/existing-api.md +++ b/existing-api.md @@ -853,8 +853,8 @@ shape) and check permissions with additionally required for `alter: true` (403 `Permission denied for alter-table`). Immutable database → 403 `Database is immutable`. -- **Request** — requires `Content-Type: application/json` (else 400 - `"Invalid content-type, must be application/json"`). Body: +- **Request** — the body is parsed as JSON regardless of the request + `Content-Type` header (invalid JSON → 400). Body: | Field | Rules | |---|---| @@ -937,7 +937,7 @@ shape) and check permissions with does not change the SQLite schema. - **Permission:** `set-column-type` (403 `Permission denied`). -- **Request** (JSON content type required): `{"column": "name", +- **Request**: `{"column": "name", "column_type": {"type": "url", "config": {...}?} | null}`. Unknown keys/invalid structure → detailed 400 errors; unknown type → 400 `"Unknown column type: x"`. Default registered types (via the diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index 9e8fa356..30dd0429 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -254,11 +254,15 @@ Concerns: executes immediately. Decide the 1.0 rule (suggestion: confirmation only for schema-destroying operations, i.e. keep as is — but document it as a deliberate contract). -- **Content-type enforcement is inconsistent:** `/-/insert`, `/-/upsert`, +- ~~**Content-type enforcement is inconsistent:** `/-/insert`, `/-/upsert`, `/-/alter`, `/-/set-column-type` demand `Content-Type: application/json` (400 otherwise); `/-/create` parses the body as JSON regardless of content type; execute-write and the query CRUD endpoints accept both JSON - and form encodings. Pick one rule for JSON-only endpoints. + and form encodings. Pick one rule for JSON-only endpoints.~~ ✅ **Done** + — the lenient rule won: JSON-only write endpoints parse the body as JSON + regardless of `Content-Type` (CSRF protection comes from the + cross-origin header checks, not content types). This also fixed a 500 on + insert when the header was absent entirely. - **JSON-vs-HTML negotiation on POST differs per endpoint:** execute-write and canned queries key off `Accept: application/json` / a `_json` body field; the write API keys off nothing (always JSON); query store keys off diff --git a/tests/test_api_write.py b/tests/test_api_write.py index 14dedb73..840bd05b 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -394,13 +394,6 @@ async def test_insert_rows_post_body_too_large(tmp_path_factory): "Invalid JSON: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)" ], ), - ( - "/data/docs/-/insert", - {}, - "invalid_content_type", - 400, - ["Invalid content-type, must be application/json"], - ), ( "/data/docs/-/insert", [], @@ -582,11 +575,7 @@ async def test_insert_or_upsert_row_errors( json=input, headers={ "Authorization": "Bearer {}".format(token), - "Content-Type": ( - "text/plain" - if special_case == "invalid_content_type" - else "application/json" - ), + "Content-Type": "application/json", }, ) diff --git a/tests/test_column_types.py b/tests/test_column_types.py index 4e553771..cd308ec9 100644 --- a/tests/test_column_types.py +++ b/tests/test_column_types.py @@ -322,12 +322,6 @@ async def test_clear_column_type_api(ds_ct): "Invalid JSON: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)" ], ), - ( - {"column": "title", "column_type": {"type": "email"}}, - "invalid_content_type", - 400, - ["Invalid content-type, must be application/json"], - ), ( [], None, @@ -413,11 +407,7 @@ async def test_set_column_type_api_errors( kwargs = { "headers": { "Authorization": f"Bearer {token}", - "Content-Type": ( - "text/plain" - if special_case == "invalid_content_type" - else "application/json" - ), + "Content-Type": "application/json", } } if special_case == "invalid_json": diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index b5f16b30..040e645a 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -662,3 +662,58 @@ async def test_debug_endpoints_use_size_and_page_parameters(ds_error_shape, endp bad_page = await ds_error_shape.client.get(base + "&_page=0", actor={"id": "root"}) data = assert_canonical_error(bad_page, 400) assert data["errors"] == ["_page must be a positive integer"] + + +# Write endpoints parse the body as JSON regardless of Content-Type + + +@pytest.mark.asyncio +async def test_insert_works_without_content_type_header(ds_error_shape): + # Previously a 500 AttributeError + response = await ds_error_shape.client.post( + "/data/docs/-/insert", + content='{"row": {"id": 1, "title": "One"}}', + actor={"id": "root"}, + ) + assert response.status_code == 201 + assert response.json()["rows"][0]["title"] == "One" + + +@pytest.mark.asyncio +async def test_insert_works_with_form_content_type(ds_error_shape): + # Previously 400 "Invalid content-type, must be application/json" + response = await ds_error_shape.client.post( + "/data/docs/-/insert", + content='{"row": {"id": 2, "title": "Two"}}', + headers={"Content-Type": "application/x-www-form-urlencoded"}, + actor={"id": "root"}, + ) + assert response.status_code == 201 + + +@pytest.mark.asyncio +async def test_insert_form_encoded_body_is_invalid_json(ds_error_shape): + response = await ds_error_shape.client.post( + "/data/docs/-/insert", + content="title=Three", + headers={"Content-Type": "application/x-www-form-urlencoded"}, + actor={"id": "root"}, + ) + data = assert_canonical_error(response, 400) + assert data["errors"][0].startswith("Invalid JSON:") + + +@pytest.mark.asyncio +async def test_alter_and_set_column_type_ignore_content_type(ds_error_shape): + alter = await ds_error_shape.client.post( + "/data/docs/-/alter", + content='{"operations": [{"op": "add_column", "args": {"name": "extra"}}]}', + actor={"id": "root"}, + ) + assert alter.status_code == 200, alter.text + sct = await ds_error_shape.client.post( + "/data/docs/-/set-column-type", + content='{"column": "title", "column_type": {"type": "textarea"}}', + actor={"id": "root"}, + ) + assert sct.status_code == 200, sct.text From 60bac9439d5674f2a8e716568804c29ff43cd43a Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 23:21:42 +0000 Subject: [PATCH 35/46] Mark shape=object item done in recommendations Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- stable-api-recommendations.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index 30dd0429..4da23934 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -390,7 +390,8 @@ Concerns: required"`. The JSON behavior masks caller bugs; return 400 on both.~~ ✅ **Done** — all data formats now return 400; the HTML SQL editor page is unchanged. -3. **`_shape=object` HTTP 200 error** (§1b) — almost certainly unintended. +3. ~~**`_shape=object` HTTP 200 error** (§1b) — almost certainly unintended.~~ + ✅ Done — now 400 (fixed with §1b). 4. ~~**Row delete 500** (§1c) — inconsistent with every sibling endpoint.~~ ✅ Done — now 400. 5. **The "SQL Interrupted" error embeds an HTML fragment in the JSON `error` From 0d962deb05a047b3cecd680983505dcf645a3fac Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 23:34:17 +0000 Subject: [PATCH 36/46] Plain text SQL Interrupted errors in JSON responses The SQL time limit error embedded an HTML fragment (paragraph, textarea and script tags) as the error string in JSON responses. DatasetteError now accepts a plain_message which the exception handler prefers for JSON error bodies; the HTML error page keeps the rich message with the SQL textarea. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/handle_exception.py | 4 +++- datasette/views/base.py | 3 +++ datasette/views/database.py | 4 ++++ datasette/views/row.py | 4 ++++ stable-api-recommendations.md | 6 ++++-- tests/test_api.py | 12 +++--------- tests/test_error_shape.py | 22 ++++++++++++++++++++++ 7 files changed, 43 insertions(+), 12 deletions(-) diff --git a/datasette/handle_exception.py b/datasette/handle_exception.py index fc290dbc..e255ddf2 100644 --- a/datasette/handle_exception.py +++ b/datasette/handle_exception.py @@ -28,6 +28,7 @@ def handle_exception(datasette, request, exception): rich.get_console().print_exception(show_locals=True) title = None + plain_message = None if isinstance(exception, Base400): status = exception.status info = {} @@ -36,6 +37,7 @@ def handle_exception(datasette, request, exception): status = exception.status info = exception.error_dict message = exception.message + plain_message = exception.plain_message if exception.message_is_html: message = Markup(message) title = exception.title @@ -50,7 +52,7 @@ def handle_exception(datasette, request, exception): add_cors_headers(headers) if request.path.split("?")[0].endswith(".json"): body = dict(info) - body.update(error_body(message, status)) + body.update(error_body(plain_message or message, status)) return Response.json(body, status=status, headers=headers) info.update( { diff --git a/datasette/views/base.py b/datasette/views/base.py index a12ae050..88d16753 100644 --- a/datasette/views/base.py +++ b/datasette/views/base.py @@ -29,12 +29,15 @@ class DatasetteError(Exception): status=500, template=None, message_is_html=False, + plain_message=None, ): self.message = message self.title = title self.error_dict = error_dict or {} self.status = status self.message_is_html = message_is_html + # Plain text used for JSON error responses when message is HTML + self.plain_message = plain_message class View: diff --git a/datasette/views/database.py b/datasette/views/database.py index c9dcfa89..10dc66ae 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -819,6 +819,10 @@ class QueryView(View): title="SQL Interrupted", status=400, message_is_html=True, + plain_message=( + "SQL query took too long. The time limit is" + " controlled by the sql_time_limit_ms setting." + ), ) except sqlite3.DatabaseError as ex: query_error = str(ex) diff --git a/datasette/views/row.py b/datasette/views/row.py index 2a103180..d9a3deeb 100644 --- a/datasette/views/row.py +++ b/datasette/views/row.py @@ -200,6 +200,10 @@ class RowView(BaseView): title="SQL Interrupted", status=400, message_is_html=True, + plain_message=( + "SQL query took too long. The time limit is" + " controlled by the sql_time_limit_ms setting." + ), ) except (sqlite3.OperationalError, InvalidSql) as e: raise DatasetteError(str(e), title="Invalid SQL", status=400) diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md index 4da23934..0d817612 100644 --- a/stable-api-recommendations.md +++ b/stable-api-recommendations.md @@ -394,9 +394,11 @@ Concerns: ✅ Done — now 400 (fixed with §1b). 4. ~~**Row delete 500** (§1c) — inconsistent with every sibling endpoint.~~ ✅ Done — now 400. -5. **The "SQL Interrupted" error embeds an HTML fragment in the JSON `error` +5. ~~**The "SQL Interrupted" error embeds an HTML fragment in the JSON `error` value** (views/database.py:805-820). Error strings in the JSON API should - be plain text. + be plain text.~~ ✅ **Done** — `DatasetteError` gained a `plain_message` + used for JSON responses; the HTML error page keeps the rich version with + the SQL textarea. §8 is now fully resolved. --- diff --git a/tests/test_api.py b/tests/test_api.py index da0ebc07..9a96f14f 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -329,14 +329,8 @@ def test_sql_time_limit(app_client_shorter_time_limit): ) assert 400 == response.status expected_message = ( - "

SQL query took too long. The time limit is controlled by the\n" - 'sql_time_limit_ms\n' - "configuration option.

\n" - '\n' - "" + "SQL query took too long. The time limit is" + " controlled by the sql_time_limit_ms setting." ) assert response.json == { "ok": False, @@ -356,7 +350,7 @@ async def test_custom_sql_time_limit(ds_client): "/fixtures/-/query.json?sql=select+sleep(0.01)&_timelimit=5", ) assert response.status_code == 400 - assert response.json()["error"].startswith("

SQL query took too long.") + assert response.json()["error"].startswith("SQL query took too long.") @pytest.mark.asyncio diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index 040e645a..44856b36 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -717,3 +717,25 @@ async def test_alter_and_set_column_type_ignore_content_type(ds_error_shape): actor={"id": "root"}, ) assert sct.status_code == 200, sct.text + + +# SQL Interrupted errors carry plain text in JSON, not an HTML fragment + + +@pytest.mark.asyncio +async def test_sql_interrupted_json_error_is_plain_text(ds_client): + response = await ds_client.get( + "/fixtures/-/query.json?sql=select+sleep(0.01)&_timelimit=5" + ) + data = assert_canonical_error(response, 400) + assert "<" not in data["error"] + assert data["error"].startswith("SQL query took too long.") + + +@pytest.mark.asyncio +async def test_sql_interrupted_html_page_keeps_rich_error(ds_client): + response = await ds_client.get( + "/fixtures/-/query?sql=select+sleep(0.01)&_timelimit=5" + ) + assert response.status_code == 400 + assert " Date: Mon, 6 Jul 2026 23:34:17 +0000 Subject: [PATCH 37/46] Remove working analysis documents existing-api.md and stable-api-recommendations.md were working documents for the 1.0 API consistency review. Their content remains available in this branch history; they are not intended to merge to main. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- existing-api.md | 1244 --------------------------------- stable-api-recommendations.md | 469 ------------- 2 files changed, 1713 deletions(-) delete mode 100644 existing-api.md delete mode 100644 stable-api-recommendations.md diff --git a/existing-api.md b/existing-api.md deleted file mode 100644 index c58f8f41..00000000 --- a/existing-api.md +++ /dev/null @@ -1,1244 +0,0 @@ -# Datasette JSON API — As Implemented - -This document describes the JSON API of this Datasette codebase (version `1.0a35`) as -derived directly from the source code. It intentionally ignores the existing `docs/` -directory: every claim below is based on the route table in `datasette/app.py` -(`Datasette._routes()`, app.py:2507-2767) and the view implementations in -`datasette/views/`. - -## Contents - -- [Cross-cutting behavior](#cross-cutting-behavior) -- [Instance endpoints](#instance-endpoints) -- [Database endpoints](#database-endpoints) -- [Table and row read endpoints](#table-and-row-read-endpoints) -- [The write API](#the-write-api) -- [Stored (canned) queries API](#stored-canned-queries-api) -- [Authentication and tokens](#authentication-and-tokens) -- [Appendix: registered actions (permissions)](#appendix-registered-actions-permissions) - ---- - -## Cross-cutting behavior - -### URL formats and content negotiation - -- Most read endpoints are registered with an optional format suffix: - `/(...)(\.(?Pjson))?$`. The bare path returns HTML; the `.json` - extension returns JSON. -- Table, row and query routes accept any `\w+` format extension; formats other - than the built-in `html`, `json`, `csv`, `blob` must be provided by a plugin - via `register_output_renderer`, otherwise the request 404s. -- HTML responses include a `Link: <...>; rel="alternate"; - type="application/json+datasette"` header pointing at the `.json` variant - (views/base.py:141-159), unless the view opts out with - `has_json_alternate = False`. -- Database, table, row and query names in paths are **tilde-encoded** - (a percent-encoding variant using `~` as the escape character; - utils/__init__.py `_TILDE_ENCODING_SAFE`). Multi-column primary keys in row - URLs are comma-separated. -- JSON responses are always compact `json.dumps` output serialized by - `CustomJSONEncoder`; there is no pretty-printing query parameter. Binary - values are serialized as `{"$base64": true, "encoded": "..."}`. -- Success content type: `application/json; charset=utf-8` - (`_shape=array&_nl=on` responses use `text/plain`). - -### Success envelope and stability marker - -Every JSON endpoint that returns an object includes `"ok": true` on -success. `JsonDataView` injects it automatically for dict responses -(views/special.py); the homepage, jump, schema, permission-debug and -autocomplete views add it explicitly. The former top-level-array endpoints -(`/-/plugins`, `/-/databases`, `/-/actions`) now return objects wrapping -their arrays (`{"ok": true, "plugins": [...]}` etc.). - -JSON endpoints that are **not part of the documented API** include a -marker key (`UNSTABLE_API_MESSAGE` in utils/__init__.py): - -```json -"unstable": "This API is not part of Datasette's stable interface and may change at any time" -``` - -Currently: the homepage (`/.json`, `/-/.json`), `/db/-/queries/analyze`, -`POST /db/-/queries/store`, `/db//-/definition`, -`/db/-/query/parameters`, `/db/-/execute-write/analyze` and the -`POST /-/permissions` playground response. - -### Error shape (canonical) - -Every JSON error response uses one canonical shape, built by `error_body()` -(utils/__init__.py): - -```json -{ - "ok": false, - "error": "all messages joined with '; '", - "errors": ["message", "..."], - "status": 404 -} -``` - -- `errors` is a list of one or more message strings (multi-message - validation errors, e.g. per-row insert errors, list them all). -- `error` is the messages joined with `"; "`. -- `status` always matches the HTTP status code. - -The shape is produced by four code paths, all delegating to `error_body()`: - -1. **Exception handler** (handle_exception.py) — `NotFound`, - `DatasetteError`, `BadRequest` etc. on `.json` paths. `DatasetteError` - `error_dict` context keys are merged in; the legacy `title` key is no - longer emitted in JSON (it survives in the HTML error template context). -2. **The `_error()` helper** (views/base.py:183-184) — the write API, - stored-query API, execute-write and permission-denied paths. -3. **JSON renderer errors** (renderer.py) — SQL errors on table/query - endpoints return HTTP 400 with the canonical keys **plus** the context - keys of the response it could not produce: - - ```json - {"ok": false, "error": "no such table: x", "errors": ["no such table: x"], - "status": 400, "rows": [], "truncated": false} - ``` - - Invalid `_shape=` values and `_shape=object` misuse (on queries or - pk-less tables) also return canonical 400 errors. -4. **Permission debug endpoints** (`/-/allowed`, `/-/rules`, `/-/check`, - POST `/-/permissions`) — canonical shape (previously bare - `{"error": ...}` objects). - -Method-not-allowed responses return HTTP 405 with the canonical shape when -the path ends in `.json` or the request content type is `application/json`; -plain text otherwise (views/base.py). - -**`Forbidden` handling:** when a view raises `Forbidden` (e.g. via -`ensure_permission`), the default `forbidden()` plugin hook returns the -canonical JSON error with status 403 when the path ends in `.json` or the -request carries an `Accept: application/json` / `Content-Type: -application/json` header; other requests get an HTML error page -(forbidden.py). Endpoints that check permissions themselves return -`_error(..., 403)` JSON directly. - -### CORS - -When Datasette is started with `--cors`, responses gain -(utils/__init__.py:1297-1302): - -``` -Access-Control-Allow-Origin: * -Access-Control-Allow-Headers: Authorization, Content-Type -Access-Control-Expose-Headers: Link -Access-Control-Allow-Methods: GET, POST, HEAD, OPTIONS -Access-Control-Max-Age: 3600 -``` - -### CSRF / cross-origin protection - -Datasette uses header-based cross-origin protection -(`CrossOriginProtectionMiddleware`, csrf.py:67-178) rather than CSRF tokens -for API calls. For non-GET/HEAD/OPTIONS requests: - -1. Requests carrying `Authorization: Bearer ...` **and no `Cookie` header** - bypass the check entirely (csrf.py:98-110). -2. Otherwise `Sec-Fetch-Site` must be `same-origin` or `none`; other values → 403. -3. If neither `Sec-Fetch-Site` nor `Origin` is present (curl, API clients), - the request passes. -4. Fallback: `Origin` must exactly match the request scheme/host/port → else 403. - -Plain JSON API clients (no cookies, no browser headers) are never blocked; -`Content-Type: application/json` itself plays no role in the CSRF decision. - -### Settings that govern the API - -From `SETTINGS` (app.py:197-287): `default_page_size` (100), -`max_returned_rows` (1000), `max_insert_rows` (100), `sql_time_limit_ms` -(1000), `default_facet_size` (30), `facet_time_limit_ms` (200), -`allow_facet` (true), `allow_download` (true), `allow_signed_tokens` (true), -`default_allow_sql` (true), `max_signed_tokens_ttl` (0), `default_cache_ttl` -(5), `allow_csv_stream` (true), `max_csv_mb` (100), `force_https_urls` -(false), `trace_debug` (false), `base_url` ("/"). - -### The JSON renderer: `_shape`, `_nl`, `_json`, `_json_infinity` - -`json_renderer` (renderer.py:31-126) processes `.json` output for table, row -and query views (but **not** for the instance/database/debug endpoints, which -build JSON directly): - -- **`_shape`** (default `objects`): - - `objects` — `{"ok": true, "rows": [{col: val}, ...], "truncated": false, ...}` - - `arrays` — same envelope, each row a list of values - - `array` — response body is a bare JSON array of row objects - - `arrayfirst` — bare JSON array of the first column's values - - `object` — table views only: an object keyed by primary-key string. - On queries or tables without primary keys: a canonical 400 error - (`_shape=object is only available on tables` / - `_shape=object not available for tables with no primary keys`). - - anything else — canonical HTTP 400 error `Invalid _shape: x` -- **`_nl=on`** — with `_shape=array` only: newline-delimited JSON, `text/plain`. -- **`_json=COLUMN`** (repeatable) — parse that column's string values with - `json.loads` so they nest as JSON; parse failures leave the value unchanged. -- **`_json_infinity=1`** — preserve `Infinity`/`-Infinity`; by default they - are replaced with `null`. -- `columns` is stripped from dict-shaped output unless `?_extra=columns` was - requested (renderer.py:110-113). -- If a SQL error occurred, `_shape` is ignored, HTTP status is 400 and the - envelope carries the canonical error keys alongside `rows`/`truncated`. - -### The `?_extra=` system - -Table, row and query JSON responses support `?_extra=` (repeatable and/or -comma-separated, extras.py:9-14) to add keys to the response. Extras are -scope-registered (`ExtraScope.TABLE` / `ROW` / `QUERY`) and only **public** -extras are available over JSON (extras.py:73-92). Unknown extra names (and -internal HTML-only names) on data formats return 400 -`Unknown _extra: `; HTML pages ignore them. The available names per -scope are listed with the relevant endpoints below. - ---- - -## Instance endpoints - -Most of these are implemented with `JsonDataView` (views/special.py:30-79): -GET-only; bare path renders an HTML page (`show_json.html`), `.json` returns -the data; permission defaults to `view-instance` and denial raises -`Forbidden` → **HTML** 403 page. - -### GET / - -Routes: `/(\.(?Pjsono?))?$` and `/-/(\.(?Pjsono?))?$` -(app.py:2517-2518); `/-` permanently redirects to `/-/`. `IndexView` -(views/index.py:22-189). `GET /.json`, `/.jsono` and `/-/.json` return JSON. - -- **Permission:** `view-instance` (denied → 403). Databases and tables are - further filtered by `view-database` / `view-table` for the actor. -- **Parameters:** `_sort=relationships` sorts each database's truncated table - list by foreign-key relationship count. -- **JSON response** (index.py:147-161) — includes `ok: true` plus: - - `databases` — a **list** of database objects (undocumented API, subject - to change). Each item: `name`, `hash` (or null), `color`, `path`, - `tables_and_views_truncated` (up to 5 items: `name`, `columns`, - `primary_keys`, `count` (int or null), `hidden`, `fts_table`, - `num_relationships_for_sorting`, `private`; view items are just - `{"name", "private"}`), `tables_and_views_more` (bool), `tables_count`, - `table_rows_sum`, `show_table_row_counts`, `hidden_table_rows_sum`, - `hidden_tables_count`, `views_count`, `private`. - - `metadata` — instance metadata object. - -### GET /-/versions(.json) - -`JsonDataView` over `Datasette._versions` (app.py:2548-2551, 2171-2245). -Permission `view-instance`. No parameters. - -Response keys: `python` (`{version, full}`), `datasette` (`{version}` plus -optional `note`), `asgi` (`"3.0"`), `uvicorn` (string or null), `sqlite` -(`{version, fts_versions, extensions, compile_options}`; `extensions` -includes `json1` and optionally `spatialite`), `pysqlite3` (only when -running under pysqlite3). - -### GET /-/plugins(.json) - -app.py:2552-2557, `Datasette._plugins` (app.py:2247-2266). Permission -`view-instance`. - -- **Parameters:** `?all=1` — include Datasette's built-in default plugins - (filtered out by default). -- **Response:** `{"ok": true, "plugins": [...]}` — each plugin is - `{"name", "static", "templates", "version", "hooks"}`, sorted by name. - -### GET /-/settings(.json) - -app.py:2558-2561. Permission `view-instance`. No parameters. Returns a flat -object mapping every setting name (see [Settings](#settings-that-govern-the-api)) -to its effective value. - -### GET /-/config(.json) - -app.py:2562-2565. Permission `view-instance`. No parameters. Returns the full -`datasette.yaml` configuration dict passed through -`redact_keys(config, ("secret", "key", "password", "token", "hash", "dsn"))` -(app.py:2502-2505) — any dict key containing one of those substrings has its -value replaced by `"***"` (utils/__init__.py:1532-1556). - -### GET /-/threads(.json) - -app.py:2566-2569, `Datasette._threads` (app.py:2268-2285). Permission -**`permissions-debug`** (exposes runtime internals). No parameters. - -Response: `num_threads`, `threads` (list of `{name, ident, daemon}`), -`num_tasks`, `tasks` (asyncio task repr strings). When the -`num_sql_threads` setting is 0 the response is exactly -`{"num_threads": 0, "threads": []}`. - -### GET /-/databases(.json) - -app.py:2570-2573, `Datasette._connected_databases` (app.py:2157-2169). -Permission `view-instance`. No parameters. - -Response: `{"ok": true, "databases": [...]}` — each database is -`{"name", "route", "path", "size", "is_mutable", "is_memory", "hash"}`. -Only databases the actor is allowed to `view-database` are listed. - -### GET /-/actor(.json) - -app.py:2574-2579, registered with `permission=None` — **accessible to any -request including anonymous**. No parameters. - -Response: `{"ok": true, "actor": {...}}` or `{"ok": true, "actor": null}` (app.py:2287-2288). - -### GET /-/actions(.json) - -app.py:2580-2589. Permission **`permissions-debug`**. No parameters. - -Response: `{"ok": true, "actions": [...]}` — each action is -`{"name", "abbr", "description", "takes_parent", "takes_child", -"resource_class", "also_requires"}`, sorted by name (app.py:2290-2304). - -### GET /-/auth-token - -`AuthTokenView` (app.py:2590-2593, views/special.py:198-217). GET only, no -`.json` variant, HTML/redirect only. - -- **Parameter:** `token` — the one-time secret printed by `datasette --root`. -- Match → invalidates the token, sets the signed `ds_actor` cookie to - `{"id": "root"}` and 302-redirects to the homepage. Mismatch or reuse → - `Forbidden` → 403 HTML. - -### GET/POST /-/create-token - -`CreateTokenView` (app.py:2594-2597, views/special.py:727-856). **HTML form -endpoint only — there is no JSON request/response mode in this codebase** -(`has_json_alternate = False`; the POST body must be form-encoded, a JSON -content type raises `BadRequest` → 400). - -- **Gates** (each failure → `Forbidden` → 403): `allow_signed_tokens` must be - on; request must have an actor with an `id`; the actor must not itself be - token-derived. -- **POST fields:** `expire_type` (`""`/`minutes`/`hours`/`days`), - `expire_duration` (positive int), plus restriction checkboxes named - `all:`, `database::`, - `resource::

{{ action.name }} {% if action.abbr %}{{ action.abbr }}{% endif %}
:`. -- **Response:** HTML page containing the new `dstok_` token. -- Programmatic alternatives: `datasette create-token` CLI or - `datasette.create_token()`. - -### GET /-/api - -`ApiExplorerView` (app.py:2598-2601, views/special.py:859-1020). HTML API -explorer, GET only. Permission `view-instance` (403 on denial). - -### GET /-/jump(.json) - -`JumpView` (app.py:2602-2605, views/special.py:1023-1201). The route allows -an optional `.json` suffix but the view **always returns JSON**. - -- **Permission:** none checked directly; results are filtered via - `allowed_resources_sql` for the current actor (default items come from the - `jump_items_sql` plugin hook). -- **Parameter:** `q` — whitespace-split terms matched as a case-insensitive - `%term1%term2%` LIKE pattern. -- **Response:** `{"ok": true, "matches": [...], "truncated": bool}`; each match: - `name`, `url`, `type` (`database`/`table`/`view`/`query`/plugin-defined), - `description`, optional `display_name`. Capped at 100 matches. - -### GET /-/schema(.json|.md) - -`InstanceSchemaView` (app.py:2610-2613, views/special.py:1257-1293). - -- **Permission:** no explicit check; only databases the actor can - `view-database` are included (others silently omitted). -- **Formats:** no extension → HTML; `.json` → - `{"ok": true, "schemas": [{"database": name, "schema": "..."}]}`; `.md` → - `text/markdown` rendering. - -### GET/POST /-/logout - -`LogoutView` (app.py:2614-2617, views/special.py:220-238). HTML endpoint. -GET renders a confirmation page (or redirects if anonymous); POST deletes the -`ds_actor` cookie and 302-redirects to `/`. - -### GET/POST /-/permissions - -`PermissionsDebugView` (app.py:2618-2621, views/special.py:241-295). No -`.json` route. Both methods require `view-instance` **and** -`permissions-debug` (403 on denial). - -- **GET** — HTML permission-check log; `?filter=all|exclude-yours|only-yours`. -- **POST** — form-encoded `actor` (JSON string), `permission`, optional - `resource_1`, `resource_2`; returns **JSON** - `{"action", "allowed", "resource": {"parent", "child", "path"}}` plus - `actor_id` when present. Errors: unknown action → 404; child without - parent → 400 (both canonical error shape). - -### GET /-/allowed(.json) - -`AllowedResourcesView` (app.py:2622-2625, views/special.py:298-460). Bare -path always renders the HTML form; `.json` returns JSON. - -- **Permission:** none — reports the **current actor's own** allowed - resources. Items gain a `reason` field if the actor also holds - `permissions-debug`. -- **Parameters:** `action` (required; missing → 400 canonical error, unknown - → 404), `parent`, `child` (requires `parent`), `_page` (default 1), - `_size` (default 50, maximum 200, accepts `max`; out-of-range → 400). -- **Response:** `{"action", "actor_id", "page", "page_size", "total", - "items": [{"parent", "child", "resource"}]}` with optional `next_url` / - `previous_url`. - -### GET /-/rules(.json) - -`PermissionRulesView` (app.py:2626-2629, views/special.py:463-584). -Permission `view-instance` **and** `permissions-debug`. Parameters and error -shapes as `/-/allowed`. Response items: -`{"parent", "child", "resource", "allow" (1|0), "reason", "source_plugin"}`. - -### GET /-/check(.json) - -`PermissionCheckView` (app.py:2630-2633, views/special.py:633-662). -Permission `permissions-debug`. Parameters `action` (required), `parent`, -`child`. Checks the **current request's actor**; response -`{"action", "allowed", "resource": {...}}` plus `actor_id`. - -### GET/POST /-/messages - -`MessagesDebugView` (app.py:2634-2637, views/special.py:703-724). HTML debug -tool for flash messages; permission `view-instance`; POST is form-encoded -(`message`, `message_type` = INFO/WARNING/ERROR/all) and 302-redirects. - -### GET /-/allow-debug - -`AllowDebugView` (app.py:2638-2641, views/special.py:665-700). GET only, HTML -only, **no permission required**. Parameters `actor` and `allow` (JSON -strings); renders the result of `actor_matches_allow()` in the page. - -### GET /-/patterns - -Pattern portfolio page (app.py:2642-2645). HTML only; not part of the JSON API. - -### GET /-/debug/autocomplete - -`AutocompleteDebugView` (app.py:2646-2649, views/special.py:94-195). HTML -debug page for the table autocomplete API; permission `view-instance` plus -`view-table` when `?database=&table=` are supplied. - ---- - -## Database endpoints - -### GET /\.db - -Downloads the raw SQLite file. Route → `database_download` -(app.py:2650-2653; views/database.py:533-570). - -- **Permission:** `view-database-download` (denied → `Forbidden` → 403 HTML). -- **Other gates:** unknown database → 404 `"Invalid database"`; in-memory - database → 404; `allow_download` off **or** mutable database → - `Forbidden("Database download is forbidden")`; no file path → 404. -- **Response:** streamed `application/octet-stream` with a - `content-disposition` attachment; immutable databases with a known hash set - `Etag` and honor `If-None-Match` → 304. - -### GET /\(.json) - -`DatabaseView` (app.py:2654-2657; views/database.py:71-277). Only `html` and -`json` formats are accepted; any other extension → 404 `"Invalid format: ..."`. - -- **Permission:** `view-database` via `check_visibility` (denied → - `Forbidden` → 403 HTML). Table/view listings are filtered by `view-table`; - stored queries by `view-query`. -- **Parameters:** - - `?sql=` — non-blank value 302-redirects to `//-/query?...` - preserving the query string and format. - - No `?_extra=` and no `_shape` support — the JSON is built directly and - returned via `Response.json`, bypassing the JSON renderer - (views/database.py:189-212). -- **JSON response** (all keys always present): - - `ok` — always `true` - - `database` — name; `private` — bool; `path` — URL path; `size` — bytes - - `tables` — list (includes hidden tables), each: - `name`, `columns` (names), `primary_keys`, `count` (int or null, - time-boxed), `count_truncated` (bool — count is a capped lower bound), - `hidden`, `fts_table`, `foreign_keys` (`{incoming: [...], outgoing: [...]}` - of `{other_table, column, other_column}`), `private` - - `hidden_count` — number of hidden tables - - `views` — list of `{name, private}` - - `queries` — **up to 5** stored queries (canonical stored-query objects, - see the stored-queries section); `queries_more` (bool); - `queries_count` (total visible) - - `allow_execute_sql` — bool for this actor - - `table_columns` — `{table: [columns]}`, empty `{}` unless - `allow_execute_sql` (views map to `[]`) - - `metadata` — database metadata dict - -### GET /\/-/query(.json) — arbitrary SQL - -`QueryView` (app.py:2691-2694; views/database.py:573-1130). The same class -also executes stored queries dispatched from the table route (see stored -queries section). - -- **Permission:** `execute-sql` on the database via `check_visibility` - (denied → `Forbidden` → 403 HTML). -- **Parameters:** - - `sql` — SQL to run. Must pass `validate_sql_select` - (utils/__init__.py:345-354): after stripping `--` comment lines it must - start with `select`, `with` or an `explain` variant, and must not contain - `pragma` (except allowlisted `pragma_*()` table-valued functions). - Failure → 400 `DatasetteError` titled `"Invalid SQL"` → JSON - `{"ok": false, "error": "Statement must be a SELECT", "status": 400, - "title": "Invalid SQL"}`. - - Any other `name=value` pair supplies the `:name` named parameter; missing - parameters default to `""`. Names starting with `_` are excluded. - - `_timelimit` — per-request SQL time limit in ms. - - `_shape`, `_nl`, `_json`, `_json_infinity` — see the JSON renderer section. - - `_extra` — QUERY-scope extras: `columns`, `debug`, `request`, - `render_cell`, `query` (`{"sql", "params"}`), `metadata`, `database`, - `database_color`, `private`, `extras`. -- **Response** (default shape): - `{"ok": true, "rows": [{col: val}, ...], "truncated": false}` plus any - requested extras. `truncated: true` when the result hit `max_returned_rows`. -- **Errors:** - - SQLite errors (e.g. `no such table`) are **not** raised — they surface as - HTTP 400 `{"ok": false, "error": "", "rows": [], "truncated": false}`. - - Time limit → 400 titled `"SQL Interrupted"` (the `error` value contains - an HTML fragment). - - `?sql=` omitted or blank → 400 `"?sql= is required"` for all data - formats (`.json`, `.csv`, plugin formats). The HTML page remains the - SQL editor. -- `.csv` streams CSV; unknown extensions → 404. - -### GET /\/-/query/parameters - -`QueryParametersView` (app.py:2687-2690; views/stored_queries.py:26-51). - -- **Permission:** `execute-sql` → 403 JSON - `{"ok": false, "errors": ["Permission denied: need execute-sql"]}`. -- **Parameters:** only `sql` (default `""`); any other key → 400 - `"Invalid keys: ..."`. -- **Response:** 200 `{"ok": true, "parameters": ["name1", ...]}`. SQL with a - parameter beginning `_` → 400 `"Magic parameters are not allowed"`. -- Responses carry `Content-Security-Policy: frame-ancestors 'none'` and - `X-Frame-Options: DENY`. - -### POST /\/-/create - -`TableCreateView` (app.py:2658; views/table_create_alter.py:785-962). -GET → 405. Body is parsed as JSON regardless of content type; invalid JSON → -400 `{"ok": false, "errors": ["Invalid JSON: ..."]}`. - -- **Permissions** (all denials → 403 canonical error JSON, - all checked at the **database** level): - - `create-table` — always required (`["Permission denied"]`) - - `insert-row` — if `rows`/`row` provided (`need insert-row`) - - `update-row` — if `replace: true` (`need update-row`) - - `alter-table` — if `alter: true` on an **existing** table - (`need alter-table`); when the table does not exist yet and rows are - supplied, alter is enabled automatically. -- **Request schema** (pydantic `CreateTableRequest`, extra keys forbidden → - 400 `"Invalid keys: a, b"`): - - `table` (required) — must match `^(?!sqlite_)[^\n]+$` - - `rows` (list of objects) / `row` (single object) — mutually exclusive - - `columns` — list of `{name, type, fk_table, fk_column, not_null, - default, default_expr}`; mutually exclusive with `rows`/`row`; `type` one - of `text`/`integer`/`float`/`blob` (default `text`); `default` and - `default_expr` mutually exclusive; `default_expr` one of - `current_timestamp`, `current_date`, `current_time`, `current_unixtime`, - `current_unixtime_ms`. At least one of `columns`/`rows`/`row` required. - - `pk` (string) / `pks` (list) — mutually exclusive. For an existing table - a differing pk → 400 `"pk cannot be changed for existing table"`. - - `ignore` / `replace` (bools) — mutually exclusive; require `row`/`rows` - and `pk`/`pks`. - - `alter` (bool) — add missing columns when inserting into an existing table. -- **Success** — **201**: - ```json - {"ok": true, "database": "...", "table": "...", - "table_url": "https://.../db/table", "table_api_url": "https://.../db/table.json", - "schema": "CREATE TABLE ...", "row_count": 2} - ``` - `row_count` only when rows were inserted. Write failures → 400 - `{"ok": false, "errors": [""]}`. Emits `create-table` / - `insert-rows` / `alter-table` events. - -### POST /\/-/execute-write - -`ExecuteWriteView` (app.py:2679-2682; views/execute_write.py:236-476). GET on -the same path renders an HTML form (requires `execute-write-sql`). - -- **Permission (POST):** `execute-write-sql` → 403 - `{"ok": false, "errors": ["Permission denied: need execute-write-sql"]}`; - immutable database → 403 `["Database is immutable"]`. -- **Per-statement permissions:** the SQL is analyzed - (`decision_for_write_sql_operation`, write_sql.py:63-189) and each - operation must pass: - - | Operation | Requirement | - |---|---| - | `select` / internal ops / function calls | ignored | - | read of a table | `view-table` on that table | - | `insert` or `update` | **all of** `insert-row`, `update-row`, `delete-row` on the table | - | `delete` | `delete-row` | - | `create table` | `create-table` on the database | - | `alter table`, `create index`, `drop index` | `alter-table` on the table | - | `drop table` | `drop-table` | - | `VACUUM`, virtual-table writes, shadow-table writes | rejected outright (403) | - | statements touching attached databases | rejected (403) | - -- **Body:** JSON (`{"sql": ..., "params": {...}}` — only those two keys) or - form-encoded (`sql` plus one field per parameter, `_sql_param_` prefix - stripped). Validation errors (400): `"SQL is required"`, - `"params must be a dictionary"`, `"Unknown parameters: a, b"`, - `"Magic parameters are not allowed"`, `"Could not analyze query: ..."`, - `"Use /-/query for read-only SQL; this endpoint only executes writes"`. -- **JSON is returned when** the body was JSON, `Accept: application/json`, or - a truthy `_json` field is present; otherwise HTML. -- **Success** — 200: - ```json - {"ok": true, "message": "Query executed, 1 row affected", "rowcount": 1, - "rows": [], "truncated": false, - "analysis": [{"operation": "insert", "database": "db", "table": "t", - "required_permission": "insert-row, update-row, delete-row", - "source": null}]} - ``` - `rows` is populated by `RETURNING` clauses. SQLite errors → 400 - `{"ok": false, "errors": [""]}`. Anti-framing headers on all - responses. - -### GET /\/-/execute-write/analyze - -`ExecuteWriteAnalyzeView` (app.py:2675-2678; views/execute_write.py:479-507). - -- **Permission:** `execute-write-sql` → 403 `errors` JSON. -- **Parameters:** only `sql` allowed (else 400 `"Invalid keys: ..."`). -- **Response** — 200 even when analysis fails (`ok: false` in body): - `{"ok", "parameters", "analysis_error", "analysis_rows": - [{operation, database, table, required_permission, source, allowed}], - "execute_disabled", "execute_disabled_reason"}`. `allowed` is a per-actor - permission check result (true/false/null). - -### GET /\/-/foreign-key-targets - -`DatabaseForeignKeyTargetsView` (app.py:2659-2662; -views/table_create_alter.py:965-1005). - -- **Parameter:** `table` (optional) — only used for the permission check. -- **Permission:** `create-table` on the database, **or** `alter-table` on - `?table=` when it names an existing table. Neither → 403 - `{"ok": false, "errors": ["Permission denied: need create-table"]}`. -- **Response:** 200 `{"ok": true, "database": "...", "targets": - [{"fk_table", "fk_column", "type"}]}` — every non-hidden table with exactly - one primary-key column; `type` is the pk's SQLite type affinity. - -### GET /\/-/schema(.json|.md) - -`DatabaseSchemaView` (app.py:2683-2686; views/special.py:1296-1329). - -- **Permission:** `view-database` (denied → `Forbidden` → 403 HTML). -- **Unknown database** → 404; for `.json`: - `{"ok": false, "error": "Database not found"}`. The permission check runs - first, so unauthorized actors cannot probe for database existence. -- **Responses:** `.json` → 200 `{"ok": true, "database": "", "schema": ""}` - (concatenated `sqlite_master.sql` joined with `;\n`); `.md` → - `text/markdown`; no extension → HTML. - ---- - -## Table and row read endpoints - -### GET /\/\.json - -Route `r"/(?P[^\/\.]+)/(?P
[^\/\.]+)(\.(?P\w+))?$"` → -`table_view` (app.py:2711-2714; views/table.py:1670). Serves both tables and -SQL views. GET/HEAD only — POST returns a plain-text 405. If the name is -neither a table nor a view but matches a stored query, the request is -dispatched to `QueryView` (views/table.py:1703-1712). - -**Permission:** `view-table` via `check_visibility`; denial raises -`Forbidden` → **HTML** 403 page even for `.json`. Unknown table → -`TableNotFound` → 404 (JSON error shape for `.json` paths). - -**Default JSON keys** (views/table.py:2308-2332 + renderer): - -| Key | Meaning | -|---|---| -| `ok` | `true` when data was retrieved without error | -| `next` | pagination token string, or `null` on the last page | -| `next_url` | absolute URL of the next page, or `null` on the last page | -| `rows` | list of row objects `{column: value}` (default `_shape=objects`) | -| `truncated` | always present; `false` for table pages | - -`columns` is computed but removed unless `?_extra=columns` was requested. -When there is a next page the response carries a -`Link: ; rel="next"` header (views/table.py:1911-1912). - -**`?_extra=` options** (TABLE scope; registry -views/table_extras.py:1197-1235; unknown names silently ignored): - -| `_extra=` | Returns | -|---|---| -| `count` | total matching-row count, computed with a `limit 10001` subquery so it caps at 10001; `null` with `_nocount` or on count timeout. Requesting `count` implicitly includes `count_truncated` | -| `count_truncated` | `true` when `count` hit the counting limit (the real count is at least the reported value) | -| `count_sql` | the SQL used for the count | -| `facet_results` | `{"results": {name: facet}, "timed_out": [...]}`; each facet: `{name, type, hideable, toggle_url, results: [{value, label, count, toggle_url, selected}], truncated}` | -| `facets_timed_out` | facet names that exceeded `facet_time_limit_ms` | -| `suggested_facets` | `[{name, toggle_url, (type)}]`; empty when suggestion is disabled or paginating | -| `human_description_en` | English description of filters + sort | -| `next_url` | absolute URL of the next page or `null` | -| `columns` | column names of the returned rows | -| `all_columns` | all table columns regardless of `_col`/`_nocol` | -| `primary_keys` | pk column names (empty for rowid tables and views) | -| `display_columns` | HTML-oriented column metadata | -| `render_cell` | per-row plugin-rendered HTML strings | -| `debug` | `{url_vars, resolved, nofacet, nosuggest}` — explicitly unstable | -| `request` | `{url, path, full_path, host, args}` | -| `query` | `{sql, params}` of the main query | -| `column_types` | `{column: {type, config}}` assigned column types | -| `set_column_type_ui` | UI helper, `null` unless actor has `set-column-type` | -| `metadata` | table metadata dict including column descriptions | -| `extras` | self-describing list of all available extras | -| `database`, `table`, `database_color` | identity/display values | -| `renderers` | `{format_name: url}` of formats that can render this data | -| `custom_table_templates` | template lookup list | -| `sorted_facet_results` | facets as a display-ordered list | -| `table_definition` | `CREATE TABLE` SQL | -| `view_definition` | `CREATE VIEW` SQL, `null` for tables | -| `is_view` | boolean | -| `private` | `true` if visible to this actor but not anonymously | -| `expandable_columns` | `[[foreign_key, label_column_or_null], ...]` | -| `form_hidden_args` | pairs of `_`-prefixed args for HTML forms | - -Non-public extras (`actions`, `filters`, `display_rows`) are HTML-only and -never appear in JSON. `_extra=_html` expands to the full HTML bundle -(views/table_extras.py:1162-1194). Any `_facet*` argument implicitly adds -`facet_results`; `_shape=object` implicitly adds `primary_keys` -(views/table.py:2252-2256). There is **no** `filtered_table_rows_count` -extra — it was replaced by `count`. - -**Column filters `?__=`** (filters.py:260-427). Any -querystring key not starting with `_` is a filter; bare `?column=value` means -`exact`. Columns whose names start with `_` can be filtered as -`?_col__exact=`. Operators: - -| op | SQL | -|---|---| -| `exact` | `"col" = :p` (default) | -| `not` | `"col" != :p` | -| `contains` / `notcontains` | `like '%v%'` / `not like '%v%'` | -| `endswith` / `startswith` | `like '%v'` / `like 'v%'` | -| `gt` / `gte` / `lt` / `lte` | `>` `>=` `<` `<=` (numeric strings cast to int) | -| `like` / `notlike` | raw `like` / `not like` pattern | -| `glob` | `glob` | -| `in` / `notin` | comma-separated list, or JSON array if the value starts with `[` | -| `arraycontains` / `arraynotcontains` | `[not] in (select value from json_each("col"))` (requires JSON1) | -| `date` | `date("col") = :p` | -| `isnull` / `notnull` | `is null` / `is not null` (no value) | -| `isblank` / `notblank` | `(is null or = '')` / opposite (no value) | - -**Special (underscore) parameters:** - -| Param | Behavior | -|---|---| -| `_where=SQL` | extra raw where clause (repeatable); requires `execute-sql` else 403 `"_where= is not allowed"` | -| `_search=q` | FTS against the table's FTS table | -| `_search_=q` | FTS restricted to one column; 400 if invalid | -| `_searchmode=raw` | pass the query straight to `match` | -| `_fts_table=` / `_fts_pk=` | override the FTS table / pk used for joins | -| `_through={"table","column","value"}` | filter via an incoming foreign key (repeatable, JSON value) | -| `_sort=col` / `_sort_desc=col` | sort; 400 if both given or column not sortable | -| `_next=token` | pagination token | -| `_size=N\|max` | page size; default `default_page_size` (100); `max` = `max_returned_rows` (1000); 400 on invalid | -| `_col=name` (repeatable) | return only pks + these columns; 400 on invalid | -| `_nocol=name` (repeatable) | exclude columns; 400 if invalid or a pk | -| `_labels=on` | expand every FK column into `{"value", "label"}` | -| `_label=col` (repeatable) | expand only the named FK column(s) | -| `_facet=col` | request a facet; 400 `"_facet= is not allowed"` when `allow_facet` off | -| `_facet_array=col` / `_facet_date=col` | typed facets | -| `_facet_size=N\|max` | facet bucket count, default 30, capped at `max_returned_rows` | -| `_nocount=1` | skip count (`count` extra → null) | -| `_nofacet=1` | skip facets and suggestions | -| `_nosuggest=1` | skip facet suggestions only | -| `_shape=` | see renderer section; `array`/`object` also force `_nocount` and `_nofacet` | -| `_nl=on` | NDJSON with `_shape=array` | -| `_json=col` / `_json_infinity=1` | renderer options | -| `_timelimit=ms` | custom SQL time limit | -| `_ttl=seconds` | `Cache-Control: max-age=N` (`0` → `no-cache`); default `default_cache_ttl` (5) | -| `_trace=1` | append `_trace` key (requires `trace_debug` setting) | -| `_extra=` | see above | - -**Pagination** is keyset-based for tables: `page_size + 1` rows are fetched; -`next` is built from the last row of the page — comma-joined tilde-encoded -primary-key values, prefixed by the sort value when sorted (`$null` for null -sort values) (views/table.py:2041-2111, 2421-2482). `next_url` is the -absolute URL with `_next` replaced. - -### GET /\/\.json (SQL views) - -Same code path with `is_view=True`. Differences: - -- No primary keys: `primary_keys` → `[]`; `_shape=object` fails; base query - has no `order by`. -- **Pagination is offset-based**: `_next` is an integer offset applied as - `limit N offset M` (views/table.py:2047-2049, 2438-2439) — unlike the - keyset tokens used for tables. -- `view_definition` returns the `CREATE VIEW` SQL; `table_definition` is null. - -### GET /\/\/\.json - -`RowView` (app.py:2715-2718; views/row.py:137). `` is comma-separated -tilde-encoded primary key values (rowid for rowid tables). - -- **Permission:** `view-table` (denied → `Forbidden` → 403 HTML). Missing row - → 404 `"Record not found: [...]"`. -- **Default JSON keys:** `ok`, `database`, `table`, `rows` (single-element - list), `primary_keys`, `primary_key_values`, `query_ms`, - `truncated: false`; `columns` only with `?_extra=columns`. -- **`?_extra=` (ROW scope):** `columns`, `primary_keys`, `render_cell`, - `debug`, `request`, `query`, `column_types`, `metadata`, `extras`, - `database`, `table`, `database_color`, `private`, `foreign_key_tables` - (incoming FKs with `count` and `link`; single-pk rows only). -- **Foreign-key label expansion does not apply to row JSON** — `_labels` has - no effect here; expansion happens only in the HTML path - (views/row.py:445-475). -- `_shape`, `_json`, `_nl`, `_json_infinity`, `_ttl` apply. - -### The .blob format - -`//
/.blob?_blob_column=col` (also on query pages) — -fetches raw binary bytes (blob_renderer.py:10-61). `_blob_column` required -(400 if missing/invalid); optional `_blob_hash` must equal the value's -SHA-256 (else 400 `"Link has expired..."`). Returns `application/binary` as a -download attachment. In JSON output, binary cells appear as -`{"$base64": true, "encoded": "..."}`. - -### GET /\/\/-/schema(.json|.md) - -`TableSchemaView` (app.py:2751-2754; views/special.py:1332-1378). - -- **Permission:** `view-table` via `ensure_permission` (denied → 403 HTML). -- **Responses:** `.json` → 200 `{"ok": true, "database", "table", "schema"}`; - `.md` → `text/markdown`; no extension → HTML. Missing table → 404 - `{"ok": false, "error": "Table not found"}` for `.json`. - -### GET /\/\/-/fragment - -`TableFragmentView` (app.py:2739-2742; views/table.py:1385-1418). -**HTML-only** — returns the `_table.html` partial; no JSON variant. Accepts -table querystring parameters plus `_row=` to render a single row. - -### GET /\/\/-/autocomplete - -`TableAutocompleteView` (app.py:2743-2746; views/table.py:1492-1595). Tables -only — views get 400 `"Autocomplete is only available for tables"`. - -- **Permission:** `view-table` (denied → `Forbidden` → 403). -- **Parameters:** `q` (matched with escaped `LIKE %q%` against pk columns and - the label column) and `_initial` (truthy: with empty `q`, return the 10 - most recent rows). Neither → `{"ok": true, "rows": []}`. -- **Response:** `{"rows": [{"pks": {pk_name: value}, "label": "..."}]}` — max - 10 items; 500 ms query budget with fallbacks, timing out to - `{"ok": true, "rows": []}`. - ---- - -## The write API - -All write endpoints return errors via `_error()` (the canonical error -shape) and check permissions with -`datasette.allowed()` directly, so their 403s are JSON (unlike the -`Forbidden`-raising read endpoints). Routes: app.py:2719-2762. - -### POST /\/\/-/insert - -`TableInsertView` (views/table.py:907-1194). - -- **Permissions:** `insert-row` on the table (denied → 403 - `["Permission denied"]`); `update-row` additionally required for - `replace: true` (403 `need update-row to use "replace"`); `alter-table` - additionally required for `alter: true` (403 - `Permission denied for alter-table`). Immutable database → 403 - `Database is immutable`. -- **Request** — the body is parsed as JSON regardless of the request - `Content-Type` header (invalid JSON → 400). Body: - - | Field | Rules | - |---|---| - | `row` | single object; mutually exclusive with `rows`; forces `return: true` | - | `rows` | list of objects; max `max_insert_rows` (default 100), else 400 `"Too many rows, maximum allowed is 100"` | - | `ignore` | skip rows whose pk already exists; mutually exclusive with `replace` | - | `replace` | replace rows with matching pks (needs `update-row`) | - | `alter` | add missing columns (needs `alter-table`) | - | `return` | include inserted rows in the response | - - One of `row`/`rows` required. Unknown keys → 400 `"Invalid parameter: ..."`. - Unless `alter`, row keys must be existing columns → per-row 400 - `"Row 0 has invalid columns: x, y"`. Values are validated against assigned - column types. -- **Response** — **201** `{"ok": true}`; with `return: true` also `rows` - (the rows as stored, re-fetched by rowid). SQLite errors during the write → - 400 with the message. Emits `insert-rows` (and possibly `alter-table`) - events. - -### POST /\/\/-/upsert - -`TableUpsertView` — subclasses insert (views/table.py:1197-1201). - -- **Permissions:** **both** `insert-row` and `update-row` (403 - `need both insert-row and update-row`); `alter: true` needs `alter-table`. -- **Request:** same as insert, except `ignore`/`replace` are rejected (400 - `"Upsert does not support ignore or replace"`) and **every row must contain - the table's primary key(s)** (per-row 400 - `Row 0 is missing primary key column(s): "id"` / `has null primary key`). -- **Response** — **200** (note: insert returns 201) `{"ok": true}`; with - `return: true`, `rows` re-fetched by pk. Emits `upsert-rows`. - -### POST /\/\/-/alter - -`TableAlterView` (views/table_create_alter.py:1130-1353). - -- **Permission:** `alter-table` (403 `need alter-table`); immutable → 403. -- **Request:** `{"operations": [{"op": ..., "args": {...}}, ...]}` — a - non-empty list, validated by pydantic (extra keys forbidden anywhere; - errors → 400 `location: message`): - - | `op` | `args` | - |---|---| - | `add_column` | `name` (required), `type` (`text`/`integer`/`float`/`blob`, default `text`), `not_null`, `default` xor `default_expr`; `not_null: true` requires a default | - | `rename_column` | `name`, `to` | - | `rename_table` | `to` (must not start `sqlite_`) | - | `alter_column` | `name` + at least one of `type`, `not_null`, `default`, `default_expr` | - | `drop_column` | `name` | - | `set_primary_key` | `columns` (non-empty list) | - | `reorder_columns` | `columns` (non-empty list) | - | `add_foreign_key` | `column`, `fk_table`, optional `fk_column` | - | `drop_foreign_key` | `column` | - | `set_foreign_keys` | `foreign_keys`: list of `{column, fk_table, fk_column?}` | - - `default_expr` must be one of the five `current_*` keywords. Operations are - applied in a single write transaction; any failure → 400. -- **Response** — 200: - ```json - {"ok": true, "database": "...", "table": "", - "table_url": "...", "table_api_url": "...", - "altered": true, "schema": "...", "before_schema": "...", - "operations_applied": 2} - ``` - -### POST /\/\/-/drop - -`TableDropView` (views/table.py:1320-1382). - -- **Permission:** `drop-table` (403 `Permission denied`); immutable → 403. -- **Confirmation flow:** without `{"confirm": true}` in the body, nothing is - dropped and a 200 preview is returned: - `{"ok": true, "database", "table", "row_count", - "message": "Pass \"confirm\": true to confirm"}`. With `confirm: true` → - 200 `{"ok": true}`. Emits `drop-table`. - -### POST /\/\/-/set-column-type - -`TableSetColumnTypeView` (views/table.py:1204-1317). Assigns a Datasette -*column type* (metadata stored in the internal `column_types` table) — it -does not change the SQLite schema. - -- **Permission:** `set-column-type` (403 `Permission denied`). -- **Request**: `{"column": "name", - "column_type": {"type": "url", "config": {...}?} | null}`. Unknown - keys/invalid structure → detailed 400 errors; unknown type → 400 - `"Unknown column type: x"`. Default registered types (via the - `register_column_types` hook): `url`, `email`, `json`, `textarea`. -- **Response** — 200 `{"ok": true, "database", "table", "column", - "column_type": {...} | null}`. - -### GET /\/\/-/foreign-key-suggestions - -`TableForeignKeySuggestionsView` (views/table_create_alter.py:1008-1127). -**GET only** (read-only despite living beside the write endpoints). - -- **Permission:** `alter-table` (403 `need alter-table`); views → 400 - `"Cannot suggest foreign keys for a view"`. -- **Response** — 200: `{"ok": true, "database", "table", - "row_check": {attempted, status, row_limit, sampled_rows, checked_options}, - "columns": [{column, type, affinity, current, - "suggestions": [{fk_table, fk_column, confidence, sampled_values, reasons}], - "options": [...]}]}`. Samples up to 500 rows within 50 ms/200 ms budgets. - -### POST /\/\/\/-/update - -`RowUpdateView` (views/row.py:781-870). - -- **Permissions:** `update-row` (403 `Permission denied`); `alter: true` - additionally requires `alter-table` (403 - `Permission denied for alter-table`). -- **404s:** `Database not found: x` / `Table not found: x` / - `Record not found: [pks]`. -- **Request:** `{"update": {column: value, ...}, "return"?: true, - "alter"?: true}`. Missing/non-dict `update` → 400 - `"JSON must contain an update dictionary"`; unknown keys → 400 - `"Invalid keys: ..."`; write failures (bad column, constraint violation) → - 400 with the message. -- **Response** — 200 `{"ok": true}`; with `return: true`, - `{"ok": true, "rows": [{...}]}` — a single-item list, matching - insert/upsert. Emits `update-row`. - -### POST /\/\/\/-/delete - -`RowDeleteView` (views/row.py:738-778). - -- **Permission:** `delete-row` (403 `Permission denied`). 404s as update. -- **Request:** no body required (any body is ignored — there is no - confirmation step, unlike table drop). -- **Response** — 200 `{"ok": true}`; with `?_redirect_to_table` a `redirect` - key is added. A failure during the write returns 400 with the message, - matching update. Emits `delete-row`. - ---- - -## Stored (canned) queries API - -Stored queries live in the internal database's `queries` table -(utils/internal_db.py:116-133). Queries defined in `datasette.yaml` are -synced in at startup with `source="config"` and `is_trusted` defaulting to -true; queries created via the API get `source="user"`, `is_trusted=false`, -`owner_id` = actor id. - -**Canonical stored-query JSON object** (`stored_query_to_dict`, -stored_queries.py:55-80): - -```json -{ - "database": "...", "name": "...", "sql": "...", - "title": null, "description": null, "description_html": null, - "hide_sql": false, "fragment": null, - "parameters": ["p"], - "is_write": false, "is_private": true, "is_trusted": false, - "source": "user", "owner_id": "...", - "on_success_message": null, "on_success_message_sql": null, - "on_success_redirect": null, - "on_error_message": null, "on_error_redirect": null, - "private": true -} -``` - -`private` appears only in list responses. On input (create/update and -`datasette.yaml`), `params` is accepted as an alias for `parameters`. - -**Default permission rules for queries** (default_permissions/defaults.py): -`view-query` is default-allow, but private queries are visible only to their -owner; the owner may `update-query`/`delete-query` their `source='user'` -queries. - -### GET /-/queries(.json) and GET /\/-/queries(.json) - -`GlobalQueryListView` / `QueryListView` (app.py:2606-2609, 2663-2666; -views/stored_queries.py:69-238). The global variant lists queries across all -databases (`database`/`database_color` are null, `show_database` true). - -- **Permissions:** no single gate; results filtered per query by - `view-query` (private queries appear only for their owner). -- **Parameters:** `_size` (default 20 HTML / **50 JSON**; accepts `max`; - values over `max_returned_rows` or non-integers → 400, matching table - `_size` semantics), `_next` (cursor), `q` (substring search over - name/title/description/sql), `is_write` / `is_private` (booleans; invalid → - 400 `"is_write must be 0 or 1"`), `source`, `owner_id`. -- **Response** — 200: - `{"ok": true, "database", "database_color", "queries": [...], "next", - "next_url", "limit", "show_private_note", - "show_trusted_note", "query_list_path", "show_database", - "facets": [{title, items: [{label, count, href, active}]}], - "filters": {q, is_write, is_private, source, owner_id}}`. - -### GET /\/-/queries/analyze - -`QueryCreateAnalyzeView` (app.py:2667-2670; views/stored_queries.py:290-322). -**GET only** despite being an "analyze" action — POST → 405. - -- **Permissions:** `execute-sql` then `store-query` (each denial → 403 - `errors` JSON). -- **Parameters:** only `sql` (others → 400 `"Invalid keys: ..."`). -- **Response** — 200: `{"ok", "parameters", "analysis_error", - "analysis_rows": [{operation, database, table, required_permission, - source, allowed}], "has_sql", "analysis_is_write", "save_disabled"}`. - -### POST /\/-/queries/store - -`QueryStoreView` (app.py:2671-2674; views/stored_queries.py:325-388). GET on -the same path renders the HTML create form. - -- **Permissions:** `execute-sql` + `store-query` (403 `errors` JSON). -- **Request:** JSON bodies must wrap the fields: - `{"query": {...fields...}}`; form bodies pass fields flat. Fields: - `name` (required; `^[^/\.\n]+$`; conflicts with tables/views or existing - queries → 400), `sql` (required; read SQL must pass `validate_sql_select`; - write SQL must pass per-operation permission checks), `title`, - `description`, `hide_sql`, `fragment`, `parameters`/`params` (must exactly - match the SQL's named parameters; magic parameters rejected), - `is_private` (**default true**), and — only for write SQL — - `on_success_message`, `on_success_redirect`, `on_error_message`, - `on_error_redirect`. `is_write` is derived from SQL analysis; - `is_trusted`, `description_html` and `on_success_message_sql` cannot be - set through this API. -- **Response:** JSON request → **201** `{"ok": true, "query": {...}}`; form - request → 302 redirect. - -### GET /\/\/-/definition - -`QueryDefinitionView` (app.py:2695-2698; views/stored_queries.py:391-408). - -- **Permission:** `view-query` (403 `["Permission denied"]`). -- **Response:** 200 `{"ok": true, "query": {...}}`; 404 - `["Query not found: x"]`. - -### GET/POST /\/\/-/edit - -`QueryEditView` (app.py:2699-2702) — **HTML form endpoint** -(`has_json_alternate = False`), not part of the JSON API. Programmatic -updates use `/-/update`. - -### POST /\/\/-/update - -`QueryUpdateView` (app.py:2703-2706; views/stored_queries.py:411-465). - -- **Permissions:** `update-query` (403 `need update-query`); trusted queries - → 403 `"Trusted queries cannot be updated using the API"`; changing `sql` - additionally requires `execute-sql`. -- **Request:** `{"update": {...partial fields...}, "return"?: true}` — other - top-level keys → 400. Updatable fields: `sql`, `title`, `description`, - `hide_sql`, `fragment`, `parameters`/`params`, `is_private`, `on_*` - fields (write SQL only). New SQL is re-analyzed and `is_write` recomputed. -- **Response:** 200 `{"ok": true}` (plus `query` with `return: true`); 404 - `"Query not found: x"`. - -### POST /\/\/-/delete - -`QueryDeleteView` (app.py:2707-2710; views/stored_queries.py:594-644). GET -renders an HTML confirmation page. - -- **Permission:** `delete-query` (403 `need delete-query`). Trusted - queries → 403 `"Trusted queries cannot be deleted using the API"`, - matching update. -- **Response:** JSON request → 200 `{"ok": true}`; form → 302; 404 - `"Query not found: x"`. No `confirm` field required (unlike table drop). - -### GET/POST /\/\(.json) — executing a stored query - -No dedicated route: the table route resolves the name, and on `TableNotFound` -the request is dispatched to `QueryView` when a stored query matches -(views/table.py:1698-1712). Covers both config-defined and API-stored -queries. - -**GET (read queries)** — `QueryView.get` (views/database.py:695-1130): - -- **Permissions:** `view-query` (denied → `Forbidden` → 403 HTML). Read - queries then require `execute-sql` unless `is_trusted`. Write queries are - **not executed** on GET — JSON returns empty `rows`; HTML shows a POST form. -- **Parameters:** each named `:param` is read from the query string (missing - → `""`); `_timelimit`; renderer options (`_shape`, `_nl`, `_json`, - `_json_infinity`); `_extra` (QUERY scope). -- **Response:** `{"ok": true, "rows": [...], "truncated": false}` + extras. - SQL errors → 400 with `error` in the envelope. - -**POST (write queries)** — `QueryView.post` (views/database.py:574-693): - -- **Permissions:** `view-query`; then, unless `is_trusted`: - `execute-write-sql` on the database **plus** per-operation write - permissions (same table as `/-/execute-write`). Rejection → 403 - `{"ok": false, "message": "...", "redirect": null}` for JSON clients. - Immutable database → 403. -- **Body:** form-encoded or JSON `param=value` pairs (values coerced to - strings). -- **JSON is returned when** `Accept: application/json`, `?_json=1`, or a - `_json` body field is present; otherwise 302 + flash message. -- **Magic parameters** (`:__`, resolved server-side; registered - via `register_magic_parameters`, default_magic_parameters.py): - `_now_epoch`, `_now_date_utc`, `_now_datetime_utc`, `_actor_`, - `_random_chars_`, `_cookie_`, `_header_` (underscores → - hyphens). User-stored queries cannot contain magic parameters — they are a - feature of config/trusted queries. -- **Response:** success → 200 - `{"ok": true, "message": "...", "redirect": "..."|null}` — `message` - honors `on_success_message_sql` / `on_success_message`, falling back to - `"Query executed"` or `"Query executed, N rows affected"`. SQL failure → - **400** canonical error (message honors `on_error_message`) plus a - `redirect` context key from `on_error_redirect`. Operation rejection - (`QueryWriteRejected`, e.g. VACUUM) → 403 canonical error plus - `redirect: null`. - ---- - -## Authentication and tokens - -### Bearer tokens (`dstok_`) - -Signed API tokens are sent as `Authorization: Bearer dstok_...`. The -`actor_from_signed_api_token` hook (default_permissions/tokens.py:25-40) -passes the token to `datasette.verify_token()`, which tries every handler -registered via `register_token_handler`; the default is -`SignedTokenHandler` (tokens.py:117-193). - -- **Format:** `dstok_` + itsdangerous-signed payload (namespace `token`) - containing `a` (actor id), `t` (creation Unix time), optional `d` - (duration seconds), optional `_r` (restrictions). -- **Verification:** a `dstok_`-prefixed token that fails verification — - `allow_signed_tokens` off, invalid signature, missing/non-integer `t`, - malformed `d`, or expired — raises `TokenInvalid`, and the request fails - with **401**, the canonical error body and a - `WWW-Authenticate: Bearer error="invalid_token"` header (even if a valid - `ds_actor` cookie is also present). Tokens with prefixes no registered - handler recognizes are ignored (they may belong to an auth plugin). The - effective duration is `d` capped by `max_signed_tokens_ttl` (default 0 = - no cap; a non-zero setting also imposes a TTL on tokens without `d`). -- **Resulting actor:** `{"id": , "token": "dstok"}` plus `"_r"` and - `"token_expires"` when applicable. - -**Restrictions (`_r`)** (default_permissions/restrictions.py): - -- `"a"`: list of actions allowed on any resource -- `"d"`: `{database_name: [actions]}` -- `"r"`: `{database_name: {table_name: [actions]}}` - -Actions are stored as abbreviations when available (see appendix); checks -accept either the full name or the abbreviation. Restrictions are an -allowlist filter layered on top of normal permission resolution — a -restricted token can never do more than its allowlist, and never more than -the underlying actor could do anyway. - -### Token creation - -- **`/-/create-token`** is an HTML form endpoint only (see the instance - section) — there is no JSON API to mint tokens in this codebase. -- Programmatic alternatives: the `datasette create-token` CLI command and - the `datasette.create_token()` Python API. -- `/-/auth-token` is the one-time `--root` login mechanism, unrelated to API - tokens. - -### Cookie authentication - -Browser sessions use the signed `ds_actor` cookie (set by `/-/auth-token`, -plugins, or login flows; cleared by `/-/logout`). API POSTs from browsers are -subject to the cross-origin checks described in -[CSRF](#csrf--cross-origin-protection). - ---- - -## Appendix: registered actions (permissions) - -From `datasette/default_actions.py` (registered via the `register_actions` -hook). Token restrictions store the abbreviation when available. - -| Action | Abbr | Resource level | Notes | -|---|---|---|---| -| `view-instance` | `vi` | global | | -| `permissions-debug` | `pd` | global | gates the debug endpoints | -| `debug-menu` | `dm` | global | UI only | -| `view-database` | `vd` | database | | -| `view-database-download` | `vdd` | database | `also_requires="view-database"` | -| `execute-sql` | `es` | database | `also_requires="view-database"`; denied when the `default_allow_sql` setting is off | -| `execute-write-sql` | `ews` | database | `also_requires="view-database"` | -| `create-table` | `ct` | database | | -| `store-query` | `sq` | database | `also_requires="execute-sql"` | -| `view-table` | `vt` | table | | -| `insert-row` | `ir` | table | | -| `delete-row` | `dr` | table | | -| `update-row` | `ur` | table | | -| `alter-table` | `at` | table | | -| `set-column-type` | `sct` | table | | -| `drop-table` | `dt` | table | | -| `view-query` | `vq` | query | default-allow; private queries restricted to their owner | -| `update-query` | `uq` | query | query owner allowed by default (source=`user` only) | -| `delete-query` | `dq` | query | query owner allowed by default (source=`user` only) | diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md deleted file mode 100644 index 0d817612..00000000 --- a/stable-api-recommendations.md +++ /dev/null @@ -1,469 +0,0 @@ -# Datasette 1.0 Stable API — Consistency and Completeness Review - -This review is based on `existing-api.md`, which documents the JSON API as -actually implemented in this codebase (`1.0a35`), derived from source. The -goal here is to identify everything that should be made consistent, fixed, or -explicitly scoped out **before** the 1.0 stability promise takes effect — -because after 1.0, every inconsistency below becomes a compatibility -commitment. - -Findings are grouped by theme. Each carries a priority: - -- **P1 — should block 1.0**: breaking to fix later, or a correctness/security - concern. -- **P2 — strongly recommended**: fixable later only via awkward additive - changes. -- **P3 — nice to have / documentation decision**: can be resolved by - documenting the behavior as intentional. - ---- - -## 1. Error responses: four shapes is three too many (P1) — ✅ IMPLEMENTED - -> **Status:** implemented. All four shapes now delegate to a shared -> `error_body()` helper (`datasette/utils/__init__.py`) producing -> `{"ok": false, "error": "", "errors": [...], "status": }`. -> The `title` key is no longer emitted in JSON; the bare `{"error": ...}` -> debug-endpoint shape is gone; `_shape=object` misuse now returns HTTP 400 -> (part of §1b). Covered by `tests/test_error_shape.py` and documented in -> the "Error responses" section of `docs/json_api.rst`. §1a (`Forbidden` → -> JSON) and §1b (write canned-query 200) are now also implemented. Still -> open from this section's sub-items: the §1c status outliers. - -The API currently produces four distinct JSON error shapes depending on which -internal layer generates the error: - -| Shape | Producer | Example endpoints | -|---|---|---| -| `{"ok": false, "error", "status", "title"}` | exception handler (handle_exception.py:50-53) | 404s and `DatasetteError`s on any `.json` path | -| `{"ok": false, "errors": [...]}` | `_error()` helper (views/base.py:183-184) | all write endpoints, stored-query endpoints, execute-write | -| `{"ok": false, "error", "rows": [], "truncated": false}` | JSON renderer (renderer.py:52-56) | SQL errors on table/query reads | -| `{"error": "..."}` (no `ok`) | permission debug views (views/special.py) | `/-/allowed`, `/-/rules`, `/-/check`, POST `/-/permissions` | - -Additionally, write canned queries report failure via a **fifth** vocabulary: -`{"ok": false, "message": ..., "redirect": ...}` with HTTP **200** -(views/database.py:678-690). - -A 1.0 client cannot write a single error handler today. **Recommendation:** -pick one canonical error object — the singular/plural tension is easiest to -resolve as: - -```json -{"ok": false, "error": "human-readable summary", "errors": ["detail", "..."], "status": 400} -``` - -where `errors` is optional and `error` is always present — and route every -error path through it (including the `forbidden` and `handle_exception` -defaults). At minimum, eliminate the bare `{"error": ...}` shape and the -`status`/`title` keys nobody else emits (`title` is a template-rendering -concern that leaked into the API). - -### 1a. `Forbidden` returns an HTML 403 to JSON clients (P1) — ✅ IMPLEMENTED - -> **Status:** implemented — the default `forbidden()` hook now returns the -> canonical JSON error for requests whose path ends in `.json` or that send -> `Accept: application/json` / `Content-Type: application/json`. - -Read endpoints that deny access via `ensure_permission`/`check_visibility` -raise `Forbidden`, and the default `forbidden()` hook renders an **HTML error -page even for `.json` requests** (forbidden.py:4-19, app.py:2895-2904). So: - -- `GET /db/table.json` without `view-table` → 403 **HTML** -- `POST /db/table/-/insert` without `insert-row` → 403 **JSON** - -A JSON client gets unparseable output precisely when it most needs a -machine-readable answer. **Recommendation:** the default forbidden handler -must return the canonical JSON error when the path ends in `.json` or the -request prefers JSON, mirroring `handle_exception`. - -### 1b. Errors that return HTTP 200 (P1) — ✅ IMPLEMENTED - -> **Status:** implemented. `_shape=object` misuse returns 400 (done with -> §1), and write canned-query SQL failures now return **400** with the -> canonical error shape (plus the `redirect` context key); the -> `QueryWriteRejected` 403 branch also uses the canonical shape. - -- `_shape=object` on a query or pk-less table → `{"ok": false, "error": - "_shape=object is only available on tables"}` with **200** - (renderer.py:73-90), while an unknown `_shape` value returns **400** - (renderer.py:101-108). Same class of error, different status. -- Write canned-query SQL failure → **200** `{"ok": false, "message": ...}` - (views/database.py:683-690), while the equivalent failure on - `/-/execute-write` returns **400**. - -**Recommendation:** all `ok: false` responses should carry a 4xx/5xx status. -(`/-/execute-write/analyze` returning `ok: false` with 200 for "analysis -completed, SQL is invalid" is defensible but should then not reuse the `ok` -key — see §2.) - -### 1c. Wrong-status outliers (P2) — ✅ IMPLEMENTED - -- ~~Row **delete** write failures return **500** (views/row.py:757) while row - **update** write failures return **400** (views/row.py:832-835). Same - failure class, different status; pick 400 (or 409 for constraint - violations) for both.~~ ✅ **Done** — delete now returns 400, matching - update and the rest of the write API. -- ~~Invalid or expired bearer tokens silently degrade the request to anonymous, - so clients see a 403 permission error (or worse, anonymous-permitted data) - rather than a 401 (tokens.py:147-193). For 1.0, a malformed/expired - `Authorization: Bearer dstok_...` header should produce **401** with a - distinguishable error, so clients can tell "renew your token" apart from - "you lack permission".~~ ✅ **Done** — token handlers can raise - `TokenInvalid`; Datasette responds 401 with the canonical body and a - `WWW-Authenticate: Bearer error="invalid_token"` header. Unrecognized - token prefixes still fall through to anonymous so auth plugins keep - working. - ---- - -## 2. Success envelope: `ok` is not universal, arrays are not extensible (P1/P2) — ✅ IMPLEMENTED (§2a-2c open) - -> **Status:** recommendations 1-3 are implemented. Every JSON-object -> success response now includes `"ok": true` (`JsonDataView` injects it for -> dict responses; homepage, jump, schema, permission-debug and autocomplete -> views set it explicitly), and the three top-level-array endpoints now -> return objects: `/-/plugins` → `{"ok": true, "plugins": [...]}`, -> `/-/databases` → `{"ok": true, "databases": [...]}`, `/-/actions` → -> `{"ok": true, "actions": [...]}`. Covered by -> `tests/test_success_envelope.py`. The sub-findings §2a (collection -> representations), §2b (`_extra`/`_shape` coverage) and §2c (count -> truncation) remain open. - -Endpoints disagree about the success envelope: - -- **Have `ok: true`:** table/row/query reads, database view, all write - endpoints, stored-query endpoints, `/-/allowed`-style debug data. -- **No `ok` key:** `/-/versions`, `/-/settings`, `/-/config`, `/-/threads`, - `/-/actor`, `/-/jump`, `/-/schema` variants (`{"database", "schema"}`, - `{"schemas": [...]}`), table `/-/schema.json`, `/-/autocomplete` - (`{"rows": []}`), homepage `/.json`. -- **Top-level JSON arrays:** `/-/plugins`, `/-/databases`, `/-/actions` - (app.py:2247-2304). A top-level array can never grow a sibling key - (pagination, warnings, `ok`) without a breaking change. - -**Recommendations:** - -1. (P1) Wrap the three array endpoints in objects before 1.0: - `{"ok": true, "plugins": [...]}` etc. This is the single cheapest - future-proofing fix in this list. -2. (P2) Add `ok: true` to every JSON-object success response, or explicitly - document that `ok` only exists on data endpoints. Half-consistency is the - worst outcome. -3. (P2) `/db/-/schema.json` (`{"database", "schema"}`) and - `/db/table/-/schema.json` should match the envelope style of their sibling - endpoints (they are also the only data endpoints whose 404 uses the - exception shape but whose success has no `ok`). - -### 2a. Collection representations disagree (P2) - -- ~~Homepage `/.json` returns `databases` as an **object keyed by name** - (index.py:147-161); `/-/databases.json` returns an **array**; the database - page returns `tables` as an array. Choose arrays-of-objects everywhere - (objects-keyed-by-name break when names need ordering or pagination).~~ - ✅ **Done** — the homepage returns a list, matching `/-/databases.json`. - The homepage JSON remains deliberately undocumented. -- ~~Insert/upsert with `return: true` respond with `rows` (plural, list); row - update with `return: true` responds with `row` (singular, object) - (views/row.py:837-844). Pick one (`rows` everywhere, even for one row, - matches the read API).~~ ✅ **Done** — row update now returns - `rows: [{...}]`. - -### 2b. `_extra`/`_shape` support is uneven (P2) — partially implemented - -> **Status:** unknown `_extra` names on data formats now return 400 -> `Unknown _extra: ` (HTML pages still ignore them). Extending -> extras/shaping to database/instance scope remains open. - -The extras system (`?_extra=`, scope-registered) is the 1.0 mechanism for -response shaping — but it only exists on table, row and query endpoints. The -database view builds JSON by hand and supports **neither `_extra` nor -`_shape`** (views/database.py:189-212); the homepage likewise. Either extend -extras to database/instance scope before 1.0 or document clearly that shaping -is a table/row/query feature. Also decide the contract for **unknown -`_extra` names, which are currently silently ignored** (extras.py:116-122) — -silent ignoring means typos return the default payload with no signal; -recommend a 400 or a `warnings` key. - -### 2c. Count truncation is invisible in JSON (P2) — ✅ IMPLEMENTED - -> **Status:** implemented — a public `count_truncated` extra now exists and -> is implicitly included whenever `count` is requested. - -The `count` extra is computed with a `limit 10001` subquery, so `count: -10001` actually means "at least 10001" — the `count_truncated` flag exists -but only in the HTML template context, never in JSON (views/table.py: -2334-2337). Expose it (e.g. make `count` be `null` + add `count_estimate`, -or add `count_truncated` to the JSON) before clients start trusting the -number. - ---- - -## 3. Pagination: three mechanisms, two contracts (P2) — partially implemented - -> **Status:** `next_url` now accompanies `next` in the default table JSON -> keys (previously it required `?_extra=next_url`), so every response with -> a `next` token also carries the ready-to-follow URL. Pagination tokens -> are deliberately left undocumented as to their internal structure. -> `_size` is now the single page-size parameter with uniform table-style -> semantics everywhere: query lists accept `max` and 400 on out-of-range -> values (previously silently clamped), and the `/-/allowed` and -> `/-/rules` debug endpoints renamed `page`/`page_size` to -> `_page`/`_size` with the same validation (400 instead of silent -> capping at 200). `has_more` has been **removed** from the query-list -> JSON — `next: null` is the single end-of-results signal everywhere, -> keeping default response keys minimal (`total` remains a debug-endpoint -> nicety). Fixing this also uncovered and fixed a bug where the query -> list's JSON `next_url` pointed at the HTML page (it dropped the `.json` -> extension) and was relative where the table `next_url` is absolute. -> §3 is now fully resolved. - -| Endpoint | Mechanism | Token | Extras | -|---|---|---|---| -| Table `.json` | keyset | tilde-encoded pk/sort values in `_next` | `next` always in body, `next_url` via `_extra`, `Link: rel=next` header | -| SQL view `.json` | **offset** | integer in the same `_next` parameter | same envelope | -| `/-/queries` lists | keyset | cursor in `_next` | `next`, `next_url`, **`has_more`** in body | -| `/-/allowed`, `/-/rules` | **page numbers** | `page`/`page_size` | `total`, `next_url`, `previous_url` | - -Concerns: - -1. The same `_next` parameter means "start after key" on tables but "row - offset" on views. Offset pagination over views is also O(n) and skews - under concurrent writes. If unifiable, unify; if not, document loudly. -2. `has_more` exists on query lists but not table pages; `total` exists on - debug endpoints but not elsewhere. Standardize the pagination block - (suggest: `next`, `next_url` — nullable — everywhere; treat `has_more` as - `next != null`). -3. Page-size parameters: `_size` (default 100, `max` keyword allowed) on - tables; `_size` (default 50 JSON, clamped 1–1000, no `max` keyword) on - query lists; `page_size` (default 50, silently capped at 200) on debug - endpoints. Align names, defaults and the cap behavior (silent capping vs - 400) as far as practical. - ---- - -## 4. HTTP semantics (P2) - -- ~~**201 vs 200:** insert → 201, upsert → 200 (views/table.py:1194), create - table → 201, store query → 201. Insert-201/upsert-200 is defensible - (upsert may not create) but it is undocumented subtlety; state it, or - return 200 for both with an explicit `created` count.~~ ✅ **Done** — - documented as deliberate in the upsert docs. -- **Destructive-action confirmation is asymmetric:** table drop requires - `{"confirm": true}` and has a preview response (views/table.py:1346-1365); - row delete executes immediately and ignores the body; query delete - executes immediately. Decide the 1.0 rule (suggestion: confirmation only - for schema-destroying operations, i.e. keep as is — but document it as a - deliberate contract). -- ~~**Content-type enforcement is inconsistent:** `/-/insert`, `/-/upsert`, - `/-/alter`, `/-/set-column-type` demand `Content-Type: application/json` - (400 otherwise); `/-/create` parses the body as JSON regardless of - content type; execute-write and the query CRUD endpoints accept both JSON - and form encodings. Pick one rule for JSON-only endpoints.~~ ✅ **Done** - — the lenient rule won: JSON-only write endpoints parse the body as JSON - regardless of `Content-Type` (CSRF protection comes from the - cross-origin header checks, not content types). This also fixed a 500 on - insert when the header was absent entirely. -- **JSON-vs-HTML negotiation on POST differs per endpoint:** execute-write - and canned queries key off `Accept: application/json` / a `_json` body - field; the write API keys off nothing (always JSON); query store keys off - request content type. A single documented rule ("responses are JSON if the - request body was JSON or `Accept: application/json`") would cover all of - them. -- **Endpoints named like actions but served over GET:** - `/-/queries/analyze`, `/-/execute-write/analyze`, - `/-/foreign-key-suggestions`, `/-/query/parameters` are all GET (correct, - they are reads) — fine, but `analyze` under a POST-shaped path invites - wrong calls; make sure 405 responses for POST on these return the JSON 405 - shape (they do only when the path ends `.json` or content type is JSON — - a JSON POST to `/-/queries/analyze` gets JSON, a form POST gets text). - ---- - -## 5. Naming and parameter conventions (P2/P3) - -- ~~**`params` and `parameters` are duplicate keys** in every stored-query - object (stored_queries.py:55-80). Delete one before 1.0 (suggest keeping - `parameters`; the write side already accepts both on input).~~ - ✅ **Done** — output objects carry only `parameters` (matching - `/-/query/parameters` and the analyze endpoints); `params` remains an - accepted input alias for API creation and `datasette.yaml` config. -- **Three names for the same concept across error/message payloads:** - `error`, `errors`, `message`. See §1. -- ~~**Boolean query parameters have at least three grammars:** `_nl=on`, - `_labels=on/off`, `?all=1`, `is_write=1|0|true|false|t|f|yes|no|on|off`, - `_nocount=1`. Adopt one accepted set (the query-list parser at - query_helpers.py:81-94 is a good candidate) and apply it everywhere.~~ - ✅ **Documented** — the JSON API docs state the canonical grammar - (`on/true/1`, `off/false/0`), which `value_as_boolean` already accepts - everywhere it is used. -- ~~**`.jsono`** survives on the homepage route (identical output to `.json`) - and as a row-view redirect. Remove it at 1.0; it is pure legacy.~~ - ✅ Removed: the homepage routes only accept `.json` and the row-view - redirect is gone. -- **`_json` is overloaded:** on GET it is a renderer option naming a column - to parse as JSON (repeatable); on canned-query POST a `_json` body field - forces a JSON response. Two unrelated meanings for one name. -- The reserved `/-/` namespace is applied consistently across routes — this - is in good shape. The one gap: table names matching `^-$`-adjacent shapes - are protected by tilde-encoding; keep a test asserting `/-/` can never be - shadowed by user data. - ---- - -## 6. Permissions and security consistency (P1/P2) - -- ~~**(P1) `/-/databases.json` ignores per-database permissions** — it lists - every attached database (name, path on disk, size) to any actor holding - `view-instance` (app.py:2157-2169), while the homepage and every other - endpoint filter by `view-database`. On a public instance with private - databases this leaks filesystem paths and database names. Filter it, or - gate it behind `permissions-debug`.~~ ✅ **Done** — the endpoint now - filters through `allowed_resources("view-database", actor)`. -- ~~**(P2) `/db/-/schema` checks existence before permission** - (views/special.py:1308-1317): an actor without `view-database` can - distinguish "database exists" (403) from "does not exist" (404). - Standardize on permission-check-first (as the table view does) so - unauthorized actors get a uniform response.~~ ✅ **Done** — permission is - checked first; the table schema view also now 404s (instead of a 500 - KeyError) for an unknown database. -- ~~**(P2) `/-/threads` exposes runtime internals** (thread idents, asyncio - task reprs including file paths) behind only `view-instance`. Consider - `permissions-debug`, alongside `/-/actions` which already requires it.~~ - ✅ **Done** — `/-/threads` now requires `permissions-debug`. -- ~~**(P3) `/-/config` redaction is substring-based** on six key names - (app.py:2502-2505); plugins storing secrets under other names leak. Worth - a note in plugin authoring docs plus a `redact_keys` plugin hook.~~ - ✅ **Documented** — the plugin secrets docs now advise naming keys to - match the redaction substrings (a `redact_keys` hook remains a possible - future addition). -- **(P3) Database-level checks on `/-/create`** (insert-row/update-row - checked against `DatabaseResource`, not the about-to-exist table — - table_create_alter.py:819-856) vs table-level checks on `/-/insert`. - Correct by necessity, but document that a token restricted to - table-level `ir` cannot use `/-/create` with rows. - ---- - -## 7. Completeness gaps for a 1.0 JSON API (P2/P3) - -1. **(P2) No JSON API to create tokens.** `/-/create-token` is an HTML form - only (`has_json_alternate = False`, form-encoded POST). Any automation - that wants to mint scoped tokens must shell out to `datasette - create-token`. An intentional JSON mode (actor-authenticated, same - restriction vocabulary) rounds out the write API story — or explicitly - document token minting as CLI/Python-only. -2. **(P2) Row JSON cannot expand foreign-key labels.** `_labels` works on - table JSON but is silently ignored on row JSON (views/row.py:445-475 - expands only for HTML). Either support it or return 400 for unsupported - parameters; silent ignoring is the worst option (see also §2b on unknown - `_extra` values). -3. **(P2) No machine-readable "which write features does this instance/table - support" endpoint.** Clients must probe (`/-/insert` on an immutable - database → 403). The API explorer computes exactly this data for HTML - (views/special.py:863-990); exposing it as JSON would let clients degrade - gracefully. (`/-/allowed.json` covers the permission half already.) -4. **(P3) Table list pagination.** `/db.json` inlines all tables (with - counts) and the homepage truncates to 5 per database; a 10,000-table - database has no paginated table listing. Acceptable for 1.0 if - documented; the internal catalog tables would support a real endpoint - later. -5. **(P3) `Link: rel=next` header** exists on table JSON only. Harmless, but - either add it to the other paginated endpoints or drop it from the - contract (`Access-Control-Expose-Headers: Link` suggests it is meant to - be part of the API). - ---- - -## 8. Behavior that looks like a bug and should be resolved before freezing - -1. ~~**Trusted queries: update is blocked, delete is not.** - `QueryUpdateView` rejects `is_trusted` queries with 403 - (stored_queries.py:426-427) but `QueryDeleteView.post` never checks - `is_trusted` — an actor with `delete-query` can delete a config-defined - trusted query via the API (it will resync on restart, making the - behavior confusing rather than catastrophic). Align delete with update.~~ - ✅ **Done** — both the POST endpoint and the HTML confirmation page now - return 403 `"Trusted queries cannot be deleted using the API"`; - `datasette.remove_query()` remains available for internal use. -2. ~~**GET `/db/-/query` with no `?sql=` returns 200 `{"ok": true, "rows": - []}`** while `.csv` on the same request returns 400 `"?sql= is - required"`. The JSON behavior masks caller bugs; return 400 on both.~~ - ✅ **Done** — all data formats now return 400; the HTML SQL editor page - is unchanged. -3. ~~**`_shape=object` HTTP 200 error** (§1b) — almost certainly unintended.~~ - ✅ Done — now 400 (fixed with §1b). -4. ~~**Row delete 500** (§1c) — inconsistent with every sibling endpoint.~~ - ✅ Done — now 400. -5. ~~**The "SQL Interrupted" error embeds an HTML fragment in the JSON `error` - value** (views/database.py:805-820). Error strings in the JSON API should - be plain text.~~ ✅ **Done** — `DatasetteError` gained a `plain_message` - used for JSON responses; the HTML error page keeps the rich version with - the SQL textarea. §8 is now fully resolved. - ---- - -## 9. Define stability tiers explicitly (P1 — documentation, not code) — ✅ IMPLEMENTED - -> **Status:** implemented. Undocumented JSON endpoints self-describe with -> an `"unstable"` marker key, and `docs/json_api.rst` now opens with an -> "API stability" section (`json_api_stability`) declaring the 1.x -> promise: documented endpoints/keys are stable with additive-only -> changes, pagination tokens are opaque, the error format and token -> restriction semantics are stable, and the exempt tiers (marker-key -> endpoints, debug/support endpoints, explicitly-unstable keys) are -> listed. Cross-referenced from the introspection and permission-debug -> docs. - -Not everything under `/-/` can or should carry a 1.0 guarantee. Recommend -shipping 1.0 with an explicit three-tier contract, per endpoint: - -- **Stable (semver-protected):** table/row/query reads (`.json`, `_shape`, - `_extra` public names, filters, pagination tokens as opaque strings), the - write API (`/-/insert`, `/-/upsert`, `/-/alter`, `/-/drop`, - `/-/set-column-type`, row `/-/update`, `/-/delete`, `/-/create`, - `/-/execute-write`), stored-query CRUD + execution, `/-/versions`, - `/-/plugins`, `/-/settings`, `/-/actor`, `/-/databases`, schema endpoints, - token format & restriction semantics (`_r` abbreviations are wire format - now — they are stored inside issued tokens and cannot change silently). -- **Unstable/debug (documented as exempt):** `/-/threads`, `/-/actions`, - `/-/permissions`, `/-/allowed`, `/-/rules`, `/-/check`, `/-/messages`, - `/-/allow-debug`, `/-/patterns`, `/-/debug/autocomplete`, the `debug` and - `request` extras (the `debug` extra already self-describes as unstable), - `/-/api` and `/-/jump` (UI support endpoints), `/-/autocomplete` and - `/-/fragment` (UI support), `/-/foreign-key-suggestions` and - `/-/foreign-key-targets` (heuristic outputs). -- **Internal:** anything HTML-only (`/-/edit`, `/-/create-token`, - `/-/logout`, `/-/auth-token`). - -Two details make tiering urgent rather than optional: - -- **Extras are enumerable by clients** (`?_extra=extras` self-describes the - registry), so every public extra name is de-facto API. Mark each extra - stable or unstable in its class definition and surface that in the - `extras` output. -- **Pagination tokens leak implementation** (tilde-encoded pk values for - tables, plain integers for views). Declare them opaque now so the view - token can become keyset later without a "breaking" change. - ---- - -## 10. Summary of P1 items (the pre-1.0 checklist) - -1. ~~One canonical JSON error shape; retire the other three (§1).~~ ✅ Done. -2. ~~`Forbidden` → JSON 403 for JSON requests (§1a).~~ ✅ Done. -3. ~~No `ok: false` with HTTP 200 (§1b: `_shape=object`, write canned-query - SQL errors).~~ ✅ Done. -4. ~~Wrap `/-/plugins`, `/-/databases`, `/-/actions` top-level arrays in - objects (§2).~~ ✅ Done. -5. ~~Filter `/-/databases.json` by `view-database` or gate it behind - `permissions-debug` (§6).~~ ✅ Done. -6. ~~401 (not silent-anonymous) for invalid/expired bearer tokens (§1c).~~ - ✅ Done. -7. ~~Publish explicit stability tiers, including extras and pagination-token - opacity (§9).~~ ✅ Done. -8. Resolve the looks-like-a-bug list (§8), especially ~~trusted-query delete - and row-delete 500~~ (both done). - -Everything in P2 is worth doing now because each item is breaking-to-fix -later; each P3 can be resolved by a sentence of documentation declaring the -current behavior intentional. From 8b159144a58992f8fe1f5a0489a3566f57e8d20c Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 23:48:49 +0000 Subject: [PATCH 38/46] Add Response.error() for JSON errors in the standard format Response.error(messages, status=400) builds a JSON error response in Datasette's standard error format, alongside Response.json/html/text. messages can be a single string or a list. All internal error response construction now uses it - the private views.base._error() helper is gone and the verbose Response.json(error_body(...), status=...) sites are converted. error_body() remains for the cases that merge the error keys into a larger payload (the JSON renderer, handle_exception and the permission debug payload builders). Since Response is public plugin API, plugins that build JSON endpoints now have an obvious way to return errors in the canonical shape. Documented in the internals documentation, including the guidance to raise Forbidden/NotFound/BadRequest/DatasetteError instead when the error should content-negotiate. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/app.py | 5 +- datasette/forbidden.py | 4 +- datasette/utils/asgi.py | 14 ++++- datasette/views/base.py | 9 +--- datasette/views/execute_write.py | 16 +++--- datasette/views/row.py | 32 ++++++------ datasette/views/special.py | 18 ++----- datasette/views/stored_queries.py | 70 ++++++++++++++----------- datasette/views/table.py | 74 ++++++++++++++------------- datasette/views/table_create_alter.py | 50 +++++++++--------- docs/internals.rst | 6 ++- tests/test_internals_response.py | 24 +++++++++ 12 files changed, 182 insertions(+), 140 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 463c9be2..9c7e768b 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -113,7 +113,6 @@ from .utils import ( detect_json1, add_cors_headers, display_actor, - error_body, escape_css_string, escape_sqlite, find_spatialite, @@ -2958,9 +2957,7 @@ class DatasetteRouter: headers = {"www-authenticate": 'Bearer error="invalid_token"'} if self.ds.cors: add_cors_headers(headers) - response = Response.json( - error_body([str(exception)], 401), status=401, headers=headers - ) + response = Response.error([str(exception)], 401, headers=headers) await response.asgi_send(send) async def handle_404(self, request, send, exception=None): diff --git a/datasette/forbidden.py b/datasette/forbidden.py index 3a81ac4f..91b1ff96 100644 --- a/datasette/forbidden.py +++ b/datasette/forbidden.py @@ -1,5 +1,5 @@ from datasette import hookimpl, Response -from .utils import add_cors_headers, error_body +from .utils import add_cors_headers @hookimpl(trylast=True) @@ -13,7 +13,7 @@ def forbidden(datasette, request, message): headers = {} if datasette.cors: add_cors_headers(headers) - return Response.json(error_body(message, 403), status=403, headers=headers) + return Response.error(message, 403, headers=headers) return Response.html( await datasette.render_template( "error.html", diff --git a/datasette/utils/asgi.py b/datasette/utils/asgi.py index 7777308f..610b86f2 100644 --- a/datasette/utils/asgi.py +++ b/datasette/utils/asgi.py @@ -1,6 +1,6 @@ import json from typing import Optional -from datasette.utils import MultiParams, calculate_etag, sha256_file +from datasette.utils import MultiParams, calculate_etag, error_body, sha256_file from datasette.utils.multipart import ( parse_form_data, MultipartParseError, @@ -575,6 +575,18 @@ class Response: content_type="application/json; charset=utf-8", ) + @classmethod + def error(cls, messages, status=400, headers=None): + """ + A JSON error response using Datasette's standard error format. + + messages can be a single string or a list of strings. For errors + that should content-negotiate between JSON and HTML, raise + Forbidden, NotFound, BadRequest or DatasetteError instead and let + Datasette's error handling hooks build the response. + """ + return cls.json(error_body(messages, status), status=status, headers=headers) + @classmethod def redirect(cls, path, status=302, headers=None): headers = headers or {} diff --git a/datasette/views/base.py b/datasette/views/base.py index 88d16753..66e14a6d 100644 --- a/datasette/views/base.py +++ b/datasette/views/base.py @@ -5,7 +5,6 @@ import sys from datasette.utils.asgi import Request from datasette.utils import ( add_cors_headers, - error_body, EscapeHtmlWriter, InvalidSql, LimitedWriter, @@ -53,7 +52,7 @@ class View: request.path.endswith(".json") or request.headers.get("content-type") == "application/json" ): - response = Response.json(error_body("Method not allowed", 405), status=405) + response = Response.error("Method not allowed", 405) else: response = Response.text("Method not allowed", status=405) return response @@ -92,7 +91,7 @@ class BaseView: request.path.endswith(".json") or request.headers.get("content-type") == "application/json" ): - response = Response.json(error_body("Method not allowed", 405), status=405) + response = Response.error("Method not allowed", 405) else: response = Response.text("Method not allowed", status=405) return response @@ -180,10 +179,6 @@ class BaseView: return view -def _error(messages, status=400): - return Response.json(error_body(messages, status), status=status) - - async def stream_csv(datasette, fetch_data, request, database): kwargs = {} stream = request.args.get("_stream") diff --git a/datasette/views/execute_write.py b/datasette/views/execute_write.py index 6806e69d..dd35b127 100644 --- a/datasette/views/execute_write.py +++ b/datasette/views/execute_write.py @@ -5,7 +5,7 @@ from datasette.resources import DatabaseResource from datasette.utils import UNSTABLE_API_MESSAGE, sqlite3 from datasette.utils.asgi import Response -from .base import BaseView, _error +from .base import BaseView from .database import display_rows as display_query_rows from .query_helpers import ( QueryValidationError, @@ -348,7 +348,7 @@ class ExecuteWriteView(BaseView): ) if not db.is_mutable: return _block_framing( - _error( + Response.error( ["Cannot execute write SQL because this database is immutable."], 403, ) @@ -367,10 +367,10 @@ class ExecuteWriteView(BaseView): actor=request.actor, ): return _block_framing( - _error(["Permission denied: need execute-write-sql"], 403) + Response.error(["Permission denied: need execute-write-sql"], 403) ) if not db.is_mutable: - return _block_framing(_error(["Database is immutable"], 403)) + return _block_framing(Response.error(["Database is immutable"], 403)) data = {} is_json = request.headers.get("content-type", "").startswith("application/json") @@ -384,7 +384,7 @@ class ExecuteWriteView(BaseView): ) except QueryValidationError as ex: if _wants_json(request, is_json, data): - return _block_framing(_error([ex.message], ex.status)) + return _block_framing(Response.error([ex.message], ex.status)) if ex.flash: self.ds.add_message(request, ex.message, self.ds.ERROR) return await self._render_form( @@ -405,7 +405,7 @@ class ExecuteWriteView(BaseView): except sqlite3.DatabaseError as ex: message = str(ex) if wants_json: - return _block_framing(_error([message], 400)) + return _block_framing(Response.error([message], 400)) return await self._render_form( request, db, @@ -488,13 +488,13 @@ class ExecuteWriteAnalyzeView(BaseView): actor=request.actor, ): return _block_framing( - _error(["Permission denied: need execute-write-sql"], 403) + Response.error(["Permission denied: need execute-write-sql"], 403) ) invalid_keys = set(request.args) - {"sql"} if invalid_keys: return _block_framing( - _error( + Response.error( ["Invalid keys: {}".format(", ".join(sorted(invalid_keys)))], 400, ) diff --git a/datasette/views/row.py b/datasette/views/row.py index d9a3deeb..c90a3bbe 100644 --- a/datasette/views/row.py +++ b/datasette/views/row.py @@ -12,7 +12,7 @@ from datasette.utils.asgi import NotFound, Forbidden, PayloadTooLarge, Response from datasette.database import QueryInterrupted from datasette.events import UpdateRowEvent, DeleteRowEvent from datasette.resources import TableResource -from .base import BaseView, DatasetteError, _error, stream_csv +from .base import BaseView, DatasetteError, stream_csv from datasette.utils import ( add_cors_headers, await_me_maybe, @@ -715,11 +715,13 @@ async def _resolve_row_and_check_permission(datasette, request, permission): try: resolved = await datasette.resolve_row(request) except DatabaseNotFound as e: - return False, _error(["Database not found: {}".format(e.database_name)], 404) + return False, Response.error( + ["Database not found: {}".format(e.database_name)], 404 + ) except TableNotFound as e: - return False, _error(["Table not found: {}".format(e.table)], 404) + return False, Response.error(["Table not found: {}".format(e.table)], 404) except RowNotFound as e: - return False, _error(["Record not found: {}".format(e.pk_values)], 404) + return False, Response.error(["Record not found: {}".format(e.pk_values)], 404) # Ensure user has permission to delete this row if not await datasette.allowed( @@ -727,7 +729,7 @@ async def _resolve_row_and_check_permission(datasette, request, permission): resource=TableResource(database=resolved.db.name, table=resolved.table), actor=request.actor, ): - return False, _error(["Permission denied"], 403) + return False, Response.error(["Permission denied"], 403) return True, resolved @@ -752,7 +754,7 @@ class RowDeleteView(BaseView): try: await resolved.db.execute_write_fn(delete_row, request=request) except Exception as e: - return _error([str(e)], 400) + return Response.error([str(e)], 400) await self.ds.track_event( DeleteRowEvent( @@ -791,24 +793,24 @@ class RowUpdateView(BaseView): try: data = await request.json() except json.JSONDecodeError as e: - return _error(["Invalid JSON: {}".format(e)]) + return Response.error(["Invalid JSON: {}".format(e)]) except PayloadTooLarge as e: - return _error([str(e)], 413) + return Response.error([str(e)], 413) if not isinstance(data, dict): - return _error(["JSON must be a dictionary"]) + return Response.error(["JSON must be a dictionary"]) if "update" not in data or not isinstance(data["update"], dict): - return _error(["JSON must contain an update dictionary"]) + return Response.error(["JSON must contain an update dictionary"]) invalid_keys = set(data.keys()) - {"update", "return", "alter"} if invalid_keys: - return _error(["Invalid keys: {}".format(", ".join(invalid_keys))]) + return Response.error(["Invalid keys: {}".format(", ".join(invalid_keys))]) update = data["update"] try: update = decode_write_json_row(update) except WriteJsonValueError as e: - return _error([str(e)], 400) + return Response.error([str(e)], 400) # Validate column types from datasette.views.table import _validate_column_types @@ -817,7 +819,7 @@ class RowUpdateView(BaseView): self.ds, resolved.db.name, resolved.table, [update] ) if ct_errors: - return _error(ct_errors, 400) + return Response.error(ct_errors, 400) alter = data.get("alter") if alter and not await self.ds.allowed( @@ -825,7 +827,7 @@ class RowUpdateView(BaseView): resource=TableResource(database=resolved.db.name, table=resolved.table), actor=request.actor, ): - return _error(["Permission denied for alter-table"], 403) + return Response.error(["Permission denied for alter-table"], 403) def update_row(conn): sqlite_utils.Database(conn)[resolved.table].update( @@ -835,7 +837,7 @@ class RowUpdateView(BaseView): try: await resolved.db.execute_write_fn(update_row, request=request) except Exception as e: - return _error([str(e)], 400) + return Response.error([str(e)], 400) result = {"ok": True} returned_row = None diff --git a/datasette/views/special.py b/datasette/views/special.py index 9386440c..c13191a1 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -501,13 +501,9 @@ class PermissionRulesView(BaseView): # JSON API - action parameter is required action = request.args.get("action") if not action: - return Response.json( - error_body("action parameter is required", 400), status=400 - ) + return Response.error("action parameter is required", 400) if action not in self.ds.actions: - return Response.json( - error_body(f"Unknown action: {action}", 404), status=404 - ) + return Response.error(f"Unknown action: {action}", 404) actor = request.actor if isinstance(request.actor, dict) else None @@ -516,15 +512,13 @@ class PermissionRulesView(BaseView): if page < 1: raise ValueError except ValueError: - return Response.json( - error_body("_page must be a positive integer", 400), status=400 - ) + return Response.error("_page must be a positive integer", 400) try: page_size = parse_size_limit( request.args.get("_size"), default=50, maximum=200 ) except ValueError as ex: - return Response.json(error_body(str(ex), 400), status=400) + return Response.error(str(ex), 400) offset = (page - 1) * page_size from datasette.utils.actions_sql import build_permission_rules_sql @@ -673,9 +667,7 @@ class PermissionCheckView(BaseView): # JSON API - action parameter is required action = request.args.get("action") if not action: - return Response.json( - error_body("action parameter is required", 400), status=400 - ) + return Response.error("action parameter is required", 400) parent = request.args.get("parent") child = request.args.get("child") diff --git a/datasette/views/stored_queries.py b/datasette/views/stored_queries.py index 3273d13c..d64f37d2 100644 --- a/datasette/views/stored_queries.py +++ b/datasette/views/stored_queries.py @@ -5,7 +5,7 @@ from datasette.stored_queries import stored_query_to_dict from datasette.utils import UNSTABLE_API_MESSAGE, sqlite3, tilde_decode from datasette.utils.asgi import Response -from .base import BaseView, _error +from .base import BaseView from .query_helpers import ( QueryValidationError, _as_bool, @@ -34,12 +34,14 @@ class QueryParametersView(BaseView): resource=DatabaseResource(db.name), actor=request.actor, ): - return _block_framing(_error(["Permission denied: need execute-sql"], 403)) + return _block_framing( + Response.error(["Permission denied: need execute-sql"], 403) + ) invalid_keys = set(request.args) - {"sql"} if invalid_keys: return _block_framing( - _error( + Response.error( ["Invalid keys: {}".format(", ".join(sorted(invalid_keys)))], 400, ) @@ -47,7 +49,7 @@ class QueryParametersView(BaseView): try: parameters = _derived_query_parameters(request.args.get("sql") or "") except QueryValidationError as ex: - return _block_framing(_error([ex.message], ex.status)) + return _block_framing(Response.error([ex.message], ex.status)) return _block_framing( Response.json( { @@ -95,7 +97,7 @@ class QueryListView(BaseView): is_write = _as_optional_bool(request.args.get("is_write"), "is_write") is_private = _as_optional_bool(request.args.get("is_private"), "is_private") except QueryValidationError as ex: - return _error([ex.message], ex.status) + return Response.error([ex.message], ex.status) page = await self.ds.list_queries( database, @@ -306,18 +308,22 @@ class QueryCreateAnalyzeView(BaseView): resource=DatabaseResource(db.name), actor=request.actor, ): - return _block_framing(_error(["Permission denied: need execute-sql"], 403)) + return _block_framing( + Response.error(["Permission denied: need execute-sql"], 403) + ) if not await self.ds.allowed( action="store-query", resource=DatabaseResource(db.name), actor=request.actor, ): - return _block_framing(_error(["Permission denied: need store-query"], 403)) + return _block_framing( + Response.error(["Permission denied: need store-query"], 403) + ) invalid_keys = set(request.args) - {"sql"} if invalid_keys: return _block_framing( - _error( + Response.error( ["Invalid keys: {}".format(", ".join(sorted(invalid_keys)))], 400, ) @@ -352,13 +358,13 @@ class QueryStoreView(QueryCreateView): resource=DatabaseResource(db.name), actor=request.actor, ): - return _error(["Permission denied: need execute-sql"], 403) + return Response.error(["Permission denied: need execute-sql"], 403) if not await self.ds.allowed( action="store-query", resource=DatabaseResource(db.name), actor=request.actor, ): - return _error(["Permission denied: need store-query"], 403) + return Response.error(["Permission denied: need store-query"], 403) is_json = False query_data = {} @@ -375,7 +381,7 @@ class QueryStoreView(QueryCreateView): return await self._error_response( request, db, query_data, ex.message, ex.status ) - return _error([ex.message], ex.status) + return Response.error([ex.message], ex.status) prepared.pop("analysis") name = prepared.pop("name") @@ -384,7 +390,7 @@ class QueryStoreView(QueryCreateView): except sqlite3.IntegrityError as ex: if not is_json and isinstance(query_data, dict): return await self._error_response(request, db, query_data, str(ex), 400) - return _error([str(ex)], 400) + return Response.error([str(ex)], 400) query = await self.ds.get_query(db.name, name) assert query is not None @@ -409,13 +415,13 @@ class QueryDefinitionView(BaseView): query_name = tilde_decode(request.url_vars["query"]) query = await self.ds.get_query(db.name, query_name) if query is None: - return _error(["Query not found: {}".format(query_name)], 404) + return Response.error(["Query not found: {}".format(query_name)], 404) if not await self.ds.allowed( action="view-query", resource=QueryResource(db.name, query_name), actor=request.actor, ): - return _error(["Permission denied"], 403) + return Response.error(["Permission denied"], 403) return Response.json( { "ok": True, @@ -433,15 +439,17 @@ class QueryUpdateView(BaseView): query_name = tilde_decode(request.url_vars["query"]) existing = await self.ds.get_query(db.name, query_name) if existing is None: - return _error(["Query not found: {}".format(query_name)], 404) + return Response.error(["Query not found: {}".format(query_name)], 404) if not await self.ds.allowed( action="update-query", resource=QueryResource(db.name, query_name), actor=request.actor, ): - return _error(["Permission denied: need update-query"], 403) + return Response.error(["Permission denied: need update-query"], 403) if existing.is_trusted: - return _error(["Trusted queries cannot be updated using the API"], 403) + return Response.error( + ["Trusted queries cannot be updated using the API"], 403 + ) try: data, _ = await _json_or_form_payload(request) @@ -467,7 +475,7 @@ class QueryUpdateView(BaseView): self.ds, request, db, existing, update ) except QueryValidationError as ex: - return _error([ex.message], ex.status) + return Response.error([ex.message], ex.status) await self.ds.update_query(db.name, query_name, **update_kwargs) if data.get("return"): @@ -524,32 +532,32 @@ class QueryEditView(BaseView): async def get(self, request): db, query_name, existing = await self._load(request) if existing is None: - return _error(["Query not found: {}".format(query_name)], 404) + return Response.error(["Query not found: {}".format(query_name)], 404) await self.ds.ensure_permission( action="update-query", resource=QueryResource(db.name, query_name), actor=request.actor, ) if existing.is_trusted: - return _error(["Trusted queries cannot be edited"], 403) + return Response.error(["Trusted queries cannot be edited"], 403) return await self._render_form(request, db, existing) async def post(self, request): db, query_name, existing = await self._load(request) if existing is None: - return _error(["Query not found: {}".format(query_name)], 404) + return Response.error(["Query not found: {}".format(query_name)], 404) if not await self.ds.allowed( action="update-query", resource=QueryResource(db.name, query_name), actor=request.actor, ): - return _error(["Permission denied: need update-query"], 403) + return Response.error(["Permission denied: need update-query"], 403) if existing.is_trusted: - return _error(["Trusted queries cannot be edited"], 403) + return Response.error(["Trusted queries cannot be edited"], 403) data, _ = await _json_or_form_payload(request) if not isinstance(data, dict): - return _error(["Invalid form submission"], 400) + return Response.error(["Invalid form submission"], 400) sql = data.get("sql") sql = existing.sql if sql is None else sql.strip() title = data.get("title") or "" @@ -621,14 +629,16 @@ class QueryDeleteView(BaseView): async def get(self, request): db, query_name, existing = await self._load(request) if existing is None: - return _error(["Query not found: {}".format(query_name)], 404) + return Response.error(["Query not found: {}".format(query_name)], 404) await self.ds.ensure_permission( action="delete-query", resource=QueryResource(db.name, query_name), actor=request.actor, ) if existing.is_trusted: - return _error(["Trusted queries cannot be deleted using the API"], 403) + return Response.error( + ["Trusted queries cannot be deleted using the API"], 403 + ) return await self.render( ["query_delete.html"], request, @@ -643,15 +653,17 @@ class QueryDeleteView(BaseView): async def post(self, request): db, query_name, existing = await self._load(request) if existing is None: - return _error(["Query not found: {}".format(query_name)], 404) + return Response.error(["Query not found: {}".format(query_name)], 404) if not await self.ds.allowed( action="delete-query", resource=QueryResource(db.name, query_name), actor=request.actor, ): - return _error(["Permission denied: need delete-query"], 403) + return Response.error(["Permission denied: need delete-query"], 403) if existing.is_trusted: - return _error(["Trusted queries cannot be deleted using the API"], 403) + return Response.error( + ["Trusted queries cannot be deleted using the API"], 403 + ) data, is_json = await _json_or_form_payload(request) await self.ds.remove_query(db.name, query_name) diff --git a/datasette/views/table.py b/datasette/views/table.py index 80ad1aab..8ce4b711 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -60,7 +60,7 @@ from dataclasses import dataclass, field from datasette.extras import ExtraScope from . import Context, from_extra -from .base import BaseView, DatasetteError, _error, stream_csv +from .base import BaseView, DatasetteError, stream_csv from .database import QueryView from .table_create_alter import ( ALTER_TABLE_COLUMN_TYPES, @@ -1035,7 +1035,7 @@ class TableInsertView(BaseView): try: resolved = await self.ds.resolve_table(request) except NotFound as e: - return _error([e.args[0]], 404) + return Response.error([e.args[0]], 404) db = resolved.db database_name = db.name table_name = resolved.table @@ -1043,7 +1043,7 @@ class TableInsertView(BaseView): # Table must exist (may handle table creation in the future) db = self.ds.get_database(database_name) if not await db.table_exists(table_name): - return _error(["Table not found: {}".format(table_name)], 404) + return Response.error(["Table not found: {}".format(table_name)], 404) if upsert: # Must have insert-row AND upsert-row permissions @@ -1059,7 +1059,7 @@ class TableInsertView(BaseView): actor=request.actor, ) ): - return _error( + return Response.error( ["Permission denied: need both insert-row and update-row"], 403 ) else: @@ -1069,10 +1069,10 @@ class TableInsertView(BaseView): resource=TableResource(database=database_name, table=table_name), actor=request.actor, ): - return _error(["Permission denied"], 403) + return Response.error(["Permission denied"], 403) if not db.is_mutable: - return _error(["Database is immutable"], 403) + return Response.error(["Database is immutable"], 403) pks = await db.primary_keys(table_name) @@ -1081,20 +1081,20 @@ class TableInsertView(BaseView): request, db, table_name, pks, upsert ) except PayloadTooLarge as e: - return _error([str(e)], 413) + return Response.error([str(e)], 413) if errors: - return _error(errors, 400) + return Response.error(errors, 400) try: rows = decode_write_json_rows(rows) except WriteJsonValueError as e: - return _error([str(e)], 400) + return Response.error([str(e)], 400) # Validate column types ct_errors = await _validate_column_types( self.ds, database_name, table_name, rows ) if ct_errors: - return _error(ct_errors, 400) + return Response.error(ct_errors, 400) num_rows = len(rows) @@ -1108,14 +1108,16 @@ class TableInsertView(BaseView): alter = extras.get("alter") if upsert and (ignore or replace): - return _error(["Upsert does not support ignore or replace"], 400) + return Response.error(["Upsert does not support ignore or replace"], 400) if replace and not await self.ds.allowed( action="update-row", resource=TableResource(database=database_name, table=table_name), actor=request.actor, ): - return _error(['Permission denied: need update-row to use "replace"'], 403) + return Response.error( + ['Permission denied: need update-row to use "replace"'], 403 + ) initial_schema = None if alter: @@ -1125,7 +1127,7 @@ class TableInsertView(BaseView): resource=TableResource(database=database_name, table=table_name), actor=request.actor, ): - return _error(["Permission denied for alter-table"], 403) + return Response.error(["Permission denied for alter-table"], 403) # Track initial schema to check if it changed later initial_schema = await db.execute_fn( lambda conn: sqlite_utils.Database(conn)[table_name].schema @@ -1165,7 +1167,7 @@ class TableInsertView(BaseView): try: rows = await db.execute_write_fn(insert_or_upsert_rows, request=request) except Exception as e: - return _error([str(e)]) + return Response.error([str(e)]) result = {"ok": True} if should_return: if upsert: @@ -1246,7 +1248,7 @@ class TableSetColumnTypeView(BaseView): try: resolved = await self.ds.resolve_table(request) except NotFound as e: - return _error([e.args[0]], 404) + return Response.error([e.args[0]], 404) database_name = resolved.db.name table_name = resolved.table @@ -1256,39 +1258,39 @@ class TableSetColumnTypeView(BaseView): resource=TableResource(database=database_name, table=table_name), actor=request.actor, ): - return _error(["Permission denied"], 403) + return Response.error(["Permission denied"], 403) try: data = await request.json() except json.JSONDecodeError as e: - return _error(["Invalid JSON: {}".format(e)], 400) + return Response.error(["Invalid JSON: {}".format(e)], 400) except PayloadTooLarge as e: - return _error([str(e)], 413) + return Response.error([str(e)], 413) if not isinstance(data, dict): - return _error(["JSON must be a dictionary"], 400) + return Response.error(["JSON must be a dictionary"], 400) invalid_keys = set(data.keys()) - {"column", "column_type"} if invalid_keys: - return _error( + return Response.error( ['Invalid parameter: "{}"'.format('", "'.join(sorted(invalid_keys)))], 400, ) if "column" not in data: - return _error(['"column" is required'], 400) + return Response.error(['"column" is required'], 400) column = data["column"] if not isinstance(column, str): - return _error(['"column" must be a string'], 400) + return Response.error(['"column" must be a string'], 400) if "column_type" not in data: - return _error(['"column_type" is required'], 400) + return Response.error(['"column_type" is required'], 400) column_details = await self.ds._get_resource_column_details( database_name, table_name ) if column not in column_details: - return _error(["Column not found: {}".format(column)], 400) + return Response.error(["Column not found: {}".format(column)], 400) column_type_data = data["column_type"] if column_type_data is None: @@ -1305,11 +1307,11 @@ class TableSetColumnTypeView(BaseView): ) if not isinstance(column_type_data, dict): - return _error(['"column_type" must be an object or null'], 400) + return Response.error(['"column_type" must be an object or null'], 400) invalid_column_type_keys = set(column_type_data.keys()) - {"type", "config"} if invalid_column_type_keys: - return _error( + return Response.error( [ 'Invalid column_type parameter: "{}"'.format( '", "'.join(sorted(invalid_column_type_keys)) @@ -1319,24 +1321,24 @@ class TableSetColumnTypeView(BaseView): ) if "type" not in column_type_data: - return _error(['"column_type.type" is required'], 400) + return Response.error(['"column_type.type" is required'], 400) column_type = column_type_data["type"] if not isinstance(column_type, str): - return _error(['"column_type.type" must be a string'], 400) + return Response.error(['"column_type.type" must be a string'], 400) config = column_type_data.get("config") if config is not None and not isinstance(config, dict): - return _error(['"column_type.config" must be a dictionary'], 400) + return Response.error(['"column_type.config" must be a dictionary'], 400) if column_type not in self.ds._column_types: - return _error(["Unknown column type: {}".format(column_type)], 400) + return Response.error(["Unknown column type: {}".format(column_type)], 400) try: await self.ds.set_column_type( database_name, table_name, column, column_type, config ) except ValueError as e: - return _error([str(e)], 400) + return Response.error([str(e)], 400) return Response.json( { @@ -1360,22 +1362,22 @@ class TableDropView(BaseView): try: resolved = await self.ds.resolve_table(request) except NotFound as e: - return _error([e.args[0]], 404) + return Response.error([e.args[0]], 404) db = resolved.db database_name = db.name table_name = resolved.table # Table must exist db = self.ds.get_database(database_name) if not await db.table_exists(table_name): - return _error(["Table not found: {}".format(table_name)], 404) + return Response.error(["Table not found: {}".format(table_name)], 404) if not await self.ds.allowed( action="drop-table", resource=TableResource(database=database_name, table=table_name), actor=request.actor, ): - return _error(["Permission denied"], 403) + return Response.error(["Permission denied"], 403) if not db.is_mutable: - return _error(["Database is immutable"], 403) + return Response.error(["Database is immutable"], 403) confirm = False try: data = await request.json() @@ -1383,7 +1385,7 @@ class TableDropView(BaseView): except json.JSONDecodeError: pass except PayloadTooLarge as e: - return _error([str(e)], 413) + return Response.error([str(e)], 413) if not confirm: return Response.json( diff --git a/datasette/views/table_create_alter.py b/datasette/views/table_create_alter.py index c3e06c29..4deeafcc 100644 --- a/datasette/views/table_create_alter.py +++ b/datasette/views/table_create_alter.py @@ -29,7 +29,7 @@ from datasette.utils import ( from datasette.utils.asgi import NotFound, PayloadTooLarge, Response from datasette.utils.sqlite import sqlite_hidden_table_names -from .base import BaseView, _error +from .base import BaseView CREATE_TABLE_COLUMN_TYPES = ["text", "integer", "float", "blob"] CREATE_TABLE_SQLITE_TYPES = { @@ -805,22 +805,22 @@ class TableCreateView(BaseView): resource=DatabaseResource(database=database_name), actor=request.actor, ): - return _error(["Permission denied"], 403) + return Response.error(["Permission denied"], 403) try: data = await request.json() except json.JSONDecodeError as e: - return _error(["Invalid JSON: {}".format(e)]) + return Response.error(["Invalid JSON: {}".format(e)]) except PayloadTooLarge as e: - return _error([str(e)], 413) + return Response.error([str(e)], 413) if not isinstance(data, dict): - return _error(["JSON must be an object"]) + return Response.error(["JSON must be an object"]) try: create_request = CreateTableRequest.model_validate(data) except ValidationError as e: - return _error(_create_table_pydantic_errors(e)) + return Response.error(_create_table_pydantic_errors(e)) ignore = create_request.ignore replace = create_request.replace @@ -832,7 +832,7 @@ class TableCreateView(BaseView): resource=DatabaseResource(database=database_name), actor=request.actor, ): - return _error(["Permission denied: need update-row"], 403) + return Response.error(["Permission denied: need update-row"], 403) table_name = create_request.table table_exists = await db.table_exists(table_name) @@ -846,11 +846,11 @@ class TableCreateView(BaseView): resource=DatabaseResource(database=database_name), actor=request.actor, ): - return _error(["Permission denied: need insert-row"], 403) + return Response.error(["Permission denied: need insert-row"], 403) try: rows = decode_write_json_rows(rows) except WriteJsonValueError as e: - return _error([str(e)], 400) + return Response.error([str(e)], 400) alter = False if rows: @@ -865,7 +865,9 @@ class TableCreateView(BaseView): resource=DatabaseResource(database=database_name), actor=request.actor, ): - return _error(["Permission denied: need alter-table"], 403) + return Response.error( + ["Permission denied: need alter-table"], 403 + ) alter = True pk = create_request.pk @@ -881,7 +883,7 @@ class TableCreateView(BaseView): elif len(actual_pks) > 1 and pks and set(pks) != set(actual_pks): bad_pks = True if bad_pks: - return _error(["pk cannot be changed for existing table"]) + return Response.error(["pk cannot be changed for existing table"]) pks = actual_pks initial_schema = None @@ -924,7 +926,7 @@ class TableCreateView(BaseView): try: schema = await db.execute_write_fn(create_table, request=request) except Exception as e: - return _error([str(e)]) + return Response.error([str(e)]) if initial_schema is not None and initial_schema != schema: await self.ds.track_event( @@ -999,7 +1001,7 @@ class DatabaseForeignKeyTargetsView(BaseView): actor=request.actor, ) if not (can_create_table or can_alter_table): - return _error(["Permission denied: need create-table"], 403) + return Response.error(["Permission denied: need create-table"], 403) hidden_tables = await db.execute_fn( lambda conn: set(sqlite_hidden_table_names(conn)) @@ -1028,21 +1030,21 @@ class TableForeignKeySuggestionsView(BaseView): try: resolved = await self.ds.resolve_table(request) except NotFound as e: - return _error([e.args[0]], 404) + return Response.error([e.args[0]], 404) db = resolved.db database_name = db.name table_name = resolved.table if resolved.is_view: - return _error(["Cannot suggest foreign keys for a view"], 400) + return Response.error(["Cannot suggest foreign keys for a view"], 400) if not await self.ds.allowed( action="alter-table", resource=TableResource(database=database_name, table=table_name), actor=request.actor, ): - return _error(["Permission denied: need alter-table"], 403) + return Response.error(["Permission denied: need alter-table"], 403) source_columns, targets, current_by_column = await db.execute_fn( lambda conn: _foreign_key_suggestion_metadata(conn, table_name) @@ -1150,7 +1152,7 @@ class TableAlterView(BaseView): try: resolved = await self.ds.resolve_table(request) except NotFound as e: - return _error([e.args[0]], 404) + return Response.error([e.args[0]], 404) db = resolved.db database_name = db.name @@ -1161,25 +1163,25 @@ class TableAlterView(BaseView): resource=TableResource(database=database_name, table=table_name), actor=request.actor, ): - return _error(["Permission denied: need alter-table"], 403) + return Response.error(["Permission denied: need alter-table"], 403) if not db.is_mutable: - return _error(["Database is immutable"], 403) + return Response.error(["Database is immutable"], 403) try: data = await request.json() except json.JSONDecodeError as e: - return _error(["Invalid JSON: {}".format(e)], 400) + return Response.error(["Invalid JSON: {}".format(e)], 400) except PayloadTooLarge as e: - return _error([str(e)], 413) + return Response.error([str(e)], 413) if not isinstance(data, dict): - return _error(["JSON must be a dictionary"], 400) + return Response.error(["JSON must be a dictionary"], 400) try: alter_request = AlterTableRequest.model_validate(data) except ValidationError as e: - return _error(_pydantic_errors(e), 400) + return Response.error(_pydantic_errors(e), 400) def alter_table(conn): before_schema = _table_schema_from_conn(conn, table_name) @@ -1328,7 +1330,7 @@ class TableAlterView(BaseView): alter_table, request=request ) except Exception as e: - return _error([str(e)], 400) + return Response.error([str(e)], 400) altered = before_schema != after_schema if altered: diff --git a/docs/internals.rst b/docs/internals.rst index e5e7aca5..7258e6f3 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -284,7 +284,7 @@ For example: content_type="application/xml; charset=utf-8", ) -The quickest way to create responses is using the ``Response.text(...)``, ``Response.html(...)``, ``Response.json(...)`` or ``Response.redirect(...)`` helper methods: +The quickest way to create responses is using the ``Response.text(...)``, ``Response.html(...)``, ``Response.json(...)``, ``Response.error(...)`` or ``Response.redirect(...)`` helper methods: .. code-block:: python @@ -295,6 +295,8 @@ The quickest way to create responses is using the ``Response.text(...)``, ``Resp text_response = Response.text( "This will become utf-8 encoded text" ) + # A JSON error in Datasette's standard error format: + error_response = Response.error("Cannot do that", 400) # Redirects are served as 302, unless you pass status=301: redirect_response = Response.redirect( "https://latest.datasette.io/" @@ -304,6 +306,8 @@ Each of these responses will use the correct corresponding content-type - ``text Each of the helper methods take optional ``status=`` and ``headers=`` arguments, documented above. +``Response.error(messages, status=400)`` returns a JSON error in the :ref:`standard Datasette error format `. ``messages`` can be a single string or a list of strings. Use this for JSON-only endpoints; if your error should content-negotiate between JSON and HTML, raise ``Forbidden``, ``NotFound``, ``BadRequest`` or ``DatasetteError`` instead and Datasette's error handling will build the appropriate response. + .. _internals_response_asgi_send: Returning a response with .asgi_send(send) diff --git a/tests/test_internals_response.py b/tests/test_internals_response.py index 820b20b2..2366dcde 100644 --- a/tests/test_internals_response.py +++ b/tests/test_internals_response.py @@ -1,4 +1,5 @@ from datasette.utils.asgi import Response +import json import pytest @@ -52,3 +53,26 @@ async def test_response_set_cookie(): }, {"type": "http.response.body", "body": b""}, ] == events + + +def test_response_error_single_message(): + response = Response.error("Method not allowed", 405) + assert response.status == 405 + assert response.content_type == "application/json; charset=utf-8" + assert json.loads(response.body) == { + "ok": False, + "error": "Method not allowed", + "errors": ["Method not allowed"], + "status": 405, + } + + +def test_response_error_message_list_and_default_status(): + response = Response.error(["First problem", "Second problem"]) + assert response.status == 400 + assert json.loads(response.body) == { + "ok": False, + "error": "First problem; Second problem", + "errors": ["First problem", "Second problem"], + "status": 400, + } From 4874c292860c7ad67da2a730d6e58dde2718adda Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 23:59:44 +0000 Subject: [PATCH 39/46] Remove the next_url extra - the key is always present next_url became a default table JSON key alongside next, making the extra a no-op. Requesting ?_extra=next_url now returns the standard unknown-extra 400 error. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/views/table.py | 6 +++++- datasette/views/table_extras.py | 17 ----------------- docs/json_api.rst | 13 +------------ docs/template_context.rst | 2 +- tests/test_api.py | 2 +- tests/test_error_shape.py | 8 ++++++++ tests/test_table_api.py | 18 +++++++----------- 7 files changed, 23 insertions(+), 43 deletions(-) diff --git a/datasette/views/table.py b/datasette/views/table.py index 8ce4b711..f7edd744 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -107,7 +107,6 @@ class TableContext(Context): human_description_en: str = from_extra() is_view: bool = from_extra() metadata: dict = from_extra() - next_url: str = from_extra() primary_keys: list = from_extra() private: bool = from_extra() query: dict = from_extra() @@ -124,6 +123,11 @@ class TableContext(Context): metadata={"help": "True if the data for this page was retrieved without errors"} ) next: str = field(metadata={"help": "Pagination token for the next page, or None"}) + next_url: str = field( + metadata={ + "help": "Full URL for the next page of results, or None if there are no more pages. See :ref:`json_api_pagination`." + } + ) count_truncated: bool = field( metadata={ "help": "True if ``count`` is a capped lower bound rather than an exact total, because Datasette stopped counting after its configured row-count limit." diff --git a/datasette/views/table_extras.py b/datasette/views/table_extras.py index e36e8161..50002d92 100644 --- a/datasette/views/table_extras.py +++ b/datasette/views/table_extras.py @@ -321,21 +321,6 @@ class HumanDescriptionEnExtra(Extra): return human_description_en -class NextUrlExtra(Extra): - description = "Full URL for the next page of results" - example = ExtraExample( - "/fixtures/facetable.json?_size=1&_extra=next_url", - note=( - "``null`` if there are no more pages of results. " - "See :ref:`json_api_pagination`." - ), - ) - scopes = {ExtraScope.TABLE} - - async def resolve(self, context): - return context.next_url - - class ColumnsExtra(Extra): description = "List of column names returned by this table, row or query." example = ExtraExample("/fixtures/facetable.json?_extra=columns") @@ -1250,7 +1235,6 @@ TABLE_EXTRA_BUNDLES = { "count", "count_sql", "human_description_en", - "next_url", "metadata", "query", "columns", @@ -1286,7 +1270,6 @@ TABLE_EXTRA_CLASSES = [ SuggestedFacetsExtra, FacetInstancesProvider, HumanDescriptionEnExtra, - NextUrlExtra, ColumnsExtra, AllColumnsExtra, PrimaryKeysExtra, diff --git a/docs/json_api.rst b/docs/json_api.rst index 893af634..ec60a3a9 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -333,7 +333,7 @@ These can be repeated or comma-separated: :: - ?_extra=columns&_extra=count,next_url + ?_extra=columns&_extra=count,count_sql Requesting an ``_extra`` name that does not exist returns a ``400`` error in the :ref:`standard error format `, for example ``{"ok": false, "error": "Unknown _extra: nope", ...}``. @@ -437,17 +437,6 @@ The available table extras are listed below. "where state = \"CA\" sorted by pk" -``next_url`` - Full URL for the next page of results - - ``GET /fixtures/facetable.json?_size=1&_extra=next_url`` - - ``null`` if there are no more pages of results. See :ref:`json_api_pagination`. - - .. code-block:: json - - "http://localhost/fixtures/facetable.json?_size=1&_extra=next_url&_next=1" - ``columns`` List of column names returned by this table, row or query. diff --git a/docs/template_context.rst b/docs/template_context.rst index e9447058..e445b335 100644 --- a/docs/template_context.rst +++ b/docs/template_context.rst @@ -329,7 +329,7 @@ Many of these keys are shared with the :ref:`JSON API ` for this page. Pagination token for the next page, or None ``next_url`` - ``str`` - Full URL for the next page of results + Full URL for the next page of results, or None if there are no more pages. See :ref:`json_api_pagination`. ``ok`` - ``bool`` True if the data for this page was retrieved without errors diff --git a/tests/test_api.py b/tests/test_api.py index 9a96f14f..a15a507c 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -722,7 +722,7 @@ def test_config_cache_size(app_client_larger_cache_size): def test_config_force_https_urls(): with make_app_client(settings={"force_https_urls": True}) as client: response = client.get( - "/fixtures/facetable.json?_size=3&_facet=state&_extra=next_url,suggested_facets" + "/fixtures/facetable.json?_size=3&_facet=state&_extra=suggested_facets" ) assert response.json["next_url"].startswith("https://") assert response.json["facet_results"]["results"]["state"]["results"][0][ diff --git a/tests/test_error_shape.py b/tests/test_error_shape.py index 44856b36..768814fd 100644 --- a/tests/test_error_shape.py +++ b/tests/test_error_shape.py @@ -739,3 +739,11 @@ async def test_sql_interrupted_html_page_keeps_rich_error(ds_client): ) assert response.status_code == 400 assert " Date: Tue, 7 Jul 2026 00:25:09 +0000 Subject: [PATCH 40/46] /-/jump is a stable documented endpoint, not a debug exemption Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- docs/introspection.rst | 2 +- docs/json_api.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/introspection.rst b/docs/introspection.rst index ea94b70d..b78e4860 100644 --- a/docs/introspection.rst +++ b/docs/introspection.rst @@ -9,7 +9,7 @@ Each of these pages can be viewed in your browser. Add ``.json`` to the URL to g JSON responses that return an object include an ``"ok": true`` key, consistent with the rest of the :ref:`JSON API `. -The introspection endpoints documented on this page are covered by the :ref:`JSON API stability promise `, with the exception of the debug endpoints ``/-/threads``, ``/-/actions`` and ``/-/jump``, whose shapes may change in future releases. +The introspection endpoints documented on this page are covered by the :ref:`JSON API stability promise `, with the exception of the debug endpoints ``/-/threads`` and ``/-/actions``, whose shapes may change in future releases. .. _JsonDataView_metadata: diff --git a/docs/json_api.rst b/docs/json_api.rst index ec60a3a9..a96fd73d 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -48,7 +48,7 @@ Some JSON endpoints are **exempt** from this promise: debug playground. - Debug and support endpoints are documented so you can use them, but their JSON shapes are not frozen: :ref:`/-/threads `, - :ref:`/-/actions `, :ref:`/-/jump `, + :ref:`/-/actions `, the :ref:`permission debug endpoints ` (``/-/allowed``, ``/-/rules``, ``/-/check``) and the :ref:`table autocomplete endpoint `. From b23fc4ec48cbd68a072379a03aec3af7245084e3 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 00:32:29 +0000 Subject: [PATCH 41/46] Unreleased release notes for the JSON API consistency review Documents the canonical error format, the ok/envelope changes, the array-to-object endpoint conversions, 401s for invalid tokens, the pagination and page-size unification, removed legacy keys and formats, and the new Response.error(), TokenInvalid, count_truncated and unstable-marker APIs. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- docs/changelog.rst | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/docs/changelog.rst b/docs/changelog.rst index 4541981f..3230575d 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -17,6 +17,42 @@ Unreleased - The table and row JSON APIs now support ``?_extra=column_details`` for returning SQLite schema details for columns, including declared type, SQLite affinity, primary key, ``NOT NULL``, default and hidden-column metadata. - POST bodies that Datasette reads fully into memory - such as JSON submitted to the write API - are now capped by the new :ref:`setting_max_post_body_bytes` setting, defaulting to 2MB. Oversized requests are rejected with an HTTP 413 error as soon as the limit is exceeded, protecting smaller servers from memory exhaustion. File uploads are unaffected - ``request.form()`` streams those to disk and has its own separate limits. +This release also includes the results of a detailed consistency review of Datasette's JSON API in preparation for the 1.0 stable release. Several of these changes are backwards-incompatible with previous 1.0 alphas. The new :ref:`API stability documentation ` describes exactly which parts of the JSON API are covered by the 1.0 stability promise. + +JSON API: breaking changes +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- JSON error responses now use a single canonical format across every endpoint: ``{"ok": false, "error": "...", "errors": [...], "status": 400}``. The ``error`` key joins all error messages together, ``errors`` is the full list of messages and ``status`` always matches the HTTP status code. The legacy ``title`` key is no longer included in JSON errors (it remains available to the HTML error template), and endpoints that previously returned bare ``{"error": ...}`` objects have been updated. See :ref:`json_api_errors`. +- Every JSON object success response now includes ``"ok": true``, including introspection endpoints such as ``/-/versions`` and ``/-/settings``. +- ``/-/plugins.json``, ``/-/databases.json`` and ``/-/actions.json`` now return objects - ``{"ok": true, "plugins": [...]}`` and equivalents - instead of top-level JSON arrays, so these responses can gain additional keys in the future without a breaking change. The ``datasette plugins`` CLI command still outputs a plain array. +- ``/-/databases`` now only lists databases the current actor is allowed to view. It previously listed every attached database, including their filesystem paths, to any actor with ``view-instance``. +- Requests with an invalid or expired ``Authorization: Bearer`` token now receive a ``401`` status with the standard error body and a ``WWW-Authenticate: Bearer error="invalid_token"`` header, instead of being silently treated as unauthenticated. Bearer tokens that no registered token handler recognizes are still ignored, so authentication plugins with their own token formats keep working. Plugin :ref:`token handlers ` can raise the new ``datasette.TokenInvalid`` exception to trigger the same behavior. +- Permission errors for JSON requests now return the standard JSON error format with a ``403`` status. The default forbidden handling previously rendered an HTML error page even for ``.json`` requests. +- ``POST`` to a write canned query now returns a ``400`` error when the SQL fails to execute, instead of a ``200`` status with ``"ok": false`` in the body. The error response includes the standard error keys plus a ``"redirect"`` key. +- The :ref:`row update API ` with ``"return": true`` now responds with a ``"rows"`` list, matching insert and upsert, instead of a singular ``"row"`` object. +- Row delete write failures - such as a constraint violation raised by a trigger - now return ``400`` instead of ``500``, matching the other write endpoints. +- ``//-/query.json`` with a missing or blank ``?sql=`` parameter now returns a ``400`` error, as the CSV format already did, instead of a ``200`` with empty rows. +- Unknown ``?_extra=`` names now return a ``400`` error for JSON and other data formats, instead of being silently ignored. HTML pages continue to ignore unknown names. +- Table JSON responses now include ``next_url`` alongside ``next`` by default - both are ``null`` on the final page. The now-redundant ``?_extra=next_url`` parameter has been removed. +- The stored query list JSON no longer includes ``has_more`` - ``"next": null`` is the end-of-results signal across the whole API. This change also uncovered and fixed a bug where the query list ``next_url`` pointed at the HTML page and was a relative path; it is now an absolute URL that preserves the requested format. +- Stored query JSON objects no longer duplicate the list of parameter names as both ``params`` and ``parameters`` - only ``parameters`` remains in the output. ``params`` is still accepted as an input alias when creating or updating queries. +- Page size parameters are now consistent across the API: the stored query lists accept ``?_size=max`` and return a ``400`` error for values over the maximum instead of silently clamping them, and the ``/-/allowed`` and ``/-/rules`` permission debug endpoints renamed their ``page`` and ``page_size`` parameters to ``_page`` and ``_size``, matching the underscore grammar used by every other Datasette system parameter. +- ``/-/threads`` now requires the ``permissions-debug`` permission, since it exposes runtime internals such as file paths. It previously only required ``view-instance``. +- Trusted stored queries - those defined in configuration - can no longer be deleted through the JSON API or web interface, matching the existing restriction on editing them. +- The ``//-/schema`` endpoints now check the ``view-database`` permission before checking whether the database exists, so unauthorized actors can no longer probe for the existence of databases. +- SQL time limit errors in JSON responses are now a plain text message. The error string previously embedded an HTML fragment. +- The undocumented homepage JSON at ``/.json`` now returns ``databases`` as a list of objects rather than an object keyed by database name, matching every other collection in the API. +- The legacy ``.jsono`` format extension, long since superseded by ``?_shape=``, has been removed. + +JSON API: other improvements +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- The :ref:`write API ` endpoints now parse the request body as JSON regardless of the ``Content-Type`` header, so ``curl -d`` invocations work without remembering to set it. Invalid JSON is a ``400`` error. Cross-site request forgery remains prevented by Datasette's ``Origin`` and ``Sec-Fetch-Site`` checks. This also fixes a ``500`` error from the insert API when the ``Content-Type`` header was missing entirely. +- New ``Response.error(messages, status=400)`` helper for plugins that need to return a JSON error in Datasette's standard format. See :ref:`internals_response`. +- New ``count_truncated`` extra for table JSON, included automatically whenever ``count`` is requested. ``true`` means the count reached Datasette's counting limit and the real number of rows may be higher. See :ref:`json_api_extra`. +- JSON endpoints that are not part of the documented stable API now declare themselves with an ``"unstable"`` key in their responses, making the stability tier machine-readable. +- New documentation covering the grammar for :ref:`boolean query string arguments `, the reason :ref:`upsert ` returns ``200`` where insert returns ``201``, and advice for plugin authors on :ref:`naming secret configuration keys ` so that ``/-/config`` redacts them automatically. + .. _v1_0_a35: 1.0a35 (2026-06-23) From b83b12dd7abc5b0fce8353792d796bf3943e4a1a Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 01:13:33 +0000 Subject: [PATCH 42/46] Remove params input alias from the query create and update APIs The alias existed so API payloads could mirror the params key used by queries defined in datasette.yaml, but it was undocumented and untested, and the create endpoint is not part of the stable API. The API now only accepts parameters - sending params is a 400 Invalid keys error. The documented params key for queries in configuration is unchanged. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- datasette/views/query_helpers.py | 7 +++--- docs/changelog.rst | 2 +- tests/test_queries.py | 41 ++++++++++++++++++++++++++++++++ 3 files changed, 45 insertions(+), 5 deletions(-) diff --git a/datasette/views/query_helpers.py b/datasette/views/query_helpers.py index e9f85b6d..588891d4 100644 --- a/datasette/views/query_helpers.py +++ b/datasette/views/query_helpers.py @@ -33,7 +33,6 @@ _query_fields = { "hide_sql", "fragment", "parameters", - "params", "is_private", "on_success_message", "on_success_redirect", @@ -540,7 +539,7 @@ async def _prepare_query_create(datasette, request, db, data): raise QueryValidationError("Writable query fields require writable SQL") parameters = _coerce_query_parameters( - data.get("parameters", data.get("params")), + data.get("parameters"), derived, ) return { @@ -585,9 +584,9 @@ async def _prepare_query_update(datasette, request, db, existing: StoredQuery, u actor=request.actor, ) - if "parameters" in update or "params" in update: + if "parameters" in update: parameters = _coerce_query_parameters( - update.get("parameters", update.get("params")), + update.get("parameters"), derived, ) elif "sql" in update: diff --git a/docs/changelog.rst b/docs/changelog.rst index 3230575d..0079aaf2 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -35,7 +35,7 @@ JSON API: breaking changes - Unknown ``?_extra=`` names now return a ``400`` error for JSON and other data formats, instead of being silently ignored. HTML pages continue to ignore unknown names. - Table JSON responses now include ``next_url`` alongside ``next`` by default - both are ``null`` on the final page. The now-redundant ``?_extra=next_url`` parameter has been removed. - The stored query list JSON no longer includes ``has_more`` - ``"next": null`` is the end-of-results signal across the whole API. This change also uncovered and fixed a bug where the query list ``next_url`` pointed at the HTML page and was a relative path; it is now an absolute URL that preserves the requested format. -- Stored query JSON objects no longer duplicate the list of parameter names as both ``params`` and ``parameters`` - only ``parameters`` remains in the output. ``params`` is still accepted as an input alias when creating or updating queries. +- Stored query JSON objects no longer duplicate the list of parameter names as both ``params`` and ``parameters`` - only ``parameters`` remains. The query create and update APIs no longer accept ``params`` as an input alias either; ``params`` is still the documented key for :ref:`queries defined in configuration `. - Page size parameters are now consistent across the API: the stored query lists accept ``?_size=max`` and return a ``400`` error for values over the maximum instead of silently clamping them, and the ``/-/allowed`` and ``/-/rules`` permission debug endpoints renamed their ``page`` and ``page_size`` parameters to ``_page`` and ``_size``, matching the underscore grammar used by every other Datasette system parameter. - ``/-/threads`` now requires the ``permissions-debug`` permission, since it exposes runtime internals such as file paths. It previously only required ``view-instance``. - Trusted stored queries - those defined in configuration - can no longer be deleted through the JSON API or web interface, matching the existing restriction on editing them. diff --git a/tests/test_queries.py b/tests/test_queries.py index c25ec358..1cda740e 100644 --- a/tests/test_queries.py +++ b/tests/test_queries.py @@ -1125,6 +1125,47 @@ async def test_query_update_api_rejects_config_only_fields(): assert query.on_success_message_sql is None +@pytest.mark.asyncio +async def test_query_api_rejects_params_alias(): + # "params" is a datasette.yaml configuration key, not an API input - + # the API only accepts "parameters" + ds = Datasette(memory=True, default_deny=True) + ds.root_enabled = True + db = ds.add_memory_database("query_params_alias", name="data") + await db.execute_write("create table dogs (id integer primary key, name text)") + await ds.invoke_startup() + + store_response = await ds.client.post( + "/data/-/queries/store", + actor={"id": "root"}, + json={ + "query": { + "name": "by_name", + "sql": "select * from dogs where name = :name", + "params": ["name"], + } + }, + ) + assert store_response.status_code == 400 + assert store_response.json()["errors"] == ["Invalid keys: params"] + assert await ds.get_query("data", "by_name") is None + + await ds.add_query( + "data", + "editable", + "select * from dogs where name = :name", + source="user", + owner_id="root", + ) + update_response = await ds.client.post( + "/data/editable/-/update", + actor={"id": "root"}, + json={"update": {"params": ["name"]}}, + ) + assert update_response.status_code == 400 + assert update_response.json()["errors"] == ["Invalid keys: params"] + + @pytest.mark.asyncio async def test_query_update_api_rejects_trusted_queries_but_internal_update_allowed(): ds = Datasette( From 57ce1a059fce43b32cc54ca4afeeb645e7a6f5af Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 01:13:33 +0000 Subject: [PATCH 43/46] Tighten unstable marker release note Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- docs/changelog.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index 0079aaf2..60918ddf 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -50,7 +50,7 @@ JSON API: other improvements - The :ref:`write API ` endpoints now parse the request body as JSON regardless of the ``Content-Type`` header, so ``curl -d`` invocations work without remembering to set it. Invalid JSON is a ``400`` error. Cross-site request forgery remains prevented by Datasette's ``Origin`` and ``Sec-Fetch-Site`` checks. This also fixes a ``500`` error from the insert API when the ``Content-Type`` header was missing entirely. - New ``Response.error(messages, status=400)`` helper for plugins that need to return a JSON error in Datasette's standard format. See :ref:`internals_response`. - New ``count_truncated`` extra for table JSON, included automatically whenever ``count`` is requested. ``true`` means the count reached Datasette's counting limit and the real number of rows may be higher. See :ref:`json_api_extra`. -- JSON endpoints that are not part of the documented stable API now declare themselves with an ``"unstable"`` key in their responses, making the stability tier machine-readable. +- JSON endpoints that are not part of the documented stable API now declare themselves with an ``"unstable"`` key in their responses. - New documentation covering the grammar for :ref:`boolean query string arguments `, the reason :ref:`upsert ` returns ``200`` where insert returns ``201``, and advice for plugin authors on :ref:`naming secret configuration keys ` so that ``/-/config`` redacts them automatically. .. _v1_0_a35: From 4a853cb10ca6fd0c1aaecf0af7dd26665424c540 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 05:18:54 +0000 Subject: [PATCH 44/46] Use UNSTABLE_API_MESSAGE constant in tests Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- tests/test_api.py | 6 ++---- tests/test_permissions.py | 6 ++---- tests/test_queries.py | 6 ++---- tests/test_success_envelope.py | 15 +++++---------- 4 files changed, 11 insertions(+), 22 deletions(-) diff --git a/tests/test_api.py b/tests/test_api.py index a15a507c..235d394b 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -1,5 +1,6 @@ from datasette.app import Datasette from datasette.plugins import DEFAULT_PLUGINS +from datasette.utils import UNSTABLE_API_MESSAGE from datasette.utils.sqlite import sqlite_version from datasette.version import __version__ from .fixtures import make_app_client, EXPECTED_PLUGINS @@ -251,10 +252,7 @@ def test_no_files_uses_memory_database(app_client_no_files): assert response.status == 200 assert { "ok": True, - "unstable": ( - "This API is not part of Datasette's stable interface" - " and may change at any time" - ), + "unstable": UNSTABLE_API_MESSAGE, "databases": [ { "name": "_memory", diff --git a/tests/test_permissions.py b/tests/test_permissions.py index f8d2c808..88fe577f 100644 --- a/tests/test_permissions.py +++ b/tests/test_permissions.py @@ -3,6 +3,7 @@ from asgiref.sync import async_to_sync from datasette.app import Datasette from datasette.cli import cli from datasette.default_permissions import restrictions_allow_action +from datasette.utils import UNSTABLE_API_MESSAGE from .fixtures import assert_permissions_checked, make_app_client from click.testing import CliRunner from bs4 import BeautifulSoup as Soup @@ -740,10 +741,7 @@ async def test_actor_restricted_permissions( } expected = { "ok": True, - "unstable": ( - "This API is not part of Datasette's stable interface" - " and may change at any time" - ), + "unstable": UNSTABLE_API_MESSAGE, "action": permission, "allowed": expected_result, "resource": expected_resource, diff --git a/tests/test_queries.py b/tests/test_queries.py index 1cda740e..a7e492eb 100644 --- a/tests/test_queries.py +++ b/tests/test_queries.py @@ -8,6 +8,7 @@ from bs4 import BeautifulSoup as Soup from datasette.app import Datasette from datasette.resources import DatabaseResource, QueryResource from datasette.stored_queries import StoredQuery, StoredQueryPage +from datasette.utils import UNSTABLE_API_MESSAGE from datasette.utils.asgi import Forbidden from datasette.utils.sqlite import sqlite3, supports_returning @@ -2183,10 +2184,7 @@ async def test_query_parameters_endpoint_uses_get_sql_only(): assert response.status_code == 200 assert response.json() == { "ok": True, - "unstable": "{}".format( - "This API is not part of Datasette's stable interface" - " and may change at any time" - ), + "unstable": UNSTABLE_API_MESSAGE, "parameters": ["name", "id"], } assert permission_denied_response.status_code == 403 diff --git a/tests/test_success_envelope.py b/tests/test_success_envelope.py index 68b042e5..3c413d73 100644 --- a/tests/test_success_envelope.py +++ b/tests/test_success_envelope.py @@ -8,7 +8,7 @@ objects separately - see /-/plugins, /-/databases, /-/actions.) import pytest from datasette.app import Datasette -from datasette.utils import sqlite3 +from datasette.utils import sqlite3, UNSTABLE_API_MESSAGE @pytest.fixture @@ -114,11 +114,6 @@ async def test_actions_json_is_object(ds_envelope): assert "view-instance" in {action["name"] for action in data["actions"]} -UNSTABLE_MESSAGE = ( - "This API is not part of Datasette's stable interface and may change at any time" -) - - @pytest.mark.asyncio @pytest.mark.parametrize( "path", @@ -137,7 +132,7 @@ async def test_undocumented_endpoints_report_unstable(ds_client, path): finally: ds_client.ds.root_enabled = False assert response.status_code == 200 - assert response.json()["unstable"] == UNSTABLE_MESSAGE + assert response.json()["unstable"] == UNSTABLE_API_MESSAGE @pytest.mark.asyncio @@ -148,12 +143,12 @@ async def test_query_store_and_definition_report_unstable(ds_envelope): actor={"id": "root"}, ) assert store.status_code == 201 - assert store.json()["unstable"] == UNSTABLE_MESSAGE + assert store.json()["unstable"] == UNSTABLE_API_MESSAGE definition = await ds_envelope.client.get( "/data/unstable_check/-/definition", actor={"id": "root"} ) assert definition.status_code == 200 - assert definition.json()["unstable"] == UNSTABLE_MESSAGE + assert definition.json()["unstable"] == UNSTABLE_API_MESSAGE @pytest.mark.asyncio @@ -164,7 +159,7 @@ async def test_permissions_post_reports_unstable(ds_envelope): actor={"id": "root"}, ) assert response.status_code == 200 - assert response.json()["unstable"] == UNSTABLE_MESSAGE + assert response.json()["unstable"] == UNSTABLE_API_MESSAGE @pytest.mark.asyncio From be25d6e3e41117a2217a986f11ca9467eac6f558 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 05:18:54 +0000 Subject: [PATCH 45/46] Remove test_query_list_json_signals_pagination_via_next_only Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- tests/test_queries.py | 24 ------------------------ 1 file changed, 24 deletions(-) diff --git a/tests/test_queries.py b/tests/test_queries.py index a7e492eb..ffa948a9 100644 --- a/tests/test_queries.py +++ b/tests/test_queries.py @@ -3866,27 +3866,3 @@ async def test_stored_query_json_uses_parameters_not_params(): query = [q for q in listing["queries"] if q["name"] == "with_params"][0] assert query["parameters"] == ["name", "age"] assert "params" not in query - - -@pytest.mark.asyncio -async def test_query_list_json_signals_pagination_via_next_only(): - ds = Datasette(memory=True) - ds.add_memory_database("query_list_next_only", name="data") - await ds.invoke_startup() - for i in range(3): - await ds.add_query( - "data", - name="q{}".format(i), - sql="select {}".format(i), - ) - first = (await ds.client.get("/data/-/queries.json?_size=2")).json() - assert "has_more" not in first - assert first["next"] is not None - assert first["next_url"] is not None - # The internal test client cannot follow absolute URLs - next_path = first["next_url"].replace("http://localhost", "") - assert next_path.startswith("/data/-/queries.json?") - last = (await ds.client.get(next_path)).json() - assert "has_more" not in last - assert last["next"] is None - assert last["next_url"] is None From b7bbde04bed8efad028d646d636a4364c31cf618 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 05:18:54 +0000 Subject: [PATCH 46/46] Link the consistency review release note to PR #2824 Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GrHZSypDfMnym1tM5XJAFZ --- docs/changelog.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index 60918ddf..d48ab152 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -17,7 +17,7 @@ Unreleased - The table and row JSON APIs now support ``?_extra=column_details`` for returning SQLite schema details for columns, including declared type, SQLite affinity, primary key, ``NOT NULL``, default and hidden-column metadata. - POST bodies that Datasette reads fully into memory - such as JSON submitted to the write API - are now capped by the new :ref:`setting_max_post_body_bytes` setting, defaulting to 2MB. Oversized requests are rejected with an HTTP 413 error as soon as the limit is exceeded, protecting smaller servers from memory exhaustion. File uploads are unaffected - ``request.form()`` streams those to disk and has its own separate limits. -This release also includes the results of a detailed consistency review of Datasette's JSON API in preparation for the 1.0 stable release. Several of these changes are backwards-incompatible with previous 1.0 alphas. The new :ref:`API stability documentation ` describes exactly which parts of the JSON API are covered by the 1.0 stability promise. +This release also includes the results of a `detailed consistency review `__ of Datasette's JSON API in preparation for the 1.0 stable release. Several of these changes are backwards-incompatible with previous 1.0 alphas. The new :ref:`API stability documentation ` describes exactly which parts of the JSON API are covered by the 1.0 stability promise. JSON API: breaking changes ~~~~~~~~~~~~~~~~~~~~~~~~~~