diff --git a/existing-api.md b/existing-api.md deleted file mode 100644 index c58f8f41..00000000 --- a/existing-api.md +++ /dev/null @@ -1,1244 +0,0 @@ -# Datasette JSON API — As Implemented - -This document describes the JSON API of this Datasette codebase (version `1.0a35`) as -derived directly from the source code. It intentionally ignores the existing `docs/` -directory: every claim below is based on the route table in `datasette/app.py` -(`Datasette._routes()`, app.py:2507-2767) and the view implementations in -`datasette/views/`. - -## Contents - -- [Cross-cutting behavior](#cross-cutting-behavior) -- [Instance endpoints](#instance-endpoints) -- [Database endpoints](#database-endpoints) -- [Table and row read endpoints](#table-and-row-read-endpoints) -- [The write API](#the-write-api) -- [Stored (canned) queries API](#stored-canned-queries-api) -- [Authentication and tokens](#authentication-and-tokens) -- [Appendix: registered actions (permissions)](#appendix-registered-actions-permissions) - ---- - -## Cross-cutting behavior - -### URL formats and content negotiation - -- Most read endpoints are registered with an optional format suffix: - `/(...)(\.(?Pjson))?$`. The bare path returns HTML; the `.json` - extension returns JSON. -- Table, row and query routes accept any `\w+` format extension; formats other - than the built-in `html`, `json`, `csv`, `blob` must be provided by a plugin - via `register_output_renderer`, otherwise the request 404s. -- HTML responses include a `Link: <...>; rel="alternate"; - type="application/json+datasette"` header pointing at the `.json` variant - (views/base.py:141-159), unless the view opts out with - `has_json_alternate = False`. -- Database, table, row and query names in paths are **tilde-encoded** - (a percent-encoding variant using `~` as the escape character; - utils/__init__.py `_TILDE_ENCODING_SAFE`). Multi-column primary keys in row - URLs are comma-separated. -- JSON responses are always compact `json.dumps` output serialized by - `CustomJSONEncoder`; there is no pretty-printing query parameter. Binary - values are serialized as `{"$base64": true, "encoded": "..."}`. -- Success content type: `application/json; charset=utf-8` - (`_shape=array&_nl=on` responses use `text/plain`). - -### Success envelope and stability marker - -Every JSON endpoint that returns an object includes `"ok": true` on -success. `JsonDataView` injects it automatically for dict responses -(views/special.py); the homepage, jump, schema, permission-debug and -autocomplete views add it explicitly. The former top-level-array endpoints -(`/-/plugins`, `/-/databases`, `/-/actions`) now return objects wrapping -their arrays (`{"ok": true, "plugins": [...]}` etc.). - -JSON endpoints that are **not part of the documented API** include a -marker key (`UNSTABLE_API_MESSAGE` in utils/__init__.py): - -```json -"unstable": "This API is not part of Datasette's stable interface and may change at any time" -``` - -Currently: the homepage (`/.json`, `/-/.json`), `/db/-/queries/analyze`, -`POST /db/-/queries/store`, `/db//-/definition`, -`/db/-/query/parameters`, `/db/-/execute-write/analyze` and the -`POST /-/permissions` playground response. - -### Error shape (canonical) - -Every JSON error response uses one canonical shape, built by `error_body()` -(utils/__init__.py): - -```json -{ - "ok": false, - "error": "all messages joined with '; '", - "errors": ["message", "..."], - "status": 404 -} -``` - -- `errors` is a list of one or more message strings (multi-message - validation errors, e.g. per-row insert errors, list them all). -- `error` is the messages joined with `"; "`. -- `status` always matches the HTTP status code. - -The shape is produced by four code paths, all delegating to `error_body()`: - -1. **Exception handler** (handle_exception.py) — `NotFound`, - `DatasetteError`, `BadRequest` etc. on `.json` paths. `DatasetteError` - `error_dict` context keys are merged in; the legacy `title` key is no - longer emitted in JSON (it survives in the HTML error template context). -2. **The `_error()` helper** (views/base.py:183-184) — the write API, - stored-query API, execute-write and permission-denied paths. -3. **JSON renderer errors** (renderer.py) — SQL errors on table/query - endpoints return HTTP 400 with the canonical keys **plus** the context - keys of the response it could not produce: - - ```json - {"ok": false, "error": "no such table: x", "errors": ["no such table: x"], - "status": 400, "rows": [], "truncated": false} - ``` - - Invalid `_shape=` values and `_shape=object` misuse (on queries or - pk-less tables) also return canonical 400 errors. -4. **Permission debug endpoints** (`/-/allowed`, `/-/rules`, `/-/check`, - POST `/-/permissions`) — canonical shape (previously bare - `{"error": ...}` objects). - -Method-not-allowed responses return HTTP 405 with the canonical shape when -the path ends in `.json` or the request content type is `application/json`; -plain text otherwise (views/base.py). - -**`Forbidden` handling:** when a view raises `Forbidden` (e.g. via -`ensure_permission`), the default `forbidden()` plugin hook returns the -canonical JSON error with status 403 when the path ends in `.json` or the -request carries an `Accept: application/json` / `Content-Type: -application/json` header; other requests get an HTML error page -(forbidden.py). Endpoints that check permissions themselves return -`_error(..., 403)` JSON directly. - -### CORS - -When Datasette is started with `--cors`, responses gain -(utils/__init__.py:1297-1302): - -``` -Access-Control-Allow-Origin: * -Access-Control-Allow-Headers: Authorization, Content-Type -Access-Control-Expose-Headers: Link -Access-Control-Allow-Methods: GET, POST, HEAD, OPTIONS -Access-Control-Max-Age: 3600 -``` - -### CSRF / cross-origin protection - -Datasette uses header-based cross-origin protection -(`CrossOriginProtectionMiddleware`, csrf.py:67-178) rather than CSRF tokens -for API calls. For non-GET/HEAD/OPTIONS requests: - -1. Requests carrying `Authorization: Bearer ...` **and no `Cookie` header** - bypass the check entirely (csrf.py:98-110). -2. Otherwise `Sec-Fetch-Site` must be `same-origin` or `none`; other values → 403. -3. If neither `Sec-Fetch-Site` nor `Origin` is present (curl, API clients), - the request passes. -4. Fallback: `Origin` must exactly match the request scheme/host/port → else 403. - -Plain JSON API clients (no cookies, no browser headers) are never blocked; -`Content-Type: application/json` itself plays no role in the CSRF decision. - -### Settings that govern the API - -From `SETTINGS` (app.py:197-287): `default_page_size` (100), -`max_returned_rows` (1000), `max_insert_rows` (100), `sql_time_limit_ms` -(1000), `default_facet_size` (30), `facet_time_limit_ms` (200), -`allow_facet` (true), `allow_download` (true), `allow_signed_tokens` (true), -`default_allow_sql` (true), `max_signed_tokens_ttl` (0), `default_cache_ttl` -(5), `allow_csv_stream` (true), `max_csv_mb` (100), `force_https_urls` -(false), `trace_debug` (false), `base_url` ("/"). - -### The JSON renderer: `_shape`, `_nl`, `_json`, `_json_infinity` - -`json_renderer` (renderer.py:31-126) processes `.json` output for table, row -and query views (but **not** for the instance/database/debug endpoints, which -build JSON directly): - -- **`_shape`** (default `objects`): - - `objects` — `{"ok": true, "rows": [{col: val}, ...], "truncated": false, ...}` - - `arrays` — same envelope, each row a list of values - - `array` — response body is a bare JSON array of row objects - - `arrayfirst` — bare JSON array of the first column's values - - `object` — table views only: an object keyed by primary-key string. - On queries or tables without primary keys: a canonical 400 error - (`_shape=object is only available on tables` / - `_shape=object not available for tables with no primary keys`). - - anything else — canonical HTTP 400 error `Invalid _shape: x` -- **`_nl=on`** — with `_shape=array` only: newline-delimited JSON, `text/plain`. -- **`_json=COLUMN`** (repeatable) — parse that column's string values with - `json.loads` so they nest as JSON; parse failures leave the value unchanged. -- **`_json_infinity=1`** — preserve `Infinity`/`-Infinity`; by default they - are replaced with `null`. -- `columns` is stripped from dict-shaped output unless `?_extra=columns` was - requested (renderer.py:110-113). -- If a SQL error occurred, `_shape` is ignored, HTTP status is 400 and the - envelope carries the canonical error keys alongside `rows`/`truncated`. - -### The `?_extra=` system - -Table, row and query JSON responses support `?_extra=` (repeatable and/or -comma-separated, extras.py:9-14) to add keys to the response. Extras are -scope-registered (`ExtraScope.TABLE` / `ROW` / `QUERY`) and only **public** -extras are available over JSON (extras.py:73-92). Unknown extra names (and -internal HTML-only names) on data formats return 400 -`Unknown _extra: `; HTML pages ignore them. The available names per -scope are listed with the relevant endpoints below. - ---- - -## Instance endpoints - -Most of these are implemented with `JsonDataView` (views/special.py:30-79): -GET-only; bare path renders an HTML page (`show_json.html`), `.json` returns -the data; permission defaults to `view-instance` and denial raises -`Forbidden` → **HTML** 403 page. - -### GET / - -Routes: `/(\.(?Pjsono?))?$` and `/-/(\.(?Pjsono?))?$` -(app.py:2517-2518); `/-` permanently redirects to `/-/`. `IndexView` -(views/index.py:22-189). `GET /.json`, `/.jsono` and `/-/.json` return JSON. - -- **Permission:** `view-instance` (denied → 403). Databases and tables are - further filtered by `view-database` / `view-table` for the actor. -- **Parameters:** `_sort=relationships` sorts each database's truncated table - list by foreign-key relationship count. -- **JSON response** (index.py:147-161) — includes `ok: true` plus: - - `databases` — a **list** of database objects (undocumented API, subject - to change). Each item: `name`, `hash` (or null), `color`, `path`, - `tables_and_views_truncated` (up to 5 items: `name`, `columns`, - `primary_keys`, `count` (int or null), `hidden`, `fts_table`, - `num_relationships_for_sorting`, `private`; view items are just - `{"name", "private"}`), `tables_and_views_more` (bool), `tables_count`, - `table_rows_sum`, `show_table_row_counts`, `hidden_table_rows_sum`, - `hidden_tables_count`, `views_count`, `private`. - - `metadata` — instance metadata object. - -### GET /-/versions(.json) - -`JsonDataView` over `Datasette._versions` (app.py:2548-2551, 2171-2245). -Permission `view-instance`. No parameters. - -Response keys: `python` (`{version, full}`), `datasette` (`{version}` plus -optional `note`), `asgi` (`"3.0"`), `uvicorn` (string or null), `sqlite` -(`{version, fts_versions, extensions, compile_options}`; `extensions` -includes `json1` and optionally `spatialite`), `pysqlite3` (only when -running under pysqlite3). - -### GET /-/plugins(.json) - -app.py:2552-2557, `Datasette._plugins` (app.py:2247-2266). Permission -`view-instance`. - -- **Parameters:** `?all=1` — include Datasette's built-in default plugins - (filtered out by default). -- **Response:** `{"ok": true, "plugins": [...]}` — each plugin is - `{"name", "static", "templates", "version", "hooks"}`, sorted by name. - -### GET /-/settings(.json) - -app.py:2558-2561. Permission `view-instance`. No parameters. Returns a flat -object mapping every setting name (see [Settings](#settings-that-govern-the-api)) -to its effective value. - -### GET /-/config(.json) - -app.py:2562-2565. Permission `view-instance`. No parameters. Returns the full -`datasette.yaml` configuration dict passed through -`redact_keys(config, ("secret", "key", "password", "token", "hash", "dsn"))` -(app.py:2502-2505) — any dict key containing one of those substrings has its -value replaced by `"***"` (utils/__init__.py:1532-1556). - -### GET /-/threads(.json) - -app.py:2566-2569, `Datasette._threads` (app.py:2268-2285). Permission -**`permissions-debug`** (exposes runtime internals). No parameters. - -Response: `num_threads`, `threads` (list of `{name, ident, daemon}`), -`num_tasks`, `tasks` (asyncio task repr strings). When the -`num_sql_threads` setting is 0 the response is exactly -`{"num_threads": 0, "threads": []}`. - -### GET /-/databases(.json) - -app.py:2570-2573, `Datasette._connected_databases` (app.py:2157-2169). -Permission `view-instance`. No parameters. - -Response: `{"ok": true, "databases": [...]}` — each database is -`{"name", "route", "path", "size", "is_mutable", "is_memory", "hash"}`. -Only databases the actor is allowed to `view-database` are listed. - -### GET /-/actor(.json) - -app.py:2574-2579, registered with `permission=None` — **accessible to any -request including anonymous**. No parameters. - -Response: `{"ok": true, "actor": {...}}` or `{"ok": true, "actor": null}` (app.py:2287-2288). - -### GET /-/actions(.json) - -app.py:2580-2589. Permission **`permissions-debug`**. No parameters. - -Response: `{"ok": true, "actions": [...]}` — each action is -`{"name", "abbr", "description", "takes_parent", "takes_child", -"resource_class", "also_requires"}`, sorted by name (app.py:2290-2304). - -### GET /-/auth-token - -`AuthTokenView` (app.py:2590-2593, views/special.py:198-217). GET only, no -`.json` variant, HTML/redirect only. - -- **Parameter:** `token` — the one-time secret printed by `datasette --root`. -- Match → invalidates the token, sets the signed `ds_actor` cookie to - `{"id": "root"}` and 302-redirects to the homepage. Mismatch or reuse → - `Forbidden` → 403 HTML. - -### GET/POST /-/create-token - -`CreateTokenView` (app.py:2594-2597, views/special.py:727-856). **HTML form -endpoint only — there is no JSON request/response mode in this codebase** -(`has_json_alternate = False`; the POST body must be form-encoded, a JSON -content type raises `BadRequest` → 400). - -- **Gates** (each failure → `Forbidden` → 403): `allow_signed_tokens` must be - on; request must have an actor with an `id`; the actor must not itself be - token-derived. -- **POST fields:** `expire_type` (`""`/`minutes`/`hours`/`days`), - `expire_duration` (positive int), plus restriction checkboxes named - `all:`, `database::`, - `resource:::`. -- **Response:** HTML page containing the new `dstok_` token. -- Programmatic alternatives: `datasette create-token` CLI or - `datasette.create_token()`. - -### GET /-/api - -`ApiExplorerView` (app.py:2598-2601, views/special.py:859-1020). HTML API -explorer, GET only. Permission `view-instance` (403 on denial). - -### GET /-/jump(.json) - -`JumpView` (app.py:2602-2605, views/special.py:1023-1201). The route allows -an optional `.json` suffix but the view **always returns JSON**. - -- **Permission:** none checked directly; results are filtered via - `allowed_resources_sql` for the current actor (default items come from the - `jump_items_sql` plugin hook). -- **Parameter:** `q` — whitespace-split terms matched as a case-insensitive - `%term1%term2%` LIKE pattern. -- **Response:** `{"ok": true, "matches": [...], "truncated": bool}`; each match: - `name`, `url`, `type` (`database`/`table`/`view`/`query`/plugin-defined), - `description`, optional `display_name`. Capped at 100 matches. - -### GET /-/schema(.json|.md) - -`InstanceSchemaView` (app.py:2610-2613, views/special.py:1257-1293). - -- **Permission:** no explicit check; only databases the actor can - `view-database` are included (others silently omitted). -- **Formats:** no extension → HTML; `.json` → - `{"ok": true, "schemas": [{"database": name, "schema": "..."}]}`; `.md` → - `text/markdown` rendering. - -### GET/POST /-/logout - -`LogoutView` (app.py:2614-2617, views/special.py:220-238). HTML endpoint. -GET renders a confirmation page (or redirects if anonymous); POST deletes the -`ds_actor` cookie and 302-redirects to `/`. - -### GET/POST /-/permissions - -`PermissionsDebugView` (app.py:2618-2621, views/special.py:241-295). No -`.json` route. Both methods require `view-instance` **and** -`permissions-debug` (403 on denial). - -- **GET** — HTML permission-check log; `?filter=all|exclude-yours|only-yours`. -- **POST** — form-encoded `actor` (JSON string), `permission`, optional - `resource_1`, `resource_2`; returns **JSON** - `{"action", "allowed", "resource": {"parent", "child", "path"}}` plus - `actor_id` when present. Errors: unknown action → 404; child without - parent → 400 (both canonical error shape). - -### GET /-/allowed(.json) - -`AllowedResourcesView` (app.py:2622-2625, views/special.py:298-460). Bare -path always renders the HTML form; `.json` returns JSON. - -- **Permission:** none — reports the **current actor's own** allowed - resources. Items gain a `reason` field if the actor also holds - `permissions-debug`. -- **Parameters:** `action` (required; missing → 400 canonical error, unknown - → 404), `parent`, `child` (requires `parent`), `_page` (default 1), - `_size` (default 50, maximum 200, accepts `max`; out-of-range → 400). -- **Response:** `{"action", "actor_id", "page", "page_size", "total", - "items": [{"parent", "child", "resource"}]}` with optional `next_url` / - `previous_url`. - -### GET /-/rules(.json) - -`PermissionRulesView` (app.py:2626-2629, views/special.py:463-584). -Permission `view-instance` **and** `permissions-debug`. Parameters and error -shapes as `/-/allowed`. Response items: -`{"parent", "child", "resource", "allow" (1|0), "reason", "source_plugin"}`. - -### GET /-/check(.json) - -`PermissionCheckView` (app.py:2630-2633, views/special.py:633-662). -Permission `permissions-debug`. Parameters `action` (required), `parent`, -`child`. Checks the **current request's actor**; response -`{"action", "allowed", "resource": {...}}` plus `actor_id`. - -### GET/POST /-/messages - -`MessagesDebugView` (app.py:2634-2637, views/special.py:703-724). HTML debug -tool for flash messages; permission `view-instance`; POST is form-encoded -(`message`, `message_type` = INFO/WARNING/ERROR/all) and 302-redirects. - -### GET /-/allow-debug - -`AllowDebugView` (app.py:2638-2641, views/special.py:665-700). GET only, HTML -only, **no permission required**. Parameters `actor` and `allow` (JSON -strings); renders the result of `actor_matches_allow()` in the page. - -### GET /-/patterns - -Pattern portfolio page (app.py:2642-2645). HTML only; not part of the JSON API. - -### GET /-/debug/autocomplete - -`AutocompleteDebugView` (app.py:2646-2649, views/special.py:94-195). HTML -debug page for the table autocomplete API; permission `view-instance` plus -`view-table` when `?database=&table=` are supplied. - ---- - -## Database endpoints - -### GET /\.db - -Downloads the raw SQLite file. Route → `database_download` -(app.py:2650-2653; views/database.py:533-570). - -- **Permission:** `view-database-download` (denied → `Forbidden` → 403 HTML). -- **Other gates:** unknown database → 404 `"Invalid database"`; in-memory - database → 404; `allow_download` off **or** mutable database → - `Forbidden("Database download is forbidden")`; no file path → 404. -- **Response:** streamed `application/octet-stream` with a - `content-disposition` attachment; immutable databases with a known hash set - `Etag` and honor `If-None-Match` → 304. - -### GET /\(.json) - -`DatabaseView` (app.py:2654-2657; views/database.py:71-277). Only `html` and -`json` formats are accepted; any other extension → 404 `"Invalid format: ..."`. - -- **Permission:** `view-database` via `check_visibility` (denied → - `Forbidden` → 403 HTML). Table/view listings are filtered by `view-table`; - stored queries by `view-query`. -- **Parameters:** - - `?sql=` — non-blank value 302-redirects to `//-/query?...` - preserving the query string and format. - - No `?_extra=` and no `_shape` support — the JSON is built directly and - returned via `Response.json`, bypassing the JSON renderer - (views/database.py:189-212). -- **JSON response** (all keys always present): - - `ok` — always `true` - - `database` — name; `private` — bool; `path` — URL path; `size` — bytes - - `tables` — list (includes hidden tables), each: - `name`, `columns` (names), `primary_keys`, `count` (int or null, - time-boxed), `count_truncated` (bool — count is a capped lower bound), - `hidden`, `fts_table`, `foreign_keys` (`{incoming: [...], outgoing: [...]}` - of `{other_table, column, other_column}`), `private` - - `hidden_count` — number of hidden tables - - `views` — list of `{name, private}` - - `queries` — **up to 5** stored queries (canonical stored-query objects, - see the stored-queries section); `queries_more` (bool); - `queries_count` (total visible) - - `allow_execute_sql` — bool for this actor - - `table_columns` — `{table: [columns]}`, empty `{}` unless - `allow_execute_sql` (views map to `[]`) - - `metadata` — database metadata dict - -### GET /\/-/query(.json) — arbitrary SQL - -`QueryView` (app.py:2691-2694; views/database.py:573-1130). The same class -also executes stored queries dispatched from the table route (see stored -queries section). - -- **Permission:** `execute-sql` on the database via `check_visibility` - (denied → `Forbidden` → 403 HTML). -- **Parameters:** - - `sql` — SQL to run. Must pass `validate_sql_select` - (utils/__init__.py:345-354): after stripping `--` comment lines it must - start with `select`, `with` or an `explain` variant, and must not contain - `pragma` (except allowlisted `pragma_*()` table-valued functions). - Failure → 400 `DatasetteError` titled `"Invalid SQL"` → JSON - `{"ok": false, "error": "Statement must be a SELECT", "status": 400, - "title": "Invalid SQL"}`. - - Any other `name=value` pair supplies the `:name` named parameter; missing - parameters default to `""`. Names starting with `_` are excluded. - - `_timelimit` — per-request SQL time limit in ms. - - `_shape`, `_nl`, `_json`, `_json_infinity` — see the JSON renderer section. - - `_extra` — QUERY-scope extras: `columns`, `debug`, `request`, - `render_cell`, `query` (`{"sql", "params"}`), `metadata`, `database`, - `database_color`, `private`, `extras`. -- **Response** (default shape): - `{"ok": true, "rows": [{col: val}, ...], "truncated": false}` plus any - requested extras. `truncated: true` when the result hit `max_returned_rows`. -- **Errors:** - - SQLite errors (e.g. `no such table`) are **not** raised — they surface as - HTTP 400 `{"ok": false, "error": "", "rows": [], "truncated": false}`. - - Time limit → 400 titled `"SQL Interrupted"` (the `error` value contains - an HTML fragment). - - `?sql=` omitted or blank → 400 `"?sql= is required"` for all data - formats (`.json`, `.csv`, plugin formats). The HTML page remains the - SQL editor. -- `.csv` streams CSV; unknown extensions → 404. - -### GET /\/-/query/parameters - -`QueryParametersView` (app.py:2687-2690; views/stored_queries.py:26-51). - -- **Permission:** `execute-sql` → 403 JSON - `{"ok": false, "errors": ["Permission denied: need execute-sql"]}`. -- **Parameters:** only `sql` (default `""`); any other key → 400 - `"Invalid keys: ..."`. -- **Response:** 200 `{"ok": true, "parameters": ["name1", ...]}`. SQL with a - parameter beginning `_` → 400 `"Magic parameters are not allowed"`. -- Responses carry `Content-Security-Policy: frame-ancestors 'none'` and - `X-Frame-Options: DENY`. - -### POST /\/-/create - -`TableCreateView` (app.py:2658; views/table_create_alter.py:785-962). -GET → 405. Body is parsed as JSON regardless of content type; invalid JSON → -400 `{"ok": false, "errors": ["Invalid JSON: ..."]}`. - -- **Permissions** (all denials → 403 canonical error JSON, - all checked at the **database** level): - - `create-table` — always required (`["Permission denied"]`) - - `insert-row` — if `rows`/`row` provided (`need insert-row`) - - `update-row` — if `replace: true` (`need update-row`) - - `alter-table` — if `alter: true` on an **existing** table - (`need alter-table`); when the table does not exist yet and rows are - supplied, alter is enabled automatically. -- **Request schema** (pydantic `CreateTableRequest`, extra keys forbidden → - 400 `"Invalid keys: a, b"`): - - `table` (required) — must match `^(?!sqlite_)[^\n]+$` - - `rows` (list of objects) / `row` (single object) — mutually exclusive - - `columns` — list of `{name, type, fk_table, fk_column, not_null, - default, default_expr}`; mutually exclusive with `rows`/`row`; `type` one - of `text`/`integer`/`float`/`blob` (default `text`); `default` and - `default_expr` mutually exclusive; `default_expr` one of - `current_timestamp`, `current_date`, `current_time`, `current_unixtime`, - `current_unixtime_ms`. At least one of `columns`/`rows`/`row` required. - - `pk` (string) / `pks` (list) — mutually exclusive. For an existing table - a differing pk → 400 `"pk cannot be changed for existing table"`. - - `ignore` / `replace` (bools) — mutually exclusive; require `row`/`rows` - and `pk`/`pks`. - - `alter` (bool) — add missing columns when inserting into an existing table. -- **Success** — **201**: - ```json - {"ok": true, "database": "...", "table": "...", - "table_url": "https://.../db/table", "table_api_url": "https://.../db/table.json", - "schema": "CREATE TABLE ...", "row_count": 2} - ``` - `row_count` only when rows were inserted. Write failures → 400 - `{"ok": false, "errors": [""]}`. Emits `create-table` / - `insert-rows` / `alter-table` events. - -### POST /\/-/execute-write - -`ExecuteWriteView` (app.py:2679-2682; views/execute_write.py:236-476). GET on -the same path renders an HTML form (requires `execute-write-sql`). - -- **Permission (POST):** `execute-write-sql` → 403 - `{"ok": false, "errors": ["Permission denied: need execute-write-sql"]}`; - immutable database → 403 `["Database is immutable"]`. -- **Per-statement permissions:** the SQL is analyzed - (`decision_for_write_sql_operation`, write_sql.py:63-189) and each - operation must pass: - - | Operation | Requirement | - |---|---| - | `select` / internal ops / function calls | ignored | - | read of a table | `view-table` on that table | - | `insert` or `update` | **all of** `insert-row`, `update-row`, `delete-row` on the table | - | `delete` | `delete-row` | - | `create table` | `create-table` on the database | - | `alter table`, `create index`, `drop index` | `alter-table` on the table | - | `drop table` | `drop-table` | - | `VACUUM`, virtual-table writes, shadow-table writes | rejected outright (403) | - | statements touching attached databases | rejected (403) | - -- **Body:** JSON (`{"sql": ..., "params": {...}}` — only those two keys) or - form-encoded (`sql` plus one field per parameter, `_sql_param_` prefix - stripped). Validation errors (400): `"SQL is required"`, - `"params must be a dictionary"`, `"Unknown parameters: a, b"`, - `"Magic parameters are not allowed"`, `"Could not analyze query: ..."`, - `"Use /-/query for read-only SQL; this endpoint only executes writes"`. -- **JSON is returned when** the body was JSON, `Accept: application/json`, or - a truthy `_json` field is present; otherwise HTML. -- **Success** — 200: - ```json - {"ok": true, "message": "Query executed, 1 row affected", "rowcount": 1, - "rows": [], "truncated": false, - "analysis": [{"operation": "insert", "database": "db", "table": "t", - "required_permission": "insert-row, update-row, delete-row", - "source": null}]} - ``` - `rows` is populated by `RETURNING` clauses. SQLite errors → 400 - `{"ok": false, "errors": [""]}`. Anti-framing headers on all - responses. - -### GET /\/-/execute-write/analyze - -`ExecuteWriteAnalyzeView` (app.py:2675-2678; views/execute_write.py:479-507). - -- **Permission:** `execute-write-sql` → 403 `errors` JSON. -- **Parameters:** only `sql` allowed (else 400 `"Invalid keys: ..."`). -- **Response** — 200 even when analysis fails (`ok: false` in body): - `{"ok", "parameters", "analysis_error", "analysis_rows": - [{operation, database, table, required_permission, source, allowed}], - "execute_disabled", "execute_disabled_reason"}`. `allowed` is a per-actor - permission check result (true/false/null). - -### GET /\/-/foreign-key-targets - -`DatabaseForeignKeyTargetsView` (app.py:2659-2662; -views/table_create_alter.py:965-1005). - -- **Parameter:** `table` (optional) — only used for the permission check. -- **Permission:** `create-table` on the database, **or** `alter-table` on - `?table=` when it names an existing table. Neither → 403 - `{"ok": false, "errors": ["Permission denied: need create-table"]}`. -- **Response:** 200 `{"ok": true, "database": "...", "targets": - [{"fk_table", "fk_column", "type"}]}` — every non-hidden table with exactly - one primary-key column; `type` is the pk's SQLite type affinity. - -### GET /\/-/schema(.json|.md) - -`DatabaseSchemaView` (app.py:2683-2686; views/special.py:1296-1329). - -- **Permission:** `view-database` (denied → `Forbidden` → 403 HTML). -- **Unknown database** → 404; for `.json`: - `{"ok": false, "error": "Database not found"}`. The permission check runs - first, so unauthorized actors cannot probe for database existence. -- **Responses:** `.json` → 200 `{"ok": true, "database": "", "schema": ""}` - (concatenated `sqlite_master.sql` joined with `;\n`); `.md` → - `text/markdown`; no extension → HTML. - ---- - -## Table and row read endpoints - -### GET /\/\.json - -Route `r"/(?P[^\/\.]+)/(?P
[^\/\.]+)(\.(?P\w+))?$"` → -`table_view` (app.py:2711-2714; views/table.py:1670). Serves both tables and -SQL views. GET/HEAD only — POST returns a plain-text 405. If the name is -neither a table nor a view but matches a stored query, the request is -dispatched to `QueryView` (views/table.py:1703-1712). - -**Permission:** `view-table` via `check_visibility`; denial raises -`Forbidden` → **HTML** 403 page even for `.json`. Unknown table → -`TableNotFound` → 404 (JSON error shape for `.json` paths). - -**Default JSON keys** (views/table.py:2308-2332 + renderer): - -| Key | Meaning | -|---|---| -| `ok` | `true` when data was retrieved without error | -| `next` | pagination token string, or `null` on the last page | -| `next_url` | absolute URL of the next page, or `null` on the last page | -| `rows` | list of row objects `{column: value}` (default `_shape=objects`) | -| `truncated` | always present; `false` for table pages | - -`columns` is computed but removed unless `?_extra=columns` was requested. -When there is a next page the response carries a -`Link: ; rel="next"` header (views/table.py:1911-1912). - -**`?_extra=` options** (TABLE scope; registry -views/table_extras.py:1197-1235; unknown names silently ignored): - -| `_extra=` | Returns | -|---|---| -| `count` | total matching-row count, computed with a `limit 10001` subquery so it caps at 10001; `null` with `_nocount` or on count timeout. Requesting `count` implicitly includes `count_truncated` | -| `count_truncated` | `true` when `count` hit the counting limit (the real count is at least the reported value) | -| `count_sql` | the SQL used for the count | -| `facet_results` | `{"results": {name: facet}, "timed_out": [...]}`; each facet: `{name, type, hideable, toggle_url, results: [{value, label, count, toggle_url, selected}], truncated}` | -| `facets_timed_out` | facet names that exceeded `facet_time_limit_ms` | -| `suggested_facets` | `[{name, toggle_url, (type)}]`; empty when suggestion is disabled or paginating | -| `human_description_en` | English description of filters + sort | -| `next_url` | absolute URL of the next page or `null` | -| `columns` | column names of the returned rows | -| `all_columns` | all table columns regardless of `_col`/`_nocol` | -| `primary_keys` | pk column names (empty for rowid tables and views) | -| `display_columns` | HTML-oriented column metadata | -| `render_cell` | per-row plugin-rendered HTML strings | -| `debug` | `{url_vars, resolved, nofacet, nosuggest}` — explicitly unstable | -| `request` | `{url, path, full_path, host, args}` | -| `query` | `{sql, params}` of the main query | -| `column_types` | `{column: {type, config}}` assigned column types | -| `set_column_type_ui` | UI helper, `null` unless actor has `set-column-type` | -| `metadata` | table metadata dict including column descriptions | -| `extras` | self-describing list of all available extras | -| `database`, `table`, `database_color` | identity/display values | -| `renderers` | `{format_name: url}` of formats that can render this data | -| `custom_table_templates` | template lookup list | -| `sorted_facet_results` | facets as a display-ordered list | -| `table_definition` | `CREATE TABLE` SQL | -| `view_definition` | `CREATE VIEW` SQL, `null` for tables | -| `is_view` | boolean | -| `private` | `true` if visible to this actor but not anonymously | -| `expandable_columns` | `[[foreign_key, label_column_or_null], ...]` | -| `form_hidden_args` | pairs of `_`-prefixed args for HTML forms | - -Non-public extras (`actions`, `filters`, `display_rows`) are HTML-only and -never appear in JSON. `_extra=_html` expands to the full HTML bundle -(views/table_extras.py:1162-1194). Any `_facet*` argument implicitly adds -`facet_results`; `_shape=object` implicitly adds `primary_keys` -(views/table.py:2252-2256). There is **no** `filtered_table_rows_count` -extra — it was replaced by `count`. - -**Column filters `?__=`** (filters.py:260-427). Any -querystring key not starting with `_` is a filter; bare `?column=value` means -`exact`. Columns whose names start with `_` can be filtered as -`?_col__exact=`. Operators: - -| op | SQL | -|---|---| -| `exact` | `"col" = :p` (default) | -| `not` | `"col" != :p` | -| `contains` / `notcontains` | `like '%v%'` / `not like '%v%'` | -| `endswith` / `startswith` | `like '%v'` / `like 'v%'` | -| `gt` / `gte` / `lt` / `lte` | `>` `>=` `<` `<=` (numeric strings cast to int) | -| `like` / `notlike` | raw `like` / `not like` pattern | -| `glob` | `glob` | -| `in` / `notin` | comma-separated list, or JSON array if the value starts with `[` | -| `arraycontains` / `arraynotcontains` | `[not] in (select value from json_each("col"))` (requires JSON1) | -| `date` | `date("col") = :p` | -| `isnull` / `notnull` | `is null` / `is not null` (no value) | -| `isblank` / `notblank` | `(is null or = '')` / opposite (no value) | - -**Special (underscore) parameters:** - -| Param | Behavior | -|---|---| -| `_where=SQL` | extra raw where clause (repeatable); requires `execute-sql` else 403 `"_where= is not allowed"` | -| `_search=q` | FTS against the table's FTS table | -| `_search_=q` | FTS restricted to one column; 400 if invalid | -| `_searchmode=raw` | pass the query straight to `match` | -| `_fts_table=` / `_fts_pk=` | override the FTS table / pk used for joins | -| `_through={"table","column","value"}` | filter via an incoming foreign key (repeatable, JSON value) | -| `_sort=col` / `_sort_desc=col` | sort; 400 if both given or column not sortable | -| `_next=token` | pagination token | -| `_size=N\|max` | page size; default `default_page_size` (100); `max` = `max_returned_rows` (1000); 400 on invalid | -| `_col=name` (repeatable) | return only pks + these columns; 400 on invalid | -| `_nocol=name` (repeatable) | exclude columns; 400 if invalid or a pk | -| `_labels=on` | expand every FK column into `{"value", "label"}` | -| `_label=col` (repeatable) | expand only the named FK column(s) | -| `_facet=col` | request a facet; 400 `"_facet= is not allowed"` when `allow_facet` off | -| `_facet_array=col` / `_facet_date=col` | typed facets | -| `_facet_size=N\|max` | facet bucket count, default 30, capped at `max_returned_rows` | -| `_nocount=1` | skip count (`count` extra → null) | -| `_nofacet=1` | skip facets and suggestions | -| `_nosuggest=1` | skip facet suggestions only | -| `_shape=` | see renderer section; `array`/`object` also force `_nocount` and `_nofacet` | -| `_nl=on` | NDJSON with `_shape=array` | -| `_json=col` / `_json_infinity=1` | renderer options | -| `_timelimit=ms` | custom SQL time limit | -| `_ttl=seconds` | `Cache-Control: max-age=N` (`0` → `no-cache`); default `default_cache_ttl` (5) | -| `_trace=1` | append `_trace` key (requires `trace_debug` setting) | -| `_extra=` | see above | - -**Pagination** is keyset-based for tables: `page_size + 1` rows are fetched; -`next` is built from the last row of the page — comma-joined tilde-encoded -primary-key values, prefixed by the sort value when sorted (`$null` for null -sort values) (views/table.py:2041-2111, 2421-2482). `next_url` is the -absolute URL with `_next` replaced. - -### GET /\/\.json (SQL views) - -Same code path with `is_view=True`. Differences: - -- No primary keys: `primary_keys` → `[]`; `_shape=object` fails; base query - has no `order by`. -- **Pagination is offset-based**: `_next` is an integer offset applied as - `limit N offset M` (views/table.py:2047-2049, 2438-2439) — unlike the - keyset tokens used for tables. -- `view_definition` returns the `CREATE VIEW` SQL; `table_definition` is null. - -### GET /\/\/\.json - -`RowView` (app.py:2715-2718; views/row.py:137). `` is comma-separated -tilde-encoded primary key values (rowid for rowid tables). - -- **Permission:** `view-table` (denied → `Forbidden` → 403 HTML). Missing row - → 404 `"Record not found: [...]"`. -- **Default JSON keys:** `ok`, `database`, `table`, `rows` (single-element - list), `primary_keys`, `primary_key_values`, `query_ms`, - `truncated: false`; `columns` only with `?_extra=columns`. -- **`?_extra=` (ROW scope):** `columns`, `primary_keys`, `render_cell`, - `debug`, `request`, `query`, `column_types`, `metadata`, `extras`, - `database`, `table`, `database_color`, `private`, `foreign_key_tables` - (incoming FKs with `count` and `link`; single-pk rows only). -- **Foreign-key label expansion does not apply to row JSON** — `_labels` has - no effect here; expansion happens only in the HTML path - (views/row.py:445-475). -- `_shape`, `_json`, `_nl`, `_json_infinity`, `_ttl` apply. - -### The .blob format - -`//
/.blob?_blob_column=col` (also on query pages) — -fetches raw binary bytes (blob_renderer.py:10-61). `_blob_column` required -(400 if missing/invalid); optional `_blob_hash` must equal the value's -SHA-256 (else 400 `"Link has expired..."`). Returns `application/binary` as a -download attachment. In JSON output, binary cells appear as -`{"$base64": true, "encoded": "..."}`. - -### GET /\/\/-/schema(.json|.md) - -`TableSchemaView` (app.py:2751-2754; views/special.py:1332-1378). - -- **Permission:** `view-table` via `ensure_permission` (denied → 403 HTML). -- **Responses:** `.json` → 200 `{"ok": true, "database", "table", "schema"}`; - `.md` → `text/markdown`; no extension → HTML. Missing table → 404 - `{"ok": false, "error": "Table not found"}` for `.json`. - -### GET /\/\/-/fragment - -`TableFragmentView` (app.py:2739-2742; views/table.py:1385-1418). -**HTML-only** — returns the `_table.html` partial; no JSON variant. Accepts -table querystring parameters plus `_row=` to render a single row. - -### GET /\/\/-/autocomplete - -`TableAutocompleteView` (app.py:2743-2746; views/table.py:1492-1595). Tables -only — views get 400 `"Autocomplete is only available for tables"`. - -- **Permission:** `view-table` (denied → `Forbidden` → 403). -- **Parameters:** `q` (matched with escaped `LIKE %q%` against pk columns and - the label column) and `_initial` (truthy: with empty `q`, return the 10 - most recent rows). Neither → `{"ok": true, "rows": []}`. -- **Response:** `{"rows": [{"pks": {pk_name: value}, "label": "..."}]}` — max - 10 items; 500 ms query budget with fallbacks, timing out to - `{"ok": true, "rows": []}`. - ---- - -## The write API - -All write endpoints return errors via `_error()` (the canonical error -shape) and check permissions with -`datasette.allowed()` directly, so their 403s are JSON (unlike the -`Forbidden`-raising read endpoints). Routes: app.py:2719-2762. - -### POST /\/\/-/insert - -`TableInsertView` (views/table.py:907-1194). - -- **Permissions:** `insert-row` on the table (denied → 403 - `["Permission denied"]`); `update-row` additionally required for - `replace: true` (403 `need update-row to use "replace"`); `alter-table` - additionally required for `alter: true` (403 - `Permission denied for alter-table`). Immutable database → 403 - `Database is immutable`. -- **Request** — the body is parsed as JSON regardless of the request - `Content-Type` header (invalid JSON → 400). Body: - - | Field | Rules | - |---|---| - | `row` | single object; mutually exclusive with `rows`; forces `return: true` | - | `rows` | list of objects; max `max_insert_rows` (default 100), else 400 `"Too many rows, maximum allowed is 100"` | - | `ignore` | skip rows whose pk already exists; mutually exclusive with `replace` | - | `replace` | replace rows with matching pks (needs `update-row`) | - | `alter` | add missing columns (needs `alter-table`) | - | `return` | include inserted rows in the response | - - One of `row`/`rows` required. Unknown keys → 400 `"Invalid parameter: ..."`. - Unless `alter`, row keys must be existing columns → per-row 400 - `"Row 0 has invalid columns: x, y"`. Values are validated against assigned - column types. -- **Response** — **201** `{"ok": true}`; with `return: true` also `rows` - (the rows as stored, re-fetched by rowid). SQLite errors during the write → - 400 with the message. Emits `insert-rows` (and possibly `alter-table`) - events. - -### POST /\/\/-/upsert - -`TableUpsertView` — subclasses insert (views/table.py:1197-1201). - -- **Permissions:** **both** `insert-row` and `update-row` (403 - `need both insert-row and update-row`); `alter: true` needs `alter-table`. -- **Request:** same as insert, except `ignore`/`replace` are rejected (400 - `"Upsert does not support ignore or replace"`) and **every row must contain - the table's primary key(s)** (per-row 400 - `Row 0 is missing primary key column(s): "id"` / `has null primary key`). -- **Response** — **200** (note: insert returns 201) `{"ok": true}`; with - `return: true`, `rows` re-fetched by pk. Emits `upsert-rows`. - -### POST /\/\/-/alter - -`TableAlterView` (views/table_create_alter.py:1130-1353). - -- **Permission:** `alter-table` (403 `need alter-table`); immutable → 403. -- **Request:** `{"operations": [{"op": ..., "args": {...}}, ...]}` — a - non-empty list, validated by pydantic (extra keys forbidden anywhere; - errors → 400 `location: message`): - - | `op` | `args` | - |---|---| - | `add_column` | `name` (required), `type` (`text`/`integer`/`float`/`blob`, default `text`), `not_null`, `default` xor `default_expr`; `not_null: true` requires a default | - | `rename_column` | `name`, `to` | - | `rename_table` | `to` (must not start `sqlite_`) | - | `alter_column` | `name` + at least one of `type`, `not_null`, `default`, `default_expr` | - | `drop_column` | `name` | - | `set_primary_key` | `columns` (non-empty list) | - | `reorder_columns` | `columns` (non-empty list) | - | `add_foreign_key` | `column`, `fk_table`, optional `fk_column` | - | `drop_foreign_key` | `column` | - | `set_foreign_keys` | `foreign_keys`: list of `{column, fk_table, fk_column?}` | - - `default_expr` must be one of the five `current_*` keywords. Operations are - applied in a single write transaction; any failure → 400. -- **Response** — 200: - ```json - {"ok": true, "database": "...", "table": "", - "table_url": "...", "table_api_url": "...", - "altered": true, "schema": "...", "before_schema": "...", - "operations_applied": 2} - ``` - -### POST /\/\/-/drop - -`TableDropView` (views/table.py:1320-1382). - -- **Permission:** `drop-table` (403 `Permission denied`); immutable → 403. -- **Confirmation flow:** without `{"confirm": true}` in the body, nothing is - dropped and a 200 preview is returned: - `{"ok": true, "database", "table", "row_count", - "message": "Pass \"confirm\": true to confirm"}`. With `confirm: true` → - 200 `{"ok": true}`. Emits `drop-table`. - -### POST /\/\/-/set-column-type - -`TableSetColumnTypeView` (views/table.py:1204-1317). Assigns a Datasette -*column type* (metadata stored in the internal `column_types` table) — it -does not change the SQLite schema. - -- **Permission:** `set-column-type` (403 `Permission denied`). -- **Request**: `{"column": "name", - "column_type": {"type": "url", "config": {...}?} | null}`. Unknown - keys/invalid structure → detailed 400 errors; unknown type → 400 - `"Unknown column type: x"`. Default registered types (via the - `register_column_types` hook): `url`, `email`, `json`, `textarea`. -- **Response** — 200 `{"ok": true, "database", "table", "column", - "column_type": {...} | null}`. - -### GET /\/\/-/foreign-key-suggestions - -`TableForeignKeySuggestionsView` (views/table_create_alter.py:1008-1127). -**GET only** (read-only despite living beside the write endpoints). - -- **Permission:** `alter-table` (403 `need alter-table`); views → 400 - `"Cannot suggest foreign keys for a view"`. -- **Response** — 200: `{"ok": true, "database", "table", - "row_check": {attempted, status, row_limit, sampled_rows, checked_options}, - "columns": [{column, type, affinity, current, - "suggestions": [{fk_table, fk_column, confidence, sampled_values, reasons}], - "options": [...]}]}`. Samples up to 500 rows within 50 ms/200 ms budgets. - -### POST /\/\/\/-/update - -`RowUpdateView` (views/row.py:781-870). - -- **Permissions:** `update-row` (403 `Permission denied`); `alter: true` - additionally requires `alter-table` (403 - `Permission denied for alter-table`). -- **404s:** `Database not found: x` / `Table not found: x` / - `Record not found: [pks]`. -- **Request:** `{"update": {column: value, ...}, "return"?: true, - "alter"?: true}`. Missing/non-dict `update` → 400 - `"JSON must contain an update dictionary"`; unknown keys → 400 - `"Invalid keys: ..."`; write failures (bad column, constraint violation) → - 400 with the message. -- **Response** — 200 `{"ok": true}`; with `return: true`, - `{"ok": true, "rows": [{...}]}` — a single-item list, matching - insert/upsert. Emits `update-row`. - -### POST /\/\/\/-/delete - -`RowDeleteView` (views/row.py:738-778). - -- **Permission:** `delete-row` (403 `Permission denied`). 404s as update. -- **Request:** no body required (any body is ignored — there is no - confirmation step, unlike table drop). -- **Response** — 200 `{"ok": true}`; with `?_redirect_to_table` a `redirect` - key is added. A failure during the write returns 400 with the message, - matching update. Emits `delete-row`. - ---- - -## Stored (canned) queries API - -Stored queries live in the internal database's `queries` table -(utils/internal_db.py:116-133). Queries defined in `datasette.yaml` are -synced in at startup with `source="config"` and `is_trusted` defaulting to -true; queries created via the API get `source="user"`, `is_trusted=false`, -`owner_id` = actor id. - -**Canonical stored-query JSON object** (`stored_query_to_dict`, -stored_queries.py:55-80): - -```json -{ - "database": "...", "name": "...", "sql": "...", - "title": null, "description": null, "description_html": null, - "hide_sql": false, "fragment": null, - "parameters": ["p"], - "is_write": false, "is_private": true, "is_trusted": false, - "source": "user", "owner_id": "...", - "on_success_message": null, "on_success_message_sql": null, - "on_success_redirect": null, - "on_error_message": null, "on_error_redirect": null, - "private": true -} -``` - -`private` appears only in list responses. On input (create/update and -`datasette.yaml`), `params` is accepted as an alias for `parameters`. - -**Default permission rules for queries** (default_permissions/defaults.py): -`view-query` is default-allow, but private queries are visible only to their -owner; the owner may `update-query`/`delete-query` their `source='user'` -queries. - -### GET /-/queries(.json) and GET /\/-/queries(.json) - -`GlobalQueryListView` / `QueryListView` (app.py:2606-2609, 2663-2666; -views/stored_queries.py:69-238). The global variant lists queries across all -databases (`database`/`database_color` are null, `show_database` true). - -- **Permissions:** no single gate; results filtered per query by - `view-query` (private queries appear only for their owner). -- **Parameters:** `_size` (default 20 HTML / **50 JSON**; accepts `max`; - values over `max_returned_rows` or non-integers → 400, matching table - `_size` semantics), `_next` (cursor), `q` (substring search over - name/title/description/sql), `is_write` / `is_private` (booleans; invalid → - 400 `"is_write must be 0 or 1"`), `source`, `owner_id`. -- **Response** — 200: - `{"ok": true, "database", "database_color", "queries": [...], "next", - "next_url", "limit", "show_private_note", - "show_trusted_note", "query_list_path", "show_database", - "facets": [{title, items: [{label, count, href, active}]}], - "filters": {q, is_write, is_private, source, owner_id}}`. - -### GET /\/-/queries/analyze - -`QueryCreateAnalyzeView` (app.py:2667-2670; views/stored_queries.py:290-322). -**GET only** despite being an "analyze" action — POST → 405. - -- **Permissions:** `execute-sql` then `store-query` (each denial → 403 - `errors` JSON). -- **Parameters:** only `sql` (others → 400 `"Invalid keys: ..."`). -- **Response** — 200: `{"ok", "parameters", "analysis_error", - "analysis_rows": [{operation, database, table, required_permission, - source, allowed}], "has_sql", "analysis_is_write", "save_disabled"}`. - -### POST /\/-/queries/store - -`QueryStoreView` (app.py:2671-2674; views/stored_queries.py:325-388). GET on -the same path renders the HTML create form. - -- **Permissions:** `execute-sql` + `store-query` (403 `errors` JSON). -- **Request:** JSON bodies must wrap the fields: - `{"query": {...fields...}}`; form bodies pass fields flat. Fields: - `name` (required; `^[^/\.\n]+$`; conflicts with tables/views or existing - queries → 400), `sql` (required; read SQL must pass `validate_sql_select`; - write SQL must pass per-operation permission checks), `title`, - `description`, `hide_sql`, `fragment`, `parameters`/`params` (must exactly - match the SQL's named parameters; magic parameters rejected), - `is_private` (**default true**), and — only for write SQL — - `on_success_message`, `on_success_redirect`, `on_error_message`, - `on_error_redirect`. `is_write` is derived from SQL analysis; - `is_trusted`, `description_html` and `on_success_message_sql` cannot be - set through this API. -- **Response:** JSON request → **201** `{"ok": true, "query": {...}}`; form - request → 302 redirect. - -### GET /\/\/-/definition - -`QueryDefinitionView` (app.py:2695-2698; views/stored_queries.py:391-408). - -- **Permission:** `view-query` (403 `["Permission denied"]`). -- **Response:** 200 `{"ok": true, "query": {...}}`; 404 - `["Query not found: x"]`. - -### GET/POST /\/\/-/edit - -`QueryEditView` (app.py:2699-2702) — **HTML form endpoint** -(`has_json_alternate = False`), not part of the JSON API. Programmatic -updates use `/-/update`. - -### POST /\/\/-/update - -`QueryUpdateView` (app.py:2703-2706; views/stored_queries.py:411-465). - -- **Permissions:** `update-query` (403 `need update-query`); trusted queries - → 403 `"Trusted queries cannot be updated using the API"`; changing `sql` - additionally requires `execute-sql`. -- **Request:** `{"update": {...partial fields...}, "return"?: true}` — other - top-level keys → 400. Updatable fields: `sql`, `title`, `description`, - `hide_sql`, `fragment`, `parameters`/`params`, `is_private`, `on_*` - fields (write SQL only). New SQL is re-analyzed and `is_write` recomputed. -- **Response:** 200 `{"ok": true}` (plus `query` with `return: true`); 404 - `"Query not found: x"`. - -### POST /\/\/-/delete - -`QueryDeleteView` (app.py:2707-2710; views/stored_queries.py:594-644). GET -renders an HTML confirmation page. - -- **Permission:** `delete-query` (403 `need delete-query`). Trusted - queries → 403 `"Trusted queries cannot be deleted using the API"`, - matching update. -- **Response:** JSON request → 200 `{"ok": true}`; form → 302; 404 - `"Query not found: x"`. No `confirm` field required (unlike table drop). - -### GET/POST /\/\(.json) — executing a stored query - -No dedicated route: the table route resolves the name, and on `TableNotFound` -the request is dispatched to `QueryView` when a stored query matches -(views/table.py:1698-1712). Covers both config-defined and API-stored -queries. - -**GET (read queries)** — `QueryView.get` (views/database.py:695-1130): - -- **Permissions:** `view-query` (denied → `Forbidden` → 403 HTML). Read - queries then require `execute-sql` unless `is_trusted`. Write queries are - **not executed** on GET — JSON returns empty `rows`; HTML shows a POST form. -- **Parameters:** each named `:param` is read from the query string (missing - → `""`); `_timelimit`; renderer options (`_shape`, `_nl`, `_json`, - `_json_infinity`); `_extra` (QUERY scope). -- **Response:** `{"ok": true, "rows": [...], "truncated": false}` + extras. - SQL errors → 400 with `error` in the envelope. - -**POST (write queries)** — `QueryView.post` (views/database.py:574-693): - -- **Permissions:** `view-query`; then, unless `is_trusted`: - `execute-write-sql` on the database **plus** per-operation write - permissions (same table as `/-/execute-write`). Rejection → 403 - `{"ok": false, "message": "...", "redirect": null}` for JSON clients. - Immutable database → 403. -- **Body:** form-encoded or JSON `param=value` pairs (values coerced to - strings). -- **JSON is returned when** `Accept: application/json`, `?_json=1`, or a - `_json` body field is present; otherwise 302 + flash message. -- **Magic parameters** (`:__`, resolved server-side; registered - via `register_magic_parameters`, default_magic_parameters.py): - `_now_epoch`, `_now_date_utc`, `_now_datetime_utc`, `_actor_`, - `_random_chars_`, `_cookie_`, `_header_` (underscores → - hyphens). User-stored queries cannot contain magic parameters — they are a - feature of config/trusted queries. -- **Response:** success → 200 - `{"ok": true, "message": "...", "redirect": "..."|null}` — `message` - honors `on_success_message_sql` / `on_success_message`, falling back to - `"Query executed"` or `"Query executed, N rows affected"`. SQL failure → - **400** canonical error (message honors `on_error_message`) plus a - `redirect` context key from `on_error_redirect`. Operation rejection - (`QueryWriteRejected`, e.g. VACUUM) → 403 canonical error plus - `redirect: null`. - ---- - -## Authentication and tokens - -### Bearer tokens (`dstok_`) - -Signed API tokens are sent as `Authorization: Bearer dstok_...`. The -`actor_from_signed_api_token` hook (default_permissions/tokens.py:25-40) -passes the token to `datasette.verify_token()`, which tries every handler -registered via `register_token_handler`; the default is -`SignedTokenHandler` (tokens.py:117-193). - -- **Format:** `dstok_` + itsdangerous-signed payload (namespace `token`) - containing `a` (actor id), `t` (creation Unix time), optional `d` - (duration seconds), optional `_r` (restrictions). -- **Verification:** a `dstok_`-prefixed token that fails verification — - `allow_signed_tokens` off, invalid signature, missing/non-integer `t`, - malformed `d`, or expired — raises `TokenInvalid`, and the request fails - with **401**, the canonical error body and a - `WWW-Authenticate: Bearer error="invalid_token"` header (even if a valid - `ds_actor` cookie is also present). Tokens with prefixes no registered - handler recognizes are ignored (they may belong to an auth plugin). The - effective duration is `d` capped by `max_signed_tokens_ttl` (default 0 = - no cap; a non-zero setting also imposes a TTL on tokens without `d`). -- **Resulting actor:** `{"id": , "token": "dstok"}` plus `"_r"` and - `"token_expires"` when applicable. - -**Restrictions (`_r`)** (default_permissions/restrictions.py): - -- `"a"`: list of actions allowed on any resource -- `"d"`: `{database_name: [actions]}` -- `"r"`: `{database_name: {table_name: [actions]}}` - -Actions are stored as abbreviations when available (see appendix); checks -accept either the full name or the abbreviation. Restrictions are an -allowlist filter layered on top of normal permission resolution — a -restricted token can never do more than its allowlist, and never more than -the underlying actor could do anyway. - -### Token creation - -- **`/-/create-token`** is an HTML form endpoint only (see the instance - section) — there is no JSON API to mint tokens in this codebase. -- Programmatic alternatives: the `datasette create-token` CLI command and - the `datasette.create_token()` Python API. -- `/-/auth-token` is the one-time `--root` login mechanism, unrelated to API - tokens. - -### Cookie authentication - -Browser sessions use the signed `ds_actor` cookie (set by `/-/auth-token`, -plugins, or login flows; cleared by `/-/logout`). API POSTs from browsers are -subject to the cross-origin checks described in -[CSRF](#csrf--cross-origin-protection). - ---- - -## Appendix: registered actions (permissions) - -From `datasette/default_actions.py` (registered via the `register_actions` -hook). Token restrictions store the abbreviation when available. - -| Action | Abbr | Resource level | Notes | -|---|---|---|---| -| `view-instance` | `vi` | global | | -| `permissions-debug` | `pd` | global | gates the debug endpoints | -| `debug-menu` | `dm` | global | UI only | -| `view-database` | `vd` | database | | -| `view-database-download` | `vdd` | database | `also_requires="view-database"` | -| `execute-sql` | `es` | database | `also_requires="view-database"`; denied when the `default_allow_sql` setting is off | -| `execute-write-sql` | `ews` | database | `also_requires="view-database"` | -| `create-table` | `ct` | database | | -| `store-query` | `sq` | database | `also_requires="execute-sql"` | -| `view-table` | `vt` | table | | -| `insert-row` | `ir` | table | | -| `delete-row` | `dr` | table | | -| `update-row` | `ur` | table | | -| `alter-table` | `at` | table | | -| `set-column-type` | `sct` | table | | -| `drop-table` | `dt` | table | | -| `view-query` | `vq` | query | default-allow; private queries restricted to their owner | -| `update-query` | `uq` | query | query owner allowed by default (source=`user` only) | -| `delete-query` | `dq` | query | query owner allowed by default (source=`user` only) | diff --git a/stable-api-recommendations.md b/stable-api-recommendations.md deleted file mode 100644 index 0d817612..00000000 --- a/stable-api-recommendations.md +++ /dev/null @@ -1,469 +0,0 @@ -# Datasette 1.0 Stable API — Consistency and Completeness Review - -This review is based on `existing-api.md`, which documents the JSON API as -actually implemented in this codebase (`1.0a35`), derived from source. The -goal here is to identify everything that should be made consistent, fixed, or -explicitly scoped out **before** the 1.0 stability promise takes effect — -because after 1.0, every inconsistency below becomes a compatibility -commitment. - -Findings are grouped by theme. Each carries a priority: - -- **P1 — should block 1.0**: breaking to fix later, or a correctness/security - concern. -- **P2 — strongly recommended**: fixable later only via awkward additive - changes. -- **P3 — nice to have / documentation decision**: can be resolved by - documenting the behavior as intentional. - ---- - -## 1. Error responses: four shapes is three too many (P1) — ✅ IMPLEMENTED - -> **Status:** implemented. All four shapes now delegate to a shared -> `error_body()` helper (`datasette/utils/__init__.py`) producing -> `{"ok": false, "error": "", "errors": [...], "status": }`. -> The `title` key is no longer emitted in JSON; the bare `{"error": ...}` -> debug-endpoint shape is gone; `_shape=object` misuse now returns HTTP 400 -> (part of §1b). Covered by `tests/test_error_shape.py` and documented in -> the "Error responses" section of `docs/json_api.rst`. §1a (`Forbidden` → -> JSON) and §1b (write canned-query 200) are now also implemented. Still -> open from this section's sub-items: the §1c status outliers. - -The API currently produces four distinct JSON error shapes depending on which -internal layer generates the error: - -| Shape | Producer | Example endpoints | -|---|---|---| -| `{"ok": false, "error", "status", "title"}` | exception handler (handle_exception.py:50-53) | 404s and `DatasetteError`s on any `.json` path | -| `{"ok": false, "errors": [...]}` | `_error()` helper (views/base.py:183-184) | all write endpoints, stored-query endpoints, execute-write | -| `{"ok": false, "error", "rows": [], "truncated": false}` | JSON renderer (renderer.py:52-56) | SQL errors on table/query reads | -| `{"error": "..."}` (no `ok`) | permission debug views (views/special.py) | `/-/allowed`, `/-/rules`, `/-/check`, POST `/-/permissions` | - -Additionally, write canned queries report failure via a **fifth** vocabulary: -`{"ok": false, "message": ..., "redirect": ...}` with HTTP **200** -(views/database.py:678-690). - -A 1.0 client cannot write a single error handler today. **Recommendation:** -pick one canonical error object — the singular/plural tension is easiest to -resolve as: - -```json -{"ok": false, "error": "human-readable summary", "errors": ["detail", "..."], "status": 400} -``` - -where `errors` is optional and `error` is always present — and route every -error path through it (including the `forbidden` and `handle_exception` -defaults). At minimum, eliminate the bare `{"error": ...}` shape and the -`status`/`title` keys nobody else emits (`title` is a template-rendering -concern that leaked into the API). - -### 1a. `Forbidden` returns an HTML 403 to JSON clients (P1) — ✅ IMPLEMENTED - -> **Status:** implemented — the default `forbidden()` hook now returns the -> canonical JSON error for requests whose path ends in `.json` or that send -> `Accept: application/json` / `Content-Type: application/json`. - -Read endpoints that deny access via `ensure_permission`/`check_visibility` -raise `Forbidden`, and the default `forbidden()` hook renders an **HTML error -page even for `.json` requests** (forbidden.py:4-19, app.py:2895-2904). So: - -- `GET /db/table.json` without `view-table` → 403 **HTML** -- `POST /db/table/-/insert` without `insert-row` → 403 **JSON** - -A JSON client gets unparseable output precisely when it most needs a -machine-readable answer. **Recommendation:** the default forbidden handler -must return the canonical JSON error when the path ends in `.json` or the -request prefers JSON, mirroring `handle_exception`. - -### 1b. Errors that return HTTP 200 (P1) — ✅ IMPLEMENTED - -> **Status:** implemented. `_shape=object` misuse returns 400 (done with -> §1), and write canned-query SQL failures now return **400** with the -> canonical error shape (plus the `redirect` context key); the -> `QueryWriteRejected` 403 branch also uses the canonical shape. - -- `_shape=object` on a query or pk-less table → `{"ok": false, "error": - "_shape=object is only available on tables"}` with **200** - (renderer.py:73-90), while an unknown `_shape` value returns **400** - (renderer.py:101-108). Same class of error, different status. -- Write canned-query SQL failure → **200** `{"ok": false, "message": ...}` - (views/database.py:683-690), while the equivalent failure on - `/-/execute-write` returns **400**. - -**Recommendation:** all `ok: false` responses should carry a 4xx/5xx status. -(`/-/execute-write/analyze` returning `ok: false` with 200 for "analysis -completed, SQL is invalid" is defensible but should then not reuse the `ok` -key — see §2.) - -### 1c. Wrong-status outliers (P2) — ✅ IMPLEMENTED - -- ~~Row **delete** write failures return **500** (views/row.py:757) while row - **update** write failures return **400** (views/row.py:832-835). Same - failure class, different status; pick 400 (or 409 for constraint - violations) for both.~~ ✅ **Done** — delete now returns 400, matching - update and the rest of the write API. -- ~~Invalid or expired bearer tokens silently degrade the request to anonymous, - so clients see a 403 permission error (or worse, anonymous-permitted data) - rather than a 401 (tokens.py:147-193). For 1.0, a malformed/expired - `Authorization: Bearer dstok_...` header should produce **401** with a - distinguishable error, so clients can tell "renew your token" apart from - "you lack permission".~~ ✅ **Done** — token handlers can raise - `TokenInvalid`; Datasette responds 401 with the canonical body and a - `WWW-Authenticate: Bearer error="invalid_token"` header. Unrecognized - token prefixes still fall through to anonymous so auth plugins keep - working. - ---- - -## 2. Success envelope: `ok` is not universal, arrays are not extensible (P1/P2) — ✅ IMPLEMENTED (§2a-2c open) - -> **Status:** recommendations 1-3 are implemented. Every JSON-object -> success response now includes `"ok": true` (`JsonDataView` injects it for -> dict responses; homepage, jump, schema, permission-debug and autocomplete -> views set it explicitly), and the three top-level-array endpoints now -> return objects: `/-/plugins` → `{"ok": true, "plugins": [...]}`, -> `/-/databases` → `{"ok": true, "databases": [...]}`, `/-/actions` → -> `{"ok": true, "actions": [...]}`. Covered by -> `tests/test_success_envelope.py`. The sub-findings §2a (collection -> representations), §2b (`_extra`/`_shape` coverage) and §2c (count -> truncation) remain open. - -Endpoints disagree about the success envelope: - -- **Have `ok: true`:** table/row/query reads, database view, all write - endpoints, stored-query endpoints, `/-/allowed`-style debug data. -- **No `ok` key:** `/-/versions`, `/-/settings`, `/-/config`, `/-/threads`, - `/-/actor`, `/-/jump`, `/-/schema` variants (`{"database", "schema"}`, - `{"schemas": [...]}`), table `/-/schema.json`, `/-/autocomplete` - (`{"rows": []}`), homepage `/.json`. -- **Top-level JSON arrays:** `/-/plugins`, `/-/databases`, `/-/actions` - (app.py:2247-2304). A top-level array can never grow a sibling key - (pagination, warnings, `ok`) without a breaking change. - -**Recommendations:** - -1. (P1) Wrap the three array endpoints in objects before 1.0: - `{"ok": true, "plugins": [...]}` etc. This is the single cheapest - future-proofing fix in this list. -2. (P2) Add `ok: true` to every JSON-object success response, or explicitly - document that `ok` only exists on data endpoints. Half-consistency is the - worst outcome. -3. (P2) `/db/-/schema.json` (`{"database", "schema"}`) and - `/db/table/-/schema.json` should match the envelope style of their sibling - endpoints (they are also the only data endpoints whose 404 uses the - exception shape but whose success has no `ok`). - -### 2a. Collection representations disagree (P2) - -- ~~Homepage `/.json` returns `databases` as an **object keyed by name** - (index.py:147-161); `/-/databases.json` returns an **array**; the database - page returns `tables` as an array. Choose arrays-of-objects everywhere - (objects-keyed-by-name break when names need ordering or pagination).~~ - ✅ **Done** — the homepage returns a list, matching `/-/databases.json`. - The homepage JSON remains deliberately undocumented. -- ~~Insert/upsert with `return: true` respond with `rows` (plural, list); row - update with `return: true` responds with `row` (singular, object) - (views/row.py:837-844). Pick one (`rows` everywhere, even for one row, - matches the read API).~~ ✅ **Done** — row update now returns - `rows: [{...}]`. - -### 2b. `_extra`/`_shape` support is uneven (P2) — partially implemented - -> **Status:** unknown `_extra` names on data formats now return 400 -> `Unknown _extra: ` (HTML pages still ignore them). Extending -> extras/shaping to database/instance scope remains open. - -The extras system (`?_extra=`, scope-registered) is the 1.0 mechanism for -response shaping — but it only exists on table, row and query endpoints. The -database view builds JSON by hand and supports **neither `_extra` nor -`_shape`** (views/database.py:189-212); the homepage likewise. Either extend -extras to database/instance scope before 1.0 or document clearly that shaping -is a table/row/query feature. Also decide the contract for **unknown -`_extra` names, which are currently silently ignored** (extras.py:116-122) — -silent ignoring means typos return the default payload with no signal; -recommend a 400 or a `warnings` key. - -### 2c. Count truncation is invisible in JSON (P2) — ✅ IMPLEMENTED - -> **Status:** implemented — a public `count_truncated` extra now exists and -> is implicitly included whenever `count` is requested. - -The `count` extra is computed with a `limit 10001` subquery, so `count: -10001` actually means "at least 10001" — the `count_truncated` flag exists -but only in the HTML template context, never in JSON (views/table.py: -2334-2337). Expose it (e.g. make `count` be `null` + add `count_estimate`, -or add `count_truncated` to the JSON) before clients start trusting the -number. - ---- - -## 3. Pagination: three mechanisms, two contracts (P2) — partially implemented - -> **Status:** `next_url` now accompanies `next` in the default table JSON -> keys (previously it required `?_extra=next_url`), so every response with -> a `next` token also carries the ready-to-follow URL. Pagination tokens -> are deliberately left undocumented as to their internal structure. -> `_size` is now the single page-size parameter with uniform table-style -> semantics everywhere: query lists accept `max` and 400 on out-of-range -> values (previously silently clamped), and the `/-/allowed` and -> `/-/rules` debug endpoints renamed `page`/`page_size` to -> `_page`/`_size` with the same validation (400 instead of silent -> capping at 200). `has_more` has been **removed** from the query-list -> JSON — `next: null` is the single end-of-results signal everywhere, -> keeping default response keys minimal (`total` remains a debug-endpoint -> nicety). Fixing this also uncovered and fixed a bug where the query -> list's JSON `next_url` pointed at the HTML page (it dropped the `.json` -> extension) and was relative where the table `next_url` is absolute. -> §3 is now fully resolved. - -| Endpoint | Mechanism | Token | Extras | -|---|---|---|---| -| Table `.json` | keyset | tilde-encoded pk/sort values in `_next` | `next` always in body, `next_url` via `_extra`, `Link: rel=next` header | -| SQL view `.json` | **offset** | integer in the same `_next` parameter | same envelope | -| `/-/queries` lists | keyset | cursor in `_next` | `next`, `next_url`, **`has_more`** in body | -| `/-/allowed`, `/-/rules` | **page numbers** | `page`/`page_size` | `total`, `next_url`, `previous_url` | - -Concerns: - -1. The same `_next` parameter means "start after key" on tables but "row - offset" on views. Offset pagination over views is also O(n) and skews - under concurrent writes. If unifiable, unify; if not, document loudly. -2. `has_more` exists on query lists but not table pages; `total` exists on - debug endpoints but not elsewhere. Standardize the pagination block - (suggest: `next`, `next_url` — nullable — everywhere; treat `has_more` as - `next != null`). -3. Page-size parameters: `_size` (default 100, `max` keyword allowed) on - tables; `_size` (default 50 JSON, clamped 1–1000, no `max` keyword) on - query lists; `page_size` (default 50, silently capped at 200) on debug - endpoints. Align names, defaults and the cap behavior (silent capping vs - 400) as far as practical. - ---- - -## 4. HTTP semantics (P2) - -- ~~**201 vs 200:** insert → 201, upsert → 200 (views/table.py:1194), create - table → 201, store query → 201. Insert-201/upsert-200 is defensible - (upsert may not create) but it is undocumented subtlety; state it, or - return 200 for both with an explicit `created` count.~~ ✅ **Done** — - documented as deliberate in the upsert docs. -- **Destructive-action confirmation is asymmetric:** table drop requires - `{"confirm": true}` and has a preview response (views/table.py:1346-1365); - row delete executes immediately and ignores the body; query delete - executes immediately. Decide the 1.0 rule (suggestion: confirmation only - for schema-destroying operations, i.e. keep as is — but document it as a - deliberate contract). -- ~~**Content-type enforcement is inconsistent:** `/-/insert`, `/-/upsert`, - `/-/alter`, `/-/set-column-type` demand `Content-Type: application/json` - (400 otherwise); `/-/create` parses the body as JSON regardless of - content type; execute-write and the query CRUD endpoints accept both JSON - and form encodings. Pick one rule for JSON-only endpoints.~~ ✅ **Done** - — the lenient rule won: JSON-only write endpoints parse the body as JSON - regardless of `Content-Type` (CSRF protection comes from the - cross-origin header checks, not content types). This also fixed a 500 on - insert when the header was absent entirely. -- **JSON-vs-HTML negotiation on POST differs per endpoint:** execute-write - and canned queries key off `Accept: application/json` / a `_json` body - field; the write API keys off nothing (always JSON); query store keys off - request content type. A single documented rule ("responses are JSON if the - request body was JSON or `Accept: application/json`") would cover all of - them. -- **Endpoints named like actions but served over GET:** - `/-/queries/analyze`, `/-/execute-write/analyze`, - `/-/foreign-key-suggestions`, `/-/query/parameters` are all GET (correct, - they are reads) — fine, but `analyze` under a POST-shaped path invites - wrong calls; make sure 405 responses for POST on these return the JSON 405 - shape (they do only when the path ends `.json` or content type is JSON — - a JSON POST to `/-/queries/analyze` gets JSON, a form POST gets text). - ---- - -## 5. Naming and parameter conventions (P2/P3) - -- ~~**`params` and `parameters` are duplicate keys** in every stored-query - object (stored_queries.py:55-80). Delete one before 1.0 (suggest keeping - `parameters`; the write side already accepts both on input).~~ - ✅ **Done** — output objects carry only `parameters` (matching - `/-/query/parameters` and the analyze endpoints); `params` remains an - accepted input alias for API creation and `datasette.yaml` config. -- **Three names for the same concept across error/message payloads:** - `error`, `errors`, `message`. See §1. -- ~~**Boolean query parameters have at least three grammars:** `_nl=on`, - `_labels=on/off`, `?all=1`, `is_write=1|0|true|false|t|f|yes|no|on|off`, - `_nocount=1`. Adopt one accepted set (the query-list parser at - query_helpers.py:81-94 is a good candidate) and apply it everywhere.~~ - ✅ **Documented** — the JSON API docs state the canonical grammar - (`on/true/1`, `off/false/0`), which `value_as_boolean` already accepts - everywhere it is used. -- ~~**`.jsono`** survives on the homepage route (identical output to `.json`) - and as a row-view redirect. Remove it at 1.0; it is pure legacy.~~ - ✅ Removed: the homepage routes only accept `.json` and the row-view - redirect is gone. -- **`_json` is overloaded:** on GET it is a renderer option naming a column - to parse as JSON (repeatable); on canned-query POST a `_json` body field - forces a JSON response. Two unrelated meanings for one name. -- The reserved `/-/` namespace is applied consistently across routes — this - is in good shape. The one gap: table names matching `^-$`-adjacent shapes - are protected by tilde-encoding; keep a test asserting `/-/` can never be - shadowed by user data. - ---- - -## 6. Permissions and security consistency (P1/P2) - -- ~~**(P1) `/-/databases.json` ignores per-database permissions** — it lists - every attached database (name, path on disk, size) to any actor holding - `view-instance` (app.py:2157-2169), while the homepage and every other - endpoint filter by `view-database`. On a public instance with private - databases this leaks filesystem paths and database names. Filter it, or - gate it behind `permissions-debug`.~~ ✅ **Done** — the endpoint now - filters through `allowed_resources("view-database", actor)`. -- ~~**(P2) `/db/-/schema` checks existence before permission** - (views/special.py:1308-1317): an actor without `view-database` can - distinguish "database exists" (403) from "does not exist" (404). - Standardize on permission-check-first (as the table view does) so - unauthorized actors get a uniform response.~~ ✅ **Done** — permission is - checked first; the table schema view also now 404s (instead of a 500 - KeyError) for an unknown database. -- ~~**(P2) `/-/threads` exposes runtime internals** (thread idents, asyncio - task reprs including file paths) behind only `view-instance`. Consider - `permissions-debug`, alongside `/-/actions` which already requires it.~~ - ✅ **Done** — `/-/threads` now requires `permissions-debug`. -- ~~**(P3) `/-/config` redaction is substring-based** on six key names - (app.py:2502-2505); plugins storing secrets under other names leak. Worth - a note in plugin authoring docs plus a `redact_keys` plugin hook.~~ - ✅ **Documented** — the plugin secrets docs now advise naming keys to - match the redaction substrings (a `redact_keys` hook remains a possible - future addition). -- **(P3) Database-level checks on `/-/create`** (insert-row/update-row - checked against `DatabaseResource`, not the about-to-exist table — - table_create_alter.py:819-856) vs table-level checks on `/-/insert`. - Correct by necessity, but document that a token restricted to - table-level `ir` cannot use `/-/create` with rows. - ---- - -## 7. Completeness gaps for a 1.0 JSON API (P2/P3) - -1. **(P2) No JSON API to create tokens.** `/-/create-token` is an HTML form - only (`has_json_alternate = False`, form-encoded POST). Any automation - that wants to mint scoped tokens must shell out to `datasette - create-token`. An intentional JSON mode (actor-authenticated, same - restriction vocabulary) rounds out the write API story — or explicitly - document token minting as CLI/Python-only. -2. **(P2) Row JSON cannot expand foreign-key labels.** `_labels` works on - table JSON but is silently ignored on row JSON (views/row.py:445-475 - expands only for HTML). Either support it or return 400 for unsupported - parameters; silent ignoring is the worst option (see also §2b on unknown - `_extra` values). -3. **(P2) No machine-readable "which write features does this instance/table - support" endpoint.** Clients must probe (`/-/insert` on an immutable - database → 403). The API explorer computes exactly this data for HTML - (views/special.py:863-990); exposing it as JSON would let clients degrade - gracefully. (`/-/allowed.json` covers the permission half already.) -4. **(P3) Table list pagination.** `/db.json` inlines all tables (with - counts) and the homepage truncates to 5 per database; a 10,000-table - database has no paginated table listing. Acceptable for 1.0 if - documented; the internal catalog tables would support a real endpoint - later. -5. **(P3) `Link: rel=next` header** exists on table JSON only. Harmless, but - either add it to the other paginated endpoints or drop it from the - contract (`Access-Control-Expose-Headers: Link` suggests it is meant to - be part of the API). - ---- - -## 8. Behavior that looks like a bug and should be resolved before freezing - -1. ~~**Trusted queries: update is blocked, delete is not.** - `QueryUpdateView` rejects `is_trusted` queries with 403 - (stored_queries.py:426-427) but `QueryDeleteView.post` never checks - `is_trusted` — an actor with `delete-query` can delete a config-defined - trusted query via the API (it will resync on restart, making the - behavior confusing rather than catastrophic). Align delete with update.~~ - ✅ **Done** — both the POST endpoint and the HTML confirmation page now - return 403 `"Trusted queries cannot be deleted using the API"`; - `datasette.remove_query()` remains available for internal use. -2. ~~**GET `/db/-/query` with no `?sql=` returns 200 `{"ok": true, "rows": - []}`** while `.csv` on the same request returns 400 `"?sql= is - required"`. The JSON behavior masks caller bugs; return 400 on both.~~ - ✅ **Done** — all data formats now return 400; the HTML SQL editor page - is unchanged. -3. ~~**`_shape=object` HTTP 200 error** (§1b) — almost certainly unintended.~~ - ✅ Done — now 400 (fixed with §1b). -4. ~~**Row delete 500** (§1c) — inconsistent with every sibling endpoint.~~ - ✅ Done — now 400. -5. ~~**The "SQL Interrupted" error embeds an HTML fragment in the JSON `error` - value** (views/database.py:805-820). Error strings in the JSON API should - be plain text.~~ ✅ **Done** — `DatasetteError` gained a `plain_message` - used for JSON responses; the HTML error page keeps the rich version with - the SQL textarea. §8 is now fully resolved. - ---- - -## 9. Define stability tiers explicitly (P1 — documentation, not code) — ✅ IMPLEMENTED - -> **Status:** implemented. Undocumented JSON endpoints self-describe with -> an `"unstable"` marker key, and `docs/json_api.rst` now opens with an -> "API stability" section (`json_api_stability`) declaring the 1.x -> promise: documented endpoints/keys are stable with additive-only -> changes, pagination tokens are opaque, the error format and token -> restriction semantics are stable, and the exempt tiers (marker-key -> endpoints, debug/support endpoints, explicitly-unstable keys) are -> listed. Cross-referenced from the introspection and permission-debug -> docs. - -Not everything under `/-/` can or should carry a 1.0 guarantee. Recommend -shipping 1.0 with an explicit three-tier contract, per endpoint: - -- **Stable (semver-protected):** table/row/query reads (`.json`, `_shape`, - `_extra` public names, filters, pagination tokens as opaque strings), the - write API (`/-/insert`, `/-/upsert`, `/-/alter`, `/-/drop`, - `/-/set-column-type`, row `/-/update`, `/-/delete`, `/-/create`, - `/-/execute-write`), stored-query CRUD + execution, `/-/versions`, - `/-/plugins`, `/-/settings`, `/-/actor`, `/-/databases`, schema endpoints, - token format & restriction semantics (`_r` abbreviations are wire format - now — they are stored inside issued tokens and cannot change silently). -- **Unstable/debug (documented as exempt):** `/-/threads`, `/-/actions`, - `/-/permissions`, `/-/allowed`, `/-/rules`, `/-/check`, `/-/messages`, - `/-/allow-debug`, `/-/patterns`, `/-/debug/autocomplete`, the `debug` and - `request` extras (the `debug` extra already self-describes as unstable), - `/-/api` and `/-/jump` (UI support endpoints), `/-/autocomplete` and - `/-/fragment` (UI support), `/-/foreign-key-suggestions` and - `/-/foreign-key-targets` (heuristic outputs). -- **Internal:** anything HTML-only (`/-/edit`, `/-/create-token`, - `/-/logout`, `/-/auth-token`). - -Two details make tiering urgent rather than optional: - -- **Extras are enumerable by clients** (`?_extra=extras` self-describes the - registry), so every public extra name is de-facto API. Mark each extra - stable or unstable in its class definition and surface that in the - `extras` output. -- **Pagination tokens leak implementation** (tilde-encoded pk values for - tables, plain integers for views). Declare them opaque now so the view - token can become keyset later without a "breaking" change. - ---- - -## 10. Summary of P1 items (the pre-1.0 checklist) - -1. ~~One canonical JSON error shape; retire the other three (§1).~~ ✅ Done. -2. ~~`Forbidden` → JSON 403 for JSON requests (§1a).~~ ✅ Done. -3. ~~No `ok: false` with HTTP 200 (§1b: `_shape=object`, write canned-query - SQL errors).~~ ✅ Done. -4. ~~Wrap `/-/plugins`, `/-/databases`, `/-/actions` top-level arrays in - objects (§2).~~ ✅ Done. -5. ~~Filter `/-/databases.json` by `view-database` or gate it behind - `permissions-debug` (§6).~~ ✅ Done. -6. ~~401 (not silent-anonymous) for invalid/expired bearer tokens (§1c).~~ - ✅ Done. -7. ~~Publish explicit stability tiers, including extras and pagination-token - opacity (§9).~~ ✅ Done. -8. Resolve the looks-like-a-bug list (§8), especially ~~trusted-query delete - and row-delete 500~~ (both done). - -Everything in P2 is worth doing now because each item is breaking-to-fix -later; each P3 can be resolved by a sentence of documentation declaring the -current behavior intentional.