Commit graph

3,197 commits

Author SHA1 Message Date
Simon Willison
3f7d389caf Refine column field plugin API and documentation
- Simplify JavaScript column field context:
  - expose `isPk` instead of `isPrimaryKey`
  - expose `defaultExpression` instead of separate SQLite default flags
  - remove value/default state from plugin context
- Update field helper behavior:
  - `setValue()` no longer dispatches input/change events
  - remove dispatch options and `resetValue()`
  - add `markClean()` for plugin-normalized initial values
  - track clean field state for reliable dirty detection

Also:

- Prompt before closing row insert/edit dialogs when there are unsaved changes
- Map declared SQLite types to affinities, returning `BLOB` for typeless columns and `NUMERIC` for numeric/date/boolean-like declarations
2026-06-14 15:09:24 -07:00
Simon Willison
c083e44561 Ignore .playwright-mcp
Codex Desktop likes using this folder.
2026-06-14 14:21:04 -07:00
Simon Willison
841a2536ea Tighten row edit field plugin API
Replace the value/valueType/originalValue/originalValueType fields on makeColumnField() contexts with an explicit field object API for reading, writing, resetting, comparing and validating field values.

Normalize columnType to {type, config}, rename the SQLite default metadata so it is clearly SQLite-specific, and document that plugins submit only string, number, boolean or null values. Plugins that need structured data should serialize it themselves instead of relying on Datasette to special-case JSON.

Move the built-in json column type behavior onto the same plugin API used by external plugins: validate the textarea with field.setValidity() as the value changes, but submit plain text. Harden row edit value comparison so fixing invalid JSON in an existing row is not blocked by the original invalid value.

Update the JavaScript plugin documentation and Node-based tests for the revised field contract.
2026-06-14 13:58:55 -07:00
Simon Willison
b2de8b5d2e First draft of makeColumnField() plugin hook 2026-06-14 11:57:13 -07:00
Simon Willison
c405b56223 Tweak modal color scheme to be more Datasette 2026-06-14 11:56:51 -07:00
Simon Willison
35d7e3cab8 Fixed some tests for the new autocomplete work 2026-06-14 08:03:28 -07:00
Simon Willison
574290fb23 Add foreign key autocomplete to row forms
Expose single-primary-key foreign key autocomplete URLs in table page metadata and load the autocomplete component when needed.

Enhance insert and edit dialogs to wrap foreign-key inputs with the autocomplete web component, show linked selected-row labels, reserve metadata space, and keep the dropdown as a fixed overlay above modal chrome.

Add an explicit _initial=1 autocomplete mode for empty-field starter suggestions while keeping blank q responses empty by default, with tests for the endpoint and table metadata.
2026-06-14 07:30:34 -07:00
Simon Willison
aa5fb7be3d Autocomplete widget and /-/debug/autocomplete test page 2026-06-13 22:59:37 -07:00
Simon Willison
b868f7d4c3 /db/table/-/autocomplete?q= JSON endpoint
Needed to help implement edit foreign key reference.
2026-06-13 22:40:03 -07:00
Simon Willison
5490c7b794 textarea column type
Shows as multiline edit in edit/insert dialog
2026-06-13 22:18:45 -07:00
Simon Willison
2b61c916d0 Handle hidden label columns in row actions 2026-06-13 22:01:26 -07:00
Simon Willison
e1e67e912a Docs for /<database>/<table>/-/fragment 2026-06-13 21:58:16 -07:00
Simon Willison
a7cd746613 Use label column in 'Inserted X' messages 2026-06-13 21:48:28 -07:00
Simon Willison
e91c646ee6 Use column label, if available in edit/delete dialog 2026-06-13 21:43:48 -07:00
Simon Willison
5bf4cf8860 Add insert row UI to table pages
Add a permission-gated Insert row button to mutable table pages and expose the metadata needed by the client-side UI, including the insert API path, table name, primary keys, editable columns, defaults, nullability, and column type information.

Reuse the existing row edit modal for inserts. Insert submissions now use the JSON API with return=true, derive the new row's tilde-encoded row path from the returned primary key values, fetch the matching table fragment, and insert the rendered row into the current table. Successful inserts and updates now show mutation status messages above the table.

Support SQLite defaults in insert forms by showing default expressions as non-editable values with Set value / Use default controls. Keep those controls aligned and stable so toggling between default and custom values does not shift the modal layout.

Refine the edit modal at the same time: send only changed fields on update, skip the update API entirely when nothing changed, clear stale mutation status for no-op saves, and simplify modal headings so insert/edit context is shown in the bold title instead of duplicated summary text.

Add tests for the insert button and metadata, including omitted integer primary keys, default values, table names, and compound primary keys.
2026-06-13 21:34:45 -07:00
Simon Willison
e50d176722 Add in-place table row edit and delete UI
Use a compact data-row attribute on table row fragments and derive row API URLs in JavaScript from a page-level table URL. Add a /-/fragment endpoint so edited rows can be re-rendered with the active table template and render_cell hooks, then replaced in place after a successful save.

Document the custom _table.html data-row contract and cover the fragment endpoint, base_url handling, and row markup with tests.
2026-06-13 18:41:00 -07:00
Simon Willison
ad3456dc4a Display of edit modal (no save yet) 2026-06-13 14:48:44 -07:00
Simon Willison
20824bd707 Delete icon on table page now works 2026-06-13 14:40:29 -07:00
Simon Willison
de5f72dd88 hash busting thing on table.js
Addd for Codex Desktop, which has real trouble with edits
to files like this as the in-app browser does not seem to
have a cache-busting reload option.
2026-06-13 14:38:49 -07:00
Simon Willison
e3a1f19057 Edit/delete icon prototype 2026-06-13 14:19:39 -07:00
Simon Willison
f2927a1647 Fix for gen.throw(*sys.exc_info()) warning
Closes #2776
2026-06-13 11:15:47 -07:00
Simon Willison
d473dc565f
datasette.allowed_many() method and per-request permission check cache
Merge pull request #2775
2026-06-13 11:13:08 -07:00
Simon Willison
ab19b0382b Removed note from permission_resources_sql
Refs https://github.com/simonw/datasette/pull/2775/changes#r3408385197
2026-06-13 11:12:31 -07:00
Simon Willison
86334d233d Switch to CTE to handle 600+ actions at once
GPT-5.5 xhigh in Codex spotted this problem and fixed it with a CTE:

https://gisthost.github.io/?46076499ee685acddc988ff6b47a74b0
2026-06-13 11:09:28 -07:00
Simon Willison
d4cb8b464b Fix for trace_child_tasks exception handling
I had Claude Fable 5 review our use of contextvar and
it spotted this place where exceptions were not
correctly handled.
2026-06-12 13:21:58 -07:00
Simon Willison
bb59c61c9f Request-scoped permission check cache
Adds a per-request cache for permission check results, plus wiring that
resolves action permissions in bulk before plugin hooks need them:

- New _permission_check_cache contextvar, set to a fresh dict for each
  request by DatasetteRouter and reset when the request ends. Keys
  include the full serialized actor, so actors differing in any field
  (e.g. token restrictions) never share entries. SkipPermissions mode
  bypasses the cache entirely.
- datasette.allowed_many() now consults the cache and stores its
  results there, so repeated datasette.allowed() checks within one
  request resolve without further SQL.
- Table pages resolve all registered table-level actions against the
  current table and all database-level actions against its database
  (database pages likewise) in batched queries before invoking the
  table_actions/database_actions plugin hooks - allowed() calls made
  inside those hooks are then served from the cache with no plugin
  changes required. Actions with no permission rules from any plugin
  are resolved to False without touching the database.

Benchmarks (benchmarks/) with a simulated 12-plugin ecosystem making
18 checks per table page show 34 -> 13 internal-DB queries per page;
with 2ms-per-query internal DB latency (modelling Datasette Cloud)
table page time drops from 77.9ms to 27.6ms - the caching layer
accounts for ~91% of that improvement over allowed_many() alone.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 13:11:17 -07:00
Simon Willison
88878b4184 datasette.allowed_many() method 2026-06-12 12:51:40 -07:00
Simon Willison
fa86ac7b11
Clearer examples and descriptions for JSON API extras (#2773)
Review of the generated ?_extra= documentation found several extras
with no example output or with examples that needed explanation:

- extras: now shows an abbreviated example of the toggle list and has
  a clearer description (which also improves the live API output)
- set_column_type_ui: example of the shape seen with set-column-type
  permission, plus a note that it is null otherwise
- column_types: live example generated from a table with an assigned
  column type instead of an empty {}
- metadata: live table example now demonstrates a table description
  and column descriptions; row and query examples gained explanatory
  notes
- expandable_columns, foreign_key_tables, facets_timed_out, next_url,
  renderers: notes explaining the shape of their output

Also added docs_note cross-references to the relevant documentation:
facets, pagination, render_cell and register_output_renderer plugin
hooks, column type configuration and API, metadata, custom templates,
permissions and foreign key label expansion. foreign_key_tables is
now flagged as potentially executing additional queries.

https://claude.ai/code/session_01EfjBe6E817m9XNFW7EX3Vm

Co-authored-by: Claude <noreply@anthropic.com>
2026-06-11 19:41:24 -07:00
Simon Willison
1d4212122e Add release date for 1.0a33 2026-06-11 10:36:16 -07:00
Simon Willison
993169ae49 Release 1.0a33 1.0a33
Refs #2735, #2677, #2680, #2711, #2756, #2761, #2768, #2754
2026-06-11 08:24:37 -07:00
Simon Willison
4e9556cc24
Redesign and document extras mechanism to cover rows and queries in addition to tables
Merge PR #2769
2026-06-11 07:43:18 -07:00
Simon Willison
26f3b20e58 Fix to our pytest plugin to better support pytest-cov
Refs https://github.com/simonw/datasette/pulls#issuecomment-4681621052
2026-06-11 07:29:27 -07:00
Simon Willison
648a34ce81 Fix for test I broke in 92848c06 refs #2754 2026-06-11 07:13:07 -07:00
Simon Willison
9adb541674 Use asyncinject 0.7 results= seeding for per-request extras context
asyncinject 0.7 fixed the parallel executor stalling when every
initially-ready node is a seeded value, and made seeded values take
precedence over registered functions. That lets the shared per-scope
registries receive the per-request context directly via
resolve_multi(results={'context': ...}) instead of the
contextvars.ContextVar workaround.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 06:42:08 -07:00
Simon Willison
92848c06b8 Stop facet counts from wrapping (#2754)
ul.tight-bullets li uses word-break: break-all so long facet labels can
wrap, but that also let the count number break across lines. Wrap each
count in a span.facet-count with white-space: nowrap so the label can
still wrap while the count stays on one line.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 23:43:32 -07:00
Simon Willison
154ea483ea Pass columns and rows to can_render for canned queries (#2711)
The HTML branch of QueryView built an empty data dict before looping
over register_output_renderer can_render callbacks, so renderers that
depend on the result columns or rows (e.g. datasette-atom,
datasette-ics) never appeared as export options for canned queries.
Populate data with the executed query's rows, columns, SQL and query
name.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 23:36:28 -07:00
Simon Willison
d5141a5778 Fix /-/check 500 for query actions (#2756)
_check_permission_for_actor() constructed child resources with
resource_class(database=parent, table=child), but QueryResource takes a
"query" argument, not "table", so /-/check?action=delete-query (and
view-query / update-query) raised TypeError. Construct the resource
positionally so it works for any child resource class.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 23:36:28 -07:00
Simon Willison
c31bb55011 Add regression test for --default-deny index 500 (#2644)
datasette --default-deny --root with no config file previously 500'd on
the instance and database index pages: rendering them computes is_private
(include_is_private=True), which references the anon_rules CTE, but that
CTE was only defined when anonymous permission rules existed.

This was fixed by the empty-anon_rules fallback added in 4b5fac9c; this
commit adds a regression test that fails without that fallback (SQLite
"no such table: anon_rules" -> 500).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 23:36:28 -07:00
Simon Willison
1c514d69f6 Prevent open redirect via backslash in path (#2680)
asgi_send_redirect() only collapsed leading forward slashes, so a path
like /\example.com/ produced a Location of /\example.com. Browsers
normalise backslashes to forward slashes, turning that into the
protocol-relative //example.com and redirecting off-site. Collapse any
run of leading slashes and backslashes to a single slash.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 23:36:28 -07:00
Simon Willison
9622662132 Fix SQL injection via bracket escape bypass in escape_sqlite() (#2677)
escape_sqlite() wrapped identifiers in [brackets] without escaping any ]
characters inside the string. Since SQLite does not support escaping ]
within bracket quoting, an identifier containing ] could break out and
inject arbitrary SQL. Fall back to double-quote quoting (doubling any
embedded ") when the identifier contains ].

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 23:36:28 -07:00
Simon Willison
4edea3ad26 Build extras registries once per scope instead of per request
ExtraRegistry.resolve() previously constructed a fresh asyncinject
Registry on every table, row and query request - instantiating all
~37 Extra classes and re-running inspect.signature reflection over
each resolve method every time. The Extra classes are stateless, so
the asyncinject Registry for each scope is now built lazily once and
shared, along with the allowed-name sets.

The per-request context reaches the shared registry through a
contextvars.ContextVar provider rather than resolve_multi(results=...)
seeding: asyncinject's parallel executor never schedules anything when
the only initially-ready node is an unregistered pre-seeded value, so
seeding would have stalled every resolution. asyncio tasks copy the
caller's context, which keeps concurrent resolves isolated - covered
by a new test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 23:04:12 -07:00
Simon Willison
cfafa5b37f Use plain set literals for Extra scopes
frozenset({...}) was immutability ceremony for class attributes that
nothing mutates. scopes = {ExtraScope.TABLE} reads cleaner.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 22:56:40 -07:00
Simon Willison
a1b6a6976d Remove dead weight from the extras machinery
- TableExtraContext.next_value, RowExtraContext.resolved and
  QueryExtraContext.stored_query/stored_query_write/error had no
  readers - drop the fields and the arguments that populated them
- Extra.documentation() and the stable classvar were unused parallel
  descriptions of what the docs generator reads directly
- ExtraRegistry.resolve no longer carries an always-true membership
  guard (resolve_multi returns every requested registered name)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 22:55:28 -07:00
Simon Willison
6babd23cec QueryView: only resolve extras for renderer formats, single metadata path
Extras were resolved before the format dispatch, so a .csv request
carrying ?_extra= parameters paid for extras (including per-cell
render_cell plugin calls) whose results were then discarded, and the
HTML path duplicated the stored-query metadata derivation. Extras now
resolve inside the renderer-dispatch branch only, and both consumers
share a query_metadata() helper that no longer fetches database
metadata just to throw it away for stored queries.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 22:53:00 -07:00
Simon Willison
bbf0424c45 Changelog for row/query extras and related fixes
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 22:51:25 -07:00
Simon Willison
b635dc53f4 Make filters, actions and display_rows extras internal
These three extras return values that exist for the HTML templates -
a Filters instance, an async function and markupsafe/sqlite3.Row data
- so requesting them on a .json page returned a 500 serialization
error, while the generated documentation and ?_extra=extras both
advertised them as API surface. They are now public=False: ignored
like any unknown name on JSON requests, omitted from the docs and the
extras list, and still resolved for the HTML view via the new
include_internal flag on ExtraRegistry.resolve().

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 22:50:44 -07:00
Simon Willison
8f888515b6 Fix _extra=query to report the params that were actually bound
QueryExtra re-derived named parameters from the SQL with a regex,
which missed parameters declared in a stored query's params list,
reported magic _-prefixed parameters with raw querystring values that
were never bound, and echoed the entire querystring when no SQL was
present. QueryView now passes its named_parameter_values dict - the
parameters it actually bound - through QueryExtraContext.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 22:47:26 -07:00
Simon Willison
ab62ec96d1 Fix _extra=private for arbitrary SQL query pages
QueryView hardcoded private=False unless the request was for a stored
query, so /db/-/query.json?_extra=private reported false even when
execute-sql was restricted to the authenticated actor. Use
check_visibility() like the table and row views do.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 22:45:13 -07:00
Simon Willison
df8a61450b Remove hasattr/getattr probing from multi-scope extras
TableExtraContext, RowExtraContext and QueryExtraContext now share
normalized table_name, is_view, pks and query_name fields (defaulting
to None/False where inapplicable) so DebugExtra, RenderCellExtra and
RenderersExtra can read them directly. RenderCellExtra uses
context.columns in every scope - the table and row views both derive
columns from results.description so output is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 21:57:02 -07:00
Simon Willison
d825d8c4f3 Remove _get_extras() shim in favor of extra_names_from_request()
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 21:53:41 -07:00