Commit graph

3,168 commits

Author SHA1 Message Date
Simon Willison
993169ae49 Release 1.0a33 1.0a33
Refs #2735, #2677, #2680, #2711, #2756, #2761, #2768, #2754
2026-06-11 08:24:37 -07:00
Simon Willison
4e9556cc24
Redesign and document extras mechanism to cover rows and queries in addition to tables
Merge PR #2769
2026-06-11 07:43:18 -07:00
Simon Willison
26f3b20e58 Fix to our pytest plugin to better support pytest-cov
Refs https://github.com/simonw/datasette/pulls#issuecomment-4681621052
2026-06-11 07:29:27 -07:00
Simon Willison
648a34ce81 Fix for test I broke in 92848c06 refs #2754 2026-06-11 07:13:07 -07:00
Simon Willison
9adb541674 Use asyncinject 0.7 results= seeding for per-request extras context
asyncinject 0.7 fixed the parallel executor stalling when every
initially-ready node is a seeded value, and made seeded values take
precedence over registered functions. That lets the shared per-scope
registries receive the per-request context directly via
resolve_multi(results={'context': ...}) instead of the
contextvars.ContextVar workaround.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 06:42:08 -07:00
Simon Willison
92848c06b8 Stop facet counts from wrapping (#2754)
ul.tight-bullets li uses word-break: break-all so long facet labels can
wrap, but that also let the count number break across lines. Wrap each
count in a span.facet-count with white-space: nowrap so the label can
still wrap while the count stays on one line.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 23:43:32 -07:00
Simon Willison
154ea483ea Pass columns and rows to can_render for canned queries (#2711)
The HTML branch of QueryView built an empty data dict before looping
over register_output_renderer can_render callbacks, so renderers that
depend on the result columns or rows (e.g. datasette-atom,
datasette-ics) never appeared as export options for canned queries.
Populate data with the executed query's rows, columns, SQL and query
name.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 23:36:28 -07:00
Simon Willison
d5141a5778 Fix /-/check 500 for query actions (#2756)
_check_permission_for_actor() constructed child resources with
resource_class(database=parent, table=child), but QueryResource takes a
"query" argument, not "table", so /-/check?action=delete-query (and
view-query / update-query) raised TypeError. Construct the resource
positionally so it works for any child resource class.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 23:36:28 -07:00
Simon Willison
c31bb55011 Add regression test for --default-deny index 500 (#2644)
datasette --default-deny --root with no config file previously 500'd on
the instance and database index pages: rendering them computes is_private
(include_is_private=True), which references the anon_rules CTE, but that
CTE was only defined when anonymous permission rules existed.

This was fixed by the empty-anon_rules fallback added in 4b5fac9c; this
commit adds a regression test that fails without that fallback (SQLite
"no such table: anon_rules" -> 500).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 23:36:28 -07:00
Simon Willison
1c514d69f6 Prevent open redirect via backslash in path (#2680)
asgi_send_redirect() only collapsed leading forward slashes, so a path
like /\example.com/ produced a Location of /\example.com. Browsers
normalise backslashes to forward slashes, turning that into the
protocol-relative //example.com and redirecting off-site. Collapse any
run of leading slashes and backslashes to a single slash.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 23:36:28 -07:00
Simon Willison
9622662132 Fix SQL injection via bracket escape bypass in escape_sqlite() (#2677)
escape_sqlite() wrapped identifiers in [brackets] without escaping any ]
characters inside the string. Since SQLite does not support escaping ]
within bracket quoting, an identifier containing ] could break out and
inject arbitrary SQL. Fall back to double-quote quoting (doubling any
embedded ") when the identifier contains ].

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 23:36:28 -07:00
Simon Willison
4edea3ad26 Build extras registries once per scope instead of per request
ExtraRegistry.resolve() previously constructed a fresh asyncinject
Registry on every table, row and query request - instantiating all
~37 Extra classes and re-running inspect.signature reflection over
each resolve method every time. The Extra classes are stateless, so
the asyncinject Registry for each scope is now built lazily once and
shared, along with the allowed-name sets.

The per-request context reaches the shared registry through a
contextvars.ContextVar provider rather than resolve_multi(results=...)
seeding: asyncinject's parallel executor never schedules anything when
the only initially-ready node is an unregistered pre-seeded value, so
seeding would have stalled every resolution. asyncio tasks copy the
caller's context, which keeps concurrent resolves isolated - covered
by a new test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 23:04:12 -07:00
Simon Willison
cfafa5b37f Use plain set literals for Extra scopes
frozenset({...}) was immutability ceremony for class attributes that
nothing mutates. scopes = {ExtraScope.TABLE} reads cleaner.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 22:56:40 -07:00
Simon Willison
a1b6a6976d Remove dead weight from the extras machinery
- TableExtraContext.next_value, RowExtraContext.resolved and
  QueryExtraContext.stored_query/stored_query_write/error had no
  readers - drop the fields and the arguments that populated them
- Extra.documentation() and the stable classvar were unused parallel
  descriptions of what the docs generator reads directly
- ExtraRegistry.resolve no longer carries an always-true membership
  guard (resolve_multi returns every requested registered name)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 22:55:28 -07:00
Simon Willison
6babd23cec QueryView: only resolve extras for renderer formats, single metadata path
Extras were resolved before the format dispatch, so a .csv request
carrying ?_extra= parameters paid for extras (including per-cell
render_cell plugin calls) whose results were then discarded, and the
HTML path duplicated the stored-query metadata derivation. Extras now
resolve inside the renderer-dispatch branch only, and both consumers
share a query_metadata() helper that no longer fetches database
metadata just to throw it away for stored queries.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 22:53:00 -07:00
Simon Willison
bbf0424c45 Changelog for row/query extras and related fixes
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 22:51:25 -07:00
Simon Willison
b635dc53f4 Make filters, actions and display_rows extras internal
These three extras return values that exist for the HTML templates -
a Filters instance, an async function and markupsafe/sqlite3.Row data
- so requesting them on a .json page returned a 500 serialization
error, while the generated documentation and ?_extra=extras both
advertised them as API surface. They are now public=False: ignored
like any unknown name on JSON requests, omitted from the docs and the
extras list, and still resolved for the HTML view via the new
include_internal flag on ExtraRegistry.resolve().

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 22:50:44 -07:00
Simon Willison
8f888515b6 Fix _extra=query to report the params that were actually bound
QueryExtra re-derived named parameters from the SQL with a regex,
which missed parameters declared in a stored query's params list,
reported magic _-prefixed parameters with raw querystring values that
were never bound, and echoed the entire querystring when no SQL was
present. QueryView now passes its named_parameter_values dict - the
parameters it actually bound - through QueryExtraContext.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 22:47:26 -07:00
Simon Willison
ab62ec96d1 Fix _extra=private for arbitrary SQL query pages
QueryView hardcoded private=False unless the request was for a stored
query, so /db/-/query.json?_extra=private reported false even when
execute-sql was restricted to the authenticated actor. Use
check_visibility() like the table and row views do.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 22:45:13 -07:00
Simon Willison
df8a61450b Remove hasattr/getattr probing from multi-scope extras
TableExtraContext, RowExtraContext and QueryExtraContext now share
normalized table_name, is_view, pks and query_name fields (defaulting
to None/False where inapplicable) so DebugExtra, RenderCellExtra and
RenderersExtra can read them directly. RenderCellExtra uses
context.columns in every scope - the table and row views both derive
columns from results.description so output is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 21:57:02 -07:00
Simon Willison
d825d8c4f3 Remove _get_extras() shim in favor of extra_names_from_request()
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 21:53:41 -07:00
Simon Willison
f4b4506035 Remove legacy ?_extras= row parameter
The pre-1.0 ?_extras= (plural) parameter was kept for backwards
compatibility with the old row JSON API. ?_extra= is the documented
mechanism now that row pages share the extras registry.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 21:49:23 -07:00
Viraat Das
3c1012dcc2
Fix write query failing when a named parameter is called :sql (#2765)
Closes #2761
2026-06-10 20:15:03 -07:00
Simon Willison
d8605ef4c2 Fix execute_isolated_fn() against immutable databases
execute_isolated_fn() always opened its temporary connection with
write=True, which is not allowed for immutable databases - so APIs
that rely on it, like SQL analysis when storing a query, failed.

An immutable database can never receive writes, so there is no write
queue to block: in that case the function now opens a read-only
connection and runs it on the executor, bypassing the write thread
entirely. Mutable databases keep the existing write-thread behavior.

Also fixed a latent bug in the write thread where a connect() failure
for an isolated task would crash the thread instead of delivering the
exception back to the caller.

Closes #2768

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 20:04:55 -07:00
Simon Willison
4d6daa175a Add row and query JSON extras 2026-06-09 02:56:27 -07:00
Simon Willison
0fa872d438 Add debug and request JSON extra examples 2026-06-08 21:20:06 -07:00
Simon Willison
22f80b8196 Clarify render_cell JSON extra example 2026-06-08 21:13:53 -07:00
Simon Willison
79c8aff31d Add generated examples for table JSON extras 2026-06-08 21:10:58 -07:00
Simon Willison
111eeaf370 Document table JSON extras from metadata 2026-06-08 20:56:00 -07:00
Simon Willison
17bbe6855c Refactor table JSON extras into classes 2026-06-08 20:52:10 -07:00
Simon Willison
03f1ffdf8f Centralize JSON extra parsing 2026-06-08 20:45:01 -07:00
Simon Willison
6eaa9e3199
Web UI to edit and delete stored queries (#2764)
* Add web UI to edit and delete stored queries

Stored query pages now offer Edit and Delete actions in the query
actions menu, gated by the update-query and delete-query permissions.

- New QueryEditView (GET/POST at /<db>/<query>/-/edit) renders a
  pre-filled form for editing a query's title, description, SQL and
  privacy, reusing the create-query analysis UI. Changing the SQL still
  requires execute-sql; metadata-only edits do not.
- QueryDeleteView gains a GET confirmation page and HTML form POST that
  redirects to the query list, while keeping the existing JSON API.
- New default query_actions hook adds the Edit/Delete links for stored
  (non-config, non-trusted) queries the actor is allowed to manage.

Permission semantics (already enforced by default_query_permissions_sql)
are surfaced in the UI: owners can always edit/delete their queries;
non-private queries can be edited/deleted by any actor with the relevant
permission; private queries remain owner-only.

Shared the create-query form styles into _query_form_styles.html so the
edit form can reuse them.

Animated demo: https://github.com/simonw/datasette/pull/2764#issuecomment-4655694668

Closes #2760

https://claude.ai/code/session_019GU9g3pZAERukLKYNa4uAL
2026-06-08 20:19:47 -07:00
Simon Willison
911954347e Release 1.0a32 1.0a32
Refs #2757, #2759, #2762, #2763
2026-05-31 16:21:24 -07:00
Simon Willison
f9f3465582 Better empty state message
Root user was being told they didn't have permission when actually
the problem was there were no tables at all.
2026-05-31 16:15:52 -07:00
Simon Willison
b1f3e4368c
Fixes for SQL write with RETURNING (#2763)
* Fix for execute write returning, closes #2762
* Fix stored write returning rowcount message
* Add configurable execute_write returning limit
* Return rows/truncated from execute query if it used RETURNING
* INSERT ... RETURNING shows rows in /-/execute-write
* Skip RETURNING tests if SQLite version does not support it

Screenshot: https://github.com/simonw/datasette/issues/2762#issuecomment-4588111545
2026-05-31 16:15:34 -07:00
Simon Willison
1558ab7989 Fix remaining base_url issues 2026-05-30 22:48:04 -07:00
Simon Willison
d657fb4315 Fix double-prefixed export links with base_url
Use the router-stripped route_path when building request-derived export
URLs, so table, row, and query JSON/CSV links do not apply base_url twice.

Keep urls.path() behavior unchanged, and add coverage for both /prefix/
exports and a /data/ base_url with a data database.

Closes #2759
2026-05-30 22:41:54 -07:00
Simon Willison
81a4df8a3e Fix for /-/jump with base_url set, closes #2757 2026-05-30 12:25:23 -07:00
Simon Willison
c1476a48d8 Release 1.0a31 1.0a31
Refs #2712, #2735, #2742, #2743, #2747, #2748
2026-05-28 20:29:57 -07:00
Simon Willison
72cf476d1d Tidied up release notes ready to ship
Refs #2741, #2749
2026-05-28 20:28:24 -07:00
Simon Willison
9e377e8b90 Only show valid SQL write templates
Closes #2753

Demo: https://github.com/simonw/datasette/issues/2753#issuecomment-4570071413
2026-05-28 20:01:56 -07:00
Simon Willison
52729faa54 /<database>/-/query.json and changelog docs 2026-05-28 20:01:56 -07:00
Simon Willison
e5b6166fa3 Nicer UI around Execute Write SQL denied
Refs https://github.com/simonw/datasette/issues/2753#issuecomment-4569117665
2026-05-28 20:01:56 -07:00
Simon Willison
6a998610ee
datasette inspect now counts 10,000+ tables correctly (#2752)
Closes #2712

Refs https://github.com/simonw/datasette/pull/2721#issuecomment-4568966383
2026-05-28 15:52:51 -07:00
Simon Willison
74324cb849 Improved docs for user-facing SQL query pages
- /database-name/-/execute-write
- /-/queries
2026-05-28 15:46:27 -07:00
Simon Willison
b6e9b18990 datasette.yml can no longer set a query to private
Private means it has an owner, and the config does not let
you say who the owner is - plus configured queries should
not be possible to edit or delete in the UI so having an
owner makes even less sense.

You can still make configured queries visible to specific
people using regular view-query permissions.
2026-05-28 15:37:48 -07:00
Simon Willison
dd73eb018d
Analyze write SQL for full set of SQLite operations
PR #2749  - https://github.com/simonw/datasette/pull/2749
2026-05-28 15:25:28 -07:00
Simon Willison
cd838daef4 Refactor tests a bit 2026-05-28 15:22:21 -07:00
Simon Willison
0b7c26c6c8 Refactored write decision tests 2026-05-28 12:09:20 -07:00
Simon Willison
17f45b884b Clarify ignored write SQL operation tests
Split the combined ignored-operation decision test into separate internal-operation and select-statement cases.

Assert the decision reason for each case instead of checking the shared base class, so the tests document why those operations are ignored.
2026-05-28 12:06:57 -07:00