Commit graph

2,870 commits

Author SHA1 Message Date
Simon Willison
8bc9b1ee03
/-/schema and /db/-/schema and /db/table/-/schema pages (plus .json/.md)
* Add schema endpoints for databases, instances, and tables

Closes: #2586

This commit adds new endpoints to view database schemas in multiple formats:

- /-/schema - View schemas for all databases (HTML, JSON, MD)
- /database/-/schema - View schema for a specific database (HTML, JSON, MD)
- /database/table/-/schema - View schema for a specific table (JSON, MD)

Features:
- Supports HTML, JSON, and Markdown output formats
- Respects view-database and view-table permissions
- Uses group_concat(sql, ';' || CHAR(10)) from sqlite_master to retrieve schemas
- Includes comprehensive tests covering all formats and permission checks

The JSON endpoints return:
- Instance level: {"schemas": [{"database": "name", "schema": "sql"}, ...]}
- Database level: {"database": "name", "schema": "sql"}
- Table level: {"database": "name", "table": "name", "schema": "sql"}

Markdown format provides formatted output with headings and SQL code blocks.

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-07 12:01:23 -08:00
Simon Willison
1df4028d78 add_memory_database(memory_name, name=None, route=None) 2025-11-05 15:18:17 -08:00
Simon Willison
257e1c1b1b Release 1.0a21 1.0a21
Refs #2429, #2511, #2578, #2583
2025-11-05 13:51:58 -08:00
Simon Willison
d814e81b32
datasette.client.get(..., skip_permission_checks=True)
Closes #2580
2025-11-05 13:38:01 -08:00
Simon Willison
ec99bb46f8 stable-docs YAML workflow, refs #2582 2025-11-05 10:51:46 -08:00
Simon Willison
3c2254463b
Release notes for 0.65.2
Adding those to main. Refs #2579
2025-11-05 10:25:37 -08:00
Simon Willison
f12f6cc2ab
Get publish cloudrun working with latest Cloud Run (#2581)
Refs:
- #2511

Filter out bad services, refs:
- https://github.com/simonw/datasette/pull/2581#issuecomment-3492243400
2025-11-05 09:28:41 -08:00
Simon Willison
12016342e7 Fix test_metadata_yaml I broke in #2578 2025-11-04 18:40:58 -08:00
Simon Willison
b4385a3ff7 Made test_serve_with_get_headers a bit more forgiving 2025-11-04 18:39:25 -08:00
Simon Willison
ce464da34b datasette --get --headers option, closes #2578 2025-11-04 18:12:15 -08:00
Simon Willison
9f74dc22a8 Run cog with --extra test
Previously it kept on adding stuff to cli-reference.rst
that came from other plugins installed for my global environment
2025-11-04 18:11:24 -08:00
Simon Willison
8b371495dc Move open redirect fix to asgi_send_redirect, refs #2429
See https://github.com/simonw/datasette/pull/2500#issuecomment-3488632278
2025-11-04 17:08:06 -08:00
James Jefferies
f257ca6edb
Fix for open redirect - identified in Issue 2429 (#2500)
* Issue 2429 indicates the possiblity of an open redirect

The 404 processing ends up redirecting a request with multiple path
slashes to that site, i.e.

https://my-site//shedcode.co.uk will redirect to https://shedcode.co.uk

This commit uses a regular expression to remove the multiple leading
slashes before redirecting.
2025-11-04 17:04:12 -08:00
Simon Willison
295e4a2e87 Pin to httpx<1.0
Refs https://github.com/encode/httpx/issues/3635
Closes #2576
2025-11-03 15:05:17 -08:00
Simon Willison
95a1fef280 Release 1.0a20 1.0a20
Refs #2488, #2495, #2503, #2505, #2509, #2510, #2513, #2515, #2517, #2519, #2520, #2521,
#2524, #2525, #2526, #2528, #2530, #2531, #2534, #2537, #2543, #2544, #2550, #2551,
#2555, #2558, #2561, #2562, #2564, #2565, #2567, #2569, #2570, #2571, #2574
2025-11-03 14:47:24 -08:00
Simon Willison
dc3f9fe9e4 Python 3.10, not 3.8 2025-11-03 14:42:59 -08:00
Simon Willison
5d4dfcec6b Fix for link from changelog not working
Annoyingly we now get a warning in the docs build about a duplicate label,
but it seems harmless enough.
2025-11-03 14:38:57 -08:00
Simon Willison
b3b8c5831b Fixed some broken reference links on upgrade guide 2025-11-03 14:34:29 -08:00
Simon Willison
b212895b97 Updated release notes for 1.0a20
Refs #2550
2025-11-03 14:27:41 -08:00
Simon Willison
18fd373a8f
New PermissionSQL.restriction_sql mechanism for actor restrictions
Implement INTERSECT-based actor restrictions to prevent permission bypass

Actor restrictions are now implemented as SQL filters using INTERSECT rather
than as deny/allow permission rules. This ensures restrictions act as hard
limits that cannot be overridden by other permission plugins or config blocks.

Previously, actor restrictions (_r in actor dict) were implemented by 
generating permission rules with deny/allow logic. This approach had a 
critical flaw: database-level config allow blocks could bypass table-level 
restrictions, granting access to tables not in the actor's allowlist.

The new approach separates concerns:

- Permission rules determine what's allowed based on config and plugins
- Restriction filters limit the result set to only allowlisted resources
- Restrictions use INTERSECT to ensure all restriction criteria are met
- Database-level restrictions (parent, NULL) properly match all child tables

Implementation details:

- Added restriction_sql field to PermissionSQL dataclass
- Made PermissionSQL.sql optional to support restriction-only plugins
- Updated actor_restrictions_sql() to return restriction filters instead of rules
- Modified SQL builders to apply restrictions via INTERSECT and EXISTS clauses

Closes #2572
2025-11-03 14:17:51 -08:00
Simon Willison
c76c3e6e6f facet_suggest_time_limit_ms 200ms in tests, closes #2574 2025-11-03 11:52:12 -08:00
Simon Willison
fa978ec100 More upgrade tips, written by Claude Code
Refs #2549

From the datasette-atom upgrade, https://gistpreview.github.io/?d5047e04bbd9c20c59437916e21754ae
2025-11-02 12:02:45 -08:00
Simon Willison
2459285052 Additional upgrade notes by Codex CLI
Refs https://github.com/simonw/datasette/issues/2549#issuecomment-3477398336

Refs #2564
2025-11-01 20:32:42 -07:00
Simon Willison
506ce5b0ac Remove docs for obsolete register_permissions() hook, refs #2528
Also removed docs for datasette.get_permission() method which no longer exists.
2025-11-01 20:23:37 -07:00
Simon Willison
063bf7a96f Action() is kw_only, abbr= is optional, closes #2571 2025-11-01 20:20:17 -07:00
Simon Willison
7e09e1bf1b Removed obsolete actor ID v.s. actor dict code, refs #2570 2025-11-01 19:30:56 -07:00
Simon Willison
e37aa37edc Further refactor to collapse some utility functions
Refs #2570
2025-11-01 19:28:31 -07:00
Simon Willison
b8cee8768e Completed upgrade guide, closes #2564 2025-11-01 18:57:56 -07:00
Simon Willison
5c16c6687d Split permissions_resources_sql() into 5 for readability
Also remove an obsolete test that caused trouble with the new split plugin hook.

Closes #2570
2025-11-01 18:38:47 -07:00
Simon Willison
a528555e84
Additional actor restriction should not grant access to additional actions (#2569)
Closes #2568
2025-11-01 18:38:29 -07:00
Simon Willison
2b962beaeb Fix permissions_execute_sql warnings in documentation 2025-11-01 11:52:23 -07:00
Simon Willison
5705ce0d95
Move takes_child/takes_parent information from Action to Resource (#2567)
Simplified Action by moving takes_child/takes_parent logic to Resource

- Removed InstanceResource - global actions are now simply those with resource_class=None
- Resource.parent_class - Replaced parent_name: str with parent_class: type[Resource] | None for direct class references
- Simplified Action dataclass - No more redundant fields, everything is derived from the Resource class structure
- Validation - The __init_subclass__ method now checks parent_class.parent_class to enforce the 2-level hierarchy

Closes #2563
2025-11-01 11:35:08 -07:00
Simon Willison
1f8995e776 upgrade-1.0a20.md, refs #2564
And another Markdown conversion, refs #2565
2025-10-31 19:13:41 -07:00
Simon Willison
47e4060469 Enable MyST Markdown docs, port events.rst, refs #2565 2025-10-31 16:38:04 -07:00
Simon Willison
48982a0ff5 Mark 1.0a20 unreleased
Refs #2550
2025-10-31 16:12:54 -07:00
Simon Willison
223dcc7c0e Remove unused link 2025-10-31 16:11:53 -07:00
Simon Willison
3184bfae54 Release notes for 1.0a20, refs #2550 2025-10-31 15:37:30 -07:00
Simon Willison
e5f392ae7a datasette.allowed_resources_sql() returns namedtuple 2025-10-31 15:07:37 -07:00
Simon Willison
400fa08e4c
Add keyset pagination to allowed_resources() (#2562)
* Add keyset pagination to allowed_resources()

This replaces the unbounded list return with PaginatedResources,
which supports efficient keyset pagination for handling thousands
of resources.

Closes #2560

Changes:
- allowed_resources() now returns PaginatedResources instead of list
- Added limit (1-1000, default 100) and next (keyset token) parameters
- Added include_reasons parameter (replaces allowed_resources_with_reasons)
- Removed allowed_resources_with_reasons() method entirely
- PaginatedResources.all() async generator for automatic pagination
- Uses tilde-encoding for tokens (matching table pagination)
- Updated all callers to use .resources accessor
- Updated documentation with new API and examples

The PaginatedResources object has:
- resources: List of Resource objects for current page
- next: Token for next page (None if no more results)
- all(): Async generator that yields all resources across pages

Example usage:
    page = await ds.allowed_resources("view-table", actor, limit=100)
    for table in page.resources:
        print(table.child)

    # Iterate all pages automatically
    async for table in page.all():
        print(table.child)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-31 14:50:46 -07:00
Simon Willison
b7ef968c6f Fixed some rST labels I broke 2025-10-31 09:15:39 -07:00
Simon Willison
ba654b5576 Forbid same DB passed twice or via config_dir, closes #2561 2025-10-30 21:40:09 -07:00
Simon Willison
e4be95b16c
Update permissions documentation for new action system (#2551) 2025-10-30 17:59:54 -07:00
Simon Willison
87aa798148 Permission tabs include allow debug page
Closes #2559
2025-10-30 17:54:07 -07:00
Simon Willison
6a71bde37f
Permissions SQL API improvements (#2558)
* Neater design for PermissionSQL class, refs #2556
  - source is now automatically set to the source plugin
  - params is optional
* PermissionSQL.allow() and PermissionSQL.deny() shortcuts

Closes #2556

* Filter out temp database from attached_databases()

Refs https://github.com/simonw/datasette/issues/2557#issuecomment-3470510837
2025-10-30 15:48:46 -07:00
Simon Willison
5247856bd4 Filter out temp database from attached_databases()
Refs https://github.com/simonw/datasette/issues/2557#issuecomment-3470510837
2025-10-30 15:48:10 -07:00
Simon Willison
ce4b0794b2
Ported setup.py to pyproject.toml (#2555)
* Ported setup.py to pyproject.toml, refs #2553

* Make fixtures tests less flaky

The in-memory fixtures table was being shared between different
instances of the test client, leading to occasional errors when
running the full test suite.
2025-10-30 10:41:41 -07:00
Simon Willison
53e6a72a95 Move black to YAML, not pytest 2025-10-30 10:40:46 -07:00
Simon Willison
1289eb0589 Fix SQLite locking issue in execute_write_script
The execute_write_script() method was causing SQLite database locking
errors when multiple executescript() calls ran in quick succession.

Root cause: SQLite's executescript() method has special behavior - it
implicitly commits any pending transaction and operates in autocommit
mode. However, execute_write_script() was passing these calls through
execute_write_fn() with the default transaction=True, which wrapped
the executescript() call in a transaction context (with conn:).

This created a conflict where sequential executescript() calls would
cause the second call to fail with "OperationalError: database table
is locked: sqlite_master" because the sqlite_master table was still
locked from the first operation's implicit commit.

Fix: Pass transaction=False to execute_write_fn() since executescript()
manages its own transactions and should not be wrapped in an additional
transaction context.

This was causing test_hook_extra_body_script to fail because the
internal database initialization (which calls executescript twice in
succession) would fail, preventing the application from rendering
pages correctly.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-30 10:30:09 -07:00
Simon Willison
5da3c9f4bd Better display of recent permissions checks, refs #2543 2025-10-30 10:28:04 -07:00
Simon Willison
b018eb3171 Simplified the code for the permission debug pages
Decided not to use as much JavaScript

Used Codex CLI for this. Refs #2543
2025-10-30 10:28:04 -07:00