Remove scratch database APIs from the connection proposal

This commit is contained in:
Simon Willison 2026-10-07 12:25:44 -07:00
commit fb2b4a3ad8
13 changed files with 17 additions and 2647 deletions

View file

@ -140,8 +140,6 @@ Use ``--internal PATH`` or the ``DATASETTE_INTERNAL`` environment variable to pe
--ssl-certfile TEXT SSL certificate file
--internal PATH Path to a persistent Datasette internal SQLite
database
--scratch-dir DIRECTORY Directory for scratch databases, which then
survive a restart
--help Show this message and exit.

View file

@ -1395,163 +1395,12 @@ The ``name`` and ``route`` parameters are optional and work the same way as they
This removes a database that has been previously added. ``name=`` is the unique name of that database.
.. _datasette_scratch_databases:
Scratch databases
-----------------
A *scratch database* is a SQLite file that a plugin (or a user, through a plugin) creates, copies data into, changes, uses for a while and eventually throws away. Scratch databases are served like any other database - ``/name``, the JSON API, the write API - and they survive a restart of Datasette.
They live in the **scratch directory**:
- ``datasette --scratch-dir path/to/scratch`` (or the ``DATASETTE_SCRATCH_DIR`` environment variable, or ``Datasette(scratch_dir=...)``). The directory is created if it does not exist.
- In :ref:`config_dir`, a ``scratch/`` directory inside the configuration directory is used if it exists and either is empty or was created by Datasette (it contains ``.datasette-scratch.db``). A ``scratch/`` folder of your own that holds other files is left alone, with a warning in the log. If another Datasette instance is already using it - a server, while you run ``datasette mydir --get ...`` - the second instance starts without its scratch databases, with a warning.
- Otherwise scratch databases are **temporary**: they go in a temporary directory that is created the first time a plugin creates one and deleted when Datasette shuts down, the same way the internal database is temporary unless ``--internal`` is used. ``datasette.scratch_dir`` is ``None`` until then. Plugins can use the same API either way.
Use a directory just for scratch databases. Only one Datasette instance can use a scratch directory at a time: a second one given the same ``--scratch-dir`` fails to start with an error. The first time Datasette opens a directory it adopts any SQLite files already in it, but never deletes anything: leftover ``-wal``, ``-shm`` or ``-journal`` files without their database are only removed once the directory holds Datasette's registry.
The directory contains one ``<name>.db`` file per scratch database, plus a registry (``.datasette-scratch.db``) that records each database's owner, metadata, creation time and when it was last used. Scratch databases are created in WAL mode (rollback journal mode with ``--nolock``).
**Startup.** Existing scratch databases are attached when Datasette starts without opening any of them: Datasette reads the registry and lists the directory. Like any database, a scratch database gets connections and a write thread only while it is being used, and they are closed again after :ref:`setting_connection_idle_timeout_ms`. With a persistent internal database (``--internal``) and a clean shutdown, the table catalog for scratch databases is restored at startup without opening them either. With the default temporary internal database each scratch database is opened briefly, once, at startup to read its schema - as are, after a crash, the ones that were written to since their schema was last read. ``datasette serve`` does not run its startup check on scratch databases: a scratch database whose file cannot be read is reported in the log and does not stop the server from starting. Pages that list every database, such as the index page, read scratch databases' tables from the catalog and do not open them.
**The catalog** of a scratch database's tables is kept current by its own writes (schema watch mode ``"owned"``): a table created through any of the write methods is in the catalog before the write returns. If another process writes to a scratch database file directly, Datasette will not notice schema changes.
**Copying files by hand.** The directory is the source of truth. A ``.db`` file copied into the scratch directory while Datasette is stopped is adopted as a scratch database at the next startup, with no owner or metadata; its name must follow the naming rules below. A scratch database file deleted by hand is dropped from the registry at the next startup. Do not add, remove or rename files while Datasette is running. Copying a scratch database's ``.db`` file out of the directory gives you an ordinary SQLite database: nothing Datasette-specific is stored inside it.
**Crashes.** Creating, deleting and renaming are recorded in the registry before the files are touched, so if Datasette is killed part-way through, the next startup either finishes the operation (delete, rename) or undoes it (create). It also removes ``-wal``, ``-shm`` and ``-journal`` files whose database file is missing, since SQLite would otherwise apply an old ``-wal`` file to a new database created later under the same name.
**Names** are 1 to 64 characters long, use only letters, digits, ``_`` and ``-``, and start with a letter or a digit. They are unique among all attached databases, and case-insensitively unique among scratch databases (``Foo`` and ``foo`` would be the same file on macOS and Windows).
**Permissions.** Scratch databases are subject to the same permission checks as every other database. Recording an owner does not restrict who can see a scratch database: plugins that create scratch databases for one user should also restrict access to them, for example with the :ref:`plugin_hook_permission_resources_sql` plugin hook.
Scratch databases are not attached to the ``_memory`` database by ``--crossdb``, because they can be deleted at any time. To combine data from several databases in one query, ``ATTACH`` them inside your own write function, as in the recipe below.
.. _datasette_create_scratch_database:
await .create_scratch_database(name=None, \*, actor=None, metadata=None)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
``name`` - string, optional
The name of the new database. Defaults to a random name like ``scratch_3fa85f64``.
``actor`` - dictionary or string, optional
The actor the database is created for. Its ``"id"`` (or the string itself) is recorded as the owner.
``metadata`` - dictionary, optional
Anything JSON-serializable that the plugin wants to remember about this database, returned by :ref:`datasette_list_scratch_databases`.
Creates an empty database file in the scratch directory, attaches it and returns its ``ScratchDatabase`` object, a subclass of :ref:`internals_database` with extra ``.owner``, ``.created``, ``.last_used`` and ``.scratch_metadata`` attributes and ``.is_scratch`` set to ``True``. Raises ``ValueError`` for an invalid name and ``datasette.scratch.ScratchDatabaseExists`` if a database with that name already exists.
.. code-block:: python
db = await datasette.create_scratch_database(
actor=request.actor, metadata={"source": "my-plugin"}
)
await db.execute_write(
"create table notes (id integer primary key, body text)"
)
.. _datasette_delete_scratch_database:
await .delete_scratch_database(name)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Detaches the scratch database and deletes its files (including ``-wal`` and ``-shm``), its catalog entries and its registry entry. Raises ``datasette.scratch.ScratchDatabaseNotFound`` if there is no scratch database with that name.
Reads and writes that are already running when ``delete_scratch_database()`` is called are allowed to finish: the delete waits for them. Reads and writes that are queued but have not started, including ``block=False`` writes, and any call made afterwards using the old ``Database`` object, raise ``datasette.scratch.ScratchDatabaseDeleted`` (a subclass of ``DatasetteClosedError``).
.. _datasette_rename_scratch_database:
await .rename_scratch_database(name, new_name)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Renames a scratch database, keeping its data, owner and metadata, and returns the new ``ScratchDatabase`` object. This detaches the database and waits for running queries in the same way as deleting it, so calls using the old object raise ``ScratchDatabaseDeleted``. Raises ``ScratchDatabaseExists`` if ``new_name`` is taken.
.. _datasette_list_scratch_databases:
await .list_scratch_databases()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Returns a list of ``datasette.scratch.ScratchDatabaseInfo`` objects, sorted by name, with these attributes:
- ``name`` and ``path``
- ``created`` and ``last_used`` - Unix timestamps. ``last_used`` is the time of the most recent read or write. It is kept in memory and written to the registry about once a minute and when Datasette shuts down, so after a crash it can be up to a minute old.
- ``size`` - in bytes, including the ``-wal`` file
- ``owner`` - the actor ID passed to ``create_scratch_database()``, or ``None``
- ``metadata`` - the dictionary passed to ``create_scratch_database()``
- ``attached`` - ``False`` if the database could not be attached because another database already uses its name
A plugin that cleans up scratch databases nobody has used for a week could do this:
.. code-block:: python
import time
for info in await datasette.list_scratch_databases():
if info.last_used < time.time() - 7 * 24 * 60 * 60:
await datasette.delete_scratch_database(info.name)
.. _datasette_scratch_copy_recipe:
Copying data into a scratch database
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The fastest way to copy a table into a scratch database is to ``ATTACH`` the source database read-only inside one write function and copy with ``INSERT ... SELECT``. The rows never pass through Python, and 10,000 rows take around 10ms.
.. code-block:: python
from pathlib import Path
from datasette.utils import escape_sqlite
async def copy_table(
datasette, source_name, table, scratch
):
source = datasette.get_database(source_name)
source_uri = Path(source.path).resolve().as_uri() + (
"?mode=ro" if source.is_mutable else "?immutable=1"
)
def copy_in(conn):
conn.execute(
"ATTACH DATABASE ? AS source", [source_uri]
)
try:
with conn:
conn.execute("BEGIN IMMEDIATE")
create_sql = conn.execute(
"select sql from source.sqlite_master "
"where type = 'table' and name = ?",
[table],
).fetchone()[0]
conn.execute(create_sql)
conn.execute(
"insert into main.{t} select * from source.{t}".format(
t=escape_sqlite(table)
)
)
finally:
conn.execute("DETACH DATABASE source")
await scratch.execute_write_fn(
copy_in, transaction=False
)
Notes on this recipe:
- It works for mutable source databases (``?mode=ro``) and immutable ones (``?immutable=1``). It needs a database file: plain in-memory databases cannot be attached this way.
- Pass ``transaction=False`` and manage the transaction yourself, as here. SQLite cannot ``DETACH`` a database that was read inside a transaction that is still open, so with the default ``transaction=True`` the ``DETACH`` fails and the attached database stays on the write connection.
- ``DETACH`` in a ``finally`` block: an ``ATTACH`` lasts as long as the connection, and the write connection is shared by every later write to that database (see :ref:`database_connections`).
- Reusing the original ``CREATE TABLE`` statement keeps the primary key, column types and constraints. ``CREATE TABLE ... AS SELECT`` would lose them.
- With a mutable source the ``INSERT ... SELECT`` reads one consistent snapshot of it. If the source uses a rollback journal rather than WAL, writes to it wait until the copy has finished.
- The table is in the catalog, and visible in Datasette, when ``execute_write_fn()`` returns.
.. _datasette_close:
.close()
--------
Release all resources held by this ``Datasette`` instance. This calls :ref:`database_close` on every attached database (including the internal database), shuts down the thread pool executor used to run SQL queries, and unlinks the temporary file used to back the internal database if one was created. It also records when each :ref:`scratch database <datasette_scratch_databases>` was last used and deletes the scratch directory if it was a temporary one.
Release all resources held by this ``Datasette`` instance. This calls :ref:`database_close` on every attached database (including the internal database), shuts down the thread pool executor used to run SQL queries, and unlinks the temporary file used to back the internal database if one was created.
``close()`` is synchronous, idempotent and one-way: after a call to ``close()`` any attempt to use the Datasette instance to execute SQL will raise a ``datasette.database.DatasetteClosedError`` exception. A closed ``Datasette`` cannot be reopened — callers that need a fresh instance should construct a new one.

View file

@ -56,7 +56,6 @@ The files that can be included in this directory are as follows. All are optiona
* ``templates/`` - a directory containing :ref:`customization_custom_templates`
* ``plugins/`` - a directory containing plugins, see :ref:`writing_plugins_one_off`
* ``static/`` - a directory containing static files - these will be served from ``/static/filename.txt``, see :ref:`customization_static_files`
* ``scratch/`` - a directory for :ref:`scratch databases <datasette_scratch_databases>`, used instead of a temporary directory if it exists
Settings
--------