Document connection budgets, resource errors, and lifecycle changes

docs/changelog.rst

@@ -4,6 +4,22 @@
 Changelog
 =========
 
+.. _changelog_unreleased:
+
+Unreleased
+----------
+
+Connection management
+~~~~~~~~~~~~~~~~~~~~~
+
+Datasette now pools read connections and bounds writer threads and queued writes across the instance, allowing it to serve many database files without keeping each one open. (:pr:`2989`)
+
+- New settings default to :ref:`32 pooled file read connections <setting_max_open_connections>`, :ref:`8 user database writers <setting_max_write_connections>`, :ref:`256 pending writes <setting_max_pending_writes>` and a :ref:`five-second write queue timeout <setting_write_queue_timeout_ms>`. Connections and writer threads retire after :ref:`30 seconds idle <setting_connection_idle_timeout_ms>` or under capacity pressure. These limits do not impose a hard cap on total memory or file descriptors. Named in-memory databases retain writer slots to preserve their contents.
+- Index and database listings use the internal schema catalog. The new :ref:`schema watch settings <setting_schema_watch_interval_ms>` control detection of changes made outside Datasette. Immutable table row counts are now calculated when needed instead of at startup.
+- Connections, cursors, blob handles and dump iterators supplied to plugin callbacks expire when the callback finishes. Using an expired resource, or returning a live resource from ``execute_fn()``, raises ``datasette.database.ConnectionLeaseError``. Fetch results inside the callback and use :ref:`plugin_hook_prepare_connection` for setup needed on every managed connection. See :ref:`database_connection_leases` for details.
+- Full write queues and admission timeouts now raise explicit :ref:`resource errors <database_resource_errors>`, with HTTP 503 responses for requests. The new :ref:`plugin_hook_write_task_completed` hook reports completion or failure of accepted non-blocking writes. Plugins that create background tasks should observe their results so admission failures are handled.
+- Graceful shutdown drains accepted writes subject to their admission deadlines and a per-database shutdown timeout. Running callbacks retain ownership of their connections until they finish; see :ref:`database_close`.
+
 .. _v1_0_a41:
 
 1.0a41 (2026-09-24)

docs/internals.rst

@@ -1400,7 +1400,7 @@ This removes a database that has been previously added. ``name=`` is the unique
 .close()
 --------
 
-Release all resources held by this ``Datasette`` instance. This calls :ref:`database_close` on every attached database (including the internal database), shuts down the thread pool executor used to run SQL queries, and unlinks the temporary file used to back the internal database if one was created.
+Release resources held by this ``Datasette`` instance. This stops new submissions to every user database before draining their accepted writes, then closes the internal database and shuts down the thread pool executor used to run SQL queries. Each database follows the deadlines and cleanup rules in :ref:`database_close`, including deferring cleanup of connections and temporary files still owned by running write callbacks.
 
 ``close()`` is synchronous, idempotent and one-way: after a call to ``close()`` any attempt to use the Datasette instance to execute SQL will raise a ``datasette.database.DatasetteClosedError`` exception. A closed ``Datasette`` cannot be reopened — callers that need a fresh instance should construct a new one.
 
@@ -1800,10 +1800,10 @@ Application lifecycle
 Datasette guarantees a fixed sequence of events between the moment a ``Datasette`` instance is constructed and the moment its resources are released:
 
 1. ``Datasette(...)`` — the constructor runs synchronously and does not run plugin hooks.
-2. **Startup** — ``await datasette.invoke_startup()`` runs once, even if several event loops call it at the same time (the others wait for it to finish): it populates the internal database's catalog of table schemas (:ref:`internals_internal`), loads canned queries and column type configuration, then calls every registered :ref:`plugin_hook_startup` hook, in plugin registration order. When Datasette is being served, table-count precomputation for immutable databases runs immediately before this, as part of the same startup sequence.
+2. **Startup** — ``await datasette.invoke_startup()`` runs once, even if several event loops call it at the same time (the others wait for it to finish): it populates the internal database's catalog of table schemas (:ref:`internals_internal`), loads canned queries and column type configuration, then calls every registered :ref:`plugin_hook_startup` hook, in plugin registration order. Immutable table counts are calculated when first requested, unless supplied using :ref:`performance_inspect`.
 3. **Background-task launch** — once *every* ``startup`` hook has finished (not before), every task registered with :ref:`datasette_add_background_task` — by any plugin — is launched. A task registered by one plugin's ``startup`` hook can safely depend on state set up by another plugin's ``startup`` hook, because launch only happens after the whole round of hooks completes.
 4. **Serving** — the instance handles requests (or, for headless or CLI use, does whatever the embedding program does with it).
-5. **Shutdown** — triggered by the ASGI ``lifespan.shutdown`` event (Ctrl-C, ``SIGTERM``) or the end of a ``datasette serve`` process: every :ref:`plugin_hook_shutdown` hook runs first, while background tasks are still alive, so a plugin can tell its own task to wind down gracefully; every still-running background task is then cancelled and given a five-second grace period to actually stop; finally every database connection is released via :ref:`datasette_close`.
+5. **Shutdown** — triggered by the ASGI ``lifespan.shutdown`` event (Ctrl-C, ``SIGTERM``) or the end of a ``datasette serve`` process: every :ref:`plugin_hook_shutdown` hook runs first, while background tasks are still alive, so a plugin can tell its own task to wind down gracefully; every still-running background task is then cancelled and given a five-second grace period to actually stop; finally database resources are cleaned up via :ref:`datasette_close`.
 
 .. admonition:: Startup hooks run on the event loop that serves requests
 
@@ -2069,9 +2069,9 @@ Datasette is designed to serve anything from one database to thousands of databa
 
 **Schema changes** made by Datasette's own writes are detected as soon as the write finishes, and databases that other processes may change are polled - see :ref:`setting_schema_watch_interval_ms`. If a database file is replaced (for example by renaming a new file over it) or deleted, connections to the old file are discarded. Once Datasette has seen a database file exist it will not create it again: writes to a deleted file fail with an error rather than silently creating a new empty database. A database added with :ref:`datasette_add_database` whose file does not exist yet is created by its first write.
 
-**Pages that cover many databases** - the index page, ``allowed_resources()`` listings and their derived-table permission rules, ``/-/api`` and the like - read what they need about each database's tables from the :ref:`catalog tables in the internal database <internals_internal>` instead of opening every database. Plugins that summarise many databases should do the same: a query against ``catalog_tables``, ``catalog_columns`` or ``catalog_foreign_keys`` costs the same however many databases are attached, while calling methods such as ``db.table_names()`` on every database opens a connection to each of them. Use the live methods on a single :ref:`Database <internals_database>` when you need the database's own current answer.
+**Pages that cover many databases** - the index page, ``allowed_resources()`` listings and their derived-table permission rules, ``/-/api`` and the like - read what they need about each database's tables from the :ref:`catalog tables in the internal database <internals_internal>` instead of opening every database. Plugins that summarise many databases should do the same: querying ``catalog_tables``, ``catalog_columns`` or ``catalog_foreign_keys`` avoids opening each database, while calling methods such as ``db.table_names()`` on every database needs a connection to each of them. Catalog queries still cost time and memory according to the data they process, and plugins must apply appropriate permission checks before exposing catalog data to an actor. Use the live methods on a single :ref:`Database <internals_database>` when you need the database's own current answer.
 
-With :ref:`setting_num_sql_threads` set to ``0`` (for example in Pyodide) there are no threads: reads and writes run on the event loop, read connections are still pooled and closed when idle, and the write connection stays open until the database is closed.
+With :ref:`setting_num_sql_threads` set to ``0`` (for example in Pyodide) there are no threads: reads and writes run on the event loop, read connections are still pooled and closed when idle, and retained write connections are bounded by :ref:`setting_max_write_connections`. Idle file write connections can be evicted when another database needs capacity; named in-memory connections are retained to preserve their contents.
 
 Every new connection is set up by the :ref:`plugin_hook_prepare_connection` plugin hook, except isolated connections, the brief connections used to check a file's schema version, and connections to the internal database. Because connections are opened and closed as needed, anything set up on a connection - functions, temporary tables, ``ATTACH``, ``PRAGMA`` settings - only lasts as long as that connection, and is not seen by other connections to the same database. Use ``prepare_connection`` for anything every connection needs.
 
@@ -2094,6 +2094,26 @@ Do not rely on state set up on the connection by an earlier call (see above). Le
 
 The lent connection cannot be used as the *target* of another connection's ``backup()`` method, because that C function needs the real connection object. Using it as the source, ``conn.backup(other)``, works.
 
+.. _database_resource_errors:
+
+Resource errors and retries
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The exceptions in ``datasette.write_budget`` distinguish capacity failures before execution from errors raised after a callback has started:
+
+``DatabaseResourceError``
+    Base class for resource admission failures. Its ``code`` is ``"database_resource_unavailable"`` and ``execution_started`` is ``False``. Reads raise this with the message ``"Timed out waiting for a read connection"`` if they cannot obtain pooled connection capacity within five seconds. Connection opening can also raise it for resource exhaustion before preparation or execution starts.
+``DatabaseQueueFull``
+    A subclass of ``DatabaseResourceError`` with ``code="database_queue_full"`` and message ``"Too many queued database writes; try again later"``. The pending write limit was reached, so this submission was not accepted.
+``DatabaseAdmissionTimeout``
+    A subclass of ``DatabaseResourceError`` with ``code="database_admission_timeout"`` and message ``"Write expired before execution started"``. A queued write was accepted but expired before execution started.
+``DatabaseReentrancyError``
+    A ``RuntimeError`` with ``code="database_reentrant_operation"``. A callback attempted a nested Datasette database operation that could deadlock with finite capacity. Move the dependent operation outside the callback instead of retrying it in place.
+
+Resource admission errors become HTTP 503 responses with ``Retry-After: 1``; JSON responses include ``code`` and ``execution_started: false``. A caller can retry the database operation after an admission error because it has not started. This guarantee applies to that database operation, not to earlier work in the same request. Plugins that create background tasks must observe their results and handle errors; an exception in an unobserved task cannot produce a useful error response for its caller.
+
+Other errors do not carry that guarantee. A transaction may already have committed when catalog refresh or event dispatch fails. Cancelling a calling request after execution starts does not cancel the write. Datasette never automatically retries a callback; only connection opening can be retried once, before preparation hooks or the callback run.
+
 .. _database_constructor:
 
 Database(ds, path=None, is_mutable=True, is_memory=False, memory_name=None, is_temp_disk=False)
@@ -2390,7 +2410,7 @@ For example, archive an article and record the change in an audit log:
 
     await database.execute_write_fn(archive_article)
 
-If you specify ``block=False`` the method becomes fire-and-forget, queueing your function to be executed and then allowing your code after the call to ``.execute_write_fn()`` to continue running while the underlying thread waits for an opportunity to run your function. A UUID representing the queued task will be returned. Any exceptions in your code will be silently swallowed.
+With ``block=False``, a UUID is returned when the task is accepted into the queue; this does not mean that the write has succeeded. A full queue raises ``DatabaseQueueFull`` immediately. Later failures are logged, and accepted tasks report success or failure through :ref:`plugin_hook_write_task_completed`, including expiration before execution. Delivery requires the submitting event loop to remain running and is not durable. With :ref:`setting_num_sql_threads` set to ``0``, the write runs inline before the UUID is returned and failures can raise directly from the call. See :ref:`database_resource_errors` before implementing retries.
 
 .. _database_execute_isolated_fn:
 
@@ -2416,7 +2436,11 @@ The return value of the function will be returned by this method. Any exceptions
 db.close()
 ----------
 
-Release all resources held by this ``Database`` instance. This waits for queries that are already running, shuts down the write thread (if one is running) after it has finished the writes already queued, closes the write connection, and closes this database's pooled read connections.
+Stop accepting new work and release resources held by this ``Database`` instance. Accepted writes continue in FIFO order through normal writer admission, including tasks still waiting for their first writer. Their existing :ref:`admission deadlines <setting_write_queue_timeout_ms>` remain in effect, so accepted writes can still expire before starting.
+
+Shutdown waits up to ten seconds per database for writers to drain. At that deadline, queued user writes that have not started are cancelled with ``DatasetteClosedError``. A callback already running is not interrupted: its thread keeps ownership of its connection and closes it when the callback finishes. Temporary database files are retained until their owning writer exits. A shutdown timeout is reported to standard error.
+
+Running read queries are waited for separately before their connections are closed, so the ten-second writer deadline is not a deadline for the entire ``close()`` call. Database shutdown does not make the write queue durable; process termination can still prevent queued work and completion notifications from finishing.
 
 After ``db.close()`` has been called, any further call to :ref:`database_execute`, :ref:`database_execute_fn`, :ref:`database_execute_write`, :ref:`database_execute_write_fn`, :ref:`database_execute_write_many`, :ref:`database_execute_write_script` or :ref:`database_execute_isolated_fn` will raise a ``datasette.database.DatasetteClosedError`` exception.
 

docs/plugin_hooks.rst

@@ -57,9 +57,9 @@ arguments and can be called like this::
 
     select random_integer(1, 10);
 
-``prepare_connection()`` hooks are not called for Datasette's :ref:`internal database <internals_internal>`.
+``prepare_connection()`` hooks are not called for Datasette's :ref:`internal database <internals_internal>`, :ref:`isolated callbacks <database_execute_isolated_fn>` or the brief connections used only to check a file's schema version.
 
-The hook runs exactly once for every connection Datasette opens: pooled read connections, write connections and the short-lived connections used to rebuild the schema catalog. Datasette closes connections that have been idle for :ref:`setting_connection_idle_timeout_ms` and caps the number of open read connections with :ref:`setting_max_open_connections`, opening new ones as they are needed, so expect the hook to run many times over the life of a server - keep it fast. Anything a plugin needs on every connection (functions, collations, authorizers, ``PRAGMA`` settings) belongs here rather than in a write function, because connections are not shared and do not live forever.
+For pooled read connections, managed write connections and the short-lived connections used to rebuild the schema catalog, the hook runs once when each connection is opened. Datasette closes connections that have been idle for :ref:`setting_connection_idle_timeout_ms` and caps read connections with :ref:`setting_max_open_connections` and writers with :ref:`setting_max_write_connections`, opening new ones as needed, so expect the hook to run many times over the life of a server - keep it fast. Anything a plugin needs on every managed connection (functions, collations, authorizers, ``PRAGMA`` settings) belongs here rather than in a write function, because connections are not shared and do not live forever.
 
 Examples: `datasette-jellyfish <https://datasette.io/plugins/datasette-jellyfish>`__, `datasette-jq <https://datasette.io/plugins/datasette-jq>`__, `datasette-haversine <https://datasette.io/plugins/datasette-haversine>`__, `datasette-rure <https://datasette.io/plugins/datasette-rure>`__
 
@@ -2613,4 +2613,6 @@ write_task_completed(datasette, database, task_id, exception)
 
 Called after an accepted ``execute_write_fn(..., block=False)`` task finishes, fails, or expires before execution. ``database`` is the database name and ``task_id`` is the UUID returned on acceptance. ``exception`` is ``None`` on success, otherwise the exception describing the failure. ``DatabaseAdmissionTimeout`` means execution never started; other execution failures must not be assumed safe to retry. An immediately rejected submission does not trigger this hook because no task was accepted.
 
+Success means the callback and its transaction completed; it does not guarantee catalog refresh or delivery of events queued with ``track_event``. With :ref:`setting_num_sql_threads` set to ``0``, the callback runs inline: failures raise directly from the call, and successful calls return a task ID and schedule this notification.
+
 The hook may return an awaitable. Delivery runs on the submitting event loop and is not durable: stopping that loop or the process can prevent notification. Failures are also logged with ``database``, ``task_id`` and ``error_code`` fields. A failing completion hook is logged and does not retry the write or change its result.

docs/settings.rst

@@ -248,7 +248,7 @@ Default: ``8``
 
 Maximum simultaneous user database writer threads and their retained connections. Must be greater than zero. Each database preserves FIFO write order. When other databases are waiting, a writer closes its connection and releases its slot after its current callback completes. A retiring writer counts against the limit until its thread exits. Idle writers also release their slots under pressure, even if :ref:`setting_connection_idle_timeout_ms` is 0.
 
-The internal catalog has a separate writer, and file schema scans are serialized. Mutable isolated callbacks occupy a writer slot without retaining an additional ordinary file connection. Named in-memory writers occupy non-evictable slots to preserve their data; waiting writes can time out if these consume all capacity. In non-threaded mode, the same setting bounds retained writer connections, evicting an idle file connection before opening another.
+The internal catalog has a separate writer, and file schema scans are serialized. Mutable isolated callbacks occupy a writer slot without retaining an additional ordinary file connection. Named in-memory writers occupy non-evictable slots to preserve their data; waiting writes can time out if these consume all capacity. Schema catalog scans also use these writers, so serving read-only queries to named in-memory databases can consume writer slots. Allow capacity for those databases plus any file writers that need to run alongside them. In non-threaded mode, the same setting bounds retained writer connections, evicting an idle file connection before opening another.
 
 .. _setting_max_pending_writes:
 
@@ -326,7 +326,7 @@ Default: ``1000``
 Datasette keeps a catalog of every attached database's tables, columns, indexes and foreign keys in its :ref:`internal database <internals_internal>`. Each database has a *schema watch mode* that decides how that catalog is kept current:
 
 ``owned``
-    Only Datasette changes this database's schema. After every write Datasette makes to it, Datasette checks whether the schema changed and, if it did, updates the catalog before the write call returns. Owned databases are never polled.
+    Only Datasette changes this database's schema. After every write Datasette makes to it, Datasette checks whether the schema changed. Blocking writes normally refresh a changed schema in the catalog before returning. If a refresh fails transiently, Datasette retains the previous catalog and retries later; the write may already have committed. Owned databases are never polled.
 ``external``
     Other processes may change the file too - for example ``sqlite-utils`` adding a table while Datasette is serving it. On top of the checks after Datasette's own writes, the file is polled every ``schema_watch_interval_ms`` milliseconds. Each check is a ``stat()`` of the database file and its ``-wal`` and ``-journal`` files; a database is only opened if those changed, and its catalog is only rebuilt if its schema changed. Databases whose file is replaced (for example by renaming another file over it) or deleted are noticed too: open connections to the old file are discarded and the catalog is updated.
 ``immutable``