mirror of
https://github.com/simonw/datasette.git
synced 2026-11-17 07:38:32 +01:00
datasette.create_scratch_database() / delete_scratch_database() / rename_scratch_database() / list_scratch_databases() manage SQLite files in a scratch directory (--scratch-dir, config_dir/scratch, or a temporary directory created on first use). - Registry: a SQLite sidecar (.datasette-scratch.db) with intent states (creating/deleting/renaming); startup reconciles it with the directory, which is the source of truth (hand-copied files are adopted, hand-deleted ones dropped, orphan -wal/-shm/-journal removed). flock() per directory. - Scratch databases are attached at startup without opening them, use schema watch mode "owned", and track last_used in memory (flushed to the registry at most once a minute and on close). - Delete/rename: queued reads and writes fail with ScratchDatabaseDeleted, running ones finish; files are unlinked only once no connection (including the SchemaWatcher's, now counted) has them open. - Datasette.close() records final fingerprints of owned file databases in a persistent internal database, so restarts reuse their catalog rows without opening them. - Scratch databases are not attached to crossdb _memory and are skipped by check_databases(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
1055 lines
40 KiB
Python
1055 lines
40 KiB
Python
"""
|
|
Keeps the ``_internal`` catalog (``catalog_databases``, ``catalog_tables``,
|
|
``catalog_columns`` ...) current without checking every attached database on
|
|
the request path.
|
|
|
|
Every attached database has a *schema watch mode*:
|
|
|
|
``owned``
|
|
Every schema change goes through Datasette. After each write task the
|
|
write thread reads ``PRAGMA schema_version`` on the connection it just
|
|
used (microseconds, same thread). If the value moved, the catalog for
|
|
that one database is rebuilt before the write call returns, so callers
|
|
get read-your-writes on the catalog. Owned databases are never polled.
|
|
|
|
``external``
|
|
The file can also be changed by other processes (sqlite-utils, cron
|
|
jobs...). On top of the write-path check, a background task stats the
|
|
database file, its ``-wal`` and its ``-journal`` every
|
|
``schema_watch_interval_ms``. Only a database whose fingerprint
|
|
``(st_dev, st_ino, st_size, st_mtime_ns, st_ctime_ns)`` changed - or
|
|
whose previous fingerprint was *racily clean* - gets a
|
|
``PRAGMA schema_version`` check on a short-lived connection, and only a
|
|
changed ``schema_version`` (or a replaced or deleted file) triggers a
|
|
catalog rebuild for that database.
|
|
|
|
``immutable``
|
|
Scanned once at startup and never again.
|
|
|
|
Defaults: files passed to ``Datasette(files=...)`` / the CLI use the
|
|
``default_schema_watch`` setting (``external`` unless set to ``owned``),
|
|
immutable files are ``immutable``, anything added later with
|
|
``datasette.add_database()`` is ``owned``. Override per database with
|
|
``databases: {name: {schema_watch: owned|external|immutable}}`` in
|
|
``datasette.yaml`` or ``add_database(..., schema_watch=...)``.
|
|
|
|
Replaced and deleted files
|
|
--------------------------
|
|
A sweep that sees a new inode or a missing file calls
|
|
``Database._invalidate_connections()``: the database's connection generation
|
|
is bumped, idle pooled read connections are closed, leased ones are
|
|
discarded when their callback returns, and the write thread reopens its
|
|
connection before the next write. Once Datasette has seen a file exist,
|
|
write connections open it with ``mode=rw``, so a deleted file is never
|
|
silently recreated.
|
|
|
|
Racily clean fingerprints
|
|
-------------------------
|
|
Like git's index: a fingerprint whose newest timestamp is within
|
|
``RACY_WINDOW_NS`` of the moment it was taken could be followed by another
|
|
write inside the same timestamp tick that leaves size, inode and mtime
|
|
identical. Such a fingerprint is flagged ``racy`` and the next sweep runs
|
|
the ``PRAGMA schema_version`` check even though the fingerprint looks
|
|
unchanged. The stat is always taken *before* the pragma, so a write landing
|
|
between the two either shows up in the pragma or changes the next stat.
|
|
|
|
All catalog reads use one short-lived connection per database inside a
|
|
single read transaction, so ``schema_version`` and the rows written to the
|
|
catalog describe the same snapshot. The watcher never keeps a connection
|
|
open.
|
|
"""
|
|
|
|
import asyncio
|
|
import json
|
|
import logging
|
|
import os
|
|
import time
|
|
|
|
from .utils import sqlite3
|
|
from .utils.internal_db import CATALOG_TABLES, collect_schema, write_catalog_entries
|
|
|
|
logger = logging.getLogger("datasette.schema_watcher")
|
|
|
|
MODES = ("owned", "external", "immutable")
|
|
# Valid values for the default_schema_watch setting
|
|
DEFAULT_MODES = ("external", "owned")
|
|
# Passed by Datasette.__init__ for files=/CLI databases: resolved to the
|
|
# default_schema_watch setting once settings are available
|
|
FILES_DEFAULT = "_files_default"
|
|
# Timestamps closer than this to the time of the stat are treated as racy.
|
|
# 2s covers coarse filesystems (FAT: 2s, HFS+: 1s) and clock skew on
|
|
# network filesystems; ext4/xfs/tmpfs on Linux are far finer than this.
|
|
RACY_WINDOW_NS = 2_000_000_000
|
|
# Databases per thread hop / per internal-DB transaction for bulk scans
|
|
SCAN_CHUNK = 64
|
|
SCAN_CONCURRENCY = 1
|
|
# Directories with at least this many watched files are listed once per
|
|
# sweep with os.scandir() instead of probing -wal/-journal with stat()
|
|
SCANDIR_THRESHOLD = 8
|
|
PRAGMA_BUSY_TIMEOUT_MS = 200
|
|
# Read the schema cookie from the file header instead of opening a
|
|
# connection when no -wal/-journal has content
|
|
USE_HEADER_CHECK = True
|
|
|
|
|
|
class WatchState:
|
|
__slots__ = (
|
|
"catalog_version",
|
|
"db",
|
|
"error",
|
|
"fp",
|
|
"fp_racy",
|
|
"missing",
|
|
"mode",
|
|
"name",
|
|
"needs_scan",
|
|
"notified_version",
|
|
"removed",
|
|
"requested_mode",
|
|
"scan_future",
|
|
"stats",
|
|
)
|
|
|
|
def __init__(self, db, name, requested_mode):
|
|
self.db = db
|
|
self.name = name
|
|
self.requested_mode = requested_mode
|
|
self.mode = None
|
|
self.fp = None
|
|
self.fp_racy = True
|
|
self.catalog_version = None
|
|
self.notified_version = None
|
|
self.needs_scan = True
|
|
self.scan_future = None
|
|
self.missing = False
|
|
self.removed = False
|
|
self.error = None
|
|
self.stats = {"checks": 0, "scans": 0, "write_detections": 0}
|
|
|
|
@property
|
|
def is_file(self):
|
|
db = self.db
|
|
return bool(db.path) and not db.is_memory
|
|
|
|
def __repr__(self):
|
|
return f"<WatchState {self.name!r} mode={self.mode} version={self.catalog_version}>"
|
|
|
|
|
|
def _fp_main(st):
|
|
return (st.st_dev, st.st_ino, st.st_size, st.st_mtime_ns, st.st_ctime_ns)
|
|
|
|
|
|
def _fp_side(st):
|
|
# No ctime for -wal/-journal: when running as root SQLite fchown()s
|
|
# these files to match the database every time a connection opens
|
|
# them, so their ctime moves on every open - including our own checks
|
|
return (st.st_ino, st.st_size, st.st_mtime_ns)
|
|
|
|
|
|
def fingerprint(path, present=None):
|
|
"""(main, wal, journal) stat tuples; None for a file that does not exist.
|
|
|
|
``present`` is an optional set of file names in the directory (from
|
|
scandir) used to skip stat() calls for -wal/-journal files that are not
|
|
there.
|
|
"""
|
|
path = os.fspath(path)
|
|
try:
|
|
main = _fp_main(os.stat(path))
|
|
except FileNotFoundError:
|
|
main = None
|
|
parts = [main]
|
|
base = os.path.basename(path) if present is not None else None
|
|
for suffix in ("-wal", "-journal"):
|
|
if present is not None and (base + suffix) not in present:
|
|
parts.append(None)
|
|
continue
|
|
try:
|
|
st = os.stat(path + suffix)
|
|
except FileNotFoundError:
|
|
parts.append(None)
|
|
continue
|
|
# An empty -wal holds no frames (and an empty -journal nothing to
|
|
# roll back): the database is exactly the main file, same as when
|
|
# the file is absent. Read-only opens of a WAL database create an
|
|
# empty -wal, which must not look like a change.
|
|
parts.append(_fp_side(st) if st.st_size else None)
|
|
return tuple(parts)
|
|
|
|
|
|
def newest_timestamp(fp):
|
|
main = fp[0]
|
|
newest = max(main[3], main[4]) if main is not None else 0
|
|
for part in fp[1:]:
|
|
if part is not None:
|
|
newest = max(newest, part[2])
|
|
return newest
|
|
|
|
|
|
def is_racy(fp, t_ns):
|
|
return newest_timestamp(fp) >= t_ns - RACY_WINDOW_NS
|
|
|
|
|
|
SQLITE_HEADER = b"SQLite format 3\x00"
|
|
|
|
|
|
def header_schema_version(path):
|
|
"""The schema cookie (offset 40 of the database header) read straight
|
|
from the file - what PRAGMA schema_version returns - without opening a
|
|
SQLite connection or taking any lock.
|
|
|
|
Only meaningful when there is no non-empty -wal or -journal: then the
|
|
main file is the whole database. The caller must have taken the
|
|
fingerprint *before* calling this; a commit in flight at that moment has
|
|
already bumped the mtime it saw, so the fingerprint is racy and the next
|
|
sweep looks again.
|
|
"""
|
|
try:
|
|
fd = os.open(path, os.O_RDONLY)
|
|
except OSError:
|
|
return None
|
|
try:
|
|
header = os.pread(fd, 100, 0)
|
|
except OSError:
|
|
return None
|
|
finally:
|
|
os.close(fd)
|
|
if len(header) < 100 or not header.startswith(SQLITE_HEADER):
|
|
return None
|
|
return int.from_bytes(header[40:44], "big")
|
|
|
|
|
|
def _fp_to_json(fp, t_ns, closed=False):
|
|
data = {"fp": fp, "t": t_ns}
|
|
if closed:
|
|
# Taken by Datasette.close() after every connection to an owned
|
|
# database had been closed: nothing in this process can change the
|
|
# file after the stat, so the racily-clean rule does not apply
|
|
data["closed"] = True
|
|
return json.dumps(data)
|
|
|
|
|
|
def _fp_from_json(value):
|
|
"""(fp, t_ns, closed) - (None, None, False) if unreadable."""
|
|
try:
|
|
data = json.loads(value)
|
|
fp = tuple(tuple(p) if p is not None else None for p in data["fp"])
|
|
return fp, int(data["t"]), bool(data.get("closed"))
|
|
except Exception: # noqa: BLE001
|
|
return None, None, False
|
|
|
|
|
|
def store_closing_fingerprints(internal_path, rows):
|
|
"""Write SchemaWatcher.closing_fingerprints() to a persistent internal
|
|
database that has already been closed (Datasette.close() is synchronous
|
|
and the internal database's write thread is gone by then). Only rows
|
|
whose stored schema_version still matches are updated."""
|
|
conn = sqlite3.connect(internal_path, isolation_level=None)
|
|
try:
|
|
conn.execute("BEGIN IMMEDIATE")
|
|
conn.executemany(
|
|
"UPDATE catalog_databases SET fingerprint = ? "
|
|
"WHERE database_name = ? AND schema_version = ?",
|
|
[(fp_json, name, version) for name, version, fp_json in rows],
|
|
)
|
|
conn.execute("COMMIT")
|
|
finally:
|
|
conn.close()
|
|
|
|
|
|
class SchemaWatcher:
|
|
def __init__(self, ds):
|
|
self.ds = ds
|
|
self.states = {}
|
|
self._pending = set() # states registered after startup, not yet scanned
|
|
self._pending_removals = set() # names whose catalog rows must go
|
|
self._task = None
|
|
self._spawned = set()
|
|
self._last_sweep = 0.0
|
|
self.counters = {
|
|
"sweeps": 0,
|
|
"sweep_seconds": 0.0,
|
|
"stats": 0,
|
|
"pragma_checks": 0,
|
|
"header_checks": 0,
|
|
"scans": 0,
|
|
"write_detections": 0,
|
|
"replaced": 0,
|
|
"missing": 0,
|
|
"busy": 0,
|
|
"restored_from_persisted": 0,
|
|
}
|
|
|
|
# ------------------------------------------------------------------
|
|
# configuration
|
|
# ------------------------------------------------------------------
|
|
@property
|
|
def interval_s(self):
|
|
ms = self.ds.setting("schema_watch_interval_ms")
|
|
return (ms or 0) / 1000.0
|
|
|
|
def _config_mode(self, name):
|
|
config = getattr(self.ds, "config", None) or {}
|
|
db_config = (config.get("databases") or {}).get(name) or {}
|
|
return db_config.get("schema_watch")
|
|
|
|
def _default_mode_for_files(self):
|
|
from .utils import StartupError
|
|
|
|
mode = self.ds.setting("default_schema_watch") or "external"
|
|
if mode not in DEFAULT_MODES:
|
|
raise StartupError(
|
|
f"Invalid default_schema_watch setting {mode!r}, expected one of "
|
|
f"{', '.join(DEFAULT_MODES)}"
|
|
)
|
|
return mode
|
|
|
|
def _resolve_mode(self, state):
|
|
db = state.db
|
|
mode = self._config_mode(state.name) or state.requested_mode
|
|
if mode is None:
|
|
mode = "owned"
|
|
elif mode == FILES_DEFAULT:
|
|
mode = self._default_mode_for_files()
|
|
if mode not in MODES:
|
|
from .utils import StartupError
|
|
|
|
raise StartupError(
|
|
f"Invalid schema_watch mode {mode!r} for database {state.name!r}, "
|
|
f"expected one of {', '.join(MODES)}"
|
|
)
|
|
if not db.is_mutable:
|
|
# Nothing can change an immutable database
|
|
mode = "immutable"
|
|
elif mode == "external" and not state.is_file:
|
|
# There is no file to stat: rely on the write path
|
|
mode = "owned"
|
|
return mode
|
|
|
|
def configure(self):
|
|
"""Resolve modes once ds.config is available (end of __init__)."""
|
|
# Validate the setting even when no database uses it
|
|
self._default_mode_for_files()
|
|
for state in self.states.values():
|
|
state.mode = self._resolve_mode(state)
|
|
|
|
# ------------------------------------------------------------------
|
|
# registration
|
|
# ------------------------------------------------------------------
|
|
def register(self, db, requested_mode=None):
|
|
if requested_mode not in (None, FILES_DEFAULT) and requested_mode not in MODES:
|
|
raise ValueError(
|
|
f"schema_watch must be one of {', '.join(MODES)}, not {requested_mode!r}"
|
|
)
|
|
state = WatchState(db, db.name, requested_mode)
|
|
if hasattr(self.ds, "config"):
|
|
state.mode = self._resolve_mode(state)
|
|
old = self.states.get(db.name)
|
|
if old is not None:
|
|
old.removed = True
|
|
self.states[db.name] = state
|
|
db._watch_state = state
|
|
self._pending_removals.discard(db.name)
|
|
if self.ds.internal_db_created:
|
|
self._pending.add(state)
|
|
self._spawn(self.refresh([state]))
|
|
return state
|
|
|
|
def unregister(self, name):
|
|
state = self.states.pop(name, None)
|
|
if state is None:
|
|
return
|
|
state.removed = True
|
|
state.db._watch_state = None
|
|
self._pending.discard(state)
|
|
if self.ds.internal_db_created:
|
|
self._pending_removals.add(name)
|
|
self._spawn(self._delete_catalog([name]))
|
|
|
|
# ------------------------------------------------------------------
|
|
# task helpers
|
|
# ------------------------------------------------------------------
|
|
async def _off_loop(self, fn, *args):
|
|
"""Run blocking stat()/scan work in a worker thread - or inline with
|
|
num_sql_threads=0, which must work where threads cannot be started
|
|
at all (Pyodide)."""
|
|
if self.ds.executor is None:
|
|
return fn(*args)
|
|
return await asyncio.to_thread(fn, *args)
|
|
|
|
def _spawn(self, coro):
|
|
try:
|
|
loop = asyncio.get_running_loop()
|
|
except RuntimeError:
|
|
# No loop (sync caller): the work stays pending and is picked
|
|
# up by the next flush/sweep
|
|
coro.close()
|
|
return None
|
|
task = loop.create_task(coro)
|
|
self._spawned.add(task)
|
|
|
|
def _done(t):
|
|
self._spawned.discard(t)
|
|
if not t.cancelled() and t.exception() is not None:
|
|
logger.error("schema watcher task failed", exc_info=t.exception())
|
|
|
|
task.add_done_callback(_done)
|
|
return task
|
|
|
|
def loop_running(self):
|
|
task = self._task
|
|
if task is None or task.done():
|
|
return False
|
|
try:
|
|
return task.get_loop() is asyncio.get_running_loop()
|
|
except RuntimeError:
|
|
return False
|
|
|
|
async def start(self):
|
|
"""Start the polling loop (idempotent). Runs one sweep first so the
|
|
catalog is current when the first request is served."""
|
|
if self.interval_s <= 0 or self.ds._closed or self.loop_running():
|
|
return
|
|
loop = asyncio.get_running_loop()
|
|
# Claim the slot before awaiting so concurrent start() calls no-op
|
|
self._task = loop.create_task(self._run(), name="datasette-schema-watcher")
|
|
if self.ds.internal_db_created:
|
|
try:
|
|
await self.ds._refresh_schemas(background=True)
|
|
except Exception:
|
|
logger.exception("schema watcher initial sweep failed")
|
|
|
|
async def _run(self):
|
|
# Wait one interval first: start() has just swept
|
|
while not self.ds._closed:
|
|
await asyncio.sleep(self.interval_s)
|
|
if self.ds._closed:
|
|
return
|
|
try:
|
|
await self.ds._refresh_schemas(background=True)
|
|
except asyncio.CancelledError:
|
|
raise
|
|
except Exception:
|
|
logger.exception("schema watcher sweep failed")
|
|
|
|
def stop(self):
|
|
tasks = [t for t in [self._task, *self._spawned] if t is not None]
|
|
self._task = None
|
|
for task in tasks:
|
|
if task.done():
|
|
continue
|
|
loop = task.get_loop()
|
|
try:
|
|
running = asyncio.get_running_loop()
|
|
except RuntimeError:
|
|
running = None
|
|
if running is loop:
|
|
task.cancel()
|
|
elif not loop.is_closed():
|
|
try:
|
|
loop.call_soon_threadsafe(task.cancel)
|
|
except RuntimeError:
|
|
pass
|
|
|
|
async def astop(self):
|
|
tasks = [t for t in [self._task, *self._spawned] if t is not None]
|
|
self.stop()
|
|
loop = asyncio.get_running_loop()
|
|
mine = [t for t in tasks if t.get_loop() is loop and not t.done()]
|
|
if mine:
|
|
await asyncio.wait(mine, timeout=5)
|
|
|
|
# ------------------------------------------------------------------
|
|
# request path: O(1) unless add/remove_database left work pending
|
|
# ------------------------------------------------------------------
|
|
async def on_request(self):
|
|
if (self._pending or self._pending_removals) and self.ds.internal_db_created:
|
|
await self.flush_pending()
|
|
if (
|
|
not self.loop_running()
|
|
and self.interval_s > 0
|
|
and self.ds.internal_db_created
|
|
):
|
|
await self.start()
|
|
|
|
async def flush_pending(self):
|
|
if self._pending_removals:
|
|
await self._delete_catalog(list(self._pending_removals))
|
|
if self._pending:
|
|
await self.refresh(list(self._pending))
|
|
|
|
# ------------------------------------------------------------------
|
|
# write path
|
|
# ------------------------------------------------------------------
|
|
def check_after_write(self, state, conn, loop=None):
|
|
"""Called on the thread that owns ``conn`` right after a write task.
|
|
|
|
``loop`` is the event loop to notify (None when already on it)."""
|
|
try:
|
|
version = conn.execute("PRAGMA schema_version").fetchone()[0]
|
|
except Exception: # noqa: BLE001
|
|
return
|
|
if version == state.catalog_version or version == state.notified_version:
|
|
return
|
|
state.notified_version = version
|
|
if loop is None:
|
|
self._on_write_schema_change(state)
|
|
else:
|
|
try:
|
|
loop.call_soon_threadsafe(self._on_write_schema_change, state)
|
|
except RuntimeError:
|
|
pass
|
|
|
|
def _on_write_schema_change(self, state):
|
|
if state.removed:
|
|
return
|
|
state.needs_scan = True
|
|
state.stats["write_detections"] += 1
|
|
self.counters["write_detections"] += 1
|
|
if self.ds.internal_db_created:
|
|
self._spawn(self.refresh([state]))
|
|
|
|
async def after_write(self, state):
|
|
"""Awaited by blocking writes: gives read-your-writes on the catalog."""
|
|
if not self.ds.internal_db_created or state.removed:
|
|
return
|
|
try:
|
|
await self.refresh([state])
|
|
except Exception:
|
|
logger.exception("catalog refresh after write to %r failed", state.name)
|
|
|
|
# ------------------------------------------------------------------
|
|
# sweeps (stat prefilter)
|
|
# ------------------------------------------------------------------
|
|
async def sweep(self, *, background):
|
|
"""Check databases for changes made outside Datasette.
|
|
|
|
background=True (the polling loop): external-mode files only.
|
|
background=False (explicit refresh_schemas()): every mutable
|
|
database - files via the stat prefilter, named in-memory databases
|
|
via a PRAGMA schema_version check.
|
|
"""
|
|
t0 = time.perf_counter()
|
|
if self._pending_removals:
|
|
await self._delete_catalog(list(self._pending_removals))
|
|
candidates = []
|
|
for state in list(self.states.values()):
|
|
if state.mode == "immutable" or state.mode is None:
|
|
if state.needs_scan:
|
|
candidates.append(state)
|
|
continue
|
|
if background and state.mode != "external":
|
|
continue
|
|
candidates.append(state)
|
|
outcomes = []
|
|
file_candidates = [s for s in candidates if s.is_file]
|
|
if file_candidates:
|
|
outcomes = await self._off_loop(self._sweep_sync, file_candidates)
|
|
for state in candidates:
|
|
if not state.is_file and state.db.memory_name and not state.removed:
|
|
outcomes.append(await self._memory_pragma_check(state))
|
|
to_scan = [s for s in self._pending if not s.removed]
|
|
to_clear = []
|
|
for state, kind, fp, t_ns, version in outcomes:
|
|
if state.removed:
|
|
continue
|
|
self.counters["stats"] += 1
|
|
if fp is not None:
|
|
state.fp = fp
|
|
state.fp_racy = is_racy(fp, t_ns)
|
|
if fp[0] is not None:
|
|
state.db._file_seen = True
|
|
if kind == "clean":
|
|
pass
|
|
elif kind == "missing":
|
|
self.counters["missing"] += 1
|
|
if not state.missing:
|
|
state.missing = True
|
|
state.db._invalidate_connections()
|
|
to_clear.append(state)
|
|
elif kind == "replaced":
|
|
self.counters["replaced"] += 1
|
|
state.missing = False
|
|
state.db._invalidate_connections()
|
|
state.needs_scan = True
|
|
elif kind == "busy":
|
|
self.counters["busy"] += 1
|
|
state.fp_racy = True # check again next sweep
|
|
elif kind in ("checked", "header"):
|
|
self.counters[
|
|
"pragma_checks" if kind == "checked" else "header_checks"
|
|
] += 1
|
|
state.stats["checks"] += 1
|
|
if version != state.catalog_version:
|
|
state.needs_scan = True
|
|
elif kind == "error":
|
|
state.fp_racy = True
|
|
if state.needs_scan:
|
|
to_scan.append(state)
|
|
if to_clear:
|
|
await self._clear_missing(to_clear)
|
|
if to_scan:
|
|
await self.refresh(to_scan)
|
|
self.counters["sweeps"] += 1
|
|
self.counters["sweep_seconds"] += time.perf_counter() - t0
|
|
self._last_sweep = time.monotonic()
|
|
|
|
def _sweep_sync(self, states):
|
|
"""Runs in a worker thread. Reads state, never writes it."""
|
|
by_dir = {}
|
|
for state in states:
|
|
if state.is_file:
|
|
by_dir.setdefault(
|
|
os.path.dirname(os.path.abspath(state.db.path)), []
|
|
).append(state)
|
|
out = []
|
|
for directory, dir_states in by_dir.items():
|
|
present = None
|
|
if len(dir_states) >= SCANDIR_THRESHOLD:
|
|
try:
|
|
with os.scandir(directory) as it:
|
|
present = {entry.name for entry in it}
|
|
except OSError:
|
|
present = None
|
|
for state in dir_states:
|
|
t_ns = time.time_ns()
|
|
fp = fingerprint(state.db.path, present)
|
|
prev = state.fp
|
|
if fp == prev and not state.fp_racy and not state.needs_scan:
|
|
out.append((state, "clean", fp, t_ns, None))
|
|
continue
|
|
if fp[0] is None:
|
|
out.append((state, "missing", fp, t_ns, None))
|
|
continue
|
|
if prev is None or prev[0] is None or prev[0][:2] != fp[0][:2]:
|
|
# New inode/device (atomic replace, delete+recreate) or
|
|
# first sight: the schema_version alone cannot be trusted
|
|
out.append((state, "replaced", fp, t_ns, None))
|
|
continue
|
|
if USE_HEADER_CHECK and fp[1] is None and fp[2] is None:
|
|
# No -wal/-journal content: read the cookie from the
|
|
# header. No connection, no SHARED lock that could make
|
|
# a concurrent writer back off.
|
|
version = header_schema_version(os.fspath(state.db.path))
|
|
if version is not None:
|
|
out.append((state, "header", fp, t_ns, version))
|
|
continue
|
|
out.append(self._pragma_check(state, fp, t_ns))
|
|
return out
|
|
|
|
async def _memory_pragma_check(self, state):
|
|
def _version(conn):
|
|
return conn.execute("PRAGMA schema_version").fetchone()[0]
|
|
|
|
try:
|
|
version = await state.db._execute_on_write_connection(_version)
|
|
except Exception: # noqa: BLE001
|
|
return (state, "error", None, None, None)
|
|
return (state, "checked", None, None, version)
|
|
|
|
def _pragma_check(self, state, fp, t_ns):
|
|
db = state.db
|
|
try:
|
|
conn = self._connect(db, prepare=False)
|
|
except Exception: # noqa: BLE001
|
|
return (state, "error", fp, t_ns, None)
|
|
try:
|
|
conn.execute(f"PRAGMA busy_timeout={PRAGMA_BUSY_TIMEOUT_MS}")
|
|
version = conn.execute("PRAGMA schema_version").fetchone()[0]
|
|
return (state, "checked", fp, t_ns, version)
|
|
except sqlite3.OperationalError as e:
|
|
msg = str(e)
|
|
if "locked" in msg or "busy" in msg:
|
|
return (state, "busy", fp, t_ns, None)
|
|
return (state, "error", fp, t_ns, None)
|
|
except Exception: # noqa: BLE001
|
|
return (state, "error", fp, t_ns, None)
|
|
finally:
|
|
self._close(db, conn)
|
|
|
|
# ------------------------------------------------------------------
|
|
# scanning + storing
|
|
# ------------------------------------------------------------------
|
|
def _connect(self, db, prepare):
|
|
# Untracked: Database.close() on the event loop thread must not close
|
|
# this connection while a worker thread is using it (that segfaults).
|
|
# It is always closed by _close() in the same thread that opened it.
|
|
# It is counted instead, so that deleting a scratch database's files
|
|
# can wait for it (and raises once the database has been closed).
|
|
db._untracked_connection_opened()
|
|
try:
|
|
conn = db.connect(track=False)
|
|
except BaseException:
|
|
db._untracked_connection_closed()
|
|
raise
|
|
try:
|
|
if prepare:
|
|
self.ds._prepare_connection(conn, db.name)
|
|
else:
|
|
conn.row_factory = None
|
|
except BaseException:
|
|
self._close(db, conn)
|
|
raise
|
|
return conn
|
|
|
|
@staticmethod
|
|
def _close(db, conn):
|
|
try:
|
|
conn.close()
|
|
finally:
|
|
db._untracked_connection_closed()
|
|
|
|
def _scan_sync(self, state):
|
|
"""Read schema_version + full schema in one read transaction."""
|
|
db = state.db
|
|
fp = t_ns = None
|
|
if state.is_file:
|
|
t_ns = time.time_ns()
|
|
fp = fingerprint(db.path)
|
|
if fp[0] is None:
|
|
return {"state": state, "missing": True, "fp": fp, "t": t_ns}
|
|
conn = self._connect(db, prepare=True)
|
|
try:
|
|
conn.row_factory = sqlite3.Row
|
|
conn.execute("BEGIN")
|
|
try:
|
|
version = conn.execute("PRAGMA schema_version").fetchone()[0]
|
|
schema = collect_schema(conn, db.name)
|
|
finally:
|
|
conn.rollback()
|
|
finally:
|
|
self._close(db, conn)
|
|
return {
|
|
"state": state,
|
|
"fp": fp,
|
|
"t": t_ns,
|
|
"version": version,
|
|
"schema": schema,
|
|
}
|
|
|
|
async def _scan_on_write_connection(self, state):
|
|
name = state.db.name
|
|
|
|
def _scan(conn):
|
|
version = conn.execute("PRAGMA schema_version").fetchone()[0]
|
|
# The write thread's post-task schema check must not re-notify
|
|
# for the version this scan is about to store
|
|
state.notified_version = version
|
|
return version, collect_schema(conn, name)
|
|
|
|
try:
|
|
version, schema = await state.db._execute_on_write_connection(_scan)
|
|
except Exception as e: # noqa: BLE001
|
|
return {"state": state, "error": e}
|
|
return {
|
|
"state": state,
|
|
"fp": None,
|
|
"t": None,
|
|
"version": version,
|
|
"schema": schema,
|
|
}
|
|
|
|
def _scan_chunk_sync(self, states):
|
|
results = []
|
|
for state in states:
|
|
try:
|
|
results.append(self._scan_sync(state))
|
|
except Exception as e: # noqa: BLE001
|
|
results.append({"state": state, "error": e})
|
|
return results
|
|
|
|
async def refresh(self, states):
|
|
"""Rebuild the catalog for every state that needs it, coalescing with
|
|
scans already in flight for the same database."""
|
|
loop = asyncio.get_running_loop()
|
|
pending = list(states)
|
|
for _ in range(10):
|
|
mine = []
|
|
waits = []
|
|
for state in pending:
|
|
fut = state.scan_future
|
|
if fut is not None and not fut.done():
|
|
if fut.get_loop() is loop:
|
|
waits.append(fut)
|
|
continue
|
|
# Scan started on an event loop that has gone away
|
|
state.scan_future = None
|
|
state.needs_scan = True
|
|
if state.needs_scan and not state.removed:
|
|
state.needs_scan = False
|
|
state.scan_future = loop.create_future()
|
|
mine.append(state)
|
|
if mine:
|
|
try:
|
|
await self._scan_and_store(mine)
|
|
finally:
|
|
for state in mine:
|
|
fut = state.scan_future
|
|
if fut is not None and not fut.done():
|
|
fut.set_result(None)
|
|
state.scan_future = None
|
|
if waits:
|
|
await asyncio.wait(waits)
|
|
pending = [
|
|
s
|
|
for s in pending
|
|
if not s.removed
|
|
and (
|
|
s.needs_scan
|
|
or (s.scan_future is not None and not s.scan_future.done())
|
|
)
|
|
and s.error is None
|
|
]
|
|
if not pending:
|
|
break
|
|
for state in states:
|
|
# Failed scans are retried by sweeps, not by every request
|
|
if not state.needs_scan or state.error is not None:
|
|
self._pending.discard(state)
|
|
|
|
async def _scan_and_store(self, states):
|
|
memory = [s for s in states if not s.is_file and s.db.memory_name]
|
|
if memory:
|
|
results = []
|
|
for state in memory:
|
|
results.append(await self._scan_on_write_connection(state))
|
|
await self._store(results)
|
|
states = [s for s in states if s not in memory]
|
|
if not states:
|
|
return
|
|
chunks = [states[i : i + SCAN_CHUNK] for i in range(0, len(states), SCAN_CHUNK)]
|
|
semaphore = asyncio.Semaphore(SCAN_CONCURRENCY)
|
|
|
|
async def one(chunk):
|
|
async with semaphore:
|
|
results = await self._off_loop(self._scan_chunk_sync, chunk)
|
|
await self._store(results)
|
|
|
|
if len(chunks) == 1:
|
|
await one(chunks[0])
|
|
else:
|
|
await asyncio.gather(*(one(c) for c in chunks))
|
|
|
|
async def _store(self, results):
|
|
entries = []
|
|
missing = []
|
|
stored = []
|
|
for r in results:
|
|
state = r["state"]
|
|
if state.removed:
|
|
continue
|
|
if "error" in r:
|
|
if state.db._closed:
|
|
# Closed (or deleted) while the scan was running
|
|
continue
|
|
state.error = r["error"]
|
|
state.needs_scan = True
|
|
logger.warning(
|
|
"Could not read schema of database %r: %s", state.name, r["error"]
|
|
)
|
|
continue
|
|
state.error = None
|
|
if r.get("missing"):
|
|
missing.append(state)
|
|
state.fp = r["fp"]
|
|
state.fp_racy = True
|
|
continue
|
|
db = state.db
|
|
fp_json = _fp_to_json(r["fp"], r["t"]) if r["fp"] is not None else None
|
|
entries.append(
|
|
(
|
|
state.name,
|
|
str(db.path) if db.path is not None else None,
|
|
db.is_memory,
|
|
r["version"],
|
|
fp_json,
|
|
r["schema"],
|
|
)
|
|
)
|
|
stored.append(r)
|
|
if entries:
|
|
watcher = self
|
|
|
|
def _write(conn):
|
|
# Skip databases removed while we were scanning
|
|
live = [e for e in entries if e[0] in watcher.states]
|
|
write_catalog_entries(conn, live)
|
|
|
|
await self.ds.get_internal_database().execute_write_fn(_write)
|
|
for r in stored:
|
|
state = r["state"]
|
|
state.catalog_version = r["version"]
|
|
state.missing = False
|
|
if r["fp"] is not None:
|
|
state.fp = r["fp"]
|
|
state.fp_racy = is_racy(r["fp"], r["t"])
|
|
if r["fp"][0] is not None:
|
|
state.db._file_seen = True
|
|
state.stats["scans"] += 1
|
|
self.counters["scans"] += 1
|
|
if missing:
|
|
for state in missing:
|
|
if not state.missing:
|
|
state.missing = True
|
|
state.db._invalidate_connections()
|
|
await self._clear_missing(missing)
|
|
|
|
async def _clear_missing(self, states):
|
|
names = [s.name for s in states]
|
|
|
|
def _clear(conn):
|
|
for name in names:
|
|
for table in CATALOG_TABLES[:-1]:
|
|
conn.execute(f"DELETE FROM {table} WHERE database_name = ?", [name])
|
|
conn.execute(
|
|
"UPDATE catalog_databases SET schema_version = NULL, fingerprint = NULL "
|
|
"WHERE database_name = ?",
|
|
[name],
|
|
)
|
|
|
|
await self.ds.get_internal_database().execute_write_fn(_clear)
|
|
for state in states:
|
|
state.catalog_version = None
|
|
|
|
async def _delete_catalog(self, names):
|
|
watcher = self
|
|
|
|
def _delete(conn):
|
|
for name in names:
|
|
if name in watcher.states:
|
|
# Re-added under the same name; its own scan replaces rows
|
|
continue
|
|
for table in CATALOG_TABLES:
|
|
conn.execute(f"DELETE FROM {table} WHERE database_name = ?", [name])
|
|
|
|
await self.ds.get_internal_database().execute_write_fn(_delete)
|
|
for name in names:
|
|
self._pending_removals.discard(name)
|
|
|
|
# ------------------------------------------------------------------
|
|
# startup
|
|
# ------------------------------------------------------------------
|
|
async def initial_scan(self):
|
|
"""Full catalog build at startup.
|
|
|
|
Deletes catalog rows for databases that are not attached (a
|
|
persistent --internal database can carry rows from a previous run),
|
|
then scans every database - except those whose persisted fingerprint
|
|
proves the file is unchanged since the catalog was written.
|
|
"""
|
|
internal = self.ds.get_internal_database()
|
|
names = list(self.states.keys())
|
|
names_json = json.dumps(names)
|
|
|
|
def _delete_stale(conn):
|
|
for table in CATALOG_TABLES:
|
|
conn.execute(
|
|
f"DELETE FROM {table} WHERE database_name IS NULL OR database_name NOT IN "
|
|
"(SELECT value FROM json_each(?))",
|
|
[names_json],
|
|
)
|
|
|
|
await internal.execute_write_fn(_delete_stale)
|
|
self._pending_removals.clear()
|
|
states = [s for s in self.states.values() if not s.removed]
|
|
persisted = {}
|
|
if not internal.is_temp_disk:
|
|
rows = await internal.execute(
|
|
"select database_name, path, schema_version, fingerprint from catalog_databases"
|
|
)
|
|
persisted = {row["database_name"]: row for row in rows.rows}
|
|
to_scan = []
|
|
if persisted:
|
|
reuse = await self._off_loop(self._reusable_sync, states, persisted)
|
|
for state in states:
|
|
hit = reuse.get(state.name)
|
|
if hit is None:
|
|
to_scan.append(state)
|
|
continue
|
|
fp, t_ns, version = hit
|
|
state.db._file_seen = True
|
|
state.fp = fp
|
|
state.fp_racy = is_racy(fp, t_ns)
|
|
state.catalog_version = version
|
|
state.needs_scan = False
|
|
self.counters["restored_from_persisted"] += 1
|
|
else:
|
|
to_scan = states
|
|
for state in to_scan:
|
|
state.needs_scan = True
|
|
if to_scan:
|
|
await self.refresh(to_scan)
|
|
self._pending.clear()
|
|
|
|
def _reusable_sync(self, states, persisted):
|
|
"""Databases whose stored fingerprint matches the file right now and
|
|
was not racy when it was stored. Stat calls only - no connections."""
|
|
out = {}
|
|
for state in states:
|
|
row = persisted.get(state.name)
|
|
if row is None or not state.is_file or row["fingerprint"] is None:
|
|
continue
|
|
if row["path"] != str(state.db.path) or row["schema_version"] is None:
|
|
continue
|
|
stored_fp, stored_t, closed = _fp_from_json(row["fingerprint"])
|
|
if stored_fp is None or (is_racy(stored_fp, stored_t) and not closed):
|
|
continue
|
|
t_ns = time.time_ns()
|
|
fp = fingerprint(state.db.path)
|
|
if fp == stored_fp:
|
|
out[state.name] = (fp, t_ns, row["schema_version"])
|
|
return out
|
|
|
|
# ------------------------------------------------------------------
|
|
# shutdown
|
|
# ------------------------------------------------------------------
|
|
def closing_fingerprints(self):
|
|
"""Called by Datasette.close() once every attached database has been
|
|
closed. For each owned file database whose catalog is current,
|
|
returns ``(name, schema_version, fingerprint_json)`` for the file as
|
|
it is now. Owned databases are only changed through Datasette, so a
|
|
restart with a persistent internal database can reuse their catalog
|
|
rows without opening them - even if data was written (or the -wal
|
|
checkpointed by the final close) after the catalog was last built.
|
|
Stat calls only."""
|
|
out = []
|
|
for state in list(self.states.values()):
|
|
if (
|
|
state.mode != "owned"
|
|
or not state.is_file
|
|
or state.removed
|
|
or state.missing
|
|
or state.needs_scan
|
|
or state.scan_future is not None
|
|
or state.error is not None
|
|
or state.catalog_version is None
|
|
or state.notified_version not in (None, state.catalog_version)
|
|
):
|
|
continue
|
|
t_ns = time.time_ns()
|
|
try:
|
|
fp = fingerprint(state.db.path)
|
|
except OSError:
|
|
continue
|
|
if fp[0] is None:
|
|
continue
|
|
out.append(
|
|
(state.name, state.catalog_version, _fp_to_json(fp, t_ns, closed=True))
|
|
)
|
|
return out
|
|
|
|
# ------------------------------------------------------------------
|
|
# introspection
|
|
# ------------------------------------------------------------------
|
|
def status(self):
|
|
modes = {}
|
|
for state in self.states.values():
|
|
modes[state.mode] = modes.get(state.mode, 0) + 1
|
|
return {
|
|
"interval_s": self.interval_s,
|
|
"loop_running": self.loop_running(),
|
|
"modes": modes,
|
|
"pending": len(self._pending),
|
|
"pending_removals": len(self._pending_removals),
|
|
**self.counters,
|
|
}
|