mirror of
https://github.com/simonw/datasette.git
synced 2026-10-29 06:37:01 +01:00
da523c30e5tests/connection_torture.py drives one Datasette from several event loops (threads) with the smallest pool, 1ms idle timeout and 10ms schema polling through every connection operation, plus add/remove_database, scratch databases, external schema changes, replaced/deleted files, cancellation, Database.close()/Datasette.close() mid-flight and non-threaded mode. One scenario per process, so crashes and hangs are results. It checks for hangs, crashes, undocumented exceptions, per-database write order, lost writes, read-your-writes, catalog freshness, and fds/threads/the instance released after close(). tests/test_connection_torture.py runs a short fixed-seed subset of every scenario. Fixes for what it found: - A cursor that left the thread owning its connection (the cursor returned by execute_write_many()/execute_write_script(), one returned by a write callback, one rejected by execute_fn(), or one held by an exception's frames) could be freed on the event loop thread while the connection ran the same SQL again; on Python < 3.12 freeing it resets the shared cached statement, and the other query failed with "bad parameter or other API misuse". Such cursors are now closed on the owning thread. Pre-existing on main (51 failures per 1,800 multi-loop execute_write_many calls). - invoke_startup() is single-flight across event loops (a concurrent.futures future, datasette.utils.inflight) instead of _startup_sequence() holding an asyncio.Lock: a second loop got "bound to a different event loop" or hung, and two loops could run the startup hooks twice. A request made from a startup hook no longer re-runs or deadlocks startup. - The index page takes one snapshot of ds.databases: a database removed, renamed or closed mid-request, or whose file was deleted, no longer turns the page into a 500. Database.size is 0 for a deleted file. The scan-coalescing helpers move to datasette/utils/inflight.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
69 lines
2.2 KiB
Python
69 lines
2.2 KiB
Python
"""
|
|
A fast, fixed-seed subset of the connection torture harness
|
|
(tests/connection_torture.py). Each scenario runs in its own process, so a
|
|
segfault or a hang fails one test instead of killing the test run.
|
|
|
|
Several event loops (threads) drive one Datasette with extreme settings
|
|
(smallest pool, 1ms idle timeout, 10ms schema polling) through reads,
|
|
writes of every kind, DDL, isolated functions, analyze_sql, cancellation,
|
|
add/remove_database, scratch databases, external schema changes, files
|
|
replaced and deleted, Database.close() and Datasette.close() mid-flight,
|
|
and non-threaded mode. See the harness docstring for what is checked.
|
|
"""
|
|
|
|
import json
|
|
import os
|
|
import signal
|
|
import subprocess
|
|
import sys
|
|
|
|
import pytest
|
|
|
|
HARNESS = os.path.join(os.path.dirname(__file__), "connection_torture.py")
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"scenario,seed",
|
|
[
|
|
("mixed", 101),
|
|
("lifecycle", 102),
|
|
("close", 103),
|
|
("cancel", 104),
|
|
("pool", 105),
|
|
("nothreads", 106),
|
|
("writes", 107),
|
|
],
|
|
)
|
|
def test_connection_torture(scenario, seed):
|
|
try:
|
|
proc = subprocess.run(
|
|
[
|
|
sys.executable,
|
|
HARNESS,
|
|
scenario,
|
|
"--seed",
|
|
str(seed),
|
|
"--duration",
|
|
"1.5",
|
|
],
|
|
capture_output=True,
|
|
text=True,
|
|
timeout=150,
|
|
check=False,
|
|
)
|
|
except subprocess.TimeoutExpired as e: # pragma: no cover
|
|
pytest.fail(f"{scenario} hung:\n{(e.stderr or '')[-4000:]}")
|
|
if proc.returncode < 0: # pragma: no cover
|
|
pytest.fail(
|
|
f"{scenario} crashed with {signal.Signals(-proc.returncode).name}:\n"
|
|
f"{proc.stderr[-4000:]}"
|
|
)
|
|
summary = json.loads(proc.stdout.strip().splitlines()[-1])
|
|
problems = "\n".join(
|
|
f"[{p['kind']}] {p['message'][:600]}" for p in summary["problems"]
|
|
)
|
|
assert proc.returncode == 0, f"{scenario} seed {seed}:\n{problems}"
|
|
assert summary["ok"]
|
|
# It did something (the slow-query scenarios do a few dozen operations
|
|
# in 1.5s on a busy machine, the others hundreds)
|
|
assert sum(summary["counts"].values()) > 10
|