Torture harness, and the fixes it found
tests/connection_torture.py drives one Datasette from several event loops
(threads) with the smallest pool, 1ms idle timeout and 10ms schema polling
through every connection operation, plus add/remove_database, scratch
databases, external schema changes, replaced/deleted files, cancellation,
Database.close()/Datasette.close() mid-flight and non-threaded mode. One
scenario per process, so crashes and hangs are results. It checks for hangs,
crashes, undocumented exceptions, per-database write order, lost writes,
read-your-writes, catalog freshness, and fds/threads/the instance released
after close(). tests/test_connection_torture.py runs a short fixed-seed
subset of every scenario.
Fixes for what it found:
- A cursor that left the thread owning its connection (the cursor returned
by execute_write_many()/execute_write_script(), one returned by a write
callback, one rejected by execute_fn(), or one held by an exception's
frames) could be freed on the event loop thread while the connection ran
the same SQL again; on Python < 3.12 freeing it resets the shared cached
statement, and the other query failed with "bad parameter or other API
misuse". Such cursors are now closed on the owning thread. Pre-existing on
main (51 failures per 1,800 multi-loop execute_write_many calls).
- invoke_startup() is single-flight across event loops (a concurrent.futures
future, datasette.utils.inflight) instead of _startup_sequence() holding
an asyncio.Lock: a second loop got "bound to a different event loop" or
hung, and two loops could run the startup hooks twice. A request made
from a startup hook no longer re-runs or deadlocks startup.
- The index page takes one snapshot of ds.databases: a database removed,
renamed or closed mid-request, or whose file was deleted, no longer turns
the page into a 500. Database.size is 0 for a deleted file.
The scan-coalescing helpers move to datasette/utils/inflight.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
2026-10-04 09:07:48 +00:00
|
|
|
"""
|
|
|
|
|
A fast, fixed-seed subset of the connection torture harness
|
|
|
|
|
(tests/connection_torture.py). Each scenario runs in its own process, so a
|
|
|
|
|
segfault or a hang fails one test instead of killing the test run.
|
|
|
|
|
|
|
|
|
|
Several event loops (threads) drive one Datasette with extreme settings
|
|
|
|
|
(smallest pool, 1ms idle timeout, 10ms schema polling) through reads,
|
|
|
|
|
writes of every kind, DDL, isolated functions, analyze_sql, cancellation,
|
2026-10-07 12:25:44 -07:00
|
|
|
add/remove_database, external schema changes, files
|
Torture harness, and the fixes it found
tests/connection_torture.py drives one Datasette from several event loops
(threads) with the smallest pool, 1ms idle timeout and 10ms schema polling
through every connection operation, plus add/remove_database, scratch
databases, external schema changes, replaced/deleted files, cancellation,
Database.close()/Datasette.close() mid-flight and non-threaded mode. One
scenario per process, so crashes and hangs are results. It checks for hangs,
crashes, undocumented exceptions, per-database write order, lost writes,
read-your-writes, catalog freshness, and fds/threads/the instance released
after close(). tests/test_connection_torture.py runs a short fixed-seed
subset of every scenario.
Fixes for what it found:
- A cursor that left the thread owning its connection (the cursor returned
by execute_write_many()/execute_write_script(), one returned by a write
callback, one rejected by execute_fn(), or one held by an exception's
frames) could be freed on the event loop thread while the connection ran
the same SQL again; on Python < 3.12 freeing it resets the shared cached
statement, and the other query failed with "bad parameter or other API
misuse". Such cursors are now closed on the owning thread. Pre-existing on
main (51 failures per 1,800 multi-loop execute_write_many calls).
- invoke_startup() is single-flight across event loops (a concurrent.futures
future, datasette.utils.inflight) instead of _startup_sequence() holding
an asyncio.Lock: a second loop got "bound to a different event loop" or
hung, and two loops could run the startup hooks twice. A request made
from a startup hook no longer re-runs or deadlocks startup.
- The index page takes one snapshot of ds.databases: a database removed,
renamed or closed mid-request, or whose file was deleted, no longer turns
the page into a 500. Database.size is 0 for a deleted file.
The scan-coalescing helpers move to datasette/utils/inflight.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
2026-10-04 09:07:48 +00:00
|
|
|
replaced and deleted, Database.close() and Datasette.close() mid-flight,
|
|
|
|
|
and non-threaded mode. See the harness docstring for what is checked.
|
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
import json
|
|
|
|
|
import os
|
|
|
|
|
import signal
|
|
|
|
|
import subprocess
|
|
|
|
|
import sys
|
|
|
|
|
|
|
|
|
|
import pytest
|
|
|
|
|
|
|
|
|
|
HARNESS = os.path.join(os.path.dirname(__file__), "connection_torture.py")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
|
|
|
"scenario,seed",
|
|
|
|
|
[
|
|
|
|
|
("mixed", 101),
|
|
|
|
|
("lifecycle", 102),
|
|
|
|
|
("close", 103),
|
|
|
|
|
("cancel", 104),
|
|
|
|
|
("pool", 105),
|
|
|
|
|
("nothreads", 106),
|
|
|
|
|
("writes", 107),
|
|
|
|
|
],
|
|
|
|
|
)
|
|
|
|
|
def test_connection_torture(scenario, seed):
|
|
|
|
|
try:
|
|
|
|
|
proc = subprocess.run(
|
|
|
|
|
[
|
|
|
|
|
sys.executable,
|
|
|
|
|
HARNESS,
|
|
|
|
|
scenario,
|
|
|
|
|
"--seed",
|
|
|
|
|
str(seed),
|
|
|
|
|
"--duration",
|
|
|
|
|
"1.5",
|
|
|
|
|
],
|
|
|
|
|
capture_output=True,
|
|
|
|
|
text=True,
|
|
|
|
|
timeout=150,
|
|
|
|
|
check=False,
|
|
|
|
|
)
|
|
|
|
|
except subprocess.TimeoutExpired as e: # pragma: no cover
|
|
|
|
|
pytest.fail(f"{scenario} hung:\n{(e.stderr or '')[-4000:]}")
|
|
|
|
|
if proc.returncode < 0: # pragma: no cover
|
|
|
|
|
pytest.fail(
|
|
|
|
|
f"{scenario} crashed with {signal.Signals(-proc.returncode).name}:\n"
|
|
|
|
|
f"{proc.stderr[-4000:]}"
|
|
|
|
|
)
|
2026-10-06 15:08:20 -07:00
|
|
|
lines = proc.stdout.strip().splitlines()
|
|
|
|
|
if not lines:
|
|
|
|
|
pytest.fail(
|
|
|
|
|
f"{scenario} exited {proc.returncode} without a summary:\n"
|
|
|
|
|
f"{proc.stderr[-4000:]}"
|
|
|
|
|
)
|
|
|
|
|
summary = json.loads(lines[-1])
|
Torture harness, and the fixes it found
tests/connection_torture.py drives one Datasette from several event loops
(threads) with the smallest pool, 1ms idle timeout and 10ms schema polling
through every connection operation, plus add/remove_database, scratch
databases, external schema changes, replaced/deleted files, cancellation,
Database.close()/Datasette.close() mid-flight and non-threaded mode. One
scenario per process, so crashes and hangs are results. It checks for hangs,
crashes, undocumented exceptions, per-database write order, lost writes,
read-your-writes, catalog freshness, and fds/threads/the instance released
after close(). tests/test_connection_torture.py runs a short fixed-seed
subset of every scenario.
Fixes for what it found:
- A cursor that left the thread owning its connection (the cursor returned
by execute_write_many()/execute_write_script(), one returned by a write
callback, one rejected by execute_fn(), or one held by an exception's
frames) could be freed on the event loop thread while the connection ran
the same SQL again; on Python < 3.12 freeing it resets the shared cached
statement, and the other query failed with "bad parameter or other API
misuse". Such cursors are now closed on the owning thread. Pre-existing on
main (51 failures per 1,800 multi-loop execute_write_many calls).
- invoke_startup() is single-flight across event loops (a concurrent.futures
future, datasette.utils.inflight) instead of _startup_sequence() holding
an asyncio.Lock: a second loop got "bound to a different event loop" or
hung, and two loops could run the startup hooks twice. A request made
from a startup hook no longer re-runs or deadlocks startup.
- The index page takes one snapshot of ds.databases: a database removed,
renamed or closed mid-request, or whose file was deleted, no longer turns
the page into a 500. Database.size is 0 for a deleted file.
The scan-coalescing helpers move to datasette/utils/inflight.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwkLo2rJWidpkDVT3HkvNv
2026-10-04 09:07:48 +00:00
|
|
|
problems = "\n".join(
|
|
|
|
|
f"[{p['kind']}] {p['message'][:600]}" for p in summary["problems"]
|
|
|
|
|
)
|
|
|
|
|
assert proc.returncode == 0, f"{scenario} seed {seed}:\n{problems}"
|
|
|
|
|
assert summary["ok"]
|
|
|
|
|
# It did something (the slow-query scenarios do a few dozen operations
|
|
|
|
|
# in 1.5s on a busy machine, the others hundreds)
|
|
|
|
|
assert sum(summary["counts"].values()) > 10
|