Improve db.query() error handling and transaction safety

db.query() rejected statements roll back, RETURNING commits immediately.

Fixes two transaction bugs in db.query():

- A non-row-returning statement such as an UPDATE was executed and
  auto-committed by execute() before the ValueError was raised, so the
  write took effect despite being rejected. query() now runs the
  statement inside a savepoint and rolls it back before raising, so a
  rejected statement has no effect on the database - in every
  connection mode, including autocommit=True.

- INSERT ... RETURNING only committed once the returned generator was
  fully exhausted, so calling query() without iterating - or partially
  iterating with next() - left the transaction open and the write could
  be rolled back on close. The write is now completed and committed at
  call time, as the documentation already promised. Plain SELECTs are
  still fetched lazily.

Transaction control statements, VACUUM, ATTACH and DETACH never return
rows, so query() now rejects them without executing them. PRAGMA
statements skip the savepoint guard because some of them - such as
pragma journal_mode=wal - refuse to run inside a transaction.

Claude-Session: https://claude.ai/code/session_012U3iRfJoTZ5vd22cBSF2nJ
This commit is contained in:
Simon Willison 2026-07-04 17:40:48 -07:00 committed by GitHub
commit 0566a9f128
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 171 additions and 26 deletions

View file

@ -229,9 +229,9 @@ The ``db.query(sql)`` function executes a SQL query and returns an iterator over
# {'name': 'Cleo'}
# {'name': 'Pancakes'}
The SQL query is executed as soon as ``db.query()`` is called. The resulting rows are fetched lazily as you iterate, so large result sets are not loaded into memory all at once. Because execution is immediate, an error in your SQL will raise an exception straight away, and a statement such as ``INSERT ... RETURNING`` will take effect even if you do not iterate over its results.
The SQL query is executed as soon as ``db.query()`` is called. The resulting rows are fetched lazily as you iterate, so large result sets are not loaded into memory all at once. Because execution is immediate, an error in your SQL will raise an exception straight away, and a statement such as ``INSERT ... RETURNING`` will take effect - and be committed, unless a transaction is open - even if you do not iterate over its results.
``db.query()`` can only be used with SQL that returns rows. Passing a statement that returns no rows - an ``INSERT`` or ``UPDATE`` without a ``RETURNING`` clause, for example - will raise a ``ValueError``. Use :ref:`db.execute() <python_api_execute>` for those statements instead.
``db.query()`` can only be used with SQL that returns rows. Passing a statement that returns no rows - an ``INSERT`` or ``UPDATE`` without a ``RETURNING`` clause, for example - will raise a ``ValueError``. The rejected statement is rolled back, so it has no effect on the database. Use :ref:`db.execute() <python_api_execute>` for those statements instead.
.. _python_api_execute:
@ -356,7 +356,7 @@ If a transaction is open - because the call happens inside a ``db.atomic()`` blo
db.execute("insert into news (headline) values (?)", ["Cat unimpressed"])
# Both rows committed together
One corner case: a row-returning write such as ``INSERT ... RETURNING`` executed through ``db.execute()`` cannot be auto-committed, because its rows have not been read yet - call ``db.commit()`` after fetching them, or use :ref:`db.query() <python_api_query>` for those statements, which commits automatically once you have iterated over the results.
One corner case: a row-returning write such as ``INSERT ... RETURNING`` executed through ``db.execute()`` cannot be auto-committed, because its rows have not been read yet - call ``db.commit()`` after fetching them, or use :ref:`db.query() <python_api_query>` for those statements, which executes the write and commits it immediately.
.. _python_api_transactions_manual: