mirror of
https://github.com/simonw/sqlite-utils.git
synced 2026-07-22 17:04:31 +02:00
db.execute() write statements now commit automatically
Raw write statements previously opened an implicit transaction that stayed open until something committed it - the write was visible on the same connection, making it look saved, but was silently rolled back when the connection closed. execute() now commits any implicit transaction it opens, so raw writes behave like every other write in the library: committed as soon as they run, unless an explicit transaction (db.begin() or db.atomic()) is open, in which case they join it. Row-returning writes such as INSERT ... RETURNING via db.query() commit once their rows have been iterated. atomic(), commit() and rollback() now issue literal COMMIT/ROLLBACK statements, which have identical semantics in every sqlite3 connection mode. The CLI bulk command uses db.atomic() instead of 'with db.conn:'. This simplifies the transaction contract to: everything commits immediately unless you explicitly opened a transaction. Docs updated throughout; changelog and upgrading guide cover the breaking change for code that relied on rolling back uncommitted execute() writes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UnLnhsH25Nnv7LHhekUfPd
This commit is contained in:
Claude
parent
bcd9a26560
commit
f7ff3e2027
7 changed files with 114 additions and 33 deletions
|
|
@ -11,6 +11,7 @@ Unreleased
|
|||
|
||||
Breaking changes:
|
||||
|
||||
- Write statements executed with ``db.execute()`` are now committed automatically, unless a transaction is already open in which case they join it. Previously they opened an implicit transaction that stayed open until something committed it - writes appeared to work when read on the same connection but were silently rolled back when the connection closed. Code that relied on rolling back uncommitted ``db.execute()`` writes should use the new ``db.begin()`` method to open an explicit transaction first. The transaction model is documented in full at :ref:`python_api_transactions`.
|
||||
- ``db.query()`` now executes its SQL as soon as it is called, rather than waiting until the returned generator is first iterated. Rows are still fetched lazily during iteration. SQL errors are now raised at the call site, statements such as ``INSERT ... RETURNING`` take effect without needing to iterate over their results, and passing a statement that returns no rows - previously a silent no-op - now raises a ``ValueError`` recommending ``db.execute()`` instead.
|
||||
- Python API validation errors now raise ``ValueError`` instead of ``AssertionError``. Previously invalid arguments - such as ``create_table()`` with no columns, ``transform()`` on a table that does not exist, or passing both ``ignore=True`` and ``replace=True`` - were rejected using bare ``assert`` statements, which are silently skipped when Python runs with the ``-O`` flag. Code that caught ``AssertionError`` for these cases should catch ``ValueError`` instead.
|
||||
- ``table.upsert()`` and ``table.upsert_all()`` now raise ``PrimaryKeyRequired`` if a record is missing a value for any primary key column, or has a value of ``None`` for one. Previously such records - which can never match an existing row - were quietly inserted as brand new rows, or triggered a confusing ``KeyError`` after the insert had already taken place.
|
||||
|
|
|
|||
|
|
@ -152,14 +152,13 @@ Exiting the block is equivalent to calling ``db.close()``: the connection is
|
|||
closed and any transaction still open at that point is rolled back. This
|
||||
matches SQLite's own behavior when a connection closes.
|
||||
|
||||
This rarely matters in practice. Every method in this library that writes to
|
||||
the database commits its own changes before returning, so a transaction can
|
||||
only be left open here if you opened it yourself - through raw ``db.execute()``
|
||||
writes or an explicit ``BEGIN``. In that case the decision to commit stays
|
||||
with you: committing automatically on exit could silently persist
|
||||
half-finished work, for example if your code returned early from the block.
|
||||
Commit explicitly with ``db.commit()``, or wrap your writes in
|
||||
``db.atomic()`` which commits for you.
|
||||
This rarely matters in practice. Everything that writes to the database -
|
||||
including raw ``db.execute()`` statements - commits automatically, so a
|
||||
transaction can only be open here if you explicitly started one with
|
||||
``db.begin()`` and have not yet committed it. In that case the decision to
|
||||
commit stays with you: committing automatically on exit could silently
|
||||
persist half-finished work, for example if your code returned early from the
|
||||
block. Call ``db.commit()`` when the work is complete.
|
||||
|
||||
Note this differs from the ``sqlite3.Connection`` context manager in the
|
||||
standard library, which commits on success but does not close the connection.
|
||||
|
|
@ -264,8 +263,8 @@ The ``db.execute()`` and ``db.executescript()`` methods provide wrappers around
|
|||
|
||||
Other cursor methods such as ``.fetchone()`` and ``.fetchall()`` are also available, see the `standard library documentation <https://docs.python.org/3/library/sqlite3.html#sqlite3.Cursor>`__.
|
||||
|
||||
.. warning::
|
||||
Unlike the table methods described elsewhere in this documentation, ``db.execute()`` does **not** commit. If you use it to modify the database you are responsible for committing the change yourself - see :ref:`python_api_transactions_execute`.
|
||||
.. note::
|
||||
Write statements executed this way are committed automatically, unless a transaction is already open in which case they become part of it - see :ref:`python_api_transactions_execute`.
|
||||
|
||||
.. _python_api_parameters:
|
||||
|
||||
|
|
@ -308,13 +307,12 @@ Every method in this library that writes to the database - ``insert()``, ``upser
|
|||
db.table("news").insert({"headline": "Dog wins award"})
|
||||
# The new row is already saved - no commit() required
|
||||
|
||||
You never need to call ``commit()`` after using these methods, and you do not need to close the database to persist your changes.
|
||||
The same applies to raw SQL executed with :ref:`db.execute() <python_api_transactions_execute>` - a write statement is committed as soon as it has run.
|
||||
|
||||
There are exactly three situations where you need to think about transactions:
|
||||
You never need to call ``commit()``, and you do not need to close the database to persist your changes. There are exactly two situations where you need to think about transactions:
|
||||
|
||||
1. You want to group several write operations together, so they either all succeed or all fail - use :ref:`db.atomic() <python_api_atomic>`.
|
||||
2. You are executing raw SQL writes with ``db.execute()``, which does *not* commit automatically - see :ref:`python_api_transactions_execute`.
|
||||
3. You are :ref:`managing a transaction yourself <python_api_transactions_manual>`, in which case the library will never commit it for you.
|
||||
2. You are :ref:`managing a transaction yourself <python_api_transactions_manual>` with ``db.begin()``, in which case nothing is committed until you commit - the library will never commit a transaction you opened.
|
||||
|
||||
.. _python_api_atomic:
|
||||
|
||||
|
|
@ -352,21 +350,23 @@ The transaction is opened with a deferred ``BEGIN`` - SQLite takes the necessary
|
|||
Raw SQL writes with db.execute()
|
||||
--------------------------------
|
||||
|
||||
:ref:`db.execute() <python_api_execute>` is a thin wrapper around the underlying ``sqlite3`` connection, and it follows that connection's rules rather than this library's: a write statement opens a transaction that stays open until something commits it. Commit explicitly with ``db.commit()``:
|
||||
Write statements executed with :ref:`db.execute() <python_api_execute>` follow the same rule as everything else: they are committed automatically as soon as they have run.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
db.execute("insert into news (headline) values (?)", ["Dog wins award"])
|
||||
db.commit()
|
||||
# Already committed
|
||||
|
||||
Or wrap the calls in ``db.atomic()``, which commits for you:
|
||||
If a transaction is open - because the call happens inside a ``db.atomic()`` block, or after ``db.begin()`` - the statement becomes part of that transaction instead, and commits when the transaction commits:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
with db.atomic():
|
||||
db.execute("insert into news (headline) values (?)", ["Dog wins award"])
|
||||
db.execute("insert into news (headline) values (?)", ["Cat unimpressed"])
|
||||
# Both rows committed together
|
||||
|
||||
If you do neither, the write can be deceptive: reads on the same connection will see the new row, making it look saved, but the open transaction will be rolled back when the connection closes and the row will be gone.
|
||||
One corner case: a row-returning write such as ``INSERT ... RETURNING`` executed through ``db.execute()`` cannot be auto-committed, because its rows have not been read yet - call ``db.commit()`` after fetching them, or use :ref:`db.query() <python_api_query>` for those statements, which commits automatically once you have iterated over the results.
|
||||
|
||||
.. _python_api_transactions_manual:
|
||||
|
||||
|
|
@ -384,7 +384,7 @@ You can take full manual control using the ``db.begin()``, ``db.commit()`` and `
|
|||
else:
|
||||
db.rollback()
|
||||
|
||||
``db.begin()`` raises ``sqlite3.OperationalError`` if a transaction is already open. ``db.commit()`` and ``db.rollback()`` do nothing if there is no open transaction. A raw write executed with ``db.execute()`` (as above) opens a transaction implicitly, without needing ``db.begin()``.
|
||||
``db.begin()`` raises ``sqlite3.OperationalError`` if a transaction is already open. ``db.commit()`` and ``db.rollback()`` do nothing if there is no open transaction.
|
||||
|
||||
The library will never commit a transaction you opened. If you call write methods such as ``insert()`` - or use ``db.atomic()`` - while your transaction is open, they participate in it using SQLite savepoints instead of committing: exiting an ``atomic()`` block releases its savepoint, but nothing is saved to disk until you commit the outer transaction yourself. If you roll back, their changes are rolled back too.
|
||||
|
||||
|
|
|
|||
|
|
@ -76,9 +76,10 @@ Python API changes
|
|||
|
||||
**Transaction behavior is now well-defined.** 4.0 introduces the :ref:`db.atomic() <python_api_atomic>` context manager and uses it consistently for every write operation - the full model is described in :ref:`python_api_transactions`. Changes you may notice:
|
||||
|
||||
- Write statements executed with raw ``db.execute()`` calls now commit automatically, unless a transaction is already open in which case they join it. Previously they opened an implicit transaction that nothing committed - if your code used ``db.execute()`` for writes and relied on ``db.conn.rollback()`` to undo them, open an explicit transaction with the new ``db.begin()`` method first.
|
||||
- Multi-step operations such as ``table.transform()`` no longer commit an existing transaction you have open - they use savepoints inside it instead.
|
||||
- ``db.enable_wal()`` and ``db.disable_wal()`` raise a ``sqlite_utils.db.TransactionError`` if called while a transaction is open, instead of silently committing it.
|
||||
- Using ``Database`` as a context manager (``with Database(path) as db:``) closes the connection on exit *without* committing - uncommitted changes from raw ``db.execute()`` calls are rolled back.
|
||||
- Using ``Database`` as a context manager (``with Database(path) as db:``) closes the connection on exit *without* committing - a transaction you explicitly opened with ``db.begin()`` and did not commit is rolled back.
|
||||
- ``Database()`` rejects connections created with the Python 3.12+ ``sqlite3.connect(..., autocommit=True)`` or ``autocommit=False`` options, raising ``sqlite_utils.db.TransactionError``. On those connections every write the library made was silently discarded when the connection closed.
|
||||
|
||||
Packaging changes
|
||||
|
|
|
|||
|
|
@ -1140,7 +1140,7 @@ def insert_upsert_implementation(
|
|||
else:
|
||||
doc_chunks = [docs]
|
||||
for doc_chunk in doc_chunks:
|
||||
with db.conn:
|
||||
with db.atomic():
|
||||
db.conn.cursor().executemany(bulk_sql, doc_chunk)
|
||||
return
|
||||
|
||||
|
|
|
|||
|
|
@ -324,6 +324,16 @@ CREATE TABLE IF NOT EXISTS "{}"(
|
|||
""".strip()
|
||||
|
||||
|
||||
_TRANSACTION_CONTROL_PREFIXES = (
|
||||
"BEGIN",
|
||||
"COMMIT",
|
||||
"END",
|
||||
"ROLLBACK",
|
||||
"SAVEPOINT",
|
||||
"RELEASE",
|
||||
)
|
||||
|
||||
|
||||
class Database:
|
||||
"""
|
||||
Wrapper for a SQLite database connection that adds a variety of useful utility methods.
|
||||
|
|
@ -462,13 +472,13 @@ class Database:
|
|||
try:
|
||||
yield self
|
||||
except BaseException:
|
||||
self.conn.rollback()
|
||||
self.conn.execute("ROLLBACK")
|
||||
raise
|
||||
else:
|
||||
try:
|
||||
self.conn.commit()
|
||||
self.conn.execute("COMMIT")
|
||||
except BaseException:
|
||||
self.conn.rollback()
|
||||
self.conn.execute("ROLLBACK")
|
||||
raise
|
||||
|
||||
def begin(self) -> None:
|
||||
|
|
@ -487,14 +497,16 @@ class Database:
|
|||
"""
|
||||
Commit the current transaction. Does nothing if no transaction is open.
|
||||
"""
|
||||
self.conn.commit()
|
||||
if self.conn.in_transaction:
|
||||
self.conn.execute("COMMIT")
|
||||
|
||||
def rollback(self) -> None:
|
||||
"""
|
||||
Roll back the current transaction, discarding its changes. Does nothing
|
||||
if no transaction is open.
|
||||
"""
|
||||
self.conn.rollback()
|
||||
if self.conn.in_transaction:
|
||||
self.conn.execute("ROLLBACK")
|
||||
|
||||
@contextlib.contextmanager
|
||||
def ensure_autocommit_off(self) -> Generator[None, None, None]:
|
||||
|
|
@ -648,6 +660,7 @@ class Database:
|
|||
:raises ValueError: if the SQL statement does not return rows - use
|
||||
:meth:`execute` for those statements instead
|
||||
"""
|
||||
was_in_transaction = self.conn.in_transaction
|
||||
cursor = self.execute(sql, params or tuple())
|
||||
if cursor.description is None:
|
||||
raise ValueError(
|
||||
|
|
@ -659,6 +672,11 @@ class Database:
|
|||
def rows() -> Generator[dict, None, None]:
|
||||
for row in cursor:
|
||||
yield dict(zip(keys, row))
|
||||
if not was_in_transaction and self.conn.in_transaction:
|
||||
# A row-returning write such as INSERT ... RETURNING opened
|
||||
# an implicit transaction - commit it now that its rows have
|
||||
# been consumed
|
||||
self.conn.execute("COMMIT")
|
||||
|
||||
return rows()
|
||||
|
||||
|
|
@ -668,16 +686,33 @@ class Database:
|
|||
"""
|
||||
Execute SQL query and return a ``sqlite3.Cursor``.
|
||||
|
||||
A write statement - ``INSERT``, ``UPDATE``, ``CREATE TABLE`` and so on -
|
||||
is committed automatically, unless a transaction is already open, in
|
||||
which case it becomes part of that transaction. See
|
||||
:ref:`python_api_transactions`.
|
||||
|
||||
:param sql: SQL query to execute
|
||||
:param parameters: Parameters to use in that query - an iterable for ``where id = ?``
|
||||
parameters, or a dictionary for ``where id = :id``
|
||||
"""
|
||||
if self._tracer:
|
||||
self._tracer(sql, parameters)
|
||||
was_in_transaction = self.conn.in_transaction
|
||||
if parameters is not None:
|
||||
return self.conn.execute(sql, parameters)
|
||||
cursor = self.conn.execute(sql, parameters)
|
||||
else:
|
||||
return self.conn.execute(sql)
|
||||
cursor = self.conn.execute(sql)
|
||||
if (
|
||||
not was_in_transaction
|
||||
and self.conn.in_transaction
|
||||
and cursor.description is None
|
||||
and not sql.lstrip().upper().startswith(_TRANSACTION_CONTROL_PREFIXES)
|
||||
):
|
||||
# The statement opened an implicit transaction - commit it, so
|
||||
# that execute() behaves consistently with the rest of the
|
||||
# library and identically across connection modes
|
||||
self.conn.execute("COMMIT")
|
||||
return cursor
|
||||
|
||||
def executescript(self, sql: str) -> sqlite3.Cursor:
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -181,13 +181,13 @@ def test_atomic_inside_manual_transaction_uses_savepoint(fresh_db):
|
|||
fresh_db["t"].insert({"id": 2}, pk="id")
|
||||
# Nothing is committed until the user's own transaction commits
|
||||
assert fresh_db.conn.in_transaction
|
||||
fresh_db.conn.rollback()
|
||||
fresh_db.rollback()
|
||||
assert [r["id"] for r in fresh_db["t"].rows] == [1]
|
||||
# And with a commit instead, the atomic block's writes persist
|
||||
fresh_db.execute("begin")
|
||||
with fresh_db.atomic():
|
||||
fresh_db["t"].insert({"id": 3}, pk="id")
|
||||
fresh_db.conn.commit()
|
||||
fresh_db.commit()
|
||||
assert [r["id"] for r in fresh_db["t"].rows] == [1, 3]
|
||||
|
||||
|
||||
|
|
@ -221,3 +221,44 @@ def test_commit_and_rollback_without_transaction_are_noops(fresh_db):
|
|||
fresh_db.commit()
|
||||
fresh_db.rollback()
|
||||
assert not fresh_db.conn.in_transaction
|
||||
|
||||
|
||||
def test_execute_write_commits_immediately(tmpdir):
|
||||
path = str(tmpdir / "test.db")
|
||||
db = Database(path)
|
||||
db["t"].insert({"id": 1}, pk="id")
|
||||
db.execute("insert into t (id) values (2)")
|
||||
# No implicit transaction is left open
|
||||
assert not db.conn.in_transaction
|
||||
# A completely separate connection sees the row straight away
|
||||
other = sqlite3.connect(path)
|
||||
assert other.execute("select count(*) from t").fetchone()[0] == 2
|
||||
other.close()
|
||||
db.close()
|
||||
|
||||
|
||||
def test_execute_write_respects_explicit_transaction(fresh_db):
|
||||
fresh_db["t"].insert({"id": 1}, pk="id")
|
||||
fresh_db.begin()
|
||||
fresh_db.execute("insert into t (id) values (2)")
|
||||
# Still inside the explicit transaction - not committed
|
||||
assert fresh_db.conn.in_transaction
|
||||
fresh_db.rollback()
|
||||
assert [r["id"] for r in fresh_db["t"].rows] == [1]
|
||||
|
||||
|
||||
def test_query_returning_commits_after_iteration(tmpdir):
|
||||
if sqlite3.sqlite_version_info < (3, 35, 0):
|
||||
import pytest as _pytest
|
||||
|
||||
_pytest.skip("RETURNING requires SQLite 3.35.0 or higher")
|
||||
path = str(tmpdir / "test.db")
|
||||
db = Database(path)
|
||||
db["t"].insert({"id": 1}, pk="id")
|
||||
rows = list(db.query("insert into t (id) values (2) returning id"))
|
||||
assert rows == [{"id": 2}]
|
||||
assert not db.conn.in_transaction
|
||||
other = sqlite3.connect(path)
|
||||
assert other.execute("select count(*) from t").fetchone()[0] == 2
|
||||
other.close()
|
||||
db.close()
|
||||
|
|
|
|||
|
|
@ -35,14 +35,17 @@ def test_database_context_manager(tmpdir):
|
|||
path = str(tmpdir / "test.db")
|
||||
with Database(path) as db:
|
||||
db["t"].insert({"id": 1})
|
||||
# A raw write left uncommitted on purpose:
|
||||
# Raw writes commit automatically too
|
||||
db.execute("insert into t (id) values (2)")
|
||||
# An explicitly opened transaction left uncommitted on purpose:
|
||||
db.begin()
|
||||
db.execute("insert into t (id) values (3)")
|
||||
# The connection is closed...
|
||||
with pytest.raises(sqlite3.ProgrammingError):
|
||||
db.execute("select 1")
|
||||
# ... and the uncommitted change was rolled back, not committed
|
||||
# ... and the open explicit transaction was rolled back, not committed
|
||||
db2 = Database(path)
|
||||
assert [r["id"] for r in db2["t"].rows] == [1]
|
||||
assert [r["id"] for r in db2["t"].rows] == [1, 2]
|
||||
db2.close()
|
||||
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue