Merge remote-tracking branch 'origin/main' into say-5/document-memory-attrs

# Conflicts:
#	tests/test_constructor.py

docs/python-api.rst

@@ -33,6 +33,8 @@ Here's how to create a new SQLite database file containing a new ``chickens`` ta
         "color": "black",
     }])
 
+The inserted rows are saved to the database file straight away - methods like ``insert_all()`` commit their own changes, so no ``commit()`` call is needed. See :ref:`python_api_transactions` for how this works.
+
 You can loop through those rows like this:
 
 .. code-block:: python
@@ -93,6 +95,8 @@ Instead of a file path you can pass in an existing SQLite connection:
 
     db = Database(sqlite3.connect("my_database.db"))
 
+The connection must use Python's default transaction handling. Connections created with the Python 3.12+ ``sqlite3.connect(..., autocommit=True)`` or ``autocommit=False`` options are rejected with a ``sqlite_utils.db.TransactionError`` - see :ref:`python_api_transactions_modes`.
+
 If you want to create an in-memory database, you can do so like this:
 
 .. code-block:: python
@@ -152,6 +156,12 @@ The ``Database`` object also works as a context manager, which will automaticall
         db["my_table"].insert({"name": "Example"})
     # Connection is automatically closed here
 
+Exiting the block is equivalent to calling ``db.close()``: the connection is closed and any transaction still open at that point is rolled back. This matches SQLite's own behavior when a connection closes.
+
+This rarely matters in practice. Everything that writes to the database - including raw ``db.execute()`` statements - commits automatically, so a transaction can only be open here if you explicitly started one with ``db.begin()`` and have not yet committed it. In that case the decision to commit stays with you: committing automatically on exit could silently persist half-finished work, for example if your code returned early from the block. Call ``db.commit()`` when the work is complete.
+
+Note this differs from the ``sqlite3.Connection`` context manager in the standard library, which commits on success but does not close the connection. See :ref:`python_api_transactions` for the full transaction model.
+
 .. _python_api_attach:
 
 Attaching additional databases
@@ -227,6 +237,23 @@ The ``db.query(sql)`` function executes a SQL query and returns an iterator over
     # {'name': 'Cleo'}
     # {'name': 'Pancakes'}
 
+The SQL query is executed as soon as ``db.query()`` is called. The resulting rows are fetched lazily as you iterate, so large result sets are not loaded into memory all at once. Because execution is immediate, an error in your SQL will raise an exception straight away, and a statement such as ``INSERT ... RETURNING`` will take effect - and be committed, unless a transaction is open - even if you do not iterate over its results.
+
+``db.query()`` can only be used with SQL that returns rows. Passing a statement that returns no rows - an ``INSERT`` or ``UPDATE`` without a ``RETURNING`` clause, for example - will raise a ``ValueError``. The rejected statement is rolled back, so it has no effect on the database. Use :ref:`db.execute() <python_api_execute>` for those statements instead.
+
+There is one exception to the rolled-back guarantee: a ``PRAGMA`` statement that returns no rows, such as ``PRAGMA user_version = 5``, still raises a ``ValueError`` but will already have taken effect. Some PRAGMA statements refuse to run inside a transaction, so PRAGMAs are executed outside the savepoint that is used to roll back other rejected statements. Use ``db.execute()`` for PRAGMA statements that do not return rows.
+
+If a query returns more than one column with the same name - a join between two tables that share column names, for example - later occurrences are renamed with a numeric suffix, so every value is included in the dictionary:
+
+.. code-block:: python
+
+    row = next(db.query("select 1 as id, 2 as id, 3 as id"))
+    print(row)
+    # Outputs:
+    # {'id': 1, 'id_2': 2, 'id_3': 3}
+
+A suffix that would collide with another column in the query is skipped - ``select 1 as id, 2 as id, 3 as id_2`` returns ``{'id': 1, 'id_3': 2, 'id_2': 3}``. The same renaming is applied by ``table.rows_where()`` and ``table.search()``.
+
 .. _python_api_execute:
 
 db.execute(sql, params)
@@ -247,6 +274,9 @@ The ``db.execute()`` and ``db.executescript()`` methods provide wrappers around
 
 Other cursor methods such as ``.fetchone()`` and ``.fetchall()`` are also available, see the `standard library documentation <https://docs.python.org/3/library/sqlite3.html#sqlite3.Cursor>`__.
 
+.. note::
+    Write statements executed this way are committed automatically, unless a transaction is already open in which case they become part of it - see :ref:`python_api_transactions_execute`.
+
 .. _python_api_parameters:
 
 Passing parameters
@@ -275,6 +305,114 @@ Named parameters using ``:name`` can be filled using a dictionary:
 
 In this example ``next()`` is used to retrieve the first result in the iterator returned by the ``db.query()`` method.
 
+.. _python_api_transactions:
+
+Transactions and saving your changes
+====================================
+
+Every method in this library that writes to the database - ``insert()``, ``upsert()``, ``update()``, ``delete()``, ``delete_where()``, ``transform()``, ``create_table()``, ``create_index()``, ``enable_fts()`` and the rest - runs inside its own transaction and commits it before returning. Your changes are saved to disk as soon as the method call finishes:
+
+.. code-block:: python
+
+    db = Database("data.db")
+    db.table("news").insert({"headline": "Dog wins award"})
+    # The new row is already saved - no commit() required
+
+The same applies to raw SQL executed with :ref:`db.execute() <python_api_transactions_execute>` - a write statement is committed as soon as it has run.
+
+You never need to call ``commit()``, and you do not need to close the database to persist your changes. There are exactly two situations where you need to think about transactions:
+
+1. You want to group several write operations together, so they either all succeed or all fail - use :ref:`db.atomic() <python_api_atomic>`.
+2. You are :ref:`managing a transaction yourself <python_api_transactions_manual>` with ``db.begin()``, in which case nothing is committed until you commit - the library will never commit a transaction you opened.
+
+.. _python_api_atomic:
+
+Grouping changes with db.atomic()
+---------------------------------
+
+Use ``db.atomic()`` to group multiple operations in a single transaction:
+
+.. code-block:: python
+
+    with db.atomic():
+        db.table("dogs").insert({"id": 1, "name": "Cleo"}, pk="id")
+        db.table("dogs").insert({"id": 2, "name": "Pancakes"})
+
+The transaction commits when the block exits. If an exception is raised, changes made inside the block will be rolled back.
+
+``db.atomic()`` can be nested. Nested blocks use SQLite savepoints, so an exception in an inner block can roll back to that savepoint without rolling back the entire outer transaction:
+
+.. code-block:: python
+
+    with db.atomic():
+        db.table("dogs").insert({"id": 1, "name": "Cleo"}, pk="id")
+        try:
+            with db.atomic():
+                db.table("dogs").insert({"id": 2, "name": "Pancakes"})
+                raise ValueError("skip this one")
+        except ValueError:
+            pass
+        db.table("dogs").insert({"id": 3, "name": "Marnie"})
+
+The transaction is opened with a deferred ``BEGIN`` - SQLite takes the necessary locks when the first statement inside the block runs.
+
+.. _python_api_transactions_execute:
+
+Raw SQL writes with db.execute()
+--------------------------------
+
+Write statements executed with :ref:`db.execute() <python_api_execute>` follow the same rule as everything else: they are committed automatically as soon as they have run.
+
+.. code-block:: python
+
+    db.execute("insert into news (headline) values (?)", ["Dog wins award"])
+    # Already committed
+
+If a transaction is open - because the call happens inside a ``db.atomic()`` block, or after ``db.begin()`` - the statement becomes part of that transaction instead, and commits when the transaction commits:
+
+.. code-block:: python
+
+    with db.atomic():
+        db.execute("insert into news (headline) values (?)", ["Dog wins award"])
+        db.execute("insert into news (headline) values (?)", ["Cat unimpressed"])
+    # Both rows committed together
+
+One corner case: a row-returning write such as ``INSERT ... RETURNING`` executed through ``db.execute()`` cannot be auto-committed, because its rows have not been read yet - call ``db.commit()`` after fetching them, or use :ref:`db.query() <python_api_query>` for those statements, which executes the write and commits it immediately.
+
+.. _python_api_transactions_manual:
+
+Managing transactions yourself
+------------------------------
+
+You can take full manual control using the ``db.begin()``, ``db.commit()`` and ``db.rollback()`` methods:
+
+.. code-block:: python
+
+    db.begin()
+    db.table("news").insert({"headline": "Dog wins award"})
+    if all_looks_good:
+        db.commit()
+    else:
+        db.rollback()
+
+``db.begin()`` raises ``sqlite3.OperationalError`` if a transaction is already open. ``db.commit()`` and ``db.rollback()`` do nothing if there is no open transaction.
+
+The library will never commit a transaction you opened. If you call write methods such as ``insert()`` - or use ``db.atomic()`` - while your transaction is open, they participate in it using SQLite savepoints instead of committing: exiting an ``atomic()`` block releases its savepoint, but nothing is saved to disk until you commit the outer transaction yourself. If you roll back, their changes are rolled back too.
+
+Two related safeguards to be aware of:
+
+- ``db.enable_wal()`` and ``db.disable_wal()`` raise a ``sqlite_utils.db.TransactionError`` if called while a transaction is open, because changing the journal mode would commit it as a side effect.
+- Closing the database - explicitly with ``db.close()``, or by exiting a ``with Database(...) as db:`` block - rolls back any transaction that is still open, see :ref:`python_api_close`.
+
+.. _python_api_transactions_modes:
+
+Supported connection modes
+--------------------------
+
+``db.atomic()`` and the automatic per-method transactions require a connection in Python's default transaction handling mode. Passing a connection created with the Python 3.12+ ``sqlite3.connect(..., autocommit=True)`` or ``autocommit=False`` options to ``Database()`` raises a ``sqlite_utils.db.TransactionError``.
+
+This is because ``commit()`` and ``rollback()`` behave differently on those connections - under ``autocommit=True`` they are documented no-ops - which would cause every write made by this library to be silently discarded when the connection closed, rather than failing loudly.
+
 .. _python_api_table:
 
 Accessing tables
@@ -700,6 +838,61 @@ You can leave off the third item in the tuple to have the referenced column auto
         ("author_id", "authors")
     ])
 
+.. _python_api_compound_foreign_keys:
+
+Compound foreign keys
+~~~~~~~~~~~~~~~~~~~~~
+
+To create a compound (multi-column) foreign key, use tuples of column names in place of the single column names:
+
+.. code-block:: python
+
+    db.table("courses").create({
+        "course_code": str,
+        "campus_name": str,
+        "dept_code": str,
+    }, pk="course_code", foreign_keys=[
+        (("campus_name", "dept_code"), "departments", ("campus_name", "dept_code"))
+    ])
+
+This creates a table-level constraint:
+
+.. code-block:: sql
+
+    CREATE TABLE "courses" (
+       "course_code" TEXT PRIMARY KEY,
+       "campus_name" TEXT,
+       "dept_code" TEXT,
+       FOREIGN KEY ("campus_name", "dept_code") REFERENCES "departments"("campus_name", "dept_code")
+    )
+
+As with single columns, you can leave off the tuple of other columns to reference the compound primary key of the other table:
+
+.. code-block:: python
+
+    foreign_keys=[
+        (("campus_name", "dept_code"), "departments")
+    ]
+
+To specify ``ON DELETE`` or ``ON UPDATE`` actions, pass ``ForeignKey`` objects instead:
+
+.. code-block:: python
+
+    from sqlite_utils.db import ForeignKey
+
+    db.table("books").create({
+        "id": int,
+        "author_id": int,
+    }, pk="id", foreign_keys=[
+        ForeignKey(
+            table="books", column="author_id",
+            other_table="authors", other_column="id",
+            on_delete="CASCADE",
+        )
+    ])
+
+Foreign key actions are preserved by :ref:`table.transform() <python_api_transform>` - prior to sqlite-utils 4.0 they were silently dropped when a table was transformed.
+
 .. _python_api_table_configuration:
 
 Table configuration options
@@ -963,8 +1156,7 @@ You can delete all records in a table that match a specific WHERE statement usin
 
     >>> db = sqlite_utils.Database("dogs.db")
     >>> # Delete every dog with age less than 3
-    >>> with db.conn:
-    >>>     db.table("dogs").delete_where("age < ?", [3])
+    >>> db.table("dogs").delete_where("age < ?", [3])
 
 Calling ``table.delete_where()`` with no other arguments will delete every row in the table.
 
@@ -997,6 +1189,8 @@ Note that the ``pk`` and ``column_order`` parameters here are optional if you ar
 
 An ``upsert_all()`` method is also available, which behaves like ``insert_all()`` but performs upserts instead.
 
+Every record passed to ``upsert()`` or ``upsert_all()`` must include a value for each primary key column - a record without one could never match an existing row, so a ``sqlite_utils.db.PrimaryKeyRequired`` exception is raised instead of quietly inserting a new row.
+
 .. note::
     ``.upsert()`` and ``.upsert_all()`` in sqlite-utils 1.x worked like ``.insert(..., replace=True)`` and ``.insert_all(..., replace=True)`` do in 2.x. See `issue #66 <https://github.com/simonw/sqlite-utils/issues/66>`__ for details of this change.
 
@@ -1095,6 +1289,8 @@ To create a species record with a note on when it was first seen, you can use th
 
 The first time this is called the record will be created for ``name="Palm"``. Any subsequent calls with that name will ignore the second argument, even if it includes different values.
 
+``None`` values are matched correctly: calling ``.lookup()`` a second time with the same values will return the primary key of the existing row even if some of those values are ``None``.
+
 ``.lookup()`` also accepts keyword arguments, which are passed through to the :ref:`insert() method <python_api_creating_tables>` and can be used to influence the shape of the created table. Supported parameters are:
 
 - ``pk`` - which defaults to ``id``
@@ -1140,6 +1336,8 @@ To extract the ``species`` column out to a separate ``Species`` table, you can d
         "species": "Common Juniper"
     }, extracts={"species": "Species"})
 
+``None`` values are not extracted: no record is created for them in the lookup table and the column value stays ``null``.
+
 .. _python_api_m2m:
 
 Working with many-to-many relationships
@@ -1408,6 +1606,26 @@ To ignore the case where the key already exists, use ``ignore=True``:
 
     db.table("books").add_foreign_key("author_id", "authors", "id", ignore=True)
 
+To add a compound foreign key, pass tuples of columns:
+
+.. code-block:: python
+
+    db.table("courses").add_foreign_key(
+        ("campus_name", "dept_code"), "departments", ("campus_name", "dept_code")
+    )
+
+As with single columns, omitting the other columns will use the compound primary key of the other table. ``other_table`` must always be specified for a compound foreign key.
+
+Use ``on_delete=`` and ``on_update=`` to specify ``ON DELETE`` and ``ON UPDATE`` actions for the foreign key:
+
+.. code-block:: python
+
+    db.table("books").add_foreign_key(
+        "author_id", "authors", "id", on_delete="CASCADE"
+    )
+
+This creates a foreign key with an ``ON DELETE CASCADE`` clause, so deleting an author will also delete their books (provided foreign key enforcement is enabled with ``PRAGMA foreign_keys = ON``). Valid actions are ``"SET NULL"``, ``"SET DEFAULT"``, ``"CASCADE"``, ``"RESTRICT"`` and the default ``"NO ACTION"``.
+
 .. _python_api_add_foreign_keys:
 
 Adding multiple foreign key constraints at once
@@ -1426,6 +1644,8 @@ Here's an example adding two foreign keys at once:
 
 This method runs the same checks as ``.add_foreign_keys()`` and will raise ``sqlite_utils.db.AlterError`` if those checks fail.
 
+Foreign keys that already exist are silently skipped, so repeated calls are idempotent - but only if they match exactly. Requesting a foreign key that exists with different ``ON DELETE``/``ON UPDATE`` actions raises ``AlterError``: use ``table.transform()`` to change the actions of an existing foreign key.
+
 .. _python_api_index_foreign_keys:
 
 Adding indexes for all foreign keys
@@ -1437,6 +1657,8 @@ If you want to ensure that every foreign key column in your database has a corre
 
     db.index_foreign_keys()
 
+Compound foreign keys get a single composite index across their columns.
+
 .. _python_api_drop:
 
 Dropping a table or view
@@ -1639,6 +1861,16 @@ This example drops two foreign keys - the one from ``places.country`` to ``count
         drop_foreign_keys=("country", "continent")
     )
 
+A bare column name drops any foreign key that column participates in, including compound foreign keys. To target a compound foreign key precisely, pass a tuple of its columns:
+
+.. code-block:: python
+
+    db.table("courses").transform(
+        drop_foreign_keys=[("campus_name", "dept_code")]
+    )
+
+Renaming a column with ``rename=`` updates any foreign keys that use it, and dropping a column with ``drop=`` also drops any foreign keys it participates in - for a compound foreign key this removes the whole constraint.
+
 .. _python_api_transform_sql:
 
 Custom transformations with .transform_sql()
@@ -1806,6 +2038,8 @@ This produces a lookup table like so:
         "latin" TEXT
     )
 
+Rows where every extracted column is ``null`` are not extracted: no record is created for them in the lookup table and their foreign key column is left as ``null``. When extracting multiple columns, rows where at least one of the extracted columns has a value will be extracted as usual.
+
 .. _python_api_hash:
 
 Setting an ID based on the hash of the row contents
@@ -1979,7 +2213,7 @@ The ``db.iterdump()`` method returns a sequence of SQL strings representing a co
 
 This uses the `sqlite3.Connection.iterdump() <https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.iterdump>`__ method.
 
-If you are using ``pysqlite3`` or ``sqlean.py`` the underlying method may be missing. If you install the `sqlite-dump <https://pypi.org/project/sqlite-dump/>`__ package then the ``db.iterdump()`` method will use that implementation instead:
+If you are using ``pysqlite3`` the underlying method may be missing. If you install the `sqlite-dump <https://pypi.org/project/sqlite-dump/>`__ package then the ``db.iterdump()`` method will use that implementation instead:
 
 .. code-block:: bash
 
@@ -2086,17 +2320,39 @@ Almost all SQLite tables have a ``rowid`` column, but a table with no explicitly
 .foreign_keys
 -------------
 
-The ``.foreign_keys`` property returns any foreign key relationships for the table, as a list of ``ForeignKey(table, column, other_table, other_column)`` named tuples. It is not available on views.
+The ``.foreign_keys`` property returns any foreign key relationships for the table, as a list of ``ForeignKey`` objects. It is not available on views.
+
+Each ``ForeignKey`` has the following attributes:
+
+``table``
+    The table the foreign key is defined on.
+``column``
+    The column on this table, or ``None`` for a compound foreign key.
+``other_table``
+    The table being referenced.
+``other_column``
+    The referenced column, or ``None`` for a compound foreign key.
+``columns``
+    A tuple of the columns on this table, always populated (a one-item tuple for single-column foreign keys).
+``other_columns``
+    A tuple of the referenced columns.
+``is_compound``
+    ``True`` if this is a compound (multi-column) foreign key.
+``on_delete``
+    The ``ON DELETE`` action, e.g. ``"CASCADE"`` - ``"NO ACTION"`` if not set.
+``on_update``
+    The ``ON UPDATE`` action - ``"NO ACTION"`` if not set.
+
+``ForeignKey`` was a ``namedtuple`` prior to sqlite-utils 4.0. It is now a dataclass and can no longer be unpacked or indexed as a tuple - access its fields by name instead. See :ref:`upgrading_3_to_4` for details.
 
 ::
 
     >>> db.table("Street_Tree_List").foreign_keys
-    [ForeignKey(table='Street_Tree_List', column='qLegalStatus', other_table='qLegalStatus', other_column='id'),
-     ForeignKey(table='Street_Tree_List', column='qCareAssistant', other_table='qCareAssistant', other_column='id'),
-     ForeignKey(table='Street_Tree_List', column='qSiteInfo', other_table='qSiteInfo', other_column='id'),
-     ForeignKey(table='Street_Tree_List', column='qSpecies', other_table='qSpecies', other_column='id'),
-     ForeignKey(table='Street_Tree_List', column='qCaretaker', other_table='qCaretaker', other_column='id'),
-     ForeignKey(table='Street_Tree_List', column='PlantType', other_table='PlantType', other_column='id')]
+    [ForeignKey(table='Street_Tree_List', column='qLegalStatus', other_table='qLegalStatus', other_column='id', columns=('qLegalStatus',), other_columns=('id',), is_compound=False, on_delete='NO ACTION', on_update='NO ACTION'),
+     ForeignKey(table='Street_Tree_List', column='qCareAssistant', other_table='qCareAssistant', other_column='id', columns=('qCareAssistant',), other_columns=('id',), is_compound=False, on_delete='NO ACTION', on_update='NO ACTION'),
+     ...]
+
+Compound foreign keys - defined with ``FOREIGN KEY (col_a, col_b) REFERENCES other(col_a, col_b)`` - are returned as a single ``ForeignKey`` with ``is_compound=True``, ``column`` and ``other_column`` set to ``None``, and the participating columns available in the ``columns`` and ``other_columns`` tuples.
 
 .. _python_api_introspection_schema:
 
@@ -2676,6 +2932,8 @@ You can disable WAL mode using ``.disable_wal()``:
 
     Database("my_database.db").disable_wal()
 
+The journal mode can only be changed outside of a transaction. Calling either method while a transaction is open - inside a ``db.atomic()`` block, for example - raises a ``sqlite_utils.db.TransactionError``, unless the database is already in the requested mode in which case the call is a no-op.
+
 You can check the current journal mode for a database using the ``journal_mode`` property:
 
 .. code-block:: python