remote/sqlite-utils
1
0
Fork 0
mirror of https://github.com/simonw/sqlite-utils.git synced 2026-07-23 01:14:31 +02:00
Code Issues Projects Releases Packages Wiki Activity

table.foreign_keys now handles compound foreign keys, refs #594

Browse source 
This is a breaking change, since the return type is now a dataclass
and not a namedtuple.
This commit is contained in:
Simon Willison 2026-07-05 13:59:25 -07:00
commit f10459cffb
5 changed files with 232 additions and 36 deletions
Download patch file Download diff file Expand all files Collapse all files

9
docs/changelog.rst
View file

@ -4,6 +4,15 @@
Changelog
===========
.. _v_unreleased:
Unreleased
----------
Breaking changes:
- ``table.foreign_keys`` now returns ``ForeignKey`` objects that are dataclasses rather than ``namedtuple`` instances, so they can no longer be unpacked or indexed as ``(table, column, other_table, other_column)`` tuples - access their fields by name instead. Compound (multi-column) foreign keys are now represented as a single ``ForeignKey`` with ``is_compound=True`` and populated ``columns``/``other_columns`` lists, where ``column`` and ``other_column`` are ``None``. Previously they were returned as one ``ForeignKey`` per column, misleadingly suggesting several independent foreign keys. See :ref:`upgrading_3_to_4` for details. (:issue:`594`)
.. _v4_0rc2:
4.0rc2 (2026-07-04)

32
docs/python-api.rst
View file

@ -2204,17 +2204,35 @@ Almost all SQLite tables have a ``rowid`` column, but a table with no explicitly
.foreign_keys
-------------
The ``.foreign_keys`` property returns any foreign key relationships for the table, as a list of ``ForeignKey(table, column, other_table, other_column)`` named tuples. It is not available on views.
The ``.foreign_keys`` property returns any foreign key relationships for the table, as a list of ``ForeignKey`` objects. It is not available on views.
Each ``ForeignKey`` has the following attributes:
``table``
The table the foreign key is defined on.
``column``
The column on this table, or ``None`` for a compound foreign key.
``other_table``
The table being referenced.
``other_column``
The referenced column, or ``None`` for a compound foreign key.
``columns``
A list of the columns on this table, always populated (a single-item list for single-column foreign keys).
``other_columns``
A list of the referenced columns.
``is_compound``
``True`` if this is a compound (multi-column) foreign key.
``ForeignKey`` was a ``namedtuple`` prior to sqlite-utils 4.0. It is now a dataclass and can no longer be unpacked or indexed as a tuple - access its fields by name instead. See :ref:`upgrading_3_to_4` for details.
::
>>> db.table("Street_Tree_List").foreign_keys
[ForeignKey(table='Street_Tree_List', column='qLegalStatus', other_table='qLegalStatus', other_column='id'),
ForeignKey(table='Street_Tree_List', column='qCareAssistant', other_table='qCareAssistant', other_column='id'),
ForeignKey(table='Street_Tree_List', column='qSiteInfo', other_table='qSiteInfo', other_column='id'),
ForeignKey(table='Street_Tree_List', column='qSpecies', other_table='qSpecies', other_column='id'),
ForeignKey(table='Street_Tree_List', column='qCaretaker', other_table='qCaretaker', other_column='id'),
ForeignKey(table='Street_Tree_List', column='PlantType', other_table='PlantType', other_column='id')]
[ForeignKey(table='Street_Tree_List', column='qLegalStatus', other_table='qLegalStatus', other_column='id', columns=['qLegalStatus'], other_columns=['id'], is_compound=False),
ForeignKey(table='Street_Tree_List', column='qCareAssistant', other_table='qCareAssistant', other_column='id', columns=['qCareAssistant'], other_columns=['id'], is_compound=False),
...]
Compound foreign keys - defined with ``FOREIGN KEY (col_a, col_b) REFERENCES other(col_a, col_b)`` - are returned as a single ``ForeignKey`` with ``is_compound=True``, ``column`` and ``other_column`` set to ``None``, and the participating columns available in the ``columns`` and ``other_columns`` lists.
.. _python_api_introspection_schema:

16
docs/upgrading.rst
View file

@ -79,6 +79,22 @@ Python API changes
**View.enable_fts() has been removed.** The ``View`` class previously had an ``enable_fts()`` method that existed only to raise ``NotImplementedError`` - full-text search is not supported for views. Calling it now raises ``AttributeError`` like any other missing method.
**ForeignKey is now a dataclass, not a namedtuple.** The ``ForeignKey`` objects returned by ``table.foreign_keys`` gained three new fields - ``columns``, ``other_columns`` and ``is_compound`` - so that compound (multi-column) foreign keys can be represented as a single object. To make room for those fields cleanly ``ForeignKey`` is now a dataclass rather than a ``namedtuple``, so it can no longer be unpacked or indexed as a tuple. Access its fields by name instead:
.. code-block:: python
# 3.x - tuple unpacking, no longer works:
for table, column, other_table, other_column in db["courses"].foreign_keys:
...
# 4.0 - access fields by name:
for fk in db["courses"].foreign_keys:
fk.table, fk.column, fk.other_table, fk.other_column
Attempting the old unpacking or ``fk[0]`` indexing now raises ``TypeError``, so any code using those patterns will fail loudly rather than silently misbehave.
Compound foreign keys - previously returned as one ``ForeignKey`` per column, misleadingly suggesting several independent single-column keys - are now returned as a single ``ForeignKey`` with ``is_compound=True``. For these the scalar ``column`` and ``other_column`` fields are ``None``; use the ``columns`` and ``other_columns`` lists instead. Single-column foreign keys are unaffected apart from the class change: ``column``/``other_column`` behave as before and ``columns``/``other_columns`` are single-item lists.
**Validation errors raise ValueError.** Invalid arguments to Python API methods - for example ``create_table()`` with no columns, or ``ignore=True`` together with ``replace=True`` - now raise ``ValueError``. They previously raised ``AssertionError`` from bare ``assert`` statements, which were silently skipped under ``python -O``.
**Transaction behavior is now well-defined.** 4.0 introduces the :ref:`db.atomic() <python_api_atomic>` context manager and uses it consistently for every write operation - the full model is described in :ref:`python_api_transactions`. Changes you may notice:

129
sqlite_utils/db.py
View file

@ -11,6 +11,7 @@ from .utils import (
)
import binascii
from collections import namedtuple
from dataclasses import dataclass, field
from collections.abc import Mapping
import contextlib
import datetime
@ -161,9 +162,43 @@ Summary information about a column, see :ref:`python_api_analyze_column`.
The ``N`` least common values as a list of ``(value, count)`` tuples, or ``None`` if the table is entirely distinct
or if the number of distinct values is less than N (since they will already have been returned in ``most_common``)
"""
ForeignKey = namedtuple(
"ForeignKey", ("table", "column", "other_table", "other_column")
)
@dataclass(order=True)
class ForeignKey:
"""
A foreign key defined on a table.
For single-column foreign keys ``column`` and ``other_column`` hold the
column names, and ``columns``/``other_columns`` are single-item lists.
For compound (multi-column) foreign keys ``column`` and ``other_column``
are ``None`` - use ``columns`` and ``other_columns`` instead, and check
``is_compound``.
Prior to sqlite-utils 4.0 this was a ``namedtuple`` and could be unpacked
or indexed as ``(table, column, other_table, other_column)``. It is now a
dataclass - access its fields by name instead.
"""
table: str
column: Optional[str]
other_table: str
other_column: Optional[str]
columns: List[str] = field(default_factory=list)
other_columns: List[str] = field(default_factory=list)
is_compound: bool = False
def __post_init__(self):
# Populate columns/other_columns for single-column foreign keys
if not self.columns:
self.columns = [self.column] if self.column is not None else []
if not self.other_columns:
self.other_columns = (
[self.other_column] if self.other_column is not None else []
)
Index = namedtuple("Index", ("seq", "name", "unique", "origin", "partial", "columns"))
XIndex = namedtuple("XIndex", ("name", "columns"))
XIndexColumn = namedtuple(
@ -1124,7 +1159,7 @@ class Database:
if not all(isinstance(fk, (tuple, list)) for fk in foreign_keys):
raise ValueError("foreign_keys= should be a list of tuples")
fks = []
for tuple_or_list in foreign_keys:
for tuple_or_list in cast(Iterable[Sequence[str]], foreign_keys):
if len(tuple_or_list) == 4:
if tuple_or_list[0] != name:
raise ValueError(
@ -1234,7 +1269,7 @@ class Database:
pk = hash_id
# Soundness check foreign_keys point to existing tables
for fk in foreign_keys:
if fk.other_table == name and columns.get(fk.other_column):
if fk.other_table == name and columns.get(cast(str, fk.other_column)):
continue
if fk.other_column != "rowid" and not any(
c for c in self[fk.other_table].columns if c.name == fk.other_column
@ -1269,7 +1304,7 @@ class Database:
foreign_keys_by_column[column_name].other_table
),
quote_identifier(
foreign_keys_by_column[column_name].other_column
cast(str, foreign_keys_by_column[column_name].other_column)
),
)
)
@ -1499,7 +1534,9 @@ class Database:
"""
# foreign_keys is a list of explicit 4-tuples
if not all(
len(fk) == 4 and isinstance(fk, (list, tuple)) for fk in foreign_keys
isinstance(fk, ForeignKey)
or (isinstance(fk, (list, tuple)) and len(fk) == 4)
for fk in foreign_keys
):
raise ValueError(
"foreign_keys must be a list of 4-tuples, "
@ -1509,7 +1546,16 @@ class Database:
foreign_keys_to_create = []
# Verify that all tables and columns exist
for table, column, other_table, other_column in foreign_keys:
for fk in foreign_keys:
if isinstance(fk, ForeignKey):
table, column, other_table, other_column = (
fk.table,
fk.column,
fk.other_table,
fk.other_column,
)
else:
table, column, other_table, other_column = fk
if not self.table(table).exists():
raise AlterError("No such table: {}".format(table))
table_obj = self.table(table)
@ -1555,8 +1601,11 @@ class Database:
i.columns[0] for i in table.indexes if len(i.columns) == 1
}
for fk in table.foreign_keys:
if fk.column not in existing_indexes:
table.create_index([fk.column], find_unique_name=True)
# Compound foreign keys expose their columns via fk.columns;
# single-column keys yield a one-item list
for column in fk.columns:
if column not in existing_indexes:
table.create_index([column], find_unique_name=True)
def vacuum(self) -> None:
"Run a SQLite ``VACUUM`` against the database."
@ -1907,21 +1956,40 @@ class Table(Queryable):
@property
def foreign_keys(self) -> List["ForeignKey"]:
"List of foreign keys defined on this table."
fks = []
"""
List of foreign keys defined on this table.
Compound (multi-column) foreign keys are returned as a single
``ForeignKey`` with ``is_compound=True`` and populated
``columns``/``other_columns`` lists.
"""
# PRAGMA foreign_key_list returns one row per column, grouped by "id"
# with "seq" giving the column order within a compound foreign key.
by_id: Dict[int, list] = {}
for row in self.db.execute(
"PRAGMA foreign_key_list({})".format(quote_identifier(self.name))
).fetchall():
if row is not None:
id, seq, table_name, from_, to_, on_update, on_delete, match = row
fks.append(
ForeignKey(
table=self.name,
column=from_,
other_table=table_name,
other_column=to_,
)
by_id.setdefault(id, []).append((seq, table_name, from_, to_))
fks = []
for id in sorted(by_id):
rows = sorted(by_id[id]) # order columns by seq
other_table = rows[0][1]
columns = [row[2] for row in rows]
other_columns = [row[3] for row in rows]
is_compound = len(rows) > 1
fks.append(
ForeignKey(
table=self.name,
column=None if is_compound else columns[0],
other_table=other_table,
other_column=None if is_compound else other_columns[0],
columns=columns,
other_columns=other_columns,
is_compound=is_compound,
)
)
return fks
@property
@ -2254,17 +2322,20 @@ class Table(Queryable):
else:
# Construct foreign_keys from current, plus add_foreign_keys, minus drop_foreign_keys
create_table_foreign_keys = []
for table, column, other_table, other_column in self.foreign_keys:
# Copy over old foreign keys, unless we are dropping them
if (drop_foreign_keys is None) or (column not in drop_foreign_keys):
create_table_foreign_keys.append(
ForeignKey(
table,
rename.get(column) or column,
other_table,
other_column,
for fk in self.foreign_keys:
# Expand compound foreign keys into per-column references;
# for single-column keys this iterates exactly once
for column, other_column in zip(fk.columns, fk.other_columns):
# Copy over old foreign keys, unless we are dropping them
if (drop_foreign_keys is None) or (column not in drop_foreign_keys):
create_table_foreign_keys.append(
ForeignKey(
fk.table,
rename.get(column) or column,
fk.other_table,
other_column,
)
)
)
# Add new foreign keys
if add_foreign_keys is not None:
for fk in self.db.resolve_foreign_keys(self.name, add_foreign_keys):

82
tests/test_foreign_keys.py Normal file
View file

@ -0,0 +1,82 @@
"""Tests for reading compound (multi-column) foreign keys - issue #594."""
import pytest
from sqlite_utils import Database
COMPOUND_SCHEMA = """
CREATE TABLE departments (
campus_name TEXT NOT NULL,
dept_code TEXT NOT NULL,
dept_name TEXT,
PRIMARY KEY (campus_name, dept_code)
);
CREATE TABLE courses (
course_code TEXT PRIMARY KEY,
course_name TEXT,
campus_name TEXT NOT NULL,
dept_code TEXT NOT NULL,
FOREIGN KEY (campus_name, dept_code)
REFERENCES departments(campus_name, dept_code)
);
"""
@pytest.fixture
def compound_db():
db = Database(memory=True)
db.executescript(COMPOUND_SCHEMA)
return db
def test_compound_foreign_key(compound_db):
fks = compound_db["courses"].foreign_keys
assert len(fks) == 1
fk = fks[0]
assert fk.is_compound is True
assert fk.table == "courses"
assert fk.other_table == "departments"
assert fk.columns == ["campus_name", "dept_code"]
assert fk.other_columns == ["campus_name", "dept_code"]
# Scalar column/other_column can't sensibly hold a compound key
assert fk.column is None
assert fk.other_column is None
def test_single_foreign_key_gets_columns_fields(fresh_db):
fresh_db["authors"].insert({"id": 1, "name": "Sally"}, pk="id")
fresh_db["books"].insert({"title": "Hedgehogs", "author_id": 1})
fresh_db["books"].add_foreign_key("author_id", "authors", "id")
fk = fresh_db["books"].foreign_keys[0]
assert fk.is_compound is False
assert fk.column == "author_id"
assert fk.other_column == "id"
assert fk.columns == ["author_id"]
assert fk.other_columns == ["id"]
def test_foreign_key_no_longer_unpacks_as_tuple(fresh_db):
# Clean break in 4.0: ForeignKey is a dataclass, not a namedtuple, so the
# old tuple unpacking and indexing patterns now fail hard.
fresh_db["authors"].insert({"id": 1, "name": "Sally"}, pk="id")
fresh_db["books"].insert({"title": "Hedgehogs", "author_id": 1})
fresh_db["books"].add_foreign_key("author_id", "authors", "id")
fk = fresh_db["books"].foreign_keys[0]
with pytest.raises(TypeError):
table, column, other_table, other_column = fk
with pytest.raises(TypeError):
fk[0]
def test_foreign_keys_are_sortable(fresh_db):
fresh_db["authors"].insert({"id": 1, "name": "Sally"}, pk="id")
fresh_db["categories"].insert({"id": 1, "name": "Wildlife"}, pk="id")
fresh_db["books"].insert({"title": "Hedgehogs", "author_id": 1, "category_id": 1})
fresh_db.add_foreign_keys(
[
("books", "author_id", "authors", "id"),
("books", "category_id", "categories", "id"),
]
)
fks = sorted(fresh_db["books"].foreign_keys)
assert fks[0].column == "author_id"
assert fks[1].column == "category_id"