Closes#781, #783
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900618685
* Fix rowid pk and last_rowid regressions in insert/upsert
Two behaviour regressions in the 4.0 insert/upsert rewrite broke callers
(notably Datasette's write API) that operate on tables without an explicit
primary key. Both are fixed here with regression tests.
1. rowid (and its aliases _rowid_/oid) were rejected as a primary key.
Table.pks already reports ["rowid"] for a rowid table, but the new pk
validation raised InvalidColumns because rowid is not listed among the
table's columns, and the insert success path then raised KeyError when
looking up the pk value. rowid aliases are now accepted for rowid tables
and resolve directly to the rowid.
2. An ignored insert (INSERT OR IGNORE that matched an existing row) no
longer populated last_rowid, and only set last_pk when an explicit pk=
was passed. It now locates the existing conflicting row by its primary
key values and reports that row's rowid and pk, rather than relying on
the connection's last inserted rowid.
Add a shared ROWID_ALIASES constant for the rowid alias names.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E7af8SxFZqiCerJB6MqKnY
A RAISE(ROLLBACK) trigger or INSERT OR ROLLBACK conflict rolls back the
entire transaction and destroys every savepoint. The cleanup paths in
atomic() and query() then raised OperationalError ("no such savepoint" /
"cannot rollback - no transaction is active"), masking the original
IntegrityError - breaking user code that catches sqlite3.IntegrityError.
Cleanup now checks conn.in_transaction first: if the error already
destroyed the transaction there is nothing left to undo, and the
original exception propagates.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
SQLite 3.23.1 rejects a UTF-8 byte order mark before the first token,
so the BOM variant of the execute()-prefixed-BEGIN test now skips when
the SQLite version does not accept a leading BOM. And versions before
3.36 allowed selecting rowid from a view, returning NULL, rather than
raising an error - the pks_and_rows_where() view test now accepts
either behavior.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The docs promise --list will not create the database file or the
_sqlite_migrations table, but legacy sqlite_migrate.Migrations classes
create the table (in the legacy schema) from their pending()/applied()
methods. The listing now runs inside a transaction that is rolled back,
keeping --list read-only regardless of what the migration class does.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
sqlite-utils insert db t - --pk badcol and sqlite-utils extract db t
nosuchcol dumped raw InvalidColumns tracebacks - the insert error
handling caught NoTable and OperationalError but not the InvalidColumns
introduced for #732, and the extract command had no handling at all
(including for NoTable when pointed at a view). Both now exit with
click-style Error: messages.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The lookup-table insert relied on INSERT OR IGNORE and the unique index
to dedupe against existing rows, but SQLite unique indexes treat NULLs
as distinct - extracting a second table into the same lookup table
re-inserted every NULL-containing value, growing orphan rows on each
extract. The insert now also has an IS-based NOT EXISTS guard, matching
how the foreign keys themselves are resolved.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The existing-foreign-key dedup compared only columns and other_table, so
requesting an FK that already exists with different ON DELETE/ON UPDATE
actions was a silent no-op - and with add_foreign_key() raising "already
exists", there was no signal that the requested actions were dropped.
An exact match (including actions) is still skipped for idempotency;
a mismatch now raises AlterError pointing at table.transform().
Also validates that compound foreign keys passed as 4-tuples have the
same number of columns on both sides - extra other-columns were being
silently discarded, e.g. ("t", ("a",), "other", ("a", "b")) created a
single-column key referencing just "a".
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Assigning conn.isolation_level commits any pending transaction as a side
effect, so entering the context manager with a transaction open silently
committed the caller's work and made a later rollback() a no-op. All
internal callers already ensure no transaction is open; the public API
now enforces it, matching enable_wal() and disable_wal().
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The CLI validated --stop-before names against both pending and applied
migrations, but Migrations.apply() only looked for the stop name among
pending ones - naming an applied migration passed validation and then
silently applied every migration after it, the exact outcome the option
exists to prevent. apply() now raises ValueError before applying
anything if a stop_before name matches an applied migration in its set;
names not in the set are still ignored, since unqualified CLI values are
offered to every set. The migrate command reports it as a clean error.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The InvalidColumns check added for #732 fired before alter=True had a
chance to add the missing pk column - a regression from 3.x, where
insert({"id": 5, "a": 2}, pk="id", alter=True) against a table without
an id column worked. With alter=True the check is now deferred until the
first batch of record keys is known: a pk column supplied by the records
passes, one found in neither the table nor the records still raises
InvalidColumns before anything is inserted. An empty record iterator
with alter=True returns without error, matching the 3.x no-op.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Type detection is the 4.0 default for CSV/TSV data, and the detected-type
transform ran even when the target table already existed - inserting a
CSV into a table with a TEXT zip column converted the column to INTEGER,
corrupting values with leading zeros ('01234' became 1234) with no
warning. Detected types now only apply to tables the command created.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The previous compound-pk-ordering commit switched pks_and_rows_where()
to Table-only properties, but the method is defined on Queryable and
views exposed it too - calling it on a View raised AttributeError.
Restored Queryable-safe logic, and stopped double-quoting the
synthesized rowid column: SQLite turns a double-quoted identifier that
does not resolve into a string literal, so on a view the generated
select "rowid" silently produced the string 'rowid' and a confusing
KeyError, where 3.x's [rowid] quoting raised OperationalError cleanly.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
resolve_foreign_keys() used all-or-nothing isinstance checks, so mixing
ForeignKey objects with tuples raised "foreign_keys= should be a list of
tuples" - a regression from 3.x, where ForeignKey was a namedtuple and
passed the tuple check. Each item is now normalized individually, which
also allows bare column-name strings in a mixed list.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The namedtuple-to-dataclass change made ForeignKey unhashable, breaking
set(table.foreign_keys) and dict-key usage that worked in 3.x. frozen=True
restores immutability and hashability. Equality and hashing cover all
compared fields including on_delete/on_update - two foreign keys differing
only in their actions are different constraints.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PRAGMA table_info sets is_pk to the 1-based position of each column
within the PRIMARY KEY, which can differ from table column order.
table.pks previously returned table column order, so an implicit
compound FOREIGN KEY ... REFERENCES other was introspected with its
referenced columns inverted, and transform() baked that inverted order
into the rewritten schema - failing with IntegrityError on valid data,
or silently reversing the constraint with foreign keys off.
table.pks, compound foreign key guessing (create, add_foreign_key) and
transform() now all use declaration order, and transform() no longer
reorders a compound PRIMARY KEY (b, a) into table column order.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
db.query() promises that a statement rejected with ValueError is rolled
back and has no effect. PRAGMA statements are the exception: some of them
refuse to run inside a transaction, so they execute outside the savepoint
guard - a row-less PRAGMA such as "PRAGMA user_version = 5" therefore
takes effect despite the ValueError. Documenting this as a known
limitation rather than fixing it, since a fix would need a hardcoded list
of row-returning PRAGMAs.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- table.extract() and the sqlite-utils extract command now skip rows
where every extracted column is null: the new foreign key column is
left null and no all-null record is added to the lookup table. Rows
with at least one non-null extracted column are extracted as before.
- The extracts= option to insert() and friends keeps None values as
null instead of creating a lookup record for them - previously each
insert batch added a duplicate null row to the lookup table.
- table.lookup() compares values using IS rather than = so lookup
values containing None match existing rows correctly, instead of
inserting a duplicate row on every call.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix transform() corrupting TRUE/FALSE/NULL column defaults
quote_default_value() passed keyword-literal defaults (TRUE, FALSE, NULL)
through self.quote(), wrapping them in quotes. So transform() rebuilt a
column declared "INTEGER DEFAULT TRUE" as "INTEGER DEFAULT 'TRUE'", and a
later default insert stored the text 'TRUE' instead of the integer 1
(likewise 'FALSE' for 0 and 'NULL' for null) - silent value corruption.
Return these keyword literals unquoted, as already done for the
CURRENT_TIME/DATE/TIMESTAMP literals.
PR #764
Column names passed to Python API methods are now resolved against the
table schema case-insensitively, mirroring how SQLite itself compares
identifiers (ASCII-only case folding). Fixes KeyError populating
last_pk from insert()/upsert(), silently ignored transform()
rename/drop/types options, duplicate column errors from
create_table(transform=True), redundant lookup() indexes, and
case-sensitive foreign key validation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Queries returning duplicate column names - e.g. joins between tables
sharing column names - silently lost values because rows were built
with dict(zip(keys, row)), where the last duplicate wins.
Later occurrences are now renamed with a numeric suffix: id, id
becomes id, id_2 - skipping any suffix that would collide with a
real column in the same query.
The new utils.dedupe_keys() helper transforms the key list once per
query, so the per-row dict construction is unchanged and there is no
measurable performance impact.
Applied in Database.query() (including the PRAGMA and RETURNING
paths), Table.rows_where(), Table.search() and the CLI's JSON output.
CSV, TSV and table output keep the original duplicate headers.
Closes#624
Fixes formatting left behind by afbfd95, which was merged while the
Black check on its pull request was still failing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
table.add_foreign_key() now accepts on_delete= and on_update= to
create foreign keys with ON DELETE/ON UPDATE actions:
table.add_foreign_key("author_id", "authors", "id", on_delete="CASCADE")
Works for compound foreign keys too.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test_create_table_compound_foreign_key_enforced caught the stdlib
sqlite3.IntegrityError, but sqlean's IntegrityError is not a subclass
of it. Import sqlite3 from sqlite_utils.utils like the other test
modules, so the right exception is used whichever backend is installed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ForeignKey objects passed to db.add_foreign_keys() were flattened to
plain (table, column, other_table, other_column) tuples before being
handed to transform(), silently discarding their on_delete and
on_update actions. The method now carries ForeignKey objects through
to transform() intact - tuple inputs are converted to ForeignKey
objects up front, which also simplifies the validation loop.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ForeignKey.columns and .other_columns are now tuples rather than lists,
both when introspected and when constructed directly (lists are
normalized in __post_init__). Tuples are now the documented form for
specifying compound foreign keys everywhere - in create_table()
foreign_keys=, add_foreign_key() and drop_foreign_keys=:
foreign_keys=[(("campus_name", "dept_code"), "departments")]
Lists continue to work as input but are no longer documented.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New ForeignKeyColumns type alias - a column name or a list/tuple of
column names - used by add_foreign_key(), add_foreign_keys() (via the
ForeignKeyTuple alias, replacing its Any slots) and the
ForeignKeyIndicator union, which it also simplifies. Tuples now
type-check anywhere lists are accepted.
ForeignKey.__post_init__ normalizes tuple columns/other_columns to
lists so direct construction with tuples compares equal to
introspected foreign keys.
Refs https://github.com/simonw/sqlite-utils/pull/770/changes#r3525703477
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
For foreign keys declared as "REFERENCES other_table" with no explicit
columns, PRAGMA foreign_key_list returns None for the referenced
column. The foreign_keys property now resolves these to the other
table's primary key columns, so other_column=None unambiguously
indicates a compound foreign key.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ForeignKey gains on_delete and on_update fields (default "NO ACTION"),
populated from PRAGMA foreign_key_list. create_table_sql() renders the
corresponding clauses for both inline and compound table-level foreign
keys, which means table.transform() now preserves actions such as
ON DELETE CASCADE - previously they were silently stripped whenever a
table was transformed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
table.add_foreign_key() and db.add_foreign_keys() now accept lists of
column names to create compound foreign keys:
table.add_foreign_key(
["campus_name", "dept_code"], "departments"
)
Omitting the other columns uses the compound primary key of the other
table. Duplicate detection compares the full column lists.
db.index_foreign_keys() now creates a single composite index across the
columns of a compound foreign key, rather than an index per column.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Compound foreign keys are passed through table.transform() intact
instead of being degraded to per-column single foreign keys:
- rename= applies to each member column of a compound key
- drop= of any member column drops the whole constraint, matching
the existing single-column behavior
- drop_foreign_keys= accepts a bare column name (drops any foreign
key that column participates in) or a tuple of columns (drops the
compound key with exactly those columns)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
create_table() and friends now accept compound foreign keys, specified
as lists of column names in the existing tuple forms:
foreign_keys=[
(["campus_name", "dept_code"], "departments"),
(["campus_name", "dept_code"], "departments", ["campus_name", "dept_code"]),
]
The two-item form guesses the compound primary key of the other table.
Compound keys are rendered as table-level FOREIGN KEY constraints,
while single-column keys keep their inline REFERENCES clauses.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
sorted() on a foreign_keys list containing both compound (column=None)
and single-column ForeignKeys raised TypeError because dataclass
ordering compared None against str. column/other_column are now
excluded from comparison - ordering and equality use the always
populated columns/other_columns lists instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
query() and execute() classified statements using
sql.lstrip().upper().startswith(...), which a leading SQL comment
defeated: db.query('/* c */ COMMIT') inside db.atomic() committed
the caller's transaction and masked the ValueError with a 'no such
savepoint' error, comment-prefixed PRAGMAs ran inside the savepoint
guard where journal mode changes are refused, and a comment-prefixed
BEGIN passed to db.execute() was instantly auto-committed.
The new _first_keyword() helper skips leading whitespace and -- or
/* */ comments - the only things SQLite's tokenizer allows before
the first token - and returns that token for exact comparison, so
keyword detection now matches what SQLite itself will see.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
db.query() rejected statements roll back, RETURNING commits immediately.
Fixes two transaction bugs in db.query():
- A non-row-returning statement such as an UPDATE was executed and
auto-committed by execute() before the ValueError was raised, so the
write took effect despite being rejected. query() now runs the
statement inside a savepoint and rolls it back before raising, so a
rejected statement has no effect on the database - in every
connection mode, including autocommit=True.
- INSERT ... RETURNING only committed once the returned generator was
fully exhausted, so calling query() without iterating - or partially
iterating with next() - left the transaction open and the write could
be rolled back on close. The write is now completed and committed at
call time, as the documentation already promised. Plain SELECTs are
still fetched lazily.
Transaction control statements, VACUUM, ATTACH and DETACH never return
rows, so query() now rejects them without executing them. PRAGMA
statements skip the savepoint guard because some of them - such as
pragma journal_mode=wal - refuse to run inside a transaction.
Claude-Session: https://claude.ai/code/session_012U3iRfJoTZ5vd22cBSF2nJ
parametrize received a generator from Path.glob(), which pytest 10
will reject. sorted() materializes the list and makes the parameter
order deterministic as a bonus.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UnLnhsH25Nnv7LHhekUfPd
Runs the entire test suite against connections created with the
Python 3.12+ sqlite3.connect(autocommit=True) mode, by patching
connect() to default autocommit=True for internally-created
connections. Errors on older Python versions. The full suite
passes in both modes with no test changes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UnLnhsH25Nnv7LHhekUfPd
Raw write statements previously opened an implicit transaction that
stayed open until something committed it - the write was visible on
the same connection, making it look saved, but was silently rolled
back when the connection closed. execute() now commits any implicit
transaction it opens, so raw writes behave like every other write
in the library: committed as soon as they run, unless an explicit
transaction (db.begin() or db.atomic()) is open, in which case they
join it.
Row-returning writes such as INSERT ... RETURNING via db.query()
commit once their rows have been iterated. atomic(), commit() and
rollback() now issue literal COMMIT/ROLLBACK statements, which have
identical semantics in every sqlite3 connection mode. The CLI bulk
command uses db.atomic() instead of 'with db.conn:'.
This simplifies the transaction contract to: everything commits
immediately unless you explicitly opened a transaction. Docs
updated throughout; changelog and upgrading guide cover the
breaking change for code that relied on rolling back uncommitted
execute() writes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UnLnhsH25Nnv7LHhekUfPd
