Columns declared with type JSON now decode automatically wherever nested
JSON is already supported (query/rows/search/memory output, and the
Python API), without needing --json-cols. Uses sqlite3's built-in
detect_types=PARSE_DECLTYPES + a registered JSON converter, so it works
transparently through joins and aliases. Flat formats (CSV/TSV/table/raw)
re-serialize decoded values back to JSON text instead of a Python repr.
Also adds "JSON" as a first-class column type for create()/add_column()/
transform(), and fixes column_affinity("JSON") so transform() doesn't
rewrite JSON columns as REAL.
UpdateWrapper counted decoded characters (len(line)) against a byte-based
file length, so utf-16-le input (2 bytes/char) only ever reported ~50%
progress even after reading finished, looking like a stall. Track the
underlying binary stream's position instead so progress is honest for any
encoding, and add regression tests covering a clean utf-16-le insert plus
a genuinely corrupt one that must still raise a visible error.
Reuses the same _compile_code mechanism as insert/upsert --convert and the
convert command (row variable, r. recipes, --import for extra modules)
instead of a bespoke mini-language. Lets sqlar-style imports (or any other
per-row transform) happen in one insert-files call rather than needing a
separate convert pass afterwards.
Closes#597
Adds a --sqlar option that defaults insert-files to the name/mode/mtime/sz/data
schema used by SQLite's own sqlar archive format, plus a content_sqlar column
type that zlib-compresses content the same way sqlar_compress() does (only
when compression actually shrinks the data). Reuses the existing -c coldef
parsing added for fixed-literal metadata columns, no changes needed there.
Existing BLOB/TEXT content/content_text imports are untouched.
Closes#141
The CLI can render query/table output as CSV via --csv, but the Python
API had no equivalent. sqlite_utils.utils.rows_to_csv() takes rows from
db.query() or table.rows/rows_where() and writes CSV to a file-like
object, or returns it as a string, with a header row by default (skip
with no_headers=True) to mirror the CLI's behavior.
Extends the -c/--column spec so a column can be populated with a
literal value for every inserted file, e.g. -c file_type:text:gif,
without a separate update pass. Existing colname:coldef shorthands
and compound --pk behavior are unchanged.
Closes#140
* Fix _decode_default_value to unescape doubled single quotes in string defaults
SQLite stores string defaults with single quotes doubled (e.g. DEFAULT 'O''Brien'
is stored as the literal "'O''Brien'" in sqlite_master). The previous code
stripped the outer quotes with value[1:-1] but never converted '' back to ',
so default_values returned the raw escaped form instead of the true string value.
* Test for doubled single quotes in string defaults
* Automated upgrades by Ruff
uvx --with 'ruff>=0.16.0' ruff check . --fix --unsafe-fixes
* Fix remaining Ruff errors with GPT-5.6 Sol high
https://gist.github.com/simonw/6da7906a9fea6e90da131c21a9055199
* Fix flake E501 long lines
* New Protocol for migrations to make ty happy
Closes#781, #783
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900618685
* Fix rowid pk and last_rowid regressions in insert/upsert
Two behaviour regressions in the 4.0 insert/upsert rewrite broke callers
(notably Datasette's write API) that operate on tables without an explicit
primary key. Both are fixed here with regression tests.
1. rowid (and its aliases _rowid_/oid) were rejected as a primary key.
Table.pks already reports ["rowid"] for a rowid table, but the new pk
validation raised InvalidColumns because rowid is not listed among the
table's columns, and the insert success path then raised KeyError when
looking up the pk value. rowid aliases are now accepted for rowid tables
and resolve directly to the rowid.
2. An ignored insert (INSERT OR IGNORE that matched an existing row) no
longer populated last_rowid, and only set last_pk when an explicit pk=
was passed. It now locates the existing conflicting row by its primary
key values and reports that row's rowid and pk, rather than relying on
the connection's last inserted rowid.
Add a shared ROWID_ALIASES constant for the rowid alias names.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E7af8SxFZqiCerJB6MqKnY
A RAISE(ROLLBACK) trigger or INSERT OR ROLLBACK conflict rolls back the
entire transaction and destroys every savepoint. The cleanup paths in
atomic() and query() then raised OperationalError ("no such savepoint" /
"cannot rollback - no transaction is active"), masking the original
IntegrityError - breaking user code that catches sqlite3.IntegrityError.
Cleanup now checks conn.in_transaction first: if the error already
destroyed the transaction there is nothing left to undo, and the
original exception propagates.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The docs promise --list will not create the database file or the
_sqlite_migrations table, but legacy sqlite_migrate.Migrations classes
create the table (in the legacy schema) from their pending()/applied()
methods. The listing now runs inside a transaction that is rolled back,
keeping --list read-only regardless of what the migration class does.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
sqlite-utils insert db t - --pk badcol and sqlite-utils extract db t
nosuchcol dumped raw InvalidColumns tracebacks - the insert error
handling caught NoTable and OperationalError but not the InvalidColumns
introduced for #732, and the extract command had no handling at all
(including for NoTable when pointed at a view). Both now exit with
click-style Error: messages.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The lookup-table insert relied on INSERT OR IGNORE and the unique index
to dedupe against existing rows, but SQLite unique indexes treat NULLs
as distinct - extracting a second table into the same lookup table
re-inserted every NULL-containing value, growing orphan rows on each
extract. The insert now also has an IS-based NOT EXISTS guard, matching
how the foreign keys themselves are resolved.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The existing-foreign-key dedup compared only columns and other_table, so
requesting an FK that already exists with different ON DELETE/ON UPDATE
actions was a silent no-op - and with add_foreign_key() raising "already
exists", there was no signal that the requested actions were dropped.
An exact match (including actions) is still skipped for idempotency;
a mismatch now raises AlterError pointing at table.transform().
Also validates that compound foreign keys passed as 4-tuples have the
same number of columns on both sides - extra other-columns were being
silently discarded, e.g. ("t", ("a",), "other", ("a", "b")) created a
single-column key referencing just "a".
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Assigning conn.isolation_level commits any pending transaction as a side
effect, so entering the context manager with a transaction open silently
committed the caller's work and made a later rollback() a no-op. All
internal callers already ensure no transaction is open; the public API
now enforces it, matching enable_wal() and disable_wal().
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The CLI validated --stop-before names against both pending and applied
migrations, but Migrations.apply() only looked for the stop name among
pending ones - naming an applied migration passed validation and then
silently applied every migration after it, the exact outcome the option
exists to prevent. apply() now raises ValueError before applying
anything if a stop_before name matches an applied migration in its set;
names not in the set are still ignored, since unqualified CLI values are
offered to every set. The migrate command reports it as a clean error.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The InvalidColumns check added for #732 fired before alter=True had a
chance to add the missing pk column - a regression from 3.x, where
insert({"id": 5, "a": 2}, pk="id", alter=True) against a table without
an id column worked. With alter=True the check is now deferred until the
first batch of record keys is known: a pk column supplied by the records
passes, one found in neither the table nor the records still raises
InvalidColumns before anything is inserted. An empty record iterator
with alter=True returns without error, matching the 3.x no-op.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Type detection is the 4.0 default for CSV/TSV data, and the detected-type
transform ran even when the target table already existed - inserting a
CSV into a table with a TEXT zip column converted the column to INTEGER,
corrupting values with leading zeros ('01234' became 1234) with no
warning. Detected types now only apply to tables the command created.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The previous compound-pk-ordering commit switched pks_and_rows_where()
to Table-only properties, but the method is defined on Queryable and
views exposed it too - calling it on a View raised AttributeError.
Restored Queryable-safe logic, and stopped double-quoting the
synthesized rowid column: SQLite turns a double-quoted identifier that
does not resolve into a string literal, so on a view the generated
select "rowid" silently produced the string 'rowid' and a confusing
KeyError, where 3.x's [rowid] quoting raised OperationalError cleanly.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
resolve_foreign_keys() used all-or-nothing isinstance checks, so mixing
ForeignKey objects with tuples raised "foreign_keys= should be a list of
tuples" - a regression from 3.x, where ForeignKey was a namedtuple and
passed the tuple check. Each item is now normalized individually, which
also allows bare column-name strings in a mixed list.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The namedtuple-to-dataclass change made ForeignKey unhashable, breaking
set(table.foreign_keys) and dict-key usage that worked in 3.x. frozen=True
restores immutability and hashability. Equality and hashing cover all
compared fields including on_delete/on_update - two foreign keys differing
only in their actions are different constraints.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PRAGMA table_info sets is_pk to the 1-based position of each column
within the PRIMARY KEY, which can differ from table column order.
table.pks previously returned table column order, so an implicit
compound FOREIGN KEY ... REFERENCES other was introspected with its
referenced columns inverted, and transform() baked that inverted order
into the rewritten schema - failing with IntegrityError on valid data,
or silently reversing the constraint with foreign keys off.
table.pks, compound foreign key guessing (create, add_foreign_key) and
transform() now all use declaration order, and transform() no longer
reorders a compound PRIMARY KEY (b, a) into table column order.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
db.query() promises that a statement rejected with ValueError is rolled
back and has no effect. PRAGMA statements are the exception: some of them
refuse to run inside a transaction, so they execute outside the savepoint
guard - a row-less PRAGMA such as "PRAGMA user_version = 5" therefore
takes effect despite the ValueError. Documenting this as a known
limitation rather than fixing it, since a fix would need a hardcoded list
of row-returning PRAGMAs.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- table.extract() and the sqlite-utils extract command now skip rows
where every extracted column is null: the new foreign key column is
left null and no all-null record is added to the lookup table. Rows
with at least one non-null extracted column are extracted as before.
- The extracts= option to insert() and friends keeps None values as
null instead of creating a lookup record for them - previously each
insert batch added a duplicate null row to the lookup table.
- table.lookup() compares values using IS rather than = so lookup
values containing None match existing rows correctly, instead of
inserting a duplicate row on every call.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix transform() corrupting TRUE/FALSE/NULL column defaults
quote_default_value() passed keyword-literal defaults (TRUE, FALSE, NULL)
through self.quote(), wrapping them in quotes. So transform() rebuilt a
column declared "INTEGER DEFAULT TRUE" as "INTEGER DEFAULT 'TRUE'", and a
later default insert stored the text 'TRUE' instead of the integer 1
(likewise 'FALSE' for 0 and 'NULL' for null) - silent value corruption.
Return these keyword literals unquoted, as already done for the
CURRENT_TIME/DATE/TIMESTAMP literals.
PR #764
Column names passed to Python API methods are now resolved against the
table schema case-insensitively, mirroring how SQLite itself compares
identifiers (ASCII-only case folding). Fixes KeyError populating
last_pk from insert()/upsert(), silently ignored transform()
rename/drop/types options, duplicate column errors from
create_table(transform=True), redundant lookup() indexes, and
case-sensitive foreign key validation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Queries returning duplicate column names - e.g. joins between tables
sharing column names - silently lost values because rows were built
with dict(zip(keys, row)), where the last duplicate wins.
Later occurrences are now renamed with a numeric suffix: id, id
becomes id, id_2 - skipping any suffix that would collide with a
real column in the same query.
The new utils.dedupe_keys() helper transforms the key list once per
query, so the per-row dict construction is unchanged and there is no
measurable performance impact.
Applied in Database.query() (including the PRAGMA and RETURNING
paths), Table.rows_where(), Table.search() and the CLI's JSON output.
CSV, TSV and table output keep the original duplicate headers.
Closes#624
table.add_foreign_key() now accepts on_delete= and on_update= to
create foreign keys with ON DELETE/ON UPDATE actions:
table.add_foreign_key("author_id", "authors", "id", on_delete="CASCADE")
Works for compound foreign keys too.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
