Values not natively serializable (set, enum.Enum, project-specific types)
previously always fell back to repr() when written to a JSON column, with
no way to reconstruct them on read. json_default is passed through to
json.dumps() as default= wherever the library serializes dict/list/tuple
values; json_object_hook is passed through to json.loads() as object_hook=
when deserialize_json=True. Both are Database-level, matching the existing
deserialize_json option, since every write/read path already threads
through self.db.
Fixes#521
Values inserted as dict/list are serialized to JSON TEXT columns, but rows
fetched via .rows/.rows_where()/.get()/.search()/.query() previously always
came back as raw strings. This adds an opt-in constructor flag that parses
any string value that looks like a JSON object or array back into a
dict/list, without changing the default (raw string) behavior.
Fixes#612
Wires the extracts= Python API feature through the CLI so imports can
create lookup tables in one step: --extract species for a table named
after the column, or --extract species:Species for a custom name.
Closes#352
Columns declared with type JSON now decode automatically wherever nested
JSON is already supported (query/rows/search/memory output, and the
Python API), without needing --json-cols. Uses sqlite3's built-in
detect_types=PARSE_DECLTYPES + a registered JSON converter, so it works
transparently through joins and aliases. Flat formats (CSV/TSV/table/raw)
re-serialize decoded values back to JSON text instead of a Python repr.
Also adds "JSON" as a first-class column type for create()/add_column()/
transform(), and fixes column_affinity("JSON") so transform() doesn't
rewrite JSON columns as REAL.
UpdateWrapper counted decoded characters (len(line)) against a byte-based
file length, so utf-16-le input (2 bytes/char) only ever reported ~50%
progress even after reading finished, looking like a stall. Track the
underlying binary stream's position instead so progress is honest for any
encoding, and add regression tests covering a clean utf-16-le insert plus
a genuinely corrupt one that must still raise a visible error.
Reuses the same _compile_code mechanism as insert/upsert --convert and the
convert command (row variable, r. recipes, --import for extra modules)
instead of a bespoke mini-language. Lets sqlar-style imports (or any other
per-row transform) happen in one insert-files call rather than needing a
separate convert pass afterwards.
Closes#597
Adds a --sqlar option that defaults insert-files to the name/mode/mtime/sz/data
schema used by SQLite's own sqlar archive format, plus a content_sqlar column
type that zlib-compresses content the same way sqlar_compress() does (only
when compression actually shrinks the data). Reuses the existing -c coldef
parsing added for fixed-literal metadata columns, no changes needed there.
Existing BLOB/TEXT content/content_text imports are untouched.
Closes#141
The CLI can render query/table output as CSV via --csv, but the Python
API had no equivalent. sqlite_utils.utils.rows_to_csv() takes rows from
db.query() or table.rows/rows_where() and writes CSV to a file-like
object, or returns it as a string, with a header row by default (skip
with no_headers=True) to mirror the CLI's behavior.
Extends the -c/--column spec so a column can be populated with a
literal value for every inserted file, e.g. -c file_type:text:gif,
without a separate update pass. Existing colname:coldef shorthands
and compound --pk behavior are unchanged.
Closes#140
* Fix _decode_default_value to unescape doubled single quotes in string defaults
SQLite stores string defaults with single quotes doubled (e.g. DEFAULT 'O''Brien'
is stored as the literal "'O''Brien'" in sqlite_master). The previous code
stripped the outer quotes with value[1:-1] but never converted '' back to ',
so default_values returned the raw escaped form instead of the true string value.
* Test for doubled single quotes in string defaults
* Automated upgrades by Ruff
uvx --with 'ruff>=0.16.0' ruff check . --fix --unsafe-fixes
* Fix remaining Ruff errors with GPT-5.6 Sol high
https://gist.github.com/simonw/6da7906a9fea6e90da131c21a9055199
* Fix flake E501 long lines
* New Protocol for migrations to make ty happy
> Add a test that covers what happens if you run transform against a table with ON CASCADE DELETE for one of its foreign keys - those records should not be deleted during the transform even though the table is dropped as part of that procedure
Closes#781, #783
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900618685
* Fix rowid pk and last_rowid regressions in insert/upsert
Two behaviour regressions in the 4.0 insert/upsert rewrite broke callers
(notably Datasette's write API) that operate on tables without an explicit
primary key. Both are fixed here with regression tests.
1. rowid (and its aliases _rowid_/oid) were rejected as a primary key.
Table.pks already reports ["rowid"] for a rowid table, but the new pk
validation raised InvalidColumns because rowid is not listed among the
table's columns, and the insert success path then raised KeyError when
looking up the pk value. rowid aliases are now accepted for rowid tables
and resolve directly to the rowid.
2. An ignored insert (INSERT OR IGNORE that matched an existing row) no
longer populated last_rowid, and only set last_pk when an explicit pk=
was passed. It now locates the existing conflicting row by its primary
key values and reports that row's rowid and pk, rather than relying on
the connection's last inserted rowid.
Add a shared ROWID_ALIASES constant for the rowid alias names.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E7af8SxFZqiCerJB6MqKnY
A RAISE(ROLLBACK) trigger or INSERT OR ROLLBACK conflict rolls back the
entire transaction and destroys every savepoint. The cleanup paths in
atomic() and query() then raised OperationalError ("no such savepoint" /
"cannot rollback - no transaction is active"), masking the original
IntegrityError - breaking user code that catches sqlite3.IntegrityError.
Cleanup now checks conn.in_transaction first: if the error already
destroyed the transaction there is nothing left to undo, and the
original exception propagates.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
SQLite 3.23.1 rejects a UTF-8 byte order mark before the first token,
so the BOM variant of the execute()-prefixed-BEGIN test now skips when
the SQLite version does not accept a leading BOM. And versions before
3.36 allowed selecting rowid from a view, returning NULL, rather than
raising an error - the pks_and_rows_where() view test now accepts
either behavior.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The docs promise --list will not create the database file or the
_sqlite_migrations table, but legacy sqlite_migrate.Migrations classes
create the table (in the legacy schema) from their pending()/applied()
methods. The listing now runs inside a transaction that is rolled back,
keeping --list read-only regardless of what the migration class does.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
sqlite-utils insert db t - --pk badcol and sqlite-utils extract db t
nosuchcol dumped raw InvalidColumns tracebacks - the insert error
handling caught NoTable and OperationalError but not the InvalidColumns
introduced for #732, and the extract command had no handling at all
(including for NoTable when pointed at a view). Both now exit with
click-style Error: messages.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
