The JSON/CSV branch used 'fn(doc) or doc', so any falsy result was
discarded and the unconverted row inserted. Exit code 0, no warning:
$ echo '[{"id": 1, "name": "Cleo"}]' | sqlite-utils insert d.db t - --convert '{}'
$ sqlite-utils 'select * from d.db' # -> the original row, not {}
The realistic shape of this is a filter:
--convert 'row if row["ok"] else {}' # meant to drop the bad rows
# every row inserted, including the ones meant to be dropped
'--lines' runs a different expression twelve lines up and has no such
fallback, so the same --convert code is rejected there and silently
accepted here:
--convert 0 JSON input -> exit 0, rows inserted
--convert 0 --lines -> exit 1, Rows must all be dictionaries, got: 0
test_insert_convert_error_messages already asserts that a non-dict
return is rejected (--convert 1); 0, False, [] and "" were the outliers,
purely because they are falsy.
None is the one falsy result that has to keep meaning 'modify doc in
place' - a --convert that is a bare statement returns it, which
test_insert_convert_row_modifying_in_place covers - so the fallback is
now keyed on 'is None' rather than truthiness.
Tests: 7 new cases in tests/test_cli_insert.py, all failing on 6bc1d33.
Suite 1505 passed, 16 skipped. black, flake8, mypy clean. pyright
reports one error and ty two diagnostics, both identical on unmodified
main (tests/test_constructor.py:74 and two unused-type-ignore comments
in db.py/utils.py) and none in the files this touches.
Fable 5.1 explains:
> apt on the runner image has stale package lists. It tried to download libminizip1t64_1.3.dfsg-3.1ubuntu2.1 from security.ubuntu.com and got a 404, because Ubuntu has since published a newer build of that package and pulled the old .deb from the mirror.
* transform: coerce empty strings to NULL when converting TEXT columns to numeric types
When a TEXT column is transformed to INTEGER, FLOAT, or REAL and a row
contains an empty string, the empty string is now converted to NULL during
the INSERT...SELECT copy, matching the expected behavior described in #488.
Fixes#488
* Emit LIMIT -1 when offset is used without limit, closes#816
SQLite requires a LIMIT clause to appear before OFFSET, so passing offset
without limit generated invalid SQL such as:
select * from "t" offset 2
which raised OperationalError: near "2": syntax error.
A negative limit means "no upper bound" in SQLite, so "limit -1 offset N"
returns all rows from position N onwards.
Fixed in three places that build LIMIT/OFFSET SQL:
- Queryable.rows_where() - also covers pks_and_rows_where()
- Table.search_sql() - also covers search()
- the "sqlite-utils rows" CLI command
* Remove duplicate comments
---------
Co-authored-by: ethanhawkes-gif <259455325+ethanhawkes-gif@users.noreply.github.com>
The tokenize value passed to Table.enable_fts() was interpolated directly
into the CREATE VIRTUAL TABLE statement inside a single-quoted string
literal. A value containing a single quote could break out of that literal
and inject arbitrary SQL, which executes via executescript(). This is
reachable from the CLI via 'enable-fts --tokenize'.
Route the value through the existing Database.quote() helper so SQLite
itself escapes it. Legitimate tokenizers such as 'porter' are unaffected.
Adds a regression test.
* Fix _decode_default_value to unescape doubled single quotes in string defaults
SQLite stores string defaults with single quotes doubled (e.g. DEFAULT 'O''Brien'
is stored as the literal "'O''Brien'" in sqlite_master). The previous code
stripped the outer quotes with value[1:-1] but never converted '' back to ',
so default_values returned the raw escaped form instead of the true string value.
* Test for doubled single quotes in string defaults
* Automated upgrades by Ruff
uvx --with 'ruff>=0.16.0' ruff check . --fix --unsafe-fixes
* Fix remaining Ruff errors with GPT-5.6 Sol high
https://gist.github.com/simonw/6da7906a9fea6e90da131c21a9055199
* Fix flake E501 long lines
* New Protocol for migrations to make ty happy
> Add a test that covers what happens if you run transform against a table with ON CASCADE DELETE for one of its foreign keys - those records should not be deleted during the transform even though the table is dropped as part of that procedure
