* transform: coerce empty strings to NULL when converting TEXT columns to numeric types
When a TEXT column is transformed to INTEGER, FLOAT, or REAL and a row
contains an empty string, the empty string is now converted to NULL during
the INSERT...SELECT copy, matching the expected behavior described in #488.
Fixes#488
* Emit LIMIT -1 when offset is used without limit, closes#816
SQLite requires a LIMIT clause to appear before OFFSET, so passing offset
without limit generated invalid SQL such as:
select * from "t" offset 2
which raised OperationalError: near "2": syntax error.
A negative limit means "no upper bound" in SQLite, so "limit -1 offset N"
returns all rows from position N onwards.
Fixed in three places that build LIMIT/OFFSET SQL:
- Queryable.rows_where() - also covers pks_and_rows_where()
- Table.search_sql() - also covers search()
- the "sqlite-utils rows" CLI command
* Remove duplicate comments
---------
Co-authored-by: ethanhawkes-gif <259455325+ethanhawkes-gif@users.noreply.github.com>
The tokenize value passed to Table.enable_fts() was interpolated directly
into the CREATE VIRTUAL TABLE statement inside a single-quoted string
literal. A value containing a single quote could break out of that literal
and inject arbitrary SQL, which executes via executescript(). This is
reachable from the CLI via 'enable-fts --tokenize'.
Route the value through the existing Database.quote() helper so SQLite
itself escapes it. Legitimate tokenizers such as 'porter' are unaffected.
Adds a regression test.
* Fix _decode_default_value to unescape doubled single quotes in string defaults
SQLite stores string defaults with single quotes doubled (e.g. DEFAULT 'O''Brien'
is stored as the literal "'O''Brien'" in sqlite_master). The previous code
stripped the outer quotes with value[1:-1] but never converted '' back to ',
so default_values returned the raw escaped form instead of the true string value.
* Test for doubled single quotes in string defaults
* Automated upgrades by Ruff
uvx --with 'ruff>=0.16.0' ruff check . --fix --unsafe-fixes
* Fix remaining Ruff errors with GPT-5.6 Sol high
https://gist.github.com/simonw/6da7906a9fea6e90da131c21a9055199
* Fix flake E501 long lines
* New Protocol for migrations to make ty happy
> Add a test that covers what happens if you run transform against a table with ON CASCADE DELETE for one of its foreign keys - those records should not be deleted during the transform even though the table is dropped as part of that procedure
For #439.
`file_progress()` sets the progress bar length to the file size in
bytes (`os.path.getsize(file.name)`), but `UpdateWrapper.__iter__`
called `update(len(line))`, which is the decoded character count.
With UTF-16-LE input every character is 2 bytes, so the bar capped
at 50%; UTF-32 capped at 25%; etc.
Simon noted in the issue that the obvious fix (calling `.tell()` on
the wrapped text stream) doesn't work because text mode disables it
during iteration. The underlying binary buffer doesn't have that
restriction though, so this tracks progress against
`TextIOWrapper.buffer.tell()` when the wrapped object exposes one.
For raw binary streams (no `.buffer` attribute) we keep the old
behaviour, which was already byte-accurate.
Added six regression tests in tests/test_utils.py covering UTF-8,
UTF-16-LE, BOM-prefixed UTF-16, the sniff-style `BufferedReader` chain,
a raw binary fallback, and the `.read()` path used by the JSON loader.
Each asserts that the sum of update() calls equals the on-disk file
size, which is what `click.progressbar` needs to reach 100%.
