Three ways a column containing nulls produced a wrong answer, all from
nulls not being accounted for.
1. The "single distinct value" shortcut. count(distinct col) skips
nulls in SQL, so a column holding nulls plus one other value reports
num_distinct == 1 and took the shortcut, which reports that one value
with a count of total_rows:
insert [{'c': None}, {'c': None}, {'c': 'a'}, {'c': 'a'}]
analyze_column('c').most_common # was [(None, 4)]
select c, count(*) group by c # is [(None, 2), ('a', 2)]
The shortcut now requires that the column has no nulls, so those
columns fall through to the group by like any other.
2. The tie-break sort compared the values directly, so a null and a
string with the same count raised TypeError:
analyze_column('c', common_limit=5)
# TypeError: '<' not supported between instances of 'NoneType' and 'str'
sort_key() keeps nulls out of the comparison and sorts them last,
which keeps the existing ordering of non-null ties - the test for
'Terryterryterry' ahead of 'Kumar' at equal counts still passes.
3. truncate() turned a null into the string 'None', so a null was
indistinguishable from a column really holding 'None'. That is the
path analyze-tables takes, since it passes value_truncate=80.
num_distinct itself is unchanged: it still follows SQL and reports 0 for
an all-null column, which test_analyze_table_column_all_nulls pins. But
num_distinct can therefore be lower than len(most_common) when a column
holds nulls alongside all-distinct values - {'c': None, 'a', 'b'} gives
num_distinct 2 and three most_common entries. Whether null should count
as a distinct value is a question about what the number means, so I have
left it alone rather than change an output the tests already pin.
Tests: 6 new cases in tests/test_analyze.py, all failing on 6bc1d33.
Suite 1506 passed, 16 skipped. mypy, pyright, flake8, ty, black and
cog --check clean.
Fable 5.1 explains:
> apt on the runner image has stale package lists. It tried to download libminizip1t64_1.3.dfsg-3.1ubuntu2.1 from security.ubuntu.com and got a 404, because Ubuntu has since published a newer build of that package and pulled the old .deb from the mirror.
* transform: coerce empty strings to NULL when converting TEXT columns to numeric types
When a TEXT column is transformed to INTEGER, FLOAT, or REAL and a row
contains an empty string, the empty string is now converted to NULL during
the INSERT...SELECT copy, matching the expected behavior described in #488.
Fixes#488
* Emit LIMIT -1 when offset is used without limit, closes#816
SQLite requires a LIMIT clause to appear before OFFSET, so passing offset
without limit generated invalid SQL such as:
select * from "t" offset 2
which raised OperationalError: near "2": syntax error.
A negative limit means "no upper bound" in SQLite, so "limit -1 offset N"
returns all rows from position N onwards.
Fixed in three places that build LIMIT/OFFSET SQL:
- Queryable.rows_where() - also covers pks_and_rows_where()
- Table.search_sql() - also covers search()
- the "sqlite-utils rows" CLI command
* Remove duplicate comments
---------
Co-authored-by: ethanhawkes-gif <259455325+ethanhawkes-gif@users.noreply.github.com>
The tokenize value passed to Table.enable_fts() was interpolated directly
into the CREATE VIRTUAL TABLE statement inside a single-quoted string
literal. A value containing a single quote could break out of that literal
and inject arbitrary SQL, which executes via executescript(). This is
reachable from the CLI via 'enable-fts --tokenize'.
Route the value through the existing Database.quote() helper so SQLite
itself escapes it. Legitimate tokenizers such as 'porter' are unaffected.
Adds a regression test.
* Fix _decode_default_value to unescape doubled single quotes in string defaults
SQLite stores string defaults with single quotes doubled (e.g. DEFAULT 'O''Brien'
is stored as the literal "'O''Brien'" in sqlite_master). The previous code
stripped the outer quotes with value[1:-1] but never converted '' back to ',
so default_values returned the raw escaped form instead of the true string value.
* Test for doubled single quotes in string defaults
* Automated upgrades by Ruff
uvx --with 'ruff>=0.16.0' ruff check . --fix --unsafe-fixes
* Fix remaining Ruff errors with GPT-5.6 Sol high
https://gist.github.com/simonw/6da7906a9fea6e90da131c21a9055199
* Fix flake E501 long lines
* New Protocol for migrations to make ty happy
> Add a test that covers what happens if you run transform against a table with ON CASCADE DELETE for one of its foreign keys - those records should not be deleted during the transform even though the table is dropped as part of that procedure
