Three ways a column containing nulls produced a wrong answer, all from
nulls not being accounted for.
1. The "single distinct value" shortcut. count(distinct col) skips
nulls in SQL, so a column holding nulls plus one other value reports
num_distinct == 1 and took the shortcut, which reports that one value
with a count of total_rows:
insert [{'c': None}, {'c': None}, {'c': 'a'}, {'c': 'a'}]
analyze_column('c').most_common # was [(None, 4)]
select c, count(*) group by c # is [(None, 2), ('a', 2)]
The shortcut now requires that the column has no nulls, so those
columns fall through to the group by like any other.
2. The tie-break sort compared the values directly, so a null and a
string with the same count raised TypeError:
analyze_column('c', common_limit=5)
# TypeError: '<' not supported between instances of 'NoneType' and 'str'
sort_key() keeps nulls out of the comparison and sorts them last,
which keeps the existing ordering of non-null ties - the test for
'Terryterryterry' ahead of 'Kumar' at equal counts still passes.
3. truncate() turned a null into the string 'None', so a null was
indistinguishable from a column really holding 'None'. That is the
path analyze-tables takes, since it passes value_truncate=80.
num_distinct itself is unchanged: it still follows SQL and reports 0 for
an all-null column, which test_analyze_table_column_all_nulls pins. But
num_distinct can therefore be lower than len(most_common) when a column
holds nulls alongside all-distinct values - {'c': None, 'a', 'b'} gives
num_distinct 2 and three most_common entries. Whether null should count
as a distinct value is a question about what the number means, so I have
left it alone rather than change an output the tests already pin.
Tests: 6 new cases in tests/test_analyze.py, all failing on 6bc1d33.
Suite 1506 passed, 16 skipped. mypy, pyright, flake8, ty, black and
cog --check clean.
Closes#559Closes#235
Refs https://github.com/simonw/llm/issues/60
- Uses `sqlean` in place of `sqlite3` if `sqlean.py` is installed
- Uses `sqlite-dump` if available and `conn.iterdump()` does not exist
- New `with db.ensure_autocommit_off()` method for ensuring autocommit is off, used by `enable_wal()` and `disable_wal()`.
