mirror of
https://github.com/simonw/sqlite-utils.git
synced 2026-10-23 17:57:03 +02:00
8e5dfdd409Three ways a column containing nulls produced a wrong answer, all from
nulls not being accounted for.
1. The "single distinct value" shortcut. count(distinct col) skips
nulls in SQL, so a column holding nulls plus one other value reports
num_distinct == 1 and took the shortcut, which reports that one value
with a count of total_rows:
insert [{'c': None}, {'c': None}, {'c': 'a'}, {'c': 'a'}]
analyze_column('c').most_common # was [(None, 4)]
select c, count(*) group by c # is [(None, 2), ('a', 2)]
The shortcut now requires that the column has no nulls, so those
columns fall through to the group by like any other.
2. The tie-break sort compared the values directly, so a null and a
string with the same count raised TypeError:
analyze_column('c', common_limit=5)
# TypeError: '<' not supported between instances of 'NoneType' and 'str'
sort_key() keeps nulls out of the comparison and sorts them last,
which keeps the existing ordering of non-null ties - the test for
'Terryterryterry' ahead of 'Kumar' at equal counts still passes.
3. truncate() turned a null into the string 'None', so a null was
indistinguishable from a column really holding 'None'. That is the
path analyze-tables takes, since it passes value_truncate=80.
num_distinct itself is unchanged: it still follows SQL and reports 0 for
an all-null column, which test_analyze_table_column_all_nulls pins. But
num_distinct can therefore be lower than len(most_common) when a column
holds nulls alongside all-distinct values - {'c': None, 'a', 'b'} gives
num_distinct 2 and three most_common entries. Whether null should count
as a distinct value is a question about what the number means, so I have
left it alone rather than change an output the tests already pin.
Tests: 6 new cases in tests/test_analyze.py, all failing on
|
||
|---|---|---|
| .. | ||
| __init__.py | ||
| __main__.py | ||
| cli.py | ||
| create_table_parser.py | ||
| db.py | ||
| hookspecs.py | ||
| migrations.py | ||
| plugins.py | ||
| py.typed | ||
| recipes.py | ||
| utils.py | ||