mirror of
https://github.com/simonw/sqlite-utils.git
synced 2026-10-23 17:57:03 +02:00
8e5dfdd409Three ways a column containing nulls produced a wrong answer, all from
nulls not being accounted for.
1. The "single distinct value" shortcut. count(distinct col) skips
nulls in SQL, so a column holding nulls plus one other value reports
num_distinct == 1 and took the shortcut, which reports that one value
with a count of total_rows:
insert [{'c': None}, {'c': None}, {'c': 'a'}, {'c': 'a'}]
analyze_column('c').most_common # was [(None, 4)]
select c, count(*) group by c # is [(None, 2), ('a', 2)]
The shortcut now requires that the column has no nulls, so those
columns fall through to the group by like any other.
2. The tie-break sort compared the values directly, so a null and a
string with the same count raised TypeError:
analyze_column('c', common_limit=5)
# TypeError: '<' not supported between instances of 'NoneType' and 'str'
sort_key() keeps nulls out of the comparison and sorts them last,
which keeps the existing ordering of non-null ties - the test for
'Terryterryterry' ahead of 'Kumar' at equal counts still passes.
3. truncate() turned a null into the string 'None', so a null was
indistinguishable from a column really holding 'None'. That is the
path analyze-tables takes, since it passes value_truncate=80.
num_distinct itself is unchanged: it still follows SQL and reports 0 for
an all-null column, which test_analyze_table_column_all_nulls pins. But
num_distinct can therefore be lower than len(most_common) when a column
holds nulls alongside all-distinct values - {'c': None, 'a', 'b'} gives
num_distinct 2 and three most_common entries. Whether null should count
as a distinct value is a question about what the number means, so I have
left it alone rather than change an output the tests already pin.
Tests: 6 new cases in tests/test_analyze.py, all failing on
|
||
|---|---|---|
| .. | ||
| sniff | ||
| __init__.py | ||
| conftest.py | ||
| ext.c | ||
| test_analyze.py | ||
| test_analyze_tables.py | ||
| test_atomic.py | ||
| test_attach.py | ||
| test_cli.py | ||
| test_cli_bulk.py | ||
| test_cli_convert.py | ||
| test_cli_insert.py | ||
| test_cli_memory.py | ||
| test_cli_migrate.py | ||
| test_column_affinity.py | ||
| test_column_casing.py | ||
| test_constructor.py | ||
| test_conversions.py | ||
| test_convert.py | ||
| test_create.py | ||
| test_create_table_parser.py | ||
| test_create_view.py | ||
| test_default_value.py | ||
| test_delete.py | ||
| test_docs.py | ||
| test_duplicate.py | ||
| test_enable_counts.py | ||
| test_extract.py | ||
| test_extracts.py | ||
| test_foreign_keys.py | ||
| test_fts.py | ||
| test_get.py | ||
| test_gis.py | ||
| test_hypothesis.py | ||
| test_insert_files.py | ||
| test_introspect.py | ||
| test_list_mode.py | ||
| test_lookup.py | ||
| test_m2m.py | ||
| test_migrations.py | ||
| test_mutator_transactions.py | ||
| test_plugins.py | ||
| test_query.py | ||
| test_recipes.py | ||
| test_recreate.py | ||
| test_register_function.py | ||
| test_rows.py | ||
| test_rows_from_file.py | ||
| test_sniff.py | ||
| test_suggest_column_types.py | ||
| test_tracer.py | ||
| test_transform.py | ||
| test_update.py | ||
| test_upsert.py | ||
| test_utils.py | ||
| test_wal.py | ||