remote/sqlite-utils
1
0
Fork 0
mirror of https://github.com/simonw/sqlite-utils.git synced 2026-09-19 23:14:09 +02:00
Code Issues Projects Releases Packages Wiki Activity

Don't extract NULL values into a lookup row (#186)

Browse source 
`table.extract()` built its lookup table with
`INSERT OR IGNORE ... SELECT DISTINCT <cols> FROM <table>`, which included
the all-NULL combination. That created a spurious lookup row for NULL and
pointed every NULL source row at it, instead of leaving those rows with a
NULL foreign key.

Before:
    db["creatures"].extract("type")
    # type lookup: [{"id": 1, "type": None}, {"id": 2, "type": "dog"}]
    # creatures:   Simon -> type_id=1, Natalie -> type_id=1, Cleo -> type_id=2

After:
    # type lookup: [{"id": 1, "type": "dog"}]
    # creatures:   Simon -> type_id=None, Natalie -> type_id=None, Cleo -> type_id=1

A row whose extracted columns are entirely NULL represents "no value", so it
now keeps a NULL foreign key and no lookup row is created for it. The fix adds
a `WHERE NOT (<col> IS NULL AND ...)` guard to the lookup INSERT; the existing
`IS`-based foreign-key UPDATE then leaves those rows NULL automatically (the
subquery finds no matching lookup row).

For multi-column extracts, only the fully-NULL combination is skipped — a
partial-NULL combination (some extracted columns set, others NULL) is a
genuine distinct value and is still extracted and shared between matching rows.

Updates test_extract_works_with_null_values to assert the corrected behaviour
and adds regression tests for the single-column and multi-column cases.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Johnson K C 2026-06-07 16:23:58 -04:00
commit dad463e9c2
2 changed files with 55 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

10
sqlite_utils/db.py
View file

@ -2267,14 +2267,22 @@ class Table(Queryable):
)
lookup_columns = [(rename.get(col) or col) for col in columns]
lookup_table.create_index(lookup_columns, unique=True, if_not_exists=True)
# Don't create a lookup row for the all-NULL combination: a row whose
# extracted columns are entirely NULL represents "no value", so it
# should keep a NULL foreign key rather than point at a NULL lookup row
# (#186). Rows with a partial NULL (some extracted columns set, others
# NULL) are a genuine distinct value and are still extracted.
self.db.execute(
"INSERT OR IGNORE INTO {} ({lookup_columns}) SELECT DISTINCT {table_cols} FROM {}".format(
"INSERT OR IGNORE INTO {} ({lookup_columns}) SELECT DISTINCT {table_cols} FROM {} WHERE NOT ({all_null})".format(
quote_identifier(table),
quote_identifier(self.name),
lookup_columns=", ".join(
quote_identifier(c) for c in lookup_columns
),
table_cols=", ".join(quote_identifier(c) for c in columns),
all_null=" AND ".join(
"{} IS NULL".format(quote_identifier(c)) for c in columns
),
)
)