remote/sqlite-utils
1
0
Fork 0
mirror of https://github.com/simonw/sqlite-utils.git synced 2026-07-23 01:14:31 +02:00
Code Issues Projects Releases Packages Wiki Activity

extract() no longer duplicates NULL-containing rows in shared lookups

Browse source 
The lookup-table insert relied on INSERT OR IGNORE and the unique index
to dedupe against existing rows, but SQLite unique indexes treat NULLs
as distinct - extracting a second table into the same lookup table
re-inserted every NULL-containing value, growing orphan rows on each
extract. The insert now also has an IS-based NOT EXISTS guard, matching
how the foreign keys themselves are resolved.

Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Simon Willison 2026-07-06 21:55:21 -07:00
commit 8572d1e39c
3 changed files with 49 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

18
sqlite_utils/db.py
View file

@ -2919,8 +2919,22 @@ class Table(Queryable):
all_columns_are_null = " AND ".join(
"{} IS NULL".format(quote_identifier(c)) for c in columns
)
# INSERT OR IGNORE dedupes against the unique index, but unique
# indexes treat NULLs as distinct - the NOT EXISTS guard uses IS
# comparison so NULL-containing rows match existing lookup rows
# instead of being inserted again
already_in_lookup = " AND ".join(
"{lookup}.{lookup_col} IS {source}.{source_col}".format(
lookup=quote_identifier(table),
lookup_col=quote_identifier(rename.get(column) or column),
source=quote_identifier(self.name),
source_col=quote_identifier(column),
)
for column in columns
)
self.db.execute(
"INSERT OR IGNORE INTO {} ({lookup_columns}) SELECT DISTINCT {table_cols} FROM {} WHERE NOT ({all_null})".format(
"INSERT OR IGNORE INTO {} ({lookup_columns}) SELECT DISTINCT {table_cols} FROM {} "
"WHERE NOT ({all_null}) AND NOT EXISTS (SELECT 1 FROM {lookup} WHERE {already_in_lookup})".format(
quote_identifier(table),
quote_identifier(self.name),
lookup_columns=", ".join(
@ -2928,6 +2942,8 @@ class Table(Queryable):
),
table_cols=", ".join(quote_identifier(c) for c in columns),
all_null=all_columns_are_null,
lookup=quote_identifier(table),
already_in_lookup=already_in_lookup,
)
)