Preserve duplicate column names in query results

Queries returning duplicate column names - e.g. joins between tables
sharing column names - silently lost values because rows were built
with dict(zip(keys, row)), where the last duplicate wins.

Later occurrences are now renamed with a numeric suffix: id, id
becomes id, id_2 - skipping any suffix that would collide with a
real column in the same query.

The new utils.dedupe_keys() helper transforms the key list once per
query, so the per-row dict construction is unchanged and there is no
measurable performance impact.

Applied in Database.query() (including the PRAGMA and RETURNING
paths), Table.rows_where(), Table.search() and the CLI's JSON output.
CSV, TSV and table output keep the original duplicate headers.

Closes #624
This commit is contained in:
Simon Willison 2026-07-05 21:20:39 -07:00 committed by GitHub
commit 07b603e562
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
10 changed files with 128 additions and 4 deletions

View file

@ -34,6 +34,7 @@ from .utils import (
OperationalError,
_compile_code,
chunks,
dedupe_keys,
file_progress,
find_spatialite,
flatten as _flatten,
@ -3493,6 +3494,10 @@ FILE_COLUMNS = {
def output_rows(iterator, headers, nl, arrays, json_cols):
# Duplicate column names would collide as dictionary keys, so rename
# later occurrences id, id -> id, id_2 - CSV and table output keep
# the original duplicate headers since they never build dictionaries
headers = dedupe_keys(headers)
# We have to iterate two-at-a-time so we can know if we
# should output a trailing comma or if we have reached
# the last row.