Preserve duplicate column names in query results

Queries returning duplicate column names - e.g. joins between tables
sharing column names - silently lost values because rows were built
with dict(zip(keys, row)), where the last duplicate wins.

Later occurrences are now renamed with a numeric suffix: id, id
becomes id, id_2 - skipping any suffix that would collide with a
real column in the same query.

The new utils.dedupe_keys() helper transforms the key list once per
query, so the per-row dict construction is unchanged and there is no
measurable performance impact.

Applied in Database.query() (including the PRAGMA and RETURNING
paths), Table.rows_where(), Table.search() and the CLI's JSON output.
CSV, TSV and table output keep the original duplicate headers.

Closes #624
This commit is contained in:
Simon Willison 2026-07-05 21:20:39 -07:00 committed by GitHub
commit 07b603e562
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
10 changed files with 128 additions and 4 deletions

View file

@ -1,5 +1,6 @@
from .utils import (
chunks,
dedupe_keys,
hash_record,
sqlite3,
OperationalError,
@ -786,7 +787,7 @@ class Database:
cursor = self.conn.execute(sql, *args)
if cursor.description is None:
raise ValueError(message)
keys = [d[0] for d in cursor.description]
keys = dedupe_keys(d[0] for d in cursor.description)
return (dict(zip(keys, row)) for row in cursor)
# Execute inside a savepoint, so a statement that turns out not to
# return rows can be rolled back before the ValueError is raised
@ -796,7 +797,7 @@ class Database:
cursor = self.conn.execute(sql, *args)
if cursor.description is None:
raise ValueError(message)
keys = [d[0] for d in cursor.description]
keys = dedupe_keys(d[0] for d in cursor.description)
try:
self.conn.execute('RELEASE "sqlite_utils_query"')
released = True
@ -1865,7 +1866,7 @@ class Queryable:
if offset is not None:
sql += " offset {}".format(offset)
cursor = self.db.execute(sql, where_args or [])
columns = [c[0] for c in cursor.description]
columns = dedupe_keys(c[0] for c in cursor.description)
for row in cursor:
yield dict(zip(columns, row))
@ -3398,7 +3399,7 @@ class Table(Queryable):
),
args,
)
columns = [c[0] for c in cursor.description]
columns = dedupe_keys(c[0] for c in cursor.description)
for row in cursor:
yield dict(zip(columns, row))