Preserve duplicate column names in query results

Queries returning duplicate column names - e.g. joins between tables
sharing column names - silently lost values because rows were built
with dict(zip(keys, row)), where the last duplicate wins.

Later occurrences are now renamed with a numeric suffix: id, id
becomes id, id_2 - skipping any suffix that would collide with a
real column in the same query.

The new utils.dedupe_keys() helper transforms the key list once per
query, so the per-row dict construction is unchanged and there is no
measurable performance impact.

Applied in Database.query() (including the PRAGMA and RETURNING
paths), Table.rows_where(), Table.search() and the CLI's JSON output.
CSV, TSV and table output keep the original duplicate headers.

Closes #624
This commit is contained in:
Simon Willison 2026-07-05 21:20:39 -07:00 committed by GitHub
commit 07b603e562
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
10 changed files with 128 additions and 4 deletions

View file

@ -746,6 +746,26 @@ def test_query_json_empty(db_path):
assert result.output.strip() == "[]"
def test_query_json_duplicate_columns_are_deduped(db_path):
# https://github.com/simonw/sqlite-utils/issues/624
result = CliRunner().invoke(
cli.cli,
[db_path, "select 1 as id, 2 as id, 'x' as value, 'y' as value"],
)
assert result.output.strip() == (
'[{"id": 1, "id_2": 2, "value": "x", "value_2": "y"}]'
)
def test_query_csv_duplicate_columns_are_preserved(db_path):
# CSV output should keep the duplicate headers, not rename them
result = CliRunner().invoke(
cli.cli,
[db_path, "select 1 as id, 2 as id", "--csv"],
)
assert result.output.replace("\r", "").strip() == "id,id\n1,2"
def test_query_invalid_function(db_path):
result = CliRunner().invoke(
cli.cli, [db_path, "select bad()", "--functions", "def invalid_python"]