Commit graph

9 commits

Author SHA1 Message Date
LeSingh1
625d004872
Fix progress bar against non-UTF-8 encodings
For #439.

`file_progress()` sets the progress bar length to the file size in
bytes (`os.path.getsize(file.name)`), but `UpdateWrapper.__iter__`
called `update(len(line))`, which is the decoded character count.
With UTF-16-LE input every character is 2 bytes, so the bar capped
at 50%; UTF-32 capped at 25%; etc.

Simon noted in the issue that the obvious fix (calling `.tell()` on
the wrapped text stream) doesn't work because text mode disables it
during iteration. The underlying binary buffer doesn't have that
restriction though, so this tracks progress against
`TextIOWrapper.buffer.tell()` when the wrapped object exposes one.

For raw binary streams (no `.buffer` attribute) we keep the old
behaviour, which was already byte-accurate.

Added six regression tests in tests/test_utils.py covering UTF-8,
UTF-16-LE, BOM-prefixed UTF-16, the sniff-style `BufferedReader` chain,
a raw binary fallback, and the `.read()` path used by the JSON loader.
Each asserts that the sum of update() calls equals the on-disk file
size, which is what `click.progressbar` needs to reach 100%.
2026-07-10 07:30:48 -07:00
Simon Willison
07b603e562
Preserve duplicate column names in query results
Queries returning duplicate column names - e.g. joins between tables
sharing column names - silently lost values because rows were built
with dict(zip(keys, row)), where the last duplicate wins.

Later occurrences are now renamed with a numeric suffix: id, id
becomes id, id_2 - skipping any suffix that would collide with a
real column in the same query.

The new utils.dedupe_keys() helper transforms the key list once per
query, so the per-row dict construction is unchanged and there is no
measurable performance impact.

Applied in Database.query() (including the PRAGMA and RETURNING
paths), Table.rows_where(), Table.search() and the CLI's JSON output.
CSV, TSV and table output keep the original duplicate headers.

Closes #624
2026-07-05 21:20:39 -07:00
Simon Willison
34e75ed0dd sqlite_utils.utils.flatten() function, closes #500 2022-10-18 11:00:35 -07:00
Simon Willison
9d1bac4a99 Test for maximize_csv_field_size_limit, refs #442 2022-06-14 14:23:54 -07:00
Simon Willison
931b1e1513 .insert(hash_id_columns=) parameter, closes #343 2022-03-01 16:00:51 -08:00
Chris Amico
ee11274fcb
New spatialite helper methods, closes #79
- db.init_spatialite()
- table.add_geometry_column()
- table.create_spatial_index()

Co-authored-by: Simon Willison <swillison@gmail.com>
2022-02-03 21:55:09 -08:00
Simon Willison
a8f9cc6f64 Add test for chunks(), refs #364 2022-01-08 13:16:52 -08:00
Simon Willison
bf4c6b7c82 find_spatialite() utility function, closes #135 2020-08-21 13:30:02 -07:00
Simon Willison
1a61a6d3d6 Ability to insert base64 binary data as JSON, closes #126 2020-07-26 20:59:34 -07:00