remote/sqlite-utils
1
0
Fork 0
mirror of https://github.com/simonw/sqlite-utils.git synced 2026-10-30 06:57:50 +01:00
Code Issues Projects Releases Packages Wiki Activity

Preserve carriage returns in quoted CSV/TSV fields (#895)

Browse source 
When reading CSV or TSV files through rows_from_file() or the CLI insert command, io.TextIOWrapper defaulted to universal-newline mode (newline=None), causing carriage returns (\r\n or lone \r) embedded inside quoted fields to be silently translated to \n before csv.reader could process them.

Setting newline='' in TextIOWrapper preserves carriage returns in quoted fields according to the Python standard library csv module specification.

Fixes #895
This commit is contained in:
agustin18 2026-10-03 23:08:20 +00:00
commit c27092220d
4 changed files with 49 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

7
sqlite_utils/cli.py
View file

@ -1226,11 +1226,14 @@ def insert_upsert_implementation(
# The --sniff option needs us to buffer the file to peek ahead
sniff_buffer = None
decoded_buffer = None
newline = "" if (csv or tsv) else None
if sniff:
sniff_buffer = io.BufferedReader(file, buffer_size=4096)
decoded_buffer = io.TextIOWrapper(sniff_buffer, encoding=encoding)
decoded_buffer = io.TextIOWrapper(
sniff_buffer, encoding=encoding, newline=newline
)
else:
decoded_buffer = io.TextIOWrapper(file, encoding=encoding)
decoded_buffer = io.TextIOWrapper(file, encoding=encoding, newline=newline)
tracker = None
with file_progress(decoded_buffer, silent=silent) as decoded: