Type detection is the 4.0 default for CSV/TSV data, and the detected-type
transform ran even when the target table already existed - inserting a
CSV into a table with a TEXT zip column converted the column to INTEGER,
corrupting values with leading zeros ('01234' became 1234) with no
warning. Detected types now only apply to tables the command created.
Refs https://github.com/simonw/sqlite-utils/issues/769#issuecomment-4900034150
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- table.extract() and the sqlite-utils extract command now skip rows
where every extracted column is null: the new foreign key column is
left null and no all-null record is added to the lookup table. Rows
with at least one non-null extracted column are extracted as before.
- The extracts= option to insert() and friends keeps None values as
null instead of creating a lookup record for them - previously each
insert batch added a duplicate null row to the lookup table.
- table.lookup() compares values using IS rather than = so lookup
values containing None match existing rows correctly, instead of
inserting a duplicate row on every call.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Queries returning duplicate column names - e.g. joins between tables
sharing column names - silently lost values because rows were built
with dict(zip(keys, row)), where the last duplicate wins.
Later occurrences are now renamed with a numeric suffix: id, id
becomes id, id_2 - skipping any suffix that would collide with a
real column in the same query.
The new utils.dedupe_keys() helper transforms the key list once per
query, so the per-row dict construction is unchanged and there is no
measurable performance impact.
Applied in Database.query() (including the PRAGMA and RETURNING
paths), Table.rows_where(), Table.search() and the CLI's JSON output.
CSV, TSV and table output keep the original duplicate headers.
Closes#624
This now works:
echo '{"date": "13th January 2025"}' | sqlite-utils insert dates.db dates -
sqlite-utils convert dates.db dates date r.parsedate
sqlite-utils rows dates.db dates
Previously the command would have been:
sqlite-utils convert dates.db dates date 'r.parsedate(value)'
The `--detect-types` option is now automatically turned on for all commands that deal with CSV or CSV.
A new `--no-detect-types` option can be used to have all columns treated as text.
Closes#679
Closes#659
The --functions option now accepts:
- File paths ending in .py (e.g., --functions my_funcs.py)
- Multiple invocations (e.g., --functions foo.py --functions 'def bar(): ...')
- Inline Python code (existing behavior)
Implementation follows the same pattern as llm's --functions flag
(simonw/llm@a880c123).
Changes:
- Added multiple=True to --functions Click option in query, bulk, and memory commands
- Modified _register_functions() to detect and read .py files
- Updated _maybe_register_functions() to iterate over multiple function sources
- Removed unused bytes/bytearray handling
- Added comprehensive tests for file paths and multiple invocations
- Updated documentation with examples
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Shorter help for --functions
---------
Co-authored-by: Claude <noreply@anthropic.com>
* sqlite-utils tui command if Trogon is installed, closes#545
* Documentation for trogon TUI
* Screenshot of TUI
* Ignore trogon mypy error
* only run flake8 on Python 3.8 or higher, closes#550
