`ValueTracker.evaluate()` used `if not value` to skip NULL values, but
this also skipped empty strings (`""`), `0`, and other falsy values.
A CSV column containing only empty strings was incorrectly inferred as
INTEGER (the initial default) instead of TEXT.
Replace the falsy check with an explicit `if value is None` so that only
NULL is skipped and all other values — including `""` — are evaluated
against the type tests.
Queries returning duplicate column names - e.g. joins between tables
sharing column names - silently lost values because rows were built
with dict(zip(keys, row)), where the last duplicate wins.
Later occurrences are now renamed with a numeric suffix: id, id
becomes id, id_2 - skipping any suffix that would collide with a
real column in the same query.
The new utils.dedupe_keys() helper transforms the key list once per
query, so the per-row dict construction is unchanged and there is no
measurable performance impact.
Applied in Database.query() (including the PRAGMA and RETURNING
paths), Table.rows_where(), Table.search() and the CLI's JSON output.
CSV, TSV and table output keep the original duplicate headers.
Closes#624
* Add comprehensive type annotations
- mypy.ini: expanded configuration with module-specific settings
- hookspecs.py: type annotations for hook functions
- plugins.py: typed get_plugins() return value
- recipes.py: full type annotations for parsedate, parsedatetime, jsonsplit
- utils.py: extensive type annotations including Row type alias,
TypeTracker, ValueTracker, and all utility functions
- db.py: type annotations for Database methods (__exit__,
ensure_autocommit_off, tracer, register_function, etc.) and
Queryable class methods
- tests/test_docs.py: updated to match new signature display format
* Fix type errors caught by ty check
- Add type: ignore comments for external library type stub limitations
(csv.reader, click.progressbar, IOBase.name, Callable.__name__)
- Change Iterable to Sequence for SQL where_args parameters
- Use db.table() instead of db[name] for proper Table return type
- Fix rebuild_fts return type from None to Table
- Update test_tracer to expect fewer queries (optimization side effect)
* Fix mypy type errors
- Add type: ignore comments for runtime-valid patterns mypy can't verify
- Fix new_column_types annotation to Dict[str, Set[type]]
- Add type: ignore for Default sentinel values passed to create_table
* mypy skip tests directory
* Fix CI: exclude typing imports from recipe docs, skip mypy on tests
- Add Callable and Optional to exclusion list in _generate_convert_help()
- Regenerate docs/cli-reference.rst with cog
- Add [mypy-tests.*] ignore_errors = True to skip test type errors
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* Fix type warning for pipe.stdout possibly being None
Add conditional check before calling .read() on pipe.stdout since
Popen can return None for stdout.
* Use ctx.meta instead of dynamic attribute for database cleanup
Click's Context.meta dictionary is the proper way to store arbitrary
data on the context object, avoiding type checker warnings about
dynamic attribute assignment.
* Add assert for tables.callback before calling
Click's callback attribute is typed as Optional[Callable], so add
assert to satisfy type checker that it's not None.
* Fix type errors in cli.py and db.py
- Add type annotation for Database.conn to fix context manager errors
- Convert exception objects to str() when raising ClickException
- Handle None return from find_spatialite() with proper error message
* Fix remaining type errors in cli.py
- Add typing import and type annotations for dict kwargs
- Use db.table() instead of db[] for extract command
- Fix missing str() conversion for exception
* Fix type errors in db.py
- Add type annotation for Database.conn
- Add type: ignore for optional sqlite_dump import
- Update execute/query parameter types to Sequence|Dict for sqlite3 compatibility
- Use getattr for fn.__name__ access to handle callables without __name__
- Handle None return from find_spatialite() with OSError
- Fix pk_values assignment to use local variable
* Add type: ignore for optional pysqlite3 and sqlean imports
These are alternative sqlite3 implementations that may not be installed.
* Fix type errors in tests and plugins
- Add type: ignore for monkey-patching Database.__init__ in conftest
- Fix CLI test to pass string "2" instead of integer to Click invoke
- Add type: ignore for optional sqlean import
- Fix add_geometry_column test to use "XY" instead of integer 2
- Add type: ignore for click.Context as context manager
- Add type: ignore for enable_fts test that intentionally omits argument
- Add type: ignore for sys._called_from_test dynamic attribute
- Fix rows_from_file test type error for intentional wrong argument
- Handle None from pm.get_hookcallers in plugins.py
* Use db.table() instead of db[] for Table-specific operations
Changes db[table] to db.table(table) in CLI commands where we know
we're working with tables, not views. This resolves most of the
Table | View disambiguation type warnings since db.table() returns
Table directly rather than Table | View.
* Fix remaining type warnings in sqlite_utils package
- Add assert for sniff_buffer not being None
- Handle cursor.fetchone() potentially returning None
- Use db.table() for counts_table and index_foreign_keys
- Add type: ignore for cursor union type in raw mode
* Ran Black
* Run ty in CI
* ty check sqlite_utils
* Skip running ty on Windows
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
This now works:
echo '{"date": "13th January 2025"}' | sqlite-utils insert dates.db dates -
sqlite-utils convert dates.db dates date r.parsedate
sqlite-utils rows dates.db dates
Previously the command would have been:
sqlite-utils convert dates.db dates date 'r.parsedate(value)'
Closes#559Closes#235
Refs https://github.com/simonw/llm/issues/60
- Uses `sqlean` in place of `sqlite3` if `sqlean.py` is installed
- Uses `sqlite-dump` if available and `conn.iterdump()` does not exist
- New `with db.ensure_autocommit_off()` method for ensuring autocommit is off, used by `enable_wal()` and `disable_wal()`.
* progressbar for inserts/upserts of other file formats, closes#485
* Pin to Python 3.10.6 for the moment as workaround for mypy error
Co-authored-by: Simon Willison <swillison@gmail.com>
