A record missing a primary key value - or with None for one - can
never match an existing row, because a NULL primary key never
satisfies ON CONFLICT. Such records were previously inserted as
brand new rows (silently, for multi-record upsert_all calls) or
triggered a KeyError after the insert had already happened. Both
cases, including upserting an empty record, now raise
PrimaryKeyRequired before any SQL is executed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UnLnhsH25Nnv7LHhekUfPd
Both ran their INSERT INTO fts(fts) statements via a bare execute()
with no commit, leaving the connection inside an open implicit
transaction - the FTS operation and all subsequent writes were then
silently rolled back when the connection closed. Both are now
wrapped in db.atomic(), matching delete_where() and the other
write operations.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UnLnhsH25Nnv7LHhekUfPd
Previously query() was a generator function, so nothing - including
the SQL itself - ran until the result was first iterated. A write
statement passed to query() silently did nothing, and SQL errors
surfaced far from the call site. The SQL now executes as soon as
query() is called, while rows are still fetched lazily during
iteration.
Statements that return no rows now raise a ValueError directing
callers to execute() instead. As a side effect, statements like
INSERT ... RETURNING now work naturally with query().
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UnLnhsH25Nnv7LHhekUfPd
drop-table on a view name silently dropped the view, and drop-view
on a table name silently dropped the table, because both used
db[name].drop() which dispatches on the actual object type. They
now use db.table() / db.view() and exit with an explanatory error
pointing at the correct command. --ignore still exits cleanly but
no longer drops anything.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UnLnhsH25Nnv7LHhekUfPd
Migrations.apply() now wraps each migration function and its
_sqlite_migrations tracking row in db.atomic(), so they commit
together: a failing migration rolls back cleanly, is not recorded,
and stays pending, eliminating the double-apply hazard where a
partially-failed migration left committed side effects behind.
Migrations that cannot run inside a transaction (VACUUM, journal
mode changes, manual transaction management) can opt out by
registering with @migrations(transactional=False).
Also fixed the test fixtures which used db.query() for INSERT
statements - query() is a lazy generator, so those inserts never
actually executed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UnLnhsH25Nnv7LHhekUfPd
Table.delete_where() ran its DELETE via a bare execute() with no
commit, unlike Table.delete() which wraps in db.atomic(). The
connection was left with an open implicit transaction, so the
deletion (and all subsequent writes, including later atomic()
blocks which switched to savepoint mode) was silently rolled back
when the connection closed.
Wrap the DELETE in db.atomic() and remove the now-unnecessary
atomic() wrapper from the delete_where documentation example.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UnLnhsH25Nnv7LHhekUfPd
Assessment of the CREATE TABLE parser as the foundation for 4.1
transform() improvements: fit for 4.1, two tokenizer bugs to fix
first (comments, numeric literal defaults), test wiring, and
design decisions around round-trip fidelity and API visibility.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UnLnhsH25Nnv7LHhekUfPd
Consolidated findings from a final review before the stable 4.0
release: five release blockers, semantic decisions to lock in now,
polish items, and verified-sound areas.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UnLnhsH25Nnv7LHhekUfPd
* fix: defer plugin entrypoint loading until runtime
Fixessimonw/sqlite-utils#713
* Lazy-load plugins when listing them
Keep plugin entrypoints from loading at module import time, but preserve the existing behavior of get_plugins() by loading entrypoints the first time plugins are listed outside tests.
---------
Co-authored-by: Simon Willison <swillison@gmail.com>
When inserting a CSV file that contains only a header row (no data rows),
the --detect-types option would crash with an AssertionError because it
tried to transform a table that didn't exist.
This fix adds a check to ensure the table exists before attempting to
apply type transformations. The fix is applied in two places:
1. insert_upsert_implementation() for the insert command
2. memory() command for CSV/TSV files
Added test case: test_insert_csv_headers_only
Co-authored-by: Test User <test@example.com>
* Add comprehensive type annotations
- mypy.ini: expanded configuration with module-specific settings
- hookspecs.py: type annotations for hook functions
- plugins.py: typed get_plugins() return value
- recipes.py: full type annotations for parsedate, parsedatetime, jsonsplit
- utils.py: extensive type annotations including Row type alias,
TypeTracker, ValueTracker, and all utility functions
- db.py: type annotations for Database methods (__exit__,
ensure_autocommit_off, tracer, register_function, etc.) and
Queryable class methods
- tests/test_docs.py: updated to match new signature display format
* Fix type errors caught by ty check
- Add type: ignore comments for external library type stub limitations
(csv.reader, click.progressbar, IOBase.name, Callable.__name__)
- Change Iterable to Sequence for SQL where_args parameters
- Use db.table() instead of db[name] for proper Table return type
- Fix rebuild_fts return type from None to Table
- Update test_tracer to expect fewer queries (optimization side effect)
* Fix mypy type errors
- Add type: ignore comments for runtime-valid patterns mypy can't verify
- Fix new_column_types annotation to Dict[str, Set[type]]
- Add type: ignore for Default sentinel values passed to create_table
* mypy skip tests directory
* Fix CI: exclude typing imports from recipe docs, skip mypy on tests
- Add Callable and Optional to exclusion list in _generate_convert_help()
- Regenerate docs/cli-reference.rst with cog
- Add [mypy-tests.*] ignore_errors = True to skip test type errors
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* Fix type warning for pipe.stdout possibly being None
Add conditional check before calling .read() on pipe.stdout since
Popen can return None for stdout.
* Use ctx.meta instead of dynamic attribute for database cleanup
Click's Context.meta dictionary is the proper way to store arbitrary
data on the context object, avoiding type checker warnings about
dynamic attribute assignment.
* Add assert for tables.callback before calling
Click's callback attribute is typed as Optional[Callable], so add
assert to satisfy type checker that it's not None.
* Fix type errors in cli.py and db.py
- Add type annotation for Database.conn to fix context manager errors
- Convert exception objects to str() when raising ClickException
- Handle None return from find_spatialite() with proper error message
* Fix remaining type errors in cli.py
- Add typing import and type annotations for dict kwargs
- Use db.table() instead of db[] for extract command
- Fix missing str() conversion for exception
* Fix type errors in db.py
- Add type annotation for Database.conn
- Add type: ignore for optional sqlite_dump import
- Update execute/query parameter types to Sequence|Dict for sqlite3 compatibility
- Use getattr for fn.__name__ access to handle callables without __name__
- Handle None return from find_spatialite() with OSError
- Fix pk_values assignment to use local variable
* Add type: ignore for optional pysqlite3 and sqlean imports
These are alternative sqlite3 implementations that may not be installed.
* Fix type errors in tests and plugins
- Add type: ignore for monkey-patching Database.__init__ in conftest
- Fix CLI test to pass string "2" instead of integer to Click invoke
- Add type: ignore for optional sqlean import
- Fix add_geometry_column test to use "XY" instead of integer 2
- Add type: ignore for click.Context as context manager
- Add type: ignore for enable_fts test that intentionally omits argument
- Add type: ignore for sys._called_from_test dynamic attribute
- Fix rows_from_file test type error for intentional wrong argument
- Handle None from pm.get_hookcallers in plugins.py
* Use db.table() instead of db[] for Table-specific operations
Changes db[table] to db.table(table) in CLI commands where we know
we're working with tables, not views. This resolves most of the
Table | View disambiguation type warnings since db.table() returns
Table directly rather than Table | View.
* Fix remaining type warnings in sqlite_utils package
- Add assert for sniff_buffer not being None
- Handle cursor.fetchone() potentially returning None
- Use db.table() for counts_table and index_foreign_keys
- Add type: ignore for cursor union type in raw mode
* Ran Black
* Run ty in CI
* ty check sqlite_utils
* Skip running ty on Windows
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
This now works:
echo '{"date": "13th January 2025"}' | sqlite-utils insert dates.db dates -
sqlite-utils convert dates.db dates date r.parsedate
sqlite-utils rows dates.db dates
Previously the command would have been:
sqlite-utils convert dates.db dates date 'r.parsedate(value)'
The `--detect-types` option is now automatically turned on for all commands that deal with CSV or CSV.
A new `--no-detect-types` option can be used to have all columns treated as text.
Closes#679
Closes#659
The --functions option now accepts:
- File paths ending in .py (e.g., --functions my_funcs.py)
- Multiple invocations (e.g., --functions foo.py --functions 'def bar(): ...')
- Inline Python code (existing behavior)
Implementation follows the same pattern as llm's --functions flag
(simonw/llm@a880c123).
Changes:
- Added multiple=True to --functions Click option in query, bulk, and memory commands
- Modified _register_functions() to detect and read .py files
- Updated _maybe_register_functions() to iterate over multiple function sources
- Removed unused bytes/bytearray handling
- Added comprehensive tests for file paths and multiple invocations
- Updated documentation with examples
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Shorter help for --functions
---------
Co-authored-by: Claude <noreply@anthropic.com>
When configuration parameters like pk, foreign_keys, not_null, etc.
are passed to Table.create(), they are now stored in self._defaults
so that subsequent operations on the same Table instance (like insert,
upsert) will use those settings automatically.
Previously, calling db["table"].insert({...}, pk="id") would create
the table correctly but subsequent calls to insert() or upsert() on
the same table object would not know about the pk setting, causing
issues like upsert() failing with "requires a pk" error.
Closes#655🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
