Commit graph

215 commits

Author SHA1 Message Date
Claude
6c88067ab7
Explain why the Database context manager rolls back instead of committing
Expands the closing-a-database docs: exit is equivalent to close(),
open transactions can only come from manual transaction control,
auto-committing could silently persist half-finished work, and the
behavior deliberately differs from sqlite3.Connection's own context
manager.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UnLnhsH25Nnv7LHhekUfPd
2026-07-04 22:25:28 +00:00
Claude
70055d7432
Document the transaction model as a first-class concept
New top-level 'Transactions and saving your changes' section in the
Python API documentation, stating the fundamental contract up
front: every write method commits its own changes before returning,
no commit() call is ever needed. The db.atomic() documentation
moves under it, alongside new subsections covering the two cases
where the user owns the commit - raw db.execute() writes and
manually managed transactions - and the supported connection modes.

Cross-referenced from Getting started (the first insert example now
says the rows are saved immediately), a warning on db.execute(),
the closing-a-database section and the upgrading guide.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UnLnhsH25Nnv7LHhekUfPd
2026-07-04 20:16:15 +00:00
Claude
fd867282b3
Document atomic() deferred BEGIN, savepoint behavior and connection modes
Documents that atomic() opens a deferred BEGIN, that calling it
while the connection is already inside a transaction produces a
savepoint whose changes only persist when the outer transaction
commits, and that Python 3.12+ autocommit=True/False connections
are not supported. Adds a test pinning the savepoint-inside-manual-
BEGIN behavior.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UnLnhsH25Nnv7LHhekUfPd
2026-07-04 19:06:44 +00:00
Claude
a6fcb5af62
Pin the Database context manager contract
with Database(...) closes the connection on exit without
committing, so uncommitted changes are rolled back. This was the
existing behavior but was undocumented and untested - it is now
stated in the docs and pinned by a test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UnLnhsH25Nnv7LHhekUfPd
2026-07-04 19:05:59 +00:00
Claude
ffec11cfb7
enable_wal() and disable_wal() refuse to run inside a transaction
Changing the journal mode assigns conn.isolation_level, which
commits any open transaction as a side effect - silently breaking
the rollback guarantee of atomic() blocks and of user-managed
transactions. Both methods now raise RuntimeError if a transaction
is open. Calling them when the database is already in the requested
mode remains a no-op.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UnLnhsH25Nnv7LHhekUfPd
2026-07-04 18:46:09 +00:00
Claude
d6753a9f57
upsert() now requires a value for every primary key column
A record missing a primary key value - or with None for one - can
never match an existing row, because a NULL primary key never
satisfies ON CONFLICT. Such records were previously inserted as
brand new rows (silently, for multi-record upsert_all calls) or
triggered a KeyError after the insert had already happened. Both
cases, including upserting an empty record, now raise
PrimaryKeyRequired before any SQL is executed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UnLnhsH25Nnv7LHhekUfPd
2026-07-04 18:43:37 +00:00
Claude
45af93d463
db.query() now executes its SQL immediately
Previously query() was a generator function, so nothing - including
the SQL itself - ran until the result was first iterated. A write
statement passed to query() silently did nothing, and SQL errors
surfaced far from the call site. The SQL now executes as soon as
query() is called, while rows are still fetched lazily during
iteration.

Statements that return no rows now raise a ValueError directing
callers to execute() instead. As a side effect, statements like
INSERT ... RETURNING now work naturally with query().

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UnLnhsH25Nnv7LHhekUfPd
2026-07-04 18:24:52 +00:00
Claude
b17c37144e
Fix delete_where() leaving the connection in an open transaction
Table.delete_where() ran its DELETE via a bare execute() with no
commit, unlike Table.delete() which wraps in db.atomic(). The
connection was left with an open implicit transaction, so the
deletion (and all subsequent writes, including later atomic()
blocks which switched to savepoint mode) was silently rolled back
when the connection closed.

Wrap the DELETE in db.atomic() and remove the now-unnecessary
atomic() wrapper from the delete_where documentation example.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UnLnhsH25Nnv7LHhekUfPd
2026-07-04 17:57:35 +00:00
Simon Willison
6729ea3f60
db.atomic() method
Closes #755
2026-06-21 10:43:19 -07:00
Simon Willison
fd5b09f64b
Database as a context manager, fixed many pytest warnings
* Database can now work as a context manager
* Claude Code helped fix a ton of .close() warnings

https://gistpreview.github.io/?730f0c5dc38528a1dd0615f330bd5481

* New autouse fixture to help with test warnings

Refs https://github.com/simonw/sqlite-utils/issues/692#issuecomment-3644371889

* Fix all remaining resource warnings

https://gistpreview.github.io/?0bb8e869b82f6ff0db647de755182502

Closes #692
2025-12-11 16:56:12 -08:00
Simon Willison
c872d27bb3
Use REAL not FLOAT as SQLite column type (#680)
* Use REAL not FLOAT as SQLite column type, refs #645
* Fix for REAL columns by CSV --detect-types

Refs https://github.com/simonw/sqlite-utils/issues/645#issuecomment-3568947189

* Removed note about strict and REAL
2025-11-23 21:37:59 -08:00
Simon Willison
fb93452ea8
Use double quotes not braces for tables and columns (#678)
Closes #677
2025-11-23 20:43:26 -08:00
Simon Willison
96fab69256 Removed convert skip_false and --skip-false, closes #542 2025-11-23 15:40:28 -08:00
Simon Willison
7ffd5052e9 table.insert_all() and table.upsert_all() now take generators of lists/tuples
Closes #672, PR #673
2025-11-23 12:17:23 -08:00
Simon Willison
094b010fd8
db.table() only returns tables, added db.view()
* db.table() only returns tables, added db.view(), closes #657
* Massive documentation update for db.table()

Refs #656
2025-05-08 23:19:36 -07:00
Simon Willison
8e7d018fa2
Use ON CONFLICT for upsert, refs #652
* New upsert implementation, refs #652
* supports_strict now caches on self._supports_strict

PR: https://github.com/simonw/sqlite-utils/pull/653
2025-05-08 20:37:49 -07:00
Simon Willison
04107d3fea Remove note from docs about 3.8 and deterministic, closes #646 2024-11-23 14:49:15 -08:00
Simon Willison
4dc2e2e9c8 Use REAL for floating point columns if table is strict, closes #644
Refs #645
2024-11-23 14:27:21 -08:00
liunux4odoo
7423296ec7
include_rank parameter for Table.search
* Add `include_rank` parameter to `Table.search`
* Test for .search(include_rank)
* Docs for table.search(include_rank)

https://github.com/simonw/sqlite-utils/pull/628

Refs #480

---------

Co-authored-by: Simon Willison <swillison@gmail.com>
2024-11-23 12:34:27 -08:00
Mat Miller
42230709f7
Recreate indexes when calling transform when possible (#634)
* Recreate indexes when calling transform when possible and raise an error when they cannot be retained automatically
* Docs for sqlite_utils.db.TransformError

Co-authored-by: Simon Willison <swillison@gmail.com>
2024-11-23 12:17:15 -08:00
Thomas
896411099e
Documented the use of delete_where (#630)
Documented the use of delete_where, as shown in https://github.com/simonw/sqlite-utils/issues/159
2024-07-18 11:34:24 -07:00
Simon Willison
5bd7aec4d2
Test against Python 3.13 pre-release (#619)
* Test against Python 3.13 pre-release

* Skip tests for numpy on Python 3.13

Refs https://github.com/simonw/sqlite-utils/pull/619#issuecomment-1998798451

* Try to avoid Python 3.13 cog differences

* Hide \b characters in cli-reference

* Fixed .rST warning
2024-03-14 21:14:10 -07:00
Simon Willison
f29189a3dd db.supports_strict docs, closes #344 2023-12-07 21:22:27 -08:00
Taj Khattra
1500c19bd0
Add more STRICT table support (#604)
* Add more STRICT table support per https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982014776.
* Make `table.transform()` preserve STRICT mode.
* Fix mypy failures in PR #604
* Link to SQLITE strict page in a few places
2023-12-07 21:05:27 -08:00
Simon Willison
b2e0cd066d Test and docs for timedelta support, refs #522 2023-11-03 17:56:34 -07:00
Simon Willison
622c3a5a7d Fixed Markdown that should be rST 2023-09-08 18:25:46 -07:00
Simon Willison
5d123f031f
Fixed bug in replacing foreign key constraints example 2023-09-06 15:33:06 -07:00
Simon Willison
98cd11a81b
Link docs to sqlite-utils-fast-fks 2023-08-18 10:45:12 -07:00
Simon Willison
509857ee87
.add_foreign_keys() uses .transform() instead of PRAGMA writable_schema
Closes #577

This should solve all sorts of problems seen by users of platforms that throw errors on writable_schema.

Also added `add_foreign_keys=` and `foreign_keys=` parameters to `table.transform()`.
2023-08-17 17:48:08 -07:00
Simon Willison
1dc6b5aa64 Note about execute_plugins=False, refs #575 2023-08-07 11:47:34 -07:00
Simon Willison
61aaa69815 .transform(..., keep_table=name) parameter, closes #571
Also type hints for the transform_sql() method
2023-07-22 15:32:09 -07:00
Simon Willison
2c12c01346 Caling .transform() also reformats the schema, closes #564 2023-07-22 15:15:30 -07:00
Simon Willison
82e8cd3667 db.rename_table() method, refs #565 2023-07-22 12:38:31 -07:00
Simon Willison
80b5fa7f12 Documentation for replace=True/ignore=True, refs #568 2023-07-22 12:19:01 -07:00
Simon Willison
f5c63088e1
Use sqlean if available in environment (#560)
Closes #559
Closes #235

Refs https://github.com/simonw/llm/issues/60

- Uses `sqlean` in place of `sqlite3` if `sqlean.py` is installed
- Uses `sqlite-dump` if available and `conn.iterdump()` does not exist
- New `with db.ensure_autocommit_off()` method for ensuring autocommit is off, used by `enable_wal()` and `disable_wal()`.
2023-06-25 16:25:51 -07:00
Simon Willison
d2a7b15b2b
Analyze tables options: --common-limit, --no-most, --no-least
Closes #544
2023-05-21 09:19:30 -07:00
Simon Willison
e0ec4c3451 --no-skip-false option, plus docs - closes #527 2023-05-08 14:03:20 -07:00
Simon Willison
365f62520f will, not may - refs #468 2022-08-27 20:38:30 -07:00
Simon Willison
104f37fa4d
db[table].create(..., transform=True) and create-table --transform
Closes #467
2022-08-27 16:17:55 -07:00
Simon Willison
36ffcafb1a table.default_values property, closes #475
Refs #468
2022-08-27 15:42:58 -07:00
Simon Willison
1491b66dd7 register_function(name=...) argument, closes #458 2022-07-27 17:13:49 -07:00
Simon Willison
c710ade644 sqlite-utils duplicate command, closes #454, refs #449
Also made it so .duplicate() raises new NoTable exception rather than raising
an AssertionError if the source table does not exist.
2022-07-15 14:45:47 -07:00
Simon Willison
da030d49fd Documentation for .duplicate(), refs #449 2022-07-15 14:29:52 -07:00
Simon Willison
773f2b6b20 Documented TypeTracker, closes #445 2022-06-20 12:46:49 -07:00
Simon Willison
1b09538bc6 where= and where_args= parameters to search() and search_sql()
Closes #441
2022-06-14 14:54:35 -07:00
Simon Willison
0b6aba696d Documentation for maximize_csv_field_size_limit, closes #442 2022-06-14 14:31:45 -07:00
Simon Willison
ce670e2d44 Docs for rows_from_file
Closes #440, closes #443
2022-06-14 13:12:42 -07:00
Simon Willison
397183debd Switch docs theme to Furo, refs #435 2022-05-20 14:52:00 -07:00
Simon Willison
7f56f90d30
Fixed rST mistake 2022-03-01 23:01:07 -08:00
Simon Willison
d25cdd37a3 db.sqlite_version property and fix for deterministic=True on SQLite 3.8.3
Closes #408
2022-03-01 16:37:53 -08:00