remote/sqlite-utils
1
0
Fork 0
mirror of https://github.com/simonw/sqlite-utils.git synced 2026-07-24 09:54:31 +02:00
Code Issues Projects Releases Packages Wiki Activity

Merge remote-tracking branch 'origin/main' into say-5/document-memory-attrs

Browse source 
# Conflicts:
#	tests/test_constructor.py
This commit is contained in:
Simon Willison 2026-07-07 19:34:36 -07:00
commit e38d701fb1
62 changed files with 7227 additions and 816 deletions
Download patch file Download diff file Expand all files Collapse all files

166
docs/changelog.rst
View file

@ -4,14 +4,174 @@
Changelog
===========
.. _v_unreleased:
Unreleased
----------
- ``sqlite-utils query`` can now read the SQL query from standard input by passing ``-`` in place of the query, for example ``echo "select * from dogs" | sqlite-utils query dogs.db -``. (:issue:`765`)
- ``sqlite-utils insert`` and ``sqlite-utils upsert`` now accept a ``--code`` option for :ref:`providing a block of Python code <cli_insert_code>` (or a path to a ``.py`` file) that defines a ``rows()`` function or ``rows`` iterable of rows to insert, as an alternative to importing from a file. (:issue:`684`)
- ``sqlite-utils insert`` and ``sqlite-utils upsert`` now accept ``--type column-name type`` to :ref:`override the type automatically chosen when the table is created <cli_insert_csv_tsv_column_types>`. This is useful for CSV or TSV columns such as ZIP codes that look like integers but should be stored as ``TEXT`` to preserve leading zeros. (:issue:`131`)
.. _v4_0:
4.0 (2026-07-07)
----------------
The 4.0 release includes some minor backwards-incompatible fixes (hence the major version number bump) and introduces three major new features:
- :ref:`Database migrations <migrations>`, providing a structured mechanism for evolving a project's schema over time. (:issue:`752`)
- :ref:`Nested transaction support <python_api_atomic>` via ``db.atomic()``, plus numerous improvements to how transactions work across the library. (:issue:`755`)
- Support for :ref:`compound foreign keys <python_api_compound_foreign_keys>`, including creation, transformation and introspection through :ref:`table.foreign_keys <python_api_introspection_foreign_keys>`. (:issue:`594`)
Other notable changes include:
- Upserts now use SQLite's ``INSERT ... ON CONFLICT ... DO UPDATE SET`` syntax, detect existing table primary keys automatically and reject records that are missing required primary key values. (:issue:`652`)
- ``db.query()`` now executes immediately and rejects statements that do not return rows; use ``db.execute()`` for writes and DDL.
- CSV and TSV imports now detect column types by default, while inserts into existing tables preserve those tables' column types. (:issue:`679`)
- Foreign key handling now preserves ``ON DELETE``/``ON UPDATE`` actions during transforms and resolves referenced primary keys more accurately. (:issue:`530`)
- Column names passed to Python API methods are now matched case-insensitively, mirroring SQLite's own identifier behavior. (:issue:`760`)
- The command-line tool now emits UTF-8 JSON output by default, with ``--ascii`` available to restore escaped output. (:issue:`625`)
- ``table.extract()`` and ``extracts=`` no longer create lookup table records for all-``null`` values. (:issue:`186`)
See :ref:`upgrading_3_to_4` for details on backwards-incompatible changes.
The detailed release notes for the features and fixes shipped during the 4.0 pre-release cycle are available in :ref:`4.0a0 <v4_0a0>`, :ref:`4.0a1 <v4_0a1>`, :ref:`4.0rc1 <v4_0rc1>`, :ref:`4.0rc2 <v4_0rc2>`, :ref:`4.0rc3 <v4_0rc3>` and :ref:`4.0rc4 <v4_0rc4>`.
Bug fixes since 4.0rc4
~~~~~~~~~~~~~~~~~~~~~~
- Fixed 4.0 regressions in ``insert``/``upsert`` against tables that use SQLite's implicit ``rowid`` primary key. Passing ``pk="rowid"``, ``pk="_rowid_"`` or ``pk="oid"`` now works again for rowid tables, and ``last_pk`` is set correctly. (:issue:`781`)
- Fixed ``insert(..., ignore=True)`` and ``insert_all(..., ignore=True)`` so an ignored insert that conflicts with an existing primary key row now reports that existing row in ``last_rowid`` and ``last_pk`` where possible. This also works for compound primary keys and list-mode inserts. (:issue:`783`)
.. _v4_0rc4:
4.0rc4 (2026-07-06)
-------------------
- **Breaking change**: ``table.extract()`` - and the ``sqlite-utils extract`` command - no longer extract rows where every extracted column is ``null``. Those rows now keep a ``null`` value in the new foreign key column instead of pointing at an all-``null`` record in the lookup table. When extracting multiple columns, rows are still extracted if at least one of the columns has a value. (:issue:`186`)
- The ``extracts=`` option to ``table.insert()`` and friends no longer creates a lookup table record for ``None`` values - the column value stays ``null``. Previously every batch of inserted rows containing a ``None`` value would add a duplicate ``null`` record to the lookup table.
- Fixed a bug where ``table.lookup()`` inserted a duplicate row on every call if any of the lookup values were ``None``. Lookup values are now compared using ``IS`` so that ``None`` values match existing rows correctly.
- JSON output from the command-line tool no longer escapes non-ASCII characters, so ``sqlite-utils data.db "select '日本語' as text"`` now outputs ``[{"text": "日本語"}]``. This matches how values were already stored by ``insert`` and how CSV/TSV output already behaved. A new ``--ascii`` option restores the previous behavior of escaping non-ASCII characters, for output destinations that cannot handle UTF-8 - see :ref:`cli_query_json_ascii`. The option is available on the ``query``, ``rows``, ``search``, ``tables``, ``views``, ``triggers``, ``indexes`` and ``memory`` commands. The ``convert --multi --dry-run`` preview and ``plugins`` output also no longer escape non-ASCII characters. (:issue:`625`)
- ``--no-headers`` now omits the header row from ``--fmt`` and ``--table`` output, not just CSV and TSV output. (:issue:`566`)
- ``table.insert_all(..., pk=...)`` now raises ``InvalidColumns`` if ``pk=`` names columns that do not exist in an existing table. Previously this behaved inconsistently, with single-row inserts raising a ``KeyError`` while other row counts succeeded. (:issue:`732`)
- Fixed an ``IndexError`` from ``table.insert(..., pk=..., ignore=True)`` when an ignored insert followed writes to another table on the same connection. ``last_pk`` is now populated from the explicit primary key value instead of looking up a stale ``lastrowid``. (:issue:`554`)
- Fixed a bug where a failed write statement executed with ``db.execute()`` left the driver's implicit transaction open. Every subsequent write then joined that phantom transaction, which nothing committed, so their work was silently rolled back when the connection was closed. The implicit transaction opened by a failed statement is now rolled back before the exception is raised. A failed write inside a transaction opened with ``db.begin()`` or ``db.atomic()`` leaves that transaction open and untouched, as before.
- Fixed a bug where transaction-control statements prefixed with an empty statement - ``db.query("; COMMIT")`` - or a UTF-8 byte order mark slipped past the check that rejects them, committing the caller's open transaction before raising a confusing ``OperationalError``. The keyword scanner used by ``db.query()`` and ``db.execute()`` now skips leading ``;`` and byte order marks, matching what the ``sqlite3`` driver tolerates before the first token, so these statements are rejected with a ``ValueError`` without being executed. The same fix means ``db.execute("; BEGIN")`` no longer auto-commits the transaction it just opened.
- Documented a limitation of ``db.query()``: a ``PRAGMA`` statement that returns no rows raises a ``ValueError`` but still takes effect, because PRAGMA statements run outside the savepoint guard used to roll back other rejected statements. Use ``db.execute()`` for row-less PRAGMA statements.
- Fixed exception masking when a statement destroys the enclosing transaction. An error such as a ``RAISE(ROLLBACK)`` trigger or ``INSERT OR ROLLBACK`` conflict rolls back the whole transaction, destroying every savepoint - the cleanup in ``db.atomic()`` and ``db.query()`` then failed with ``OperationalError: no such savepoint`` (or ``cannot rollback - no transaction is active``), hiding the original ``IntegrityError`` from code that tried to catch it. Cleanup now checks whether a transaction is still open first, so the original exception propagates.
- ``sqlite-utils migrate --list`` is now read-only even when the migrations file uses the legacy ``sqlite_migrate.Migrations`` class, whose listing methods create the ``_sqlite_migrations`` table as a side effect. The listing now runs inside a transaction that is rolled back.
- ``sqlite-utils insert ... --pk <missing column>`` and ``sqlite-utils extract <missing column>`` now show a clean ``Error:`` message instead of a raw Python traceback. The ``extract`` command also shows a clean error when pointed at a view.
- Fixed a bug where running ``table.extract()`` more than once against the same lookup table inserted duplicate rows for values containing ``null`` - SQLite unique indexes treat ``NULL`` values as distinct, so ``INSERT OR IGNORE`` alone could not dedupe them. Each repeat extract added another copy that nothing referenced. The insert now uses an ``IS``-based ``NOT EXISTS`` guard so ``null``-containing rows match existing lookup rows.
- ``db.add_foreign_keys()`` no longer silently ignores requested ``ON DELETE``/``ON UPDATE`` actions when a foreign key with the same columns already exists - it raises ``AlterError`` suggesting ``table.transform()``, since the actions of an existing foreign key cannot be changed in place. Exact duplicates, including actions, are still skipped so repeated calls stay idempotent. The method also now validates that compound foreign keys have the same number of columns on both sides, instead of silently discarding the extra columns.
- ``db.ensure_autocommit_on()`` now raises ``TransactionError`` if called while a transaction is open. Assigning ``isolation_level`` commits any pending transaction as a side effect, so entering the block silently committed the caller's open transaction and made a later ``rollback()`` a no-op.
- ``sqlite-utils migrate --stop-before`` now exits with an error if the named migration has already been applied. Previously the name passed validation but was only checked against pending migrations, so every migration after it was silently applied - the exact outcome ``--stop-before`` exists to prevent. ``Migrations.apply(db, stop_before=...)`` raises ``ValueError`` in the same situation, before applying anything.
- Fixed a regression where ``table.insert(..., pk=..., alter=True)`` raised ``InvalidColumns`` if the primary key column did not exist in the table yet. With ``alter=True`` the check now waits until the record keys are known, so a pk column supplied by the records is added by the alter as it was in 3.x. A pk column found in neither the table nor the records still raises ``InvalidColumns``.
- Fixed a bug where inserting CSV or TSV data into an existing table rewrote that table's column types to match the incoming file. Type detection is the default in 4.0, so ``sqlite-utils insert data.db places places.csv --csv`` against a table with a ``TEXT`` zip code column would convert the column to ``INTEGER`` and corrupt values with leading zeros - ``"01234"`` became ``1234``. Detected types are now only applied when the ``insert`` or ``upsert`` command creates the table.
- Fixed ``pks_and_rows_where()`` raising ``AttributeError`` when called on a view, and no longer double-quotes the synthesized ``rowid`` column in its generated SQL - SQLite turns a double-quoted identifier that does not resolve into a string literal, which on a view produced a confusing ``KeyError`` instead of the ``OperationalError`` raised in 3.x. Compound primary keys returned by this method now follow ``PRIMARY KEY`` declaration order.
- The ``foreign_keys=`` argument to ``create()`` and ``insert()`` accepts a mixed list of ``ForeignKey`` objects, tuples and column name strings again. In 4.0 pre-releases mixing ``ForeignKey`` objects with tuples raised a ``ValueError`` - a regression from 3.x, where ``ForeignKey`` was a ``namedtuple`` and passed the tuple checks.
- ``ForeignKey`` objects are hashable again. The 4.0 change from ``namedtuple`` to dataclass accidentally made them unhashable, breaking patterns like ``set(table.foreign_keys)`` that worked in 3.x. ``ForeignKey`` is now a frozen dataclass - immutable and hashable, like the namedtuple was.
- Fixed a bug where compound primary key columns were returned in table column order instead of ``PRIMARY KEY`` declaration order. For a table declared as ``CREATE TABLE other (b TEXT, a TEXT, PRIMARY KEY (a, b))`` an implicit ``FOREIGN KEY (x, y) REFERENCES other`` was introspected as referencing ``(b, a)`` when SQLite resolves it as ``(a, b)`` - running ``transform()`` on such a table then rewrote the schema with the inverted column order, silently reversing the meaning of the constraint and causing foreign key errors on valid data. ``table.pks``, compound foreign key guessing and ``transform()`` now all use the primary key declaration order, and ``transform()`` no longer reorders a compound ``PRIMARY KEY (b, a)`` into table column order.
.. _v4_0rc3:
4.0rc3 (2026-07-05)
-------------------
Breaking changes
~~~~~~~~~~~~~~~~
- :ref:`table.foreign_keys <python_api_introspection_foreign_keys>` now returns ``ForeignKey`` objects that are dataclasses rather than ``namedtuple`` instances, so they can no longer be unpacked or indexed as ``(table, column, other_table, other_column)`` tuples - access their fields by name instead. Compound (multi-column) foreign keys are now represented as a single ``ForeignKey`` with ``is_compound=True`` and populated ``columns``/``other_columns`` tuples, where ``column`` and ``other_column`` are ``None``. Previously they were returned as one ``ForeignKey`` per column, misleadingly suggesting several independent foreign keys. See :ref:`upgrading_3_to_4` for details. (:issue:`594`)
- Removed support for using ``sqlean.py`` as a drop-in replacement for the Python standard library ``sqlite3`` module. ``sqlite-utils`` will now use ``pysqlite3`` if it is installed, otherwise it will use ``sqlite3`` from the standard library.
- The ``db.ensure_autocommit_off()`` context manager has been renamed to ``db.ensure_autocommit_on()``, because the old name described the opposite of what it did. The method temporarily puts the connection into driver-level autocommit mode - by setting ``isolation_level = None`` - so that statements such as ``PRAGMA journal_mode=wal`` can run outside of an implicit transaction. (:issue:`705`)
Compound foreign key support
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Tables can now be created with :ref:`compound foreign keys <python_api_compound_foreign_keys>`, by passing tuples of column names in ``foreign_keys=``: ``foreign_keys=[(("campus_name", "dept_code"), "departments")]``. The referenced columns default to the compound primary key of the other table. Compound keys are rendered as table-level ``FOREIGN KEY`` constraints in the generated schema.
- ``table.transform()`` now preserves compound foreign keys, applying any column renames to them. Dropping a column that is part of a compound foreign key drops the whole constraint, matching the existing single-column behavior. ``drop_foreign_keys=`` accepts a bare column name - dropping any foreign key that column participates in - or a tuple of columns to target a compound key precisely.
- ``table.add_foreign_key()`` and ``db.add_foreign_keys()`` accept tuples of column names to add a compound foreign key to an existing table.
- ``db.index_foreign_keys()`` creates a single composite index for a compound foreign key.
Other foreign key improvements
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- ``ForeignKey`` now exposes ``on_delete`` and ``on_update`` fields reflecting the foreign key's ``ON DELETE``/``ON UPDATE`` actions, and ``table.transform()`` preserves those actions. Previously a transform silently stripped clauses such as ``ON DELETE CASCADE`` from the table schema.
- ``table.add_foreign_key()`` accepts new ``on_delete=`` and ``on_update=`` parameters for creating foreign keys with actions, e.g. ``table.add_foreign_key("author_id", "authors", "id", on_delete="CASCADE")``. (:issue:`530`)
- Foreign keys declared as ``REFERENCES other_table`` with no explicit column are now resolved to the other table's primary key by ``table.foreign_keys``, instead of reporting ``other_column=None``.
- Fixed a ``TypeError`` when sorting ``ForeignKey`` objects where some were compound.
Case-insensitive column matching
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Column names passed to Python API methods are now matched against the table schema case-insensitively, mirroring how SQLite itself treats identifiers. Previously many methods accepted mixed-case identifiers in the SQL they generated but then failed - or silently did nothing - when performing Python-side comparisons against the schema. (:issue:`760`) Fixes include:
- ``table.insert()`` and ``table.upsert()`` now populate ``table.last_pk`` correctly when the ``pk=`` argument uses different casing to the table schema or the record keys - previously this raised a ``KeyError`` after the row had already been written.
- Upserts no longer raise or misbehave when the casing of ``pk=`` differs from the casing of the record keys. The primary key columns are correctly excluded from the generated ``DO UPDATE SET`` clause.
- ``table.transform()`` arguments ``types=``, ``rename=``, ``drop=``, ``pk=``, ``not_null=``, ``defaults=``, ``column_order=`` and ``drop_foreign_keys=`` all resolve column names case-insensitively. Previously options like ``rename={"name": "title"}`` against a column called ``Name`` were silently ignored.
- ``db.create_table(..., transform=True)`` now recognizes existing columns that differ only by case, instead of attempting to add them again and failing with ``duplicate column name``. The casing used in the existing schema is preserved.
- ``table.lookup()`` returns the primary key value even if ``pk=`` casing differs from the schema, and recognizes existing unique indexes case-insensitively instead of creating redundant ones.
- ``table.extract()`` and ``table.convert()`` - including ``multi=True`` and ``output=`` - accept column names in any casing.
- Foreign key columns are validated and recorded using the casing of the actual schema columns, in ``foreign_keys=`` when creating tables, ``db.add_foreign_keys()``, ``table.add_foreign_key()`` and ``table.add_column(fk_col=...)``. Duplicate foreign key detection is also case-insensitive.
- ``table.create()`` with ``pk=``, ``not_null=``, ``defaults=`` or ``column_order=`` referencing columns using different casing no longer creates an unwanted extra primary key column or raises a ``ValueError``.
Everything else
~~~~~~~~~~~~~~~
- Fixed a bug where ``table.transform()`` could convert ``DEFAULT TRUE``, ``DEFAULT FALSE`` and ``DEFAULT NULL`` column defaults into quoted string defaults when rebuilding a table. Thanks, `Vincent Gao <https://github.com/gaoflow>`__. (`#764 <https://github.com/simonw/sqlite-utils/pull/764>`__)
.. _v4_0rc2:
4.0rc2 (2026-07-04)
-------------------
Breaking changes:
- Write statements executed with ``db.execute()`` are now committed automatically, unless a transaction is already open in which case they join it. Previously they opened an implicit transaction that stayed open until something committed it - writes appeared to work when read on the same connection but were silently rolled back when the connection closed. Code that relied on rolling back uncommitted ``db.execute()`` writes should use the new ``db.begin()`` method to open an explicit transaction first. The transaction model is documented in full at :ref:`python_api_transactions`.
- ``db.query()`` now executes its SQL as soon as it is called, rather than waiting until the returned generator is first iterated. Rows are still fetched lazily during iteration. SQL errors are now raised at the call site, statements such as ``INSERT ... RETURNING`` are executed and committed immediately without needing to iterate over their results, and passing a statement that returns no rows - previously a silent no-op - now raises a ``ValueError`` recommending ``db.execute()`` instead. A statement rejected this way is rolled back before the error is raised, so it has no effect on the database.
- Python API validation errors now raise ``ValueError`` instead of ``AssertionError``. Previously invalid arguments - such as ``create_table()`` with no columns, ``transform()`` on a table that does not exist, or passing both ``ignore=True`` and ``replace=True`` - were rejected using bare ``assert`` statements, which are silently skipped when Python runs with the ``-O`` flag. Code that caught ``AssertionError`` for these cases should catch ``ValueError`` instead.
- ``table.upsert()`` and ``table.upsert_all()`` now raise ``PrimaryKeyRequired`` if a record is missing a value for any primary key column, or has a value of ``None`` for one. Previously such records - which can never match an existing row - were quietly inserted as brand new rows, or triggered a confusing ``KeyError`` after the insert had already taken place.
- ``db.enable_wal()`` and ``db.disable_wal()`` now raise a ``sqlite_utils.db.TransactionError`` if called while a transaction is open. Previously they would silently commit the open transaction as a side effect of changing the journal mode, breaking the rollback guarantee of ``db.atomic()`` and of user-managed transactions.
- The ``View`` class no longer has an ``enable_fts()`` method. It existed only to raise ``NotImplementedError``, since full-text search is not supported for views - calling it now raises ``AttributeError`` instead, and the method no longer appears in the API reference. The ``sqlite-utils enable-fts`` command shows a clean error when pointed at a view.
- The no-op ``-d/--detect-types`` flag has been removed from the ``insert`` and ``upsert`` commands. Type detection has been the default for CSV/TSV data since 4.0a1, so the flag did nothing - invocations using it should simply drop it. ``--no-detect-types`` remains available to disable detection.
- ``Database()`` now raises a ``sqlite_utils.db.TransactionError`` if passed a connection created with the Python 3.12+ ``sqlite3.connect(..., autocommit=True)`` or ``autocommit=False`` options. ``commit()`` and ``rollback()`` behave differently on those connections, which previously caused every write made by the library to be silently discarded when the connection closed.
Everything else:
- Fixed a bug where ``table.delete_where()``, ``table.optimize()`` and ``table.rebuild_fts()`` did not commit their changes, leaving the connection inside an open transaction. Their work - and any subsequent writes - could then be silently rolled back when the connection was closed. All three now use ``db.atomic()``, consistent with the other write methods.
- The ``sqlite-utils drop-table`` command now refuses to drop a view, and ``drop-view`` refuses to drop a table. Previously each would silently drop the wrong type of object if the name matched. Both now exit with an error suggesting the correct command to use.
- Migrations applied by the new :ref:`migrations system <migrations>` now run inside a transaction, together with the record of the migration having been applied. If a migration raises an exception its changes are rolled back and it stays pending, so it can be safely re-applied after the error is fixed. Migrations that cannot run inside a transaction, such as those executing ``VACUUM``, can opt out using ``@migrations(transactional=False)`` - see :ref:`migrations_transactions`.
- ``table.upsert()`` and ``table.upsert_all()`` now detect the primary key or compound primary key of an existing table, so the ``pk=`` argument is no longer required when upserting into a table that already has a primary key.
- ``db.table(table_name).insert({})`` can now be used to insert a row consisting entirely of default values into an existing table, using ``INSERT INTO ... DEFAULT VALUES``. (:issue:`759`)
- Improvements to the ``sqlite-utils migrate`` command: ``--stop-before`` values that do not match any known migration are now an error instead of being silently ignored, ``--stop-before`` now works correctly with migration files that still use the older ``sqlite_migrate.Migrations`` class, and ``--list`` is now a read-only operation that no longer creates the database file or the migrations tracking table. ``migrations.applied()`` now returns migrations in the order they were applied.
- New ``db.begin()``, ``db.commit()`` and ``db.rollback()`` methods for taking manual control of transactions, as an alternative to the ``db.atomic()`` context manager.
- New documentation: :ref:`python_api_transactions` describes how transactions work and when changes are committed, and a new :ref:`upgrading` page details the changes needed to move between major versions.
.. _v4_0rc1:
4.0rc1 (2026-06-21)
-------------------
- New :ref:`database migrations system <migrations>`, incorporating functionality that was previously provided by the separate `sqlite-migrate <https://github.com/simonw/sqlite-migrate>`__ plugin. Define migration sets using the new :class:`sqlite_utils.Migrations` class and apply them using the ``sqlite-utils migrate`` command or the :ref:`migrations Python API <migrations_python>`. (:issue:`752`)
- New ``db.atomic()`` :ref:`context manager providing nested transaction support <python_api_atomic>` using SQLite transactions and savepoints. Internal multi-step operations such as ``table.transform()`` now use this mechanism to avoid unexpectedly committing an existing transaction. (:issue:`755`)
- ``Database`` objects can now be :ref:`used as context managers <python_api_close>`, automatically closing the connection when the ``with`` block exits. The CLI also now closes database and file handles more reliably, resolving a number of ``ResourceWarning`` warnings. (:issue:`692`)
- The ``sqlite-utils convert`` command can now accept a direct callable reference such as ``r.parsedate`` or ``json.loads --import json`` as the conversion code, as an alternative to calling it explicitly with ``r.parsedate(value)``. (:issue:`686`)
- Fixed a bug where CSV or TSV files with only a header row could crash ``sqlite-utils insert`` and ``sqlite-utils memory`` when type detection was enabled. Thanks, `Rami Abdelrazzaq <https://github.com/RamiNoodle733>`__. (:issue:`702`, `#707 <https://github.com/simonw/sqlite-utils/pull/707>`__)
- Fixed a bug where installed plugins could be loaded while running the test suite, despite the test-mode safeguard that disables plugin loading. Thanks, `Rami Abdelrazzaq <https://github.com/RamiNoodle733>`__. (:issue:`713`, `#719 <https://github.com/simonw/sqlite-utils/pull/719>`__)
- ``table.detect_fts()`` now recognizes legacy FTS virtual tables that quote the ``content=`` table name using square brackets, allowing ``table.enable_fts(..., replace=True)`` to replace them correctly. (:issue:`694`)
- Now depends on Click 8.3.1 or later, removing compatibility workarounds for Click's ``Sentinel`` default values. (:issue:`666`)
- Improved type annotations throughout the package, with ``ty`` now run in CI. (:issue:`697`)
- Development tooling now uses ``uv`` dependency groups, with separate ``dev`` and ``docs`` groups. (:issue:`691`)
- The test suite now runs against Python 3.15-dev. (:issue:`738`)
.. _v3_39:
3.39 (2025-11-24)
-----------------
- Fixed a bug with ``sqlite-utils install`` when the tool had been installed using ``uv``. (:issue:`687`)
- The ```--functions``` argument now optionally accepts a path to a Python file as an alternative to a string full of code, and can be specified multiple times - see :ref:`cli_query_functions`. (:issue:`659`)
- ``sqlite-utils`` now requires on Python 3.10 or higher.
- The ``--functions`` argument now optionally accepts a path to a Python file as an alternative to a string full of code, and can be specified multiple times - see :ref:`cli_query_functions`. (:issue:`659`)
- ``sqlite-utils`` now requires Python 3.10 or higher.
.. _v4_0a1:
@ -182,7 +342,7 @@ This release introduces a new :ref:`plugin system <plugins>`. Read more about th
- Conversion functions passed to :ref:`table.convert(...) <python_api_convert>` can now return lists or dictionaries, which will be inserted into the database as JSON strings. (:issue:`495`)
- ``sqlite-utils install`` and ``sqlite-utils uninstall`` commands for installing packages into the same virtual environment as ``sqlite-utils``, :ref:`described here <cli_install>`. (:issue:`483`)
- New :ref:`sqlite_utils.utils.flatten() <reference_utils_flatten>` utility function. (:issue:`500`)
- Documentation on :ref:`using Just <contributing_just>` to run tests, linters and build documentation.
- Documentation on :ref:`using Just <contributing_just>` to run tests, linters and build documentation.
- Documentation now covers the :ref:`release_process` for this package.
.. _v3_29:

268
docs/cli-reference.rst
View file

@ -20,7 +20,7 @@ This page lists the ``--help`` for every ``sqlite-utils`` CLI sub-command.
"query", "memory", "insert", "upsert", "bulk", "search", "transform", "extract",
"schema", "insert-files", "analyze-tables", "convert", "tables", "views", "rows",
"triggers", "indexes", "create-database", "create-table", "create-index",
"enable-fts", "populate-fts", "rebuild-fts", "disable-fts"
"migrate", "enable-fts", "populate-fts", "rebuild-fts", "disable-fts"
]
refs = {
"query": "cli_query",
@ -49,6 +49,7 @@ This page lists the ``--help`` for every ``sqlite-utils`` CLI sub-command.
"enable-wal": "cli_wal",
"enable-counts": "cli_enable_counts",
"bulk": "cli_bulk",
"migrate": "cli_migrate",
"create-database": "cli_create_database",
"create-table": "cli_create_table",
"drop-table": "cli_drop_table",
@ -108,6 +109,10 @@ See :ref:`cli_query`.
"select * from chickens where age > :age" \
-p age 1
Pass "-" as the SQL to read the query from standard input:
echo "select * from chickens" | sqlite-utils data.db -
Options:
--attach <TEXT FILE>... Additional databases to attach - specify alias and
filepath
@ -115,24 +120,26 @@ See :ref:`cli_query`.
--arrays Output rows as arrays instead of objects
--csv Output CSV
--tsv Output TSV
--no-headers Omit CSV headers
--no-headers Omit headers from CSV/TSV and table/--fmt output
-t, --table Output as a formatted table
--fmt TEXT Table format - one of asciidoc, double_grid,
double_outline, fancy_grid, fancy_outline, github,
grid, heavy_grid, heavy_outline, html, jira,
latex, latex_booktabs, latex_longtable, latex_raw,
mediawiki, mixed_grid, mixed_outline, moinmoin,
orgtbl, outline, pipe, plain, presto, pretty,
psql, rounded_grid, rounded_outline, rst, simple,
simple_grid, simple_outline, textile, tsv,
unsafehtml, youtrack
--fmt TEXT Table format - one of asciidoc, colon_grid,
double_grid, double_outline, fancy_grid,
fancy_outline, github, grid, heavy_grid,
heavy_outline, html, jira, latex, latex_booktabs,
latex_longtable, latex_raw, mediawiki, mixed_grid,
mixed_outline, moinmoin, orgtbl, outline, pipe,
plain, presto, pretty, psql, rounded_grid,
rounded_outline, rst, simple, simple_grid,
simple_outline, textile, tsv, unsafehtml, youtrack
--json-cols Detect JSON cols and output them as JSON, not
escaped strings
--ascii Escape non-ASCII characters in JSON output as
\uXXXX
-r, --raw Raw output, first column of first row
--raw-lines Raw output, first column of each row
-p, --param <TEXT TEXT>... Named :parameters for SQL query
--functions TEXT Python code or file path defining custom SQL
functions
--functions TEXT Python code or a file path defining custom SQL
functions; can be used multiple times
--load-extension TEXT Path to SQLite extension, with optional
:entrypoint
-h, --help Show this message and exit.
@ -174,8 +181,8 @@ See :ref:`cli_memory`.
sqlite-utils memory animals.csv --schema
Options:
--functions TEXT Python code or file path defining custom SQL
functions
--functions TEXT Python code or a file path defining custom SQL
functions; can be used multiple times
--attach <TEXT FILE>... Additional databases to attach - specify alias and
filepath
--flatten Flatten nested JSON objects, so {"foo": {"bar":
@ -184,19 +191,21 @@ See :ref:`cli_memory`.
--arrays Output rows as arrays instead of objects
--csv Output CSV
--tsv Output TSV
--no-headers Omit CSV headers
--no-headers Omit headers from CSV/TSV and table/--fmt output
-t, --table Output as a formatted table
--fmt TEXT Table format - one of asciidoc, double_grid,
double_outline, fancy_grid, fancy_outline, github,
grid, heavy_grid, heavy_outline, html, jira,
latex, latex_booktabs, latex_longtable, latex_raw,
mediawiki, mixed_grid, mixed_outline, moinmoin,
orgtbl, outline, pipe, plain, presto, pretty,
psql, rounded_grid, rounded_outline, rst, simple,
simple_grid, simple_outline, textile, tsv,
unsafehtml, youtrack
--fmt TEXT Table format - one of asciidoc, colon_grid,
double_grid, double_outline, fancy_grid,
fancy_outline, github, grid, heavy_grid,
heavy_outline, html, jira, latex, latex_booktabs,
latex_longtable, latex_raw, mediawiki, mixed_grid,
mixed_outline, moinmoin, orgtbl, outline, pipe,
plain, presto, pretty, psql, rounded_grid,
rounded_outline, rst, simple, simple_grid,
simple_outline, textile, tsv, unsafehtml, youtrack
--json-cols Detect JSON cols and output them as JSON, not
escaped strings
--ascii Escape non-ASCII characters in JSON output as
\uXXXX
-r, --raw Raw output, first column of first row
--raw-lines Raw output, first column of each row
-p, --param <TEXT TEXT>... Named :parameters for SQL query
@ -221,7 +230,7 @@ See :ref:`cli_inserting_data`, :ref:`cli_insert_csv_tsv`, :ref:`cli_insert_unstr
::
Usage: sqlite-utils insert [OPTIONS] PATH TABLE FILE
Usage: sqlite-utils insert [OPTIONS] PATH TABLE [FILE]
Insert records from FILE into a table, creating the table if it does not
already exist.
@ -237,6 +246,9 @@ See :ref:`cli_inserting_data`, :ref:`cli_insert_csv_tsv`, :ref:`cli_insert_unstr
- Use --lines to write each incoming line to a column called "line"
- Use --text to write the entire input to a column called "text"
Use --type column-name type to override the type automatically chosen when the
table is created.
You can also use --convert to pass a fragment of Python code that will be used
to convert each input.
@ -263,8 +275,20 @@ See :ref:`cli_inserting_data`, :ref:`cli_insert_csv_tsv`, :ref:`cli_insert_unstr
echo 'A bunch of words' | sqlite-utils insert words.db words - \
--text --convert '({"word": w} for w in text.split())'
Instead of a FILE you can use --code to provide a block of Python code that
defines the rows to insert, as either a rows() function that yields
dictionaries or a "rows" iterable. --code can also be a path to a .py file:
sqlite-utils insert data.db creatures --code '
def rows():
yield {"id": 1, "name": "Cleo"}
yield {"id": 2, "name": "Suna"}
' --pk id
Options:
--pk TEXT Columns to use as the primary key, e.g. id
--code TEXT Python code defining a rows() function or iterable
of rows to insert
--flatten Flatten nested JSON objects, so {"a": {"b": 1}}
becomes {"a_b": 1}
--nl Expect newline-delimited JSON
@ -285,7 +309,7 @@ See :ref:`cli_inserting_data`, :ref:`cli_insert_csv_tsv`, :ref:`cli_insert_unstr
--alter Alter existing table to add any missing columns
--not-null TEXT Columns that should be created as NOT NULL
--default <TEXT TEXT>... Default value that should be set for a column
-d, --detect-types Detect types for columns in CSV/TSV data (default)
--type <TEXT CHOICE>... Column types to use when creating the table
--no-detect-types Treat all CSV/TSV columns as TEXT
--analyze Run ANALYZE at the end of this operation
--load-extension TEXT Path to SQLite extension, with optional :entrypoint
@ -307,12 +331,17 @@ See :ref:`cli_upsert`.
::
Usage: sqlite-utils upsert [OPTIONS] PATH TABLE FILE
Usage: sqlite-utils upsert [OPTIONS] PATH TABLE [FILE]
Upsert records based on their primary key. Works like 'insert' but if an
incoming record has a primary key that matches an existing record the existing
record will be updated.
If the table already exists and has a primary key, --pk can be omitted.
Use --type column-name type to override the type automatically chosen when the
table is created.
Example:
echo '[
@ -322,7 +351,8 @@ See :ref:`cli_upsert`.
Options:
--pk TEXT Columns to use as the primary key, e.g. id
[required]
--code TEXT Python code defining a rows() function or iterable
of rows to insert
--flatten Flatten nested JSON objects, so {"a": {"b": 1}}
becomes {"a_b": 1}
--nl Expect newline-delimited JSON
@ -343,7 +373,7 @@ See :ref:`cli_upsert`.
--alter Alter existing table to add any missing columns
--not-null TEXT Columns that should be created as NOT NULL
--default <TEXT TEXT>... Default value that should be set for a column
-d, --detect-types Detect types for columns in CSV/TSV data (default)
--type <TEXT CHOICE>... Column types to use when creating the table
--no-detect-types Treat all CSV/TSV columns as TEXT
--analyze Run ANALYZE at the end of this operation
--load-extension TEXT Path to SQLite extension, with optional :entrypoint
@ -376,7 +406,8 @@ See :ref:`cli_bulk`.
Options:
--batch-size INTEGER Commit every X records
--functions TEXT Python code or file path defining custom SQL functions
--functions TEXT Python code or a file path defining custom SQL
functions; can be used multiple times
--flatten Flatten nested JSON objects, so {"a": {"b": 1}} becomes
{"a_b": 1}
--nl Expect newline-delimited JSON
@ -423,18 +454,20 @@ See :ref:`cli_search`.
--arrays Output rows as arrays instead of objects
--csv Output CSV
--tsv Output TSV
--no-headers Omit CSV headers
--no-headers Omit headers from CSV/TSV and table/--fmt output
-t, --table Output as a formatted table
--fmt TEXT Table format - one of asciidoc, double_grid,
double_outline, fancy_grid, fancy_outline, github,
grid, heavy_grid, heavy_outline, html, jira, latex,
latex_booktabs, latex_longtable, latex_raw, mediawiki,
mixed_grid, mixed_outline, moinmoin, orgtbl, outline,
pipe, plain, presto, pretty, psql, rounded_grid,
rounded_outline, rst, simple, simple_grid,
simple_outline, textile, tsv, unsafehtml, youtrack
--fmt TEXT Table format - one of asciidoc, colon_grid,
double_grid, double_outline, fancy_grid, fancy_outline,
github, grid, heavy_grid, heavy_outline, html, jira,
latex, latex_booktabs, latex_longtable, latex_raw,
mediawiki, mixed_grid, mixed_outline, moinmoin, orgtbl,
outline, pipe, plain, presto, pretty, psql,
rounded_grid, rounded_outline, rst, simple,
simple_grid, simple_outline, textile, tsv, unsafehtml,
youtrack
--json-cols Detect JSON cols and output them as JSON, not escaped
strings
--ascii Escape non-ASCII characters in JSON output as \uXXXX
--load-extension TEXT Path to SQLite extension, with optional :entrypoint
-h, --help Show this message and exit.
@ -611,6 +644,11 @@ See :ref:`cli_convert`.
"value" is a variable with the column value to be converted.
CODE can also be a reference to a callable that takes the value, for example:
sqlite-utils convert my.db mytable date r.parsedate
sqlite-utils convert my.db mytable data json.loads --import json
Use "-" for CODE to read Python code from standard input.
The following common operations are available as recipe functions:
@ -624,7 +662,6 @@ See :ref:`cli_convert`.
errors: 'Optional[object]' = None) -> 'Optional[str]'
Parse a date and convert it to ISO date format: yyyy-mm-dd
- dayfirst=True: treat xx as the day in xx/yy/zz
- yearfirst=True: treat xx as the year in xx/yy/zz
- errors=r.IGNORE to ignore values that cannot be parsed
@ -634,7 +671,6 @@ See :ref:`cli_convert`.
False, errors: 'Optional[object]' = None) -> 'Optional[str]'
Parse a datetime and convert it to ISO datetime format: yyyy-mm-ddTHH:MM:SS
- dayfirst=True: treat xx as the day in xx/yy/zz
- yearfirst=True: treat xx as the year in xx/yy/zz
- errors=r.IGNORE to ignore values that cannot be parsed
@ -688,18 +724,20 @@ See :ref:`cli_tables`.
--arrays Output rows as arrays instead of objects
--csv Output CSV
--tsv Output TSV
--no-headers Omit CSV headers
--no-headers Omit headers from CSV/TSV and table/--fmt output
-t, --table Output as a formatted table
--fmt TEXT Table format - one of asciidoc, double_grid,
double_outline, fancy_grid, fancy_outline, github,
grid, heavy_grid, heavy_outline, html, jira, latex,
latex_booktabs, latex_longtable, latex_raw, mediawiki,
mixed_grid, mixed_outline, moinmoin, orgtbl, outline,
pipe, plain, presto, pretty, psql, rounded_grid,
rounded_outline, rst, simple, simple_grid,
simple_outline, textile, tsv, unsafehtml, youtrack
--fmt TEXT Table format - one of asciidoc, colon_grid,
double_grid, double_outline, fancy_grid, fancy_outline,
github, grid, heavy_grid, heavy_outline, html, jira,
latex, latex_booktabs, latex_longtable, latex_raw,
mediawiki, mixed_grid, mixed_outline, moinmoin, orgtbl,
outline, pipe, plain, presto, pretty, psql,
rounded_grid, rounded_outline, rst, simple,
simple_grid, simple_outline, textile, tsv, unsafehtml,
youtrack
--json-cols Detect JSON cols and output them as JSON, not escaped
strings
--ascii Escape non-ASCII characters in JSON output as \uXXXX
--columns Include list of columns for each table
--schema Include schema for each table
--load-extension TEXT Path to SQLite extension, with optional :entrypoint
@ -729,18 +767,20 @@ See :ref:`cli_views`.
--arrays Output rows as arrays instead of objects
--csv Output CSV
--tsv Output TSV
--no-headers Omit CSV headers
--no-headers Omit headers from CSV/TSV and table/--fmt output
-t, --table Output as a formatted table
--fmt TEXT Table format - one of asciidoc, double_grid,
double_outline, fancy_grid, fancy_outline, github,
grid, heavy_grid, heavy_outline, html, jira, latex,
latex_booktabs, latex_longtable, latex_raw, mediawiki,
mixed_grid, mixed_outline, moinmoin, orgtbl, outline,
pipe, plain, presto, pretty, psql, rounded_grid,
rounded_outline, rst, simple, simple_grid,
simple_outline, textile, tsv, unsafehtml, youtrack
--fmt TEXT Table format - one of asciidoc, colon_grid,
double_grid, double_outline, fancy_grid, fancy_outline,
github, grid, heavy_grid, heavy_outline, html, jira,
latex, latex_booktabs, latex_longtable, latex_raw,
mediawiki, mixed_grid, mixed_outline, moinmoin, orgtbl,
outline, pipe, plain, presto, pretty, psql,
rounded_grid, rounded_outline, rst, simple,
simple_grid, simple_outline, textile, tsv, unsafehtml,
youtrack
--json-cols Detect JSON cols and output them as JSON, not escaped
strings
--ascii Escape non-ASCII characters in JSON output as \uXXXX
--columns Include list of columns for each view
--schema Include schema for each view
--load-extension TEXT Path to SQLite extension, with optional :entrypoint
@ -775,19 +815,21 @@ See :ref:`cli_rows`.
--arrays Output rows as arrays instead of objects
--csv Output CSV
--tsv Output TSV
--no-headers Omit CSV headers
--no-headers Omit headers from CSV/TSV and table/--fmt output
-t, --table Output as a formatted table
--fmt TEXT Table format - one of asciidoc, double_grid,
double_outline, fancy_grid, fancy_outline, github,
grid, heavy_grid, heavy_outline, html, jira,
latex, latex_booktabs, latex_longtable, latex_raw,
mediawiki, mixed_grid, mixed_outline, moinmoin,
orgtbl, outline, pipe, plain, presto, pretty,
psql, rounded_grid, rounded_outline, rst, simple,
simple_grid, simple_outline, textile, tsv,
unsafehtml, youtrack
--fmt TEXT Table format - one of asciidoc, colon_grid,
double_grid, double_outline, fancy_grid,
fancy_outline, github, grid, heavy_grid,
heavy_outline, html, jira, latex, latex_booktabs,
latex_longtable, latex_raw, mediawiki, mixed_grid,
mixed_outline, moinmoin, orgtbl, outline, pipe,
plain, presto, pretty, psql, rounded_grid,
rounded_outline, rst, simple, simple_grid,
simple_outline, textile, tsv, unsafehtml, youtrack
--json-cols Detect JSON cols and output them as JSON, not
escaped strings
--ascii Escape non-ASCII characters in JSON output as
\uXXXX
--load-extension TEXT Path to SQLite extension, with optional
:entrypoint
-h, --help Show this message and exit.
@ -815,18 +857,20 @@ See :ref:`cli_triggers`.
--arrays Output rows as arrays instead of objects
--csv Output CSV
--tsv Output TSV
--no-headers Omit CSV headers
--no-headers Omit headers from CSV/TSV and table/--fmt output
-t, --table Output as a formatted table
--fmt TEXT Table format - one of asciidoc, double_grid,
double_outline, fancy_grid, fancy_outline, github,
grid, heavy_grid, heavy_outline, html, jira, latex,
latex_booktabs, latex_longtable, latex_raw, mediawiki,
mixed_grid, mixed_outline, moinmoin, orgtbl, outline,
pipe, plain, presto, pretty, psql, rounded_grid,
rounded_outline, rst, simple, simple_grid,
simple_outline, textile, tsv, unsafehtml, youtrack
--fmt TEXT Table format - one of asciidoc, colon_grid,
double_grid, double_outline, fancy_grid, fancy_outline,
github, grid, heavy_grid, heavy_outline, html, jira,
latex, latex_booktabs, latex_longtable, latex_raw,
mediawiki, mixed_grid, mixed_outline, moinmoin, orgtbl,
outline, pipe, plain, presto, pretty, psql,
rounded_grid, rounded_outline, rst, simple,
simple_grid, simple_outline, textile, tsv, unsafehtml,
youtrack
--json-cols Detect JSON cols and output them as JSON, not escaped
strings
--ascii Escape non-ASCII characters in JSON output as \uXXXX
--load-extension TEXT Path to SQLite extension, with optional :entrypoint
-h, --help Show this message and exit.
@ -854,18 +898,20 @@ See :ref:`cli_indexes`.
--arrays Output rows as arrays instead of objects
--csv Output CSV
--tsv Output TSV
--no-headers Omit CSV headers
--no-headers Omit headers from CSV/TSV and table/--fmt output
-t, --table Output as a formatted table
--fmt TEXT Table format - one of asciidoc, double_grid,
double_outline, fancy_grid, fancy_outline, github,
grid, heavy_grid, heavy_outline, html, jira, latex,
latex_booktabs, latex_longtable, latex_raw, mediawiki,
mixed_grid, mixed_outline, moinmoin, orgtbl, outline,
pipe, plain, presto, pretty, psql, rounded_grid,
rounded_outline, rst, simple, simple_grid,
simple_outline, textile, tsv, unsafehtml, youtrack
--fmt TEXT Table format - one of asciidoc, colon_grid,
double_grid, double_outline, fancy_grid, fancy_outline,
github, grid, heavy_grid, heavy_outline, html, jira,
latex, latex_booktabs, latex_longtable, latex_raw,
mediawiki, mixed_grid, mixed_outline, moinmoin, orgtbl,
outline, pipe, plain, presto, pretty, psql,
rounded_grid, rounded_outline, rst, simple,
simple_grid, simple_outline, textile, tsv, unsafehtml,
youtrack
--json-cols Detect JSON cols and output them as JSON, not escaped
strings
--ascii Escape non-ASCII characters in JSON output as \uXXXX
--load-extension TEXT Path to SQLite extension, with optional :entrypoint
-h, --help Show this message and exit.
@ -911,10 +957,10 @@ See :ref:`cli_create_table`.
sqlite-utils create-table my.db people \
id integer \
name text \
height float \
height real \
photo blob --pk id
Valid column types are text, integer, float and blob.
Valid column types are text, integer, real, float and blob.
Options:
--pk TEXT Column to use as primary key
@ -960,6 +1006,44 @@ See :ref:`cli_create_index`.
-h, --help Show this message and exit.
.. _cli_ref_migrate:
migrate
=======
See :ref:`cli_migrate`.
::
Usage: sqlite-utils migrate [OPTIONS] DB_PATH [MIGRATIONS]...
Apply pending database migrations.
Usage:
sqlite-utils migrate database.db
This will find the migrations.py file in the current directory or
subdirectories and apply any pending migrations.
Or pass paths to one or more migrations.py files directly:
sqlite-utils migrate database.db path/to/migrations.py
Pass --list to see a list of applied and pending migrations without applying
them.
Use --stop-before migration_set:name to stop before a migration. This option
can be used multiple times.
Options:
--stop-before TEXT Stop before applying this migration. Use set:name to
target a migration set.
--list List migrations without running them
-v, --verbose Show verbose output
-h, --help Show this message and exit.
.. _cli_ref_enable_fts:
enable-fts
@ -971,7 +1055,7 @@ See :ref:`cli_fts`.
Usage: sqlite-utils enable-fts [OPTIONS] PATH TABLE COLUMN...
Enable full-text search for specific table and columns"
Enable full-text search for specific table and columns
Example:
@ -1147,7 +1231,7 @@ See :ref:`cli_add_column`.
::
Usage: sqlite-utils add-column [OPTIONS] PATH TABLE COL_NAME
[[integer|int|float|real|text|str|blob|bytes]]
[integer|int|float|real|text|str|blob|bytes]
Add a column to the specified table

117
docs/cli.rst
View file

@ -29,6 +29,16 @@ The ``sqlite-utils query`` command lets you run queries directly against a SQLit
.. note::
In Python: :ref:`db.query() <python_api_query>` CLI reference: :ref:`sqlite-utils query <cli_ref_query>`
Pass ``-`` as the SQL query to read the query from standard input. This is useful for longer queries that would otherwise require careful shell escaping, or for piping in SQL generated by another tool:
.. code-block:: bash
echo "select * from dogs" | sqlite-utils query dogs.db -
.. code-block:: bash
sqlite-utils query dogs.db - < query.sql
.. _cli_query_json:
Returning JSON
@ -45,6 +55,8 @@ The default format returned for queries is JSON:
[{"id": 1, "age": 4, "name": "Cleo"},
{"id": 2, "age": 2, "name": "Pancakes"}]
If the query returns more than one column with the same name, later occurrences are renamed with a numeric suffix - ``select 1 as id, 2 as id`` returns ``[{"id": 1, "id_2": 2}]``. This only applies to JSON output: :ref:`CSV and TSV <cli_query_csv>` and :ref:`table <cli_query_table>` output keep the duplicate column headers unchanged.
.. _cli_query_nl:
Newline-delimited JSON
@ -109,6 +121,33 @@ If you want to pretty-print the output further, you can pipe it through ``python
}
]
.. _cli_query_json_ascii:
Unicode characters in JSON
~~~~~~~~~~~~~~~~~~~~~~~~~~
JSON output includes unicode characters directly, without escaping them:
.. code-block:: bash
sqlite-utils dogs.db "select '日本語' as text"
.. code-block:: output
[{"text": "日本語"}]
Use ``--ascii`` to escape non-ASCII characters as ``\uXXXX`` sequences instead:
.. code-block:: bash
sqlite-utils dogs.db "select '日本語' as text" --ascii
.. code-block:: output
[{"text": "\u65e5\u672c\u8a9e"}]
The ``--ascii`` option can help on systems that cannot display or process UTF-8, such as Windows consoles using a legacy code page. On Windows, setting the ``PYTHONUTF8=1`` environment variable is an alternative fix for ``UnicodeEncodeError`` crashes when redirecting output to a file.
.. _cli_query_binary_json:
Binary data in JSON
@ -257,6 +296,7 @@ Available ``--fmt`` options are:
.. ]]]
- ``asciidoc``
- ``colon_grid``
- ``double_grid``
- ``double_outline``
- ``fancy_grid``
@ -1057,6 +1097,37 @@ That will look for SpatiaLite in a set of predictable locations. To load it from
sqlite-utils create-database empty.db --init-spatialite --load-extension /path/to/spatialite.so
.. _cli_migrate:
Running migrations
==================
The ``migrate`` command applies pending Python migrations to a database. For the full migration file format and Python API, see :ref:`migrations`.
.. code-block:: bash
sqlite-utils migrate creatures.db path/to/migrations.py
If you omit the migration path it will search the current directory and subdirectories for files called ``migrations.py``:
.. code-block:: bash
sqlite-utils migrate creatures.db
Use ``--list`` to list applied and pending migrations without running them:
.. code-block:: bash
sqlite-utils migrate creatures.db --list
Use ``--stop-before`` to stop before a named migration. The option can be passed more than once, and can target a specific migration set using ``migration_set:migration_name``:
.. code-block:: bash
sqlite-utils migrate creatures.db path/to/migrations.py \
--stop-before creatures:add_weight \
--stop-before sales:drop_index
.. _cli_inserting_data:
Inserting JSON data
@ -1265,6 +1336,8 @@ A progress bar is displayed when inserting data from a file. You can hide the pr
By default, column types are automatically detected for CSV or TSV files - resulting in a mix of ``TEXT``, ``INTEGER`` and ``REAL`` columns. To disable type detection and treat all columns as ``TEXT``, use the ``--no-detect-types`` option.
Detected types are only applied when the table is created by the command. Inserting CSV or TSV data into a table that already exists leaves the existing column types unchanged - values are inserted using the table's existing schema.
For example, given a ``creatures.csv`` file containing this:
.. code-block::
@ -1293,6 +1366,25 @@ Will produce this schema with automatically detected types:
"weight" REAL
);
.. _cli_insert_csv_tsv_column_types:
Overriding column types
-----------------------
Use ``--type column-name type`` to override the type automatically chosen when the table is created. This option can be used more than once, and works with both ``insert`` and ``upsert``:
.. code-block:: bash
sqlite-utils insert places.db places places.csv --csv \
--type zipcode text \
--type score real
This is useful for values such as ZIP codes, which may look like integers but should be stored as ``TEXT`` to preserve leading zeros.
The column type should be one of ``TEXT``, ``INTEGER``, ``FLOAT``, ``REAL`` or ``BLOB``. Column types are matched case-insensitively.
As with detected column types, ``--type`` only affects tables created by the command. If the table already exists, its existing column types are left unchanged.
To disable type detection and treat all columns as TEXT, use ``--no-detect-types``:
.. code-block:: bash
@ -1488,6 +1580,27 @@ The result looks like this:
COMMIT;
.. _cli_insert_code:
Inserting rows generated by Python code
=======================================
Instead of providing a ``FILE`` to import, you can use the ``--code`` option to pass a block of Python code that generates the rows to insert. This is the command-line equivalent of calling ``db["creatures"].insert_all(rows())`` from the :ref:`Python API <python_api>`.
Your code should define either a ``rows()`` function that returns or yields dictionaries, or a ``rows`` iterable such as a list of dictionaries:
.. code-block:: bash
sqlite-utils insert data.db creatures --code '
def rows():
yield {"id": 1, "name": "Cleo"}
yield {"id": 2, "name": "Suna"}
' --pk id
``--code`` can also be given a path to a Python ``.py`` file.
The ``--code`` option works with both ``sqlite-utils insert`` and ``sqlite-utils upsert``, and composes with table options such as ``--pk``, ``--replace``, ``--alter``, ``--not-null`` and ``--default``. It cannot be combined with a ``FILE`` argument or with input format options such as ``--csv`` or ``--convert``.
.. _cli_insert_replace:
Insert-replacing data
@ -1520,6 +1633,8 @@ For example:
This will update the dog with an ID of 2 to have an age of 4, creating a new record (with a null name) if one does not exist. If a row DOES exist the name will be left as-is.
If the table already exists and has a primary key, you can omit the ``--pk`` option and ``sqlite-utils`` will use that existing primary key.
The command will fail if you reference columns that do not exist on the table. To automatically create missing columns, use the ``--alter`` option.
.. note::
@ -2225,6 +2340,8 @@ The ``sqlite-utils extract`` command can be used to extract specified columns in
Take a look at the Python API documentation for :ref:`python_api_extract` for a detailed description of how this works, including examples of table schemas before and after running an extraction operation.
Rows where every extracted column is ``null`` are not extracted - those rows get a ``null`` value in their new foreign key column and no record is created for them in the lookup table.
The command takes a database, table and one or more columns that should be extracted. To extract the ``species`` column from the ``trees`` table you would run:
.. code-block:: bash

58
docs/conf.py
View file

@ -1,8 +1,10 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
from subprocess import Popen, PIPE
from beanbag_docutils.sphinx.ext.github import github_linkcode_resolve
import inspect
from pathlib import Path
from subprocess import Popen, PIPE, check_output
import sys
# This file is execfile()d with the current directory set to its
# containing dir.
@ -45,14 +47,52 @@ extlinks = {
}
def _linkcode_git_ref():
try:
return check_output(["git", "rev-parse", "HEAD"]).decode("utf8").strip()
except Exception:
return "main"
def linkcode_resolve(domain, info):
return github_linkcode_resolve(
domain=domain,
info=info,
allowed_module_names=["sqlite_utils"],
github_org_id="simonw",
github_repo_id="sqlite-utils",
branch="main",
if domain != "py":
return None
module_name = info.get("module")
if not module_name or module_name.split(".")[0] != "sqlite_utils":
return None
module = sys.modules.get(module_name)
if module is None:
return None
obj = module
for part in info.get("fullname", "").split("."):
obj = getattr(obj, part, None)
if obj is None:
return None
if isinstance(obj, property):
obj = obj.fget
try:
obj = inspect.unwrap(obj)
source_file = inspect.getsourcefile(obj)
_, line_number = inspect.getsourcelines(obj)
except Exception:
return None
if source_file is None:
return None
try:
filename = Path(source_file).resolve().relative_to(Path(__file__).parent.parent)
except ValueError:
return None
return (
"https://github.com/simonw/sqlite-utils/blob/"
f"{_linkcode_git_ref()}/{filename}#L{line_number}"
)

2
docs/index.rst
View file

@ -36,8 +36,10 @@ Contents
installation
cli
python-api
migrations
plugins
reference
cli-reference
upgrading
contributing
changelog

10
docs/installation.rst
View file

@ -52,15 +52,15 @@ On some platforms the ability to load additional extensions (via ``conn.load_ext
You may also see the error ``sqlite3.OperationalError: table sqlite_master may not be modified`` when trying to alter an existing table.
You can work around these limitations by installing either the `pysqlite3 <https://pypi.org/project/pysqlite3/>`__ package or the `sqlean.py <https://pypi.org/project/sqlean.py/>`__ package, both of which provide drop-in replacements for the standard library ``sqlite3`` module but with a recent version of SQLite and full support for loading extensions.
You can work around these limitations by installing the `pysqlite3 <https://pypi.org/project/pysqlite3/>`__ package, which provides a drop-in replacement for the standard library ``sqlite3`` module but with a recent version of SQLite and full support for loading extensions.
To install ``sqlean.py`` (which has compiled binary wheels available for all major platforms) run the following:
To install ``pysqlite3`` run the following:
.. code-block:: bash
sqlite-utils install sqlean.py
sqlite-utils install pysqlite3
``pysqlite3`` and ``sqlean.py`` do not provide implementations of the ``.iterdump()`` method. To use that method (see :ref:`python_api_itedump`) or the ``sqlite-utils dump`` command you should also install the ``sqlite-dump`` package:
``pysqlite3`` does not provide an implementation of the ``.iterdump()`` method. To use that method (see :ref:`python_api_itedump`) or the ``sqlite-utils dump`` command you should also install the ``sqlite-dump`` package:
.. code-block:: bash
@ -87,4 +87,4 @@ For ``zsh``:
Add this code to ``~/.zshrc`` or ``~/.bashrc`` to automatically run it when you start a new shell.
See `the Click documentation <https://click.palletsprojects.com/en/8.1.x/shell-completion/>`__ for more details.
See `the Click documentation <https://click.palletsprojects.com/en/8.1.x/shell-completion/>`__ for more details.

194
docs/migrations.rst Normal file
View file

@ -0,0 +1,194 @@
.. _migrations:
=====================
Database migrations
=====================
``sqlite-utils`` includes a migration system for applying repeatable changes to SQLite database files.
A migration is a Python function that receives a :class:`sqlite_utils.Database` instance and then executes Python code to modify that database - creating or transforming tables, adding indexes, inserting rows, or any other operation supported by SQLite.
Migrations are grouped into named sets using the :class:`sqlite_utils.Migrations` class, and each applied migration is recorded in the ``_sqlite_migrations`` table in that database.
This means you can run the migrate operation multiple times and it will only apply migrations that have not previously been recorded.
.. _migrations_define:
Defining migrations
===================
Ordered migration sets are defined by first creating a :class:`sqlite_utils.Migrations` object.
Individual migrations are Python functions that are then registered with that migration set. Each migration function is passed a single argument that is a :ref:`sqlite_utils.Database <reference_db_database>` instance.
The name passed to ``Migrations("creatures")`` identifies that set of migrations. Use a name that is unique for your project, since multiple migration sets can be applied to the same database.
Here is a simple example of a ``migrations.py`` file which creates a table, then adds an extra column to that table in a second migration:
.. code-block:: python
from sqlite_utils import Migrations
migrations = Migrations("creatures")
@migrations()
def create_table(db):
db["creatures"].create(
{"id": int, "name": str, "species": str},
pk="id",
)
@migrations()
def add_weight(db):
db["creatures"].add_column("weight", float)
.. _migrations_python:
Applying migrations in Python
=============================
Once you have a ``Migrations(name)`` collection with one or more migrations registered to it, you can execute them in Python code like this:
.. code-block:: python
from sqlite_utils import Database
db = Database("creatures.db")
migrations.apply(db)
Running ``migrations.apply(db)`` repeatedly is safe. Migrations that already have a matching ``migration_set`` and ``name`` row in ``_sqlite_migrations`` will be skipped.
Migration functions are applied in the order that they were registered. The function name is used as the migration name unless you pass one explicitly:
.. code-block:: python
@migrations(name="001_create_table")
def create_table(db):
db["creatures"].create({"id": int, "name": str}, pk="id")
When you apply a set of migrations you can stop part way through by specifying a ``stop_before=`` migration name:
.. code-block:: python
migrations.apply(db, stop_before="add_weight")
.. _migrations_transactions:
Migrations and transactions
===========================
Each migration runs inside a transaction, together with the ``_sqlite_migrations`` record of it having been applied. If a migration function raises an exception, everything it did is rolled back, no record is written and the migration stays pending - so fixing the error and re-applying will run that migration again from a clean state. Migrations that completed earlier in the same ``apply()`` run stay applied.
Some operations cannot run inside a transaction, for example ``VACUUM`` or changing the journal mode with ``db.enable_wal()``. Register migrations like these with ``transactional=False``:
.. code-block:: python
@migrations(transactional=False)
def compact(db):
db.execute("VACUUM")
A migration registered with ``transactional=False`` runs without a wrapping transaction, so if it fails part way through any changes it already made will not be rolled back, and re-applying will run the whole function again.
Avoid calling ``db.commit()`` or otherwise managing transactions manually inside a transactional migration - register the migration with ``transactional=False`` if it needs to control its own transactions. Using ``with db.atomic():`` blocks inside a migration is fine: they nest as savepoints within the migration's transaction, so the migration as a whole still commits or rolls back as a single unit. See :ref:`python_api_transactions`.
Applying migrations using the CLI
=================================
Run migrations using the ``sqlite-utils migrate`` command:
.. code-block:: bash
sqlite-utils migrate creatures.db path/to/migrations.py
The first argument is the database file. The remaining arguments can be paths to migration files or directories containing migration files.
If you omit migration paths, ``sqlite-utils`` searches the current directory and subdirectories for files called ``migrations.py``:
.. code-block:: bash
sqlite-utils migrate creatures.db
You can also pass a directory. Every ``migrations.py`` file in that directory tree will be considered:
.. code-block:: bash
sqlite-utils migrate creatures.db path/to/project/
Running the command repeatedly is safe. Migrations that already have a matching ``migration_set`` and ``name`` row in ``_sqlite_migrations`` will be skipped.
Listing migrations
==================
Use ``--list`` to show applied and pending migrations without running them. This is a read-only operation - it will not create the database file or the ``_sqlite_migrations`` table:
.. code-block:: bash
sqlite-utils migrate creatures.db --list
Example output:
.. code-block:: output
Migrations for: creatures
Applied:
create_table - 2026-06-09 17:23:12.048092+00:00
add_weight - 2026-06-09 17:23:12.051249+00:00
Pending:
add_age
Stopping before a migration
===========================
When applying migrations using the CLI, you can stop before a named migration:
.. code-block:: bash
sqlite-utils migrate creatures.db path/to/migrations.py --stop-before add_weight
This applies any pending migrations before ``add_weight`` and leaves ``add_weight`` and later migrations pending. An unqualified migration name matches in any migration set.
You can also target a specific migration set using ``migration_set:migration_name``. This is useful if a migrations file contains more than one migration set, or if multiple sets use the same migration name:
.. code-block:: bash
sqlite-utils migrate creatures.db path/to/migrations.py \
--stop-before creatures:add_weight \
--stop-before sales:drop_index
The ``--stop-before`` option can be passed more than once.
If a ``--stop-before`` value does not match any known migration the command exits with an error, rather than silently applying everything. Naming a migration that has already been applied is also an error - stopping before it is impossible to honor - and no pending migrations are applied.
Verbose output
==============
Use ``--verbose`` or ``-v`` to show the schema before and after migrations are applied, plus a unified diff when the schema changes:
.. code-block:: bash
sqlite-utils migrate creatures.db --verbose
Migrating from sqlite-migrate
=============================
This system uses the same migration table format as the older `sqlite-migrate <https://github.com/simonw/sqlite-migrate>`__ package. To use existing migration files directly with ``sqlite-utils``, update their import from ``sqlite_migrate`` to ``sqlite_utils``:
.. code-block:: python
from sqlite_utils import Migrations
migration = Migrations("creatures")
@migration()
def create_table(db):
db["creatures"].create({"id": int, "name": str}, pk="id")
Python API
==========
.. autoclass:: sqlite_utils.migrations.Migrations
:members:
:undoc-members:
:exclude-members: _Migration, _AppliedMigration

7
docs/plugins.rst
View file

@ -140,9 +140,7 @@ As a special niche feature, if your plugin needs to import some files and then a
prepare_connection(conn)
~~~~~~~~~~~~~~~~~~~~~~~~
This hook is called when a new SQLite database connection is created. You can
use it to `register custom SQL functions <https://docs.python.org/2/library/sqlite3.html#sqlite3.Connection.create_function>`_,
aggregates and collations. For example:
This hook is called when a new SQLite database connection is created. You can use it to `register custom SQL functions <https://docs.python.org/2/library/sqlite3.html#sqlite3.Connection.create_function>`_, aggregates and collations. For example:
.. code-block:: python
@ -154,8 +152,7 @@ aggregates and collations. For example:
"hello", 1, lambda name: f"Hello, {name}!"
)
This registers a SQL function called ``hello`` which takes a single
argument and can be called like this:
This registers a SQL function called ``hello`` which takes a single argument and can be called like this:
.. code-block:: sql

278
docs/python-api.rst
View file

@ -33,6 +33,8 @@ Here's how to create a new SQLite database file containing a new ``chickens`` ta
"color": "black",
}])
The inserted rows are saved to the database file straight away - methods like ``insert_all()`` commit their own changes, so no ``commit()`` call is needed. See :ref:`python_api_transactions` for how this works.
You can loop through those rows like this:
.. code-block:: python
@ -93,6 +95,8 @@ Instead of a file path you can pass in an existing SQLite connection:
db = Database(sqlite3.connect("my_database.db"))
The connection must use Python's default transaction handling. Connections created with the Python 3.12+ ``sqlite3.connect(..., autocommit=True)`` or ``autocommit=False`` options are rejected with a ``sqlite_utils.db.TransactionError`` - see :ref:`python_api_transactions_modes`.
If you want to create an in-memory database, you can do so like this:
.. code-block:: python
@ -152,6 +156,12 @@ The ``Database`` object also works as a context manager, which will automaticall
db["my_table"].insert({"name": "Example"})
# Connection is automatically closed here
Exiting the block is equivalent to calling ``db.close()``: the connection is closed and any transaction still open at that point is rolled back. This matches SQLite's own behavior when a connection closes.
This rarely matters in practice. Everything that writes to the database - including raw ``db.execute()`` statements - commits automatically, so a transaction can only be open here if you explicitly started one with ``db.begin()`` and have not yet committed it. In that case the decision to commit stays with you: committing automatically on exit could silently persist half-finished work, for example if your code returned early from the block. Call ``db.commit()`` when the work is complete.
Note this differs from the ``sqlite3.Connection`` context manager in the standard library, which commits on success but does not close the connection. See :ref:`python_api_transactions` for the full transaction model.
.. _python_api_attach:
Attaching additional databases
@ -227,6 +237,23 @@ The ``db.query(sql)`` function executes a SQL query and returns an iterator over
# {'name': 'Cleo'}
# {'name': 'Pancakes'}
The SQL query is executed as soon as ``db.query()`` is called. The resulting rows are fetched lazily as you iterate, so large result sets are not loaded into memory all at once. Because execution is immediate, an error in your SQL will raise an exception straight away, and a statement such as ``INSERT ... RETURNING`` will take effect - and be committed, unless a transaction is open - even if you do not iterate over its results.
``db.query()`` can only be used with SQL that returns rows. Passing a statement that returns no rows - an ``INSERT`` or ``UPDATE`` without a ``RETURNING`` clause, for example - will raise a ``ValueError``. The rejected statement is rolled back, so it has no effect on the database. Use :ref:`db.execute() <python_api_execute>` for those statements instead.
There is one exception to the rolled-back guarantee: a ``PRAGMA`` statement that returns no rows, such as ``PRAGMA user_version = 5``, still raises a ``ValueError`` but will already have taken effect. Some PRAGMA statements refuse to run inside a transaction, so PRAGMAs are executed outside the savepoint that is used to roll back other rejected statements. Use ``db.execute()`` for PRAGMA statements that do not return rows.
If a query returns more than one column with the same name - a join between two tables that share column names, for example - later occurrences are renamed with a numeric suffix, so every value is included in the dictionary:
.. code-block:: python
row = next(db.query("select 1 as id, 2 as id, 3 as id"))
print(row)
# Outputs:
# {'id': 1, 'id_2': 2, 'id_3': 3}
A suffix that would collide with another column in the query is skipped - ``select 1 as id, 2 as id, 3 as id_2`` returns ``{'id': 1, 'id_3': 2, 'id_2': 3}``. The same renaming is applied by ``table.rows_where()`` and ``table.search()``.
.. _python_api_execute:
db.execute(sql, params)
@ -247,6 +274,9 @@ The ``db.execute()`` and ``db.executescript()`` methods provide wrappers around
Other cursor methods such as ``.fetchone()`` and ``.fetchall()`` are also available, see the `standard library documentation <https://docs.python.org/3/library/sqlite3.html#sqlite3.Cursor>`__.
.. note::
Write statements executed this way are committed automatically, unless a transaction is already open in which case they become part of it - see :ref:`python_api_transactions_execute`.
.. _python_api_parameters:
Passing parameters
@ -275,6 +305,114 @@ Named parameters using ``:name`` can be filled using a dictionary:
In this example ``next()`` is used to retrieve the first result in the iterator returned by the ``db.query()`` method.
.. _python_api_transactions:
Transactions and saving your changes
====================================
Every method in this library that writes to the database - ``insert()``, ``upsert()``, ``update()``, ``delete()``, ``delete_where()``, ``transform()``, ``create_table()``, ``create_index()``, ``enable_fts()`` and the rest - runs inside its own transaction and commits it before returning. Your changes are saved to disk as soon as the method call finishes:
.. code-block:: python
db = Database("data.db")
db.table("news").insert({"headline": "Dog wins award"})
# The new row is already saved - no commit() required
The same applies to raw SQL executed with :ref:`db.execute() <python_api_transactions_execute>` - a write statement is committed as soon as it has run.
You never need to call ``commit()``, and you do not need to close the database to persist your changes. There are exactly two situations where you need to think about transactions:
1. You want to group several write operations together, so they either all succeed or all fail - use :ref:`db.atomic() <python_api_atomic>`.
2. You are :ref:`managing a transaction yourself <python_api_transactions_manual>` with ``db.begin()``, in which case nothing is committed until you commit - the library will never commit a transaction you opened.
.. _python_api_atomic:
Grouping changes with db.atomic()
---------------------------------
Use ``db.atomic()`` to group multiple operations in a single transaction:
.. code-block:: python
with db.atomic():
db.table("dogs").insert({"id": 1, "name": "Cleo"}, pk="id")
db.table("dogs").insert({"id": 2, "name": "Pancakes"})
The transaction commits when the block exits. If an exception is raised, changes made inside the block will be rolled back.
``db.atomic()`` can be nested. Nested blocks use SQLite savepoints, so an exception in an inner block can roll back to that savepoint without rolling back the entire outer transaction:
.. code-block:: python
with db.atomic():
db.table("dogs").insert({"id": 1, "name": "Cleo"}, pk="id")
try:
with db.atomic():
db.table("dogs").insert({"id": 2, "name": "Pancakes"})
raise ValueError("skip this one")
except ValueError:
pass
db.table("dogs").insert({"id": 3, "name": "Marnie"})
The transaction is opened with a deferred ``BEGIN`` - SQLite takes the necessary locks when the first statement inside the block runs.
.. _python_api_transactions_execute:
Raw SQL writes with db.execute()
--------------------------------
Write statements executed with :ref:`db.execute() <python_api_execute>` follow the same rule as everything else: they are committed automatically as soon as they have run.
.. code-block:: python
db.execute("insert into news (headline) values (?)", ["Dog wins award"])
# Already committed
If a transaction is open - because the call happens inside a ``db.atomic()`` block, or after ``db.begin()`` - the statement becomes part of that transaction instead, and commits when the transaction commits:
.. code-block:: python
with db.atomic():
db.execute("insert into news (headline) values (?)", ["Dog wins award"])
db.execute("insert into news (headline) values (?)", ["Cat unimpressed"])
# Both rows committed together
One corner case: a row-returning write such as ``INSERT ... RETURNING`` executed through ``db.execute()`` cannot be auto-committed, because its rows have not been read yet - call ``db.commit()`` after fetching them, or use :ref:`db.query() <python_api_query>` for those statements, which executes the write and commits it immediately.
.. _python_api_transactions_manual:
Managing transactions yourself
------------------------------
You can take full manual control using the ``db.begin()``, ``db.commit()`` and ``db.rollback()`` methods:
.. code-block:: python
db.begin()
db.table("news").insert({"headline": "Dog wins award"})
if all_looks_good:
db.commit()
else:
db.rollback()
``db.begin()`` raises ``sqlite3.OperationalError`` if a transaction is already open. ``db.commit()`` and ``db.rollback()`` do nothing if there is no open transaction.
The library will never commit a transaction you opened. If you call write methods such as ``insert()`` - or use ``db.atomic()`` - while your transaction is open, they participate in it using SQLite savepoints instead of committing: exiting an ``atomic()`` block releases its savepoint, but nothing is saved to disk until you commit the outer transaction yourself. If you roll back, their changes are rolled back too.
Two related safeguards to be aware of:
- ``db.enable_wal()`` and ``db.disable_wal()`` raise a ``sqlite_utils.db.TransactionError`` if called while a transaction is open, because changing the journal mode would commit it as a side effect.
- Closing the database - explicitly with ``db.close()``, or by exiting a ``with Database(...) as db:`` block - rolls back any transaction that is still open, see :ref:`python_api_close`.
.. _python_api_transactions_modes:
Supported connection modes
--------------------------
``db.atomic()`` and the automatic per-method transactions require a connection in Python's default transaction handling mode. Passing a connection created with the Python 3.12+ ``sqlite3.connect(..., autocommit=True)`` or ``autocommit=False`` options to ``Database()`` raises a ``sqlite_utils.db.TransactionError``.
This is because ``commit()`` and ``rollback()`` behave differently on those connections - under ``autocommit=True`` they are documented no-ops - which would cause every write made by this library to be silently discarded when the connection closed, rather than failing loudly.
.. _python_api_table:
Accessing tables
@ -700,6 +838,61 @@ You can leave off the third item in the tuple to have the referenced column auto
("author_id", "authors")
])
.. _python_api_compound_foreign_keys:
Compound foreign keys
~~~~~~~~~~~~~~~~~~~~~
To create a compound (multi-column) foreign key, use tuples of column names in place of the single column names:
.. code-block:: python
db.table("courses").create({
"course_code": str,
"campus_name": str,
"dept_code": str,
}, pk="course_code", foreign_keys=[
(("campus_name", "dept_code"), "departments", ("campus_name", "dept_code"))
])
This creates a table-level constraint:
.. code-block:: sql
CREATE TABLE "courses" (
"course_code" TEXT PRIMARY KEY,
"campus_name" TEXT,
"dept_code" TEXT,
FOREIGN KEY ("campus_name", "dept_code") REFERENCES "departments"("campus_name", "dept_code")
)
As with single columns, you can leave off the tuple of other columns to reference the compound primary key of the other table:
.. code-block:: python
foreign_keys=[
(("campus_name", "dept_code"), "departments")
]
To specify ``ON DELETE`` or ``ON UPDATE`` actions, pass ``ForeignKey`` objects instead:
.. code-block:: python
from sqlite_utils.db import ForeignKey
db.table("books").create({
"id": int,
"author_id": int,
}, pk="id", foreign_keys=[
ForeignKey(
table="books", column="author_id",
other_table="authors", other_column="id",
on_delete="CASCADE",
)
])
Foreign key actions are preserved by :ref:`table.transform() <python_api_transform>` - prior to sqlite-utils 4.0 they were silently dropped when a table was transformed.
.. _python_api_table_configuration:
Table configuration options
@ -963,8 +1156,7 @@ You can delete all records in a table that match a specific WHERE statement usin
>>> db = sqlite_utils.Database("dogs.db")
>>> # Delete every dog with age less than 3
>>> with db.conn:
>>> db.table("dogs").delete_where("age < ?", [3])
>>> db.table("dogs").delete_where("age < ?", [3])
Calling ``table.delete_where()`` with no other arguments will delete every row in the table.
@ -997,6 +1189,8 @@ Note that the ``pk`` and ``column_order`` parameters here are optional if you ar
An ``upsert_all()`` method is also available, which behaves like ``insert_all()`` but performs upserts instead.
Every record passed to ``upsert()`` or ``upsert_all()`` must include a value for each primary key column - a record without one could never match an existing row, so a ``sqlite_utils.db.PrimaryKeyRequired`` exception is raised instead of quietly inserting a new row.
.. note::
``.upsert()`` and ``.upsert_all()`` in sqlite-utils 1.x worked like ``.insert(..., replace=True)`` and ``.insert_all(..., replace=True)`` do in 2.x. See `issue #66 <https://github.com/simonw/sqlite-utils/issues/66>`__ for details of this change.
@ -1095,6 +1289,8 @@ To create a species record with a note on when it was first seen, you can use th
The first time this is called the record will be created for ``name="Palm"``. Any subsequent calls with that name will ignore the second argument, even if it includes different values.
``None`` values are matched correctly: calling ``.lookup()`` a second time with the same values will return the primary key of the existing row even if some of those values are ``None``.
``.lookup()`` also accepts keyword arguments, which are passed through to the :ref:`insert() method <python_api_creating_tables>` and can be used to influence the shape of the created table. Supported parameters are:
- ``pk`` - which defaults to ``id``
@ -1140,6 +1336,8 @@ To extract the ``species`` column out to a separate ``Species`` table, you can d
"species": "Common Juniper"
}, extracts={"species": "Species"})
``None`` values are not extracted: no record is created for them in the lookup table and the column value stays ``null``.
.. _python_api_m2m:
Working with many-to-many relationships
@ -1408,6 +1606,26 @@ To ignore the case where the key already exists, use ``ignore=True``:
db.table("books").add_foreign_key("author_id", "authors", "id", ignore=True)
To add a compound foreign key, pass tuples of columns:
.. code-block:: python
db.table("courses").add_foreign_key(
("campus_name", "dept_code"), "departments", ("campus_name", "dept_code")
)
As with single columns, omitting the other columns will use the compound primary key of the other table. ``other_table`` must always be specified for a compound foreign key.
Use ``on_delete=`` and ``on_update=`` to specify ``ON DELETE`` and ``ON UPDATE`` actions for the foreign key:
.. code-block:: python
db.table("books").add_foreign_key(
"author_id", "authors", "id", on_delete="CASCADE"
)
This creates a foreign key with an ``ON DELETE CASCADE`` clause, so deleting an author will also delete their books (provided foreign key enforcement is enabled with ``PRAGMA foreign_keys = ON``). Valid actions are ``"SET NULL"``, ``"SET DEFAULT"``, ``"CASCADE"``, ``"RESTRICT"`` and the default ``"NO ACTION"``.
.. _python_api_add_foreign_keys:
Adding multiple foreign key constraints at once
@ -1426,6 +1644,8 @@ Here's an example adding two foreign keys at once:
This method runs the same checks as ``.add_foreign_keys()`` and will raise ``sqlite_utils.db.AlterError`` if those checks fail.
Foreign keys that already exist are silently skipped, so repeated calls are idempotent - but only if they match exactly. Requesting a foreign key that exists with different ``ON DELETE``/``ON UPDATE`` actions raises ``AlterError``: use ``table.transform()`` to change the actions of an existing foreign key.
.. _python_api_index_foreign_keys:
Adding indexes for all foreign keys
@ -1437,6 +1657,8 @@ If you want to ensure that every foreign key column in your database has a corre
db.index_foreign_keys()
Compound foreign keys get a single composite index across their columns.
.. _python_api_drop:
Dropping a table or view
@ -1639,6 +1861,16 @@ This example drops two foreign keys - the one from ``places.country`` to ``count
drop_foreign_keys=("country", "continent")
)
A bare column name drops any foreign key that column participates in, including compound foreign keys. To target a compound foreign key precisely, pass a tuple of its columns:
.. code-block:: python
db.table("courses").transform(
drop_foreign_keys=[("campus_name", "dept_code")]
)
Renaming a column with ``rename=`` updates any foreign keys that use it, and dropping a column with ``drop=`` also drops any foreign keys it participates in - for a compound foreign key this removes the whole constraint.
.. _python_api_transform_sql:
Custom transformations with .transform_sql()
@ -1806,6 +2038,8 @@ This produces a lookup table like so:
"latin" TEXT
)
Rows where every extracted column is ``null`` are not extracted: no record is created for them in the lookup table and their foreign key column is left as ``null``. When extracting multiple columns, rows where at least one of the extracted columns has a value will be extracted as usual.
.. _python_api_hash:
Setting an ID based on the hash of the row contents
@ -1979,7 +2213,7 @@ The ``db.iterdump()`` method returns a sequence of SQL strings representing a co
This uses the `sqlite3.Connection.iterdump() <https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.iterdump>`__ method.
If you are using ``pysqlite3`` or ``sqlean.py`` the underlying method may be missing. If you install the `sqlite-dump <https://pypi.org/project/sqlite-dump/>`__ package then the ``db.iterdump()`` method will use that implementation instead:
If you are using ``pysqlite3`` the underlying method may be missing. If you install the `sqlite-dump <https://pypi.org/project/sqlite-dump/>`__ package then the ``db.iterdump()`` method will use that implementation instead:
.. code-block:: bash
@ -2086,17 +2320,39 @@ Almost all SQLite tables have a ``rowid`` column, but a table with no explicitly
.foreign_keys
-------------
The ``.foreign_keys`` property returns any foreign key relationships for the table, as a list of ``ForeignKey(table, column, other_table, other_column)`` named tuples. It is not available on views.
The ``.foreign_keys`` property returns any foreign key relationships for the table, as a list of ``ForeignKey`` objects. It is not available on views.
Each ``ForeignKey`` has the following attributes:
``table``
The table the foreign key is defined on.
``column``
The column on this table, or ``None`` for a compound foreign key.
``other_table``
The table being referenced.
``other_column``
The referenced column, or ``None`` for a compound foreign key.
``columns``
A tuple of the columns on this table, always populated (a one-item tuple for single-column foreign keys).
``other_columns``
A tuple of the referenced columns.
``is_compound``
``True`` if this is a compound (multi-column) foreign key.
``on_delete``
The ``ON DELETE`` action, e.g. ``"CASCADE"`` - ``"NO ACTION"`` if not set.
``on_update``
The ``ON UPDATE`` action - ``"NO ACTION"`` if not set.
``ForeignKey`` was a ``namedtuple`` prior to sqlite-utils 4.0. It is now a dataclass and can no longer be unpacked or indexed as a tuple - access its fields by name instead. See :ref:`upgrading_3_to_4` for details.
::
>>> db.table("Street_Tree_List").foreign_keys
[ForeignKey(table='Street_Tree_List', column='qLegalStatus', other_table='qLegalStatus', other_column='id'),
ForeignKey(table='Street_Tree_List', column='qCareAssistant', other_table='qCareAssistant', other_column='id'),
ForeignKey(table='Street_Tree_List', column='qSiteInfo', other_table='qSiteInfo', other_column='id'),
ForeignKey(table='Street_Tree_List', column='qSpecies', other_table='qSpecies', other_column='id'),
ForeignKey(table='Street_Tree_List', column='qCaretaker', other_table='qCaretaker', other_column='id'),
ForeignKey(table='Street_Tree_List', column='PlantType', other_table='PlantType', other_column='id')]
[ForeignKey(table='Street_Tree_List', column='qLegalStatus', other_table='qLegalStatus', other_column='id', columns=('qLegalStatus',), other_columns=('id',), is_compound=False, on_delete='NO ACTION', on_update='NO ACTION'),
ForeignKey(table='Street_Tree_List', column='qCareAssistant', other_table='qCareAssistant', other_column='id', columns=('qCareAssistant',), other_columns=('id',), is_compound=False, on_delete='NO ACTION', on_update='NO ACTION'),
...]
Compound foreign keys - defined with ``FOREIGN KEY (col_a, col_b) REFERENCES other(col_a, col_b)`` - are returned as a single ``ForeignKey`` with ``is_compound=True``, ``column`` and ``other_column`` set to ``None``, and the participating columns available in the ``columns`` and ``other_columns`` tuples.
.. _python_api_introspection_schema:
@ -2676,6 +2932,8 @@ You can disable WAL mode using ``.disable_wal()``:
Database("my_database.db").disable_wal()
The journal mode can only be changed outside of a transaction. Calling either method while a transaction is open - inside a ``db.atomic()`` block, for example - raises a ``sqlite_utils.db.TransactionError``, unless the database is already in the requested mode in which case the call is a no-op.
You can check the current journal mode for a database using the ``journal_mode`` property:
.. code-block:: python

7
docs/reference.rst
View file

@ -70,6 +70,13 @@ sqlite_utils.db.ColumnDetails
.. autoclass:: sqlite_utils.db.ColumnDetails
.. _reference_db_other_foreign_key:
sqlite_utils.db.ForeignKey
--------------------------
.. autoclass:: sqlite_utils.db.ForeignKey
sqlite_utils.utils
==================

146
docs/upgrading.rst Normal file
View file

@ -0,0 +1,146 @@
.. _upgrading:
===========
Upgrading
===========
This page describes the changes you may need to make to your own code or scripts when upgrading between major versions of ``sqlite-utils``.
For the full list of changes in every release see the :ref:`changelog`.
.. _upgrading_3_to_4:
Upgrading from 3.x to 4.0
=========================
Requirements
------------
- Python 3.10 or higher is required.
- The ``click`` dependency must be version 8.3.1 or later.
Command-line changes
--------------------
**Type detection is now the default for CSV and TSV imports.** ``sqlite-utils insert`` and ``sqlite-utils upsert`` now detect column types when importing CSV or TSV data - previously every column was created as ``TEXT`` unless you passed ``--detect-types``. To restore the old behavior pass the new ``--no-detect-types`` flag:
.. code-block:: bash
sqlite-utils insert data.db rows data.csv --csv --no-detect-types
Two related things have been removed:
- The ``SQLITE_UTILS_DETECT_TYPES`` environment variable.
- The old ``-d/--detect-types`` flag itself. Since detection is now the default the flag did nothing - remove it from any scripts that used it.
**The convert command no longer skips falsey values.** ``sqlite-utils convert`` previously skipped values that evaluated to ``False`` (empty strings, ``0``) unless you passed ``--no-skip-false``. All values are now converted and the ``--no-skip-false`` flag has been removed.
**drop-table and drop-view check the object type.** ``sqlite-utils drop-table`` now refuses to drop a view, and ``drop-view`` refuses to drop a table. Previously each would silently drop the wrong type of object if the name matched. If you relied on that (unlikely), use the matching command instead.
**sqlite-utils tui has moved to a plugin.** The optional terminal interface is now provided by the `sqlite-utils-tui <https://github.com/simonw/sqlite-utils-tui>`__ plugin:
.. code-block:: bash
sqlite-utils install sqlite-utils-tui
Python API changes
------------------
**db.query() now rejects SQL that does not return rows.** This is likely the most common change you will need to make to existing code. ``db.query()`` used to accept any SQL statement - passing one that returns no rows, such as an ``INSERT`` or ``UPDATE`` without a ``RETURNING`` clause or a ``CREATE TABLE``, did nothing at all, silently. Those statements now raise a ``ValueError``, and are rolled back so they have no effect on the database. Transaction control statements (``BEGIN``, ``COMMIT``, ``END``, ``ROLLBACK``, ``SAVEPOINT``, ``RELEASE``) plus ``VACUUM``, ``ATTACH`` and ``DETACH`` are also rejected with a ``ValueError``, without being executed at all. Use ``db.execute()`` for statements that do not return rows:
.. code-block:: python
# 3.x accepted this but silently did nothing:
db.query("update dogs set name = 'Cleopaws'")
# In 4.0 use execute() for SQL that does not return rows:
db.execute("update dogs set name = 'Cleopaws'")
**db.query() executes immediately.** ``db.query(sql)`` previously returned a generator that did not execute the SQL until you started iterating over it. The SQL now runs as soon as the method is called - rows are still fetched lazily, but errors in your SQL raise at the ``db.query()`` call site rather than on first iteration, and a write with a ``RETURNING`` clause takes effect even if you never iterate over its results.
**db.table() no longer returns views.** ``db.table(name)`` now raises a ``sqlite_utils.db.NoTable`` exception if ``name`` is a SQL view. Use the new ``db.view(name)`` method for views:
.. code-block:: python
table = db.table("my_table")
view = db.view("my_view")
``db["name"]`` still returns either a ``Table`` or a ``View`` depending on what exists in the database.
**Upserts use INSERT ... ON CONFLICT.** Upsert operations now use SQLite's ``INSERT ... ON CONFLICT SET`` syntax rather than the previous ``INSERT OR IGNORE`` followed by ``UPDATE``. If your code depends on the old behavior, pass ``use_old_upsert=True`` to the ``Database()`` constructor - see :ref:`python_api_old_upsert`.
**Upsert records must include their primary keys.** ``table.upsert()`` and ``table.upsert_all()`` now raise ``sqlite_utils.db.PrimaryKeyRequired`` if a record is missing a value for any primary key column (or has ``None`` for one). Previously such records were quietly inserted as new rows. Relatedly, ``pk=`` is now optional when the table already exists with a primary key - it is detected automatically.
**Floating point columns are now REAL.** Auto-detected floating point columns are created with the correct SQLite type ``REAL`` instead of ``FLOAT``. Code that inspects column types should expect ``REAL``.
**Generated schemas use double quotes.** Tables created by this library now wrap table and column names in ``"double-quotes"`` where they previously used ``[square-braces]``. If you compare ``table.schema`` strings against expected values you will need to update them.
**table.convert() no longer skips falsey values.** Matching the CLI change above, ``table.convert()`` now converts every value. The ``skip_false`` parameter has been removed - previously it defaulted to ``True``, skipping empty strings and other falsey values.
**Null values are no longer extracted into lookup tables.** ``table.extract()`` and the ``sqlite-utils extract`` command leave rows alone if every extracted column is ``null`` - the new foreign key column is left as ``null`` instead of pointing at an all-``null`` record in the lookup table. The ``extracts=`` insert option similarly keeps ``None`` values as ``null``. Relatedly, ``table.lookup()`` now compares values using ``IS`` so that looking up a value containing ``None`` returns the existing matching row - previously it inserted a duplicate row on every call.
**ensure_autocommit_off() is now ensure_autocommit_on().** The ``db.ensure_autocommit_off()`` context manager has been renamed to ``db.ensure_autocommit_on()``. The old name described the opposite of what the method did: it temporarily puts the connection into driver-level autocommit mode (by setting ``isolation_level = None``), so that statements such as ``PRAGMA journal_mode=wal`` can run outside of an implicit transaction. The behavior is unchanged - update any calls to use the new name.
**View.enable_fts() has been removed.** The ``View`` class previously had an ``enable_fts()`` method that existed only to raise ``NotImplementedError`` - full-text search is not supported for views. Calling it now raises ``AttributeError`` like any other missing method.
**ForeignKey is now a dataclass, not a namedtuple.** The ``ForeignKey`` objects returned by ``table.foreign_keys`` gained new fields - ``columns``, ``other_columns``, ``is_compound``, ``on_delete`` and ``on_update`` - so that compound (multi-column) foreign keys and foreign key actions can be represented. To make room for those fields cleanly ``ForeignKey`` is now a dataclass rather than a ``namedtuple``, so it can no longer be unpacked or indexed as a tuple. Access its fields by name instead:
.. code-block:: python
# 3.x - tuple unpacking, no longer works:
for table, column, other_table, other_column in db["courses"].foreign_keys:
...
# 4.0 - access fields by name:
for fk in db["courses"].foreign_keys:
fk.table, fk.column, fk.other_table, fk.other_column
Attempting the old unpacking or ``fk[0]`` indexing now raises ``TypeError``, so any code using those patterns will fail loudly rather than silently misbehave. Like the old namedtuple, ``ForeignKey`` instances are immutable and hashable - they can be collected into sets and used as dictionary keys. Note that equality now includes the ``on_delete`` and ``on_update`` actions: a ``ForeignKey`` with ``ON DELETE CASCADE`` is not equal to one without.
Compound foreign keys - previously returned as one ``ForeignKey`` per column, misleadingly suggesting several independent single-column keys - are now returned as a single ``ForeignKey`` with ``is_compound=True``. For these the scalar ``column`` and ``other_column`` fields are ``None``; use the ``columns`` and ``other_columns`` tuples instead. Single-column foreign keys are unaffected apart from the class change: ``column``/``other_column`` behave as before and ``columns``/``other_columns`` are one-item tuples.
Two related behavior changes to ``table.transform()``: compound foreign keys now survive a transform (previously they were split into separate single-column keys), and ``ON DELETE``/``ON UPDATE`` actions such as ``ON DELETE CASCADE`` are now preserved (previously they were silently stripped from the schema).
**Validation errors raise ValueError.** Invalid arguments to Python API methods - for example ``create_table()`` with no columns, or ``ignore=True`` together with ``replace=True`` - now raise ``ValueError``. They previously raised ``AssertionError`` from bare ``assert`` statements, which were silently skipped under ``python -O``.
**Transaction behavior is now well-defined.** 4.0 introduces the :ref:`db.atomic() <python_api_atomic>` context manager and uses it consistently for every write operation - the full model is described in :ref:`python_api_transactions`. Changes you may notice:
- Write statements executed with raw ``db.execute()`` calls now commit automatically, unless a transaction is already open in which case they join it. Previously they opened an implicit transaction that nothing committed - if your code used ``db.execute()`` for writes and relied on ``db.conn.rollback()`` to undo them, open an explicit transaction with the new ``db.begin()`` method first.
- Multi-step operations such as ``table.transform()`` no longer commit an existing transaction you have open - they use savepoints inside it instead.
- ``db.enable_wal()`` and ``db.disable_wal()`` raise a ``sqlite_utils.db.TransactionError`` if called while a transaction is open, instead of silently committing it.
- Using ``Database`` as a context manager (``with Database(path) as db:``) closes the connection on exit *without* committing - a transaction you explicitly opened with ``db.begin()`` and did not commit is rolled back.
- ``Database()`` rejects connections created with the Python 3.12+ ``sqlite3.connect(..., autocommit=True)`` or ``autocommit=False`` options, raising ``sqlite_utils.db.TransactionError``. On those connections every write the library made was silently discarded when the connection closed.
Packaging changes
-----------------
- ``sqlite-utils`` now uses ``pyproject.toml`` in place of ``setup.py``.
- ``pip`` is now a runtime dependency, used by the ``sqlite-utils install`` and ``uninstall`` commands.
New features to be aware of
---------------------------
Not breaking changes, but new in 4.0 and worth knowing about when you upgrade:
- A :ref:`database migrations system <migrations>`, incorporating the functionality of the ``sqlite-migrate`` plugin. If you used that plugin, the built-in system reads the same ``_sqlite_migrations`` table - your applied migrations will not run again. Update your migration files to use ``from sqlite_utils import Migrations``.
- :ref:`db.atomic() <python_api_atomic>` for nested transaction support.
- ``table.insert_all()`` and ``table.upsert_all()`` accept an iterator of lists or tuples as an alternative to dictionaries - see :ref:`python_api_insert_lists`.
.. _upgrading_2_to_3:
Upgrading from 2.x to 3.0
=========================
The 3.0 release redesigned search. The breaking changes were minor:
- ``table.search()`` returns a generator of dictionaries, sorted by relevance. It previously returned a list of tuples sorted by ``rowid``.
- The ``-c`` shortcut for ``--csv`` and the ``-f`` shortcut for ``--fmt`` were removed from the CLI - use the full option names.
.. _upgrading_1_to_2:
Upgrading from 1.x to 2.0
=========================
The 2.0 release changed the meaning of *upsert*. In 1.x, ``table.upsert()`` and ``table.upsert_all()`` actually performed ``INSERT OR REPLACE`` operations - entirely replacing the existing row. Since 2.0 an upsert updates only the columns you provide, leaving other columns untouched.
If you want the 1.x behavior, use ``table.insert(..., replace=True)`` or ``table.insert_all(..., replace=True)`` instead.