Commit graph

298 commits

Author SHA1 Message Date
Simon Willison
bfd74a35bb upsert() detects existing compound primary keys
Closes #629
2026-06-24 10:43:53 -07:00
Simon Willison
b5d0080cd1 Handle db.table(table_name).insert({}), closes #759 2026-06-22 12:00:05 -07:00
Rami Abdelrazzaq
f448b61f5e
fix: Plugins are still loaded when running tests (#719)
* fix: defer plugin entrypoint loading until runtime

Fixes simonw/sqlite-utils#713

* Lazy-load plugins when listing them

Keep plugin entrypoints from loading at module import time, but preserve the existing behavior of get_plugins() by loading entrypoints the first time plugins are listed outside tests.

---------

Co-authored-by: Simon Willison <swillison@gmail.com>
2026-06-21 16:14:45 -07:00
Simon Willison
1a28416e10 Fix for detect_fts failing on [], refs #694 2026-06-21 15:54:19 -07:00
Simon Willison
6729ea3f60
db.atomic() method
Closes #755
2026-06-21 10:43:19 -07:00
Simon Willison
8f0c06e188
Test against Python 3.15-dev, bump ty and Black (#738)
* Add Python 3.15-dev to test matrix
* Run ty check only on 3.14
* Bump Black version
* Update tabulate and use that in 
* Bump to latest ty
2026-05-17 16:52:48 -07:00
Simon Willison
8d74ffc932
More type annotations (#697)
* Add comprehensive type annotations

- mypy.ini: expanded configuration with module-specific settings
- hookspecs.py: type annotations for hook functions
- plugins.py: typed get_plugins() return value
- recipes.py: full type annotations for parsedate, parsedatetime, jsonsplit
- utils.py: extensive type annotations including Row type alias,
  TypeTracker, ValueTracker, and all utility functions
- db.py: type annotations for Database methods (__exit__,
  ensure_autocommit_off, tracer, register_function, etc.) and
  Queryable class methods
- tests/test_docs.py: updated to match new signature display format

* Fix type errors caught by ty check

- Add type: ignore comments for external library type stub limitations
  (csv.reader, click.progressbar, IOBase.name, Callable.__name__)
- Change Iterable to Sequence for SQL where_args parameters
- Use db.table() instead of db[name] for proper Table return type
- Fix rebuild_fts return type from None to Table
- Update test_tracer to expect fewer queries (optimization side effect)

* Fix mypy type errors

- Add type: ignore comments for runtime-valid patterns mypy can't verify
- Fix new_column_types annotation to Dict[str, Set[type]]
- Add type: ignore for Default sentinel values passed to create_table

* mypy skip tests directory

* Fix CI: exclude typing imports from recipe docs, skip mypy on tests

- Add Callable and Optional to exclusion list in _generate_convert_help()
- Regenerate docs/cli-reference.rst with cog
- Add [mypy-tests.*] ignore_errors = True to skip test type errors

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-16 22:11:47 -08:00
Simon Willison
871038505b
Type fixes now enforced by ty
* Fix type warning for pipe.stdout possibly being None

Add conditional check before calling .read() on pipe.stdout since
Popen can return None for stdout.

* Use ctx.meta instead of dynamic attribute for database cleanup

Click's Context.meta dictionary is the proper way to store arbitrary
data on the context object, avoiding type checker warnings about
dynamic attribute assignment.

* Add assert for tables.callback before calling

Click's callback attribute is typed as Optional[Callable], so add
assert to satisfy type checker that it's not None.

* Fix type errors in cli.py and db.py

- Add type annotation for Database.conn to fix context manager errors
- Convert exception objects to str() when raising ClickException
- Handle None return from find_spatialite() with proper error message

* Fix remaining type errors in cli.py

- Add typing import and type annotations for dict kwargs
- Use db.table() instead of db[] for extract command
- Fix missing str() conversion for exception

* Fix type errors in db.py

- Add type annotation for Database.conn
- Add type: ignore for optional sqlite_dump import
- Update execute/query parameter types to Sequence|Dict for sqlite3 compatibility
- Use getattr for fn.__name__ access to handle callables without __name__
- Handle None return from find_spatialite() with OSError
- Fix pk_values assignment to use local variable

* Add type: ignore for optional pysqlite3 and sqlean imports

These are alternative sqlite3 implementations that may not be installed.

* Fix type errors in tests and plugins

- Add type: ignore for monkey-patching Database.__init__ in conftest
- Fix CLI test to pass string "2" instead of integer to Click invoke
- Add type: ignore for optional sqlean import
- Fix add_geometry_column test to use "XY" instead of integer 2
- Add type: ignore for click.Context as context manager
- Add type: ignore for enable_fts test that intentionally omits argument
- Add type: ignore for sys._called_from_test dynamic attribute
- Fix rows_from_file test type error for intentional wrong argument
- Handle None from pm.get_hookcallers in plugins.py

* Use db.table() instead of db[] for Table-specific operations

Changes db[table] to db.table(table) in CLI commands where we know
we're working with tables, not views. This resolves most of the
Table | View disambiguation type warnings since db.table() returns
Table directly rather than Table | View.

* Fix remaining type warnings in sqlite_utils package

- Add assert for sniff_buffer not being None
- Handle cursor.fetchone() potentially returning None
- Use db.table() for counts_table and index_foreign_keys
- Add type: ignore for cursor union type in raw mode

* Ran Black

* Run ty in CI

* ty check sqlite_utils

* Skip running ty on Windows

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-16 19:34:45 -08:00
Simon Willison
fd5b09f64b
Database as a context manager, fixed many pytest warnings
* Database can now work as a context manager
* Claude Code helped fix a ton of .close() warnings

https://gistpreview.github.io/?730f0c5dc38528a1dd0615f330bd5481

* New autouse fixture to help with test warnings

Refs https://github.com/simonw/sqlite-utils/issues/692#issuecomment-3644371889

* Fix all remaining resource warnings

https://gistpreview.github.io/?0bb8e869b82f6ff0db647de755182502

Closes #692
2025-12-11 16:56:12 -08:00
Simon Willison
c872d27bb3
Use REAL not FLOAT as SQLite column type (#680)
* Use REAL not FLOAT as SQLite column type, refs #645
* Fix for REAL columns by CSV --detect-types

Refs https://github.com/simonw/sqlite-utils/issues/645#issuecomment-3568947189

* Removed note about strict and REAL
2025-11-23 21:37:59 -08:00
Simon Willison
fb93452ea8
Use double quotes not braces for tables and columns (#678)
Closes #677
2025-11-23 20:43:26 -08:00
Simon Willison
52ea7d21f4 Fix for mypy errors 2025-11-23 15:53:19 -08:00
Simon Willison
96fab69256 Removed convert skip_false and --skip-false, closes #542 2025-11-23 15:40:28 -08:00
Simon Willison
fafa966300 Table.create() now stores configuration in _defaults
When configuration parameters like pk, foreign_keys, not_null, etc.
are passed to Table.create(), they are now stored in self._defaults
so that subsequent operations on the same Table instance (like insert,
upsert) will use those settings automatically.

Previously, calling db["table"].insert({...}, pk="id") would create
the table correctly but subsequent calls to insert() or upsert() on
the same table object would not know about the pk setting, causing
issues like upsert() failing with "requires a pk" error.

Closes #655

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-23 15:18:55 -08:00
Simon Willison
7ffd5052e9 table.insert_all() and table.upsert_all() now take generators of lists/tuples
Closes #672, PR #673
2025-11-23 12:17:23 -08:00
Alex Chan
cf1b407207
Add a type hint for db.close() (#663)
Closes #662
2025-10-01 13:37:05 -07:00
Simon Willison
094b010fd8
db.table() only returns tables, added db.view()
* db.table() only returns tables, added db.view(), closes #657
* Massive documentation update for db.table()

Refs #656
2025-05-08 23:19:36 -07:00
Simon Willison
8e7d018fa2
Use ON CONFLICT for upsert, refs #652
* New upsert implementation, refs #652
* supports_strict now caches on self._supports_strict

PR: https://github.com/simonw/sqlite-utils/pull/653
2025-05-08 20:37:49 -07:00
Simon Willison
0e4e270d44
Test against multiple SQLite versions (#654)
* Test against pre-upsert SQLite 3.23.1

Borrowed from 8f86d2af6

* Try this on Python 3.9

* select ... from pragma_function_list()

Refs https://github.com/simonw/sqlite-utils/pull/654#issuecomment-2860898278

* Fix spelling error

* Compatible with latest black

* Skip plugin test that needs pragma_function_list

Refs https://github.com/simonw/sqlite-utils/pull/654#issuecomment-2860924225

* Ran cog
2025-05-07 17:49:50 -07:00
Simon Willison
4dc2e2e9c8 Use REAL for floating point columns if table is strict, closes #644
Refs #645
2024-11-23 14:27:21 -08:00
liunux4odoo
7423296ec7
include_rank parameter for Table.search
* Add `include_rank` parameter to `Table.search`
* Test for .search(include_rank)
* Docs for table.search(include_rank)

https://github.com/simonw/sqlite-utils/pull/628

Refs #480

---------

Co-authored-by: Simon Willison <swillison@gmail.com>
2024-11-23 12:34:27 -08:00
Mat Miller
42230709f7
Recreate indexes when calling transform when possible (#634)
* Recreate indexes when calling transform when possible and raise an error when they cannot be retained automatically
* Docs for sqlite_utils.db.TransformError

Co-authored-by: Simon Willison <swillison@gmail.com>
2024-11-23 12:17:15 -08:00
Simon Willison
8906f57740 Hack to support reuse of memory command, closes #643 2024-11-08 11:17:17 -08:00
Simon Willison
dc79454234
Fix for np.int8 error, refs #632
Also refs:
- https://github.com/simonw/llm/issues/531
2024-07-18 11:32:48 -07:00
Simon Willison
70cc0c91ab
Drop support for Python 3.7
* Drop support for Python 3.7, closes #616
* Bumped versions on a whole lot of GitHub Actions
2024-01-30 18:37:38 -08:00
Simon Willison
ff57a97482 Upgrade to latest Black 2024-01-30 18:18:52 -08:00
Taj Khattra
1500c19bd0
Add more STRICT table support (#604)
* Add more STRICT table support per https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982014776.
* Make `table.transform()` preserve STRICT mode.
* Fix mypy failures in PR #604
* Link to SQLITE strict page in a few places
2023-12-07 21:05:27 -08:00
Simon Willison
88bd372205 str, int, bytes aliases for column types, closes #606 2023-12-06 10:49:21 -08:00
Harald Nezbeda
347fdc865e
Map datetime.timedelta to text columns, closes #522 2023-11-03 17:49:50 -07:00
Simon Willison
b92ea4793c Fixed spelling of wrappping 2023-11-03 17:31:49 -07:00
Simon Willison
1c6ea54338
.transform() now preserves rowid values
* .transform() now preserves rowid values, refs #592
* Test transform rowids against different table types, closes #592
2023-09-08 17:45:30 -07:00
Simon Willison
509857ee87
.add_foreign_keys() uses .transform() instead of PRAGMA writable_schema
Closes #577

This should solve all sorts of problems seen by users of platforms that throw errors on writable_schema.

Also added `add_foreign_keys=` and `foreign_keys=` parameters to `table.transform()`.
2023-08-17 17:48:08 -07:00
Simon Willison
374a816c72 Database(..., execute_plugins=False) mechanism, refs #575 2023-07-22 16:06:11 -07:00
Alex Garcia
3f80a02698
prepare_connection plugin hook
Closes:
- #574

Refs #567

---------

Co-authored-by: Simon Willison <swillison@gmail.com>
2023-07-22 15:59:08 -07:00
Simon Willison
61aaa69815 .transform(..., keep_table=name) parameter, closes #571
Also type hints for the transform_sql() method
2023-07-22 15:32:09 -07:00
Simon Willison
82e8cd3667 db.rename_table() method, refs #565 2023-07-22 12:38:31 -07:00
Simon Willison
86a352f8b7 Fixed mypy error, refs #568 2023-07-22 12:27:41 -07:00
Simon Willison
58b577279f table.create(..., replace=True / ignore = True) closes #568 2023-07-22 12:15:40 -07:00
Simon Willison
f5c63088e1
Use sqlean if available in environment (#560)
Closes #559
Closes #235

Refs https://github.com/simonw/llm/issues/60

- Uses `sqlean` in place of `sqlite3` if `sqlean.py` is installed
- Uses `sqlite-dump` if available and `conn.iterdump()` does not exist
- New `with db.ensure_autocommit_off()` method for ensuring autocommit is off, used by `enable_wal()` and `disable_wal()`.
2023-06-25 16:25:51 -07:00
Simon Willison
6027f3ea69 No need to show common values if everything is null
Closes #547
2023-05-21 10:19:16 -07:00
Simon Willison
d2a7b15b2b
Analyze tables options: --common-limit, --no-most, --no-least
Closes #544
2023-05-21 09:19:30 -07:00
Simon Willison
c764a9ee8f Avoid negative hashes in lambda names, refs #543 2023-05-08 15:12:39 -07:00
Simon Willison
eebd1a26ae Removed FunctionAlreadyRegistered error, refs #526, #525 2023-05-08 14:58:28 -07:00
Simon Willison
fca3ef8cf2 Applied Black, refs #526, #525 2023-05-08 14:54:24 -07:00
Martin Carpenter
02f5c4d69d
Support repeated calls to Table.convert()
* Test repeated calls to Table.convert()
* Register Table.convert() functions under their own `lambda_hash` name
* Raise exception on registering identical function names

Refs #525
2023-05-08 14:53:58 -07:00
rhoboro
6500fed8b2
Transform no longer breaks non-string default values
Closes #509
2023-05-08 14:13:36 -07:00
Scott Perry
39ef137e67
Support self-referencing FKs in create (#537) 2023-05-08 14:10:00 -07:00
Simon Willison
455c35b512 .convert(skip_false) option, refs #527 2023-05-08 13:52:21 -07:00
Simon Willison
2376c452a5 upsert_all() now works with not_null - refs #538 2023-05-08 12:24:19 -07:00
Simon Willison
92f77c3262 Ran against updated Black 2023-03-12 14:28:43 -07:00