Commit graph

1,037 commits

Author SHA1 Message Date
Simon Willison
8f0c06e188
Test against Python 3.15-dev, bump ty and Black (#738)
* Add Python 3.15-dev to test matrix
* Run ty check only on 3.14
* Bump Black version
* Update tabulate and use that in 
* Bump to latest ty
2026-05-17 16:52:48 -07:00
Simon Willison
8d74ffc932
More type annotations (#697)
* Add comprehensive type annotations

- mypy.ini: expanded configuration with module-specific settings
- hookspecs.py: type annotations for hook functions
- plugins.py: typed get_plugins() return value
- recipes.py: full type annotations for parsedate, parsedatetime, jsonsplit
- utils.py: extensive type annotations including Row type alias,
  TypeTracker, ValueTracker, and all utility functions
- db.py: type annotations for Database methods (__exit__,
  ensure_autocommit_off, tracer, register_function, etc.) and
  Queryable class methods
- tests/test_docs.py: updated to match new signature display format

* Fix type errors caught by ty check

- Add type: ignore comments for external library type stub limitations
  (csv.reader, click.progressbar, IOBase.name, Callable.__name__)
- Change Iterable to Sequence for SQL where_args parameters
- Use db.table() instead of db[name] for proper Table return type
- Fix rebuild_fts return type from None to Table
- Update test_tracer to expect fewer queries (optimization side effect)

* Fix mypy type errors

- Add type: ignore comments for runtime-valid patterns mypy can't verify
- Fix new_column_types annotation to Dict[str, Set[type]]
- Add type: ignore for Default sentinel values passed to create_table

* mypy skip tests directory

* Fix CI: exclude typing imports from recipe docs, skip mypy on tests

- Add Callable and Optional to exclusion list in _generate_convert_help()
- Regenerate docs/cli-reference.rst with cog
- Add [mypy-tests.*] ignore_errors = True to skip test type errors

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-16 22:11:47 -08:00
Simon Willison
871038505b
Type fixes now enforced by ty
* Fix type warning for pipe.stdout possibly being None

Add conditional check before calling .read() on pipe.stdout since
Popen can return None for stdout.

* Use ctx.meta instead of dynamic attribute for database cleanup

Click's Context.meta dictionary is the proper way to store arbitrary
data on the context object, avoiding type checker warnings about
dynamic attribute assignment.

* Add assert for tables.callback before calling

Click's callback attribute is typed as Optional[Callable], so add
assert to satisfy type checker that it's not None.

* Fix type errors in cli.py and db.py

- Add type annotation for Database.conn to fix context manager errors
- Convert exception objects to str() when raising ClickException
- Handle None return from find_spatialite() with proper error message

* Fix remaining type errors in cli.py

- Add typing import and type annotations for dict kwargs
- Use db.table() instead of db[] for extract command
- Fix missing str() conversion for exception

* Fix type errors in db.py

- Add type annotation for Database.conn
- Add type: ignore for optional sqlite_dump import
- Update execute/query parameter types to Sequence|Dict for sqlite3 compatibility
- Use getattr for fn.__name__ access to handle callables without __name__
- Handle None return from find_spatialite() with OSError
- Fix pk_values assignment to use local variable

* Add type: ignore for optional pysqlite3 and sqlean imports

These are alternative sqlite3 implementations that may not be installed.

* Fix type errors in tests and plugins

- Add type: ignore for monkey-patching Database.__init__ in conftest
- Fix CLI test to pass string "2" instead of integer to Click invoke
- Add type: ignore for optional sqlean import
- Fix add_geometry_column test to use "XY" instead of integer 2
- Add type: ignore for click.Context as context manager
- Add type: ignore for enable_fts test that intentionally omits argument
- Add type: ignore for sys._called_from_test dynamic attribute
- Fix rows_from_file test type error for intentional wrong argument
- Handle None from pm.get_hookcallers in plugins.py

* Use db.table() instead of db[] for Table-specific operations

Changes db[table] to db.table(table) in CLI commands where we know
we're working with tables, not views. This resolves most of the
Table | View disambiguation type warnings since db.table() returns
Table directly rather than Table | View.

* Fix remaining type warnings in sqlite_utils package

- Add assert for sniff_buffer not being None
- Handle cursor.fetchone() potentially returning None
- Use db.table() for counts_table and index_foreign_keys
- Add type: ignore for cursor union type in raw mode

* Ran Black

* Run ty in CI

* ty check sqlite_utils

* Skip running ty on Windows

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-16 19:34:45 -08:00
Simon Willison
fd5b09f64b
Database as a context manager, fixed many pytest warnings
* Database can now work as a context manager
* Claude Code helped fix a ton of .close() warnings

https://gistpreview.github.io/?730f0c5dc38528a1dd0615f330bd5481

* New autouse fixture to help with test warnings

Refs https://github.com/simonw/sqlite-utils/issues/692#issuecomment-3644371889

* Fix all remaining resource warnings

https://gistpreview.github.io/?0bb8e869b82f6ff0db647de755182502

Closes #692
2025-12-11 16:56:12 -08:00
Simon Willison
066d0f3e8d pip install . --group dev 2025-12-11 15:33:25 -08:00
Simon Willison
29d84bc95d
dependency-groups.dev and uv in docs (#691)
* dependency-groups.dev and switch to uv in docs
* Separate dependency group for docs
2025-12-11 15:32:00 -08:00
Simon Willison
5a7f522980
sqlite-utils convert can now use callable references, closes #686
This now works:

    echo '{"date": "13th January 2025"}' | sqlite-utils insert dates.db dates -
    sqlite-utils convert dates.db dates date r.parsedate
    sqlite-utils rows dates.db dates

Previously the command would have been:

    sqlite-utils convert dates.db dates date 'r.parsedate(value)'
2025-11-29 14:43:02 -08:00
Simon Willison
f9e5fbb3bd Backport release notes for 3.39, closes #688 2025-11-24 10:46:34 -08:00
Copilot
e66e82f2c8
Remove Sentinel workaround for Click 8.3.1+ (#685)
Closes #666

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
2025-11-24 09:11:18 -08:00
Simon Willison
a0d8bc3932 Add pip as a dependency to ensure plugin installation, closes #687 2025-11-24 07:58:54 -08:00
Simon Willison
ae58faf274 Add 3.14 to the publish.yml workflow 2025-11-23 22:34:59 -08:00
Simon Willison
8c40732e4f Changelog markup fixes, refs #682 4.0a1 2025-11-23 22:32:34 -08:00
Simon Willison
9ea950e3ba Removed duplicate changelog entry, refs #682 2025-11-23 22:30:15 -08:00
Simon Willison
69ee540454 Release 4.0a1
Refs #682, #542, #645, #655, #657, #659, #672, #675, #677, #679
2025-11-23 22:27:18 -08:00
Simon Willison
6dad400e14 Release notes for 4.0a1
Refs #682, #542, #645, #655, #657, #659, #672, #675, #677, #679
2025-11-23 22:26:32 -08:00
Simon Willison
35377a874b
Detect CSV/TSV column types by default (#683)
The `--detect-types` option is now automatically turned on for all commands that deal with CSV or CSV.

A new `--no-detect-types` option can be used to have all columns treated as text.

Closes #679
2025-11-23 22:23:15 -08:00
Simon Willison
0bbc68089c
--functions can take filenames, can be used multiple times (#681)
Closes #659

The --functions option now accepts:
- File paths ending in .py (e.g., --functions my_funcs.py)
- Multiple invocations (e.g., --functions foo.py --functions 'def bar(): ...')
- Inline Python code (existing behavior)

Implementation follows the same pattern as llm's --functions flag
(simonw/llm@a880c123).

Changes:
- Added multiple=True to --functions Click option in query, bulk, and memory commands
- Modified _register_functions() to detect and read .py files
- Updated _maybe_register_functions() to iterate over multiple function sources
- Removed unused bytes/bytearray handling
- Added comprehensive tests for file paths and multiple invocations
- Updated documentation with examples

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Shorter help for --functions

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-11-23 21:46:51 -08:00
Simon Willison
c872d27bb3
Use REAL not FLOAT as SQLite column type (#680)
* Use REAL not FLOAT as SQLite column type, refs #645
* Fix for REAL columns by CSV --detect-types

Refs https://github.com/simonw/sqlite-utils/issues/645#issuecomment-3568947189

* Removed note about strict and REAL
2025-11-23 21:37:59 -08:00
Simon Willison
fb93452ea8
Use double quotes not braces for tables and columns (#678)
Closes #677
2025-11-23 20:43:26 -08:00
Simon Willison
bf1ac778a3 Use uv in the Justfile, refs #674 2025-11-23 19:50:06 -08:00
Simon Willison
d328f7e765 Fix flake8 error 2025-11-23 19:11:10 -08:00
Simon Willison
52ea7d21f4 Fix for mypy errors 2025-11-23 15:53:19 -08:00
Simon Willison
96fab69256 Removed convert skip_false and --skip-false, closes #542 2025-11-23 15:40:28 -08:00
Simon Willison
fafa966300 Table.create() now stores configuration in _defaults
When configuration parameters like pk, foreign_keys, not_null, etc.
are passed to Table.create(), they are now stored in self._defaults
so that subsequent operations on the same Table instance (like insert,
upsert) will use those settings automatically.

Previously, calling db["table"].insert({...}, pk="id") would create
the table correctly but subsequent calls to insert() or upsert() on
the same table object would not know about the pk setting, causing
issues like upsert() failing with "requires a pk" error.

Closes #655

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-23 15:18:55 -08:00
Simon Willison
d5f113de48 .gitignore uv.lock and dist 2025-11-23 14:12:18 -08:00
Simon Willison
4f12c7a452 Port from setup.py to pyproject.toml
Refs #676, closes #675
2025-11-23 14:07:11 -08:00
Benson Muite
1587feca6f Use PEP639 license metadata formatting 2025-11-23 12:18:18 -08:00
Simon Willison
7ffd5052e9 table.insert_all() and table.upsert_all() now take generators of lists/tuples
Closes #672, PR #673
2025-11-23 12:17:23 -08:00
Simon Willison
dfb2dbe967 Update GitHub Actions workflow cache configuration 2025-11-23 12:12:55 -08:00
Simon Willison
076c1a0aa2 Ran black 2025-11-23 11:59:19 -08:00
fry69
42d23f5954 test: suppress PytestUnraisableExceptionWarning in dateparse_errors 2025-11-23 11:58:06 -08:00
fry69
10957305be test: fix Python 3.14 datetime deprecation 2025-11-23 11:58:06 -08:00
Simon Willison
c479ca0f44 SQLite tests against Python 3.10 2025-10-01 13:52:10 -07:00
Simon Willison
370318c695 Lowercase for geometry types
Cog test failed in CI, but these are case insensitive anyway.
2025-10-01 13:52:10 -07:00
Simon Willison
f91e4c9e52 Run cog with --diff to show errors 2025-10-01 13:52:10 -07:00
Simon Willison
1361ed5711 A bunch of fixes for Click sentinal stuff 2025-10-01 13:52:10 -07:00
Simon Willison
bccd05c9b4 Test on 3.14 and 3.14t, drop 3.9 2025-10-01 13:52:10 -07:00
Alex Chan
cf1b407207
Add a type hint for db.close() (#663)
Closes #662
2025-10-01 13:37:05 -07:00
Simon Willison
094b010fd8
db.table() only returns tables, added db.view()
* db.table() only returns tables, added db.view(), closes #657
* Massive documentation update for db.table()

Refs #656
2025-05-08 23:19:36 -07:00
Simon Willison
d892d2ae49 Fix underline length, refs #656 4.0a0 2025-05-08 20:53:24 -07:00
Simon Willison
9e458dea7d Release 0.4a0
Refs #646, #648, #652, #654, #656
2025-05-08 20:51:48 -07:00
Simon Willison
8e7d018fa2
Use ON CONFLICT for upsert, refs #652
* New upsert implementation, refs #652
* supports_strict now caches on self._supports_strict

PR: https://github.com/simonw/sqlite-utils/pull/653
2025-05-08 20:37:49 -07:00
Simon Willison
72f6c820f6 Small documentation updates to fix warnings 2025-05-07 17:54:29 -07:00
Simon Willison
0e4e270d44
Test against multiple SQLite versions (#654)
* Test against pre-upsert SQLite 3.23.1

Borrowed from 8f86d2af6

* Try this on Python 3.9

* select ... from pragma_function_list()

Refs https://github.com/simonw/sqlite-utils/pull/654#issuecomment-2860898278

* Fix spelling error

* Compatible with latest black

* Skip plugin test that needs pragma_function_list

Refs https://github.com/simonw/sqlite-utils/pull/654#issuecomment-2860924225

* Ran cog
2025-05-07 17:49:50 -07:00
Simon Willison
88a83f0b7e
Link to JSON Lines instead
ndjson.org has an expired domain.
2025-05-04 22:50:10 -07:00
Simon Willison
0aefbb634d Removed trogon feature in favor of sqlite-util-tui
Refs #648, #545
2025-01-10 17:15:02 -08:00
Simon Willison
04107d3fea Remove note from docs about 3.8 and deterministic, closes #646 2024-11-23 14:49:15 -08:00
Simon Willison
c544f75446 Remove 3.8 from trove tags, refs #646 2024-11-23 14:48:55 -08:00
Simon Willison
1290c50f71 Drop Python 3.9, add Python 3.13 to all CI
Refs #646
2024-11-23 14:43:04 -08:00
Simon Willison
9d7da0606e Release 3.38 3.38
Refs #628, #633, #643, #644
2024-11-23 14:38:40 -08:00