Commit graph

449 commits

Author SHA1 Message Date
Simon Willison
5a7f522980
sqlite-utils convert can now use callable references, closes #686
This now works:

    echo '{"date": "13th January 2025"}' | sqlite-utils insert dates.db dates -
    sqlite-utils convert dates.db dates date r.parsedate
    sqlite-utils rows dates.db dates

Previously the command would have been:

    sqlite-utils convert dates.db dates date 'r.parsedate(value)'
2025-11-29 14:43:02 -08:00
Simon Willison
35377a874b
Detect CSV/TSV column types by default (#683)
The `--detect-types` option is now automatically turned on for all commands that deal with CSV or CSV.

A new `--no-detect-types` option can be used to have all columns treated as text.

Closes #679
2025-11-23 22:23:15 -08:00
Simon Willison
0bbc68089c
--functions can take filenames, can be used multiple times (#681)
Closes #659

The --functions option now accepts:
- File paths ending in .py (e.g., --functions my_funcs.py)
- Multiple invocations (e.g., --functions foo.py --functions 'def bar(): ...')
- Inline Python code (existing behavior)

Implementation follows the same pattern as llm's --functions flag
(simonw/llm@a880c123).

Changes:
- Added multiple=True to --functions Click option in query, bulk, and memory commands
- Modified _register_functions() to detect and read .py files
- Updated _maybe_register_functions() to iterate over multiple function sources
- Removed unused bytes/bytearray handling
- Added comprehensive tests for file paths and multiple invocations
- Updated documentation with examples

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Shorter help for --functions

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-11-23 21:46:51 -08:00
Simon Willison
c872d27bb3
Use REAL not FLOAT as SQLite column type (#680)
* Use REAL not FLOAT as SQLite column type, refs #645
* Fix for REAL columns by CSV --detect-types

Refs https://github.com/simonw/sqlite-utils/issues/645#issuecomment-3568947189

* Removed note about strict and REAL
2025-11-23 21:37:59 -08:00
Simon Willison
fb93452ea8
Use double quotes not braces for tables and columns (#678)
Closes #677
2025-11-23 20:43:26 -08:00
Simon Willison
d328f7e765 Fix flake8 error 2025-11-23 19:11:10 -08:00
Simon Willison
96fab69256 Removed convert skip_false and --skip-false, closes #542 2025-11-23 15:40:28 -08:00
Simon Willison
fafa966300 Table.create() now stores configuration in _defaults
When configuration parameters like pk, foreign_keys, not_null, etc.
are passed to Table.create(), they are now stored in self._defaults
so that subsequent operations on the same Table instance (like insert,
upsert) will use those settings automatically.

Previously, calling db["table"].insert({...}, pk="id") would create
the table correctly but subsequent calls to insert() or upsert() on
the same table object would not know about the pk setting, causing
issues like upsert() failing with "requires a pk" error.

Closes #655

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-23 15:18:55 -08:00
Simon Willison
4f12c7a452 Port from setup.py to pyproject.toml
Refs #676, closes #675
2025-11-23 14:07:11 -08:00
Simon Willison
7ffd5052e9 table.insert_all() and table.upsert_all() now take generators of lists/tuples
Closes #672, PR #673
2025-11-23 12:17:23 -08:00
fry69
42d23f5954 test: suppress PytestUnraisableExceptionWarning in dateparse_errors 2025-11-23 11:58:06 -08:00
Simon Willison
094b010fd8
db.table() only returns tables, added db.view()
* db.table() only returns tables, added db.view(), closes #657
* Massive documentation update for db.table()

Refs #656
2025-05-08 23:19:36 -07:00
Simon Willison
8e7d018fa2
Use ON CONFLICT for upsert, refs #652
* New upsert implementation, refs #652
* supports_strict now caches on self._supports_strict

PR: https://github.com/simonw/sqlite-utils/pull/653
2025-05-08 20:37:49 -07:00
Simon Willison
0e4e270d44
Test against multiple SQLite versions (#654)
* Test against pre-upsert SQLite 3.23.1

Borrowed from 8f86d2af6

* Try this on Python 3.9

* select ... from pragma_function_list()

Refs https://github.com/simonw/sqlite-utils/pull/654#issuecomment-2860898278

* Fix spelling error

* Compatible with latest black

* Skip plugin test that needs pragma_function_list

Refs https://github.com/simonw/sqlite-utils/pull/654#issuecomment-2860924225

* Ran cog
2025-05-07 17:49:50 -07:00
Simon Willison
4381390cf1 Skip this test on DBs that don't support strict, refs #644 2024-11-23 14:32:58 -08:00
Simon Willison
4dc2e2e9c8 Use REAL for floating point columns if table is strict, closes #644
Refs #645
2024-11-23 14:27:21 -08:00
liunux4odoo
7423296ec7
include_rank parameter for Table.search
* Add `include_rank` parameter to `Table.search`
* Test for .search(include_rank)
* Docs for table.search(include_rank)

https://github.com/simonw/sqlite-utils/pull/628

Refs #480

---------

Co-authored-by: Simon Willison <swillison@gmail.com>
2024-11-23 12:34:27 -08:00
Mat Miller
42230709f7
Recreate indexes when calling transform when possible (#634)
* Recreate indexes when calling transform when possible and raise an error when they cannot be retained automatically
* Docs for sqlite_utils.db.TransformError

Co-authored-by: Simon Willison <swillison@gmail.com>
2024-11-23 12:17:15 -08:00
Simon Willison
2258b431d4 Neater return_db pattern for reusing memory command, closes #643 2024-11-08 12:11:17 -08:00
Simon Willison
8906f57740 Hack to support reuse of memory command, closes #643 2024-11-08 11:17:17 -08:00
Simon Willison
1d050dcdc7 insert-files multiple --pk support, closes #621 2024-03-16 17:33:31 -07:00
Simon Willison
da92a30679 Test for insert CLI with multiple --pk, refs #621 2024-03-16 17:26:02 -07:00
Simon Willison
23be5be1dc create-table now supports multiple --pk, refs #620 2024-03-16 17:05:39 -07:00
Konstantin Baikov
1dc5da3e5d
Use context managers for file writes in tests (#618)
This removes the need to explicitly close files
and avoids accidental unclosed files
2024-03-14 21:20:19 -07:00
Taj Khattra
1500c19bd0
Add more STRICT table support (#604)
* Add more STRICT table support per https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982014776.
* Make `table.transform()` preserve STRICT mode.
* Fix mypy failures in PR #604
* Link to SQLITE strict page in a few places
2023-12-07 21:05:27 -08:00
Simon Willison
88bd372205 str, int, bytes aliases for column types, closes #606 2023-12-06 10:49:21 -08:00
Simon Willison
b2e0cd066d Test and docs for timedelta support, refs #522 2023-11-03 17:56:34 -07:00
Simon Willison
1c6ea54338
.transform() now preserves rowid values
* .transform() now preserves rowid values, refs #592
* Test transform rowids against different table types, closes #592
2023-09-08 17:45:30 -07:00
Simon Willison
56093de078 sqlite-utils transform --add-foreign-key option, closes #585 2023-08-17 18:51:04 -07:00
Simon Willison
d2bcdc00c6 Swapped the order of a bunch of pytest comparisons
When I wrote this I thought constant == value was a better assertion. I no longer think that.
2023-08-17 18:05:13 -07:00
Simon Willison
b4735f794a Remove test.utils.collapse_whitespace, closes #583 2023-08-17 17:59:09 -07:00
Simon Willison
509857ee87
.add_foreign_keys() uses .transform() instead of PRAGMA writable_schema
Closes #577

This should solve all sorts of problems seen by users of platforms that throw errors on writable_schema.

Also added `add_foreign_keys=` and `foreign_keys=` parameters to `table.transform()`.
2023-08-17 17:48:08 -07:00
Simon Willison
993029f466 flake8 fixes 2023-08-17 17:19:02 -07:00
Simon Willison
374a816c72 Database(..., execute_plugins=False) mechanism, refs #575 2023-07-22 16:06:11 -07:00
Alex Garcia
3f80a02698
prepare_connection plugin hook
Closes:
- #574

Refs #567

---------

Co-authored-by: Simon Willison <swillison@gmail.com>
2023-07-22 15:59:08 -07:00
Simon Willison
61aaa69815 .transform(..., keep_table=name) parameter, closes #571
Also type hints for the transform_sql() method
2023-07-22 15:32:09 -07:00
Simon Willison
18f190e283 sqlite-utils rename-table command, refs #565 2023-07-22 12:48:04 -07:00
Simon Willison
82e8cd3667 db.rename_table() method, refs #565 2023-07-22 12:38:31 -07:00
Simon Willison
0c563e2d13 Fixed test I broke in #568 2023-07-22 12:23:42 -07:00
Simon Willison
58b577279f table.create(..., replace=True / ignore = True) closes #568 2023-07-22 12:15:40 -07:00
Simon Willison
b379a2a0c3 Added pluggy and first hook, register_commands - refs #569, #567 2023-07-22 12:04:31 -07:00
Simon Willison
f7af23837d --empty-null option for CSV and TSV imports, closes #563 2023-07-02 22:42:26 -07:00
Simon Willison
8c739558f7 --stop-after option, closes #561 2023-06-27 11:50:04 -07:00
Simon Willison
f5c63088e1
Use sqlean if available in environment (#560)
Closes #559
Closes #235

Refs https://github.com/simonw/llm/issues/60

- Uses `sqlean` in place of `sqlite3` if `sqlean.py` is installed
- Uses `sqlite-dump` if available and `conn.iterdump()` does not exist
- New `with db.ensure_autocommit_off()` method for ensuring autocommit is off, used by `enable_wal()` and `disable_wal()`.
2023-06-25 16:25:51 -07:00
Simon Willison
718b0cba9b
Experimental TUI powered by Trogon
* sqlite-utils tui command if Trogon is installed, closes #545
* Documentation for trogon TUI
* Screenshot of TUI
* Ignore trogon mypy error
* only run flake8 on Python 3.8 or higher, closes #550
2023-05-21 11:41:56 -07:00
Simon Willison
e8c5b042e4 Validate column names in analyze-columns, closes #548 2023-05-21 10:35:48 -07:00
Simon Willison
6027f3ea69 No need to show common values if everything is null
Closes #547
2023-05-21 10:19:16 -07:00
Simon Willison
d2a7b15b2b
Analyze tables options: --common-limit, --no-most, --no-least
Closes #544
2023-05-21 09:19:30 -07:00
Simon Willison
dab23884ae Better error message if rows_from_file called with StringIO, closes #520
Refs #448
2023-05-08 15:08:02 -07:00
Simon Willison
eebd1a26ae Removed FunctionAlreadyRegistered error, refs #526, #525 2023-05-08 14:58:28 -07:00