mirror of
https://github.com/simonw/sqlite-utils.git
synced 2026-07-24 18:04:32 +02:00
sqlite-utils memory command for directly querying CSV/JSON data
* Turn SQL errors into click errors * Initial CSV-only prototype of sqlite-utils memory, refs #272 * Implement --save plus tests for --save and --dump, refs #272 * Re-arranged CLI query documentation, refs #272 * Re-organized CLI query docs, refs #272 * Docs for --save and --dump plus made SQL optional for those, refs #273 * Replaced one last :memory: example * Documented --attach option for memory command, refs #272 * Improved arrangement of CLI query documentation
This commit is contained in:
Simon Willison
GitHub
parent
eea3851d40
commit
42ec59d8ee
4 changed files with 350 additions and 52 deletions
|
|
@ -47,7 +47,7 @@
|
|||
This release adds the ability to execute queries joining data from more than one database file - similar to the cross database querying feature introduced in `Datasette 0.55 <https://docs.datasette.io/en/stable/changelog.html#v0-55>`__.
|
||||
|
||||
- The ``db.attach(alias, filepath)`` Python method can be used to attach extra databases to the same connection, see :ref:`db.attach() in the Python API documentation <python_api_attach>`. (`#113 <https://github.com/simonw/sqlite-utils/issues/113>`__)
|
||||
- The ``--attach`` option attaches extra aliased databases to run SQL queries against directly on the command-line, see :ref:`attaching additional databases in the CLI documentation <cli_attach>`. (`#236 <https://github.com/simonw/sqlite-utils/issues/236>`__)
|
||||
- The ``--attach`` option attaches extra aliased databases to run SQL queries against directly on the command-line, see :ref:`attaching additional databases in the CLI documentation <cli_query_attach>`. (`#236 <https://github.com/simonw/sqlite-utils/issues/236>`__)
|
||||
|
||||
.. _v3_5:
|
||||
|
||||
|
|
|
|||
225
docs/cli.rst
225
docs/cli.rst
|
|
@ -8,25 +8,31 @@ The ``sqlite-utils`` command-line tool can be used to manipulate SQLite database
|
|||
|
||||
.. contents:: :local:
|
||||
|
||||
.. _cli_query_json:
|
||||
.. _cli_query:
|
||||
|
||||
Running queries and returning JSON
|
||||
==================================
|
||||
Running SQL queries
|
||||
===================
|
||||
|
||||
You can execute a SQL query against a database and get the results back as JSON like this::
|
||||
The ``sqlite-utils query`` command lets you run queries directly against a SQLite database file. This is the default subcommand, so the following two examples work the same way::
|
||||
|
||||
$ sqlite-utils query dogs.db "select * from dogs"
|
||||
$ sqlite-utils dogs.db "select * from dogs"
|
||||
|
||||
.. _cli_query_json:
|
||||
|
||||
Returning JSON
|
||||
--------------
|
||||
|
||||
The default format returned for queries is JSON::
|
||||
|
||||
$ sqlite-utils dogs.db "select * from dogs"
|
||||
[{"id": 1, "age": 4, "name": "Cleo"},
|
||||
{"id": 2, "age": 2, "name": "Pancakes"}]
|
||||
|
||||
This is the default command for ``sqlite-utils``, so you can instead use this::
|
||||
.. _cli_query_nl:
|
||||
|
||||
$ sqlite-utils dogs.db "select * from dogs"
|
||||
|
||||
You can pass named parameters to the query using ``-p``::
|
||||
|
||||
$ sqlite-utils query dogs.db "select :num * :num2" -p num 5 -p num2 6
|
||||
[{":num * :num2": 30}]
|
||||
Newline-delimited JSON
|
||||
~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use ``--nl`` to get back newline-delimited JSON objects::
|
||||
|
||||
|
|
@ -34,6 +40,11 @@ Use ``--nl`` to get back newline-delimited JSON objects::
|
|||
{"id": 1, "age": 4, "name": "Cleo"}
|
||||
{"id": 2, "age": 2, "name": "Pancakes"}
|
||||
|
||||
.. _cli_query_arrays:
|
||||
|
||||
JSON arrays
|
||||
~~~~~~~~~~~
|
||||
|
||||
You can use ``--arrays`` to request arrays instead of objects::
|
||||
|
||||
$ sqlite-utils dogs.db "select * from dogs" --arrays
|
||||
|
|
@ -62,6 +73,11 @@ If you want to pretty-print the output further, you can pipe it through ``python
|
|||
}
|
||||
]
|
||||
|
||||
.. _cli_query_binary_json:
|
||||
|
||||
Binary data in JSON
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Binary strings are not valid JSON, so BLOB columns containing binary data will be returned as a JSON object containing base64 encoded data, that looks like this::
|
||||
|
||||
$ sqlite-utils dogs.db "select name, content from images" | python -mjson.tool
|
||||
|
|
@ -75,27 +91,11 @@ Binary strings are not valid JSON, so BLOB columns containing binary data will b
|
|||
}
|
||||
]
|
||||
|
||||
If you execute an ``UPDATE``, ``INSERT`` or ``DELETE`` query the command will return the number of affected rows::
|
||||
|
||||
$ sqlite-utils dogs.db "update dogs set age = 5 where name = 'Cleo'"
|
||||
[{"rows_affected": 1}]
|
||||
|
||||
You can run queries against a temporary in-memory database by passing ``:memory:`` as the filename::
|
||||
|
||||
$ sqlite-utils :memory: "select sqlite_version()"
|
||||
[{"sqlite_version()": "3.29.0"}]
|
||||
|
||||
You can load SQLite extension modules using the ``--load-extension`` option, see :ref:`cli_load_extension`.
|
||||
|
||||
::
|
||||
|
||||
$ sqlite-utils :memory: "select spatialite_version()" --load-extension=spatialite
|
||||
[{"spatialite_version()": "4.3.0a"}]
|
||||
|
||||
.. _cli_json_values:
|
||||
|
||||
Nested JSON values
|
||||
------------------
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
If one of your columns contains JSON, by default it will be returned as an escaped string::
|
||||
|
||||
|
|
@ -126,24 +126,10 @@ You can use the ``--json-cols`` option to automatically detect these JSON column
|
|||
}
|
||||
]
|
||||
|
||||
.. _cli_attach:
|
||||
|
||||
Attaching additional databases
|
||||
------------------------------
|
||||
|
||||
SQLite supports cross-database SQL queries, which can join data from tables in more than one database file.
|
||||
|
||||
You can attach one or more additional databases using the ``--attach`` option, providing an alias to use for that database and the path to the SQLite file on disk.
|
||||
|
||||
This example attaches the ``books.db`` database under the alias ``books`` and then runs a query that combines data from that database with the default ``dogs.db`` database::
|
||||
|
||||
sqlite-utils dogs.db --attach books books.db \
|
||||
'select * from sqlite_master union all select * from books.sqlite_master'
|
||||
|
||||
.. _cli_query_csv:
|
||||
|
||||
Running queries and returning CSV
|
||||
=================================
|
||||
Returning CSV or TSV
|
||||
--------------------
|
||||
|
||||
You can use the ``--csv`` option to return results as CSV::
|
||||
|
||||
|
|
@ -167,8 +153,8 @@ Use ``--tsv`` instead of ``--csv`` to get back tab-separated values::
|
|||
|
||||
.. _cli_query_table:
|
||||
|
||||
Running queries and outputting a table
|
||||
======================================
|
||||
Table-formatted output
|
||||
----------------------
|
||||
|
||||
You can use the ``--table`` option (or ``-t`` shortcut) to output query results as a table::
|
||||
|
||||
|
|
@ -192,8 +178,8 @@ For a full list of table format options, run ``sqlite-utils query --help``.
|
|||
|
||||
.. _cli_query_raw:
|
||||
|
||||
Returning raw data from a query, such as binary content
|
||||
=======================================================
|
||||
Returning raw data, such as binary content
|
||||
------------------------------------------
|
||||
|
||||
If your table contains binary data in a ``BLOB`` you can use the ``--raw`` option to output specific columns directly to standard out.
|
||||
|
||||
|
|
@ -201,6 +187,145 @@ For example, to retrieve a binary image from a ``BLOB`` column and store it in a
|
|||
|
||||
$ sqlite-utils photos.db "select contents from photos where id=1" --raw > myphoto.jpg
|
||||
|
||||
|
||||
.. _cli_query_parameters:
|
||||
|
||||
Using named parameters
|
||||
----------------------
|
||||
|
||||
You can pass named parameters to the query using ``-p``::
|
||||
|
||||
$ sqlite-utils query dogs.db "select :num * :num2" -p num 5 -p num2 6
|
||||
[{":num * :num2": 30}]
|
||||
|
||||
These will be correctly quoted and escaped in the SQL query, providing a safe way to combine other values with SQL.
|
||||
|
||||
.. _cli_query_update_insert_delete:
|
||||
|
||||
UPDATE, INSERT and DELETE
|
||||
-------------------------
|
||||
|
||||
If you execute an ``UPDATE``, ``INSERT`` or ``DELETE`` query the command will return the number of affected rows::
|
||||
|
||||
$ sqlite-utils dogs.db "update dogs set age = 5 where name = 'Cleo'"
|
||||
[{"rows_affected": 1}]
|
||||
|
||||
SQLite extensions
|
||||
-----------------
|
||||
|
||||
You can load SQLite extension modules using the ``--load-extension`` option, see :ref:`cli_load_extension`.
|
||||
|
||||
::
|
||||
|
||||
$ sqlite-utils dogs.db "select spatialite_version()" --load-extension=spatialite
|
||||
[{"spatialite_version()": "4.3.0a"}]
|
||||
|
||||
.. _cli_query_attach:
|
||||
|
||||
Attaching additional databases
|
||||
------------------------------
|
||||
|
||||
SQLite supports cross-database SQL queries, which can join data from tables in more than one database file.
|
||||
|
||||
You can attach one or more additional databases using the ``--attach`` option, providing an alias to use for that database and the path to the SQLite file on disk.
|
||||
|
||||
This example attaches the ``books.db`` database under the alias ``books`` and then runs a query that combines data from that database with the default ``dogs.db`` database::
|
||||
|
||||
sqlite-utils dogs.db --attach books books.db \
|
||||
'select * from sqlite_master union all select * from books.sqlite_master'
|
||||
|
||||
.. _cli_query_memory:
|
||||
|
||||
Querying CSV data directly using an in-memory database
|
||||
======================================================
|
||||
|
||||
The ``sqlite-utils memory`` command works similar to ``sqlite-utils query``, but allows you to execute queries against an in-memory database.
|
||||
|
||||
You can also pass this command CSV files which will be loaded into a temporary in-memory table, allowing you to execute SQL against that data without a separate step to first convert it to SQLite.
|
||||
|
||||
Without any extra arguments, this command executes SQL against the in-memory database directly::
|
||||
|
||||
$ sqlite-utils memory 'select sqlite_version()'
|
||||
[{"sqlite_version()": "3.35.5"}]
|
||||
|
||||
It takes all of the same formatting options as :ref:`sqlite-utils query <cli_query>`: ``--csv`` and ``--csv`` and ``--table`` and ``--nl``::
|
||||
|
||||
$ sqlite-utils memory 'select sqlite_version()' --csv
|
||||
sqlite_version()
|
||||
3.35.5
|
||||
$ sqlite-utils memory 'select sqlite_version()' --table --fmt grid
|
||||
+--------------------+
|
||||
| sqlite_version() |
|
||||
+====================+
|
||||
| 3.35.5 |
|
||||
+--------------------+
|
||||
|
||||
.. _cli_query_memory_csv:
|
||||
|
||||
Running queries directly against CSV
|
||||
------------------------------------
|
||||
|
||||
If you have data in CSV format you can load it into an in-memory SQLite database and run queries against it directly in a single command using ``sqlite-utils memory`` like this::
|
||||
|
||||
$ sqlite-utils memory data.csv "select * from data"
|
||||
|
||||
You can pass multiple files to the command if you want to run joins between different CSV files::
|
||||
|
||||
$ sqlite-utils memory one.csv two.csv "select * from one join two on one.id = two.other_id"
|
||||
|
||||
The in-memory tables will be named after the CSV files without their ``.csv`` extension. The tool also sets up aliases for those tables (using SQL views) as ``t1``, ``t2`` and so on, or you can use the alias ``t`` to refer to the first table::
|
||||
|
||||
$ sqlite-utils memory example.csv "select * from t"
|
||||
|
||||
To read from standard input, use ``-`` as the filename - then use ``stdin`` or ``t`` or ``t1`` as the table name::
|
||||
|
||||
$ cat example.csv | sqlite-utils memory - "select * from stdin"
|
||||
|
||||
.. _cli_query_memory_attach:
|
||||
|
||||
Joining in-memory data against existing databases using \-\-attach
|
||||
------------------------------------------------------------------
|
||||
|
||||
The :ref:`attach option <cli_query_attach>` can be used to attach database files to the in-memory connection, enabling joins between in-memory data loaded from a file and tables in existing SQLite database files. An example::
|
||||
|
||||
$ echo "id\n1\n3\n5" | sqlite-utils memory - --attach trees trees.db \
|
||||
"select * from trees.trees where rowid in (select id from stdin)"
|
||||
|
||||
Here the ``--attach trees trees.db`` option makes the ``trees.db`` database available with an alias of ``trees``.
|
||||
|
||||
``select * from trees.trees where ...`` can then query the ``trees`` table in that database.
|
||||
|
||||
The CSV data that was piped into the script is available in the ``stdin`` table, so ``... where rowid in (select id from stdin)`` can be used to return rows from the ``trees`` table that match IDs that were piped in as CSV content.
|
||||
|
||||
.. _cli_query_memory_dump_save:
|
||||
|
||||
\-\-dump and \-\-save
|
||||
---------------------
|
||||
|
||||
You can dump out the SQL used for the temporary in-memory database, complete with all imported data, using the ``--dump`` option::
|
||||
|
||||
% sqlite-utils memory dogs.csv --dump
|
||||
BEGIN TRANSACTION;
|
||||
CREATE TABLE [dogs] (
|
||||
[rowid] TEXT,
|
||||
[id] TEXT,
|
||||
[dog_age] TEXT,
|
||||
[name] TEXT
|
||||
);
|
||||
INSERT INTO "dogs" VALUES('1','1','4','Cleo');
|
||||
INSERT INTO "dogs" VALUES('2','2','2','Pancakes');
|
||||
INSERT INTO "dogs" VALUES('3','2','3','Pancakes');
|
||||
CREATE VIEW t1 AS select * from [dogs];
|
||||
CREATE VIEW t AS select * from [dogs];
|
||||
COMMIT;
|
||||
|
||||
Passing ``--save other.db`` will instead use that SQL to populate a new database file::
|
||||
|
||||
% sqlite-utils memory dogs.csv --save dogs.db
|
||||
|
||||
These features are mainly intented as debugging tools - for much more finely grained control over how data is inserted into a SQLite database file see :ref:`cli_inserting_data` and :ref:`cli_insert_csv_tsv`.
|
||||
|
||||
|
||||
.. _cli_rows:
|
||||
|
||||
Returning all rows in a table
|
||||
|
|
@ -270,7 +395,7 @@ Use ``--schema`` to include the schema of each table::
|
|||
[age] INTEGER,
|
||||
[name] TEXT)
|
||||
|
||||
The ``--nl``, ``--csv``, ``--tsv`` and ``--table`` options are all available.
|
||||
The ``--nl``, ``--csv``, ``--tsv``, ``--table`` and ``--fmt`` options are also available.
|
||||
|
||||
.. _cli_views:
|
||||
|
||||
|
|
@ -1280,5 +1405,5 @@ This option can be applied multiple times to load multiple extensions.
|
|||
|
||||
Since `SpatiaLite <https://www.gaia-gis.it/fossil/libspatialite/index>`__ is commonly used with SQLite, the value ``spatialite`` is special: it will search for SpatiaLite in the most common installation locations, saving you from needing to remember exactly where that module is located::
|
||||
|
||||
$ sqlite-utils :memory: "select spatialite_version()" --load-extension=spatialite
|
||||
$ sqlite-utils memory "select spatialite_version()" --load-extension=spatialite
|
||||
[{"spatialite_version()": "4.3.0a"}]
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue