dclient insert command (#13)

* Move making queries to own page, refs #7
* Documentation for the dclient insert feature, refs #8
* Implemented progress bar for insert, refs #8
* --pk option
* --ignore and --replace for insert, refs #8
* --batch-size option, refs #8
* First insert test, using a mock - refs #8
* insert test that exercises against an in-memory Datasette instance, refs #8
* Insert tests now cover an error case, refs #8
* Tests for --ignore and --replace, refs #8
* Tests for different formats, refs #8
* Test for --no-detect-types
* Support for --encoding, refs #8
This commit is contained in:
Simon Willison 2023-07-24 16:22:39 -07:00 committed by GitHub
commit c847badc4e
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
8 changed files with 619 additions and 8 deletions

71
docs/inserting.md Normal file
View file

@ -0,0 +1,71 @@
# Inserting data
The `dclient insert` command can be used to insert data from a local file directly into a Datasette instance, via the [Write API](https://docs.datasette.io/en/latest/json_api.html#the-json-write-api) introduced in the Datasette 1.0 alphas.
First you'll need to {ref}`authenticate <authentication>` with the instance.
To insert data from a `data.csv` file into a table called `my_table`, creating that table if it does not exist:
```bash
dclient insert https://my-private-space.datasette.cloud/data my_table data.csv --create
```
You can also pipe data into standard input:
```bash
curl -s 'https://api.github.com/repos/simonw/dclient/issues' | \
dclient insert \
https://my-private-space.datasette.cloud/data \
issues - --create
```
## Supported formats
Data can be inserted from CSV, TSV, JSON or newline-delimited JSON files.
The format of the file will be automatically detected. You can override this by using one of the following options:
- `--csv`
- `--tsv`
- `--json`
- `--nl` for newline-delimited JSON
Use `--encoding <encoding>` to specify the encoding of the file. The default is `utf-8`.
### JSON
JSON files should be formatted like this:
```json
[
{
"id": 1
"column1": "value1",
"column2": "value2"
},
{
"id": 2
"column1": "value1",
"column2": "value2"
}
]
```
Newline-delimited files like this:
```
{"id": 1, "column1": "value1", "column2": "value2"}
{"id": 2, "column1": "value1", "column2": "value2"}
```
### CSV and TSV
CSV and TSV files should have a header row containing the names of the columns.
By default, `dclient` will attempt to detect the types of the different columns in the CSV and TSV files - so if a column only ever contains numeric integers it will be stored as integers in the SQLite database.
You can disable this and have every value treated as a string using `--no-detect-types`.
### Other options
- `--create` - create the table if it doesn't already exist
- `--replace` - replace any rows with a matching primary key
- `--ignore` - ignore any rows with a matching existing primary key
- `--pk id` - set a primary key (for if the table is being created)
If you use `--create` a table will be created with rows to match the columns in your uploaded data - using the correctly detected types, unless you use `--no-detect-types` in which case every column will be of type `text`.