mirror of https://github.com/simonw/dclient.git synced 2026-07-22 17:04:32 +02:00

Simon Willison c847badc4e

* Move making queries to own page, refs #7
* Documentation for the dclient insert feature, refs #8
* Implemented progress bar for insert, refs #8
* --pk option
* --ignore and --replace for insert, refs #8
* --batch-size option, refs #8
* First insert test, using a mock - refs #8
* insert test that exercises against an in-memory Datasette instance, refs #8
* Insert tests now cover an error case, refs #8
* Tests for --ignore and --replace, refs #8
* Tests for different formats, refs #8
* Test for --no-detect-types
* Support for --encoding, refs #8

2023-07-24 16:22:39 -07:00

2.4 KiB

Raw Blame History

Inserting data

The dclient insert command can be used to insert data from a local file directly into a Datasette instance, via the Write API introduced in the Datasette 1.0 alphas.

First you'll need to {ref}authenticate <authentication> with the instance.

To insert data from a data.csv file into a table called my_table, creating that table if it does not exist:

dclient insert https://my-private-space.datasette.cloud/data my_table data.csv --create

You can also pipe data into standard input:

curl -s 'https://api.github.com/repos/simonw/dclient/issues' | \
  dclient insert \
    https://my-private-space.datasette.cloud/data \
    issues - --create

Supported formats

Data can be inserted from CSV, TSV, JSON or newline-delimited JSON files.

The format of the file will be automatically detected. You can override this by using one of the following options:

--csv
--tsv
--json
--nl for newline-delimited JSON

Use --encoding <encoding> to specify the encoding of the file. The default is utf-8.

JSON

JSON files should be formatted like this:

[
    {
        "id": 1
        "column1": "value1",
        "column2": "value2"
    },
    {
        "id": 2
        "column1": "value1",
        "column2": "value2"
    }
]

Newline-delimited files like this:

{"id": 1, "column1": "value1", "column2": "value2"}
{"id": 2, "column1": "value1", "column2": "value2"}

CSV and TSV

CSV and TSV files should have a header row containing the names of the columns.

By default, dclient will attempt to detect the types of the different columns in the CSV and TSV files - so if a column only ever contains numeric integers it will be stored as integers in the SQLite database.

You can disable this and have every value treated as a string using --no-detect-types.

Other options

--create - create the table if it doesn't already exist
--replace - replace any rows with a matching primary key
--ignore - ignore any rows with a matching existing primary key
--pk id - set a primary key (for if the table is being created)

If you use --create a table will be created with rows to match the columns in your uploaded data - using the correctly detected types, unless you use --no-detect-types in which case every column will be of type text.

2.4 KiB Raw Blame History