Compare commits

...

9 commits

Author SHA1 Message Date
Claude
bad841cb5c
Add demos/exploring-writing.md showboat demo
Executable demo covering all new commands against a live Datasette:
- create-table, insert (CSV via stdin)
- databases, tables, schema (exploration)
- rows with --table, -w filtering, --sort-desc, --col, --csv, --limit
- get (single row by PK)
- query with --table and --nl output formats
- update (single row by PK with key=value pairs)
- upsert (bulk insert-or-update from JSON)
- delete and drop with --yes

Replaces the old root-level demo.md with a proper demos/ directory.

https://claude.ai/code/session_01DoiJf9Gbvbw2asN9yvP6dx
2026-02-12 01:34:47 +00:00
Simon Willison
eadb314ee4
Update cache-dependency-path to pyproject.toml 2026-02-11 17:31:08 -08:00
Simon Willison
13ae71e08e
Update Python versions and actions in workflow 2026-02-11 17:30:30 -08:00
Claude
02dd3ae9ca
Update documentation for new commands and output formats
- Update README.md with new commands summary and uv development workflow
- Add docs/exploring.md covering databases, tables, schema, rows, get
  commands with filtering, sorting, pagination, and output format examples
- Add docs/writing.md covering create-table, upsert, update, delete, drop
- Update docs/queries.md with output format options (--csv, --tsv, --nl, --table)
- Update docs/inserting.md with cross-references to upsert/update
- Update docs/index.md toctree to include new pages
- Regenerate all cog --help blocks with current command signatures

https://claude.ai/code/session_01DoiJf9Gbvbw2asN9yvP6dx
2026-02-12 01:28:52 +00:00
Simon Willison
364c344d8d
Update Python versions and workflow actions in test.yml 2026-02-11 17:27:53 -08:00
Claude
278c3b67e2
Add follow_redirects to API GET requests and showboat demo
- Fix _api_get to follow redirects (needed for query endpoint on
  newer Datasette which redirects /{db}.json?sql= to /{db}/-/query)
- Add demo.md showing all new commands against a live Datasette instance:
  create-table, insert, databases, tables, schema, rows (with filtering,
  sorting, column selection, CSV export), get, query, update, upsert,
  delete, drop

https://claude.ai/code/session_01DoiJf9Gbvbw2asN9yvP6dx
2026-02-12 00:32:58 +00:00
Claude
5ee8b2d93c
Implement T1 and T2: exploration, CRUD commands, and output formats
Refactored shared infrastructure:
- _resolve_url(), _resolve_token(), _api_get(), _api_post() helpers
- Shared output formatters: JSON, CSV, TSV, NL-JSON, text table
- DCLIENT_TOKEN and DCLIENT_URL environment variable support

T1 - Core table browsing:
- `databases` - list databases on a Datasette instance
- `tables` - list tables with optional --schema flag
- `schema` - show CREATE TABLE SQL for database/table
- `rows` - browse rows with filtering (-w), sorting, pagination,
  column selection, faceting, full-text search, --limit, --all
- `get` - fetch a single row by primary key

T2 - Write operations:
- `upsert` - insert-or-update rows from file (CSV/TSV/JSON/JSONL)
- `update` - update a single row by PK with key=value pairs
- `delete` - delete a row by PK (with --yes to skip confirmation)
- `drop` - drop a table (shows row count, requires confirmation)
- `create-table` - create empty table with --column name type --pk

Output formats added to query and rows:
- --csv, --tsv, --nl, --json, --table flags

23 new tests covering all commands (62 total, all passing).

https://claude.ai/code/session_01DoiJf9Gbvbw2asN9yvP6dx
2026-02-12 00:22:12 +00:00
Claude
9715510431
Migrate from setup.py to pyproject.toml for uv compatibility
- Replace setup.py with pyproject.toml using uv_build backend
- Add dependency-groups.dev for test dependencies
- Configure flat layout with module-root = ""
- Fix pre-existing test failures (updated error message, optional mock)
- Add uv.lock and dist/ to .gitignore
- uv run pytest (39 passed) and uv build both work

https://claude.ai/code/session_01DoiJf9Gbvbw2asN9yvP6dx
2026-02-12 00:16:43 +00:00
Claude
2a8836b272
Add comprehensive improvement plan comparing dclient to Datasette features
Detailed feature gap analysis and tiered implementation plan covering:
- Core table browsing and schema exploration (databases, tables, schema, rows, get)
- Write operations (upsert, update, delete, drop, create-table)
- Output format improvements (CSV, TSV, table, NL-JSON)
- Authentication UX (browser login, env vars, keyring, auth status)
- Discoverability features (info, URL inference, shell completions)

https://claude.ai/code/session_01DoiJf9Gbvbw2asN9yvP6dx
2026-02-12 00:09:00 +00:00
17 changed files with 2936 additions and 178 deletions

View file

@ -12,18 +12,18 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
uses: actions/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
cache: pip
cache-dependency-path: setup.py
cache-dependency-path: pyproject.toml
- name: Install dependencies
run: |
pip install '.[test]'
pip install . --group dev
- name: Run tests
run: |
pytest
@ -34,13 +34,13 @@ jobs:
permissions:
id-token: write
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- name: Set up Python
uses: actions/setup-python@v5
uses: actions/setup-python@v6
with:
python-version: "3.12"
python-version: "3.14"
cache: pip
cache-dependency-path: setup.py
cache-dependency-path: pyproject.toml
- name: Install dependencies
run: |
pip install setuptools wheel build

View file

@ -10,18 +10,18 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
uses: actions/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
cache: pip
cache-dependency-path: setup.py
cache-dependency-path: pyproject.toml
- name: Install dependencies
run: |
pip install -e '.[test]'
pip install . --group dev
- name: Run tests
run: |
pytest

2
.gitignore vendored
View file

@ -7,3 +7,5 @@ venv
.pytest_cache
*.egg-info
.DS_Store
uv.lock
dist/

529
PLAN.md Normal file
View file

@ -0,0 +1,529 @@
# dclient Improvement Plan
## Feature Gap Analysis: dclient vs Datasette
### What dclient currently supports
| Feature | Status |
|---------|--------|
| SQL query execution (`query`) | Supported |
| Bulk data insertion (`insert`) | Supported (CSV, TSV, JSON, JSONL) |
| Actor/token introspection (`actor`) | Supported |
| URL aliases (`alias add/list/remove`) | Supported |
| Token storage (`auth add/list/remove`) | Supported |
| Bearer token authentication | Supported |
| Progress bar for inserts | Supported |
| Verbose/debug mode | Supported |
| Datasette plugin (`datasette dc`) | Supported |
### What Datasette exposes that dclient does NOT cover
#### Read Operations (high value)
| Datasette Feature | API Endpoint | dclient Status |
|-------------------|-------------|----------------|
| List databases | `GET /-/databases.json` | **Missing** |
| List tables/views in a database | `GET /{db}.json` | **Missing** |
| Browse/filter table rows | `GET /{db}/{table}.json?filters...` | **Missing** |
| Get single row by PK | `GET /{db}/{table}/{pk}.json` | **Missing** |
| Table schema inspection | `GET /{db}/{table}/-/schema.json` | **Missing** |
| Database schema inspection | `GET /{db}/-/schema.json` | **Missing** |
| Full-text search | `?_search=term` | **Missing** |
| Column filtering (exact, gt, lt, contains, etc.) | `?col__op=val` | **Missing** |
| Faceted browsing | `?_facet=col` | **Missing** |
| Sorting | `?_sort=col` / `?_sort_desc=col` | **Missing** |
| Pagination | `?_next=cursor&_size=N` | **Missing** |
| Column selection | `?_col=a&_col=b` / `?_nocol=x` | **Missing** |
| CSV/TSV export of table data | `GET /{db}/{table}.csv` | **Missing** |
| Canned queries listing | metadata in `GET /{db}.json` | **Missing** |
#### Write Operations (high value)
| Datasette Feature | API Endpoint | dclient Status |
|-------------------|-------------|----------------|
| Row upsert (insert-or-update) | `POST /{db}/{table}/-/upsert` | **Missing** |
| Update a single row | `POST /{db}/{table}/{pk}/-/update` | **Missing** |
| Delete a single row | `POST /{db}/{table}/{pk}/-/delete` | **Missing** |
| Drop a table | `POST /{db}/{table}/-/drop` | **Missing** |
| Create empty table with schema | `POST /{db}/-/create` (columns only) | **Missing** |
#### Metadata/Admin (medium value)
| Datasette Feature | API Endpoint | dclient Status |
|-------------------|-------------|----------------|
| Instance version info | `GET /-/versions.json` | **Missing** |
| Installed plugins | `GET /-/plugins.json` | **Missing** |
| Instance settings | `GET /-/settings.json` | **Missing** |
| Token creation | `POST /-/create-token` | **Missing** |
| Permission checks | `GET /-/allowed.json` | **Missing** |
---
## Proposed Plan
The plan is organized into tiers by impact and complexity, with the most
valuable additions first.
---
### Tier 1 -- Core Table Browsing & Schema Exploration
These commands turn dclient from "a way to run raw SQL" into a genuine
database exploration tool. They make up the biggest gap today: a user who
wants to see what tables exist or browse rows must either open a browser
or hand-craft SQL.
#### 1.1 `dclient databases` -- List databases
```
dclient databases <instance_url_or_alias>
```
Hits `GET {url}/-/databases.json`. Prints a table of database names, sizes,
and whether they are mutable. Add `--json` for raw JSON output.
#### 1.2 `dclient tables` -- List tables and views
```
dclient tables <db_url_or_alias> # e.g. https://latest.datasette.io/fixtures
dclient tables <db_url_or_alias> --views # include views
dclient tables <db_url_or_alias> --counts # include row counts
dclient tables <db_url_or_alias> --schema # include CREATE TABLE SQL
```
Hits `GET {db}.json`. Prints table names, row counts, and column lists.
`--json` for raw JSON.
#### 1.3 `dclient schema` -- Show table/database schema
```
dclient schema <db_url_or_alias> # full database schema
dclient schema <db_url_or_alias>/<table> # single table schema
```
Hits `GET {db}/-/schema.json` or `GET {db}/{table}/-/schema.json`.
Outputs the `CREATE TABLE` / `CREATE VIEW` SQL.
#### 1.4 `dclient rows` -- Browse table rows with filtering
This is the biggest single feature addition. It exposes Datasette's
powerful table filtering API without requiring the user to write SQL.
```
dclient rows <table_url_or_alias> [OPTIONS]
```
**Filter flags:**
```
--where 'column = value' # maps to ?column__exact=value
--where 'column > 100' # maps to ?column__gt=100
--where 'column contains foo' # maps to ?column__contains=foo
--where 'column is null' # maps to ?column__isnull
--where-sql 'age > 30' # maps to ?_where=age > 30 (raw SQL)
```
Or a simpler column-value approach:
```
-w column=value # exact
-w column__gt=100 # pass Datasette filter operators directly
```
**Other flags:**
```
--sort column # ascending sort
--sort-desc column # descending sort
--search 'full text query' # FTS search (?_search=...)
--col name --col email # select specific columns
--nocol description # exclude columns
--facet column # facet counts
--size 50 # page size (default 100)
--all # auto-paginate through all pages
--limit N # stop after N total rows
--csv # output as CSV
--tsv # output as TSV
--nl # output as newline-delimited JSON
--json # output as JSON (default)
--table # human-readable table format
```
**Pagination:**
By default, print one page of results. `--all` follows `next_url` to
fetch every page. `--limit N` caps total rows returned.
#### 1.5 `dclient get` -- Fetch a single row by primary key
```
dclient get <table_url_or_alias> <pk_value>
dclient get <table_url_or_alias> <pk1>,<pk2> # compound key
```
Hits `GET {db}/{table}/{pk}.json`. Prints the row as JSON.
---
### Tier 2 -- Write Operations
dclient already supports `insert` and `create`. These additions round out
the full CRUD lifecycle.
#### 2.1 `dclient upsert` -- Insert or update rows
```
dclient upsert <db_url_or_alias> <table> <filepath> [OPTIONS]
```
Same interface as `insert` but hits the `/-/upsert` endpoint. Requires
rows to contain primary key columns.
Shares most code with `insert`; the main difference is the endpoint URL
and the absence of `--replace`/`--ignore` flags.
#### 2.2 `dclient update` -- Update a single row
```
dclient update <table_url_or_alias> <pk_value> <key=value> [<key=value> ...]
dclient update <table_url_or_alias> <pk_value> --input file.json
```
Hits `POST {db}/{table}/{pk}/-/update` with `{"update": {...}}`.
#### 2.3 `dclient delete` -- Delete rows
```
dclient delete <table_url_or_alias> <pk_value>
dclient delete <table_url_or_alias> <pk_value> --yes # skip confirmation
```
Hits `POST {db}/{table}/{pk}/-/delete`. Prompts for confirmation unless
`--yes` is passed.
#### 2.4 `dclient drop` -- Drop a table
```
dclient drop <table_url_or_alias>
dclient drop <table_url_or_alias> --yes # skip confirmation
```
Hits `POST {db}/{table}/-/drop` with `{"confirm": true}`. Shows row count
and asks for confirmation unless `--yes`.
#### 2.5 `dclient create-table` -- Create an empty table with explicit schema
```
dclient create-table <db_url_or_alias> <table_name> \
--column id integer \
--column name text \
--column score float \
--pk id
```
Hits `POST {db}/-/create` with a `columns` array instead of `rows`.
The existing `insert --create` handles creation-with-data; this handles
the schema-only case.
---
### Tier 3 -- Output & UX Improvements
#### 3.1 Multiple output formats for `query` and `rows`
Currently `query` only outputs JSON. Add:
```
--csv # CSV output
--tsv # TSV output
--nl # newline-delimited JSON
--table # human-readable ASCII table (like sqlite-utils)
--yaml # YAML output (optional, low priority)
```
For `--table` output, use a simple column-aligned format or integrate
with `tabulate` / `rich` (consider keeping dependencies minimal).
A lightweight approach: `sqlite-utils` already has table-formatting
utilities that could be reused.
#### 3.2 Pipe-friendly defaults
Detect whether stdout is a TTY:
- **TTY**: default to `--table` (human-readable) output
- **Pipe**: default to JSON (machine-readable) output
This matches the UX convention of tools like `gh` and `jq`.
#### 3.3 `--output` / `-o` flag for writing to a file
```
dclient query ... -o results.csv --csv
dclient rows ... -o dump.json
```
#### 3.4 Streaming output for large result sets
For `--all` pagination mode and `--csv`/`--nl` formats, stream rows as
they arrive rather than buffering everything in memory. Write each page
to stdout immediately.
---
### Tier 4 -- Authentication Improvements
The current auth system works but has friction points. These changes
make it smoother.
#### 4.1 `dclient auth login` -- Browser-based authentication
```
dclient auth login <instance_url_or_alias>
```
Flow:
1. Open the user's browser to `{instance}/-/create-token`
2. Start a temporary local HTTP server to receive the token callback
3. User creates a token in the Datasette UI and pastes it, OR:
4. If the instance supports it, redirect back to the local server with the token
5. Store the token automatically via the existing auth system
For instances that don't support redirect, fall back to:
1. Open the browser to `/-/create-token`
2. Prompt the user to paste the token into the terminal
This is similar to how `gh auth login`, `gcloud auth login`, and
`heroku login` work.
#### 4.2 `dclient auth token` -- Create a token via the API
```
dclient auth token <instance_url_or_alias>
dclient auth token <instance_url_or_alias> --expires-after 3600
dclient auth token <instance_url_or_alias> --read-only
```
Hits `POST /-/create-token`. Requires an existing root token or session.
Prints the new token and optionally stores it.
#### 4.3 `dclient auth status` -- Validate stored credentials
```
dclient auth status <instance_url_or_alias>
```
Hits `/-/actor.json` with the stored token and prints:
- Whether the token is valid
- The actor identity (id, permissions)
- Token expiry if available
This is just a friendlier wrapper around the existing `actor` command,
integrated into the `auth` subgroup.
#### 4.4 Environment variable support
```
export DCLIENT_TOKEN=dstok_xxx
export DCLIENT_URL=https://my-datasette.example.com/mydb
```
Token resolution order (highest to lowest priority):
1. `--token` CLI flag
2. `DCLIENT_TOKEN` environment variable
3. Stored token in `auth.json` (existing behavior)
The `DCLIENT_URL` variable provides a default instance so users don't
have to type it every time:
```
export DCLIENT_URL=https://my-datasette.example.com/mydb
dclient tables # uses DCLIENT_URL
dclient query 'select 1' # uses DCLIENT_URL
dclient rows mytable --limit 10 # uses DCLIENT_URL
```
#### 4.5 Keyring integration (optional, lower priority)
Instead of storing tokens in plaintext `auth.json`, optionally use the
system keyring via the `keyring` Python package. This keeps tokens
encrypted at rest on macOS (Keychain), Windows (Credential Vault), and
Linux (Secret Service / KWallet).
Make this opt-in: `dclient auth add --keyring <url>`. Fall back to the
current file-based storage if `keyring` is not installed.
---
### Tier 5 -- Discoverability & Convenience
#### 5.1 `dclient info` -- Instance overview
```
dclient info <instance_url_or_alias>
```
Hits `/-/versions.json`, `/-/plugins.json`, `/-/databases.json` in
parallel. Prints a summary:
```
Datasette 1.0.1
Python 3.12.1
SQLite 3.45.0
Databases: 3 (fixtures, content, ephemeral)
Plugins: 12 installed
```
#### 5.2 URL inference and shortcuts
Allow shorthand for common patterns:
```
# These should all work equivalently:
dclient rows https://latest.datasette.io/fixtures/facetable
dclient rows latest fixtures/facetable # if 'latest' is an alias
dclient rows latest facetable # if alias points to a database URL
```
The key insight: if an alias points to `https://x.com/dbname`, then
`dclient rows <alias> <table>` should construct
`https://x.com/dbname/table.json`.
Currently the alias is just a URL prefix that gets used verbatim. Making
the alias system slightly smarter about joining paths would reduce typing
significantly.
#### 5.3 `dclient config` -- Manage configuration
```
dclient config show # print config directory and contents
dclient config path # print config directory path
dclient config edit # open config in $EDITOR
```
Useful for debugging when tokens or aliases aren't working as expected.
#### 5.4 Shell completions
Use Click's built-in shell completion support to generate completions
for bash, zsh, and fish:
```
dclient --install-completion bash
dclient --install-completion zsh
dclient --install-completion fish
```
Click 8.x supports this natively via `shell_complete`. This could also
dynamically complete alias names, database names, and table names by
querying the stored aliases.
---
### Tier 6 -- Advanced Features (lower priority)
#### 6.1 `dclient export` -- Bulk export table data
```
dclient export <table_url> -o data.csv --csv
dclient export <table_url> -o data.json --json
dclient export <table_url> -o data.db # export to local SQLite
```
This is essentially `dclient rows --all` but optimized for bulk export:
- Streams using `?_stream=1` (CSV format) when available
- For JSON, auto-paginates using `_next` cursor
- For SQLite output, pipes into `sqlite-utils insert`
#### 6.2 Parallel insert for large files
When inserting very large files, optionally send batches in parallel:
```
dclient insert <url> <table> huge.csv --parallel 4
```
Uses a thread pool to send N batches concurrently. Would require
switching from synchronous `httpx` to `httpx.AsyncClient` or using
`concurrent.futures.ThreadPoolExecutor`.
#### 6.3 `dclient diff` -- Compare local and remote data
```
dclient diff local.db remote_alias/tablename
```
Compare a local SQLite table with a remote Datasette table. Show added,
removed, and modified rows. Useful for sync workflows.
---
## Implementation Notes
### Shared infrastructure to build first
Several of the proposed commands share common patterns. Before
implementing individual commands, refactor these into shared utilities:
1. **URL resolution helper**: Centralize the alias-lookup + URL-construction
logic that is currently duplicated in every command. A single function
like `resolve_url(url_or_alias, suffix=None)` that handles aliases,
adds `.json` where appropriate, and joins path segments.
2. **Authenticated HTTP client helper**: A function or context manager
that creates an `httpx.Client` with the right `Authorization` header
and timeout, resolved from `--token` / env var / auth.json. This
eliminates the repeated token-resolution boilerplate in every command.
3. **Output formatter**: A shared output module that handles `--json`,
`--csv`, `--tsv`, `--nl`, `--table` flags consistently across all
commands that return data.
4. **Pagination helper**: A generator that follows `next_url` links to
yield all pages of results, used by `rows --all` and `export`.
### Dependency considerations
The current dependencies are minimal: `click`, `httpx`, `sqlite-utils`.
The proposed changes mostly stay within these. Potential additions:
- `tabulate` or `rich` for `--table` output (or implement a minimal
version to avoid new deps)
- `keyring` for optional encrypted token storage (optional dependency)
### Backward compatibility
All proposed changes are additive (new commands and new flags). No
existing command signatures or behaviors change. The only subtle change
would be if TTY-detection changes the default output format of `query`,
which should be gated behind a major version bump or opt-in flag.
### Testing strategy
Each new command should follow the existing test patterns:
- Unit tests with `pytest-httpx` for HTTP mocking
- Integration tests against a real in-memory Datasette instance
(as done in `test_insert.py`)
- `cogapp` for keeping `--help` output in docs in sync
---
## Priority Summary
| Priority | Feature | Rationale |
|----------|---------|-----------|
| **P0** | `databases`, `tables`, `schema` | Basic exploration; most-requested gap |
| **P0** | `rows` with filtering/sorting/pagination | Replaces need for raw SQL for common tasks |
| **P0** | Environment variable support (`DCLIENT_TOKEN`, `DCLIENT_URL`) | Near-zero effort, big UX win |
| **P1** | `get`, `update`, `delete`, `drop` | Completes CRUD lifecycle |
| **P1** | `upsert` | Natural complement to existing `insert` |
| **P1** | Multiple output formats (CSV, table, NL) | Makes `query`/`rows` output usable in pipelines |
| **P1** | `auth login` (browser-based) | Biggest auth UX improvement |
| **P2** | `create-table` (schema only) | Useful but `insert --create` covers most cases |
| **P2** | TTY-aware default output | Polish |
| **P2** | `auth status`, `auth token` | Convenience wrappers |
| **P2** | `info` | Nice discoverability |
| **P2** | URL inference / smarter aliases | Reduces typing |
| **P3** | `export` (bulk) | Covered by `rows --all` for most cases |
| **P3** | Shell completions | Polish |
| **P3** | Keyring integration | Security hardening |
| **P3** | Parallel insert | Performance optimization |
| **P3** | `diff` | Advanced workflow |

View file

@ -11,10 +11,15 @@ Much of the functionality requires Datasette 1.0a2 or higher.
## Things you can do with dclient
- Run SQL queries against Datasette and returning the results as JSON
- **Explore** databases, tables, schemas, and rows without writing SQL
- **Query** with SQL and get results as JSON, CSV, TSV, or formatted tables
- **Insert** data from CSV, TSV, JSON, or newline-delimited JSON files
- **Upsert** data (insert or update based on primary key)
- **Update** individual rows by primary key
- **Delete** rows or **drop** entire tables
- **Create tables** with explicit schemas
- Run queries against authenticated Datasette instances
- Create aliases and store authentication tokens for convenient access to Datasette
- Insert data into Datasette using the [insert API](https://docs.datasette.io/en/latest/json_api.html#the-json-write-api) (Datasette 1.0 alpha or higher)
- Create aliases and store authentication tokens for convenient access
## Installation
@ -26,36 +31,67 @@ If you want to install it in the same virtual environment as Datasette (to use i
```bash
datasette install dclient
```
## Running a query
## Quick start
```bash
# Explore a Datasette instance
dclient databases https://latest.datasette.io
dclient tables https://latest.datasette.io/fixtures
dclient schema https://latest.datasette.io/fixtures
# Browse rows with filtering and sorting
dclient rows https://latest.datasette.io/fixtures/facetable --table
dclient rows https://latest.datasette.io/fixtures/facetable -w state=CA --sort city
# Fetch a single row by primary key
dclient get https://latest.datasette.io/fixtures/facetable 1
# Run a SQL query
dclient query https://latest.datasette.io/fixtures "select * from facetable limit 1"
```
To shorten that, create an alias:
For write operations (requires authentication):
```bash
# Create a table
dclient create-table https://example.com/db mytable \
--column id integer --column name text --pk id
# Insert data from a CSV file
dclient insert https://example.com/db mytable data.csv --create
# Update a row
dclient update https://example.com/db/mytable 42 name=Alice age=30
# Upsert from a JSON file
dclient upsert https://example.com/db mytable data.json --json
# Delete a row
dclient delete https://example.com/db/mytable 42 --yes
# Drop a table
dclient drop https://example.com/db/mytable --yes
```
To shorten URLs, create an alias:
```bash
dclient alias add fixtures https://latest.datasette.io/fixtures
```
Then run it like this instead:
```bash
dclient query fixtures "select * from facetable limit 1"
```
## Documentation
Visit **[dclient.datasette.io](https://dclient.datasette.io)** for full documentation on using this tool.
## Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
To contribute to this tool, first checkout the code. Then install dependencies and run tests using [uv](https://docs.astral.sh/uv/):
```bash
cd dclient
python -m venv venv
source venv/bin/activate
uv run pytest
```
Now install the dependencies and test dependencies:
To build the package:
```bash
pip install -e '.[test]'
```
To run the tests:
```bash
pytest
uv build
```

View file

@ -1,5 +1,7 @@
import click
import csv
import httpx
import io
import json
import pathlib
from sqlite_utils.utils import rows_from_file, Format, TypeTracker, progressbar
@ -14,10 +16,234 @@ def get_config_dir():
return pathlib.Path(click.get_app_dir("io.datasette.dclient"))
# --- Shared infrastructure ---
def _load_aliases(aliases_file):
if aliases_file.exists():
aliases = json.loads(aliases_file.read_text())
else:
aliases = {}
return aliases
def _load_auths(auth_file):
if auth_file.exists():
auths = json.loads(auth_file.read_text())
else:
auths = {}
return auths
def _resolve_url(url_or_alias):
"""Resolve an alias to a URL, or return the input unchanged."""
aliases_file = get_config_dir() / "aliases.json"
aliases = _load_aliases(aliases_file)
if url_or_alias in aliases:
return aliases[url_or_alias]
return url_or_alias
def _resolve_token(url, token=None):
"""Resolve a token from CLI arg, env var, or auth.json."""
if token:
return token
env_token = click.get_current_context().obj.get("env_token") if click.get_current_context().obj else None
if env_token:
return env_token
return token_for_url(url, _load_auths(get_config_dir() / "auth.json"))
def _make_headers(token):
"""Build request headers with optional auth."""
headers = {}
if token:
headers["Authorization"] = "Bearer {}".format(token)
return headers
def _api_get(url, token=None, params=None, verbose=False):
"""Make an authenticated GET request and return parsed JSON."""
headers = _make_headers(token)
if verbose:
full_url = url
if params:
full_url += "?" + urllib.parse.urlencode(params, doseq=True)
click.echo(full_url, err=True)
response = httpx.get(url, params=params, headers=headers, timeout=40.0, follow_redirects=True)
if response.status_code != 200:
_raise_api_error(response)
try:
data = response.json()
except json.JSONDecodeError:
raise click.ClickException("Response was not valid JSON")
if isinstance(data, dict) and data.get("ok") is False:
_raise_from_data(data)
return data
def _api_post(url, data, token=None, verbose=False):
"""Make an authenticated POST request and return parsed JSON."""
headers = _make_headers(token)
headers["Content-Type"] = "application/json"
if verbose:
click.echo("POST {}".format(url), err=True)
click.echo(textwrap.indent(json.dumps(data, indent=2), " "), err=True)
response = httpx.post(url, headers=headers, json=data, timeout=40.0)
if verbose:
click.echo(str(response), err=True)
if str(response.status_code)[0] != "2":
if "json" in response.headers.get("content-type", ""):
resp_data = response.json()
if "errors" in resp_data:
raise click.ClickException("\n".join(resp_data["errors"]))
_raise_from_data(resp_data)
response.raise_for_status()
response_data = response.json()
if verbose:
click.echo(textwrap.indent(json.dumps(response_data, indent=2), " "), err=True)
return response_data
def _raise_api_error(response):
"""Raise a ClickException from a non-200 response."""
try:
data = response.json()
except json.JSONDecodeError:
raise click.ClickException(
"{} status code. Response was not valid JSON".format(response.status_code)
)
bits = []
if data.get("title"):
bits.append(data["title"])
if data.get("error"):
bits.append(data["error"])
raise click.ClickException(
"{} status code. {}".format(response.status_code, ": ".join(bits))
)
def _raise_from_data(data):
"""Raise a ClickException from a JSON error response."""
bits = []
if data.get("title"):
bits.append(data["title"])
if data.get("error"):
bits.append(data["error"])
if data.get("errors"):
bits.extend(data["errors"])
if not bits:
bits = [json.dumps(data)]
raise click.ClickException(": ".join(bits))
def _output_rows(rows, output_format, nl=False, cols=None):
"""Output rows in the specified format."""
if output_format == "json":
click.echo(json.dumps(rows, indent=2))
elif output_format == "nl":
for row in rows:
click.echo(json.dumps(row))
elif output_format == "csv":
_output_csv(rows, cols)
elif output_format == "tsv":
_output_csv(rows, cols, delimiter="\t")
elif output_format == "table":
_output_table(rows, cols)
else:
click.echo(json.dumps(rows, indent=2))
def _output_csv(rows, cols=None, delimiter=","):
"""Write rows as CSV/TSV to stdout."""
if not rows:
return
if cols is None:
cols = list(rows[0].keys())
writer = csv.DictWriter(sys.stdout, fieldnames=cols, delimiter=delimiter, extrasaction="ignore")
writer.writeheader()
writer.writerows(rows)
def _output_table(rows, cols=None):
"""Write rows as a simple aligned text table."""
if not rows:
return
if cols is None:
cols = list(rows[0].keys())
# Calculate column widths
widths = {col: len(col) for col in cols}
str_rows = []
for row in rows:
str_row = {}
for col in cols:
val = str(row.get(col, "")) if row.get(col) is not None else ""
str_row[col] = val
widths[col] = max(widths[col], len(val))
str_rows.append(str_row)
# Header
header = " ".join(col.ljust(widths[col]) for col in cols)
click.echo(header)
click.echo(" ".join("-" * widths[col] for col in cols))
for str_row in str_rows:
click.echo(" ".join(str_row[col].ljust(widths[col]) for col in cols))
def _determine_format(fmt_csv, fmt_tsv, fmt_json, fmt_nl, fmt_table, default="json"):
"""Determine output format from flags."""
if fmt_csv:
return "csv"
if fmt_tsv:
return "tsv"
if fmt_json:
return "json"
if fmt_nl:
return "nl"
if fmt_table:
return "table"
return default
# --- CLI ---
@click.group()
@click.version_option()
def cli():
@click.pass_context
def cli(ctx):
"A client CLI utility for Datasette instances"
ctx.ensure_object(dict)
import os
ctx.obj["env_token"] = os.environ.get("DCLIENT_TOKEN")
ctx.obj["env_url"] = os.environ.get("DCLIENT_URL")
def _resolve_url_with_env(url_or_alias):
"""Resolve URL, falling back to DCLIENT_URL env var."""
if url_or_alias:
return _resolve_url(url_or_alias)
ctx = click.get_current_context()
env_url = ctx.obj.get("env_url") if ctx.obj else None
if env_url:
return env_url
raise click.ClickException("No URL provided and DCLIENT_URL not set")
# --- Output format options (shared decorator) ---
def output_format_options(f):
"""Add --csv, --tsv, --json, --nl, --table output options."""
f = click.option("fmt_table", "--table", is_flag=True, help="Output as text table")(f)
f = click.option("fmt_nl", "--nl", is_flag=True, help="Output as newline-delimited JSON")(f)
f = click.option("fmt_json", "--json", is_flag=True, help="Output as JSON (default)")(f)
f = click.option("fmt_tsv", "--tsv", is_flag=True, help="Output as TSV")(f)
f = click.option("fmt_csv", "--csv", is_flag=True, help="Output as CSV")(f)
return f
# --- query command ---
@cli.command()
@ -25,7 +251,9 @@ def cli():
@click.argument("sql")
@click.option("--token", help="API token")
@click.option("-v", "--verbose", is_flag=True, help="Verbose output: show HTTP request")
def query(url_or_alias, sql, token, verbose):
@output_format_options
@click.pass_context
def query(ctx, url_or_alias, sql, token, verbose, fmt_csv, fmt_tsv, fmt_json, fmt_nl, fmt_table):
"""
Run a SQL query against a Datasette database URL
@ -38,62 +266,277 @@ def query(url_or_alias, sql, token, verbose):
https://datasette.io/content \\
'select * from news limit 10'
"""
aliases_file = get_config_dir() / "aliases.json"
aliases = _load_aliases(aliases_file)
if url_or_alias in aliases:
url = aliases[url_or_alias]
else:
url = url_or_alias
if not url_or_alias.endswith(".json"):
url = _resolve_url_with_env(url_or_alias)
if not url.endswith(".json"):
url += ".json"
if token is None:
# Maybe there's a token in auth.json?
token = token_for_url(url, _load_auths(get_config_dir() / "auth.json"))
headers = {}
if token:
headers["Authorization"] = f"Bearer {token}"
params = {"sql": sql, "_shape": "objects"}
if verbose:
click.echo(url + "?" + urllib.parse.urlencode(params), err=True)
response = httpx.get(url, params=params, headers=headers)
token = _resolve_token(url, token)
data = _api_get(url, token=token, params={"sql": sql, "_shape": "objects"}, verbose=verbose)
output_format = _determine_format(fmt_csv, fmt_tsv, fmt_json, fmt_nl, fmt_table)
_output_rows(data["rows"], output_format)
if response.status_code != 200:
# Is it valid JSON?
try:
data = response.json()
except json.JSONDecodeError:
raise click.ClickException(
"{} status code. Response was not valid JSON".format(
response.status_code
)
)
bits = []
if data.get("title"):
bits.append(data["title"])
if data.get("error"):
bits.append(data["error"])
raise click.ClickException(
"{} status code. {}".format(response.status_code, ": ".join(bits))
)
# We should have JSON now
try:
data = response.json()
except json.JSONDecodeError:
raise click.ClickException("Response was not valid JSON")
# ... but it may have a {"ok": false} error
if not data.get("ok"):
bits = []
if data.get("title"):
bits.append(data["title"])
if data.get("error"):
bits.append(data["error"])
if not bits:
bits = [json.dumps(data)]
raise click.ClickException(": ".join(bits))
# --- databases command ---
# Output results
click.echo(json.dumps(response.json()["rows"], indent=2))
@cli.command()
@click.argument("url_or_alias", required=False)
@click.option("--token", help="API token")
@click.option("-v", "--verbose", is_flag=True, help="Verbose output")
@output_format_options
@click.pass_context
def databases(ctx, url_or_alias, token, verbose, fmt_csv, fmt_tsv, fmt_json, fmt_nl, fmt_table):
"""
List databases on a Datasette instance
Example usage:
\b
dclient databases https://latest.datasette.io
"""
url = _resolve_url_with_env(url_or_alias)
# Ensure we hit the instance root
api_url = url.rstrip("/") + "/-/databases.json"
token = _resolve_token(url, token)
data = _api_get(api_url, token=token, verbose=verbose)
output_format = _determine_format(fmt_csv, fmt_tsv, fmt_json, fmt_nl, fmt_table)
if output_format == "table":
rows = []
for db in data:
rows.append({
"name": db["name"],
"path": db.get("path", ""),
"is_mutable": db.get("is_mutable", ""),
})
_output_rows(rows, "table")
elif output_format == "json":
click.echo(json.dumps(data, indent=2))
else:
_output_rows(data, output_format)
# --- tables command ---
@cli.command()
@click.argument("url_or_alias", required=False)
@click.option("--token", help="API token")
@click.option("--views", is_flag=True, help="Include views")
@click.option("--schema", is_flag=True, help="Show CREATE TABLE SQL")
@click.option("-v", "--verbose", is_flag=True, help="Verbose output")
@output_format_options
@click.pass_context
def tables(ctx, url_or_alias, token, views, schema, verbose, fmt_csv, fmt_tsv, fmt_json, fmt_nl, fmt_table):
"""
List tables in a Datasette database
Example usage:
\b
dclient tables https://latest.datasette.io/fixtures
"""
url = _resolve_url_with_env(url_or_alias)
api_url = url.rstrip("/") + ".json"
token = _resolve_token(url, token)
data = _api_get(api_url, token=token, verbose=verbose)
output_format = _determine_format(fmt_csv, fmt_tsv, fmt_json, fmt_nl, fmt_table)
result_tables = data.get("tables", []) if isinstance(data, dict) else data
if not views:
result_tables = [t for t in result_tables if not t.get("hidden")]
if schema:
# Fetch schema for the database
schema_url = url.rstrip("/") + "/-/schema.json"
schema_data = _api_get(schema_url, token=token, verbose=verbose)
schema_map = {}
if isinstance(schema_data, dict):
for item in schema_data.get("tables", []):
schema_map[item["name"]] = item.get("schema", "")
for t in result_tables:
t["schema"] = schema_map.get(t["name"], "")
if output_format == "json":
click.echo(json.dumps(result_tables, indent=2))
elif output_format == "table":
rows = []
for t in result_tables:
row = {
"name": t["name"],
"rows": str(t.get("count", t.get("rows_count", ""))),
"columns": ", ".join(t.get("columns", [])),
}
if schema:
row["schema"] = t.get("schema", "")
rows.append(row)
_output_rows(rows, "table")
else:
_output_rows(result_tables, output_format)
# --- schema command ---
@cli.command()
@click.argument("url_or_alias", required=False)
@click.option("--token", help="API token")
@click.option("-v", "--verbose", is_flag=True, help="Verbose output")
@click.pass_context
def schema(ctx, url_or_alias, token, verbose):
"""
Show the SQL schema for a database or table
Example usage:
\b
dclient schema https://latest.datasette.io/fixtures
dclient schema https://latest.datasette.io/fixtures/facetable
"""
url = _resolve_url_with_env(url_or_alias)
token = _resolve_token(url, token)
schema_url = url.rstrip("/") + "/-/schema.json"
data = _api_get(schema_url, token=token, verbose=verbose)
if isinstance(data, dict) and "tables" in data:
for t in data["tables"]:
click.echo(t.get("schema", "") + ";")
click.echo()
elif isinstance(data, dict) and "schema" in data:
click.echo(data["schema"] + ";")
else:
click.echo(json.dumps(data, indent=2))
# --- rows command ---
@cli.command()
@click.argument("url_or_alias", required=False)
@click.option("--token", help="API token")
@click.option("-w", "--where", "where_clauses", multiple=True, help="Filter: column__op=value (e.g. age__gt=30)")
@click.option("--where-sql", "where_sql", multiple=True, help="Raw SQL WHERE clause")
@click.option("--search", help="Full-text search query")
@click.option("--sort", help="Sort by column (ascending)")
@click.option("--sort-desc", help="Sort by column (descending)")
@click.option("--col", "columns", multiple=True, help="Include only these columns")
@click.option("--nocol", "nocolumns", multiple=True, help="Exclude these columns")
@click.option("--facet", "facets", multiple=True, help="Facet by column")
@click.option("--size", type=int, help="Number of rows per page")
@click.option("--all", "fetch_all", is_flag=True, help="Fetch all pages")
@click.option("--limit", type=int, help="Maximum total rows to return")
@click.option("-v", "--verbose", is_flag=True, help="Verbose output")
@output_format_options
@click.pass_context
def rows(ctx, url_or_alias, token, where_clauses, where_sql, search, sort,
sort_desc, columns, nocolumns, facets, size, fetch_all, limit, verbose,
fmt_csv, fmt_tsv, fmt_json, fmt_nl, fmt_table):
"""
Browse rows in a Datasette table with filtering and sorting
Example usage:
\b
dclient rows https://latest.datasette.io/fixtures/facetable
dclient rows https://latest.datasette.io/fixtures/facetable -w state=CA
dclient rows https://latest.datasette.io/fixtures/facetable --sort city
dclient rows https://latest.datasette.io/fixtures/facetable --search text
"""
url = _resolve_url_with_env(url_or_alias)
token = _resolve_token(url, token)
output_format = _determine_format(fmt_csv, fmt_tsv, fmt_json, fmt_nl, fmt_table)
# Build query params
params = [("_shape", "objects"), ("_extra", "next_url")]
for w in where_clauses:
if "=" in w:
key, _, value = w.partition("=")
params.append((key, value))
else:
params.append(("_where", w))
for w in where_sql:
params.append(("_where", w))
if search:
params.append(("_search", search))
if sort:
params.append(("_sort", sort))
if sort_desc:
params.append(("_sort_desc", sort_desc))
for col in columns:
params.append(("_col", col))
for col in nocolumns:
params.append(("_nocol", col))
for facet in facets:
params.append(("_facet", facet))
if size:
params.append(("_size", str(size)))
api_url = url.rstrip("/") + ".json"
all_rows = []
total = 0
page_url = api_url
while True:
data = _api_get(page_url, token=token, params=params if page_url == api_url else None, verbose=verbose)
page_rows = data.get("rows", [])
if limit:
remaining = limit - total
page_rows = page_rows[:remaining]
all_rows.extend(page_rows)
total += len(page_rows)
if limit and total >= limit:
break
next_url = data.get("next_url")
if not fetch_all or not next_url:
break
page_url = next_url
params = None # next_url includes params
# Show facets if requested
facet_results = data.get("facet_results") if facets else None
if facet_results:
click.echo("Facets:", err=True)
for facet_name, facet_info in facet_results.items():
click.echo(" {}:".format(facet_name), err=True)
results_list = facet_info.get("results", facet_info) if isinstance(facet_info, dict) else facet_info
if isinstance(results_list, list):
for item in results_list:
click.echo(" {} ({})".format(item.get("value", ""), item.get("count", "")), err=True)
click.echo(err=True)
_output_rows(all_rows, output_format)
# --- get command ---
@cli.command()
@click.argument("url_or_alias", required=False)
@click.argument("pk_values")
@click.option("--token", help="API token")
@click.option("-v", "--verbose", is_flag=True, help="Verbose output")
@click.pass_context
def get(ctx, url_or_alias, pk_values, token, verbose):
"""
Fetch a single row by primary key
Example usage:
\b
dclient get https://latest.datasette.io/fixtures/facetable 1
dclient get https://latest.datasette.io/fixtures/compound_pk a,b
"""
url = _resolve_url_with_env(url_or_alias)
token = _resolve_token(url, token)
api_url = url.rstrip("/") + "/" + pk_values + ".json"
data = _api_get(api_url, token=token, params={"_shape": "objects"}, verbose=verbose)
rows = data.get("rows", [data] if isinstance(data, dict) else data)
if rows:
click.echo(json.dumps(rows[0], indent=2))
else:
raise click.ClickException("Row not found")
# --- insert command ---
@cli.command()
@ -167,15 +610,8 @@ def insert(
https://private.datasette.cloud/data \\
mytable data.csv --pk id --create
"""
aliases_file = get_config_dir() / "aliases.json"
aliases = _load_aliases(aliases_file)
if url_or_alias in aliases:
url = aliases[url_or_alias]
else:
url = url_or_alias
if token is None:
token = token_for_url(url, _load_auths(get_config_dir() / "auth.json"))
url = _resolve_url(url_or_alias)
token = _resolve_token(url, token)
format = None
if format_csv:
@ -205,7 +641,6 @@ def insert(
raise click.ClickException(str(ex))
if format in (Format.JSON, Format.NL):
# Disable progress bar - it can't handle these formats
file_size = None
no_detect_types = True
@ -226,15 +661,12 @@ def insert(
bar.update(new_bytes)
bytes_so_far += new_bytes
except ValueError:
# File has likely been closed, so fp.tell() fails
pass
types = None
if first and not no_detect_types:
# Detect types on first batch
tracker = TypeTracker()
list(tracker.wrap(batch))
types = tracker.types
# Convert types
for row in batch:
for key, value in row.items():
if value is None:
@ -264,10 +696,311 @@ def insert(
)
# --- upsert command ---
@cli.command()
@click.argument("url_or_alias")
@click.argument("table")
@click.argument(
"filepath", type=click.Path("rb", readable=True, allow_dash=True, dir_okay=False)
)
@click.option("format_csv", "--csv", is_flag=True, help="Input is CSV")
@click.option("format_tsv", "--tsv", is_flag=True, help="Input is TSV")
@click.option("format_json", "--json", is_flag=True, help="Input is JSON")
@click.option("format_nl", "--nl", is_flag=True, help="Input is newline-delimited JSON")
@click.option("--encoding", help="Character encoding for CSV/TSV")
@click.option(
"--no-detect-types", is_flag=True, help="Don't detect column types for CSV/TSV"
)
@click.option("--alter", is_flag=True, help="Alter table to add any missing columns")
@click.option(
"--batch-size", type=int, default=100, help="Send rows in batches of this size"
)
@click.option(
"--interval", type=float, default=10, help="Send batch at least every X seconds"
)
@click.option("--token", "-t", help="API token")
@click.option("--silent", is_flag=True, help="Don't output progress")
@click.option(
"-v",
"--verbose",
is_flag=True,
help="Verbose output: show HTTP request and response",
)
def upsert(
url_or_alias,
table,
filepath,
format_csv,
format_tsv,
format_json,
format_nl,
encoding,
no_detect_types,
alter,
batch_size,
interval,
token,
silent,
verbose,
):
"""
Upsert data into a remote Datasette table
Rows with matching primary keys will be updated; others will be inserted.
Each row must include the primary key column(s).
Example usage:
\b
dclient upsert \\
https://private.datasette.cloud/data \\
mytable data.csv
"""
url = _resolve_url(url_or_alias)
token = _resolve_token(url, token)
format = None
if format_csv:
format = Format.CSV
elif format_tsv:
format = Format.TSV
elif format_json:
format = Format.JSON
elif format_nl:
format = Format.NL
if format is None and filepath == "-":
raise click.ClickException(
"An explicit format is required - e.g. --csv "
"- when reading from standard input"
)
if filepath != "-":
file_size = pathlib.Path(filepath).stat().st_size
fp = open(filepath, "rb")
else:
fp = sys.stdin.buffer
file_size = None
try:
rows, format = rows_from_file(fp, format=format, encoding=encoding)
except Exception as ex:
raise click.ClickException(str(ex))
if format in (Format.JSON, Format.NL):
file_size = None
no_detect_types = True
first = True
with progressbar(
length=file_size,
label="Upserting rows",
silent=silent or (file_size is None),
show_percent=True,
) as bar:
bytes_so_far = 0
for batch in _batches(rows, batch_size, interval=interval):
if file_size is not None:
try:
bytes_consumed_so_far = fp.tell()
new_bytes = bytes_consumed_so_far - bytes_so_far
bar.update(new_bytes)
bytes_so_far += new_bytes
except ValueError:
pass
if first and not no_detect_types:
tracker = TypeTracker()
list(tracker.wrap(batch))
types = tracker.types
for row in batch:
for key, value in row.items():
if value is None:
continue
if types[key] == "integer":
if not value:
row[key] = None
else:
row[key] = int(value)
elif types[key] == "float":
if not value:
row[key] = None
else:
row[key] = float(value)
first = False
data = {"rows": batch}
if alter:
data["alter"] = True
upsert_url = "{}/{}/-/upsert".format(url, table)
_api_post(upsert_url, data, token=token, verbose=verbose)
# --- update command ---
@cli.command()
@click.argument("url_or_alias")
@click.argument("pk_values")
@click.argument("updates", nargs=-1)
@click.option("--token", help="API token")
@click.option("--alter", is_flag=True, help="Alter table to add any missing columns")
@click.option("-v", "--verbose", is_flag=True, help="Verbose output")
def update(url_or_alias, pk_values, updates, token, alter, verbose):
"""
Update a row by primary key
Pass key=value pairs to set column values.
Example usage:
\b
dclient update https://example.com/db/table 42 name=Alice age=30
"""
url = _resolve_url(url_or_alias)
token = _resolve_token(url, token)
if not updates:
raise click.ClickException("Provide at least one key=value pair")
update_dict = {}
for pair in updates:
if "=" not in pair:
raise click.ClickException("Invalid key=value pair: {}".format(pair))
key, _, value = pair.partition("=")
# Try to parse as JSON for numbers, booleans, null
try:
value = json.loads(value)
except (json.JSONDecodeError, ValueError):
pass
update_dict[key] = value
data = {"update": update_dict}
if alter:
data["alter"] = True
api_url = "{}/{}/-/update".format(url.rstrip("/"), pk_values)
result = _api_post(api_url, data, token=token, verbose=verbose)
click.echo(json.dumps(result, indent=2))
# --- delete command ---
@cli.command()
@click.argument("url_or_alias")
@click.argument("pk_values")
@click.option("--token", help="API token")
@click.option("--yes", is_flag=True, help="Skip confirmation")
@click.option("-v", "--verbose", is_flag=True, help="Verbose output")
def delete(url_or_alias, pk_values, token, yes, verbose):
"""
Delete a row by primary key
Example usage:
\b
dclient delete https://example.com/db/table 42
dclient delete https://example.com/db/table 42 --yes
"""
url = _resolve_url(url_or_alias)
token = _resolve_token(url, token)
if not yes:
click.confirm("Delete row {}?".format(pk_values), abort=True)
api_url = "{}/{}/-/delete".format(url.rstrip("/"), pk_values)
result = _api_post(api_url, {}, token=token, verbose=verbose)
click.echo(json.dumps(result, indent=2))
# --- drop command ---
@cli.command()
@click.argument("url_or_alias")
@click.option("--token", help="API token")
def actor(url_or_alias, token):
@click.option("--yes", is_flag=True, help="Skip confirmation")
@click.option("-v", "--verbose", is_flag=True, help="Verbose output")
def drop(url_or_alias, token, yes, verbose):
"""
Drop a table
Example usage:
\b
dclient drop https://example.com/db/table
dclient drop https://example.com/db/table --yes
"""
url = _resolve_url(url_or_alias)
token = _resolve_token(url, token)
api_url = url.rstrip("/") + "/-/drop"
if not yes:
# First request to get row count
info = _api_post(api_url, {}, token=token, verbose=verbose)
row_count = info.get("row_count", "unknown")
table_name = info.get("table", url.rstrip("/").split("/")[-1])
click.confirm("Drop table {} ({} rows)?".format(table_name, row_count), abort=True)
result = _api_post(api_url, {"confirm": True}, token=token, verbose=verbose)
click.echo(json.dumps(result, indent=2))
# --- create-table command ---
@cli.command(name="create-table")
@click.argument("url_or_alias")
@click.argument("table_name")
@click.option(
"--column", "column_defs", multiple=True, nargs=2,
help="Column definition: name type (e.g. --column id integer --column name text)"
)
@click.option(
"pks", "--pk", multiple=True,
help="Column(s) to use as primary key"
)
@click.option("--token", help="API token")
@click.option("-v", "--verbose", is_flag=True, help="Verbose output")
def create_table(url_or_alias, table_name, column_defs, pks, token, verbose):
"""
Create a new empty table with explicit schema
Example usage:
\b
dclient create-table https://example.com/db mytable \\
--column id integer --column name text --pk id
"""
url = _resolve_url(url_or_alias)
token = _resolve_token(url, token)
if not column_defs:
raise click.ClickException("Provide at least one --column definition")
columns = [{"name": name, "type": typ} for name, typ in column_defs]
data = {"table": table_name, "columns": columns}
if pks:
if len(pks) == 1:
data["pk"] = pks[0]
else:
data["pks"] = list(pks)
api_url = url.rstrip("/") + "/-/create"
result = _api_post(api_url, data, token=token, verbose=verbose)
click.echo(json.dumps(result, indent=2))
# --- actor command ---
@cli.command()
@click.argument("url_or_alias", required=False)
@click.option("--token", help="API token")
@click.pass_context
def actor(ctx, url_or_alias, token):
"""
Show the actor represented by an API token
@ -276,18 +1009,12 @@ def actor(url_or_alias, token):
\b
dclient actor https://latest.datasette.io/fixtures
"""
aliases_file = get_config_dir() / "aliases.json"
aliases = _load_aliases(aliases_file)
if url_or_alias in aliases:
url = aliases[url_or_alias]
else:
url = url_or_alias
url = _resolve_url_with_env(url_or_alias)
if not (url.startswith("http://") or url.startswith("https://")):
raise click.ClickException("Invalid URL: " + url)
if token is None:
token = token_for_url(url, _load_auths(get_config_dir() / "auth.json"))
token = _resolve_token(url, token)
url_bits = url.split("/")
url_bits[-1] = "-/actor.json"
@ -299,6 +1026,9 @@ def actor(url_or_alias, token):
click.echo(json.dumps(response.json(), indent=4))
# --- alias commands ---
@cli.group()
def alias():
"Manage aliases for different instances"
@ -369,6 +1099,9 @@ def alias_remove(name):
raise click.ClickException("No such alias")
# --- auth commands ---
@cli.group()
def auth():
"Manage authentication for different instances"
@ -441,20 +1174,7 @@ def auth_remove(alias_or_url):
raise click.ClickException("No such URL or alias")
def _load_aliases(aliases_file):
if aliases_file.exists():
aliases = json.loads(aliases_file.read_text())
else:
aliases = {}
return aliases
def _load_auths(auth_file):
if auth_file.exists():
auths = json.loads(auth_file.read_text())
else:
auths = {}
return auths
# --- Internal helpers ---
def _batches(iterable, size, interval=None):
@ -521,7 +1241,6 @@ def _insert_batch(
if verbose:
click.echo(str(response), err=True)
if str(response.status_code)[0] != "2":
# Is there an error we can show?
if "/json" in response.headers["content-type"]:
data = response.json()
if "errors" in data:

322
demos/exploring-writing.md Normal file
View file

@ -0,0 +1,322 @@
# Exploring and Writing Data with dclient
*2026-02-12T01:33:47Z*
This demo walks through dclient's data exploration and write commands against a live Datasette instance running on localhost with a mutable in-memory database called `demo`.
## Creating a table
`create-table` creates an empty table with an explicit schema. We define columns and a primary key up front.
```bash
uv run dclient create-table http://127.0.0.1:8042/demo dogs --column id integer --column name text --column age integer --column breed text --pk id --token dstok_eyJhIjoicm9vdCIsInQiOjE3NzA4NTk5ODJ9.1KDQjswLBzm5-R-MLmopOIHRZls
```
```output
{
"ok": true,
"database": "demo",
"table": "dogs",
"table_url": "http://127.0.0.1:8042/demo/dogs",
"table_api_url": "http://127.0.0.1:8042/demo/dogs.json",
"schema": "CREATE TABLE [dogs] (\n [id] INTEGER PRIMARY KEY,\n [name] TEXT,\n [age] INTEGER,\n [breed] TEXT\n)"
}
```
## Inserting rows from CSV
`insert` reads CSV (or TSV, JSON, newline-delimited JSON) from a file or stdin. Here we pipe a CSV file in.
```bash
cat /tmp/dogs.csv | uv run dclient insert http://127.0.0.1:8042/demo dogs - --csv --token dstok_eyJhIjoicm9vdCIsInQiOjE3NzA4NTk5ODJ9.1KDQjswLBzm5-R-MLmopOIHRZls
```
```output
```
## Listing databases
`databases` lists every database on the instance. The `--table` flag formats the output as a text table.
```bash
uv run dclient databases http://127.0.0.1:8042 --table --token dstok_eyJhIjoicm9vdCIsInQiOjE3NzA4NTk5ODJ9.1KDQjswLBzm5-R-MLmopOIHRZls
```
```output
name path is_mutable
------- ---- ----------
_memory False
demo True
```
## Listing tables
`tables` shows all tables in a database with row counts and column names.
```bash
uv run dclient tables http://127.0.0.1:8042/demo --table --token dstok_eyJhIjoicm9vdCIsInQiOjE3NzA4NTk5ODJ9.1KDQjswLBzm5-R-MLmopOIHRZls
```
```output
name rows columns
---- ---- --------------------
dogs 5 id, name, age, breed
```
## Viewing the schema
`schema` prints the CREATE TABLE SQL for every table in the database.
```bash
uv run dclient schema http://127.0.0.1:8042/demo --token dstok_eyJhIjoicm9vdCIsInQiOjE3NzA4NTk5ODJ9.1KDQjswLBzm5-R-MLmopOIHRZls
```
```output
CREATE TABLE [dogs] (
[id] INTEGER PRIMARY KEY,
[name] TEXT,
[age] INTEGER,
[breed] TEXT
);
```
## Browsing rows
`rows` lets you explore table data without writing SQL. The default output is JSON; `--table` renders a text table.
```bash
uv run dclient rows http://127.0.0.1:8042/demo/dogs --table --token dstok_eyJhIjoicm9vdCIsInQiOjE3NzA4NTk5ODJ9.1KDQjswLBzm5-R-MLmopOIHRZls
```
```output
id name age breed
-- -------- --- ----------------
1 Cleo 5 Golden Retriever
2 Pancakes 3 Poodle
3 Fido 7 Labrador
4 Muffin 2 Corgi
5 Rex 4 German Shepherd
```
### Filtering
Use `-w` to filter using Datasette's column operators (`__gt`, `__contains`, `__exact`, etc.).
```bash
uv run dclient rows http://127.0.0.1:8042/demo/dogs --table -w age__gt=4 --token dstok_eyJhIjoicm9vdCIsInQiOjE3NzA4NTk5ODJ9.1KDQjswLBzm5-R-MLmopOIHRZls
```
```output
id name age breed
-- ---- --- ----------------
1 Cleo 5 Golden Retriever
3 Fido 7 Labrador
```
### Sorting
`--sort` and `--sort-desc` control row order.
```bash
uv run dclient rows http://127.0.0.1:8042/demo/dogs --table --sort-desc age --token dstok_eyJhIjoicm9vdCIsInQiOjE3NzA4NTk5ODJ9.1KDQjswLBzm5-R-MLmopOIHRZls
```
```output
id name age breed
-- -------- --- ----------------
3 Fido 7 Labrador
1 Cleo 5 Golden Retriever
5 Rex 4 German Shepherd
2 Pancakes 3 Poodle
4 Muffin 2 Corgi
```
### Selecting columns
`--col` picks specific columns. Here we also output as CSV, which is handy for piping into other tools.
```bash
uv run dclient rows http://127.0.0.1:8042/demo/dogs --csv --col name --col breed --token dstok_eyJhIjoicm9vdCIsInQiOjE3NzA4NTk5ODJ9.1KDQjswLBzm5-R-MLmopOIHRZls
```
```output
id,name,breed
1,Cleo,Golden Retriever
2,Pancakes,Poodle
3,Fido,Labrador
4,Muffin,Corgi
5,Rex,German Shepherd
```
### Limiting results
`--limit` caps the total number of rows returned.
```bash
uv run dclient rows http://127.0.0.1:8042/demo/dogs --table --limit 2 --sort age --token dstok_eyJhIjoicm9vdCIsInQiOjE3NzA4NTk5ODJ9.1KDQjswLBzm5-R-MLmopOIHRZls
```
```output
id name age breed
-- -------- --- ------
4 Muffin 2 Corgi
2 Pancakes 3 Poodle
```
## Fetching a single row
`get` retrieves one row by its primary key.
```bash
uv run dclient get http://127.0.0.1:8042/demo/dogs 3 --token dstok_eyJhIjoicm9vdCIsInQiOjE3NzA4NTk5ODJ9.1KDQjswLBzm5-R-MLmopOIHRZls
```
```output
{
"id": 3,
"name": "Fido",
"age": 7,
"breed": "Labrador"
}
```
## Running SQL queries
`query` executes arbitrary SQL and returns results. Here we use `--table` for a readable summary.
```bash
uv run dclient query http://127.0.0.1:8042/demo 'select breed, count(*) as count from dogs group by breed order by count desc' --table --token dstok_eyJhIjoicm9vdCIsInQiOjE3NzA4NTk5ODJ9.1KDQjswLBzm5-R-MLmopOIHRZls
```
```output
breed count
---------------- -----
Poodle 1
Labrador 1
Golden Retriever 1
German Shepherd 1
Corgi 1
```
The same query as newline-delimited JSON, useful for streaming into `jq` or other line-oriented tools.
```bash
uv run dclient query http://127.0.0.1:8042/demo 'select name, age from dogs order by age desc limit 3' --nl --token dstok_eyJhIjoicm9vdCIsInQiOjE3NzA4NTk5ODJ9.1KDQjswLBzm5-R-MLmopOIHRZls
```
```output
{"name": "Fido", "age": 7}
{"name": "Cleo", "age": 5}
{"name": "Rex", "age": 4}
```
## Updating a row
`update` modifies a single row by primary key. Pass column=value pairs as arguments. Numeric values are auto-detected.
```bash
uv run dclient update http://127.0.0.1:8042/demo/dogs 3 name=Buddy age=8 --token dstok_eyJhIjoicm9vdCIsInQiOjE3NzA4NTk5ODJ9.1KDQjswLBzm5-R-MLmopOIHRZls
```
```output
{
"ok": true
}
```
Verify the update:
```bash
uv run dclient get http://127.0.0.1:8042/demo/dogs 3 --token dstok_eyJhIjoicm9vdCIsInQiOjE3NzA4NTk5ODJ9.1KDQjswLBzm5-R-MLmopOIHRZls
```
```output
{
"id": 3,
"name": "Buddy",
"age": 8,
"breed": "Labrador"
}
```
## Upserting rows
`upsert` inserts new rows and updates existing ones that match by primary key. Here we update Cleo's age to 6 and add a new dog Luna, all in one call.
```bash
cat /tmp/upsert_dogs.json | uv run dclient upsert http://127.0.0.1:8042/demo dogs - --json --token dstok_eyJhIjoicm9vdCIsInQiOjE3NzA4NTk5ODJ9.1KDQjswLBzm5-R-MLmopOIHRZls
```
```output
```
The table now shows Cleo at age 6 and the new row for Luna:
```bash
uv run dclient rows http://127.0.0.1:8042/demo/dogs --table --sort id --token dstok_eyJhIjoicm9vdCIsInQiOjE3NzA4NTk5ODJ9.1KDQjswLBzm5-R-MLmopOIHRZls
```
```output
id name age breed
-- -------- --- ----------------
1 Cleo 6 Golden Retriever
2 Pancakes 3 Poodle
3 Buddy 8 Labrador
4 Muffin 2 Corgi
5 Rex 4 German Shepherd
6 Luna 1 Husky
```
## Deleting a row
`delete` removes a row by primary key. Use `--yes` to skip the confirmation prompt.
```bash
uv run dclient delete http://127.0.0.1:8042/demo/dogs 4 --yes --token dstok_eyJhIjoicm9vdCIsInQiOjE3NzA4NTk5ODJ9.1KDQjswLBzm5-R-MLmopOIHRZls
```
```output
{
"ok": true
}
```
Muffin (id=4) is gone:
```bash
uv run dclient rows http://127.0.0.1:8042/demo/dogs --table --sort id --token dstok_eyJhIjoicm9vdCIsInQiOjE3NzA4NTk5ODJ9.1KDQjswLBzm5-R-MLmopOIHRZls
```
```output
id name age breed
-- -------- --- ----------------
1 Cleo 6 Golden Retriever
2 Pancakes 3 Poodle
3 Buddy 8 Labrador
5 Rex 4 German Shepherd
6 Luna 1 Husky
```
## Dropping a table
`drop` removes an entire table. Without `--yes` it shows the row count and asks for confirmation.
```bash
uv run dclient drop http://127.0.0.1:8042/demo/dogs --yes --token dstok_eyJhIjoicm9vdCIsInQiOjE3NzA4NTk5ODJ9.1KDQjswLBzm5-R-MLmopOIHRZls
```
```output
{
"ok": true
}
```
The table is gone:
```bash
uv run dclient tables http://127.0.0.1:8042/demo --json --token dstok_eyJhIjoicm9vdCIsInQiOjE3NzA4NTk5ODJ9.1KDQjswLBzm5-R-MLmopOIHRZls
```
```output
[]
```

View file

@ -161,7 +161,7 @@ cog.out(
)
]]] -->
```
Usage: dclient actor [OPTIONS] URL_OR_ALIAS
Usage: dclient actor [OPTIONS] [URL_OR_ALIAS]
Show the actor represented by an API token

347
docs/exploring.md Normal file
View file

@ -0,0 +1,347 @@
# Exploring data
dclient provides several commands for exploring Datasette instances without writing SQL.
## Listing databases
Use `dclient databases` to list all databases on a Datasette instance:
```bash
dclient databases https://latest.datasette.io
```
Use `--table` for a formatted text table:
```bash
dclient databases https://latest.datasette.io --table
```
Output:
```
name path is_mutable
-------- -------- ----------
fixtures fixtures False
```
## Listing tables
Use `dclient tables` to list tables in a specific database:
```bash
dclient tables https://latest.datasette.io/fixtures
```
Use `--table` for formatted output:
```bash
dclient tables https://latest.datasette.io/fixtures --table
```
Use `--schema` to include the CREATE TABLE SQL for each table:
```bash
dclient tables https://latest.datasette.io/fixtures --schema
```
Use `--views` to include views in the output (hidden by default).
## Viewing schemas
Use `dclient schema` to show the SQL schema for a database or an individual table:
```bash
# Schema for all tables in a database
dclient schema https://latest.datasette.io/fixtures
# Schema for a specific table
dclient schema https://latest.datasette.io/fixtures/facetable
```
## Browsing rows
Use `dclient rows` to browse table data with filtering, sorting, and pagination — no SQL required:
```bash
dclient rows https://latest.datasette.io/fixtures/facetable --table
```
### Filtering
Use `-w` / `--where` to filter rows. Supports [Datasette filter operators](https://docs.datasette.io/en/stable/json_api.html#table-arguments) like `__gt`, `__contains`, `__exact`, etc.:
```bash
# Exact match
dclient rows https://example.com/db/dogs -w breed=Poodle
# Greater than
dclient rows https://example.com/db/dogs -w age__gt=4
# Multiple filters (AND)
dclient rows https://example.com/db/dogs -w age__gte=3 -w breed=Labrador
```
For raw SQL WHERE clauses, use `--where-sql`:
```bash
dclient rows https://example.com/db/dogs --where-sql "age > 3 and breed != 'Poodle'"
```
### Sorting
```bash
# Sort ascending
dclient rows https://example.com/db/dogs --sort name
# Sort descending
dclient rows https://example.com/db/dogs --sort-desc age
```
### Selecting columns
```bash
# Include only specific columns
dclient rows https://example.com/db/dogs --col name --col breed
# Exclude specific columns
dclient rows https://example.com/db/dogs --nocol id
```
### Full-text search
```bash
dclient rows https://example.com/db/articles --search "python tutorial"
```
### Faceting
```bash
dclient rows https://example.com/db/dogs --facet breed
```
Facet results are printed to stderr, with the rows on stdout.
### Pagination
By default, only the first page of results is returned. Use `--all` to fetch every page:
```bash
dclient rows https://example.com/db/dogs --all
```
Control page size with `--size` and limit total rows with `--limit`:
```bash
# 10 rows per page, max 50 rows total
dclient rows https://example.com/db/dogs --size 10 --limit 50
```
## Fetching a single row
Use `dclient get` to fetch a single row by its primary key:
```bash
dclient get https://latest.datasette.io/fixtures/facetable 1
```
Output:
```json
{
"pk": 1,
"created": "2019-01-14 08:00:00",
"planet_int": 1,
"on_earth": 1,
"state": "CA",
...
}
```
For compound primary keys, separate values with a comma:
```bash
dclient get https://example.com/db/compound_pk a,b
```
## Output formats
The `databases`, `tables`, `rows`, and `query` commands all support multiple output formats:
- `--json` — JSON (default)
- `--csv` — CSV
- `--tsv` — TSV
- `--nl` — Newline-delimited JSON
- `--table` — Formatted text table
```bash
dclient rows https://example.com/db/dogs --csv > dogs.csv
dclient rows https://example.com/db/dogs --table
```
## dclient databases --help
<!-- [[[cog
import cog
from dclient import cli
from click.testing import CliRunner
runner = CliRunner()
result = runner.invoke(cli.cli, ["databases", "--help"])
help = result.output.replace("Usage: cli", "Usage: dclient")
cog.out(
"```\n{}\n```".format(help)
)
]]] -->
```
Usage: dclient databases [OPTIONS] [URL_OR_ALIAS]
List databases on a Datasette instance
Example usage:
dclient databases https://latest.datasette.io
Options:
--token TEXT API token
-v, --verbose Verbose output
--csv Output as CSV
--tsv Output as TSV
--json Output as JSON (default)
--nl Output as newline-delimited JSON
--table Output as text table
--help Show this message and exit.
```
<!-- [[[end]]] -->
## dclient tables --help
<!-- [[[cog
import cog
from dclient import cli
from click.testing import CliRunner
runner = CliRunner()
result = runner.invoke(cli.cli, ["tables", "--help"])
help = result.output.replace("Usage: cli", "Usage: dclient")
cog.out(
"```\n{}\n```".format(help)
)
]]] -->
```
Usage: dclient tables [OPTIONS] [URL_OR_ALIAS]
List tables in a Datasette database
Example usage:
dclient tables https://latest.datasette.io/fixtures
Options:
--token TEXT API token
--views Include views
--schema Show CREATE TABLE SQL
-v, --verbose Verbose output
--csv Output as CSV
--tsv Output as TSV
--json Output as JSON (default)
--nl Output as newline-delimited JSON
--table Output as text table
--help Show this message and exit.
```
<!-- [[[end]]] -->
## dclient schema --help
<!-- [[[cog
import cog
from dclient import cli
from click.testing import CliRunner
runner = CliRunner()
result = runner.invoke(cli.cli, ["schema", "--help"])
help = result.output.replace("Usage: cli", "Usage: dclient")
cog.out(
"```\n{}\n```".format(help)
)
]]] -->
```
Usage: dclient schema [OPTIONS] [URL_OR_ALIAS]
Show the SQL schema for a database or table
Example usage:
dclient schema https://latest.datasette.io/fixtures
dclient schema https://latest.datasette.io/fixtures/facetable
Options:
--token TEXT API token
-v, --verbose Verbose output
--help Show this message and exit.
```
<!-- [[[end]]] -->
## dclient rows --help
<!-- [[[cog
import cog
from dclient import cli
from click.testing import CliRunner
runner = CliRunner()
result = runner.invoke(cli.cli, ["rows", "--help"])
help = result.output.replace("Usage: cli", "Usage: dclient")
cog.out(
"```\n{}\n```".format(help)
)
]]] -->
```
Usage: dclient rows [OPTIONS] [URL_OR_ALIAS]
Browse rows in a Datasette table with filtering and sorting
Example usage:
dclient rows https://latest.datasette.io/fixtures/facetable
dclient rows https://latest.datasette.io/fixtures/facetable -w state=CA
dclient rows https://latest.datasette.io/fixtures/facetable --sort city
dclient rows https://latest.datasette.io/fixtures/facetable --search text
Options:
--token TEXT API token
-w, --where TEXT Filter: column__op=value (e.g. age__gt=30)
--where-sql TEXT Raw SQL WHERE clause
--search TEXT Full-text search query
--sort TEXT Sort by column (ascending)
--sort-desc TEXT Sort by column (descending)
--col TEXT Include only these columns
--nocol TEXT Exclude these columns
--facet TEXT Facet by column
--size INTEGER Number of rows per page
--all Fetch all pages
--limit INTEGER Maximum total rows to return
-v, --verbose Verbose output
--csv Output as CSV
--tsv Output as TSV
--json Output as JSON (default)
--nl Output as newline-delimited JSON
--table Output as text table
--help Show this message and exit.
```
<!-- [[[end]]] -->
## dclient get --help
<!-- [[[cog
import cog
from dclient import cli
from click.testing import CliRunner
runner = CliRunner()
result = runner.invoke(cli.cli, ["get", "--help"])
help = result.output.replace("Usage: cli", "Usage: dclient")
cog.out(
"```\n{}\n```".format(help)
)
]]] -->
```
Usage: dclient get [OPTIONS] [URL_OR_ALIAS] PK_VALUES
Fetch a single row by primary key
Example usage:
dclient get https://latest.datasette.io/fixtures/facetable 1
dclient get https://latest.datasette.io/fixtures/compound_pk a,b
Options:
--token TEXT API token
-v, --verbose Verbose output
--help Show this message and exit.
```
<!-- [[[end]]] -->

View file

@ -9,7 +9,7 @@ A client CLI utility for [Datasette](https://datasette.io/) instances
## Installation
Install `dclient` using `pip` (or [pipx](https://pipxproject.github.io/pipx/):
Install `dclient` using `pip` (or [pipx](https://pipxproject.github.io/pipx/)):
```bash
pip install dclient
@ -21,16 +21,11 @@ If you also have Datasette installed in the same environment it will register it
```bash
datasette install dclient
```
This means you can run any of these commands using `datasette dt` instead, like this:
This means you can run any of these commands using `datasette dc` instead, like this:
```bash
datasette dc --help
datasette dc query https://latest.datasette.io/fixtures "select * from facetable limit 1"
```
You can install it into Datasette this way using:
```bash
datasette install dclient
```
## Contents
@ -38,8 +33,10 @@ datasette install dclient
---
maxdepth: 3
---
exploring
queries
inserting
writing
aliases
authentication
inserting
```

View file

@ -4,6 +4,8 @@ The `dclient insert` command can be used to insert data from a local file direct
First you'll need to {ref}`authenticate <authentication>` with the instance.
See also {doc}`writing` for related commands: `upsert`, `update`, `delete`, `drop`, and `create-table`.
To insert data from a `data.csv` file into a table called `my_table`, creating that table if it does not exist:
```bash
@ -138,3 +140,7 @@ Options:
```
<!-- [[[end]]] -->
:::{note}
To update existing rows rather than inserting new ones, see {doc}`writing` for the `upsert` and `update` commands.
:::

View file

@ -24,6 +24,34 @@ Output:
]
```
## Output formats
Query results default to JSON. Use these flags for other formats:
- `--csv` — CSV
- `--tsv` — TSV
- `--nl` — Newline-delimited JSON (one JSON object per line)
- `--table` — Formatted text table
```bash
dclient query https://latest.datasette.io/fixtures \
"select state, count(*) as n from facetable group by state" --table
```
Output:
```
state n
----- --
CA 10
MI 4
MO 1
```
Export as CSV:
```bash
dclient query https://latest.datasette.io/fixtures \
"select * from facetable limit 5" --csv > results.csv
```
## dclient query --help
<!-- [[[cog
import cog
@ -52,6 +80,11 @@ Usage: dclient query [OPTIONS] URL_OR_ALIAS SQL
Options:
--token TEXT API token
-v, --verbose Verbose output: show HTTP request
--csv Output as CSV
--tsv Output as TSV
--json Output as JSON (default)
--nl Output as newline-delimited JSON
--table Output as text table
--help Show this message and exit.
```

265
docs/writing.md Normal file
View file

@ -0,0 +1,265 @@
# Writing data
In addition to `insert` (documented in {doc}`inserting`), dclient provides several commands for modifying data in a Datasette instance via the [Write API](https://docs.datasette.io/en/latest/json_api.html#the-json-write-api).
All write commands require authentication. See {doc}`authentication` for how to set up tokens.
## Creating tables
Use `dclient create-table` to create a new empty table with an explicit schema:
```bash
dclient create-table https://example.com/db mytable \
--column id integer \
--column name text \
--column age integer \
--pk id
```
Output:
```json
{
"ok": true,
"database": "db",
"table": "mytable",
"schema": "CREATE TABLE [mytable] (\n [id] INTEGER PRIMARY KEY,\n [name] TEXT,\n [age] INTEGER\n)"
}
```
Use `--pk` one or more times to set compound primary keys:
```bash
dclient create-table https://example.com/db events \
--column date text --column venue text --column title text \
--pk date --pk venue
```
## Upserting data
Use `dclient upsert` to insert rows or update existing ones based on primary key. This works like `insert` but uses the [upsert API endpoint](https://docs.datasette.io/en/latest/json_api.html#upserting-rows):
```bash
dclient upsert https://example.com/db mytable data.csv
```
The same file format options as `insert` are supported: `--csv`, `--tsv`, `--json`, `--nl`, `--encoding`, `--no-detect-types`.
Upsert from standard input:
```bash
echo '[{"id": 1, "name": "Updated"}]' | \
dclient upsert https://example.com/db mytable - --json
```
Use `--alter` to add any missing columns automatically.
## Updating a single row
Use `dclient update` to update specific columns of a row by its primary key:
```bash
dclient update https://example.com/db/mytable 42 name=Alice age=30
```
Values are automatically parsed — numbers become integers/floats, `true`/`false` become booleans, `null` becomes null. Everything else is treated as a string.
Use `--alter` to allow adding new columns that don't exist yet.
## Deleting a row
Use `dclient delete` to delete a single row by primary key:
```bash
dclient delete https://example.com/db/mytable 42
```
You'll be prompted for confirmation. Use `--yes` to skip the prompt:
```bash
dclient delete https://example.com/db/mytable 42 --yes
```
## Dropping a table
Use `dclient drop` to drop an entire table:
```bash
dclient drop https://example.com/db/mytable
```
Without `--yes`, dclient will first query the table and show you the row count before asking for confirmation:
```
Drop table mytable (150 rows)? [y/N]:
```
Use `--yes` to skip confirmation:
```bash
dclient drop https://example.com/db/mytable --yes
```
## dclient create-table --help
<!-- [[[cog
import cog
from dclient import cli
from click.testing import CliRunner
runner = CliRunner()
result = runner.invoke(cli.cli, ["create-table", "--help"])
help = result.output.replace("Usage: cli", "Usage: dclient")
cog.out(
"```\n{}\n```".format(help)
)
]]] -->
```
Usage: dclient create-table [OPTIONS] URL_OR_ALIAS TABLE_NAME
Create a new empty table with explicit schema
Example usage:
dclient create-table https://example.com/db mytable \
--column id integer --column name text --pk id
Options:
--column TEXT... Column definition: name type (e.g. --column id integer
--column name text)
--pk TEXT Column(s) to use as primary key
--token TEXT API token
-v, --verbose Verbose output
--help Show this message and exit.
```
<!-- [[[end]]] -->
## dclient upsert --help
<!-- [[[cog
import cog
from dclient import cli
from click.testing import CliRunner
runner = CliRunner()
result = runner.invoke(cli.cli, ["upsert", "--help"])
help = result.output.replace("Usage: cli", "Usage: dclient")
cog.out(
"```\n{}\n```".format(help)
)
]]] -->
```
Usage: dclient upsert [OPTIONS] URL_OR_ALIAS TABLE FILEPATH
Upsert data into a remote Datasette table
Rows with matching primary keys will be updated; others will be inserted. Each
row must include the primary key column(s).
Example usage:
dclient upsert \
https://private.datasette.cloud/data \
mytable data.csv
Options:
--csv Input is CSV
--tsv Input is TSV
--json Input is JSON
--nl Input is newline-delimited JSON
--encoding TEXT Character encoding for CSV/TSV
--no-detect-types Don't detect column types for CSV/TSV
--alter Alter table to add any missing columns
--batch-size INTEGER Send rows in batches of this size
--interval FLOAT Send batch at least every X seconds
-t, --token TEXT API token
--silent Don't output progress
-v, --verbose Verbose output: show HTTP request and response
--help Show this message and exit.
```
<!-- [[[end]]] -->
## dclient update --help
<!-- [[[cog
import cog
from dclient import cli
from click.testing import CliRunner
runner = CliRunner()
result = runner.invoke(cli.cli, ["update", "--help"])
help = result.output.replace("Usage: cli", "Usage: dclient")
cog.out(
"```\n{}\n```".format(help)
)
]]] -->
```
Usage: dclient update [OPTIONS] URL_OR_ALIAS PK_VALUES [UPDATES]...
Update a row by primary key
Pass key=value pairs to set column values.
Example usage:
dclient update https://example.com/db/table 42 name=Alice age=30
Options:
--token TEXT API token
--alter Alter table to add any missing columns
-v, --verbose Verbose output
--help Show this message and exit.
```
<!-- [[[end]]] -->
## dclient delete --help
<!-- [[[cog
import cog
from dclient import cli
from click.testing import CliRunner
runner = CliRunner()
result = runner.invoke(cli.cli, ["delete", "--help"])
help = result.output.replace("Usage: cli", "Usage: dclient")
cog.out(
"```\n{}\n```".format(help)
)
]]] -->
```
Usage: dclient delete [OPTIONS] URL_OR_ALIAS PK_VALUES
Delete a row by primary key
Example usage:
dclient delete https://example.com/db/table 42
dclient delete https://example.com/db/table 42 --yes
Options:
--token TEXT API token
--yes Skip confirmation
-v, --verbose Verbose output
--help Show this message and exit.
```
<!-- [[[end]]] -->
## dclient drop --help
<!-- [[[cog
import cog
from dclient import cli
from click.testing import CliRunner
runner = CliRunner()
result = runner.invoke(cli.cli, ["drop", "--help"])
help = result.output.replace("Usage: cli", "Usage: dclient")
cog.out(
"```\n{}\n```".format(help)
)
]]] -->
```
Usage: dclient drop [OPTIONS] URL_OR_ALIAS
Drop a table
Example usage:
dclient drop https://example.com/db/table
dclient drop https://example.com/db/table --yes
Options:
--token TEXT API token
--yes Skip confirmation
-v, --verbose Verbose output
--help Show this message and exit.
```
<!-- [[[end]]] -->

42
pyproject.toml Normal file
View file

@ -0,0 +1,42 @@
[project]
name = "dclient"
version = "0.4"
description = "A client CLI utility for Datasette instances"
readme = "README.md"
requires-python = ">=3.8"
license = "Apache-2.0"
authors = [{name = "Simon Willison"}]
dependencies = [
"click",
"httpx",
"sqlite-utils",
]
[project.urls]
Homepage = "https://github.com/simonw/dclient"
Issues = "https://github.com/simonw/dclient/issues"
CI = "https://github.com/simonw/dclient/actions"
Changelog = "https://github.com/simonw/dclient/releases"
[project.scripts]
dclient = "dclient.cli:cli"
[project.entry-points.datasette]
client = "dclient.plugin"
[dependency-groups]
dev = [
"pytest",
"pytest-asyncio",
"pytest-httpx",
"cogapp",
"pytest-mock",
"datasette>=1.0a2",
]
[build-system]
requires = ["uv_build>=0.9.18,<0.10.0"]
build-backend = "uv_build"
[tool.uv.build-backend]
module-root = ""

View file

@ -1,46 +0,0 @@
from setuptools import setup
import os
VERSION = "0.4"
def get_long_description():
with open(
os.path.join(os.path.dirname(os.path.abspath(__file__)), "README.md"),
encoding="utf8",
) as fp:
return fp.read()
setup(
name="dclient",
description="A client CLI utility for Datasette instances",
long_description=get_long_description(),
long_description_content_type="text/markdown",
author="Simon Willison",
url="https://github.com/simonw/dclient",
project_urls={
"Issues": "https://github.com/simonw/dclient/issues",
"CI": "https://github.com/simonw/dclient/actions",
"Changelog": "https://github.com/simonw/dclient/releases",
},
license="Apache License, Version 2.0",
version=VERSION,
packages=["dclient"],
entry_points={
"datasette": ["client = dclient.plugin"],
"console_scripts": ["dclient = dclient.cli:cli"],
},
install_requires=["click", "httpx", "sqlite-utils"],
extras_require={
"test": [
"pytest",
"pytest-asyncio",
"pytest-httpx",
"cogapp",
"pytest-mock",
"datasette>=1.0a2",
]
},
python_requires=">=3.8",
)

View file

@ -120,7 +120,7 @@ def make_format_test(content, arg):
input_data=SIMPLE_CSV,
cmd_args=[],
table_exists=False,
expected_output="Inserting rows\nError: Table not found: table1\n",
expected_output="Inserting rows\nError: Table not found\n",
should_error=True,
expected_table_json=None,
),
@ -296,7 +296,7 @@ def test_insert_against_datasette(
return loop.run_until_complete(run())
httpx_mock.add_callback(custom_response)
httpx_mock.add_callback(custom_response, is_optional=should_error)
path = pathlib.Path(tmpdir) / "data.txt"
if isinstance(input_data, str):

506
tests/test_new_commands.py Normal file
View file

@ -0,0 +1,506 @@
"""Tests for new T1 and T2 commands: databases, tables, schema, rows, get,
upsert, update, delete, drop, create-table."""
import asyncio
import json
import pathlib
import pytest
from click.testing import CliRunner
from datasette.app import Datasette
from dclient.cli import cli
@pytest.fixture
def ds():
"""Create a Datasette instance with a memory database and sample data."""
datasette = Datasette(
config={
"permissions": {
"create-table": {"id": "*"},
"insert-row": {"id": "*"},
"update-row": {"id": "*"},
"delete-row": {"id": "*"},
"drop-table": {"id": "*"},
"alter-table": {"id": "*"},
}
}
)
db = datasette.add_memory_database("data")
loop = asyncio.get_event_loop()
async def setup():
# Drop all tables first
for table in await db.table_names():
await db.execute_write("drop table [{}]".format(table))
await db.execute_write(
"create table dogs (id integer primary key, name text, age integer)"
)
await db.execute_write(
"insert into dogs (id, name, age) values (1, 'Cleo', 5), (2, 'Pancakes', 3), (3, 'Fido', 7)"
)
loop.run_until_complete(setup())
return datasette, db, loop
@pytest.fixture
def token(ds):
datasette, _, _ = ds
return datasette.create_token("actor")
def _mock_datasette(httpx_mock, ds):
"""Set up httpx_mock to route requests through the Datasette instance."""
datasette, _, loop = ds
def custom_response(request):
async def run():
method = request.method
url_path = request.url.raw_path.decode()
if method == "GET":
response = await datasette.client.get(
url_path, headers=request.headers
)
else:
content = request.read()
try:
body = json.loads(content) if content else {}
except json.JSONDecodeError:
body = {}
response = await datasette.client.request(
method, url_path, json=body, headers=request.headers
)
import httpx as httpx_lib
return httpx_lib.Response(
status_code=response.status_code,
headers=response.headers,
content=response.content,
)
return loop.run_until_complete(run())
httpx_mock.add_callback(custom_response, is_optional=True)
# --- databases command ---
def test_databases(httpx_mock, ds, token):
_mock_datasette(httpx_mock, ds)
runner = CliRunner()
result = runner.invoke(cli, ["databases", "http://localhost/", "--token", token])
assert result.exit_code == 0
data = json.loads(result.output)
names = [d["name"] for d in data]
assert "data" in names
def test_databases_table_format(httpx_mock, ds, token):
_mock_datasette(httpx_mock, ds)
runner = CliRunner()
result = runner.invoke(
cli, ["databases", "http://localhost/", "--token", token, "--table"]
)
assert result.exit_code == 0
assert "name" in result.output
assert "data" in result.output
# --- tables command ---
def test_tables(httpx_mock, ds, token):
_mock_datasette(httpx_mock, ds)
runner = CliRunner()
result = runner.invoke(cli, ["tables", "http://localhost/data", "--token", token])
assert result.exit_code == 0
data = json.loads(result.output)
table_names = [t["name"] for t in data]
assert "dogs" in table_names
def test_tables_table_format(httpx_mock, ds, token):
_mock_datasette(httpx_mock, ds)
runner = CliRunner()
result = runner.invoke(
cli, ["tables", "http://localhost/data", "--token", token, "--table"]
)
assert result.exit_code == 0
assert "dogs" in result.output
# --- schema command ---
def test_schema(httpx_mock, ds, token):
_mock_datasette(httpx_mock, ds)
runner = CliRunner()
result = runner.invoke(cli, ["schema", "http://localhost/data", "--token", token])
assert result.exit_code == 0
assert "CREATE TABLE" in result.output
assert "dogs" in result.output
# --- rows command ---
def test_rows_basic(httpx_mock, ds, token):
_mock_datasette(httpx_mock, ds)
runner = CliRunner()
result = runner.invoke(
cli, ["rows", "http://localhost/data/dogs", "--token", token]
)
assert result.exit_code == 0
data = json.loads(result.output)
assert len(data) == 3
names = [r["name"] for r in data]
assert "Cleo" in names
def test_rows_with_filter(httpx_mock, ds, token):
_mock_datasette(httpx_mock, ds)
runner = CliRunner()
result = runner.invoke(
cli,
["rows", "http://localhost/data/dogs", "--token", token, "-w", "age__gt=4"],
)
assert result.exit_code == 0
data = json.loads(result.output)
for row in data:
assert row["age"] > 4
def test_rows_with_sort(httpx_mock, ds, token):
_mock_datasette(httpx_mock, ds)
runner = CliRunner()
result = runner.invoke(
cli,
["rows", "http://localhost/data/dogs", "--token", token, "--sort", "name"],
)
assert result.exit_code == 0
data = json.loads(result.output)
names = [r["name"] for r in data]
assert names == sorted(names)
def test_rows_with_sort_desc(httpx_mock, ds, token):
_mock_datasette(httpx_mock, ds)
runner = CliRunner()
result = runner.invoke(
cli,
[
"rows",
"http://localhost/data/dogs",
"--token",
token,
"--sort-desc",
"age",
],
)
assert result.exit_code == 0
data = json.loads(result.output)
ages = [r["age"] for r in data]
assert ages == sorted(ages, reverse=True)
def test_rows_csv(httpx_mock, ds, token):
_mock_datasette(httpx_mock, ds)
runner = CliRunner()
result = runner.invoke(
cli, ["rows", "http://localhost/data/dogs", "--token", token, "--csv"]
)
assert result.exit_code == 0
assert "id,name,age" in result.output
assert "Cleo" in result.output
def test_rows_nl(httpx_mock, ds, token):
_mock_datasette(httpx_mock, ds)
runner = CliRunner()
result = runner.invoke(
cli, ["rows", "http://localhost/data/dogs", "--token", token, "--nl"]
)
assert result.exit_code == 0
lines = [l for l in result.output.strip().split("\n") if l]
for line in lines:
parsed = json.loads(line)
assert "name" in parsed
def test_rows_table_format(httpx_mock, ds, token):
_mock_datasette(httpx_mock, ds)
runner = CliRunner()
result = runner.invoke(
cli, ["rows", "http://localhost/data/dogs", "--token", token, "--table"]
)
assert result.exit_code == 0
assert "id" in result.output
assert "name" in result.output
assert "Cleo" in result.output
# Should have header separator
assert "---" in result.output
def test_rows_with_limit(httpx_mock, ds, token):
_mock_datasette(httpx_mock, ds)
runner = CliRunner()
result = runner.invoke(
cli,
["rows", "http://localhost/data/dogs", "--token", token, "--limit", "2"],
)
assert result.exit_code == 0
data = json.loads(result.output)
assert len(data) == 2
def test_rows_with_columns(httpx_mock, ds, token):
_mock_datasette(httpx_mock, ds)
runner = CliRunner()
result = runner.invoke(
cli,
["rows", "http://localhost/data/dogs", "--token", token, "--col", "name"],
)
assert result.exit_code == 0
data = json.loads(result.output)
for row in data:
assert "name" in row
# --- get command ---
def test_get(httpx_mock, ds, token):
_mock_datasette(httpx_mock, ds)
runner = CliRunner()
result = runner.invoke(
cli, ["get", "http://localhost/data/dogs", "1", "--token", token]
)
assert result.exit_code == 0
data = json.loads(result.output)
assert data["name"] == "Cleo"
assert data["id"] == 1
# --- upsert command ---
def test_upsert(httpx_mock, ds, token, tmpdir):
_mock_datasette(httpx_mock, ds)
# Upsert: update Cleo's age and add a new dog
data = json.dumps([
{"id": 1, "name": "Cleo", "age": 6},
{"id": 4, "name": "Muffin", "age": 2},
])
path = pathlib.Path(tmpdir) / "upsert.json"
path.write_text(data)
runner = CliRunner()
result = runner.invoke(
cli,
[
"upsert",
"http://localhost/data",
"dogs",
str(path),
"--json",
"--token",
token,
],
)
assert result.exit_code == 0, result.output
# Verify
datasette, _, loop = ds
async def check():
r = await datasette.client.get("/data/dogs.json?_shape=array")
return r.json()
rows = loop.run_until_complete(check())
names = {r["name"]: r["age"] for r in rows}
assert names["Cleo"] == 6 # updated
assert names["Muffin"] == 2 # inserted
# --- update command ---
def test_update(httpx_mock, ds, token):
_mock_datasette(httpx_mock, ds)
runner = CliRunner()
result = runner.invoke(
cli,
[
"update",
"http://localhost/data/dogs",
"1",
"name=Rex",
"age=10",
"--token",
token,
],
)
assert result.exit_code == 0, result.output
data = json.loads(result.output)
assert data.get("ok") is True
# Verify the update
datasette, _, loop = ds
async def check():
r = await datasette.client.get("/data/dogs/1.json?_shape=objects")
return r.json()
row_data = loop.run_until_complete(check())
row = row_data["rows"][0]
assert row["name"] == "Rex"
assert row["age"] == 10
# --- delete command ---
def test_delete(httpx_mock, ds, token):
_mock_datasette(httpx_mock, ds)
runner = CliRunner()
result = runner.invoke(
cli,
[
"delete",
"http://localhost/data/dogs",
"3",
"--yes",
"--token",
token,
],
)
assert result.exit_code == 0, result.output
data = json.loads(result.output)
assert data.get("ok") is True
# Verify deletion
datasette, _, loop = ds
async def check():
r = await datasette.client.get("/data/dogs.json?_shape=array")
return r.json()
rows = loop.run_until_complete(check())
ids = [r["id"] for r in rows]
assert 3 not in ids
assert len(rows) == 2
# --- drop command ---
def test_drop(httpx_mock, ds, token):
_mock_datasette(httpx_mock, ds)
runner = CliRunner()
result = runner.invoke(
cli,
["drop", "http://localhost/data/dogs", "--yes", "--token", token],
)
assert result.exit_code == 0, result.output
data = json.loads(result.output)
assert data.get("ok") is True
# Verify table is gone
datasette, db, loop = ds
async def check():
return await db.table_names()
tables = loop.run_until_complete(check())
assert "dogs" not in tables
# --- create-table command ---
def test_create_table(httpx_mock, ds, token):
_mock_datasette(httpx_mock, ds)
runner = CliRunner()
result = runner.invoke(
cli,
[
"create-table",
"http://localhost/data",
"cats",
"--column",
"id",
"integer",
"--column",
"name",
"text",
"--pk",
"id",
"--token",
token,
],
)
assert result.exit_code == 0, result.output
data = json.loads(result.output)
assert data.get("ok") is True
# Verify table exists
datasette, db, loop = ds
async def check():
return await db.table_names()
tables = loop.run_until_complete(check())
assert "cats" in tables
# --- output format tests for query ---
SAMPLE_QUERY_RESPONSE = {
"ok": True,
"database": "data",
"rows": [
{"id": 1, "name": "Cleo", "age": 5},
{"id": 2, "name": "Pancakes", "age": 3},
{"id": 3, "name": "Fido", "age": 7},
],
"truncated": False,
"columns": ["id", "name", "age"],
}
def test_query_csv(httpx_mock):
httpx_mock.add_response(json=SAMPLE_QUERY_RESPONSE, status_code=200)
runner = CliRunner()
result = runner.invoke(
cli, ["query", "https://example.com/data", "select * from dogs", "--csv"]
)
assert result.exit_code == 0
assert "id,name,age" in result.output
assert "Cleo" in result.output
def test_query_nl(httpx_mock):
httpx_mock.add_response(json=SAMPLE_QUERY_RESPONSE, status_code=200)
runner = CliRunner()
result = runner.invoke(
cli, ["query", "https://example.com/data", "select * from dogs", "--nl"]
)
assert result.exit_code == 0
lines = [l for l in result.output.strip().split("\n") if l]
assert len(lines) == 3
first = json.loads(lines[0])
assert first["name"] == "Cleo"
def test_query_table_format(httpx_mock):
httpx_mock.add_response(json=SAMPLE_QUERY_RESPONSE, status_code=200)
runner = CliRunner()
result = runner.invoke(
cli, ["query", "https://example.com/data", "select * from dogs", "--table"]
)
assert result.exit_code == 0
assert "Cleo" in result.output
assert "---" in result.output