dclient/PLAN.md

dclient Improvement Plan

Feature Gap Analysis: dclient vs Datasette

What dclient currently supports

Feature	Status
SQL query execution (query)	Supported
Bulk data insertion (insert)	Supported (CSV, TSV, JSON, JSONL)
Actor/token introspection (actor)	Supported
URL aliases (alias add/list/remove)	Supported
Token storage (auth add/list/remove)	Supported
Bearer token authentication	Supported
Progress bar for inserts	Supported
Verbose/debug mode	Supported
Datasette plugin (datasette dc)	Supported

What Datasette exposes that dclient does NOT cover

Read Operations (high value)

Datasette Feature	API Endpoint	dclient Status
List databases	GET /-/databases.json	Missing
List tables/views in a database	GET /{db}.json	Missing
Browse/filter table rows	GET /{db}/{table}.json?filters...	Missing
Get single row by PK	GET /{db}/{table}/{pk}.json	Missing
Table schema inspection	GET /{db}/{table}/-/schema.json	Missing
Database schema inspection	GET /{db}/-/schema.json	Missing
Full-text search	?_search=term	Missing
Column filtering (exact, gt, lt, contains, etc.)	?col__op=val	Missing
Faceted browsing	?_facet=col	Missing
Sorting	?_sort=col / ?_sort_desc=col	Missing
Pagination	?_next=cursor&_size=N	Missing
Column selection	?_col=a&_col=b / ?_nocol=x	Missing
CSV/TSV export of table data	GET /{db}/{table}.csv	Missing
Canned queries listing	metadata in GET /{db}.json	Missing

Write Operations (high value)

Datasette Feature	API Endpoint	dclient Status
Row upsert (insert-or-update)	POST /{db}/{table}/-/upsert	Missing
Update a single row	POST /{db}/{table}/{pk}/-/update	Missing
Delete a single row	POST /{db}/{table}/{pk}/-/delete	Missing
Drop a table	POST /{db}/{table}/-/drop	Missing
Create empty table with schema	POST /{db}/-/create (columns only)	Missing

Metadata/Admin (medium value)

Datasette Feature	API Endpoint	dclient Status
Instance version info	GET /-/versions.json	Missing
Installed plugins	GET /-/plugins.json	Missing
Instance settings	GET /-/settings.json	Missing
Token creation	POST /-/create-token	Missing
Permission checks	GET /-/allowed.json	Missing

---

Proposed Plan

The plan is organized into tiers by impact and complexity, with the most valuable additions first.

---

Tier 1 -- Core Table Browsing & Schema Exploration

These commands turn dclient from "a way to run raw SQL" into a genuine database exploration tool. They make up the biggest gap today: a user who wants to see what tables exist or browse rows must either open a browser or hand-craft SQL.

1.1 dclient databases -- List databases

dclient databases <instance_url_or_alias>

Hits GET {url}/-/databases.json. Prints a table of database names, sizes, and whether they are mutable. Add --json for raw JSON output.

1.2 dclient tables -- List tables and views

dclient tables <db_url_or_alias>          # e.g. https://latest.datasette.io/fixtures
dclient tables <db_url_or_alias> --views  # include views
dclient tables <db_url_or_alias> --counts # include row counts
dclient tables <db_url_or_alias> --schema # include CREATE TABLE SQL

Hits GET {db}.json. Prints table names, row counts, and column lists. --json for raw JSON.

1.3 dclient schema -- Show table/database schema

dclient schema <db_url_or_alias>                     # full database schema
dclient schema <db_url_or_alias>/<table>             # single table schema

Hits GET {db}/-/schema.json or GET {db}/{table}/-/schema.json. Outputs the CREATE TABLE / CREATE VIEW SQL.

1.4 dclient rows -- Browse table rows with filtering

This is the biggest single feature addition. It exposes Datasette's powerful table filtering API without requiring the user to write SQL.

dclient rows <table_url_or_alias> [OPTIONS]

Filter flags:

--where 'column = value'        # maps to ?column__exact=value
--where 'column > 100'          # maps to ?column__gt=100
--where 'column contains foo'   # maps to ?column__contains=foo
--where 'column is null'        # maps to ?column__isnull
--where-sql 'age > 30'          # maps to ?_where=age > 30  (raw SQL)

Or a simpler column-value approach:

-w column=value                 # exact
-w column__gt=100               # pass Datasette filter operators directly

Other flags:

--sort column                   # ascending sort
--sort-desc column              # descending sort
--search 'full text query'      # FTS search (?_search=...)
--col name --col email          # select specific columns
--nocol description             # exclude columns
--facet column                  # facet counts
--size 50                       # page size (default 100)
--all                           # auto-paginate through all pages
--limit N                       # stop after N total rows
--csv                           # output as CSV
--tsv                           # output as TSV
--nl                            # output as newline-delimited JSON
--json                          # output as JSON (default)
--table                         # human-readable table format

Pagination: By default, print one page of results. --all follows next_url to fetch every page. --limit N caps total rows returned.

1.5 dclient get -- Fetch a single row by primary key

dclient get <table_url_or_alias> <pk_value>
dclient get <table_url_or_alias> <pk1>,<pk2>   # compound key

Hits GET {db}/{table}/{pk}.json. Prints the row as JSON.

---

Tier 2 -- Write Operations

dclient already supports insert and create. These additions round out the full CRUD lifecycle.

2.1 dclient upsert -- Insert or update rows

dclient upsert <db_url_or_alias> <table> <filepath> [OPTIONS]

Same interface as insert but hits the /-/upsert endpoint. Requires rows to contain primary key columns.

Shares most code with insert; the main difference is the endpoint URL and the absence of --replace/--ignore flags.

2.2 dclient update -- Update a single row

dclient update <table_url_or_alias> <pk_value> <key=value> [<key=value> ...]
dclient update <table_url_or_alias> <pk_value> --input file.json

Hits POST {db}/{table}/{pk}/-/update with {"update": {...}}.

2.3 dclient delete -- Delete rows

dclient delete <table_url_or_alias> <pk_value>
dclient delete <table_url_or_alias> <pk_value> --yes   # skip confirmation

Hits POST {db}/{table}/{pk}/-/delete. Prompts for confirmation unless --yes is passed.

2.4 dclient drop -- Drop a table

dclient drop <table_url_or_alias>
dclient drop <table_url_or_alias> --yes   # skip confirmation

Hits POST {db}/{table}/-/drop with {"confirm": true}. Shows row count and asks for confirmation unless --yes.

2.5 dclient create-table -- Create an empty table with explicit schema

dclient create-table <db_url_or_alias> <table_name> \
  --column id integer \
  --column name text \
  --column score float \
  --pk id

Hits POST {db}/-/create with a columns array instead of rows. The existing insert --create handles creation-with-data; this handles the schema-only case.

---

Tier 3 -- Output & UX Improvements

3.1 Multiple output formats for query and rows

Currently query only outputs JSON. Add:

--csv          # CSV output
--tsv          # TSV output
--nl           # newline-delimited JSON
--table        # human-readable ASCII table (like sqlite-utils)
--yaml         # YAML output (optional, low priority)

For --table output, use a simple column-aligned format or integrate with tabulate / rich (consider keeping dependencies minimal).

A lightweight approach: sqlite-utils already has table-formatting utilities that could be reused.

3.2 Pipe-friendly defaults

Detect whether stdout is a TTY:

• TTY: default to --table (human-readable) output
• Pipe: default to JSON (machine-readable) output

This matches the UX convention of tools like gh and jq.

3.3 --output / -o flag for writing to a file

dclient query ... -o results.csv --csv
dclient rows ... -o dump.json

3.4 Streaming output for large result sets

For --all pagination mode and --csv/--nl formats, stream rows as they arrive rather than buffering everything in memory. Write each page to stdout immediately.

---

Tier 4 -- Authentication Improvements

The current auth system works but has friction points. These changes make it smoother.

4.1 dclient auth login -- Browser-based authentication

dclient auth login <instance_url_or_alias>

Flow:

1. Open the user's browser to {instance}/-/create-token
2. Start a temporary local HTTP server to receive the token callback
3. User creates a token in the Datasette UI and pastes it, OR:
4. If the instance supports it, redirect back to the local server with the token
5. Store the token automatically via the existing auth system

For instances that don't support redirect, fall back to:

1. Open the browser to /-/create-token
2. Prompt the user to paste the token into the terminal

This is similar to how gh auth login, gcloud auth login, and heroku login work.

4.2 dclient auth token -- Create a token via the API

dclient auth token <instance_url_or_alias>
dclient auth token <instance_url_or_alias> --expires-after 3600
dclient auth token <instance_url_or_alias> --read-only

Hits POST /-/create-token. Requires an existing root token or session. Prints the new token and optionally stores it.

4.3 dclient auth status -- Validate stored credentials

dclient auth status <instance_url_or_alias>

Hits /-/actor.json with the stored token and prints:

• Whether the token is valid
• The actor identity (id, permissions)
• Token expiry if available

This is just a friendlier wrapper around the existing actor command, integrated into the auth subgroup.

4.4 Environment variable support

export DCLIENT_TOKEN=dstok_xxx
export DCLIENT_URL=https://my-datasette.example.com/mydb

Token resolution order (highest to lowest priority):

1. --token CLI flag
2. DCLIENT_TOKEN environment variable
3. Stored token in auth.json (existing behavior)

The DCLIENT_URL variable provides a default instance so users don't have to type it every time:

export DCLIENT_URL=https://my-datasette.example.com/mydb
dclient tables                    # uses DCLIENT_URL
dclient query 'select 1'         # uses DCLIENT_URL
dclient rows mytable --limit 10  # uses DCLIENT_URL

4.5 Keyring integration (optional, lower priority)

Instead of storing tokens in plaintext auth.json, optionally use the system keyring via the keyring Python package. This keeps tokens encrypted at rest on macOS (Keychain), Windows (Credential Vault), and Linux (Secret Service / KWallet).

Make this opt-in: dclient auth add --keyring <url>. Fall back to the current file-based storage if keyring is not installed.

---

Tier 5 -- Discoverability & Convenience

5.1 dclient info -- Instance overview

dclient info <instance_url_or_alias>

Hits /-/versions.json, /-/plugins.json, /-/databases.json in parallel. Prints a summary:

Datasette 1.0.1
Python 3.12.1
SQLite 3.45.0
Databases: 3 (fixtures, content, ephemeral)
Plugins: 12 installed

5.2 URL inference and shortcuts

Allow shorthand for common patterns:

# These should all work equivalently:
dclient rows https://latest.datasette.io/fixtures/facetable
dclient rows latest fixtures/facetable    # if 'latest' is an alias
dclient rows latest facetable             # if alias points to a database URL

The key insight: if an alias points to https://x.com/dbname, then dclient rows <alias> <table> should construct https://x.com/dbname/table.json.

Currently the alias is just a URL prefix that gets used verbatim. Making the alias system slightly smarter about joining paths would reduce typing significantly.

5.3 dclient config -- Manage configuration

dclient config show              # print config directory and contents
dclient config path              # print config directory path
dclient config edit              # open config in $EDITOR

Useful for debugging when tokens or aliases aren't working as expected.

5.4 Shell completions

Use Click's built-in shell completion support to generate completions for bash, zsh, and fish:

dclient --install-completion bash
dclient --install-completion zsh
dclient --install-completion fish

Click 8.x supports this natively via shell_complete. This could also dynamically complete alias names, database names, and table names by querying the stored aliases.

---

Tier 6 -- Advanced Features (lower priority)

6.1 dclient export -- Bulk export table data

dclient export <table_url> -o data.csv --csv
dclient export <table_url> -o data.json --json
dclient export <table_url> -o data.db   # export to local SQLite

This is essentially dclient rows --all but optimized for bulk export:

• Streams using ?_stream=1 (CSV format) when available
• For JSON, auto-paginates using _next cursor
• For SQLite output, pipes into sqlite-utils insert

6.2 Parallel insert for large files

When inserting very large files, optionally send batches in parallel:

dclient insert <url> <table> huge.csv --parallel 4

Uses a thread pool to send N batches concurrently. Would require switching from synchronous httpx to httpx.AsyncClient or using concurrent.futures.ThreadPoolExecutor.

6.3 dclient diff -- Compare local and remote data

dclient diff local.db remote_alias/tablename

Compare a local SQLite table with a remote Datasette table. Show added, removed, and modified rows. Useful for sync workflows.

---

Implementation Notes

Shared infrastructure to build first

Several of the proposed commands share common patterns. Before implementing individual commands, refactor these into shared utilities:

1. URL resolution helper: Centralize the alias-lookup + URL-construction logic that is currently duplicated in every command. A single function like resolve_url(url_or_alias, suffix=None) that handles aliases, adds .json where appropriate, and joins path segments.

2. Authenticated HTTP client helper: A function or context manager that creates an httpx.Client with the right Authorization header and timeout, resolved from --token / env var / auth.json. This eliminates the repeated token-resolution boilerplate in every command.

3. Output formatter: A shared output module that handles --json, --csv, --tsv, --nl, --table flags consistently across all commands that return data.

4. Pagination helper: A generator that follows next_url links to yield all pages of results, used by rows --all and export.

Dependency considerations

The current dependencies are minimal: click, httpx, sqlite-utils. The proposed changes mostly stay within these. Potential additions:

• tabulate or rich for --table output (or implement a minimal version to avoid new deps)
• keyring for optional encrypted token storage (optional dependency)

Backward compatibility

All proposed changes are additive (new commands and new flags). No existing command signatures or behaviors change. The only subtle change would be if TTY-detection changes the default output format of query, which should be gated behind a major version bump or opt-in flag.

Testing strategy

Each new command should follow the existing test patterns:

• Unit tests with pytest-httpx for HTTP mocking
• Integration tests against a real in-memory Datasette instance (as done in test_insert.py)
• cogapp for keeping --help output in docs in sync

---

Priority Summary

Priority	Feature	Rationale
P0	databases, tables, schema	Basic exploration; most-requested gap
P0	rows with filtering/sorting/pagination	Replaces need for raw SQL for common tasks
P0	Environment variable support (DCLIENT_TOKEN, DCLIENT_URL)	Near-zero effort, big UX win
P1	get, update, delete, drop	Completes CRUD lifecycle
P1	upsert	Natural complement to existing insert
P1	Multiple output formats (CSV, table, NL)	Makes query/rows output usable in pipelines
P1	auth login (browser-based)	Biggest auth UX improvement
P2	create-table (schema only)	Useful but insert --create covers most cases
P2	TTY-aware default output	Polish
P2	auth status, auth token	Convenience wrappers
P2	info	Nice discoverability
P2	URL inference / smarter aliases	Reduces typing
P3	export (bulk)	Covered by rows --all for most cases
P3	Shell completions	Polish
P3	Keyring integration	Security hardening
P3	Parallel insert	Performance optimization
P3	diff	Advanced workflow