Merge remote-tracking branch 'origin/main' into claude/json-api-docs-1-0-review-a3e83u

# Conflicts:
#	datasette/__init__.py
#	tests/test_api_write.py

docs/authentication.rst

@@ -50,7 +50,7 @@ The one exception is the "root" account, which you can sign into while using Dat
 The ``--root`` flag is designed for local development and testing. When you start Datasette with ``--root``, the root user automatically receives every permission, including:
 
 * All view permissions (``view-instance``, ``view-database``, ``view-table``, etc.)
-* All write permissions (``insert-row``, ``update-row``, ``delete-row``, ``create-table``, ``alter-table``, ``set-column-type``, ``drop-table``)
+* All write permissions (``insert-row``, ``update-row``, ``delete-row``, ``create-table``, ``create-view``, ``alter-table``, ``set-column-type``, ``drop-table``, ``drop-view``)
 * Debug permissions (``permissions-debug``, ``debug-menu``)
 * Any custom permissions defined by plugins
 
@@ -1390,6 +1390,16 @@ create-table
 
 Actor is allowed to create a database table.
 
+``resource`` - ``datasette.resources.DatabaseResource(database)``
+    ``database`` is the name of the database (string)
+
+.. _actions_create_view:
+
+create-view
+-----------
+
+Actor is allowed to create a database view.
+
 ``resource`` - ``datasette.resources.DatabaseResource(database)``
     ``database`` is the name of the database (string)
 
@@ -1429,6 +1439,18 @@ Actor is allowed to drop a database table.
 
     ``table`` is the name of the table (string)
 
+.. _actions_drop_view:
+
+drop-view
+---------
+
+Actor is allowed to drop a database view.
+
+``resource`` - ``datasette.resources.TableResource(database, table)``
+    ``database`` is the name of the database (string)
+
+    ``table`` is the name of the view (string)
+
 .. _actions_execute_sql:
 
 execute-sql

docs/binary_data.rst

@@ -12,7 +12,12 @@ Datasette includes special handling for these binary values. The Datasette inter
    :width: 311px
    :alt: Screenshot showing download links next to binary data in the table view
 
-Binary data is represented in ``.json`` exports using Base64 encoding.
+.. _binary_json_format:
+
+Binary values in JSON
+---------------------
+
+Binary data is represented in ``.json`` exports using Base64 encoding. Datasette uses this representation for every ``BLOB`` value, including binary values that could also be decoded as UTF-8 text.
 
 https://latest.datasette.io/fixtures/binary_data.json?_shape=array
 
@@ -39,6 +44,48 @@ https://latest.datasette.io/fixtures/binary_data.json?_shape=array
         }
     ]
 
+The same format can be used with the :ref:`JSON write API <json_api_write>`.
+If a column value in a ``row``, ``rows`` or ``update`` object is a JSON object with exactly ``"$base64"`` set to ``true`` and an ``"encoded"`` string, Datasette will decode that Base64 string and store the resulting bytes:
+
+.. code-block:: json
+
+    {
+        "data": {
+            "$base64": true,
+            "encoded": "FRwCx60F/g=="
+        }
+    }
+
+This works for inserts, upserts and updates. It also works when creating a table from example ``row`` or ``rows`` data: Datasette decodes the value before inferring the schema, allowing that column to be created as a ``BLOB`` column.
+
+To store a JSON object with that exact shape literally, wrap it in a ``"$raw"`` object:
+
+.. code-block:: json
+
+    {
+        "data": {
+            "$raw": {
+                "$base64": true,
+                "encoded": "FRwCx60F/g=="
+            }
+        }
+    }
+
+``"$raw"`` unwraps exactly one layer. To store a literal ``"$raw"`` object containing a Base64 object, wrap it again:
+
+.. code-block:: json
+
+    {
+        "data": {
+            "$raw": {
+                "$raw": {
+                    "$base64": true,
+                    "encoded": "FRwCx60F/g=="
+                }
+            }
+        }
+    }
+
 .. _binary_linking:
 
 Linking to binary downloads

docs/changelog.rst

@@ -4,6 +4,19 @@
 Changelog
 =========
 
+.. _unreleased:
+
+Unreleased
+----------
+
+- Table pages now offer an "Insert multiple rows" mode in the row insertion dialog. This accepts pasted TSV, CSV or JSON, previews the parsed rows before inserting them, validates unknown columns as data is pasted and displays omitted auto integer primary keys as ``auto`` in the preview. (:pr:`2813`)
+- The bulk insert UI can skip rows with existing primary keys, or update existing rows and insert new rows using the existing ``/<database>/<table>/-/upsert`` API when the actor has both :ref:`insert-row <actions_insert_row>` and :ref:`update-row <actions_update_row>` permissions. (:pr:`2813`)
+- The "Create table" dialog now includes a "Create table from data" mode. Paste TSV, CSV or JSON rows to preview inferred columns and types, choose the table name and primary key, then create the table and insert those rows in one step. (:pr:`2813`)
+- Datasette's JSON APIs now consistently encode every ``BLOB`` value using the documented :ref:`binary value JSON format <binary_json_format>`, even when the bytes could be decoded as UTF-8 text.
+- The insert and edit row dialogs now provide a dedicated control for ``BLOB`` values. Existing binary values are shown by byte size, image values under 10MB are previewed as thumbnails, and replacements can be attached, dropped or pasted into the control.
+- The table and row JSON APIs now support ``?_extra=column_details`` for returning SQLite schema details for columns, including declared type, SQLite affinity, primary key, ``NOT NULL``, default and hidden-column metadata.
+- POST bodies that Datasette reads fully into memory - such as JSON submitted to the write API - are now capped by the new :ref:`setting_max_post_body_bytes` setting, defaulting to 2MB. Oversized requests are rejected with an HTTP 413 error as soon as the limit is exceeded, protecting smaller servers from memory exhaustion. File uploads are unaffected - ``request.form()`` streams those to disk and has its own separate limits.
+
 .. _v1_0_a35:
 
 1.0a35 (2026-06-23)

docs/cli-reference.rst

@@ -244,6 +244,9 @@ These can be passed to ``datasette serve`` using ``datasette serve --setting nam
                                    custom query (default=1000)
       max_insert_rows              Maximum rows that can be inserted at a time using
                                    the bulk insert API (default=100)
+      max_post_body_bytes          Maximum size in bytes for a POST body read into
+                                   memory, e.g. JSON API requests - set 0 to disable
+                                   this limit (default=2097152)
       num_sql_threads              Number of threads in the thread pool for
                                    executing SQLite queries (default=3)
       sql_time_limit_ms            Time limit for a SQL query in milliseconds

docs/internals.rst

@@ -52,6 +52,9 @@ The request object is passed to various plugin hooks. It represents an incoming
 ``.actor`` - dictionary (str -> Any) or None
     The currently authenticated actor (see :ref:`actors <authentication_actor>`), or ``None`` if the request is unauthenticated.
 
+``.max_post_body_bytes`` - integer
+    The maximum number of bytes ``await request.post_body()`` will read into memory, or ``0`` for no limit. Set from the :ref:`setting_max_post_body_bytes` setting (default 2MB) for requests created by Datasette. Can be passed to the ``Request`` constructor as a keyword argument.
+
 The object also has the following awaitable methods:
 
 ``await request.form(files=False, ...)`` - FormData
@@ -109,9 +112,11 @@ The object also has the following awaitable methods:
 ``await request.json()`` - Any
     Returns the parsed JSON body of a request submitted by ``POST``.
 
-``await request.post_body()`` - bytes
+``await request.post_body(max_bytes=None)`` - bytes
     Returns the un-parsed body of a request submitted by ``POST`` - useful for things like incoming JSON data.
 
+    The body is read fully into memory, capped at ``request.max_post_body_bytes`` - which Datasette sets from the :ref:`setting_max_post_body_bytes` setting (default 2MB). Bodies that exceed the limit raise a ``datasette.PayloadTooLarge`` exception, which Datasette turns into an HTTP 413 error response. Pass ``max_bytes=`` to override the limit for a specific call, or ``max_bytes=0`` to disable it. ``request.post_vars()`` and ``request.json()`` read the body through this method, so the same limit applies to them.
+
 And a class method that can be used to create fake request objects for use in tests:
 
 ``fake(path_with_query_string, method="GET", scheme="http", url_vars=None)``

docs/json_api.rst

@@ -497,6 +497,25 @@ The available table extras are listed below.
           "pk"
         ]
 
+``column_details``
+    SQLite schema details for columns in this table. The dictionary maps column names to objects describing the schema for each column. (Each object has ``type`` as the declared type string returned by SQLite, or ``""`` if no type was declared; ``sqlite_type`` as the normalized SQLite affinity, one of ``TEXT``, ``INTEGER``, ``REAL``, ``BLOB`` or ``NUMERIC``; ``notnull`` as a boolean; ``default`` as the raw SQL default expression string, such as ``"42"``, ``"'hello'"`` or ``"datetime('now')"``, or ``null`` if there is no default; ``is_pk`` as a boolean; ``pk_position`` as the integer primary key position reported by SQLite, or ``0`` for columns that are not part of the primary key; and ``hidden`` as the integer value reported by SQLite's ``PRAGMA table_xinfo``. ``hidden`` is ``0`` for normal columns, ``1`` for hidden virtual table columns, ``2`` for virtual generated columns and ``3`` for stored generated columns.)
+
+    ``GET /fixtures/binary_data.json?_size=0&_extra=column_details``
+
+    .. code-block:: json
+
+        {
+          "data": {
+            "type": "BLOB",
+            "sqlite_type": "BLOB",
+            "notnull": false,
+            "default": null,
+            "is_pk": false,
+            "pk_position": 0,
+            "hidden": 0
+          }
+        }
+
 ``display_columns``
     Column metadata used by the HTML table display. Each item includes ``name``, ``sortable``, ``is_pk``, ``type``, ``notnull``, ``description``, ``column_type`` and ``column_type_config`` keys.
 
@@ -902,6 +921,25 @@ The following extras are available for row JSON responses.
           "id"
         ]
 
+``column_details``
+    SQLite schema details for columns in this table. The dictionary maps column names to objects describing the schema for each column. (Each object has ``type`` as the declared type string returned by SQLite, or ``""`` if no type was declared; ``sqlite_type`` as the normalized SQLite affinity, one of ``TEXT``, ``INTEGER``, ``REAL``, ``BLOB`` or ``NUMERIC``; ``notnull`` as a boolean; ``default`` as the raw SQL default expression string, such as ``"42"``, ``"'hello'"`` or ``"datetime('now')"``, or ``null`` if there is no default; ``is_pk`` as a boolean; ``pk_position`` as the integer primary key position reported by SQLite, or ``0`` for columns that are not part of the primary key; and ``hidden`` as the integer value reported by SQLite's ``PRAGMA table_xinfo``. ``hidden`` is ``0`` for normal columns, ``1`` for hidden virtual table columns, ``2`` for virtual generated columns and ``3`` for stored generated columns.)
+
+    ``GET /fixtures/binary_data/1.json?_extra=column_details``
+
+    .. code-block:: json
+
+        {
+          "data": {
+            "type": "BLOB",
+            "sqlite_type": "BLOB",
+            "notnull": false,
+            "default": null,
+            "is_pk": false,
+            "pk_position": 0,
+            "hidden": 0
+          }
+        }
+
 ``render_cell``
     Rendered HTML for each cell using the render_cell plugin hook (See the :ref:`render_cell() plugin hook <plugin_hook_render_cell>` documentation.)
 
@@ -1626,6 +1664,8 @@ The JSON write API
 
 Datasette provides a write API for JSON data. This is a POST-only API that requires an authenticated API token, see :ref:`CreateTokenView`. The token will need to have the specified :ref:`authentication_permissions`.
 
+The row-based write APIs can write :ref:`binary values in JSON <binary_json_format>` using Datasette's Base64 representation for BLOB data.
+
 .. _ExecuteWriteView:
 
 Executing write SQL
@@ -1756,6 +1796,8 @@ A single row can be inserted using the ``"row"`` key:
         }
     }
 
+Column values can use the :ref:`binary value JSON format <binary_json_format>` to write BLOB data.
+
 If successful, this will return a ``201`` status code and the newly inserted row, for example:
 
 .. code-block:: json
@@ -1863,6 +1905,8 @@ An upsert is an insert or update operation. If a row with a matching primary key
 
 The upsert API is mostly the same shape as the :ref:`insert API <TableInsertView>`. It requires both the :ref:`actions_insert_row` and :ref:`actions_update_row` permissions.
 
+It also accepts the same :ref:`binary value JSON format <binary_json_format>`.
+
 ::
 
     POST /<database>/<table>/-/upsert
@@ -1995,6 +2039,8 @@ To update a row, make a ``POST`` to ``/<database>/<table>/<row-pks>/-/update``.
 
 You only need to pass the columns you want to update. Any other columns will be left unchanged.
 
+Updated values can use the :ref:`binary value JSON format <binary_json_format>`.
+
 If successful, this will return a ``200`` status code and a ``{"ok": true}`` response body.
 
 Add ``"return": true`` to the request body to return the updated row:
@@ -2200,6 +2246,8 @@ Datasette will create a table with a schema that matches those rows and insert t
         "pk": "id"
     }
 
+Example rows can use the :ref:`binary value JSON format <binary_json_format>`, allowing Datasette to infer ``BLOB`` columns.
+
 Doing this requires both the :ref:`actions_create_table` and :ref:`actions_insert_row` permissions.
 
 The ``201`` response here will be similar to the ``columns`` form, but will also include the number of rows that were inserted as ``row_count``:

docs/settings.rst

@@ -125,6 +125,23 @@ You can increase or decrease this limit like so::
 
     datasette mydatabase.db --setting max_insert_rows 1000
 
+.. _setting_max_post_body_bytes:
+
+max_post_body_bytes
+~~~~~~~~~~~~~~~~~~~
+
+Maximum size in bytes for a POST body that Datasette reads fully into memory, such as JSON submitted to the :ref:`write API <json_api_write>`. Requests with larger bodies are rejected with an HTTP 413 error. Defaults to 2,097,152 (2MB).
+
+This limit exists to protect against memory exhaustion: unlike file uploads handled by ``request.form()``, which stream to disk, these bodies are held entirely in memory and parsing them as JSON can multiply their memory footprint several times over.
+
+If you increase :ref:`setting_max_insert_rows` to support larger bulk inserts you may need to increase this limit as well::
+
+    datasette mydatabase.db --setting max_post_body_bytes 10485760
+
+Set it to 0 to disable the limit entirely::
+
+    datasette mydatabase.db --setting max_post_body_bytes 0
+
 .. _setting_num_sql_threads:
 
 num_sql_threads

docs/template_context.rst

@@ -98,7 +98,7 @@ The page listing the tables, views and queries in a database, e.g. /fixtures. Re
     The color assigned to the database
 
 ``database_page_data`` - ``dict``
-    JSON data used by JavaScript on the database page. Currently ``{}`` or ``{"createTable": {...}}`` where ``createTable`` includes ``path``, ``foreignKeyTargetsPath``, ``databaseName``, ``columnTypes``, ``defaultExpressions`` and optional ``customColumnTypes``.
+    JSON data used by JavaScript on the database page. Currently ``{}`` or ``{"createTable": {...}}`` where ``createTable`` includes ``path``, ``foreignKeyTargetsPath``, ``databaseName``, ``columnTypes``, ``defaultExpressions``, ``canInsertRows`` and optional ``customColumnTypes``.
 
 ``editable`` - ``bool``
     Boolean indicating if the database is editable
@@ -389,7 +389,7 @@ Many of these keys are shared with the :ref:`JSON API <json_api>` for this page.
     SQL definition for this table
 
 ``table_insert_ui`` - ``dict``
-    Information needed to enable the row insertion UI, or ``None`` if row insertion is not available to the current actor. When present it has ``path``, ``tableName``, ``columns`` and ``primaryKeys`` keys; each column includes ``name``, ``sqlite_type``, ``notnull``, ``default``, ``has_default``, ``is_pk``, ``value_kind`` and ``column_type`` keys.
+    Information needed to enable the row insertion UI, or ``None`` if row insertion is not available to the current actor. When present it has ``path``, ``tableName``, ``columns``, ``bulkColumns``, ``primaryKeys`` and ``maxInsertRows`` keys, plus optional ``upsertPath`` if the current actor has permission to update rows. ``columns`` lists columns for the single-row insert form, while ``bulkColumns`` lists columns for the bulk insert form. Each column includes ``name``, ``sqlite_type``, ``notnull``, ``default``, ``has_default``, ``is_pk``, ``is_auto_pk``, ``value_kind`` and ``column_type`` keys.
 
 ``table_page_data`` - ``dict``
     JSON data used by JavaScript on the table page. Includes ``database``, ``table`` and ``tableUrl``, plus optional ``foreignKeys`` mapping column names to autocomplete URLs, optional ``insertRow`` data and optional ``alterTable`` data.