Fixes for SQL write with RETURNING (#2763)

* Fix for execute write returning, closes #2762
* Fix stored write returning rowcount message
* Add configurable execute_write returning limit
* Return rows/truncated from execute query if it used RETURNING
* INSERT ... RETURNING shows rows in /-/execute-write
* Skip RETURNING tests if SQLite version does not support it

Screenshot: https://github.com/simonw/datasette/issues/2762#issuecomment-4588111545

docs/internals.rst

@@ -1928,8 +1928,8 @@ Example usage:
 
 .. _database_execute_write:
 
-await db.execute_write(sql, params=None, block=True)
-----------------------------------------------------
+await db.execute_write(sql, params=None, block=True, request=None, return_all=False, returning_limit=10)
+--------------------------------------------------------------------------------------------------------
 
 SQLite only allows one database connection to write at a time. Datasette handles this for you by maintaining a queue of writes to be executed against a given database. Plugins can submit write operations to this queue and they will be executed in the order in which they are received.
 
@@ -1937,7 +1937,30 @@ This method can be used to queue up a non-SELECT SQL query to be executed agains
 
 You can pass additional SQL parameters as a tuple or dictionary.
 
-The method will block until the operation is completed, and the return value will be the return from calling ``conn.execute(...)`` using the underlying ``sqlite3`` Python library.
+The optional ``request=`` argument is used internally by Datasette to pass request context to :ref:`write_wrapper plugin hooks <plugin_hook_write_wrapper>`.
+
+The method will block until the operation is completed, and the return value will be an ``ExecuteWriteResult`` object. This imitates a subset of the ``sqlite3.Cursor`` object:
+
+``.rowcount``
+    The number of rows modified by the statement, or ``-1`` if that number is unavailable.
+
+``.lastrowid``
+    The row ID of the last modified row, as returned by ``sqlite3.Cursor.lastrowid``.
+
+``.description``
+    The same column metadata exposed by Python's `sqlite3.Cursor.description <https://docs.python.org/3/library/sqlite3.html#sqlite3.Cursor.description>`__: one tuple per returned column, or ``None`` if the statement does not return rows.
+
+``.truncated``
+    ``True`` if the statement returned more rows than ``returning_limit``.
+
+``.fetchall()``
+    Returns any rows buffered by Datasette from the statement, such as rows from SQLite's ``RETURNING`` clause. This may be limited by ``returning_limit`` unless ``return_all=True`` was used. This method empties the buffer, so calling it again will return an empty list.
+
+SQLite statements using ``RETURNING`` must have their rows consumed before the transaction can commit. Datasette will fetch up to ``returning_limit + 1`` rows before committing, store up to ``returning_limit`` rows on the result object and set ``.truncated`` if there were more. The default ``returning_limit`` is ``10``.
+
+When ``.truncated`` is ``True``, ``.rowcount`` will be ``-1``. SQLite only reports the final row count for a ``RETURNING`` statement after every returned row has been fetched, and Datasette has deliberately stopped fetching rows after ``returning_limit`` to avoid buffering a potentially large result in memory.
+
+If you need to retrieve every row returned by a statement, pass ``return_all=True``. This will buffer all returned rows in memory before committing.
 
 If you pass ``block=False`` this behavior changes to "fire and forget" - queries will be added to the write queue and executed in a separate thread while your code can continue to do other things. The method will return a UUID representing the queued task.
 

docs/json_api.rst

@@ -554,7 +554,8 @@ Datasette analyzes the SQL before executing it. The actor must have ``execute-wr
 
 Unsupported SQL operations are rejected by default. ``VACUUM`` is not allowed in arbitrary write SQL, and writes to SQLite virtual tables or shadow tables are rejected. SQL functions are allowed and are not separately restricted by Datasette permissions.
 
-A successful response includes a message, the SQLite ``rowcount`` and a summary of the operations that were executed:
+A successful response includes a message, the SQLite ``rowcount``, a ``"rows"``
+list, a ``"truncated"`` flag and a summary of the operations that were executed:
 
 The shape of the ``"analysis"`` block is not yet considered a stable API and may change in future Datasette releases.
 
@@ -564,6 +565,8 @@ The shape of the ``"analysis"`` block is not yet considered a stable API and may
         "ok": true,
         "message": "Query executed, 1 row affected",
         "rowcount": 1,
+        "rows": [],
+        "truncated": false,
         "analysis": [
             {
                 "operation": "insert",
@@ -577,6 +580,43 @@ The shape of the ``"analysis"`` block is not yet considered a stable API and may
 
 If SQLite reports ``-1`` for the row count, the message will be ``"Query executed"``.
 
+For most write statements ``"rows"`` will be an empty list and ``"truncated"``
+will be ``false``. If the SQL uses SQLite's ``RETURNING`` clause, ``"rows"``
+will contain returned rows using the same default representation as table and
+query JSON responses. ``"truncated"`` indicates if more rows were returned than
+the execute-write returning row limit, which defaults to 10:
+
+.. code-block:: json
+
+    {
+        "ok": true,
+        "message": "Query executed, 1 row affected",
+        "rowcount": 1,
+        "rows": [
+            {
+                "id": 1,
+                "name": "Cleo"
+            }
+        ],
+        "truncated": false,
+        "analysis": [
+            {
+                "operation": "insert",
+                "database": "data",
+                "table": "dogs",
+                "required_permission": "insert-row, update-row, delete-row",
+                "source": null
+            },
+            {
+                "operation": "read",
+                "database": "data",
+                "table": "dogs",
+                "required_permission": "view-table",
+                "source": null
+            }
+        ]
+    }
+
 Errors use the standard Datasette error format:
 
 .. code-block:: json