JSON output no longer escapes non-ASCII characters, new --ascii option (#777)

Closes #625

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JaHan1NhaTRAxJ9LQtSLf9

docs/changelog.rst

@@ -4,6 +4,13 @@
  Changelog
 ===========
 
+.. _unreleased:
+
+Unreleased
+----------
+
+- JSON output from the command-line tool no longer escapes non-ASCII characters, so ``sqlite-utils data.db "select '日本語' as text"`` now outputs ``[{"text": "日本語"}]``. This matches how values were already stored by ``insert`` and how CSV/TSV output already behaved. A new ``--ascii`` option restores the previous behavior of escaping non-ASCII characters, for output destinations that cannot handle UTF-8 - see :ref:`cli_query_json_ascii`. The option is available on the ``query``, ``rows``, ``search``, ``tables``, ``views``, ``triggers``, ``indexes`` and ``memory`` commands. The ``convert --multi --dry-run`` preview and ``plugins`` output also no longer escape non-ASCII characters. (:issue:`625`)
+
 .. _v4_0rc3:
 
 4.0rc3 (2026-07-05)

docs/cli-reference.rst

@@ -129,6 +129,8 @@ See :ref:`cli_query`.
                                   simple_outline, textile, tsv, unsafehtml, youtrack
       --json-cols                 Detect JSON cols and output them as JSON, not
                                   escaped strings
+      --ascii                     Escape non-ASCII characters in JSON output as
+                                  \uXXXX
       -r, --raw                   Raw output, first column of first row
       --raw-lines                 Raw output, first column of each row
       -p, --param <TEXT TEXT>...  Named :parameters for SQL query
@@ -198,6 +200,8 @@ See :ref:`cli_memory`.
                                   simple_outline, textile, tsv, unsafehtml, youtrack
       --json-cols                 Detect JSON cols and output them as JSON, not
                                   escaped strings
+      --ascii                     Escape non-ASCII characters in JSON output as
+                                  \uXXXX
       -r, --raw                   Raw output, first column of first row
       --raw-lines                 Raw output, first column of each row
       -p, --param <TEXT TEXT>...  Named :parameters for SQL query
@@ -435,6 +439,7 @@ See :ref:`cli_search`.
                              youtrack
       --json-cols            Detect JSON cols and output them as JSON, not escaped
                              strings
+      --ascii                Escape non-ASCII characters in JSON output as \uXXXX
       --load-extension TEXT  Path to SQLite extension, with optional :entrypoint
       -h, --help             Show this message and exit.
 
@@ -701,6 +706,7 @@ See :ref:`cli_tables`.
                              youtrack
       --json-cols            Detect JSON cols and output them as JSON, not escaped
                              strings
+      --ascii                Escape non-ASCII characters in JSON output as \uXXXX
       --columns              Include list of columns for each table
       --schema               Include schema for each table
       --load-extension TEXT  Path to SQLite extension, with optional :entrypoint
@@ -743,6 +749,7 @@ See :ref:`cli_views`.
                              youtrack
       --json-cols            Detect JSON cols and output them as JSON, not escaped
                              strings
+      --ascii                Escape non-ASCII characters in JSON output as \uXXXX
       --columns              Include list of columns for each view
       --schema               Include schema for each view
       --load-extension TEXT  Path to SQLite extension, with optional :entrypoint
@@ -790,6 +797,8 @@ See :ref:`cli_rows`.
                                   simple_outline, textile, tsv, unsafehtml, youtrack
       --json-cols                 Detect JSON cols and output them as JSON, not
                                   escaped strings
+      --ascii                     Escape non-ASCII characters in JSON output as
+                                  \uXXXX
       --load-extension TEXT       Path to SQLite extension, with optional
                                   :entrypoint
       -h, --help                  Show this message and exit.
@@ -830,6 +839,7 @@ See :ref:`cli_triggers`.
                              youtrack
       --json-cols            Detect JSON cols and output them as JSON, not escaped
                              strings
+      --ascii                Escape non-ASCII characters in JSON output as \uXXXX
       --load-extension TEXT  Path to SQLite extension, with optional :entrypoint
       -h, --help             Show this message and exit.
 
@@ -870,6 +880,7 @@ See :ref:`cli_indexes`.
                              youtrack
       --json-cols            Detect JSON cols and output them as JSON, not escaped
                              strings
+      --ascii                Escape non-ASCII characters in JSON output as \uXXXX
       --load-extension TEXT  Path to SQLite extension, with optional :entrypoint
       -h, --help             Show this message and exit.
 

docs/cli.rst

@@ -111,6 +111,33 @@ If you want to pretty-print the output further, you can pipe it through ``python
         }
     ]
 
+.. _cli_query_json_ascii:
+
+Unicode characters in JSON
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+JSON output includes unicode characters directly, without escaping them:
+
+.. code-block:: bash
+
+    sqlite-utils dogs.db "select '日本語' as text"
+
+.. code-block:: output
+
+    [{"text": "日本語"}]
+
+Use ``--ascii`` to escape non-ASCII characters as ``\uXXXX`` sequences instead:
+
+.. code-block:: bash
+
+    sqlite-utils dogs.db "select '日本語' as text" --ascii
+
+.. code-block:: output
+
+    [{"text": "\u65e5\u672c\u8a9e"}]
+
+The ``--ascii`` option can help on systems that cannot display or process UTF-8, such as Windows consoles using a legacy code page. On Windows, setting the ``PYTHONUTF8=1`` environment variable is an alternative fix for ``UnicodeEncodeError`` crashes when redirecting output to a file.
+
 .. _cli_query_binary_json:
 
 Binary data in JSON