Support ANY column types for strict tables

Closes #790, #820

docs/changelog.rst

@@ -10,6 +10,7 @@ Unreleased
 ----------
 
 - New ``table.checks``, ``table.column_checks`` and ``table.table_checks`` introspection properties expose column-level and table-level ``CHECK`` constraints. (:issue:`834`)
+- New ``sqlite_utils.ANY`` marker type for creating and introspecting SQLite ``ANY`` columns. The Python API and CLI can create, add and transform these columns, and ``table.transform()`` and ``table.extract()`` now preserve ``ANY`` columns and their values in ``STRICT`` tables. (:issue:`790`)
 - ``table.transform()`` now preserves ``CHECK`` constraints, including comments within their expressions. Renaming a column rewrites identifier references in checks without changing string literals or function names. Dropping a column drops a check owned by that column, and raises ``TransformError`` if a remaining check depends on it. (:issue:`762`)
 - ``table.transform()`` now preserves comments immediately before or after column definitions. These comments move with the column if it is renamed or reordered, and are removed if the column is dropped. (:issue:`762`)
 - ``table.transform(rename=...)`` now preserves explicit indexes on renamed columns by dropping and recreating those indexes against the new column names. Previously this raised a ``TransformError``. (:issue:`822`)

docs/cli-reference.rst

@@ -494,7 +494,7 @@ See :ref:`cli_transform_table`.
 
     Options:
       --type <TEXT CHOICE>...         Change column type to INTEGER, TEXT, FLOAT,
-                                      REAL or BLOB
+                                      REAL, BLOB or ANY
       --drop TEXT                     Drop this column
       --rename <TEXT TEXT>...         Rename this column to X
       -o, --column-order TEXT         Reorder columns
@@ -963,7 +963,7 @@ See :ref:`cli_create_table`.
               height real \
               photo blob --pk id
 
-      Valid column types are text, integer, real, float and blob.
+      Valid column types are text, integer, real, float, blob and any.
 
     Options:
       --pk TEXT                 Column to use as primary key
@@ -1257,7 +1257,7 @@ See :ref:`cli_add_column`.
 ::
 
     Usage: sqlite-utils add-column [OPTIONS] PATH TABLE COL_NAME
-                          [integer|int|float|real|text|str|blob|bytes]
+                          [integer|int|float|real|text|str|blob|bytes|any]
 
       Add a column to the specified table
 

docs/cli.rst

@@ -1390,7 +1390,14 @@ Use ``--type column-name type`` to override the type automatically chosen when t
 
 This is useful for values such as ZIP codes, which may look like integers but should be stored as ``TEXT`` to preserve leading zeros.
 
-The column type should be one of ``TEXT``, ``INTEGER``, ``FLOAT``, ``REAL`` or ``BLOB``. Column types are matched case-insensitively.
+The column type should be one of ``TEXT``, ``INTEGER``, ``FLOAT``, ``REAL``, ``BLOB`` or ``ANY``. Column types are matched case-insensitively.
+
+``ANY`` is especially useful with ``--strict``. An ``ANY`` column in a strict table preserves values without coercion, so text such as ``000123`` remains text instead of being converted to an integer:
+
+.. code-block:: bash
+
+    sqlite-utils insert events.db events events.csv --csv --strict \
+        --type payload any
 
 As with detected column types, ``--type`` only affects tables created by the command. If the table already exists, its existing column types are left unchanged.
 
@@ -2141,6 +2148,12 @@ You can create a table in `SQLite STRICT mode <https://www.sqlite.org/stricttabl
 
    sqlite-utils create-table mydb.db mytable id integer name text --strict
 
+Use the ``any`` type for a strict column that should accept integers, floating point values, text, binary data or null without coercion:
+
+.. code-block:: bash
+
+   sqlite-utils create-table events.db events id integer payload any --strict
+
 .. code-block:: bash
 
    sqlite-utils tables mydb.db --schema -t
@@ -2223,7 +2236,7 @@ The ``transform`` command allows you to apply complex transformations to a table
 Every option for this table (with the exception of ``--pk-none``) can be specified multiple times. The options are as follows:
 
 ``--type column-name new-type``
-    Change the type of the specified column. Valid types are ``integer``, ``text``, ``float``, ``blob``.
+    Change the type of the specified column. Valid types are ``integer``, ``text``, ``float``, ``real``, ``blob`` and ``any``.
 
 ``--drop column-name``
     Drop the specified column.

docs/python-api.rst

@@ -828,6 +828,19 @@ You can pass ``strict=True`` to create a table in ``STRICT`` mode:
         "name": str,
     }, strict=True)
 
+SQLite ``STRICT`` tables can use the ``ANY`` column type for values that should retain their exact SQLite storage class without coercion. Use the ``sqlite_utils.ANY`` marker type:
+
+.. code-block:: python
+
+    import sqlite_utils
+
+    db.table("events").create({
+        "id": int,
+        "payload": sqlite_utils.ANY,
+    }, pk="id", strict=True)
+
+An ``ANY`` column can store integers, floating point values, text, binary data or ``None``. In a ``STRICT`` table a text value such as ``"000123"`` remains text with its leading zeroes intact. SQLite also accepts ``ANY`` columns in ordinary non-``STRICT`` tables, but those columns apply numeric affinity and would store that same value as the integer ``123``.
+
 .. note::
     In the CLI: :ref:`sqlite-utils create-table <cli_create_table>`
 
@@ -1569,7 +1582,7 @@ You can specify the ``col_type`` argument either using a SQLite type as a string
 
 The ``col_type`` is optional - if you omit it the type of ``TEXT`` will be used.
 
-SQLite types you can specify are ``"TEXT"``, ``"INTEGER"``, ``"FLOAT"``, ``"REAL"`` or ``"BLOB"``.
+SQLite types you can specify are ``"TEXT"``, ``"INTEGER"``, ``"FLOAT"``, ``"REAL"``, ``"BLOB"`` or ``"ANY"``. You can use the ``sqlite_utils.ANY`` marker instead of the ``"ANY"`` string.
 
 If you pass a Python type, it will be mapped to SQLite types as shown here::
 
@@ -1582,6 +1595,7 @@ If you pass a Python type, it will be mapped to SQLite types as shown here::
     datetime.date: "TEXT"
     datetime.time: "TEXT"
     datetime.timedelta: "TEXT"
+    sqlite_utils.ANY: "ANY"
 
     # If numpy is installed
     np.int8: "INTEGER"
@@ -1831,6 +1845,8 @@ Pass ``strict=False`` to convert a strict table back to a regular non-strict tab
 
     table.transform(strict=False)
 
+If the table has ``ANY`` columns, converting it to non-strict mode can coerce text values that look numeric. For example, SQLite converts ``"000123"`` to the integer ``123`` when copying it into an ordinary ``ANY`` column. This is SQLite's documented distinction between `STRICT and ordinary ANY columns <https://www.sqlite.org/stricttables.html#the_any_datatype>`__.
+
 The default is ``strict=None``, which preserves the table's existing strict mode.
 
 Passing ``strict=True`` raises ``sqlite_utils.db.TransformError`` if the available SQLite version does not support strict tables.
@@ -2458,6 +2474,11 @@ The ``.columns_dict`` property returns a dictionary version of the columns with
     >>> db.table("PlantType").columns_dict
     {'id': <class 'int'>, 'value': <class 'str'>}
 
+SQLite ``ANY`` columns are represented by the ``sqlite_utils.ANY`` marker type::
+
+    >>> db.table("events").columns_dict
+    {'id': <class 'int'>, 'payload': <class 'sqlite_utils.utils.ANY'>}
+
 .. _python_api_introspection_default_values:
 
 .default_values