From 0b6aba696dd07814d97f563de6ad1d5daab01fd9 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 14 Jun 2022 14:31:45 -0700 Subject: [PATCH] Documentation for maximize_csv_field_size_limit, closes #442 --- docs/python-api.rst | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/docs/python-api.rst b/docs/python-api.rst index 9fe6d67..ca98317 100644 --- a/docs/python-api.rst +++ b/docs/python-api.rst @@ -2549,6 +2549,37 @@ The ``sqlite_utils.utils.rows_from_file()`` helper function can read rows (a seq .. autofunction:: sqlite_utils.utils.rows_from_file :noindex: +.. _python_api_maximize_csv_field_size_limit: + +Setting the maximum CSV field size limit +======================================== + +Sometimes when working with CSV files that include extremely long fields you may see an error that looks like this:: + + _csv.Error: field larger than field limit (131072) + +The Python standard library ``csv`` module enforces a field size limit. You can increase that limit using the ``csv.field_size_limit(new_limit)`` method (`documented here `__) but if you don't want to pick a new level you may instead want to increase it to the maximum possible. + +The maximum possible value for this is not documented, and varies between systems. + +Calling ``sqlite_utils.utils.maximize_csv_field_size_limit()`` will set the value to the highest possible for the current system: + +.. code-block:: python + + from sqlite_utils.utils import maximize_csv_field_size_limit + + maximize_csv_field_size_limit() + + +If you need to reset to the original value after calling this function you can do so like this: + +.. code-block:: python + + from sqlite_utils.utils import ORIGINAL_CSV_FIELD_SIZE_LIMIT + import csv + + csv.field_size_limit(ORIGINAL_CSV_FIELD_SIZE_LIMIT) + .. _python_api_gis: SpatiaLite helpers