Analyze tables options: --common-limit, --no-most, --no-least

Closes #544
This commit is contained in:
Simon Willison 2023-05-21 09:19:30 -07:00 committed by GitHub
commit d2a7b15b2b
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
6 changed files with 178 additions and 40 deletions

View file

@ -564,10 +564,13 @@ See :ref:`cli_analyze_tables`.
sqlite-utils analyze-tables data.db trees
Options:
-c, --column TEXT Specific columns to analyze
--save Save results to _analyze_tables table
--load-extension TEXT Path to SQLite extension, with optional :entrypoint
-h, --help Show this message and exit.
-c, --column TEXT Specific columns to analyze
--save Save results to _analyze_tables table
--common-limit INTEGER How many common values
--no-most Skip most common values
--no-least Skip least common values
--load-extension TEXT Path to SQLite extension, with optional :entrypoint
-h, --help Show this message and exit.
.. _cli_ref_convert:

View file

@ -730,11 +730,15 @@ For each column this tool displays the number of null rows, the number of blank
If you do not specify any tables every table in the database will be analyzed::
$ sqlite-utils analyze-tables github.db
sqlite-utils analyze-tables github.db
If you wish to analyze one or more specific columns, use the ``-c`` option::
$ sqlite-utils analyze-tables github.db tags -c sha
sqlite-utils analyze-tables github.db tags -c sha
To show more than 10 common values, use ``--common-limit 20``. To skip the most common or least common value analysis, use ``--no-most`` or ``--no-least``::
sqlite-utils analyze-tables github.db tags --common-limit 20 --no-least
.. _cli_analyze_tables_save:
@ -743,7 +747,7 @@ Saving the analyzed table details
``analyze-tables`` can take quite a while to run for large database files. You can save the results of the analysis to a database table called ``_analyze_tables_`` using the ``--save`` option::
$ sqlite-utils analyze-tables github.db --save
sqlite-utils analyze-tables github.db --save
The ``_analyze_tables_`` table has the following schema::

View file

@ -1115,7 +1115,26 @@ You can inspect the database to see the results like this::
Analyzing a column
==================
The ``table.analyze_column(column, common_limit=10, value_truncate=None)`` method is used by the :ref:`analyze-tables <cli_analyze_tables>` CLI command. It returns a ``ColumnDetails`` named tuple with the following fields:
The ``table.analyze_column(column)`` method is used by the :ref:`analyze-tables <cli_analyze_tables>` CLI command.
It takes the following arguments and options:
``column`` - required
The name of the column to analyze
``common_limit``
The number of most common values to return. Defaults to 10.
``value_truncate``
If set to an integer, values longer than this will be truncated to this length. Defaults to None.
``most_common``
If set to False, the ``most_common`` field of the returned ``ColumnDetails`` will be set to None. Defaults to True.
``least_common``
If set to False, the ``least_common`` field of the returned ``ColumnDetails`` will be set to None. Defaults to True.
And returns a ``ColumnDetails`` named tuple with the following fields:
``table``
The name of the table
@ -1141,10 +1160,6 @@ The ``table.analyze_column(column, common_limit=10, value_truncate=None)`` metho
``least_common``
The ``N`` least common values as a list of ``(value, count)`` tuples`, or ``None`` if the table is entirely distinct or if the number of distinct values is less than N (since they will already have been returned in ``most_common``)
``N`` defaults to 10, or you can pass a custom ``N`` using the ``common_limit`` parameter.
You can use the ``value_truncate`` parameter to truncate values in the ``most_common`` and ``least_common`` lists to a specified number of characters.
.. _python_api_add_column:
Adding columns