datasette/docs/settings.rst

.. _settings:

Settings
========

Using \-\-setting
-----------------

Datasette supports a number of settings. These can be set using the ``--setting name value`` option to ``datasette serve``.

You can set multiple settings at once like this::

    datasette mydatabase.db \
      --setting default_page_size 50 \
      --setting sql_time_limit_ms 3500 \
      --setting max_returned_rows 2000

Settings can also be specified :ref:`in the database.yaml configuration file <configuration_reference_settings>`.

.. _config_dir:

Configuration directory mode
----------------------------

Normally you configure Datasette using command-line options. For a Datasette instance with custom templates, custom plugins, a static directory and several databases this can get quite verbose::

    datasette one.db two.db \
      --metadata=metadata.json \
      --template-dir=templates/ \
      --plugins-dir=plugins \
      --static css:css

As an alternative to this, you can run Datasette in *configuration directory* mode. Create a directory with the following structure::

    # In a directory called my-app:
    my-app/one.db
    my-app/two.db
    my-app/datasette.yaml
    my-app/metadata.json
    my-app/templates/index.html
    my-app/plugins/my_plugin.py
    my-app/static/my.css

Now start Datasette by providing the path to that directory::

    datasette my-app/

Datasette will detect the files in that directory and automatically configure itself using them. It will serve all ``*.db`` files that it finds, will load ``metadata.json`` if it exists, and will load the ``templates``, ``plugins`` and ``static`` folders if they are present.

The files that can be included in this directory are as follows. All are optional.

* ``*.db`` (or ``*.sqlite3`` or ``*.sqlite``) - SQLite database files that will be served by Datasette
* ``datasette.yaml`` - :ref:`configuration` for the Datasette instance
* ``metadata.json`` - :ref:`metadata` for those databases - ``metadata.yaml`` or ``metadata.yml`` can be used as well
* ``inspect-data.json`` - the result of running ``datasette inspect *.db --inspect-file=inspect-data.json`` from the configuration directory - any database files listed here will be treated as immutable, so they should not be changed while Datasette is running
* ``templates/`` - a directory containing :ref:`customization_custom_templates`
* ``plugins/`` - a directory containing plugins, see :ref:`writing_plugins_one_off`
* ``static/`` - a directory containing static files - these will be served from ``/static/filename.txt``, see :ref:`customization_static_files`
* ``scratch/`` - a directory for :ref:`scratch databases <datasette_scratch_databases>`, used instead of a temporary directory if it exists

Settings
--------

The following options can be set using ``--setting name value``, or by storing them in the ``settings.json`` file for use with :ref:`config_dir`.

.. _setting_default_allow_sql:

default_allow_sql
~~~~~~~~~~~~~~~~~

.. [[[cog
    from settings_doc import setting_default
    setting_default(cog, "default_allow_sql")
.. ]]]

Default: ``on``

.. [[[end]]]

Should users be able to execute arbitrary SQL queries by default?

Setting this to ``off`` causes permission checks for :ref:`actions_execute_sql` to fail by default.

This setting controls the ability to submit arbitrary SQL. It does not disable structured table-browsing features that use SQL generated by Datasette, such as sorting, column filters and :ref:`facets`. Use :ref:`setting_allow_facet` to control whether users can request facets.

::

    datasette mydatabase.db --setting default_allow_sql off

Another way to achieve this is to add ``"allow_sql": false`` to your ``datasette.yaml`` file, as described in :ref:`authentication_permissions_execute_sql`. This setting offers a more convenient way to do this.

.. _setting_default_page_size:

default_page_size
~~~~~~~~~~~~~~~~~

.. [[[cog
    setting_default(cog, "default_page_size")
.. ]]]

Default: ``100``

.. [[[end]]]

The default number of rows returned by the table page. You can over-ride this on a per-page basis using the ``?_size=80`` query string parameter, provided you do not specify a value higher than the ``max_returned_rows`` setting. You can set this default using ``--setting`` like so::

    datasette mydatabase.db --setting default_page_size 50

.. _setting_sql_time_limit_ms:

sql_time_limit_ms
~~~~~~~~~~~~~~~~~

.. [[[cog
    setting_default(cog, "sql_time_limit_ms")
.. ]]]

Default: ``1000``

.. [[[end]]]

Time limit for SQL queries, in milliseconds. If a query takes longer than this to run Datasette will terminate the query and return an error.

If this time limit is too short for you, you can customize it using the ``sql_time_limit_ms`` limit - for example, to increase it to 3.5 seconds::

    datasette mydatabase.db --setting sql_time_limit_ms 3500

You can optionally set a lower time limit for an individual query using the ``?_timelimit=100`` query string argument::

    /my-database/my-table?qSpecies=44&_timelimit=100

This would set the time limit to 100ms for that specific query. This feature is useful if you are working with databases of unknown size and complexity - a query that might make perfect sense for a smaller table could take too long to execute on a table with millions of rows. By setting custom time limits you can execute queries "optimistically" - e.g. give me an exact count of rows matching this query but only if it takes less than 100ms to calculate.

.. _setting_max_returned_rows:

max_returned_rows
~~~~~~~~~~~~~~~~~

.. [[[cog
    setting_default(cog, "max_returned_rows")
.. ]]]

Default: ``1000``

.. [[[end]]]

The maximum number of rows Datasette returns at a time. If you execute a query that exceeds this limit, Datasette will truncate the result set and include a warning. You can use OFFSET/LIMIT or other methods in your SQL to implement pagination if you need to return more rows.

You can increase or decrease this limit like so::

    datasette mydatabase.db --setting max_returned_rows 2000

.. _setting_max_insert_rows:

max_insert_rows
~~~~~~~~~~~~~~~

.. [[[cog
    setting_default(cog, "max_insert_rows")
.. ]]]

Default: ``100``

.. [[[end]]]

Maximum rows that can be inserted at a time using the bulk insert API, see :ref:`TableInsertView`.

You can increase or decrease this limit like so::

    datasette mydatabase.db --setting max_insert_rows 1000

.. _setting_max_post_body_bytes:

max_post_body_bytes
~~~~~~~~~~~~~~~~~~~

.. [[[cog
    setting_default(cog, "max_post_body_bytes")
.. ]]]

Default: ``2097152``

.. [[[end]]]

Maximum size in bytes for a POST body that Datasette reads fully into memory, such as JSON submitted to the :ref:`write API <json_api_write>`. Requests with larger bodies are rejected with an HTTP 413 error.

This limit exists to protect against memory exhaustion: unlike file uploads handled by ``request.form()``, which stream to disk, these bodies are held entirely in memory and parsing them as JSON can multiply their memory footprint several times over.

If you increase :ref:`setting_max_insert_rows` to support larger bulk inserts you may need to increase this limit as well::

    datasette mydatabase.db --setting max_post_body_bytes 10485760

Set it to 0 to disable the limit entirely::

    datasette mydatabase.db --setting max_post_body_bytes 0

.. _setting_num_sql_threads:

num_sql_threads
~~~~~~~~~~~~~~~

.. [[[cog
    setting_default(cog, "num_sql_threads")
.. ]]]

Default: ``3``

.. [[[end]]]

Maximum number of threads in the thread pool Datasette uses to execute SQLite queries.

::

    datasette mydatabase.db --setting num_sql_threads 10

Setting this to 0 turns off threaded SQL queries entirely - useful for environments that do not support threading such as `Pyodide <https://pyodide.org/>`__.

.. _setting_max_open_connections:

max_open_connections
~~~~~~~~~~~~~~~~~~~~

.. [[[cog
    setting_default(cog, "max_open_connections")
.. ]]]

Default: ``128``

.. [[[end]]]

Read queries borrow a connection from a pool for the duration of one query (or one :ref:`execute_fn() <database_execute_fn>` callback) and return it afterwards. This setting caps how many pooled read connections can be open at once, across all attached file databases. When a new connection is needed and the cap has been reached, the least recently used idle connection - which may belong to any database - is closed first.

This is what lets one Datasette instance serve hundreds or thousands of database files without holding a connection, and its file descriptors, open to every one of them.

Every SQL thread can hold one read connection at a time, so a cap smaller than the number of threads would close and reopen connections for nearly every query. Datasette therefore raises the cap it uses to at least four times :ref:`setting_num_sql_threads` (12 with the default of 3 threads). ``/-/settings.json`` shows the value you configured.

Set this to 0 for no limit, in which case idle connections are only closed by :ref:`setting_connection_idle_timeout_ms`.

::

    datasette *.db --setting max_open_connections 64

The cap does not cover write connections (one per database that is being written to, see :ref:`setting_connection_idle_timeout_ms`), the short-lived connections used by :ref:`database_execute_isolated_fn` and by schema checks, or in-memory databases, which use no file descriptors. The exception is the ``_memory`` database under ``--crossdb``: each of its connections ``ATTACH``\ es up to ten database files, so those connections count against the cap and are closed when idle like file connections. Adding or removing a database closes them too, so the next cross-database query sees the current set of databases.

Each pooled connection to a file in WAL mode uses three file descriptors (the database, its ``-wal`` and its ``-shm`` file); other databases use one. Under a small file descriptor limit such as ``ulimit -n 256``, a cap of 64 keeps pooled read connections below about 200 descriptors even if every database uses WAL, leaving room for write connections and Datasette itself.

.. _setting_connection_idle_timeout_ms:

connection_idle_timeout_ms
~~~~~~~~~~~~~~~~~~~~~~~~~~

.. [[[cog
    setting_default(cog, "connection_idle_timeout_ms")
.. ]]]

Default: ``30000``

.. [[[end]]]

Connections that have not been used for this many milliseconds are closed, so a database that stops receiving traffic stops holding file descriptors, memory and threads.

- **Read connections** in the pool are closed once they have been idle this long. The next read to that database opens a new connection.
- **Write threads**: each database that receives writes gets its own write thread, holding a single write connection, and writes to that database are run one at a time by that thread. A write thread that has had no work for this long closes its connection and exits. The next write to that database starts a new thread and opens a new connection.

Opening a new connection costs around a millisecond (more if :ref:`plugin_hook_prepare_connection` hooks or extensions such as SpatiaLite have to be loaded again), which is negligible for a database that has been idle for 30 seconds, so the default keeps busy databases warm and releases forgotten ones quickly.

Anything a function attaches to a connection itself - temporary tables, ``ATTACH`` statements, ``PRAGMA`` settings, functions registered with ``conn.create_function()`` - is lost when that connection is closed, and is never shared between pooled connections. Set that up in the :ref:`plugin_hook_prepare_connection` plugin hook instead, which runs once for every new connection.

In-memory databases are exempt, since closing their only write connection would discard their contents. With :ref:`setting_num_sql_threads` set to 0 there are no threads to time out, so the write connection stays open until Datasette shuts down; read connections are still closed when idle.

Set this to 0 to keep read connections open until they are evicted by :ref:`setting_max_open_connections`, and write threads running until Datasette shuts down::

    datasette mydatabase.db --setting connection_idle_timeout_ms 0

.. _setting_schema_watch_interval_ms:

schema_watch_interval_ms
~~~~~~~~~~~~~~~~~~~~~~~~

.. [[[cog
    setting_default(cog, "schema_watch_interval_ms")
.. ]]]

Default: ``1000``

.. [[[end]]]

Datasette keeps a catalog of every attached database's tables, columns, indexes and foreign keys in its :ref:`internal database <internals_internal>`. Each database has a *schema watch mode* that decides how that catalog is kept current:

``owned``
    Only Datasette changes this database's schema. After every write Datasette makes to it, Datasette checks whether the schema changed and, if it did, updates the catalog before the write call returns. Owned databases are never polled.
``external``
    Other processes may change the file too - for example ``sqlite-utils`` adding a table while Datasette is serving it. On top of the checks after Datasette's own writes, the file is polled every ``schema_watch_interval_ms`` milliseconds. Each check is a ``stat()`` of the database file and its ``-wal`` and ``-journal`` files; a database is only opened if those changed, and its catalog is only rebuilt if its schema changed. Databases whose file is replaced (for example by renaming another file over it) or deleted are noticed too: open connections to the old file are discarded and the catalog is updated.
``immutable``
    The catalog is built once at startup. Immutable databases (opened with ``-i``) always use this mode.

Set this to ``0`` to turn polling off. External databases are then only checked after Datasette's own writes to them::

    datasette mydatabase.db --setting schema_watch_interval_ms 0

Database files, and named in-memory databases, use the :ref:`setting_default_schema_watch` mode - including those added by plugins with :ref:`datasette.add_database() <datasette_add_database>`, unless the plugin passes ``schema_watch="owned"`` (or ``"external"``). A named in-memory database in ``external`` mode is checked with ``PRAGMA schema_version`` on its write connection every ``schema_watch_interval_ms``, which notices tables created through a plugin's own connection to it. You can set the mode for an individual database in ``datasette.yaml``, which overrides both:

.. code-block:: yaml

    databases:
      mydatabase:
        schema_watch: owned   # or external, or immutable

.. _setting_default_schema_watch:

default_schema_watch
~~~~~~~~~~~~~~~~~~~~

.. [[[cog
    setting_default(cog, "default_schema_watch")
.. ]]]

Default: ``external``

.. [[[end]]]

The schema watch mode, ``external`` or ``owned``, for mutable database files and named in-memory databases that do not have a mode of their own: files passed on the command line or with ``Datasette(files=...)``, and databases added with :ref:`datasette.add_database() <datasette_add_database>` without a ``schema_watch`` argument - see :ref:`setting_schema_watch_interval_ms`.

The default, ``external``, suits the common workflow of changing a database with another tool such as ``sqlite-utils`` while Datasette is serving it: new tables and columns show up within about a second. If Datasette is the only thing that ever changes your database files, set this to ``owned`` and they will not be polled. This is worth doing for instances with thousands of databases, where polling costs a few milliseconds of CPU per second::

    datasette *.db --setting default_schema_watch owned


.. _setting_allow_facet:

allow_facet
~~~~~~~~~~~

.. [[[cog
    setting_default(cog, "allow_facet")
.. ]]]

Default: ``on``

.. [[[end]]]

Allow users to specify columns they would like to facet on using the ``?_facet=COLNAME`` URL parameter to the table view.

If disabled, facets will still be displayed if they have been specifically enabled in ``metadata.json`` configuration for the table.

Here's how to disable this feature::

    datasette mydatabase.db --setting allow_facet off

.. _setting_default_facet_size:

default_facet_size
~~~~~~~~~~~~~~~~~~

.. [[[cog
    setting_default(cog, "default_facet_size")
.. ]]]

Default: ``30``

.. [[[end]]]

The default number of unique rows returned by :ref:`facets`. You can customize it like this::

    datasette mydatabase.db --setting default_facet_size 50

.. _setting_facet_time_limit_ms:

facet_time_limit_ms
~~~~~~~~~~~~~~~~~~~

.. [[[cog
    setting_default(cog, "facet_time_limit_ms")
.. ]]]

Default: ``200``

.. [[[end]]]

The time limit in milliseconds Datasette allows for calculating a facet. You can customize it like this::

    datasette mydatabase.db --setting facet_time_limit_ms 1000

.. _setting_facet_suggest_time_limit_ms:

facet_suggest_time_limit_ms
~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. [[[cog
    setting_default(cog, "facet_suggest_time_limit_ms")
.. ]]]

Default: ``50``

.. [[[end]]]

When Datasette calculates suggested facets it needs to run a SQL query for every column in your table. This time limit, in milliseconds, applies separately to each query. If the time limit is exceeded the column will not be suggested as a facet.

You can increase this time limit like so::

    datasette mydatabase.db --setting facet_suggest_time_limit_ms 500

.. _setting_suggest_facets:

suggest_facets
~~~~~~~~~~~~~~

.. [[[cog
    setting_default(cog, "suggest_facets")
.. ]]]

Default: ``on``

.. [[[end]]]

Should Datasette calculate suggested facets? Turn this off like so::

    datasette mydatabase.db --setting suggest_facets off

.. _setting_allow_download:

allow_download
~~~~~~~~~~~~~~

.. [[[cog
    setting_default(cog, "allow_download")
.. ]]]

Default: ``on``

.. [[[end]]]

Should users be able to download the original SQLite database using a link on the database index page? Databases can only be downloaded if they are served in immutable mode and not in-memory. If downloading is unavailable for either of these reasons, the download link is hidden even if ``allow_download`` is on. To disable database downloads, use the following::

    datasette mydatabase.db --setting allow_download off

.. _setting_allow_signed_tokens:

allow_signed_tokens
~~~~~~~~~~~~~~~~~~~

.. [[[cog
    setting_default(cog, "allow_signed_tokens")
.. ]]]

Default: ``on``

.. [[[end]]]

Should users be able to create signed API tokens to access Datasette?

Use the following to turn it off::

    datasette mydatabase.db --setting allow_signed_tokens off

Turning this setting off will disable the ``/-/create-token`` page, :ref:`described here <CreateTokenView>`. It will also cause any incoming ``Authorization: Bearer dstok_...`` API tokens to be ignored.

.. _setting_max_signed_tokens_ttl:

max_signed_tokens_ttl
~~~~~~~~~~~~~~~~~~~~~

.. [[[cog
    setting_default(cog, "max_signed_tokens_ttl")
.. ]]]

Default: ``0``

.. [[[end]]]

Maximum allowed expiry time for signed API tokens created by users.

A value of ``0`` means no limit - tokens can be created that will never expire.

Set this to a value in seconds to limit the maximum expiry time. For example, to set that limit to 24 hours you would use::

    datasette mydatabase.db --setting max_signed_tokens_ttl 86400

This setting is enforced when incoming tokens are processed.

.. _setting_default_cache_ttl:

default_cache_ttl
~~~~~~~~~~~~~~~~~

.. [[[cog
    setting_default(cog, "default_cache_ttl")
.. ]]]

Default: ``5``

.. [[[end]]]

Default HTTP caching max-age header in seconds, used for ``Cache-Control: max-age=X``. Can be over-ridden on a per-request basis using the ``?_ttl=`` query string parameter. Set this to ``0`` to disable HTTP caching entirely.

::

    datasette mydatabase.db --setting default_cache_ttl 60

Dynamic responses for authenticated actors, requests with cookies or an ``Authorization`` header, and responses that set cookies use ``Cache-Control: private, no-store``. This takes precedence over ``default_cache_ttl`` and ``?_ttl=``, even when cache headers are otherwise disabled. Anonymous dynamic responses vary by ``Cookie`` and ``Authorization``. Static assets retain their own cache policy.

.. _setting_cache_size_kb:

cache_size_kb
~~~~~~~~~~~~~

.. [[[cog
    setting_default(cog, "cache_size_kb")
.. ]]]

Default: ``0``

.. [[[end]]]

Sets the amount of memory SQLite uses for its `per-connection cache <https://www.sqlite.org/pragma.html#pragma_cache_size>`_, in KB. Set this to ``0`` to use SQLite's default cache size.

::

    datasette mydatabase.db --setting cache_size_kb 5000

.. _setting_allow_csv_stream:

allow_csv_stream
~~~~~~~~~~~~~~~~

.. [[[cog
    setting_default(cog, "allow_csv_stream")
.. ]]]

Default: ``on``

.. [[[end]]]

Enables :ref:`the CSV export feature <csv_export>` where an entire table
(potentially hundreds of thousands of rows) can be exported as a single CSV
file. You can turn it off like this:

::

    datasette mydatabase.db --setting allow_csv_stream off

.. _setting_max_csv_mb:

max_csv_mb
~~~~~~~~~~

.. [[[cog
    setting_default(cog, "max_csv_mb")
.. ]]]

Default: ``100``

.. [[[end]]]

The maximum size of CSV that can be exported, in megabytes.
You can disable the limit entirely by setting this to 0:

::

    datasette mydatabase.db --setting max_csv_mb 0

.. _setting_truncate_cells_html:

truncate_cells_html
~~~~~~~~~~~~~~~~~~~

.. [[[cog
    setting_default(cog, "truncate_cells_html")
.. ]]]

Default: ``2048``

.. [[[end]]]

In the HTML table view, truncate any strings that are longer than this value.
The full value will still be available in CSV, JSON and on the individual row
HTML page. Set this to 0 to disable truncation.

::

    datasette mydatabase.db --setting truncate_cells_html 0

.. _setting_force_https_urls:

force_https_urls
~~~~~~~~~~~~~~~~

.. [[[cog
    setting_default(cog, "force_https_urls")
.. ]]]

Default: ``off``

.. [[[end]]]

Forces self-referential URLs in the JSON output to always use the ``https://``
protocol. This is useful for cases where the application itself is hosted using
HTTP but is served to the outside world via a proxy that enables HTTPS.

::

    datasette mydatabase.db --setting force_https_urls 1

.. _setting_template_debug:

template_debug
~~~~~~~~~~~~~~

.. [[[cog
    setting_default(cog, "template_debug")
.. ]]]

Default: ``off``

.. [[[end]]]

This setting enables template context debug mode, which is useful to help understand what variables are available to custom templates when you are writing them.

Enable it like this::

    datasette mydatabase.db --setting template_debug 1

Now you can add ``?_context=1`` or ``&_context=1`` to any Datasette page to see the context that was passed to that template.

Some examples:

* https://latest.datasette.io/?_context=1
* https://latest.datasette.io/fixtures?_context=1
* https://latest.datasette.io/fixtures/roadside_attractions?_context=1

.. _setting_trace_debug:

trace_debug
~~~~~~~~~~~

.. [[[cog
    setting_default(cog, "trace_debug")
.. ]]]

Default: ``off``

.. [[[end]]]

This setting enables appending ``?_trace=1`` to any page in order to see the SQL queries and other trace information that was used to generate that page.

Enable it like this::

    datasette mydatabase.db --setting trace_debug 1

Some examples:

* https://latest.datasette.io/?_trace=1
* https://latest.datasette.io/fixtures/roadside_attractions?_trace=1

See :ref:`internals_tracer` for details on how to hook into this mechanism as a plugin author.

.. _setting_base_url:

base_url
~~~~~~~~

.. [[[cog
    setting_default(cog, "base_url")
.. ]]]

Default: ``/``

.. [[[end]]]

If you are running Datasette behind a proxy, it may be useful to change the root path used for the Datasette instance.

For example, if you are sending traffic from ``https://www.example.com/tools/datasette/`` through to a proxied Datasette instance you may wish Datasette to use ``/tools/datasette/`` as its root URL.

You can do that like so::

    datasette mydatabase.db --setting base_url /tools/datasette/

.. _setting_secret:

Configuring the secret
----------------------

Datasette uses a secret string to sign secure values such as cookies.

If you do not provide a secret, Datasette will create one when it starts up. This secret will reset every time the Datasette server restarts though, so things like authentication cookies and :ref:`API tokens <CreateTokenView>` will not stay valid between restarts.

You can pass a secret to Datasette in two ways: with the ``--secret`` command-line option or by setting a ``DATASETTE_SECRET`` environment variable.

::

    datasette mydb.db --secret=SECRET_VALUE_HERE

Or::

    export DATASETTE_SECRET=SECRET_VALUE_HERE
    datasette mydb.db

One way to generate a secure random secret is to use Python like this::

    python3 -c 'import secrets; print(secrets.token_hex(32))'
    cdb19e94283a20f9d42cca50c5a4871c0aa07392db308755d60a1a5b9bb0fa52

Plugin authors can make use of this signing mechanism in their plugins using the :ref:`datasette.sign() <datasette_sign>` and :ref:`datasette.unsign() <datasette_unsign>` methods.

.. _setting_publish_secrets:

Using secrets with datasette publish
------------------------------------

The :ref:`cli_publish` and :ref:`cli_package` commands both generate a secret for you automatically when Datasette is deployed.

This means that every time you deploy a new version of a Datasette project, a new secret will be generated. This will cause signed cookies to become invalid on every fresh deploy.

You can fix this by creating a secret that will be used for multiple deploys and passing it using the ``--secret`` option::

    datasette publish cloudrun mydb.db --service=my-service --secret=cdb19e94283a20f9d42cca5