--limit= mechanism plus new limits for facets

Replaced the --max_returned_rows and --sql_time_limit_ms options to
"datasette serve" with a new --limit option, which supports a larger
list of limits.

Example usage:

	datasette serve --limit max_returned_rows:1000 \
		--limit sql_time_limit_ms:2500 \
		--limit default_facet_size:50 \
		--limit facet_time_limit_ms:1000 \
		--limit facet_suggest_time_limit_ms:500

New docs: https://datasette.readthedocs.io/en/latest/limits.html

Closes #270
Closes #264
This commit is contained in:
Simon Willison 2018-05-17 22:08:26 -07:00
commit cef9a9a870
No known key found for this signature in database
GPG key ID: 17E2DEA2588B7F52
10 changed files with 118 additions and 72 deletions

View file

@ -101,9 +101,6 @@ datasette serve options
--cors Enable CORS by serving Access-Control-Allow-
Origin: *
--page_size INTEGER Page size - default is 100
--max_returned_rows INTEGER Max allowed rows to return at once - default is
1000. Set to 0 to disable check entirely.
--sql_time_limit_ms INTEGER Max time allowed for SQL queries in ms
--load-extension PATH Path to a SQLite extension to load
--inspect-file TEXT Path to JSON file created using "datasette
inspect"
@ -113,4 +110,6 @@ datasette serve options
--plugins-dir DIRECTORY Path to directory containing custom plugins
--static STATIC MOUNT mountpoint:path-to-directory for serving static
files
--limit LIMIT Set a limit using limitname:integer
datasette.readthedocs.io/en/latest/limits.html
--help Show this message and exit.

View file

@ -22,6 +22,7 @@ Contents
facets
full_text_search
metadata
limits
custom_templates
plugins
changelog

View file

@ -132,7 +132,7 @@ Special table arguments
The Datasette table view takes a number of special querystring arguments:
``?_size=1000`` or ``?_size=max``
Sets a custom page size. This cannot exceed the ``max_returned_rows`` option
Sets a custom page size. This cannot exceed the ``max_returned_rows`` limit
passed to ``datasette serve``. Use ``max`` to get ``max_returned_rows``.
``?_sort=COLUMN``

51
docs/limits.rst Normal file
View file

@ -0,0 +1,51 @@
Limits
======
To prevent rogue, long-running queries from making a Datasette instance inaccessible to other users, Datasette imposes some limits on the SQL that you can execute.
sql_time_limit_ms
-----------------
By default, queries have a time limit of one second. If a query takes longer than this to run Datasette will terminate the query and return an error.
If this time limit is too short for you, you can customize it using the ``sql_time_limit_ms`` limit - for example, to increase it to 3.5 seconds::
datasette mydatabase.db --limit sql_time_limit_ms:3500
You can optionally set a lower time limit for an individual query using the ``_timelimit`` query string argument::
/my-database/my-table?qSpecies=44&_timelimit=100
This would set the time limit to 100ms for that specific query. This feature is useful if you are working with databases of unknown size and complexity - a query that might make perfect sense for a smaller table could take too long to execute on a table with millions of rows. By setting custom time limits you can execute queries "optimistically" - e.g. give me an exact count of rows matching this query but only if it takes less than 100ms to calculate.
max_returned_rows
-----------------
Datasette returns a maximum of 1,000 rows of data at a time. If you execute a query that returns more than 1,000 rows, Datasette will return the first 1,000 and include a warning that the result set has been truncated. You can use OFFSET/LIMIT or other methods in your SQL to implement pagination if you need to return more than 1,000 rows.
You can increase or decrease this limit like so::
datasette mydatabase.db --limit max_returned_rows:2000
default_facet_size
------------------
The default number of unique rows returned by :ref:`facets` is 30. You can customize it like this::
datasette mydatabase.db --limit default_facet_size:50
facet_time_limit_ms
-------------------
This is the time limit Datasette allows for calculating a facet, which defaults to 200ms::
datasette mydatabase.db --limit facet_time_limit_ms:1000
facet_suggest_time_limit_ms
---------------------------
When Datasette calculates suggested facets it needs to run a SQL query for every column in your table. The default for this time limit is 50ms to account for the fact that it needs to run once for every column. If the time limit is exceeded the column will not be suggested as a facet.
You can increase this time limit like so::
datasette mydatabase.db --limit facet_suggest_time_limit_ms:500

View file

@ -46,39 +46,6 @@ statements can be used to change database settings at runtime. If you need to
include the string "pragma" in a query you can do so safely using a named
parameter.
Query limits
------------
To prevent rogue, long-running queries from making a Datasette instance
inaccessible to other users, Datasette imposes some limits on the SQL that you
can execute.
By default, queries have a time limit of one second. If a query takes longer
than this to run Datasette will terminate the query and return an error.
If this time limit is too short for you, you can customize it using the
``sql_time_limit_ms`` option - for example, to increase it to 3.5 seconds::
datasette mydatabase.db --sql_time_limit_ms=3500
You can optionally set a lower time limit for an individual query using the
``_timelimit`` query string argument::
/my-database/my-table?qSpecies=44&_timelimit=100
This would set the time limit to 100ms for that specific query. This feature
is useful if you are working with databases of unknown size and complexity -
a query that might make perfect sense for a smaller table could take too long
to execute on a table with millions of rows. By setting custom time limits you
can execute queries "optimistically" - e.g. give me an exact count of rows
matching this query but only if it takes less than 100ms to calculate.
Datasette returns a maximum of 1,000 rows of data at a time. If you execute a
query that returns more than 1,000 rows, Datasette will return the first 1,000
and include a warning that the result set has been truncated. You can use
OFFSET/LIMIT or other methods in your SQL to implement pagination if you need to
return more than 1,000 rows.
Views
-----