From c5791156d92615f25696ba93dae5bb2dcc192c98 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 7 Mar 2022 14:04:10 -0800 Subject: [PATCH 001/952] Code of conduct, refs #1654 --- CODE_OF_CONDUCT.md | 128 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 128 insertions(+) create mode 100644 CODE_OF_CONDUCT.md diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 00000000..14d4c567 --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,128 @@ +# Contributor Covenant Code of Conduct + +## Our Pledge + +We as members, contributors, and leaders pledge to make participation in our +community a harassment-free experience for everyone, regardless of age, body +size, visible or invisible disability, ethnicity, sex characteristics, gender +identity and expression, level of experience, education, socio-economic status, +nationality, personal appearance, race, religion, or sexual identity +and orientation. + +We pledge to act and interact in ways that contribute to an open, welcoming, +diverse, inclusive, and healthy community. + +## Our Standards + +Examples of behavior that contributes to a positive environment for our +community include: + +* Demonstrating empathy and kindness toward other people +* Being respectful of differing opinions, viewpoints, and experiences +* Giving and gracefully accepting constructive feedback +* Accepting responsibility and apologizing to those affected by our mistakes, + and learning from the experience +* Focusing on what is best not just for us as individuals, but for the + overall community + +Examples of unacceptable behavior include: + +* The use of sexualized language or imagery, and sexual attention or + advances of any kind +* Trolling, insulting or derogatory comments, and personal or political attacks +* Public or private harassment +* Publishing others' private information, such as a physical or email + address, without their explicit permission +* Other conduct which could reasonably be considered inappropriate in a + professional setting + +## Enforcement Responsibilities + +Community leaders are responsible for clarifying and enforcing our standards of +acceptable behavior and will take appropriate and fair corrective action in +response to any behavior that they deem inappropriate, threatening, offensive, +or harmful. + +Community leaders have the right and responsibility to remove, edit, or reject +comments, commits, code, wiki edits, issues, and other contributions that are +not aligned to this Code of Conduct, and will communicate reasons for moderation +decisions when appropriate. + +## Scope + +This Code of Conduct applies within all community spaces, and also applies when +an individual is officially representing the community in public spaces. +Examples of representing our community include using an official e-mail address, +posting via an official social media account, or acting as an appointed +representative at an online or offline event. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported to the community leaders responsible for enforcement at +`swillison+datasette-code-of-conduct@gmail.com`. +All complaints will be reviewed and investigated promptly and fairly. + +All community leaders are obligated to respect the privacy and security of the +reporter of any incident. + +## Enforcement Guidelines + +Community leaders will follow these Community Impact Guidelines in determining +the consequences for any action they deem in violation of this Code of Conduct: + +### 1. Correction + +**Community Impact**: Use of inappropriate language or other behavior deemed +unprofessional or unwelcome in the community. + +**Consequence**: A private, written warning from community leaders, providing +clarity around the nature of the violation and an explanation of why the +behavior was inappropriate. A public apology may be requested. + +### 2. Warning + +**Community Impact**: A violation through a single incident or series +of actions. + +**Consequence**: A warning with consequences for continued behavior. No +interaction with the people involved, including unsolicited interaction with +those enforcing the Code of Conduct, for a specified period of time. This +includes avoiding interactions in community spaces as well as external channels +like social media. Violating these terms may lead to a temporary or +permanent ban. + +### 3. Temporary Ban + +**Community Impact**: A serious violation of community standards, including +sustained inappropriate behavior. + +**Consequence**: A temporary ban from any sort of interaction or public +communication with the community for a specified period of time. No public or +private interaction with the people involved, including unsolicited interaction +with those enforcing the Code of Conduct, is allowed during this period. +Violating these terms may lead to a permanent ban. + +### 4. Permanent Ban + +**Community Impact**: Demonstrating a pattern of violation of community +standards, including sustained inappropriate behavior, harassment of an +individual, or aggression toward or disparagement of classes of individuals. + +**Consequence**: A permanent ban from any sort of public interaction within +the community. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], +version 2.0, available at +https://www.contributor-covenant.org/version/2/0/code_of_conduct.html. + +Community Impact Guidelines were inspired by [Mozilla's code of conduct +enforcement ladder](https://github.com/mozilla/diversity). + +[homepage]: https://www.contributor-covenant.org + +For answers to common questions about this code of conduct, see the FAQ at +https://www.contributor-covenant.org/faq. Translations are available at +https://www.contributor-covenant.org/translations. From 239aed182053903ed69108776b6864d42bfe1eb4 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 15 Mar 2022 08:36:35 -0700 Subject: [PATCH 002/952] Revert "Code of conduct, refs #1654" This reverts commit c5791156d92615f25696ba93dae5bb2dcc192c98. Refs #1658 --- CODE_OF_CONDUCT.md | 128 --------------------------------------------- 1 file changed, 128 deletions(-) delete mode 100644 CODE_OF_CONDUCT.md diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md deleted file mode 100644 index 14d4c567..00000000 --- a/CODE_OF_CONDUCT.md +++ /dev/null @@ -1,128 +0,0 @@ -# Contributor Covenant Code of Conduct - -## Our Pledge - -We as members, contributors, and leaders pledge to make participation in our -community a harassment-free experience for everyone, regardless of age, body -size, visible or invisible disability, ethnicity, sex characteristics, gender -identity and expression, level of experience, education, socio-economic status, -nationality, personal appearance, race, religion, or sexual identity -and orientation. - -We pledge to act and interact in ways that contribute to an open, welcoming, -diverse, inclusive, and healthy community. - -## Our Standards - -Examples of behavior that contributes to a positive environment for our -community include: - -* Demonstrating empathy and kindness toward other people -* Being respectful of differing opinions, viewpoints, and experiences -* Giving and gracefully accepting constructive feedback -* Accepting responsibility and apologizing to those affected by our mistakes, - and learning from the experience -* Focusing on what is best not just for us as individuals, but for the - overall community - -Examples of unacceptable behavior include: - -* The use of sexualized language or imagery, and sexual attention or - advances of any kind -* Trolling, insulting or derogatory comments, and personal or political attacks -* Public or private harassment -* Publishing others' private information, such as a physical or email - address, without their explicit permission -* Other conduct which could reasonably be considered inappropriate in a - professional setting - -## Enforcement Responsibilities - -Community leaders are responsible for clarifying and enforcing our standards of -acceptable behavior and will take appropriate and fair corrective action in -response to any behavior that they deem inappropriate, threatening, offensive, -or harmful. - -Community leaders have the right and responsibility to remove, edit, or reject -comments, commits, code, wiki edits, issues, and other contributions that are -not aligned to this Code of Conduct, and will communicate reasons for moderation -decisions when appropriate. - -## Scope - -This Code of Conduct applies within all community spaces, and also applies when -an individual is officially representing the community in public spaces. -Examples of representing our community include using an official e-mail address, -posting via an official social media account, or acting as an appointed -representative at an online or offline event. - -## Enforcement - -Instances of abusive, harassing, or otherwise unacceptable behavior may be -reported to the community leaders responsible for enforcement at -`swillison+datasette-code-of-conduct@gmail.com`. -All complaints will be reviewed and investigated promptly and fairly. - -All community leaders are obligated to respect the privacy and security of the -reporter of any incident. - -## Enforcement Guidelines - -Community leaders will follow these Community Impact Guidelines in determining -the consequences for any action they deem in violation of this Code of Conduct: - -### 1. Correction - -**Community Impact**: Use of inappropriate language or other behavior deemed -unprofessional or unwelcome in the community. - -**Consequence**: A private, written warning from community leaders, providing -clarity around the nature of the violation and an explanation of why the -behavior was inappropriate. A public apology may be requested. - -### 2. Warning - -**Community Impact**: A violation through a single incident or series -of actions. - -**Consequence**: A warning with consequences for continued behavior. No -interaction with the people involved, including unsolicited interaction with -those enforcing the Code of Conduct, for a specified period of time. This -includes avoiding interactions in community spaces as well as external channels -like social media. Violating these terms may lead to a temporary or -permanent ban. - -### 3. Temporary Ban - -**Community Impact**: A serious violation of community standards, including -sustained inappropriate behavior. - -**Consequence**: A temporary ban from any sort of interaction or public -communication with the community for a specified period of time. No public or -private interaction with the people involved, including unsolicited interaction -with those enforcing the Code of Conduct, is allowed during this period. -Violating these terms may lead to a permanent ban. - -### 4. Permanent Ban - -**Community Impact**: Demonstrating a pattern of violation of community -standards, including sustained inappropriate behavior, harassment of an -individual, or aggression toward or disparagement of classes of individuals. - -**Consequence**: A permanent ban from any sort of public interaction within -the community. - -## Attribution - -This Code of Conduct is adapted from the [Contributor Covenant][homepage], -version 2.0, available at -https://www.contributor-covenant.org/version/2/0/code_of_conduct.html. - -Community Impact Guidelines were inspired by [Mozilla's code of conduct -enforcement ladder](https://github.com/mozilla/diversity). - -[homepage]: https://www.contributor-covenant.org - -For answers to common questions about this code of conduct, see the FAQ at -https://www.contributor-covenant.org/faq. Translations are available at -https://www.contributor-covenant.org/translations. From 5a353a32b9c4d75acbe3193fd72f735a8e78516a Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 15 Mar 2022 08:37:14 -0700 Subject: [PATCH 003/952] Revert "Fixed tests for urlsafe_components, refs #1650" This reverts commit bb499942c15c4e2cfa4b6afab8f8debe5948c009. Refs #1658 --- tests/test_utils.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/test_utils.py b/tests/test_utils.py index ff4f649a..1c3ab495 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -19,8 +19,8 @@ from unittest.mock import patch ("foo", ["foo"]), ("foo,bar", ["foo", "bar"]), ("123,433,112", ["123", "433", "112"]), - ("123-2C433,112", ["123,433", "112"]), - ("123-2F433-2F112", ["123/433/112"]), + ("123%2C433,112", ["123,433", "112"]), + ("123%2F433%2F112", ["123/433/112"]), ], ) def test_urlsafe_components(path, expected): From 77e718c3ffb30473759a8b1ed347f73cb2ff5cfe Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 15 Mar 2022 08:37:31 -0700 Subject: [PATCH 004/952] Revert "Fix bug with percentage redirects, close #1650" This reverts commit c85d669de387b40e667fd6942c6cc1c15b4f5964. Refs #1658 --- datasette/utils/__init__.py | 7 +------ tests/test_html.py | 4 ---- 2 files changed, 1 insertion(+), 10 deletions(-) diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index e7c9fb1c..79feeef6 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -10,7 +10,6 @@ import markupsafe import mergedeep import os import re -import secrets import shlex import tempfile import typing @@ -1173,8 +1172,4 @@ def dash_encode(s: str) -> str: @documented def dash_decode(s: str) -> str: "Decodes a dash-encoded string, so ``-2Ffoo-2Fbar`` -> ``/foo/bar``" - # Avoid accidentally decoding a %2f style sequence - temp = secrets.token_hex(16) - s = s.replace("%", temp) - decoded = urllib.parse.unquote(s.replace("-", "%")) - return decoded.replace(temp, "%") + return urllib.parse.unquote(s.replace("-", "%")) diff --git a/tests/test_html.py b/tests/test_html.py index 55d78c05..de703284 100644 --- a/tests/test_html.py +++ b/tests/test_html.py @@ -961,10 +961,6 @@ def test_no_alternate_url_json(app_client, path): "/fivethirtyeight/twitter-ratio%2Fsenators", "/fivethirtyeight/twitter-2Dratio-2Fsenators", ), - ( - "/fixtures/table%2Fwith%2Fslashes", - "/fixtures/table-2Fwith-2Fslashes", - ), # query string should be preserved ("/foo/bar%2Fbaz?id=5", "/foo/bar-2Fbaz?id=5"), ), From 645381a5ed23c016281e8c6c7d141518f91b67e5 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 15 Mar 2022 08:36:35 -0700 Subject: [PATCH 005/952] Add code of conduct again Refs #1658 --- CODE_OF_CONDUCT.md | 128 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 128 insertions(+) create mode 100644 CODE_OF_CONDUCT.md diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 00000000..14d4c567 --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,128 @@ +# Contributor Covenant Code of Conduct + +## Our Pledge + +We as members, contributors, and leaders pledge to make participation in our +community a harassment-free experience for everyone, regardless of age, body +size, visible or invisible disability, ethnicity, sex characteristics, gender +identity and expression, level of experience, education, socio-economic status, +nationality, personal appearance, race, religion, or sexual identity +and orientation. + +We pledge to act and interact in ways that contribute to an open, welcoming, +diverse, inclusive, and healthy community. + +## Our Standards + +Examples of behavior that contributes to a positive environment for our +community include: + +* Demonstrating empathy and kindness toward other people +* Being respectful of differing opinions, viewpoints, and experiences +* Giving and gracefully accepting constructive feedback +* Accepting responsibility and apologizing to those affected by our mistakes, + and learning from the experience +* Focusing on what is best not just for us as individuals, but for the + overall community + +Examples of unacceptable behavior include: + +* The use of sexualized language or imagery, and sexual attention or + advances of any kind +* Trolling, insulting or derogatory comments, and personal or political attacks +* Public or private harassment +* Publishing others' private information, such as a physical or email + address, without their explicit permission +* Other conduct which could reasonably be considered inappropriate in a + professional setting + +## Enforcement Responsibilities + +Community leaders are responsible for clarifying and enforcing our standards of +acceptable behavior and will take appropriate and fair corrective action in +response to any behavior that they deem inappropriate, threatening, offensive, +or harmful. + +Community leaders have the right and responsibility to remove, edit, or reject +comments, commits, code, wiki edits, issues, and other contributions that are +not aligned to this Code of Conduct, and will communicate reasons for moderation +decisions when appropriate. + +## Scope + +This Code of Conduct applies within all community spaces, and also applies when +an individual is officially representing the community in public spaces. +Examples of representing our community include using an official e-mail address, +posting via an official social media account, or acting as an appointed +representative at an online or offline event. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported to the community leaders responsible for enforcement at +`swillison+datasette-code-of-conduct@gmail.com`. +All complaints will be reviewed and investigated promptly and fairly. + +All community leaders are obligated to respect the privacy and security of the +reporter of any incident. + +## Enforcement Guidelines + +Community leaders will follow these Community Impact Guidelines in determining +the consequences for any action they deem in violation of this Code of Conduct: + +### 1. Correction + +**Community Impact**: Use of inappropriate language or other behavior deemed +unprofessional or unwelcome in the community. + +**Consequence**: A private, written warning from community leaders, providing +clarity around the nature of the violation and an explanation of why the +behavior was inappropriate. A public apology may be requested. + +### 2. Warning + +**Community Impact**: A violation through a single incident or series +of actions. + +**Consequence**: A warning with consequences for continued behavior. No +interaction with the people involved, including unsolicited interaction with +those enforcing the Code of Conduct, for a specified period of time. This +includes avoiding interactions in community spaces as well as external channels +like social media. Violating these terms may lead to a temporary or +permanent ban. + +### 3. Temporary Ban + +**Community Impact**: A serious violation of community standards, including +sustained inappropriate behavior. + +**Consequence**: A temporary ban from any sort of interaction or public +communication with the community for a specified period of time. No public or +private interaction with the people involved, including unsolicited interaction +with those enforcing the Code of Conduct, is allowed during this period. +Violating these terms may lead to a permanent ban. + +### 4. Permanent Ban + +**Community Impact**: Demonstrating a pattern of violation of community +standards, including sustained inappropriate behavior, harassment of an +individual, or aggression toward or disparagement of classes of individuals. + +**Consequence**: A permanent ban from any sort of public interaction within +the community. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], +version 2.0, available at +https://www.contributor-covenant.org/version/2/0/code_of_conduct.html. + +Community Impact Guidelines were inspired by [Mozilla's code of conduct +enforcement ladder](https://github.com/mozilla/diversity). + +[homepage]: https://www.contributor-covenant.org + +For answers to common questions about this code of conduct, see the FAQ at +https://www.contributor-covenant.org/faq. Translations are available at +https://www.contributor-covenant.org/translations. From c10cd48baf106659bf3f129ad7bfb2226be73821 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 7 Mar 2022 11:56:59 -0800 Subject: [PATCH 006/952] Min pytest-asyncio of 0.17 So that the asyncio_mode in pytest.ini does not produce a warning on older versions of that library. --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 8e69c2f5..e70839d6 100644 --- a/setup.py +++ b/setup.py @@ -69,7 +69,7 @@ setup( "test": [ "pytest>=5.2.2,<7.1.0", "pytest-xdist>=2.2.1,<2.6", - "pytest-asyncio>=0.10,<0.19", + "pytest-asyncio>=0.17,<0.19", "beautifulsoup4>=4.8.1,<4.11.0", "black==22.1.0", "pytest-timeout>=1.4.2,<2.2", From a35393b29cfb5b8abdc6a94e577af1c9a5c13652 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 15 Mar 2022 11:01:57 -0700 Subject: [PATCH 007/952] Tilde encoding (#1659) Closes #1657 Refs #1439 --- datasette/app.py | 11 +++++---- datasette/url_builder.py | 10 ++++---- datasette/utils/__init__.py | 37 ++++++++++++++++------------- datasette/views/base.py | 25 +++++++++++--------- datasette/views/table.py | 14 +++++++---- docs/csv_export.rst | 18 --------------- docs/internals.rst | 34 +++++++++++++-------------- tests/test_api.py | 17 ++++---------- tests/test_cli.py | 6 ++--- tests/test_html.py | 45 +++++++++++++++++++++--------------- tests/test_internals_urls.py | 2 +- tests/test_table_api.py | 9 +++++--- tests/test_table_html.py | 2 +- tests/test_utils.py | 36 +++++++++-------------------- 14 files changed, 125 insertions(+), 141 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 7abccc05..b39ef7cd 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -1211,11 +1211,14 @@ class DatasetteRouter: return await self.handle_404(request, send) async def handle_404(self, request, send, exception=None): - # If path contains % encoding, redirect to dash encoding + # If path contains % encoding, redirect to tilde encoding if "%" in request.path: - # Try the same path but with "%" replaced by "-" - # and "-" replaced with "-2D" - new_path = request.path.replace("-", "-2D").replace("%", "-") + # Try the same path but with "%" replaced by "~" + # and "~" replaced with "~7E" + # and "." replaced with "~2E" + new_path = ( + request.path.replace("~", "~7E").replace("%", "~").replace(".", "~2E") + ) if request.query_string: new_path += "?{}".format(request.query_string) await asgi_send_redirect(send, new_path) diff --git a/datasette/url_builder.py b/datasette/url_builder.py index eebfe31e..9f072462 100644 --- a/datasette/url_builder.py +++ b/datasette/url_builder.py @@ -1,4 +1,4 @@ -from .utils import dash_encode, path_with_format, HASH_LENGTH, PrefixedUrlString +from .utils import tilde_encode, path_with_format, HASH_LENGTH, PrefixedUrlString import urllib @@ -31,20 +31,20 @@ class Urls: db = self.ds.databases[database] if self.ds.setting("hash_urls") and db.hash: path = self.path( - f"{dash_encode(database)}-{db.hash[:HASH_LENGTH]}", format=format + f"{tilde_encode(database)}-{db.hash[:HASH_LENGTH]}", format=format ) else: - path = self.path(dash_encode(database), format=format) + path = self.path(tilde_encode(database), format=format) return path def table(self, database, table, format=None): - path = f"{self.database(database)}/{dash_encode(table)}" + path = f"{self.database(database)}/{tilde_encode(table)}" if format is not None: path = path_with_format(path=path, format=format) return PrefixedUrlString(path) def query(self, database, query, format=None): - path = f"{self.database(database)}/{dash_encode(query)}" + path = f"{self.database(database)}/{tilde_encode(query)}" if format is not None: path = path_with_format(path=path, format=format) return PrefixedUrlString(path) diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index 79feeef6..bd591459 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -15,6 +15,7 @@ import tempfile import typing import time import types +import secrets import shutil import urllib import yaml @@ -112,12 +113,12 @@ async def await_me_maybe(value: typing.Any) -> typing.Any: def urlsafe_components(token): - """Splits token on commas and dash-decodes each component""" - return [dash_decode(b) for b in token.split(",")] + """Splits token on commas and tilde-decodes each component""" + return [tilde_decode(b) for b in token.split(",")] def path_from_row_pks(row, pks, use_rowid, quote=True): - """Generate an optionally dash-quoted unique identifier + """Generate an optionally tilde-encoded unique identifier for a row from its primary keys.""" if use_rowid: bits = [row["rowid"]] @@ -126,7 +127,7 @@ def path_from_row_pks(row, pks, use_rowid, quote=True): row[pk]["value"] if isinstance(row[pk], dict) else row[pk] for pk in pks ] if quote: - bits = [dash_encode(str(bit)) for bit in bits] + bits = [tilde_encode(str(bit)) for bit in bits] else: bits = [str(bit) for bit in bits] @@ -1142,34 +1143,38 @@ def add_cors_headers(headers): headers["Access-Control-Expose-Headers"] = "Link" -_DASH_ENCODING_SAFE = frozenset( +_TILDE_ENCODING_SAFE = frozenset( b"ABCDEFGHIJKLMNOPQRSTUVWXYZ" b"abcdefghijklmnopqrstuvwxyz" - b"0123456789_" + b"0123456789_-" # This is the same as Python percent-encoding but I removed - # '.' and '-' and '~' + # '.' and '~' ) -class DashEncoder(dict): +class TildeEncoder(dict): # Keeps a cache internally, via __missing__ def __missing__(self, b): # Handle a cache miss, store encoded string in cache and return. - res = chr(b) if b in _DASH_ENCODING_SAFE else "-{:02X}".format(b) + res = chr(b) if b in _TILDE_ENCODING_SAFE else "~{:02X}".format(b) self[b] = res return res -_dash_encoder = DashEncoder().__getitem__ +_tilde_encoder = TildeEncoder().__getitem__ @documented -def dash_encode(s: str) -> str: - "Returns dash-encoded string - for example ``/foo/bar`` -> ``-2Ffoo-2Fbar``" - return "".join(_dash_encoder(char) for char in s.encode("utf-8")) +def tilde_encode(s: str) -> str: + "Returns tilde-encoded string - for example ``/foo/bar`` -> ``~2Ffoo~2Fbar``" + return "".join(_tilde_encoder(char) for char in s.encode("utf-8")) @documented -def dash_decode(s: str) -> str: - "Decodes a dash-encoded string, so ``-2Ffoo-2Fbar`` -> ``/foo/bar``" - return urllib.parse.unquote(s.replace("-", "%")) +def tilde_decode(s: str) -> str: + "Decodes a tilde-encoded string, so ``~2Ffoo~2Fbar`` -> ``/foo/bar``" + # Avoid accidentally decoding a %2f style sequence + temp = secrets.token_hex(16) + s = s.replace("%", temp) + decoded = urllib.parse.unquote(s.replace("~", "%")) + return decoded.replace(temp, "%") diff --git a/datasette/views/base.py b/datasette/views/base.py index 7cd385b7..1c0c3f9b 100644 --- a/datasette/views/base.py +++ b/datasette/views/base.py @@ -10,6 +10,7 @@ import pint from datasette import __version__ from datasette.database import QueryInterrupted +from datasette.utils.asgi import Request from datasette.utils import ( add_cors_headers, await_me_maybe, @@ -17,8 +18,8 @@ from datasette.utils import ( InvalidSql, LimitedWriter, call_with_supported_arguments, - dash_decode, - dash_encode, + tilde_decode, + tilde_encode, path_from_row_pks, path_with_added_args, path_with_removed_args, @@ -205,14 +206,14 @@ class DataView(BaseView): async def resolve_db_name(self, request, db_name, **kwargs): hash = None name = None - decoded_name = dash_decode(db_name) + decoded_name = tilde_decode(db_name) if decoded_name not in self.ds.databases and "-" in db_name: # No matching DB found, maybe it's a name-hash? name_bit, hash_bit = db_name.rsplit("-", 1) - if dash_decode(name_bit) not in self.ds.databases: + if tilde_decode(name_bit) not in self.ds.databases: raise NotFound(f"Database not found: {name}") else: - name = dash_decode(name_bit) + name = tilde_decode(name_bit) hash = hash_bit else: name = decoded_name @@ -235,7 +236,7 @@ class DataView(BaseView): return await db.table_exists(t) table, _format = await resolve_table_and_format( - table_and_format=dash_decode(kwargs["table_and_format"]), + table_and_format=tilde_decode(kwargs["table_and_format"]), table_exists=async_table_exists, allowed_formats=self.ds.renderers.keys(), ) @@ -243,11 +244,11 @@ class DataView(BaseView): if _format: kwargs["as_format"] = f".{_format}" elif kwargs.get("table"): - kwargs["table"] = dash_decode(kwargs["table"]) + kwargs["table"] = tilde_decode(kwargs["table"]) should_redirect = self.ds.urls.path(f"{name}-{expected}") if kwargs.get("table"): - should_redirect += "/" + dash_encode(kwargs["table"]) + should_redirect += "/" + tilde_encode(kwargs["table"]) if kwargs.get("pk_path"): should_redirect += "/" + kwargs["pk_path"] if kwargs.get("as_format"): @@ -291,6 +292,7 @@ class DataView(BaseView): if not request.args.get(key) ] if extra_parameters: + # Replace request object with a new one with modified scope if not request.query_string: new_query_string = "&".join(extra_parameters) else: @@ -300,7 +302,8 @@ class DataView(BaseView): new_scope = dict( request.scope, query_string=new_query_string.encode("latin-1") ) - request.scope = new_scope + receive = request.receive + request = Request(new_scope, receive) if stream: # Some quick soundness checks if not self.ds.setting("allow_csv_stream"): @@ -467,7 +470,7 @@ class DataView(BaseView): return await db.table_exists(t) table, _ext_format = await resolve_table_and_format( - table_and_format=dash_decode(args["table_and_format"]), + table_and_format=tilde_decode(args["table_and_format"]), table_exists=async_table_exists, allowed_formats=self.ds.renderers.keys(), ) @@ -475,7 +478,7 @@ class DataView(BaseView): args["table"] = table del args["table_and_format"] elif "table" in args: - args["table"] = dash_decode(args["table"]) + args["table"] = tilde_decode(args["table"]) return _format, args async def view_get(self, request, database, hash, correct_hash_provided, **kwargs): diff --git a/datasette/views/table.py b/datasette/views/table.py index 1d81755e..72b8e9a4 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -12,7 +12,8 @@ from datasette.utils import ( MultiParams, append_querystring, compound_keys_after_sql, - dash_encode, + tilde_decode, + tilde_encode, escape_sqlite, filters_should_redirect, is_url, @@ -143,7 +144,7 @@ class RowTableShared(DataView): '{flat_pks}'.format( base_url=base_url, database=database, - table=dash_encode(table), + table=tilde_encode(table), flat_pks=str(markupsafe.escape(pk_path)), flat_pks_quoted=path_from_row_pks(row, pks, not pks), ) @@ -200,8 +201,8 @@ class RowTableShared(DataView): link_template.format( database=database, base_url=base_url, - table=dash_encode(other_table), - link_id=dash_encode(str(value)), + table=tilde_encode(other_table), + link_id=tilde_encode(str(value)), id=str(markupsafe.escape(value)), label=str(markupsafe.escape(label)) or "-", ) @@ -346,6 +347,8 @@ class TableView(RowTableShared): write=bool(canned_query.get("write")), ) + table = tilde_decode(table) + db = self.ds.databases[database] is_view = bool(await db.get_view_definition(table)) table_exists = bool(await db.table_exists(table)) @@ -766,7 +769,7 @@ class TableView(RowTableShared): if prefix is None: prefix = "$null" else: - prefix = dash_encode(str(prefix)) + prefix = tilde_encode(str(prefix)) next_value = f"{prefix},{next_value}" added_args = {"_next": next_value} if sort: @@ -938,6 +941,7 @@ class RowView(RowTableShared): name = "row" async def data(self, request, database, hash, table, pk_path, default_labels=False): + table = tilde_decode(table) await self.check_permissions( request, [ diff --git a/docs/csv_export.rst b/docs/csv_export.rst index b1cc673c..023fa05e 100644 --- a/docs/csv_export.rst +++ b/docs/csv_export.rst @@ -59,21 +59,3 @@ truncation error message. You can increase or remove this limit using the :ref:`setting_max_csv_mb` config setting. You can also disable the CSV export feature entirely using :ref:`setting_allow_csv_stream`. - -A note on URLs --------------- - -The default URL for the CSV representation of a table is that table with -``.csv`` appended to it: - -* https://latest.datasette.io/fixtures/facetable - HTML interface -* https://latest.datasette.io/fixtures/facetable.csv - CSV export -* https://latest.datasette.io/fixtures/facetable.json - JSON API - -This pattern doesn't work for tables with names that already end in ``.csv`` or -``.json``. For those tables, you can instead use the ``_format=`` query string -parameter: - -* https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv - HTML interface -* https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv?_format=csv - CSV export -* https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv?_format=json - JSON API diff --git a/docs/internals.rst b/docs/internals.rst index d035e1f1..3d223603 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -545,7 +545,7 @@ These functions can be accessed via the ``{{ urls }}`` object in Datasette templ facetable table pragma_cache_size query -Use the ``format="json"`` (or ``"csv"`` or other formats supported by plugins) arguments to get back URLs to the JSON representation. This is usually the path with ``.json`` added on the end, but it may use ``?_format=json`` in cases where the path already includes ``.json``, for example a URL to a table named ``table.json``. +Use the ``format="json"`` (or ``"csv"`` or other formats supported by plugins) arguments to get back URLs to the JSON representation. This is the path with ``.json`` added on the end. These methods each return a ``datasette.utils.PrefixedUrlString`` object, which is a subclass of the Python ``str`` type. This allows the logic that considers the ``base_url`` setting to detect if that prefix has already been applied to the path. @@ -876,31 +876,31 @@ Utility function for calling ``await`` on a return value if it is awaitable, oth .. autofunction:: datasette.utils.await_me_maybe -.. _internals_dash_encoding: +.. _internals_tilde_encoding: -Dash encoding -------------- +Tilde encoding +-------------- -Datasette uses a custom encoding scheme in some places, called **dash encoding**. This is primarily used for table names and row primary keys, to avoid any confusion between ``/`` characters in those values and the Datasette URLs that reference them. +Datasette uses a custom encoding scheme in some places, called **tilde encoding**. This is primarily used for table names and row primary keys, to avoid any confusion between ``/`` characters in those values and the Datasette URLs that reference them. -Dash encoding uses the same algorithm as `URL percent-encoding `__, but with the ``-`` hyphen character used in place of ``%``. +Tilde encoding uses the same algorithm as `URL percent-encoding `__, but with the ``~`` tilde character used in place of ``%``. -Any character other than ``ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz 0123456789_`` will be replaced by the numeric equivalent preceded by a hyphen. For example: +Any character other than ``ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz 0123456789_-`` will be replaced by the numeric equivalent preceded by a tilde. For example: -- ``/`` becomes ``-2F`` -- ``.`` becomes ``-2E`` -- ``%`` becomes ``-25`` -- ``-`` becomes ``-2D`` -- Space character becomes ``-20`` -- ``polls/2022.primary`` becomes ``polls-2F2022-2Eprimary`` +- ``/`` becomes ``~2F`` +- ``.`` becomes ``~2E`` +- ``%`` becomes ``~25`` +- ``~`` becomes ``~7E`` +- Space character becomes ``~20`` +- ``polls/2022.primary`` becomes ``polls~2F2022~2Eprimary`` -.. _internals_utils_dash_encode: +.. _internals_utils_tilde_encode: -.. autofunction:: datasette.utils.dash_encode +.. autofunction:: datasette.utils.tilde_encode -.. _internals_utils_dash_decode: +.. _internals_utils_tilde_decode: -.. autofunction:: datasette.utils.dash_decode +.. autofunction:: datasette.utils.tilde_decode .. _internals_tracer: diff --git a/tests/test_api.py b/tests/test_api.py index dd916cf0..87d91e56 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -679,18 +679,9 @@ def test_row(app_client): assert [{"id": "1", "content": "hello"}] == response.json["rows"] -def test_row_format_in_querystring(app_client): - # regression test for https://github.com/simonw/datasette/issues/563 - response = app_client.get( - "/fixtures/simple_primary_key/1?_format=json&_shape=objects" - ) - assert response.status == 200 - assert [{"id": "1", "content": "hello"}] == response.json["rows"] - - def test_row_strange_table_name(app_client): response = app_client.get( - "/fixtures/table%2Fwith%2Fslashes.csv/3.json?_shape=objects" + "/fixtures/table~2Fwith~2Fslashes~2Ecsv/3.json?_shape=objects" ) assert response.status == 200 assert [{"pk": "3", "content": "hey"}] == response.json["rows"] @@ -942,7 +933,7 @@ def test_cors(app_client_with_cors, path, status_code): ) def test_database_with_space_in_name(app_client_two_attached_databases, path): response = app_client_two_attached_databases.get( - "/extra-20database" + path, follow_redirects=True + "/extra~20database" + path, follow_redirects=True ) assert response.status == 200 @@ -953,7 +944,7 @@ def test_common_prefix_database_names(app_client_conflicting_database_names): d["name"] for d in app_client_conflicting_database_names.get("/-/databases.json").json ] - for db_name, path in (("foo", "/foo.json"), ("foo-bar", "/foo-2Dbar.json")): + for db_name, path in (("foo", "/foo.json"), ("foo-bar", "/foo-bar.json")): data = app_client_conflicting_database_names.get(path).json assert db_name == data["database"] @@ -996,7 +987,7 @@ async def test_hidden_sqlite_stat1_table(): @pytest.mark.asyncio @pytest.mark.parametrize("db_name", ("foo", r"fo%o", "f~/c.d")) -async def test_dash_encoded_database_names(db_name): +async def test_tilde_encoded_database_names(db_name): ds = Datasette() ds.add_memory_database(db_name) response = await ds.client.get("/.json") diff --git a/tests/test_cli.py b/tests/test_cli.py index e30c2ad3..5afe72c1 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -9,7 +9,7 @@ from datasette.app import SETTINGS from datasette.plugins import DEFAULT_PLUGINS from datasette.cli import cli, serve from datasette.version import __version__ -from datasette.utils import dash_encode +from datasette.utils import tilde_encode from datasette.utils.sqlite import sqlite3 from click.testing import CliRunner import io @@ -295,12 +295,12 @@ def test_weird_database_names(ensure_eventloop, tmpdir, filename): assert result1.exit_code == 0, result1.output filename_no_stem = filename.rsplit(".", 1)[0] expected_link = '{}'.format( - dash_encode(filename_no_stem), filename_no_stem + tilde_encode(filename_no_stem), filename_no_stem ) assert expected_link in result1.output # Now try hitting that database page result2 = runner.invoke( - cli, [db_path, "--get", "/{}".format(dash_encode(filename_no_stem))] + cli, [db_path, "--get", "/{}".format(tilde_encode(filename_no_stem))] ) assert result2.exit_code == 0, result2.output diff --git a/tests/test_html.py b/tests/test_html.py index de703284..76a8423a 100644 --- a/tests/test_html.py +++ b/tests/test_html.py @@ -29,7 +29,7 @@ def test_homepage(app_client_two_attached_databases): ) # Should be two attached databases assert [ - {"href": r"/extra-20database", "text": "extra database"}, + {"href": "/extra~20database", "text": "extra database"}, {"href": "/fixtures", "text": "fixtures"}, ] == [{"href": a["href"], "text": a.text.strip()} for a in soup.select("h2 a")] # Database should show count text and attached tables @@ -44,8 +44,8 @@ def test_homepage(app_client_two_attached_databases): {"href": a["href"], "text": a.text.strip()} for a in links_p.findAll("a") ] assert [ - {"href": r"/extra-20database/searchable", "text": "searchable"}, - {"href": r"/extra-20database/searchable_view", "text": "searchable_view"}, + {"href": r"/extra~20database/searchable", "text": "searchable"}, + {"href": r"/extra~20database/searchable_view", "text": "searchable_view"}, ] == table_links @@ -139,15 +139,15 @@ def test_database_page(app_client): queries_ul = soup.find("h2", text="Queries").find_next_sibling("ul") assert queries_ul is not None assert [ - ( - "/fixtures/-F0-9D-90-9C-F0-9D-90-A2-F0-9D-90-AD-F0-9D-90-A2-F0-9D-90-9E-F0-9D-90-AC", - "𝐜𝐢𝐭𝐢𝐞𝐬", - ), ("/fixtures/from_async_hook", "from_async_hook"), ("/fixtures/from_hook", "from_hook"), ("/fixtures/magic_parameters", "magic_parameters"), ("/fixtures/neighborhood_search#fragment-goes-here", "Search neighborhoods"), ("/fixtures/pragma_cache_size", "pragma_cache_size"), + ( + "/fixtures/~F0~9D~90~9C~F0~9D~90~A2~F0~9D~90~AD~F0~9D~90~A2~F0~9D~90~9E~F0~9D~90~AC", + "𝐜𝐢𝐭𝐢𝐞𝐬", + ), ] == sorted( [(a["href"], a.text) for a in queries_ul.find_all("a")], key=lambda p: p[0] ) @@ -193,11 +193,11 @@ def test_row_redirects_with_url_hash(app_client_with_hash): def test_row_strange_table_name_with_url_hash(app_client_with_hash): - response = app_client_with_hash.get("/fixtures/table-2Fwith-2Fslashes-2Ecsv/3") + response = app_client_with_hash.get("/fixtures/table~2Fwith~2Fslashes~2Ecsv/3") assert response.status == 302 - assert response.headers["Location"].endswith("/table-2Fwith-2Fslashes-2Ecsv/3") + assert response.headers["Location"].endswith("/table~2Fwith~2Fslashes~2Ecsv/3") response = app_client_with_hash.get( - "/fixtures/table-2Fwith-2Fslashes-2Ecsv/3", follow_redirects=True + "/fixtures/table~2Fwith~2Fslashes~2Ecsv/3", follow_redirects=True ) assert response.status == 200 @@ -229,7 +229,7 @@ def test_row_page_does_not_truncate(): ["query", "db-fixtures", "query-neighborhood_search"], ), ( - "/fixtures/table%2Fwith%2Fslashes.csv", + "/fixtures/table~2Fwith~2Fslashes~2Ecsv", ["table", "db-fixtures", "table-tablewithslashescsv-fa7563"], ), ( @@ -255,7 +255,7 @@ def test_css_classes_on_body(app_client, path, expected_classes): "table-fixtures-simple_primary_key.html, *table.html", ), ( - "/fixtures/table%2Fwith%2Fslashes.csv", + "/fixtures/table~2Fwith~2Fslashes~2Ecsv", "table-fixtures-tablewithslashescsv-fa7563.html, *table.html", ), ( @@ -359,7 +359,7 @@ def test_row_links_from_other_tables(app_client, path, expected_text, expected_l ], ), ( - "/fixtures/compound_primary_key/a-2Fb,-2Ec-2Dd", + "/fixtures/compound_primary_key/a~2Fb,~2Ec~2Dd", [ [ 'a/b', @@ -816,7 +816,8 @@ def test_base_url_affects_metadata_extra_css_urls(app_client_base_url_prefix): ), ("/fixtures/pragma_cache_size", None), ( - "/fixtures/𝐜𝐢𝐭𝐢𝐞𝐬", + # /fixtures/𝐜𝐢𝐭𝐢𝐞𝐬 + "/fixtures/~F0~9D~90~9C~F0~9D~90~A2~F0~9D~90~AD~F0~9D~90~A2~F0~9D~90~9E~F0~9D~90~AC", "/fixtures?sql=select+id%2C+name+from+facet_cities+order+by+id+limit+1%3B", ), ("/fixtures/magic_parameters", None), @@ -824,6 +825,7 @@ def test_base_url_affects_metadata_extra_css_urls(app_client_base_url_prefix): ) def test_edit_sql_link_on_canned_queries(app_client, path, expected): response = app_client.get(path) + assert response.status == 200 expected_link = f'Edit SQL' if expected: assert expected_link in response.text @@ -898,8 +900,8 @@ def test_trace_correctly_escaped(app_client): # Table page ("/fixtures/facetable", "http://localhost/fixtures/facetable.json"), ( - "/fixtures/table%2Fwith%2Fslashes.csv", - "http://localhost/fixtures/table%2Fwith%2Fslashes.csv?_format=json", + "/fixtures/table~2Fwith~2Fslashes~2Ecsv", + "http://localhost/fixtures/table~2Fwith~2Fslashes~2Ecsv.json", ), # Row page ( @@ -930,6 +932,7 @@ def test_trace_correctly_escaped(app_client): ) def test_alternate_url_json(app_client, path, expected): response = app_client.get(path) + assert response.status == 200 link = response.headers["link"] assert link == '{}; rel="alternate"; type="application/json+datasette"'.format( expected @@ -959,13 +962,17 @@ def test_no_alternate_url_json(app_client, path): ( ( "/fivethirtyeight/twitter-ratio%2Fsenators", - "/fivethirtyeight/twitter-2Dratio-2Fsenators", + "/fivethirtyeight/twitter-ratio~2Fsenators", + ), + ( + "/fixtures/table%2Fwith%2Fslashes.csv", + "/fixtures/table~2Fwith~2Fslashes~2Ecsv", ), # query string should be preserved - ("/foo/bar%2Fbaz?id=5", "/foo/bar-2Fbaz?id=5"), + ("/foo/bar%2Fbaz?id=5", "/foo/bar~2Fbaz?id=5"), ), ) -def test_redirect_percent_encoding_to_dash_encoding(app_client, path, expected): +def test_redirect_percent_encoding_to_tilde_encoding(app_client, path, expected): response = app_client.get(path) assert response.status == 302 assert response.headers["location"] == expected diff --git a/tests/test_internals_urls.py b/tests/test_internals_urls.py index 16515ad6..4307789c 100644 --- a/tests/test_internals_urls.py +++ b/tests/test_internals_urls.py @@ -121,7 +121,7 @@ def test_database(ds, base_url, format, expected): ("/", "name", None, "/_memory/name"), ("/prefix/", "name", None, "/prefix/_memory/name"), ("/", "name", "json", "/_memory/name.json"), - ("/", "name.json", "json", "/_memory/name-2Ejson.json"), + ("/", "name.json", "json", "/_memory/name~2Ejson.json"), ], ) def test_table_and_query(ds, base_url, name, format, expected): diff --git a/tests/test_table_api.py b/tests/test_table_api.py index cc38d392..3ab369b3 100644 --- a/tests/test_table_api.py +++ b/tests/test_table_api.py @@ -138,13 +138,13 @@ def test_table_shape_object_compound_primary_key(app_client): response = app_client.get("/fixtures/compound_primary_key.json?_shape=object") assert response.json == { "a,b": {"pk1": "a", "pk2": "b", "content": "c"}, - "a-2Fb,-2Ec-2Dd": {"pk1": "a/b", "pk2": ".c-d", "content": "c"}, + "a~2Fb,~2Ec-d": {"pk1": "a/b", "pk2": ".c-d", "content": "c"}, } def test_table_with_slashes_in_name(app_client): response = app_client.get( - "/fixtures/table%2Fwith%2Fslashes.csv?_shape=objects&_format=json" + "/fixtures/table~2Fwith~2Fslashes~2Ecsv.json?_shape=objects" ) assert response.status == 200 data = response.json @@ -1032,7 +1032,10 @@ def test_infinity_returned_as_invalid_json_if_requested(app_client): def test_custom_query_with_unicode_characters(app_client): - response = app_client.get("/fixtures/𝐜𝐢𝐭𝐢𝐞𝐬.json?_shape=array") + # /fixtures/𝐜𝐢𝐭𝐢𝐞𝐬.json + response = app_client.get( + "/fixtures/~F0~9D~90~9C~F0~9D~90~A2~F0~9D~90~AD~F0~9D~90~A2~F0~9D~90~9E~F0~9D~90~AC.json?_shape=array" + ) assert [{"id": 1, "name": "San Francisco"}] == response.json diff --git a/tests/test_table_html.py b/tests/test_table_html.py index 77d97d80..d40f017a 100644 --- a/tests/test_table_html.py +++ b/tests/test_table_html.py @@ -565,7 +565,7 @@ def test_table_html_compound_primary_key(app_client): 'c', ], [ - 'a/b,.c-d', + 'a/b,.c-d', 'a/b', '.c-d', 'c', diff --git a/tests/test_utils.py b/tests/test_utils.py index 1c3ab495..790aadc7 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -19,8 +19,8 @@ from unittest.mock import patch ("foo", ["foo"]), ("foo,bar", ["foo", "bar"]), ("123,433,112", ["123", "433", "112"]), - ("123%2C433,112", ["123,433", "112"]), - ("123%2F433%2F112", ["123/433/112"]), + ("123~2C433,112", ["123,433", "112"]), + ("123~2F433~2F112", ["123/433/112"]), ], ) def test_urlsafe_components(path, expected): @@ -93,7 +93,7 @@ def test_path_with_replaced_args(path, args, expected): "row,pks,expected_path", [ ({"A": "foo", "B": "bar"}, ["A", "B"], "foo,bar"), - ({"A": "f,o", "B": "bar"}, ["A", "B"], "f-2Co,bar"), + ({"A": "f,o", "B": "bar"}, ["A", "B"], "f~2Co,bar"), ({"A": 123}, ["A"], "123"), ( utils.CustomRow( @@ -393,9 +393,7 @@ def test_table_columns(): ("/foo?sql=select+1", "json", {}, "/foo.json?sql=select+1"), ("/foo/bar", "json", {}, "/foo/bar.json"), ("/foo/bar", "csv", {}, "/foo/bar.csv"), - ("/foo/bar.csv", "json", {}, "/foo/bar.csv?_format=json"), ("/foo/bar", "csv", {"_dl": 1}, "/foo/bar.csv?_dl=1"), - ("/foo/b.csv", "json", {"_dl": 1}, "/foo/b.csv?_dl=1&_format=json"), ( "/sf-trees/Street_Tree_List?_search=cherry&_size=1000", "csv", @@ -410,18 +408,6 @@ def test_path_with_format(path, format, extra_qs, expected): assert expected == actual -def test_path_with_format_replace_format(): - request = Request.fake("/foo/bar.csv") - assert ( - utils.path_with_format(request=request, format="blob") - == "/foo/bar.csv?_format=blob" - ) - assert ( - utils.path_with_format(request=request, format="blob", replace_format="csv") - == "/foo/bar.blob" - ) - - @pytest.mark.parametrize( "bytes,expected", [ @@ -652,15 +638,15 @@ async def test_derive_named_parameters(sql, expected): "original,expected", ( ("abc", "abc"), - ("/foo/bar", "-2Ffoo-2Fbar"), - ("/-/bar", "-2F-2D-2Fbar"), - ("-/db-/table.csv", "-2D-2Fdb-2D-2Ftable-2Ecsv"), - (r"%~-/", "-25-7E-2D-2F"), - ("-25-7E-2D-2F", "-2D25-2D7E-2D2D-2D2F"), + ("/foo/bar", "~2Ffoo~2Fbar"), + ("/-/bar", "~2F-~2Fbar"), + ("-/db-/table.csv", "-~2Fdb-~2Ftable~2Ecsv"), + (r"%~-/", "~25~7E-~2F"), + ("~25~7E~2D~2F", "~7E25~7E7E~7E2D~7E2F"), ), ) -def test_dash_encoding(original, expected): - actual = utils.dash_encode(original) +def test_tilde_encoding(original, expected): + actual = utils.tilde_encode(original) assert actual == expected # And test round-trip - assert original == utils.dash_decode(actual) + assert original == utils.tilde_decode(actual) From 77a904fea14f743560af9cc668146339bdbbd0a9 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 15 Mar 2022 11:03:01 -0700 Subject: [PATCH 008/952] Update pytest requirement from <7.1.0,>=5.2.2 to >=5.2.2,<7.2.0 (#1656) Updates the requirements on [pytest](https://github.com/pytest-dev/pytest) to permit the latest version. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst) - [Commits](https://github.com/pytest-dev/pytest/compare/5.2.2...7.1.0) --- updated-dependencies: - dependency-name: pytest dependency-type: direct:development ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index e70839d6..4b58b8c4 100644 --- a/setup.py +++ b/setup.py @@ -67,7 +67,7 @@ setup( extras_require={ "docs": ["sphinx_rtd_theme", "sphinx-autobuild", "codespell"], "test": [ - "pytest>=5.2.2,<7.1.0", + "pytest>=5.2.2,<7.2.0", "pytest-xdist>=2.2.1,<2.6", "pytest-asyncio>=0.17,<0.19", "beautifulsoup4>=4.8.1,<4.11.0", From 30e5f0e67c38054a8087a2a4eae3fc4d1779af90 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 17 Mar 2022 14:30:02 -0700 Subject: [PATCH 009/952] Documented internals used by datasette-hashed-urls Closes #1663 --- docs/internals.rst | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/docs/internals.rst b/docs/internals.rst index 3d223603..117cb95c 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -217,12 +217,18 @@ You can create your own instance of this - for example to help write tests for a } }) +Constructor parameters include: + +- ``files=[...]`` - a list of database files to open +- ``immutables=[...]`` - a list of database files to open in immutable mode +- ``metadata={...}`` - a dictionary of :ref:`metadata` + .. _datasette_databases: .databases ---------- -Property exposing an ordered dictionary of databases currently connected to Datasette. +Property exposing a ``collections.OrderedDict`` of databases currently connected to Datasette. The dictionary keys are the name of the database that is used in the URL - e.g. ``/fixtures`` would have a key of ``"fixtures"``. The values are :ref:`internals_database` instances. @@ -582,6 +588,13 @@ The arguments are as follows: The first argument is the ``datasette`` instance you are attaching to, the second is a ``path=``, then ``is_mutable`` and ``is_memory`` are both optional arguments. +.. _database_hash: + +db.hash +------- + +If the database was opened in immutable mode, this property returns the 64 character SHA-256 hash of the database contents as a string. Otherwise it returns ``None``. + .. _database_execute: await db.execute(sql, ...) From d4f60c2388c01ddce1b16f95c16d310e037c9912 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 18 Mar 2022 17:12:03 -0700 Subject: [PATCH 010/952] Remove hashed URL mode Also simplified how view class routing works. Refs #1661 --- datasette/app.py | 2 +- datasette/views/base.py | 153 ++++++----------------------------- datasette/views/database.py | 19 +++-- datasette/views/index.py | 3 +- datasette/views/special.py | 3 +- datasette/views/table.py | 34 ++++---- tests/fixtures.py | 6 -- tests/test_api.py | 29 ------- tests/test_custom_pages.py | 42 +++++----- tests/test_html.py | 28 ------- tests/test_internals_urls.py | 18 ----- tests/test_table_api.py | 8 -- 12 files changed, 79 insertions(+), 266 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index b39ef7cd..3099ada7 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -1097,7 +1097,7 @@ class Datasette: ) add_route( TableView.as_view(self), - r"/(?P[^/]+)/(?P[^/]+?$)", + r"/(?P[^/]+)/(?P[^\/\.]+)(\.[a-zA-Z0-9_]+)?$", ) add_route( RowView.as_view(self), diff --git a/datasette/views/base.py b/datasette/views/base.py index 1c0c3f9b..e31beb19 100644 --- a/datasette/views/base.py +++ b/datasette/views/base.py @@ -122,11 +122,11 @@ class BaseView: async def delete(self, request, *args, **kwargs): return Response.text("Method not allowed", status=405) - async def dispatch_request(self, request, *args, **kwargs): + async def dispatch_request(self, request): if self.ds: await self.ds.refresh_schemas() handler = getattr(self, request.method.lower(), None) - return await handler(request, *args, **kwargs) + return await handler(request) async def render(self, templates, request, context=None): context = context or {} @@ -169,9 +169,7 @@ class BaseView: def as_view(cls, *class_args, **class_kwargs): async def view(request, send): self = view.view_class(*class_args, **class_kwargs) - return await self.dispatch_request( - request, **request.scope["url_route"]["kwargs"] - ) + return await self.dispatch_request(request) view.view_class = cls view.__doc__ = cls.__doc__ @@ -200,90 +198,14 @@ class DataView(BaseView): add_cors_headers(r.headers) return r - async def data(self, request, database, hash, **kwargs): + async def data(self, request): raise NotImplementedError - async def resolve_db_name(self, request, db_name, **kwargs): - hash = None - name = None - decoded_name = tilde_decode(db_name) - if decoded_name not in self.ds.databases and "-" in db_name: - # No matching DB found, maybe it's a name-hash? - name_bit, hash_bit = db_name.rsplit("-", 1) - if tilde_decode(name_bit) not in self.ds.databases: - raise NotFound(f"Database not found: {name}") - else: - name = tilde_decode(name_bit) - hash = hash_bit - else: - name = decoded_name - - try: - db = self.ds.databases[name] - except KeyError: - raise NotFound(f"Database not found: {name}") - - # Verify the hash - expected = "000" - if db.hash is not None: - expected = db.hash[:HASH_LENGTH] - correct_hash_provided = expected == hash - - if not correct_hash_provided: - if "table_and_format" in kwargs: - - async def async_table_exists(t): - return await db.table_exists(t) - - table, _format = await resolve_table_and_format( - table_and_format=tilde_decode(kwargs["table_and_format"]), - table_exists=async_table_exists, - allowed_formats=self.ds.renderers.keys(), - ) - kwargs["table"] = table - if _format: - kwargs["as_format"] = f".{_format}" - elif kwargs.get("table"): - kwargs["table"] = tilde_decode(kwargs["table"]) - - should_redirect = self.ds.urls.path(f"{name}-{expected}") - if kwargs.get("table"): - should_redirect += "/" + tilde_encode(kwargs["table"]) - if kwargs.get("pk_path"): - should_redirect += "/" + kwargs["pk_path"] - if kwargs.get("as_format"): - should_redirect += kwargs["as_format"] - if kwargs.get("as_db"): - should_redirect += kwargs["as_db"] - - if ( - (self.ds.setting("hash_urls") or "_hash" in request.args) - and - # Redirect only if database is immutable - not self.ds.databases[name].is_mutable - ): - return name, expected, correct_hash_provided, should_redirect - - return name, expected, correct_hash_provided, None - def get_templates(self, database, table=None): assert NotImplemented - async def get(self, request, db_name, **kwargs): - ( - database, - hash, - correct_hash_provided, - should_redirect, - ) = await self.resolve_db_name(request, db_name, **kwargs) - if should_redirect: - return self.redirect(request, should_redirect, remove_args={"_hash"}) - - return await self.view_get( - request, database, hash, correct_hash_provided, **kwargs - ) - - async def as_csv(self, request, database, hash, **kwargs): + async def as_csv(self, request, database): + kwargs = {} stream = request.args.get("_stream") # Do not calculate facets or counts: extra_parameters = [ @@ -313,9 +235,7 @@ class DataView(BaseView): kwargs["_size"] = "max" # Fetch the first page try: - response_or_template_contexts = await self.data( - request, database, hash, **kwargs - ) + response_or_template_contexts = await self.data(request) if isinstance(response_or_template_contexts, Response): return response_or_template_contexts elif len(response_or_template_contexts) == 4: @@ -367,10 +287,11 @@ class DataView(BaseView): next = None while first or (next and stream): try: + kwargs = {} if next: kwargs["_next"] = next if not first: - data, _, _ = await self.data(request, database, hash, **kwargs) + data, _, _ = await self.data(request, **kwargs) if first: if request.args.get("_header") != "off": await writer.writerow(headings) @@ -445,60 +366,39 @@ class DataView(BaseView): if not trace: content_type = "text/csv; charset=utf-8" disposition = 'attachment; filename="{}.csv"'.format( - kwargs.get("table", database) + request.url_vars.get("table", database) ) headers["content-disposition"] = disposition return AsgiStream(stream_fn, headers=headers, content_type=content_type) - async def get_format(self, request, database, args): - """Determine the format of the response from the request, from URL - parameters or from a file extension. - - `args` is a dict of the path components parsed from the URL by the router. - """ - # If ?_format= is provided, use that as the format - _format = request.args.get("_format", None) - if not _format: - _format = (args.pop("as_format", None) or "").lstrip(".") + def get_format(self, request): + # Format is the bit from the path following the ., if one exists + last_path_component = request.path.split("/")[-1] + if "." in last_path_component: + return last_path_component.split(".")[-1] else: - args.pop("as_format", None) - if "table_and_format" in args: - db = self.ds.databases[database] + return None - async def async_table_exists(t): - return await db.table_exists(t) - - table, _ext_format = await resolve_table_and_format( - table_and_format=tilde_decode(args["table_and_format"]), - table_exists=async_table_exists, - allowed_formats=self.ds.renderers.keys(), - ) - _format = _format or _ext_format - args["table"] = table - del args["table_and_format"] - elif "table" in args: - args["table"] = tilde_decode(args["table"]) - return _format, args - - async def view_get(self, request, database, hash, correct_hash_provided, **kwargs): - _format, kwargs = await self.get_format(request, database, kwargs) + async def get(self, request): + db_name = request.url_vars["db_name"] + database = tilde_decode(db_name) + _format = self.get_format(request) + data_kwargs = {} if _format == "csv": - return await self.as_csv(request, database, hash, **kwargs) + return await self.as_csv(request, database) if _format is None: # HTML views default to expanding all foreign key labels - kwargs["default_labels"] = True + data_kwargs["default_labels"] = True extra_template_data = {} start = time.perf_counter() status_code = None templates = [] try: - response_or_template_contexts = await self.data( - request, database, hash, **kwargs - ) + response_or_template_contexts = await self.data(request, **data_kwargs) if isinstance(response_or_template_contexts, Response): return response_or_template_contexts # If it has four items, it includes an HTTP status code @@ -650,10 +550,7 @@ class DataView(BaseView): ttl = request.args.get("_ttl", None) if ttl is None or not ttl.isdigit(): - if correct_hash_provided: - ttl = self.ds.setting("default_cache_ttl_hashed") - else: - ttl = self.ds.setting("default_cache_ttl") + ttl = self.ds.setting("default_cache_ttl") return self.set_response_headers(r, ttl) diff --git a/datasette/views/database.py b/datasette/views/database.py index e26706e7..48635e01 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -12,6 +12,7 @@ from datasette.utils import ( await_me_maybe, check_visibility, derive_named_parameters, + tilde_decode, to_css_class, validate_sql_select, is_url, @@ -21,7 +22,7 @@ from datasette.utils import ( sqlite3, InvalidSql, ) -from datasette.utils.asgi import AsgiFileDownload, Response, Forbidden +from datasette.utils.asgi import AsgiFileDownload, NotFound, Response, Forbidden from datasette.plugins import pm from .base import DatasetteError, DataView @@ -30,7 +31,8 @@ from .base import DatasetteError, DataView class DatabaseView(DataView): name = "database" - async def data(self, request, database, hash, default_labels=False, _size=None): + async def data(self, request, default_labels=False, _size=None): + database = tilde_decode(request.url_vars["db_name"]) await self.check_permissions( request, [ @@ -45,10 +47,13 @@ class DatabaseView(DataView): sql = request.args.get("sql") validate_sql_select(sql) return await QueryView(self.ds).data( - request, database, hash, sql, _size=_size, metadata=metadata + request, sql, _size=_size, metadata=metadata ) - db = self.ds.databases[database] + try: + db = self.ds.databases[database] + except KeyError: + raise NotFound("Database not found: {}".format(database)) table_counts = await db.table_counts(5) hidden_table_names = set(await db.hidden_table_names()) @@ -156,7 +161,8 @@ class DatabaseView(DataView): class DatabaseDownload(DataView): name = "database_download" - async def view_get(self, request, database, hash, correct_hash_present, **kwargs): + async def get(self, request): + database = tilde_decode(request.url_vars["db_name"]) await self.check_permissions( request, [ @@ -191,8 +197,6 @@ class QueryView(DataView): async def data( self, request, - database, - hash, sql, editable=True, canned_query=None, @@ -201,6 +205,7 @@ class QueryView(DataView): named_parameters=None, write=False, ): + database = tilde_decode(request.url_vars["db_name"]) params = {key: request.args.get(key) for key in request.args} if "sql" in params: params.pop("sql") diff --git a/datasette/views/index.py b/datasette/views/index.py index 18454759..311a49db 100644 --- a/datasette/views/index.py +++ b/datasette/views/index.py @@ -18,7 +18,8 @@ COUNT_DB_SIZE_LIMIT = 100 * 1024 * 1024 class IndexView(BaseView): name = "index" - async def get(self, request, as_format): + async def get(self, request): + as_format = request.url_vars["as_format"] await self.check_permission(request, "view-instance") databases = [] for name, db in self.ds.databases.items(): diff --git a/datasette/views/special.py b/datasette/views/special.py index cdd530f0..c7b5061f 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -14,7 +14,8 @@ class JsonDataView(BaseView): self.data_callback = data_callback self.needs_request = needs_request - async def get(self, request, as_format): + async def get(self, request): + as_format = request.url_vars["as_format"] await self.check_permission(request, "view-instance") if self.needs_request: data = self.data_callback(request) diff --git a/datasette/views/table.py b/datasette/views/table.py index 72b8e9a4..8bdc7417 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -271,20 +271,18 @@ class RowTableShared(DataView): class TableView(RowTableShared): name = "table" - async def post(self, request, db_name, table_and_format): + async def post(self, request): + db_name = tilde_decode(request.url_vars["db_name"]) + table = tilde_decode(request.url_vars["table"]) # Handle POST to a canned query - canned_query = await self.ds.get_canned_query( - db_name, table_and_format, request.actor - ) + canned_query = await self.ds.get_canned_query(db_name, table, request.actor) assert canned_query, "You may only POST to a canned query" return await QueryView(self.ds).data( request, - db_name, - None, canned_query["sql"], metadata=canned_query, editable=False, - canned_query=table_and_format, + canned_query=table, named_parameters=canned_query.get("params"), write=bool(canned_query.get("write")), ) @@ -325,20 +323,22 @@ class TableView(RowTableShared): async def data( self, request, - database, - hash, - table, default_labels=False, _next=None, _size=None, ): + database = tilde_decode(request.url_vars["db_name"]) + table = tilde_decode(request.url_vars["table"]) + try: + db = self.ds.databases[database] + except KeyError: + raise NotFound("Database not found: {}".format(database)) + # If this is a canned query, not a table, then dispatch to QueryView instead canned_query = await self.ds.get_canned_query(database, table, request.actor) if canned_query: return await QueryView(self.ds).data( request, - database, - hash, canned_query["sql"], metadata=canned_query, editable=False, @@ -347,9 +347,6 @@ class TableView(RowTableShared): write=bool(canned_query.get("write")), ) - table = tilde_decode(table) - - db = self.ds.databases[database] is_view = bool(await db.get_view_definition(table)) table_exists = bool(await db.table_exists(table)) @@ -940,8 +937,9 @@ async def _sql_params_pks(db, table, pk_values): class RowView(RowTableShared): name = "row" - async def data(self, request, database, hash, table, pk_path, default_labels=False): - table = tilde_decode(table) + async def data(self, request, default_labels=False): + database = tilde_decode(request.url_vars["db_name"]) + table = tilde_decode(request.url_vars["table"]) await self.check_permissions( request, [ @@ -950,7 +948,7 @@ class RowView(RowTableShared): "view-instance", ], ) - pk_values = urlsafe_components(pk_path) + pk_values = urlsafe_components(request.url_vars["pk_path"]) db = self.ds.databases[database] sql, params, pks = await _sql_params_pks(db, table, pk_values) results = await db.execute(sql, params, truncate=True) diff --git a/tests/fixtures.py b/tests/fixtures.py index 11f09c41..342a3020 100644 --- a/tests/fixtures.py +++ b/tests/fixtures.py @@ -214,12 +214,6 @@ def app_client_two_attached_databases_one_immutable(): yield client -@pytest.fixture(scope="session") -def app_client_with_hash(): - with make_app_client(settings={"hash_urls": True}, is_immutable=True) as client: - yield client - - @pytest.fixture(scope="session") def app_client_with_trace(): with make_app_client(settings={"trace_debug": True}, is_immutable=True) as client: diff --git a/tests/test_api.py b/tests/test_api.py index 87d91e56..46e41afb 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -825,35 +825,6 @@ def test_config_redirects_to_settings(app_client, path, expected_redirect): assert response.headers["Location"] == expected_redirect -@pytest.mark.parametrize( - "path,expected_redirect", - [ - ("/fixtures/facetable.json?_hash=1", "/fixtures-HASH/facetable.json"), - ( - "/fixtures/facetable.json?city_id=1&_hash=1", - "/fixtures-HASH/facetable.json?city_id=1", - ), - ], -) -def test_hash_parameter( - app_client_two_attached_databases_one_immutable, path, expected_redirect -): - # First get the current hash for the fixtures database - current_hash = app_client_two_attached_databases_one_immutable.ds.databases[ - "fixtures" - ].hash[:7] - response = app_client_two_attached_databases_one_immutable.get(path) - assert response.status == 302 - location = response.headers["Location"] - assert expected_redirect.replace("HASH", current_hash) == location - - -def test_hash_parameter_ignored_for_mutable_databases(app_client): - path = "/fixtures/facetable.json?_hash=1" - response = app_client.get(path) - assert response.status == 200 - - test_json_columns_default_expected = [ {"intval": 1, "strval": "s", "floatval": 0.5, "jsonval": '{"foo": "bar"}'} ] diff --git a/tests/test_custom_pages.py b/tests/test_custom_pages.py index 66b7437a..f2cfe394 100644 --- a/tests/test_custom_pages.py +++ b/tests/test_custom_pages.py @@ -21,61 +21,61 @@ def custom_pages_client_with_base_url(): def test_custom_pages_view_name(custom_pages_client): response = custom_pages_client.get("/about") - assert 200 == response.status - assert "ABOUT! view_name:page" == response.text + assert response.status == 200 + assert response.text == "ABOUT! view_name:page" def test_request_is_available(custom_pages_client): response = custom_pages_client.get("/request") - assert 200 == response.status - assert "path:/request" == response.text + assert response.status == 200 + assert response.text == "path:/request" def test_custom_pages_with_base_url(custom_pages_client_with_base_url): response = custom_pages_client_with_base_url.get("/prefix/request") - assert 200 == response.status - assert "path:/prefix/request" == response.text + assert response.status == 200 + assert response.text == "path:/prefix/request" def test_custom_pages_nested(custom_pages_client): response = custom_pages_client.get("/nested/nest") - assert 200 == response.status - assert "Nest!" == response.text + assert response.status == 200 + assert response.text == "Nest!" response = custom_pages_client.get("/nested/nest2") - assert 404 == response.status + assert response.status == 404 def test_custom_status(custom_pages_client): response = custom_pages_client.get("/202") - assert 202 == response.status - assert "202!" == response.text + assert response.status == 202 + assert response.text == "202!" def test_custom_headers(custom_pages_client): response = custom_pages_client.get("/headers") - assert 200 == response.status - assert "foo" == response.headers["x-this-is-foo"] - assert "bar" == response.headers["x-this-is-bar"] - assert "FOOBAR" == response.text + assert response.status == 200 + assert response.headers["x-this-is-foo"] == "foo" + assert response.headers["x-this-is-bar"] == "bar" + assert response.text == "FOOBAR" def test_custom_content_type(custom_pages_client): response = custom_pages_client.get("/atom") - assert 200 == response.status + assert response.status == 200 assert response.headers["content-type"] == "application/xml" - assert "" == response.text + assert response.text == "" def test_redirect(custom_pages_client): response = custom_pages_client.get("/redirect") - assert 302 == response.status - assert "/example" == response.headers["Location"] + assert response.status == 302 + assert response.headers["Location"] == "/example" def test_redirect2(custom_pages_client): response = custom_pages_client.get("/redirect2") - assert 301 == response.status - assert "/example" == response.headers["Location"] + assert response.status == 301 + assert response.headers["Location"] == "/example" @pytest.mark.parametrize( diff --git a/tests/test_html.py b/tests/test_html.py index 76a8423a..6e4c22b1 100644 --- a/tests/test_html.py +++ b/tests/test_html.py @@ -5,7 +5,6 @@ from .fixtures import ( # noqa app_client_base_url_prefix, app_client_shorter_time_limit, app_client_two_attached_databases, - app_client_with_hash, make_app_client, METADATA, ) @@ -101,13 +100,6 @@ def test_not_allowed_methods(): assert response.status == 405 -def test_database_page_redirects_with_url_hash(app_client_with_hash): - response = app_client_with_hash.get("/fixtures") - assert response.status == 302 - response = app_client_with_hash.get("/fixtures", follow_redirects=True) - assert "fixtures" in response.text - - def test_database_page(app_client): response = app_client.get("/fixtures") soup = Soup(response.body, "html.parser") @@ -182,26 +174,6 @@ def test_sql_time_limit(app_client_shorter_time_limit): assert expected_html_fragment in response.text -def test_row_redirects_with_url_hash(app_client_with_hash): - response = app_client_with_hash.get("/fixtures/simple_primary_key/1") - assert response.status == 302 - assert response.headers["Location"].endswith("/1") - response = app_client_with_hash.get( - "/fixtures/simple_primary_key/1", follow_redirects=True - ) - assert response.status == 200 - - -def test_row_strange_table_name_with_url_hash(app_client_with_hash): - response = app_client_with_hash.get("/fixtures/table~2Fwith~2Fslashes~2Ecsv/3") - assert response.status == 302 - assert response.headers["Location"].endswith("/table~2Fwith~2Fslashes~2Ecsv/3") - response = app_client_with_hash.get( - "/fixtures/table~2Fwith~2Fslashes~2Ecsv/3", follow_redirects=True - ) - assert response.status == 200 - - def test_row_page_does_not_truncate(): with make_app_client(settings={"truncate_cells_html": 5}) as client: response = client.get("/fixtures/facetable/1") diff --git a/tests/test_internals_urls.py b/tests/test_internals_urls.py index 4307789c..d60aafcf 100644 --- a/tests/test_internals_urls.py +++ b/tests/test_internals_urls.py @@ -1,6 +1,5 @@ from datasette.app import Datasette from datasette.utils import PrefixedUrlString -from .fixtures import app_client_with_hash import pytest @@ -147,20 +146,3 @@ def test_row(ds, base_url, format, expected): actual = ds.urls.row("_memory", "facetable", "1", format=format) assert actual == expected assert isinstance(actual, PrefixedUrlString) - - -@pytest.mark.parametrize("base_url", ["/", "/prefix/"]) -def test_database_hashed(app_client_with_hash, base_url): - ds = app_client_with_hash.ds - original_base_url = ds._settings["base_url"] - try: - ds._settings["base_url"] = base_url - db_hash = ds.get_database("fixtures").hash - assert len(db_hash) == 64 - expected = f"{base_url}fixtures-{db_hash[:7]}" - assert ds.urls.database("fixtures") == expected - assert ds.urls.table("fixtures", "name") == expected + "/name" - assert ds.urls.query("fixtures", "name") == expected + "/name" - finally: - # Reset this since fixture is shared with other tests - ds._settings["base_url"] = original_base_url diff --git a/tests/test_table_api.py b/tests/test_table_api.py index 3ab369b3..3d0a7fbd 100644 --- a/tests/test_table_api.py +++ b/tests/test_table_api.py @@ -2,7 +2,6 @@ from datasette.utils import detect_json1 from datasette.utils.sqlite import sqlite_version from .fixtures import ( # noqa app_client, - app_client_with_hash, app_client_with_trace, app_client_returned_rows_matches_page_size, generate_compound_rows, @@ -41,13 +40,6 @@ def test_table_not_exists_json(app_client): } == app_client.get("/fixtures/blah.json").json -def test_jsono_redirects_to_shape_objects(app_client_with_hash): - response_1 = app_client_with_hash.get("/fixtures/simple_primary_key.jsono") - response = app_client_with_hash.get(response_1.headers["Location"]) - assert response.status == 302 - assert response.headers["Location"].endswith("?_shape=objects") - - def test_table_shape_arrays(app_client): response = app_client.get("/fixtures/simple_primary_key.json?_shape=arrays") assert [ From 8658c66438ec71edc7e9adc495f4692b937a0f57 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 18 Mar 2022 17:19:31 -0700 Subject: [PATCH 011/952] Show error if --setting hash_urls 1 used, refs #1661 --- datasette/app.py | 15 ++++++++++----- datasette/cli.py | 21 ++++++++++++++++++--- tests/test_cli.py | 7 +++++++ 3 files changed, 35 insertions(+), 8 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 3099ada7..c1c0663d 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -118,11 +118,6 @@ SETTINGS = ( 50, "Time limit for calculating a suggested facet", ), - Setting( - "hash_urls", - False, - "Include DB file contents hash in URLs, for far-future caching", - ), Setting( "allow_facet", True, @@ -177,6 +172,16 @@ SETTINGS = ( ), Setting("base_url", "/", "Datasette URLs should use this base path"), ) +OBSOLETE_SETTINGS = { + option.name: option + for option in ( + Setting( + "hash_urls", + False, + "The hash_urls setting has been removed, try the datasette-hashed-urls plugin instead", + ), + ) +} DEFAULT_SETTINGS = {option.name: option.default for option in SETTINGS} diff --git a/datasette/cli.py b/datasette/cli.py index 61e7ce91..b94ac192 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -12,7 +12,14 @@ from subprocess import call import sys from runpy import run_module import webbrowser -from .app import Datasette, DEFAULT_SETTINGS, SETTINGS, SQLITE_LIMIT_ATTACHED, pm +from .app import ( + OBSOLETE_SETTINGS, + Datasette, + DEFAULT_SETTINGS, + SETTINGS, + SQLITE_LIMIT_ATTACHED, + pm, +) from .utils import ( StartupError, check_connection, @@ -50,8 +57,12 @@ class Config(click.ParamType): return name, value = config.split(":", 1) if name not in DEFAULT_SETTINGS: + if name in OBSOLETE_SETTINGS: + msg = OBSOLETE_SETTINGS[name].help + else: + msg = f"{name} is not a valid option (--help-settings to see all)" self.fail( - f"{name} is not a valid option (--help-settings to see all)", + msg, param, ctx, ) @@ -83,8 +94,12 @@ class Setting(CompositeParamType): def convert(self, config, param, ctx): name, value = config if name not in DEFAULT_SETTINGS: + if name in OBSOLETE_SETTINGS: + msg = OBSOLETE_SETTINGS[name].help + else: + msg = f"{name} is not a valid option (--help-settings to see all)" self.fail( - f"{name} is not a valid option (--help-settings to see all)", + msg, param, ctx, ) diff --git a/tests/test_cli.py b/tests/test_cli.py index 5afe72c1..89e8d044 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -310,3 +310,10 @@ def test_help_settings(): result = runner.invoke(cli, ["--help-settings"]) for setting in SETTINGS: assert setting.name in result.output + + +def test_help_error_on_hash_urls_setting(): + runner = CliRunner() + result = runner.invoke(cli, ["--setting", "hash_urls", 1]) + assert result.exit_code == 2 + assert 'The hash_urls setting has been removed' in result.output From 9979dcd07f9921ac30c4c0b5ea60d09cd1e10556 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 18 Mar 2022 17:25:14 -0700 Subject: [PATCH 012/952] Also remove default_cache_ttl_hashed setting, refs #1661 --- datasette/app.py | 17 +++-------------- datasette/cli.py | 16 ++++++++-------- datasette/url_builder.py | 9 +-------- tests/test_api.py | 2 -- tests/test_cli.py | 7 ++++--- 5 files changed, 16 insertions(+), 35 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index c1c0663d..f52e3283 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -134,11 +134,6 @@ SETTINGS = ( 5, "Default HTTP cache TTL (used in Cache-Control: max-age= header)", ), - Setting( - "default_cache_ttl_hashed", - 365 * 24 * 60 * 60, - "Default HTTP cache TTL for hashed URL pages", - ), Setting("cache_size_kb", 0, "SQLite cache size in KB (0 == use SQLite default)"), Setting( "allow_csv_stream", @@ -172,17 +167,11 @@ SETTINGS = ( ), Setting("base_url", "/", "Datasette URLs should use this base path"), ) +_HASH_URLS_REMOVED = "The hash_urls setting has been removed, try the datasette-hashed-urls plugin instead" OBSOLETE_SETTINGS = { - option.name: option - for option in ( - Setting( - "hash_urls", - False, - "The hash_urls setting has been removed, try the datasette-hashed-urls plugin instead", - ), - ) + "hash_urls": _HASH_URLS_REMOVED, + "default_cache_ttl_hashed": _HASH_URLS_REMOVED, } - DEFAULT_SETTINGS = {option.name: option.default for option in SETTINGS} FAVICON_PATH = app_root / "datasette" / "static" / "favicon.png" diff --git a/datasette/cli.py b/datasette/cli.py index b94ac192..3c6e1b2c 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -57,10 +57,10 @@ class Config(click.ParamType): return name, value = config.split(":", 1) if name not in DEFAULT_SETTINGS: - if name in OBSOLETE_SETTINGS: - msg = OBSOLETE_SETTINGS[name].help - else: - msg = f"{name} is not a valid option (--help-settings to see all)" + msg = ( + OBSOLETE_SETTINGS.get(name) + or f"{name} is not a valid option (--help-settings to see all)" + ) self.fail( msg, param, @@ -94,10 +94,10 @@ class Setting(CompositeParamType): def convert(self, config, param, ctx): name, value = config if name not in DEFAULT_SETTINGS: - if name in OBSOLETE_SETTINGS: - msg = OBSOLETE_SETTINGS[name].help - else: - msg = f"{name} is not a valid option (--help-settings to see all)" + msg = ( + OBSOLETE_SETTINGS.get(name) + or f"{name} is not a valid option (--help-settings to see all)" + ) self.fail( msg, param, diff --git a/datasette/url_builder.py b/datasette/url_builder.py index 9f072462..498ec85d 100644 --- a/datasette/url_builder.py +++ b/datasette/url_builder.py @@ -28,14 +28,7 @@ class Urls: return self.path("-/logout") def database(self, database, format=None): - db = self.ds.databases[database] - if self.ds.setting("hash_urls") and db.hash: - path = self.path( - f"{tilde_encode(database)}-{db.hash[:HASH_LENGTH]}", format=format - ) - else: - path = self.path(tilde_encode(database), format=format) - return path + return self.path(tilde_encode(database), format=format) def table(self, database, table, format=None): path = f"{self.database(database)}/{tilde_encode(table)}" diff --git a/tests/test_api.py b/tests/test_api.py index 46e41afb..d3c94023 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -798,14 +798,12 @@ def test_settings_json(app_client): "allow_facet": True, "suggest_facets": True, "default_cache_ttl": 5, - "default_cache_ttl_hashed": 365 * 24 * 60 * 60, "num_sql_threads": 1, "cache_size_kb": 0, "allow_csv_stream": True, "max_csv_mb": 100, "truncate_cells_html": 2048, "force_https_urls": False, - "hash_urls": False, "template_debug": False, "trace_debug": False, "base_url": "/", diff --git a/tests/test_cli.py b/tests/test_cli.py index 89e8d044..dca65f26 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -312,8 +312,9 @@ def test_help_settings(): assert setting.name in result.output -def test_help_error_on_hash_urls_setting(): +@pytest.mark.parametrize("setting", ("hash_urls", "default_cache_ttl_hashed")) +def test_help_error_on_hash_urls_setting(setting): runner = CliRunner() - result = runner.invoke(cli, ["--setting", "hash_urls", 1]) + result = runner.invoke(cli, ["--setting", setting, 1]) assert result.exit_code == 2 - assert 'The hash_urls setting has been removed' in result.output + assert "The hash_urls setting has been removed" in result.output From 32963018e7edfab1233de7c7076c428d0e5c7813 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 18 Mar 2022 17:33:06 -0700 Subject: [PATCH 013/952] Updated documentation to remove hash_urls, refs #1661 --- docs/cli-reference.rst | 4 ---- docs/performance.rst | 18 +++++++++++------- docs/settings.rst | 27 --------------------------- 3 files changed, 11 insertions(+), 38 deletions(-) diff --git a/docs/cli-reference.rst b/docs/cli-reference.rst index 155a005d..69670d8a 100644 --- a/docs/cli-reference.rst +++ b/docs/cli-reference.rst @@ -142,8 +142,6 @@ datasette serve --help-settings (default=200) facet_suggest_time_limit_ms Time limit for calculating a suggested facet (default=50) - hash_urls Include DB file contents hash in URLs, for far- - future caching (default=False) allow_facet Allow users to specify columns to facet using ?_facet= parameter (default=True) allow_download Allow users to download the original SQLite @@ -152,8 +150,6 @@ datasette serve --help-settings (default=True) default_cache_ttl Default HTTP cache TTL (used in Cache-Control: max-age= header) (default=5) - default_cache_ttl_hashed Default HTTP cache TTL for hashed URL pages - (default=31536000) cache_size_kb SQLite cache size in KB (0 == use SQLite default) (default=0) allow_csv_stream Allow .csv?_stream=1 to download all rows diff --git a/docs/performance.rst b/docs/performance.rst index bcf3208e..d37f1804 100644 --- a/docs/performance.rst +++ b/docs/performance.rst @@ -60,18 +60,22 @@ The :ref:`setting_default_cache_ttl` setting sets the default HTTP cache TTL for You can also change the cache timeout on a per-request basis using the ``?_ttl=10`` query string parameter. This can be useful when you are working with the Datasette JSON API - you may decide that a specific query can be cached for a longer time, or maybe you need to set ``?_ttl=0`` for some requests for example if you are running a SQL ``order by random()`` query. -Hashed URL mode ---------------- +datasette-hashed-urls +--------------------- -When you open a database file in immutable mode using the ``-i`` option, Datasette calculates a SHA-256 hash of the contents of that file on startup. This content hash can then optionally be used to create URLs that are guaranteed to change if the contents of the file changes in the future. This results in URLs that can then be cached indefinitely by both browsers and caching proxies - an enormous potential performance optimization. +If you open a database file in immutable mode using the ``-i`` option, you can be assured that the content of that database will not change for the lifetime of the Datasette server. -You can enable these hashed URLs in two ways: using the :ref:`setting_hash_urls` configuration setting (which affects all requests to Datasette) or via the ``?_hash=1`` query string parameter (which only applies to the current request). +The `datasette-hashed-urls plugin `__ implements an optimization where your database is served with part of the SHA-256 hash of the database contents baked into the URL. -With hashed URLs enabled, any request to e.g. ``/mydatabase/mytable`` will 302 redirect to ``mydatabase-455fe3a/mytable``. The URL containing the hash will be served with a very long cache expire header - configured using :ref:`setting_default_cache_ttl_hashed` which defaults to 365 days. +A database at ``/fixtures`` will instead be served at ``/fixtures-aa7318b``, and a year-long cache expiry header will be returned with those pages. -Since these responses are cached for a long time, you may wish to build API clients against the non-hashed version of these URLs. These 302 redirects are served extremely quickly, so this should still be a performant way to work against the Datasette API. +This will then be cached by both browsers and caching proxies such as Cloudflare or Fastly, providing a potentially significant performance boost. -If you run Datasette behind an `HTTP/2 server push `__ aware proxy such as Cloudflare Datasette will serve the 302 redirects in such a way that the redirected page will be efficiently "pushed" to the browser as part of the response, without the browser needing to make a second HTTP request to fetch the redirected resource. +To install the plugin, run the following:: + + datasette install datasette-hashed-urls .. note:: + Prior to Datasette 0.61 hashed URL mode was a core Datasette feature, enabled using the ``hash_urls`` setting. This implementation has now been removed in favor of the ``datasette-hashed-urls`` plugin. + Prior to Datasette 0.28 hashed URL mode was the default behaviour for Datasette, since all database files were assumed to be immutable and unchanging. From 0.28 onwards the default has been to treat database files as mutable unless explicitly configured otherwise. diff --git a/docs/settings.rst b/docs/settings.rst index da06d6a0..60c4b36d 100644 --- a/docs/settings.rst +++ b/docs/settings.rst @@ -178,17 +178,6 @@ Default HTTP caching max-age header in seconds, used for ``Cache-Control: max-ag datasette mydatabase.db --setting default_cache_ttl 60 -.. _setting_default_cache_ttl_hashed: - -default_cache_ttl_hashed -~~~~~~~~~~~~~~~~~~~~~~~~ - -Default HTTP caching max-age for responses served using using the :ref:`hashed-urls mechanism `. Defaults to 365 days (31536000 seconds). - -:: - - datasette mydatabase.db --setting default_cache_ttl_hashed 10000 - .. _setting_cache_size_kb: cache_size_kb @@ -251,22 +240,6 @@ HTTP but is served to the outside world via a proxy that enables HTTPS. datasette mydatabase.db --setting force_https_urls 1 -.. _setting_hash_urls: - -hash_urls -~~~~~~~~~ - -When enabled, this setting causes Datasette to append a content hash of the -database file to the URL path for every table and query within that database. - -When combined with far-future expire headers this ensures that queries can be -cached forever, safe in the knowledge that any modifications to the database -itself will result in new, uncached URL paths. - -:: - - datasette mydatabase.db --setting hash_urls 1 - .. _setting_template_debug: template_debug From 4e47a2d894b96854348343374c8e97c9d7055cf6 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 18 Mar 2022 18:37:54 -0700 Subject: [PATCH 014/952] Fixed bug where tables with a column called n caused 500 errors Closes #1228 --- datasette/facets.py | 6 +++--- tests/fixtures.py | 33 ++++++++++++++++---------------- tests/test_api.py | 1 + tests/test_csv.py | 32 +++++++++++++++---------------- tests/test_internals_database.py | 10 ++++++++++ tests/test_plugins.py | 6 ++++-- tests/test_table_api.py | 8 ++++++++ 7 files changed, 59 insertions(+), 37 deletions(-) diff --git a/datasette/facets.py b/datasette/facets.py index a1bb4a5f..b15a758c 100644 --- a/datasette/facets.py +++ b/datasette/facets.py @@ -151,10 +151,10 @@ class ColumnFacet(Facet): if column in already_enabled: continue suggested_facet_sql = """ - select {column}, count(*) as n from ( + select {column} as value, count(*) as n from ( {sql} - ) where {column} is not null - group by {column} + ) where value is not null + group by value limit {limit} """.format( column=escape_sqlite(column), sql=self.sql, limit=facet_size + 1 diff --git a/tests/fixtures.py b/tests/fixtures.py index 342a3020..e0e4ec7b 100644 --- a/tests/fixtures.py +++ b/tests/fixtures.py @@ -564,26 +564,27 @@ CREATE TABLE facetable ( tags text, complex_array text, distinct_some_null, + n text, FOREIGN KEY ("_city_id") REFERENCES [facet_cities](id) ); INSERT INTO facetable - (created, planet_int, on_earth, state, _city_id, _neighborhood, tags, complex_array, distinct_some_null) + (created, planet_int, on_earth, state, _city_id, _neighborhood, tags, complex_array, distinct_some_null, n) VALUES - ("2019-01-14 08:00:00", 1, 1, 'CA', 1, 'Mission', '["tag1", "tag2"]', '[{"foo": "bar"}]', 'one'), - ("2019-01-14 08:00:00", 1, 1, 'CA', 1, 'Dogpatch', '["tag1", "tag3"]', '[]', 'two'), - ("2019-01-14 08:00:00", 1, 1, 'CA', 1, 'SOMA', '[]', '[]', null), - ("2019-01-14 08:00:00", 1, 1, 'CA', 1, 'Tenderloin', '[]', '[]', null), - ("2019-01-15 08:00:00", 1, 1, 'CA', 1, 'Bernal Heights', '[]', '[]', null), - ("2019-01-15 08:00:00", 1, 1, 'CA', 1, 'Hayes Valley', '[]', '[]', null), - ("2019-01-15 08:00:00", 1, 1, 'CA', 2, 'Hollywood', '[]', '[]', null), - ("2019-01-15 08:00:00", 1, 1, 'CA', 2, 'Downtown', '[]', '[]', null), - ("2019-01-16 08:00:00", 1, 1, 'CA', 2, 'Los Feliz', '[]', '[]', null), - ("2019-01-16 08:00:00", 1, 1, 'CA', 2, 'Koreatown', '[]', '[]', null), - ("2019-01-16 08:00:00", 1, 1, 'MI', 3, 'Downtown', '[]', '[]', null), - ("2019-01-17 08:00:00", 1, 1, 'MI', 3, 'Greektown', '[]', '[]', null), - ("2019-01-17 08:00:00", 1, 1, 'MI', 3, 'Corktown', '[]', '[]', null), - ("2019-01-17 08:00:00", 1, 1, 'MI', 3, 'Mexicantown', '[]', '[]', null), - ("2019-01-17 08:00:00", 2, 0, 'MC', 4, 'Arcadia Planitia', '[]', '[]', null) + ("2019-01-14 08:00:00", 1, 1, 'CA', 1, 'Mission', '["tag1", "tag2"]', '[{"foo": "bar"}]', 'one', 'n1'), + ("2019-01-14 08:00:00", 1, 1, 'CA', 1, 'Dogpatch', '["tag1", "tag3"]', '[]', 'two', 'n2'), + ("2019-01-14 08:00:00", 1, 1, 'CA', 1, 'SOMA', '[]', '[]', null, null), + ("2019-01-14 08:00:00", 1, 1, 'CA', 1, 'Tenderloin', '[]', '[]', null, null), + ("2019-01-15 08:00:00", 1, 1, 'CA', 1, 'Bernal Heights', '[]', '[]', null, null), + ("2019-01-15 08:00:00", 1, 1, 'CA', 1, 'Hayes Valley', '[]', '[]', null, null), + ("2019-01-15 08:00:00", 1, 1, 'CA', 2, 'Hollywood', '[]', '[]', null, null), + ("2019-01-15 08:00:00", 1, 1, 'CA', 2, 'Downtown', '[]', '[]', null, null), + ("2019-01-16 08:00:00", 1, 1, 'CA', 2, 'Los Feliz', '[]', '[]', null, null), + ("2019-01-16 08:00:00", 1, 1, 'CA', 2, 'Koreatown', '[]', '[]', null, null), + ("2019-01-16 08:00:00", 1, 1, 'MI', 3, 'Downtown', '[]', '[]', null, null), + ("2019-01-17 08:00:00", 1, 1, 'MI', 3, 'Greektown', '[]', '[]', null, null), + ("2019-01-17 08:00:00", 1, 1, 'MI', 3, 'Corktown', '[]', '[]', null, null), + ("2019-01-17 08:00:00", 1, 1, 'MI', 3, 'Mexicantown', '[]', '[]', null, null), + ("2019-01-17 08:00:00", 2, 0, 'MC', 4, 'Arcadia Planitia', '[]', '[]', null, null) ; CREATE TABLE binary_data ( diff --git a/tests/test_api.py b/tests/test_api.py index d3c94023..421bb1fe 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -210,6 +210,7 @@ def test_database_page(app_client): "tags", "complex_array", "distinct_some_null", + "n", ], "primary_keys": ["pk"], "count": 15, diff --git a/tests/test_csv.py b/tests/test_csv.py index 8749cd8b..7fc25a09 100644 --- a/tests/test_csv.py +++ b/tests/test_csv.py @@ -24,22 +24,22 @@ world ) EXPECTED_TABLE_WITH_LABELS_CSV = """ -pk,created,planet_int,on_earth,state,_city_id,_city_id_label,_neighborhood,tags,complex_array,distinct_some_null -1,2019-01-14 08:00:00,1,1,CA,1,San Francisco,Mission,"[""tag1"", ""tag2""]","[{""foo"": ""bar""}]",one -2,2019-01-14 08:00:00,1,1,CA,1,San Francisco,Dogpatch,"[""tag1"", ""tag3""]",[],two -3,2019-01-14 08:00:00,1,1,CA,1,San Francisco,SOMA,[],[], -4,2019-01-14 08:00:00,1,1,CA,1,San Francisco,Tenderloin,[],[], -5,2019-01-15 08:00:00,1,1,CA,1,San Francisco,Bernal Heights,[],[], -6,2019-01-15 08:00:00,1,1,CA,1,San Francisco,Hayes Valley,[],[], -7,2019-01-15 08:00:00,1,1,CA,2,Los Angeles,Hollywood,[],[], -8,2019-01-15 08:00:00,1,1,CA,2,Los Angeles,Downtown,[],[], -9,2019-01-16 08:00:00,1,1,CA,2,Los Angeles,Los Feliz,[],[], -10,2019-01-16 08:00:00,1,1,CA,2,Los Angeles,Koreatown,[],[], -11,2019-01-16 08:00:00,1,1,MI,3,Detroit,Downtown,[],[], -12,2019-01-17 08:00:00,1,1,MI,3,Detroit,Greektown,[],[], -13,2019-01-17 08:00:00,1,1,MI,3,Detroit,Corktown,[],[], -14,2019-01-17 08:00:00,1,1,MI,3,Detroit,Mexicantown,[],[], -15,2019-01-17 08:00:00,2,0,MC,4,Memnonia,Arcadia Planitia,[],[], +pk,created,planet_int,on_earth,state,_city_id,_city_id_label,_neighborhood,tags,complex_array,distinct_some_null,n +1,2019-01-14 08:00:00,1,1,CA,1,San Francisco,Mission,"[""tag1"", ""tag2""]","[{""foo"": ""bar""}]",one,n1 +2,2019-01-14 08:00:00,1,1,CA,1,San Francisco,Dogpatch,"[""tag1"", ""tag3""]",[],two,n2 +3,2019-01-14 08:00:00,1,1,CA,1,San Francisco,SOMA,[],[],, +4,2019-01-14 08:00:00,1,1,CA,1,San Francisco,Tenderloin,[],[],, +5,2019-01-15 08:00:00,1,1,CA,1,San Francisco,Bernal Heights,[],[],, +6,2019-01-15 08:00:00,1,1,CA,1,San Francisco,Hayes Valley,[],[],, +7,2019-01-15 08:00:00,1,1,CA,2,Los Angeles,Hollywood,[],[],, +8,2019-01-15 08:00:00,1,1,CA,2,Los Angeles,Downtown,[],[],, +9,2019-01-16 08:00:00,1,1,CA,2,Los Angeles,Los Feliz,[],[],, +10,2019-01-16 08:00:00,1,1,CA,2,Los Angeles,Koreatown,[],[],, +11,2019-01-16 08:00:00,1,1,MI,3,Detroit,Downtown,[],[],, +12,2019-01-17 08:00:00,1,1,MI,3,Detroit,Greektown,[],[],, +13,2019-01-17 08:00:00,1,1,MI,3,Detroit,Corktown,[],[],, +14,2019-01-17 08:00:00,1,1,MI,3,Detroit,Mexicantown,[],[],, +15,2019-01-17 08:00:00,2,0,MC,4,Memnonia,Arcadia Planitia,[],[],, """.lstrip().replace( "\n", "\r\n" ) diff --git a/tests/test_internals_database.py b/tests/test_internals_database.py index 31538a24..551f67e1 100644 --- a/tests/test_internals_database.py +++ b/tests/test_internals_database.py @@ -86,6 +86,7 @@ async def test_table_exists(db, tables, exists): "tags", "complex_array", "distinct_some_null", + "n", ], ), ( @@ -204,6 +205,15 @@ async def test_table_columns(db, table, expected): is_pk=0, hidden=0, ), + Column( + cid=10, + name="n", + type="text", + notnull=0, + default_value=None, + is_pk=0, + hidden=0, + ), ], ), ( diff --git a/tests/test_plugins.py b/tests/test_plugins.py index 656f39e4..15bde962 100644 --- a/tests/test_plugins.py +++ b/tests/test_plugins.py @@ -442,6 +442,7 @@ def test_hook_register_output_renderer_all_parameters(app_client): "tags", "complex_array", "distinct_some_null", + "n", ], "rows": [ "", @@ -460,7 +461,7 @@ def test_hook_register_output_renderer_all_parameters(app_client): "", "", ], - "sql": "select pk, created, planet_int, on_earth, state, _city_id, _neighborhood, tags, complex_array, distinct_some_null from facetable order by pk limit 51", + "sql": "select pk, created, planet_int, on_earth, state, _city_id, _neighborhood, tags, complex_array, distinct_some_null, n from facetable order by pk limit 51", "query_name": None, "database": "fixtures", "table": "facetable", @@ -531,8 +532,9 @@ def test_hook_register_output_renderer_can_render(app_client): "tags", "complex_array", "distinct_some_null", + "n", ], - "sql": "select pk, created, planet_int, on_earth, state, _city_id, _neighborhood, tags, complex_array, distinct_some_null from facetable order by pk limit 51", + "sql": "select pk, created, planet_int, on_earth, state, _city_id, _neighborhood, tags, complex_array, distinct_some_null, n from facetable order by pk limit 51", "query_name": None, "database": "fixtures", "table": "facetable", diff --git a/tests/test_table_api.py b/tests/test_table_api.py index 3d0a7fbd..9db383c3 100644 --- a/tests/test_table_api.py +++ b/tests/test_table_api.py @@ -532,6 +532,7 @@ def test_table_filter_json_arraycontains(app_client): '["tag1", "tag2"]', '[{"foo": "bar"}]', "one", + "n1", ], [ 2, @@ -544,6 +545,7 @@ def test_table_filter_json_arraycontains(app_client): '["tag1", "tag3"]', "[]", "two", + "n2", ], ] @@ -565,6 +567,7 @@ def test_table_filter_json_arraynotcontains(app_client): '["tag1", "tag2"]', '[{"foo": "bar"}]', "one", + "n1", ] ] @@ -585,6 +588,7 @@ def test_table_filter_extra_where(app_client): '["tag1", "tag3"]', "[]", "two", + "n2", ] ] == response.json["rows"] @@ -958,6 +962,7 @@ def test_expand_labels(app_client): "tags": '["tag1", "tag3"]', "complex_array": "[]", "distinct_some_null": "two", + "n": "n2", }, "13": { "pk": 13, @@ -970,6 +975,7 @@ def test_expand_labels(app_client): "tags": "[]", "complex_array": "[]", "distinct_some_null": None, + "n": None, }, } == response.json @@ -1161,6 +1167,7 @@ def test_generated_columns_are_visible_in_datasette(): "tags", "complex_array", "distinct_some_null", + "n", ], ), ( @@ -1188,6 +1195,7 @@ def test_generated_columns_are_visible_in_datasette(): "tags", "complex_array", "distinct_some_null", + "n", ], ), ( From 711767bcd3c1e76a0861fe7f24069ff1c8efc97a Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 18 Mar 2022 21:03:08 -0700 Subject: [PATCH 015/952] Refactored URL routing to add tests, closes #1666 Refs #1660 --- datasette/app.py | 54 ++++++++++++++++++++----------------- datasette/utils/__init__.py | 8 ++++++ tests/test_routes.py | 34 +++++++++++++++++++++++ 3 files changed, 72 insertions(+), 24 deletions(-) create mode 100644 tests/test_routes.py diff --git a/datasette/app.py b/datasette/app.py index f52e3283..8987112c 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -60,6 +60,7 @@ from .utils import ( module_from_path, parse_metadata, resolve_env_secrets, + resolve_routes, to_css_class, ) from .utils.asgi import ( @@ -974,8 +975,7 @@ class Datasette: output.append(script) return output - def app(self): - """Returns an ASGI app function that serves the whole of Datasette""" + def _routes(self): routes = [] for routes_to_add in pm.hook.register_routes(datasette=self): @@ -1099,6 +1099,15 @@ class Datasette: + renderer_regex + r")?$", ) + return [ + # Compile any strings to regular expressions + ((re.compile(pattern) if isinstance(pattern, str) else pattern), view) + for pattern, view in routes + ] + + def app(self): + """Returns an ASGI app function that serves the whole of Datasette""" + routes = self._routes() self._register_custom_units() async def setup_db(): @@ -1129,12 +1138,7 @@ class Datasette: class DatasetteRouter: def __init__(self, datasette, routes): self.ds = datasette - routes = routes or [] - self.routes = [ - # Compile any strings to regular expressions - ((re.compile(pattern) if isinstance(pattern, str) else pattern), view) - for pattern, view in routes - ] + self.routes = routes or [] # Build a list of pages/blah/{name}.html matching expressions pattern_templates = [ filepath @@ -1187,22 +1191,24 @@ class DatasetteRouter: break scope_modifications["actor"] = actor or default_actor scope = dict(scope, **scope_modifications) - for regex, view in self.routes: - match = regex.match(path) - if match is not None: - new_scope = dict(scope, url_route={"kwargs": match.groupdict()}) - request.scope = new_scope - try: - response = await view(request, send) - if response: - self.ds._write_messages_to_response(request, response) - await response.asgi_send(send) - return - except NotFound as exception: - return await self.handle_404(request, send, exception) - except Exception as exception: - return await self.handle_500(request, send, exception) - return await self.handle_404(request, send) + + match, view = resolve_routes(self.routes, path) + + if match is None: + return await self.handle_404(request, send) + + new_scope = dict(scope, url_route={"kwargs": match.groupdict()}) + request.scope = new_scope + try: + response = await view(request, send) + if response: + self.ds._write_messages_to_response(request, response) + await response.asgi_send(send) + return + except NotFound as exception: + return await self.handle_404(request, send, exception) + except Exception as exception: + return await self.handle_500(request, send, exception) async def handle_404(self, request, send, exception=None): # If path contains % encoding, redirect to tilde encoding diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index bd591459..ccdf8ad4 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -1178,3 +1178,11 @@ def tilde_decode(s: str) -> str: s = s.replace("%", temp) decoded = urllib.parse.unquote(s.replace("~", "%")) return decoded.replace(temp, "%") + + +def resolve_routes(routes, path): + for regex, view in routes: + match = regex.match(path) + if match is not None: + return match, view + return None, None diff --git a/tests/test_routes.py b/tests/test_routes.py new file mode 100644 index 00000000..a1960f14 --- /dev/null +++ b/tests/test_routes.py @@ -0,0 +1,34 @@ +from datasette.app import Datasette +from datasette.utils import resolve_routes +import pytest + + +@pytest.fixture(scope="session") +def routes(): + ds = Datasette() + return ds._routes() + + +@pytest.mark.parametrize( + "path,expected", + ( + ("/", "IndexView"), + ("/foo", "DatabaseView"), + ("/foo.csv", "DatabaseView"), + ("/foo.json", "DatabaseView"), + ("/foo.humbug", "DatabaseView"), + ("/foo/humbug", "TableView"), + ("/foo/humbug.json", "TableView"), + ("/foo/humbug.blah", "TableView"), + ("/foo/humbug/1", "RowView"), + ("/foo/humbug/1.json", "RowView"), + ("/-/metadata.json", "JsonDataView"), + ("/-/metadata", "JsonDataView"), + ), +) +def test_routes(routes, path, expected): + match, view = resolve_routes(routes, path) + if expected is None: + assert match is None + else: + assert view.view_class.__name__ == expected From 764738dfcb16cd98b0987d443f59d5baa9d3c332 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sat, 19 Mar 2022 09:30:22 -0700 Subject: [PATCH 016/952] test_routes also now asserts matches, refs #1666 --- tests/test_routes.py | 41 +++++++++++++++++++++++++---------------- 1 file changed, 25 insertions(+), 16 deletions(-) diff --git a/tests/test_routes.py b/tests/test_routes.py index a1960f14..6718c232 100644 --- a/tests/test_routes.py +++ b/tests/test_routes.py @@ -10,25 +10,34 @@ def routes(): @pytest.mark.parametrize( - "path,expected", + "path,expected_class,expected_matches", ( - ("/", "IndexView"), - ("/foo", "DatabaseView"), - ("/foo.csv", "DatabaseView"), - ("/foo.json", "DatabaseView"), - ("/foo.humbug", "DatabaseView"), - ("/foo/humbug", "TableView"), - ("/foo/humbug.json", "TableView"), - ("/foo/humbug.blah", "TableView"), - ("/foo/humbug/1", "RowView"), - ("/foo/humbug/1.json", "RowView"), - ("/-/metadata.json", "JsonDataView"), - ("/-/metadata", "JsonDataView"), + ("/", "IndexView", {"as_format": ""}), + ("/foo", "DatabaseView", {"as_format": None, "db_name": "foo"}), + ("/foo.csv", "DatabaseView", {"as_format": ".csv", "db_name": "foo"}), + ("/foo.json", "DatabaseView", {"as_format": ".json", "db_name": "foo"}), + ("/foo.humbug", "DatabaseView", {"as_format": None, "db_name": "foo.humbug"}), + ("/foo/humbug", "TableView", {"db_name": "foo", "table": "humbug"}), + ("/foo/humbug.json", "TableView", {"db_name": "foo", "table": "humbug"}), + ("/foo/humbug.blah", "TableView", {"db_name": "foo", "table": "humbug"}), + ( + "/foo/humbug/1", + "RowView", + {"as_format": None, "db_name": "foo", "pk_path": "1", "table": "humbug"}, + ), + ( + "/foo/humbug/1.json", + "RowView", + {"as_format": ".json", "db_name": "foo", "pk_path": "1", "table": "humbug"}, + ), + ("/-/metadata.json", "JsonDataView", {"as_format": ".json"}), + ("/-/metadata", "JsonDataView", {"as_format": ""}), ), ) -def test_routes(routes, path, expected): +def test_routes(routes, path, expected_class, expected_matches): match, view = resolve_routes(routes, path) - if expected is None: + if expected_class is None: assert match is None else: - assert view.view_class.__name__ == expected + assert view.view_class.__name__ == expected_class + assert match.groupdict() == expected_matches From 61419388c134001118aaf7dfb913562d467d7913 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sat, 19 Mar 2022 09:52:08 -0700 Subject: [PATCH 017/952] Rename route match groups for consistency, refs #1667, #1660 --- datasette/app.py | 28 ++++++++++++---------------- datasette/blob_renderer.py | 4 ++-- datasette/views/base.py | 2 +- datasette/views/database.py | 6 +++--- datasette/views/index.py | 2 +- datasette/views/special.py | 2 +- datasette/views/table.py | 8 ++++---- tests/test_routes.py | 24 ++++++++++++------------ 8 files changed, 36 insertions(+), 40 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 8987112c..5259c50c 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -988,7 +988,7 @@ class Datasette: # Generate a regex snippet to match all registered renderer file extensions renderer_regex = "|".join(r"\." + key for key in self.renderers.keys()) - add_route(IndexView.as_view(self), r"/(?P(\.jsono?)?$)") + add_route(IndexView.as_view(self), r"/(?P(\.jsono?)?$)") # TODO: /favicon.ico and /-/static/ deserve far-future cache expires add_route(favicon, "/favicon.ico") @@ -1020,21 +1020,21 @@ class Datasette: ) add_route( JsonDataView.as_view(self, "metadata.json", lambda: self.metadata()), - r"/-/metadata(?P(\.json)?)$", + r"/-/metadata(?P(\.json)?)$", ) add_route( JsonDataView.as_view(self, "versions.json", self._versions), - r"/-/versions(?P(\.json)?)$", + r"/-/versions(?P(\.json)?)$", ) add_route( JsonDataView.as_view( self, "plugins.json", self._plugins, needs_request=True ), - r"/-/plugins(?P(\.json)?)$", + r"/-/plugins(?P(\.json)?)$", ) add_route( JsonDataView.as_view(self, "settings.json", lambda: self._settings), - r"/-/settings(?P(\.json)?)$", + r"/-/settings(?P(\.json)?)$", ) add_route( permanent_redirect("/-/settings.json"), @@ -1046,15 +1046,15 @@ class Datasette: ) add_route( JsonDataView.as_view(self, "threads.json", self._threads), - r"/-/threads(?P(\.json)?)$", + r"/-/threads(?P(\.json)?)$", ) add_route( JsonDataView.as_view(self, "databases.json", self._connected_databases), - r"/-/databases(?P(\.json)?)$", + r"/-/databases(?P(\.json)?)$", ) add_route( JsonDataView.as_view(self, "actor.json", self._actor, needs_request=True), - r"/-/actor(?P(\.json)?)$", + r"/-/actor(?P(\.json)?)$", ) add_route( AuthTokenView.as_view(self), @@ -1080,22 +1080,18 @@ class Datasette: PatternPortfolioView.as_view(self), r"/-/patterns$", ) - add_route( - DatabaseDownload.as_view(self), r"/(?P[^/]+?)(?P\.db)$" - ) + add_route(DatabaseDownload.as_view(self), r"/(?P[^/]+?)\.db$") add_route( DatabaseView.as_view(self), - r"/(?P[^/]+?)(?P" - + renderer_regex - + r"|.jsono|\.csv)?$", + r"/(?P[^/]+?)(?P" + renderer_regex + r"|.jsono|\.csv)?$", ) add_route( TableView.as_view(self), - r"/(?P[^/]+)/(?P
[^\/\.]+)(\.[a-zA-Z0-9_]+)?$", + r"/(?P[^/]+)/(?P
[^\/\.]+)(\.[a-zA-Z0-9_]+)?$", ) add_route( RowView.as_view(self), - r"/(?P[^/]+)/(?P
[^/]+?)/(?P[^/]+?)(?P" + r"/(?P[^/]+)/(?P
[^/]+?)/(?P[^/]+?)(?P" + renderer_regex + r")?$", ) diff --git a/datasette/blob_renderer.py b/datasette/blob_renderer.py index 217b3638..4d8c6bea 100644 --- a/datasette/blob_renderer.py +++ b/datasette/blob_renderer.py @@ -34,8 +34,8 @@ async def render_blob(datasette, database, rows, columns, request, table, view_n filename_bits = [] if table: filename_bits.append(to_css_class(table)) - if "pk_path" in request.url_vars: - filename_bits.append(request.url_vars["pk_path"]) + if "pks" in request.url_vars: + filename_bits.append(request.url_vars["pks"]) filename_bits.append(to_css_class(blob_column)) if blob_hash: filename_bits.append(blob_hash[:6]) diff --git a/datasette/views/base.py b/datasette/views/base.py index e31beb19..0bbf98bb 100644 --- a/datasette/views/base.py +++ b/datasette/views/base.py @@ -381,7 +381,7 @@ class DataView(BaseView): return None async def get(self, request): - db_name = request.url_vars["db_name"] + db_name = request.url_vars["database"] database = tilde_decode(db_name) _format = self.get_format(request) data_kwargs = {} diff --git a/datasette/views/database.py b/datasette/views/database.py index 48635e01..93bd1011 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -32,7 +32,7 @@ class DatabaseView(DataView): name = "database" async def data(self, request, default_labels=False, _size=None): - database = tilde_decode(request.url_vars["db_name"]) + database = tilde_decode(request.url_vars["database"]) await self.check_permissions( request, [ @@ -162,7 +162,7 @@ class DatabaseDownload(DataView): name = "database_download" async def get(self, request): - database = tilde_decode(request.url_vars["db_name"]) + database = tilde_decode(request.url_vars["database"]) await self.check_permissions( request, [ @@ -205,7 +205,7 @@ class QueryView(DataView): named_parameters=None, write=False, ): - database = tilde_decode(request.url_vars["db_name"]) + database = tilde_decode(request.url_vars["database"]) params = {key: request.args.get(key) for key in request.args} if "sql" in params: params.pop("sql") diff --git a/datasette/views/index.py b/datasette/views/index.py index 311a49db..f5e31181 100644 --- a/datasette/views/index.py +++ b/datasette/views/index.py @@ -19,7 +19,7 @@ class IndexView(BaseView): name = "index" async def get(self, request): - as_format = request.url_vars["as_format"] + as_format = request.url_vars["format"] await self.check_permission(request, "view-instance") databases = [] for name, db in self.ds.databases.items(): diff --git a/datasette/views/special.py b/datasette/views/special.py index c7b5061f..395ee587 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -15,7 +15,7 @@ class JsonDataView(BaseView): self.needs_request = needs_request async def get(self, request): - as_format = request.url_vars["as_format"] + as_format = request.url_vars["format"] await self.check_permission(request, "view-instance") if self.needs_request: data = self.data_callback(request) diff --git a/datasette/views/table.py b/datasette/views/table.py index 8bdc7417..ea4f24b7 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -272,7 +272,7 @@ class TableView(RowTableShared): name = "table" async def post(self, request): - db_name = tilde_decode(request.url_vars["db_name"]) + db_name = tilde_decode(request.url_vars["database"]) table = tilde_decode(request.url_vars["table"]) # Handle POST to a canned query canned_query = await self.ds.get_canned_query(db_name, table, request.actor) @@ -327,7 +327,7 @@ class TableView(RowTableShared): _next=None, _size=None, ): - database = tilde_decode(request.url_vars["db_name"]) + database = tilde_decode(request.url_vars["database"]) table = tilde_decode(request.url_vars["table"]) try: db = self.ds.databases[database] @@ -938,7 +938,7 @@ class RowView(RowTableShared): name = "row" async def data(self, request, default_labels=False): - database = tilde_decode(request.url_vars["db_name"]) + database = tilde_decode(request.url_vars["database"]) table = tilde_decode(request.url_vars["table"]) await self.check_permissions( request, @@ -948,7 +948,7 @@ class RowView(RowTableShared): "view-instance", ], ) - pk_values = urlsafe_components(request.url_vars["pk_path"]) + pk_values = urlsafe_components(request.url_vars["pks"]) db = self.ds.databases[database] sql, params, pks = await _sql_params_pks(db, table, pk_values) results = await db.execute(sql, params, truncate=True) diff --git a/tests/test_routes.py b/tests/test_routes.py index 6718c232..349ac302 100644 --- a/tests/test_routes.py +++ b/tests/test_routes.py @@ -12,26 +12,26 @@ def routes(): @pytest.mark.parametrize( "path,expected_class,expected_matches", ( - ("/", "IndexView", {"as_format": ""}), - ("/foo", "DatabaseView", {"as_format": None, "db_name": "foo"}), - ("/foo.csv", "DatabaseView", {"as_format": ".csv", "db_name": "foo"}), - ("/foo.json", "DatabaseView", {"as_format": ".json", "db_name": "foo"}), - ("/foo.humbug", "DatabaseView", {"as_format": None, "db_name": "foo.humbug"}), - ("/foo/humbug", "TableView", {"db_name": "foo", "table": "humbug"}), - ("/foo/humbug.json", "TableView", {"db_name": "foo", "table": "humbug"}), - ("/foo/humbug.blah", "TableView", {"db_name": "foo", "table": "humbug"}), + ("/", "IndexView", {"format": ""}), + ("/foo", "DatabaseView", {"format": None, "database": "foo"}), + ("/foo.csv", "DatabaseView", {"format": ".csv", "database": "foo"}), + ("/foo.json", "DatabaseView", {"format": ".json", "database": "foo"}), + ("/foo.humbug", "DatabaseView", {"format": None, "database": "foo.humbug"}), + ("/foo/humbug", "TableView", {"database": "foo", "table": "humbug"}), + ("/foo/humbug.json", "TableView", {"database": "foo", "table": "humbug"}), + ("/foo/humbug.blah", "TableView", {"database": "foo", "table": "humbug"}), ( "/foo/humbug/1", "RowView", - {"as_format": None, "db_name": "foo", "pk_path": "1", "table": "humbug"}, + {"format": None, "database": "foo", "pks": "1", "table": "humbug"}, ), ( "/foo/humbug/1.json", "RowView", - {"as_format": ".json", "db_name": "foo", "pk_path": "1", "table": "humbug"}, + {"format": ".json", "database": "foo", "pks": "1", "table": "humbug"}, ), - ("/-/metadata.json", "JsonDataView", {"as_format": ".json"}), - ("/-/metadata", "JsonDataView", {"as_format": ""}), + ("/-/metadata.json", "JsonDataView", {"format": ".json"}), + ("/-/metadata", "JsonDataView", {"format": ""}), ), ) def test_routes(routes, path, expected_class, expected_matches): From b9c2b1cfc8692b9700416db98721fa3ec982f6be Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sat, 19 Mar 2022 13:29:10 -0700 Subject: [PATCH 018/952] Consistent treatment of format in route capturing, refs #1667 Also refs #1660 --- datasette/app.py | 30 ++++++++++++------------------ tests/test_api.py | 4 ++-- tests/test_routes.py | 32 ++++++++++++++++++++++---------- 3 files changed, 36 insertions(+), 30 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 5259c50c..edef34e9 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -985,10 +985,7 @@ class Datasette: def add_route(view, regex): routes.append((regex, view)) - # Generate a regex snippet to match all registered renderer file extensions - renderer_regex = "|".join(r"\." + key for key in self.renderers.keys()) - - add_route(IndexView.as_view(self), r"/(?P(\.jsono?)?$)") + add_route(IndexView.as_view(self), r"/(\.(?Pjsono?))?$") # TODO: /favicon.ico and /-/static/ deserve far-future cache expires add_route(favicon, "/favicon.ico") @@ -1020,21 +1017,21 @@ class Datasette: ) add_route( JsonDataView.as_view(self, "metadata.json", lambda: self.metadata()), - r"/-/metadata(?P(\.json)?)$", + r"/-/metadata(\.(?Pjson))?$", ) add_route( JsonDataView.as_view(self, "versions.json", self._versions), - r"/-/versions(?P(\.json)?)$", + r"/-/versions(\.(?Pjson))?$", ) add_route( JsonDataView.as_view( self, "plugins.json", self._plugins, needs_request=True ), - r"/-/plugins(?P(\.json)?)$", + r"/-/plugins(\.(?Pjson))?$", ) add_route( JsonDataView.as_view(self, "settings.json", lambda: self._settings), - r"/-/settings(?P(\.json)?)$", + r"/-/settings(\.(?Pjson))?$", ) add_route( permanent_redirect("/-/settings.json"), @@ -1046,15 +1043,15 @@ class Datasette: ) add_route( JsonDataView.as_view(self, "threads.json", self._threads), - r"/-/threads(?P(\.json)?)$", + r"/-/threads(\.(?Pjson))?$", ) add_route( JsonDataView.as_view(self, "databases.json", self._connected_databases), - r"/-/databases(?P(\.json)?)$", + r"/-/databases(\.(?Pjson))?$", ) add_route( JsonDataView.as_view(self, "actor.json", self._actor, needs_request=True), - r"/-/actor(?P(\.json)?)$", + r"/-/actor(\.(?Pjson))?$", ) add_route( AuthTokenView.as_view(self), @@ -1080,20 +1077,17 @@ class Datasette: PatternPortfolioView.as_view(self), r"/-/patterns$", ) - add_route(DatabaseDownload.as_view(self), r"/(?P[^/]+?)\.db$") + add_route(DatabaseDownload.as_view(self), r"/(?P[^\/\.]+)\.db$") add_route( - DatabaseView.as_view(self), - r"/(?P[^/]+?)(?P" + renderer_regex + r"|.jsono|\.csv)?$", + DatabaseView.as_view(self), r"/(?P[^\/\.]+)(\.(?P\w+))?$" ) add_route( TableView.as_view(self), - r"/(?P[^/]+)/(?P
[^\/\.]+)(\.[a-zA-Z0-9_]+)?$", + r"/(?P[^\/\.]+)/(?P
[^\/\.]+)(\.(?P\w+))?$", ) add_route( RowView.as_view(self), - r"/(?P[^/]+)/(?P
[^/]+?)/(?P[^/]+?)(?P" - + renderer_regex - + r")?$", + r"/(?P[^\/\.]+)/(?P
[^/]+?)/(?P[^/]+?)(\.(?P\w+))?$", ) return [ # Compile any strings to regular expressions diff --git a/tests/test_api.py b/tests/test_api.py index 421bb1fe..253c1718 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -629,8 +629,8 @@ def test_old_memory_urls_redirect(app_client_no_files, path, expected_redirect): def test_database_page_for_database_with_dot_in_name(app_client_with_dot): - response = app_client_with_dot.get("/fixtures.dot.json") - assert 200 == response.status + response = app_client_with_dot.get("/fixtures~2Edot.json") + assert response.status == 200 def test_custom_sql(app_client): diff --git a/tests/test_routes.py b/tests/test_routes.py index 349ac302..1fa55018 100644 --- a/tests/test_routes.py +++ b/tests/test_routes.py @@ -12,14 +12,26 @@ def routes(): @pytest.mark.parametrize( "path,expected_class,expected_matches", ( - ("/", "IndexView", {"format": ""}), + ("/", "IndexView", {"format": None}), ("/foo", "DatabaseView", {"format": None, "database": "foo"}), - ("/foo.csv", "DatabaseView", {"format": ".csv", "database": "foo"}), - ("/foo.json", "DatabaseView", {"format": ".json", "database": "foo"}), - ("/foo.humbug", "DatabaseView", {"format": None, "database": "foo.humbug"}), - ("/foo/humbug", "TableView", {"database": "foo", "table": "humbug"}), - ("/foo/humbug.json", "TableView", {"database": "foo", "table": "humbug"}), - ("/foo/humbug.blah", "TableView", {"database": "foo", "table": "humbug"}), + ("/foo.csv", "DatabaseView", {"format": "csv", "database": "foo"}), + ("/foo.json", "DatabaseView", {"format": "json", "database": "foo"}), + ("/foo.humbug", "DatabaseView", {"format": "humbug", "database": "foo"}), + ( + "/foo/humbug", + "TableView", + {"database": "foo", "table": "humbug", "format": None}, + ), + ( + "/foo/humbug.json", + "TableView", + {"database": "foo", "table": "humbug", "format": "json"}, + ), + ( + "/foo/humbug.blah", + "TableView", + {"database": "foo", "table": "humbug", "format": "blah"}, + ), ( "/foo/humbug/1", "RowView", @@ -28,10 +40,10 @@ def routes(): ( "/foo/humbug/1.json", "RowView", - {"format": ".json", "database": "foo", "pks": "1", "table": "humbug"}, + {"format": "json", "database": "foo", "pks": "1", "table": "humbug"}, ), - ("/-/metadata.json", "JsonDataView", {"format": ".json"}), - ("/-/metadata", "JsonDataView", {"format": ""}), + ("/-/metadata.json", "JsonDataView", {"format": "json"}), + ("/-/metadata", "JsonDataView", {"format": None}), ), ) def test_routes(routes, path, expected_class, expected_matches): From 798f075ef9b98819fdb564f9f79c78975a0f71e8 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sat, 19 Mar 2022 13:32:29 -0700 Subject: [PATCH 019/952] Read format from route captures, closes #1667 Refs #1660 --- datasette/utils/__init__.py | 20 -------------------- datasette/views/base.py | 12 +----------- tests/test_utils.py | 25 ------------------------- 3 files changed, 1 insertion(+), 56 deletions(-) diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index ccdf8ad4..c89b9d23 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -731,26 +731,6 @@ def module_from_path(path, name): return mod -async def resolve_table_and_format( - table_and_format, table_exists, allowed_formats=None -): - if allowed_formats is None: - allowed_formats = [] - if "." in table_and_format: - # Check if a table exists with this exact name - it_exists = await table_exists(table_and_format) - if it_exists: - return table_and_format, None - - # Check if table ends with a known format - formats = list(allowed_formats) + ["csv", "jsono"] - for _format in formats: - if table_and_format.endswith(f".{_format}"): - table = table_and_format[: -(len(_format) + 1)] - return table, _format - return table_and_format, None - - def path_with_format( *, request=None, path=None, format=None, extra_qs=None, replace_format=None ): diff --git a/datasette/views/base.py b/datasette/views/base.py index 0bbf98bb..24e97d95 100644 --- a/datasette/views/base.py +++ b/datasette/views/base.py @@ -19,12 +19,10 @@ from datasette.utils import ( LimitedWriter, call_with_supported_arguments, tilde_decode, - tilde_encode, path_from_row_pks, path_with_added_args, path_with_removed_args, path_with_format, - resolve_table_and_format, sqlite3, HASH_LENGTH, ) @@ -372,18 +370,10 @@ class DataView(BaseView): return AsgiStream(stream_fn, headers=headers, content_type=content_type) - def get_format(self, request): - # Format is the bit from the path following the ., if one exists - last_path_component = request.path.split("/")[-1] - if "." in last_path_component: - return last_path_component.split(".")[-1] - else: - return None - async def get(self, request): db_name = request.url_vars["database"] database = tilde_decode(db_name) - _format = self.get_format(request) + _format = request.url_vars["format"] data_kwargs = {} if _format == "csv": diff --git a/tests/test_utils.py b/tests/test_utils.py index 790aadc7..7b41a87f 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -351,31 +351,6 @@ def test_compound_keys_after_sql(): ) -async def table_exists(table): - return table == "exists.csv" - - -@pytest.mark.asyncio -@pytest.mark.parametrize( - "table_and_format,expected_table,expected_format", - [ - ("blah", "blah", None), - ("blah.csv", "blah", "csv"), - ("blah.json", "blah", "json"), - ("blah.baz", "blah.baz", None), - ("exists.csv", "exists.csv", None), - ], -) -async def test_resolve_table_and_format( - table_and_format, expected_table, expected_format -): - actual_table, actual_format = await utils.resolve_table_and_format( - table_and_format, table_exists, ["json"] - ) - assert expected_table == actual_table - assert expected_format == actual_format - - def test_table_columns(): conn = sqlite3.connect(":memory:") conn.executescript( From 7a6654a253dee243518dc542ce4c06dbb0d0801d Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sat, 19 Mar 2022 17:11:17 -0700 Subject: [PATCH 020/952] Databases can now have a .route separate from their .name, refs #1668 --- datasette/app.py | 13 ++++++-- datasette/database.py | 1 + datasette/views/base.py | 12 +++++-- datasette/views/database.py | 18 ++++++----- datasette/views/table.py | 29 ++++++++++++----- docs/internals.rst | 11 ++++--- tests/test_internals_datasette.py | 1 + tests/test_routes.py | 52 ++++++++++++++++++++++++++++++- 8 files changed, 111 insertions(+), 26 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index edef34e9..5c8101a3 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -388,13 +388,18 @@ class Datasette: def unsign(self, signed, namespace="default"): return URLSafeSerializer(self._secret, namespace).loads(signed) - def get_database(self, name=None): + def get_database(self, name=None, route=None): + if route is not None: + matches = [db for db in self.databases.values() if db.route == route] + if not matches: + raise KeyError + return matches[0] if name is None: - # Return first no-_schemas database + # Return first database that isn't "_internal" name = [key for key in self.databases.keys() if key != "_internal"][0] return self.databases[name] - def add_database(self, db, name=None): + def add_database(self, db, name=None, route=None): new_databases = self.databases.copy() if name is None: # Pick a unique name for this database @@ -407,6 +412,7 @@ class Datasette: name = "{}_{}".format(suggestion, i) i += 1 db.name = name + db.route = route or name new_databases[name] = db # don't mutate! that causes race conditions with live import self.databases = new_databases @@ -693,6 +699,7 @@ class Datasette: return [ { "name": d.name, + "route": d.route, "path": d.path, "size": d.size, "is_mutable": d.is_mutable, diff --git a/datasette/database.py b/datasette/database.py index 6ce87215..ba594a8c 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -31,6 +31,7 @@ class Database: self, ds, path=None, is_mutable=False, is_memory=False, memory_name=None ): self.name = None + self.route = None self.ds = ds self.path = path self.is_mutable = is_mutable diff --git a/datasette/views/base.py b/datasette/views/base.py index 24e97d95..afa9eaa6 100644 --- a/datasette/views/base.py +++ b/datasette/views/base.py @@ -371,13 +371,19 @@ class DataView(BaseView): return AsgiStream(stream_fn, headers=headers, content_type=content_type) async def get(self, request): - db_name = request.url_vars["database"] - database = tilde_decode(db_name) + database_route = tilde_decode(request.url_vars["database"]) + + try: + db = self.ds.get_database(route=database_route) + except KeyError: + raise NotFound("Database not found: {}".format(database_route)) + database = db.name + _format = request.url_vars["format"] data_kwargs = {} if _format == "csv": - return await self.as_csv(request, database) + return await self.as_csv(request, database_route) if _format is None: # HTML views default to expanding all foreign key labels diff --git a/datasette/views/database.py b/datasette/views/database.py index 93bd1011..2563c5b2 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -32,7 +32,13 @@ class DatabaseView(DataView): name = "database" async def data(self, request, default_labels=False, _size=None): - database = tilde_decode(request.url_vars["database"]) + database_route = tilde_decode(request.url_vars["database"]) + try: + db = self.ds.get_database(route=database_route) + except KeyError: + raise NotFound("Database not found: {}".format(database_route)) + database = db.name + await self.check_permissions( request, [ @@ -50,11 +56,6 @@ class DatabaseView(DataView): request, sql, _size=_size, metadata=metadata ) - try: - db = self.ds.databases[database] - except KeyError: - raise NotFound("Database not found: {}".format(database)) - table_counts = await db.table_counts(5) hidden_table_names = set(await db.hidden_table_names()) all_foreign_keys = await db.get_all_foreign_keys() @@ -171,9 +172,10 @@ class DatabaseDownload(DataView): "view-instance", ], ) - if database not in self.ds.databases: + try: + db = self.ds.get_database(route=database) + except KeyError: raise DatasetteError("Invalid database", status=404) - db = self.ds.databases[database] if db.is_memory: raise DatasetteError("Cannot download in-memory databases", status=404) if not self.ds.setting("allow_download") or db.is_mutable: diff --git a/datasette/views/table.py b/datasette/views/table.py index ea4f24b7..7fa1da3a 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -272,10 +272,15 @@ class TableView(RowTableShared): name = "table" async def post(self, request): - db_name = tilde_decode(request.url_vars["database"]) + database_route = tilde_decode(request.url_vars["database"]) + try: + db = self.ds.get_database(route=database_route) + except KeyError: + raise NotFound("Database not found: {}".format(database_route)) + database = db.name table = tilde_decode(request.url_vars["table"]) # Handle POST to a canned query - canned_query = await self.ds.get_canned_query(db_name, table, request.actor) + canned_query = await self.ds.get_canned_query(database, table, request.actor) assert canned_query, "You may only POST to a canned query" return await QueryView(self.ds).data( request, @@ -327,12 +332,13 @@ class TableView(RowTableShared): _next=None, _size=None, ): - database = tilde_decode(request.url_vars["database"]) + database_route = tilde_decode(request.url_vars["database"]) table = tilde_decode(request.url_vars["table"]) try: - db = self.ds.databases[database] + db = self.ds.get_database(route=database_route) except KeyError: - raise NotFound("Database not found: {}".format(database)) + raise NotFound("Database not found: {}".format(database_route)) + database = db.name # If this is a canned query, not a table, then dispatch to QueryView instead canned_query = await self.ds.get_canned_query(database, table, request.actor) @@ -938,8 +944,13 @@ class RowView(RowTableShared): name = "row" async def data(self, request, default_labels=False): - database = tilde_decode(request.url_vars["database"]) + database_route = tilde_decode(request.url_vars["database"]) table = tilde_decode(request.url_vars["table"]) + try: + db = self.ds.get_database(route=database_route) + except KeyError: + raise NotFound("Database not found: {}".format(database_route)) + database = db.name await self.check_permissions( request, [ @@ -949,7 +960,11 @@ class RowView(RowTableShared): ], ) pk_values = urlsafe_components(request.url_vars["pks"]) - db = self.ds.databases[database] + try: + db = self.ds.get_database(route=database_route) + except KeyError: + raise NotFound("Database not found: {}".format(database_route)) + database = db.name sql, params, pks = await _sql_params_pks(db, table, pk_values) results = await db.execute(sql, params, truncate=True) columns = [r[0] for r in results.description] diff --git a/docs/internals.rst b/docs/internals.rst index 117cb95c..323256c7 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -307,14 +307,17 @@ Returns the specified database object. Raises a ``KeyError`` if the database doe .. _datasette_add_database: -.add_database(db, name=None) ----------------------------- +.add_database(db, name=None, route=None) +---------------------------------------- ``db`` - datasette.database.Database instance The database to be attached. ``name`` - string, optional - The name to be used for this database - this will be used in the URL path, e.g. ``/dbname``. If not specified Datasette will pick one based on the filename or memory name. + The name to be used for this database . If not specified Datasette will pick one based on the filename or memory name. + +``route`` - string, optional + This will be used in the URL path. If not specified, it will default to the same thing as the ``name``. The ``datasette.add_database(db)`` method lets you add a new database to the current Datasette instance. @@ -371,7 +374,7 @@ Using either of these pattern will result in the in-memory database being served ``name`` - string The name of the database to be removed. -This removes a database that has been previously added. ``name=`` is the unique name of that database, used in its URL path. +This removes a database that has been previously added. ``name=`` is the unique name of that database. .. _datasette_sign: diff --git a/tests/test_internals_datasette.py b/tests/test_internals_datasette.py index adf84be9..cc200a2d 100644 --- a/tests/test_internals_datasette.py +++ b/tests/test_internals_datasette.py @@ -55,6 +55,7 @@ async def test_datasette_constructor(): assert databases == [ { "name": "_memory", + "route": "_memory", "path": None, "size": 0, "is_mutable": False, diff --git a/tests/test_routes.py b/tests/test_routes.py index 1fa55018..dd3bc644 100644 --- a/tests/test_routes.py +++ b/tests/test_routes.py @@ -1,6 +1,7 @@ -from datasette.app import Datasette +from datasette.app import Datasette, Database from datasette.utils import resolve_routes import pytest +import pytest_asyncio @pytest.fixture(scope="session") @@ -53,3 +54,52 @@ def test_routes(routes, path, expected_class, expected_matches): else: assert view.view_class.__name__ == expected_class assert match.groupdict() == expected_matches + + +@pytest_asyncio.fixture +async def ds_with_route(): + ds = Datasette() + ds.remove_database("_memory") + db = Database(ds, is_memory=True, memory_name="route-name-db") + ds.add_database(db, name="name", route="route-name") + await db.execute_write_script( + """ + create table if not exists t (id integer primary key); + insert or replace into t (id) values (1); + """ + ) + return ds + + +@pytest.mark.asyncio +async def test_db_with_route_databases(ds_with_route): + response = await ds_with_route.client.get("/-/databases.json") + assert response.json()[0] == { + "name": "name", + "route": "route-name", + "path": None, + "size": 0, + "is_mutable": True, + "is_memory": True, + "hash": None, + } + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "path,expected_status", + ( + ("/", 200), + ("/name", 404), + ("/name/t", 404), + ("/name/t/1", 404), + ("/route-name", 200), + ("/route-name/t", 200), + ("/route-name/t/1", 200), + ), +) +async def test_db_with_route_that_does_not_match_name( + ds_with_route, path, expected_status +): + response = await ds_with_route.client.get(path) + assert response.status_code == expected_status From e10da9af3595c0a4e09c6f370103571aa4ea106e Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sat, 19 Mar 2022 17:21:56 -0700 Subject: [PATCH 021/952] alternative-route demo, refs #1668 --- .github/workflows/deploy-latest.yml | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/.github/workflows/deploy-latest.yml b/.github/workflows/deploy-latest.yml index 1ae96e89..92aa1c6b 100644 --- a/.github/workflows/deploy-latest.yml +++ b/.github/workflows/deploy-latest.yml @@ -42,6 +42,17 @@ jobs: sphinx-build -b xml . _build sphinx-to-sqlite ../docs.db _build cd .. + - name: Set up the alternate-route demo + run: | + echo ' + from datasette import hookimpl + + @hookimpl + def startup(datasette): + db = datasette.get_database("fixtures2") + db.route = "alternative-route" + ' > plugins/alternative_route.py + cp fixtures.db fixtures2.db - name: Set up Cloud Run uses: google-github-actions/setup-gcloud@master with: @@ -54,7 +65,7 @@ jobs: gcloud config set project datasette-222320 export SUFFIX="-${GITHUB_REF#refs/heads/}" export SUFFIX=${SUFFIX#-main} - datasette publish cloudrun fixtures.db extra_database.db \ + datasette publish cloudrun fixtures.db fixtures2.db extra_database.db \ -m fixtures.json \ --plugins-dir=plugins \ --branch=$GITHUB_SHA \ From cdbae2b93f441653616dd889644c63e4150ceec1 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sat, 19 Mar 2022 17:31:23 -0700 Subject: [PATCH 022/952] Fixed internal links to respect db.route, refs #1668 --- datasette/url_builder.py | 3 ++- datasette/views/table.py | 5 ++--- tests/test_routes.py | 22 +++++++++++++--------- 3 files changed, 17 insertions(+), 13 deletions(-) diff --git a/datasette/url_builder.py b/datasette/url_builder.py index 498ec85d..574bf3c1 100644 --- a/datasette/url_builder.py +++ b/datasette/url_builder.py @@ -28,7 +28,8 @@ class Urls: return self.path("-/logout") def database(self, database, format=None): - return self.path(tilde_encode(database), format=format) + db = self.ds.get_database(database) + return self.path(tilde_encode(db.route), format=format) def table(self, database, table, format=None): path = f"{self.database(database)}/{tilde_encode(table)}" diff --git a/datasette/views/table.py b/datasette/views/table.py index 7fa1da3a..8745c28a 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -141,10 +141,9 @@ class RowTableShared(DataView): "is_special_link_column": is_special_link_column, "raw": pk_path, "value": markupsafe.Markup( - '{flat_pks}'.format( + '{flat_pks}'.format( base_url=base_url, - database=database, - table=tilde_encode(table), + table_path=self.ds.urls.table(database, table), flat_pks=str(markupsafe.escape(pk_path)), flat_pks_quoted=path_from_row_pks(row, pks, not pks), ) diff --git a/tests/test_routes.py b/tests/test_routes.py index dd3bc644..211b77b5 100644 --- a/tests/test_routes.py +++ b/tests/test_routes.py @@ -61,7 +61,7 @@ async def ds_with_route(): ds = Datasette() ds.remove_database("_memory") db = Database(ds, is_memory=True, memory_name="route-name-db") - ds.add_database(db, name="name", route="route-name") + ds.add_database(db, name="original-name", route="custom-route-name") await db.execute_write_script( """ create table if not exists t (id integer primary key); @@ -75,8 +75,8 @@ async def ds_with_route(): async def test_db_with_route_databases(ds_with_route): response = await ds_with_route.client.get("/-/databases.json") assert response.json()[0] == { - "name": "name", - "route": "route-name", + "name": "original-name", + "route": "custom-route-name", "path": None, "size": 0, "is_mutable": True, @@ -90,12 +90,12 @@ async def test_db_with_route_databases(ds_with_route): "path,expected_status", ( ("/", 200), - ("/name", 404), - ("/name/t", 404), - ("/name/t/1", 404), - ("/route-name", 200), - ("/route-name/t", 200), - ("/route-name/t/1", 200), + ("/original-name", 404), + ("/original-name/t", 404), + ("/original-name/t/1", 404), + ("/custom-route-name", 200), + ("/custom-route-name/t", 200), + ("/custom-route-name/t/1", 200), ), ) async def test_db_with_route_that_does_not_match_name( @@ -103,3 +103,7 @@ async def test_db_with_route_that_does_not_match_name( ): response = await ds_with_route.client.get(path) assert response.status_code == expected_status + # There should be links to custom-route-name but none to original-name + if response.status_code == 200: + assert "/custom-route-name" in response.text + assert "/original-name" not in response.text From 5471e3c4914837de957e206d8fb80c9ec383bc2e Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sat, 19 Mar 2022 18:14:40 -0700 Subject: [PATCH 023/952] Release 0.61a0 Refs #957, #1533, #1545, #1576, #1577, #1587, #1601, #1603, #1607, #1612, #1621, #1649, #1654, #1657, #1661, #1668 --- datasette/version.py | 2 +- docs/changelog.rst | 29 +++++++++++++++++++++++++++-- docs/performance.rst | 2 ++ 3 files changed, 30 insertions(+), 3 deletions(-) diff --git a/datasette/version.py b/datasette/version.py index 91224615..ccc1e04b 100644 --- a/datasette/version.py +++ b/datasette/version.py @@ -1,2 +1,2 @@ -__version__ = "0.60.2" +__version__ = "0.61a0" __version_info__ = tuple(__version__.split(".")) diff --git a/docs/changelog.rst b/docs/changelog.rst index c58c8444..0f3d3aff 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -4,14 +4,39 @@ Changelog ========= -.. _v0_60.2: +.. _v0_61_a0: + +0.61a0 (2022-03-19) +------------------- + +- Removed hashed URL mode from Datasette. The new ``datasette-hashed-urls`` plugin can be used to achieve the same result, see :ref:`performance_hashed_urls` for details. (:issue:`1661`) +- Databases can now have a custom path within the Datasette instance that is indpendent of the database name, using the ``db.route`` property. (:issue:`1668`) +- URLs within Datasette now use a different encoding scheme for tables or databases that include "special" characters outside of the range of ``a-zA-Z0-9_-``. This scheme is explained here: :ref:`internals_tilde_encoding`. (:issue:`1657`) +- Table and row HTML pages now include a ```` element and return a ``Link: URL; rel="alternate"; type="application/json+datasette"`` HTTP header pointing to the JSON version of those pages. (:issue:`1533`) +- ``Access-Control-Expose-Headers: Link`` is now added to the CORS headers, allowing remote JavaScript to access that header. +- Canned queries are now shown at the top of the database page, directly below the SQL editor. Previously they were shown at the bottom, below the list of tables. (:issue:`1612`) +- Datasette now has a default favicon. (:issue:`1603`) +- ``sqlite_stat`` tables are now hidden by default. (:issue:`1587`) +- SpatiaLite tables ``data_licenses``, ``KNN`` and ``KNN2`` are now hidden by default. (:issue:`1601`) +- Python 3.6 is no longer supported. (:issue:`1577`) +- Tests now run against Python 3.11-dev. (:issue:`1621`) +- Fixed bug where :ref:`custom pages ` did not work on Windows. Thanks, Robert Christie. (:issue:`1545`) +- SQL query tracing mechanism now works for queries executed in ``asyncio`` sub-tasks, such as those created by ``asyncio.gather()``. (:issue:`1576`) +- :ref:`internals_tracer` mechanism is now documented. +- Common Datasette symbols can now be imported directly from the top-level ``datasette`` package, see :ref:`internals_shortcuts`. Those symbols are ``Response``, ``Forbidden``, ``NotFound``, ``hookimpl``, ``actor_matches_allow``. (:issue:`957`) +- ``/-/versions`` page now returns additional details for libraries used by SpatiaLite. (:issue:`1607`) +- Documentation now links to the `Datasette Tutorials `__. +- Datasette will now also look for SpatiaLite in ``/opt/homebrew`` - thanks, Dan Peterson. (`#1649 `__) +- Datasette is now covered by a `Code of Conduct `__. (:issue:`1654`) + +.. _v0_60_2: 0.60.2 (2022-02-07) ------------------- - Fixed a bug where Datasette would open the same file twice with two different database names if you ran ``datasette file.db file.db``. (:issue:`1632`) -.. _v0_60.1: +.. _v0_60_1: 0.60.1 (2022-01-20) ------------------- diff --git a/docs/performance.rst b/docs/performance.rst index d37f1804..89bbf5ae 100644 --- a/docs/performance.rst +++ b/docs/performance.rst @@ -60,6 +60,8 @@ The :ref:`setting_default_cache_ttl` setting sets the default HTTP cache TTL for You can also change the cache timeout on a per-request basis using the ``?_ttl=10`` query string parameter. This can be useful when you are working with the Datasette JSON API - you may decide that a specific query can be cached for a longer time, or maybe you need to set ``?_ttl=0`` for some requests for example if you are running a SQL ``order by random()`` query. +.. _performance_hashed_urls: + datasette-hashed-urls --------------------- From cb4854a435cc1418665edec2a73664ad74a32017 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sat, 19 Mar 2022 18:17:58 -0700 Subject: [PATCH 024/952] Fixed typo --- docs/changelog.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index 0f3d3aff..9f5a143c 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -10,7 +10,7 @@ Changelog ------------------- - Removed hashed URL mode from Datasette. The new ``datasette-hashed-urls`` plugin can be used to achieve the same result, see :ref:`performance_hashed_urls` for details. (:issue:`1661`) -- Databases can now have a custom path within the Datasette instance that is indpendent of the database name, using the ``db.route`` property. (:issue:`1668`) +- Databases can now have a custom path within the Datasette instance that is independent of the database name, using the ``db.route`` property. (:issue:`1668`) - URLs within Datasette now use a different encoding scheme for tables or databases that include "special" characters outside of the range of ``a-zA-Z0-9_-``. This scheme is explained here: :ref:`internals_tilde_encoding`. (:issue:`1657`) - Table and row HTML pages now include a ```` element and return a ``Link: URL; rel="alternate"; type="application/json+datasette"`` HTTP header pointing to the JSON version of those pages. (:issue:`1533`) - ``Access-Control-Expose-Headers: Link`` is now added to the CORS headers, allowing remote JavaScript to access that header. From 4a4164b81191dec35e423486a208b05a9edc65e4 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sat, 19 Mar 2022 18:23:03 -0700 Subject: [PATCH 025/952] Added another note to the 0.61a0 release notes, refs #1228 --- docs/changelog.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/changelog.rst b/docs/changelog.rst index 9f5a143c..05ad85f2 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -28,6 +28,7 @@ Changelog - Documentation now links to the `Datasette Tutorials `__. - Datasette will now also look for SpatiaLite in ``/opt/homebrew`` - thanks, Dan Peterson. (`#1649 `__) - Datasette is now covered by a `Code of Conduct `__. (:issue:`1654`) +- Fixed error caused when a table had a column named ``n``. (:issue:`1228`) .. _v0_60_2: From e627510b760198ccedba9e5af47a771e847785c9 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 21 Mar 2022 10:13:16 -0700 Subject: [PATCH 026/952] BaseView.check_permissions is now datasette.ensure_permissions, closes #1675 Refs #1660 --- datasette/app.py | 35 +++++++++++++++++++++++++++++++++++ datasette/views/base.py | 26 -------------------------- datasette/views/database.py | 12 ++++++------ datasette/views/table.py | 8 ++++---- docs/internals.rst | 26 ++++++++++++++++++++++++++ 5 files changed, 71 insertions(+), 36 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 5c8101a3..9e509e96 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -1,4 +1,5 @@ import asyncio +from typing import Sequence, Union, Tuple import asgi_csrf import collections import datetime @@ -628,6 +629,40 @@ class Datasette: ) return result + async def ensure_permissions( + self, + actor: dict, + permissions: Sequence[Union[Tuple[str, Union[str, Tuple[str, str]]], str]], + ): + """ + permissions is a list of (action, resource) tuples or 'action' strings + + Raises datasette.Forbidden() if any of the checks fail + """ + for permission in permissions: + if isinstance(permission, str): + action = permission + resource = None + elif isinstance(permission, (tuple, list)) and len(permission) == 2: + action, resource = permission + else: + assert ( + False + ), "permission should be string or tuple of two items: {}".format( + repr(permission) + ) + ok = await self.permission_allowed( + actor, + action, + resource=resource, + default=None, + ) + if ok is not None: + if ok: + return + else: + raise Forbidden(action) + async def execute( self, db_name, diff --git a/datasette/views/base.py b/datasette/views/base.py index afa9eaa6..d1e684a2 100644 --- a/datasette/views/base.py +++ b/datasette/views/base.py @@ -76,32 +76,6 @@ class BaseView: if not ok: raise Forbidden(action) - async def check_permissions(self, request, permissions): - """permissions is a list of (action, resource) tuples or 'action' strings""" - for permission in permissions: - if isinstance(permission, str): - action = permission - resource = None - elif isinstance(permission, (tuple, list)) and len(permission) == 2: - action, resource = permission - else: - assert ( - False - ), "permission should be string or tuple of two items: {}".format( - repr(permission) - ) - ok = await self.ds.permission_allowed( - request.actor, - action, - resource=resource, - default=None, - ) - if ok is not None: - if ok: - return - else: - raise Forbidden(action) - def database_color(self, database): return "ff0000" diff --git a/datasette/views/database.py b/datasette/views/database.py index 2563c5b2..69ed1233 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -39,8 +39,8 @@ class DatabaseView(DataView): raise NotFound("Database not found: {}".format(database_route)) database = db.name - await self.check_permissions( - request, + await self.ds.ensure_permissions( + request.actor, [ ("view-database", database), "view-instance", @@ -164,8 +164,8 @@ class DatabaseDownload(DataView): async def get(self, request): database = tilde_decode(request.url_vars["database"]) - await self.check_permissions( - request, + await self.ds.ensure_permissions( + request.actor, [ ("view-database-download", database), ("view-database", database), @@ -217,8 +217,8 @@ class QueryView(DataView): private = False if canned_query: # Respect canned query permissions - await self.check_permissions( - request, + await self.ds.ensure_permissions( + request.actor, [ ("view-query", (database, canned_query)), ("view-database", database), diff --git a/datasette/views/table.py b/datasette/views/table.py index 8745c28a..84169820 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -360,8 +360,8 @@ class TableView(RowTableShared): raise NotFound(f"Table not found: {table}") # Ensure user has permission to view this table - await self.check_permissions( - request, + await self.ds.ensure_permissions( + request.actor, [ ("view-table", (database, table)), ("view-database", database), @@ -950,8 +950,8 @@ class RowView(RowTableShared): except KeyError: raise NotFound("Database not found: {}".format(database_route)) database = db.name - await self.check_permissions( - request, + await self.ds.ensure_permissions( + request.actor, [ ("view-table", (database, table)), ("view-database", database), diff --git a/docs/internals.rst b/docs/internals.rst index 323256c7..12adde00 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -295,6 +295,32 @@ If neither ``metadata.json`` nor any of the plugins provide an answer to the per See :ref:`permissions` for a full list of permission actions included in Datasette core. +.. _datasette_permission_allowed: + +await .ensure_permissions(actor, permissions) +--------------------------------------------- + +``actor`` - dictionary + The authenticated actor. This is usually ``request.actor``. + +``permissions`` - list + A list of permissions to check. Each permission in that list can be a string ``action`` name or a 2-tuple of ``(action, resource)``. + +This method allows multiple permissions to be checked at onced. It raises a ``datasette.Forbidden`` exception if any of the checks are denied before one of them is explicitly granted. + +This is useful when you need to check multiple permissions at once. For example, an actor should be able to view a table if either one of the following checks returns ``True`` or not a single one of them returns ``False``: + +.. code-block:: python + + await self.ds.ensure_permissions( + request.actor, + [ + ("view-table", (database, table)), + ("view-database", database), + "view-instance", + ] + ) + .. _datasette_get_database: .get_database(name) From dfafce6d962d615d98a7080e546c7b3662ae7d34 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 21 Mar 2022 11:37:27 -0700 Subject: [PATCH 027/952] Display no-opinion permission checks on /-/permissions --- datasette/templates/permissions_debug.html | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/datasette/templates/permissions_debug.html b/datasette/templates/permissions_debug.html index d898ea8c..db709c14 100644 --- a/datasette/templates/permissions_debug.html +++ b/datasette/templates/permissions_debug.html @@ -10,6 +10,9 @@ .check-result-false { color: red; } +.check-result-no-opinion { + color: #aaa; +} .check h2 { font-size: 1em } @@ -38,6 +41,8 @@ {{ check.when }} {% if check.result %} ✓ + {% elif check.result is none %} + none {% else %} ✗ {% endif %} From 194e4f6c3fffde69eb196f8535ca45386b40ec2d Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 21 Mar 2022 11:41:56 -0700 Subject: [PATCH 028/952] Removed check_permission() from BaseView, closes #1677 Refs #1660 --- datasette/app.py | 1 + datasette/views/base.py | 10 ---------- datasette/views/database.py | 2 +- datasette/views/index.py | 2 +- datasette/views/special.py | 10 +++++----- tests/test_permissions.py | 13 ++++++++----- 6 files changed, 16 insertions(+), 22 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 9e509e96..22ae211f 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -639,6 +639,7 @@ class Datasette: Raises datasette.Forbidden() if any of the checks fail """ + assert actor is None or isinstance(actor, dict) for permission in permissions: if isinstance(permission, str): action = permission diff --git a/datasette/views/base.py b/datasette/views/base.py index d1e684a2..221e1882 100644 --- a/datasette/views/base.py +++ b/datasette/views/base.py @@ -66,16 +66,6 @@ class BaseView: response.body = b"" return response - async def check_permission(self, request, action, resource=None): - ok = await self.ds.permission_allowed( - request.actor, - action, - resource=resource, - default=True, - ) - if not ok: - raise Forbidden(action) - def database_color(self, database): return "ff0000" diff --git a/datasette/views/database.py b/datasette/views/database.py index 69ed1233..31a1839f 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -229,7 +229,7 @@ class QueryView(DataView): None, "view-query", (database, canned_query), default=True ) else: - await self.check_permission(request, "execute-sql", database) + await self.ds.ensure_permissions(request.actor, [("execute-sql", database)]) # Extract any :named parameters named_parameters = named_parameters or await derive_named_parameters( diff --git a/datasette/views/index.py b/datasette/views/index.py index f5e31181..1c391e26 100644 --- a/datasette/views/index.py +++ b/datasette/views/index.py @@ -20,7 +20,7 @@ class IndexView(BaseView): async def get(self, request): as_format = request.url_vars["format"] - await self.check_permission(request, "view-instance") + await self.ds.ensure_permissions(request.actor, ["view-instance"]) databases = [] for name, db in self.ds.databases.items(): visible, database_private = await check_visibility( diff --git a/datasette/views/special.py b/datasette/views/special.py index 395ee587..dd834528 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -16,7 +16,7 @@ class JsonDataView(BaseView): async def get(self, request): as_format = request.url_vars["format"] - await self.check_permission(request, "view-instance") + await self.ds.ensure_permissions(request.actor, ["view-instance"]) if self.needs_request: data = self.data_callback(request) else: @@ -47,7 +47,7 @@ class PatternPortfolioView(BaseView): has_json_alternate = False async def get(self, request): - await self.check_permission(request, "view-instance") + await self.ds.ensure_permissions(request.actor, ["view-instance"]) return await self.render(["patterns.html"], request=request) @@ -95,7 +95,7 @@ class PermissionsDebugView(BaseView): has_json_alternate = False async def get(self, request): - await self.check_permission(request, "view-instance") + await self.ds.ensure_permissions(request.actor, ["view-instance"]) if not await self.ds.permission_allowed(request.actor, "permissions-debug"): raise Forbidden("Permission denied") return await self.render( @@ -146,11 +146,11 @@ class MessagesDebugView(BaseView): has_json_alternate = False async def get(self, request): - await self.check_permission(request, "view-instance") + await self.ds.ensure_permissions(request.actor, ["view-instance"]) return await self.render(["messages_debug.html"], request) async def post(self, request): - await self.check_permission(request, "view-instance") + await self.ds.ensure_permissions(request.actor, ["view-instance"]) post = await request.post_vars() message = post.get("message", "") message_type = post.get("message_type") or "INFO" diff --git a/tests/test_permissions.py b/tests/test_permissions.py index 788523b0..f4169dbe 100644 --- a/tests/test_permissions.py +++ b/tests/test_permissions.py @@ -321,17 +321,20 @@ def test_permissions_debug(app_client): checks = [ { "action": div.select_one(".check-action").text, - "result": bool(div.select(".check-result-true")), + # True = green tick, False = red cross, None = gray None + "result": None + if div.select(".check-result-no-opinion") + else bool(div.select(".check-result-true")), "used_default": bool(div.select(".check-used-default")), } for div in check_divs ] - assert [ + assert checks == [ {"action": "permissions-debug", "result": True, "used_default": False}, - {"action": "view-instance", "result": True, "used_default": True}, + {"action": "view-instance", "result": None, "used_default": True}, {"action": "permissions-debug", "result": False, "used_default": True}, - {"action": "view-instance", "result": True, "used_default": True}, - ] == checks + {"action": "view-instance", "result": None, "used_default": True}, + ] @pytest.mark.parametrize( From 1a7750eb29fd15dd2eea3b9f6e33028ce441b143 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 21 Mar 2022 12:01:37 -0700 Subject: [PATCH 029/952] Documented datasette.check_visibility() method, closes #1678 --- datasette/app.py | 18 ++++++++++++++++++ datasette/utils/__init__.py | 19 ------------------- datasette/views/database.py | 10 +++------- datasette/views/index.py | 11 ++++------- docs/internals.rst | 28 +++++++++++++++++++++++++++- 5 files changed, 52 insertions(+), 34 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 22ae211f..c9eede26 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -664,6 +664,24 @@ class Datasette: else: raise Forbidden(action) + async def check_visibility(self, actor, action, resource): + """Returns (visible, private) - visible = can you see it, private = can others see it too""" + visible = await self.permission_allowed( + actor, + action, + resource=resource, + default=True, + ) + if not visible: + return False, False + private = not await self.permission_allowed( + None, + action, + resource=resource, + default=True, + ) + return visible, private + async def execute( self, db_name, diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index c89b9d23..cd8e3d61 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -1002,25 +1002,6 @@ def actor_matches_allow(actor, allow): return False -async def check_visibility(datasette, actor, action, resource, default=True): - """Returns (visible, private) - visible = can you see it, private = can others see it too""" - visible = await datasette.permission_allowed( - actor, - action, - resource=resource, - default=default, - ) - if not visible: - return False, False - private = not await datasette.permission_allowed( - None, - action, - resource=resource, - default=default, - ) - return visible, private - - def resolve_env_secrets(config, environ): """Create copy that recursively replaces {"$env": "NAME"} with values from environ""" if isinstance(config, dict): diff --git a/datasette/views/database.py b/datasette/views/database.py index 31a1839f..103bd575 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -10,7 +10,6 @@ import markupsafe from datasette.utils import ( add_cors_headers, await_me_maybe, - check_visibility, derive_named_parameters, tilde_decode, to_css_class, @@ -62,8 +61,7 @@ class DatabaseView(DataView): views = [] for view_name in await db.view_names(): - visible, private = await check_visibility( - self.ds, + visible, private = await self.ds.check_visibility( request.actor, "view-table", (database, view_name), @@ -78,8 +76,7 @@ class DatabaseView(DataView): tables = [] for table in table_counts: - visible, private = await check_visibility( - self.ds, + visible, private = await self.ds.check_visibility( request.actor, "view-table", (database, table), @@ -105,8 +102,7 @@ class DatabaseView(DataView): for query in ( await self.ds.get_canned_queries(database, request.actor) ).values(): - visible, private = await check_visibility( - self.ds, + visible, private = await self.ds.check_visibility( request.actor, "view-query", (database, query["name"]), diff --git a/datasette/views/index.py b/datasette/views/index.py index 1c391e26..aec78814 100644 --- a/datasette/views/index.py +++ b/datasette/views/index.py @@ -1,7 +1,7 @@ import hashlib import json -from datasette.utils import add_cors_headers, check_visibility, CustomJSONEncoder +from datasette.utils import add_cors_headers, CustomJSONEncoder from datasette.utils.asgi import Response from datasette.version import __version__ @@ -23,8 +23,7 @@ class IndexView(BaseView): await self.ds.ensure_permissions(request.actor, ["view-instance"]) databases = [] for name, db in self.ds.databases.items(): - visible, database_private = await check_visibility( - self.ds, + visible, database_private = await self.ds.check_visibility( request.actor, "view-database", name, @@ -36,8 +35,7 @@ class IndexView(BaseView): views = [] for view_name in await db.view_names(): - visible, private = await check_visibility( - self.ds, + visible, private = await self.ds.check_visibility( request.actor, "view-table", (name, view_name), @@ -55,8 +53,7 @@ class IndexView(BaseView): tables = {} for table in table_names: - visible, private = await check_visibility( - self.ds, + visible, private = await self.ds.check_visibility( request.actor, "view-table", (name, table), diff --git a/docs/internals.rst b/docs/internals.rst index 12adde00..f9a24fea 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -295,7 +295,7 @@ If neither ``metadata.json`` nor any of the plugins provide an answer to the per See :ref:`permissions` for a full list of permission actions included in Datasette core. -.. _datasette_permission_allowed: +.. _datasette_ensure_permissions: await .ensure_permissions(actor, permissions) --------------------------------------------- @@ -321,6 +321,32 @@ This is useful when you need to check multiple permissions at once. For example, ] ) +.. _datasette_check_visibilty: + +await .check_visibility(actor, action, resource=None) +----------------------------------------------------- + +``actor`` - dictionary + The authenticated actor. This is usually ``request.actor``. + +``action`` - string + The name of the action that is being permission checked. + +``resource`` - string or tuple, optional + The resource, e.g. the name of the database, or a tuple of two strings containing the name of the database and the name of the table. Only some permissions apply to a resource. + +This convenience method can be used to answer the question "should this item be considered private, in that it is visible to me but it is not visible to anonymous users?" + +It returns a tuple of two booleans, ``(visible, private)``. ``visible`` indicates if the actor can see this resource. ``private`` will be ``True`` if an anonymous user would not be able to view the resource. + +This example checks if the user can access a specific table, and sets ``private`` so that a padlock icon can later be displayed: + +.. code-block:: python + + visible, private = await self.ds.check_visibility( + request.actor, "view-table", (database, table) + ) + .. _datasette_get_database: .get_database(name) From 72bfd75fb7241893c931348e6aca712edc67ab04 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 21 Mar 2022 14:55:50 -0700 Subject: [PATCH 030/952] Drop n=1 threshold down to <= 20ms, closes #1679 --- datasette/utils/__init__.py | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index cd8e3d61..9109f823 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -182,15 +182,16 @@ class CustomJSONEncoder(json.JSONEncoder): def sqlite_timelimit(conn, ms): deadline = time.perf_counter() + (ms / 1000) # n is the number of SQLite virtual machine instructions that will be - # executed between each check. It's hard to know what to pick here. - # After some experimentation, I've decided to go with 1000 by default and - # 1 for time limits that are less than 50ms + # executed between each check. It takes about 0.08ms to execute 1000. + # https://github.com/simonw/datasette/issues/1679 n = 1000 - if ms < 50: + if ms <= 20: + # This mainly happens while executing our test suite n = 1 def handler(): if time.perf_counter() >= deadline: + # Returning 1 terminates the query with an error return 1 conn.set_progress_handler(handler, n) From 12f3ca79956ed9003c874f67748432adcacc6fd2 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 21 Mar 2022 18:42:03 -0700 Subject: [PATCH 031/952] google-github-actions/setup-gcloud@v0 --- .github/workflows/deploy-latest.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/deploy-latest.yml b/.github/workflows/deploy-latest.yml index 92aa1c6b..a61f6629 100644 --- a/.github/workflows/deploy-latest.yml +++ b/.github/workflows/deploy-latest.yml @@ -14,7 +14,7 @@ jobs: - name: Set up Python uses: actions/setup-python@v2 with: - python-version: 3.9 + python-version: "3.10" - uses: actions/cache@v2 name: Configure pip caching with: @@ -54,7 +54,7 @@ jobs: ' > plugins/alternative_route.py cp fixtures.db fixtures2.db - name: Set up Cloud Run - uses: google-github-actions/setup-gcloud@master + uses: google-github-actions/setup-gcloud@v0 with: version: '275.0.0' service_account_email: ${{ secrets.GCP_SA_EMAIL }} From c4c9dbd0386e46d2bf199f0ed34e4895c98cb78c Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 22 Mar 2022 09:49:26 -0700 Subject: [PATCH 032/952] google-github-actions/setup-gcloud@v0 --- .github/workflows/publish.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 3cfc67da..3e4f8146 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -85,7 +85,7 @@ jobs: sphinx-to-sqlite ../docs.db _build cd .. - name: Set up Cloud Run - uses: google-github-actions/setup-gcloud@master + uses: google-github-actions/setup-gcloud@v0 with: version: '275.0.0' service_account_email: ${{ secrets.GCP_SA_EMAIL }} From d7c793d7998388d915f8d270079c68a77a785051 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 23 Mar 2022 11:12:26 -0700 Subject: [PATCH 033/952] Release 0.61 Refs #957, #1228, #1533, #1545, #1576, #1577, #1587, #1601, #1603, #1607, #1612, #1621, #1649, #1654, #1657, #1661, #1668, #1675, #1678 --- datasette/version.py | 2 +- docs/changelog.rst | 22 ++++++++++++++-------- 2 files changed, 15 insertions(+), 9 deletions(-) diff --git a/datasette/version.py b/datasette/version.py index ccc1e04b..f9b10696 100644 --- a/datasette/version.py +++ b/datasette/version.py @@ -1,2 +1,2 @@ -__version__ = "0.61a0" +__version__ = "0.61" __version_info__ = tuple(__version__.split(".")) diff --git a/docs/changelog.rst b/docs/changelog.rst index 05ad85f2..d2de8da1 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -4,30 +4,36 @@ Changelog ========= -.. _v0_61_a0: +.. _v0_61: -0.61a0 (2022-03-19) -------------------- +0.61 (2022-03-23) +----------------- +In preparation for Datasette 1.0, this release includes two potentially backwards-incompatible changes. Hashed URL mode has been moved to a separate plugin, and the way Datasette generates URLs to databases and tables with special characters in their name such as ``/`` and ``.`` has changed. + +Datasette also now requires Python 3.7 or higher. + +- URLs within Datasette now use a different encoding scheme for tables or databases that include "special" characters outside of the range of ``a-zA-Z0-9_-``. This scheme is explained here: :ref:`internals_tilde_encoding`. (:issue:`1657`) - Removed hashed URL mode from Datasette. The new ``datasette-hashed-urls`` plugin can be used to achieve the same result, see :ref:`performance_hashed_urls` for details. (:issue:`1661`) - Databases can now have a custom path within the Datasette instance that is independent of the database name, using the ``db.route`` property. (:issue:`1668`) -- URLs within Datasette now use a different encoding scheme for tables or databases that include "special" characters outside of the range of ``a-zA-Z0-9_-``. This scheme is explained here: :ref:`internals_tilde_encoding`. (:issue:`1657`) +- Datasette is now covered by a `Code of Conduct `__. (:issue:`1654`) +- Python 3.6 is no longer supported. (:issue:`1577`) +- Tests now run against Python 3.11-dev. (:issue:`1621`) +- New :ref:`datasette.ensure_permissions(actor, permissions) ` internal method for checking multiple permissions at once. (:issue:`1675`) +- New :ref:`datasette.check_visibility(actor, action, resource=None) ` internal method for checking if a user can see a resource that would otherwise be invisible to unauthenticated users. (:issue:`1678`) - Table and row HTML pages now include a ```` element and return a ``Link: URL; rel="alternate"; type="application/json+datasette"`` HTTP header pointing to the JSON version of those pages. (:issue:`1533`) - ``Access-Control-Expose-Headers: Link`` is now added to the CORS headers, allowing remote JavaScript to access that header. - Canned queries are now shown at the top of the database page, directly below the SQL editor. Previously they were shown at the bottom, below the list of tables. (:issue:`1612`) - Datasette now has a default favicon. (:issue:`1603`) - ``sqlite_stat`` tables are now hidden by default. (:issue:`1587`) - SpatiaLite tables ``data_licenses``, ``KNN`` and ``KNN2`` are now hidden by default. (:issue:`1601`) -- Python 3.6 is no longer supported. (:issue:`1577`) -- Tests now run against Python 3.11-dev. (:issue:`1621`) -- Fixed bug where :ref:`custom pages ` did not work on Windows. Thanks, Robert Christie. (:issue:`1545`) - SQL query tracing mechanism now works for queries executed in ``asyncio`` sub-tasks, such as those created by ``asyncio.gather()``. (:issue:`1576`) - :ref:`internals_tracer` mechanism is now documented. - Common Datasette symbols can now be imported directly from the top-level ``datasette`` package, see :ref:`internals_shortcuts`. Those symbols are ``Response``, ``Forbidden``, ``NotFound``, ``hookimpl``, ``actor_matches_allow``. (:issue:`957`) - ``/-/versions`` page now returns additional details for libraries used by SpatiaLite. (:issue:`1607`) - Documentation now links to the `Datasette Tutorials `__. - Datasette will now also look for SpatiaLite in ``/opt/homebrew`` - thanks, Dan Peterson. (`#1649 `__) -- Datasette is now covered by a `Code of Conduct `__. (:issue:`1654`) +- Fixed bug where :ref:`custom pages ` did not work on Windows. Thanks, Robert Christie. (:issue:`1545`) - Fixed error caused when a table had a column named ``n``. (:issue:`1228`) .. _v0_60_2: From 0159662ab8ccb363c59647861360e0cb7a6f930d Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 23 Mar 2022 11:48:10 -0700 Subject: [PATCH 034/952] Fix for bug running ?sql= against databases with a different route, closes #1682 --- datasette/views/database.py | 7 ++++++- tests/test_routes.py | 1 + 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/datasette/views/database.py b/datasette/views/database.py index 103bd575..bdd433cc 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -203,7 +203,12 @@ class QueryView(DataView): named_parameters=None, write=False, ): - database = tilde_decode(request.url_vars["database"]) + database_route = tilde_decode(request.url_vars["database"]) + try: + db = self.ds.get_database(route=database_route) + except KeyError: + raise NotFound("Database not found: {}".format(database_route)) + database = db.name params = {key: request.args.get(key) for key in request.args} if "sql" in params: params.pop("sql") diff --git a/tests/test_routes.py b/tests/test_routes.py index 211b77b5..5ae55d21 100644 --- a/tests/test_routes.py +++ b/tests/test_routes.py @@ -94,6 +94,7 @@ async def test_db_with_route_databases(ds_with_route): ("/original-name/t", 404), ("/original-name/t/1", 404), ("/custom-route-name", 200), + ("/custom-route-name?sql=select+id+from+t", 200), ("/custom-route-name/t", 200), ("/custom-route-name/t/1", 200), ), From d431a9055e977aefe48689a2e5866ea8d3558a6c Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 23 Mar 2022 11:54:10 -0700 Subject: [PATCH 035/952] Release 0.61.1 Refs #1682 Refs https://github.com/simonw/datasette-hashed-urls/issues/13 --- datasette/version.py | 2 +- docs/changelog.rst | 7 +++++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/datasette/version.py b/datasette/version.py index f9b10696..02451a1e 100644 --- a/datasette/version.py +++ b/datasette/version.py @@ -1,2 +1,2 @@ -__version__ = "0.61" +__version__ = "0.61.1" __version_info__ = tuple(__version__.split(".")) diff --git a/docs/changelog.rst b/docs/changelog.rst index d2de8da1..03cf62b6 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -4,6 +4,13 @@ Changelog ========= +.. _v0_61_1: + +0.61.1 (2022-03-23) +------------------- + +- Fixed a bug where databases with a different route from their name (as used by the `datasette-hashed-urls plugin `__) returned errors when executing custom SQL queries. (:issue:`1682`) + .. _v0_61: 0.61 (2022-03-23) From c496f2b663ff0cef908ffaaa68b8cb63111fb5f2 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 24 Mar 2022 12:16:19 -0700 Subject: [PATCH 036/952] Don't show facet in cog menu if not allow_facet, closes #1683 --- datasette/static/table.js | 10 ++++++++-- datasette/templates/table.html | 1 + datasette/views/table.py | 3 +++ tests/test_table_html.py | 14 ++++++++++++++ 4 files changed, 26 insertions(+), 2 deletions(-) diff --git a/datasette/static/table.js b/datasette/static/table.js index 3c88cc40..096a27ac 100644 --- a/datasette/static/table.js +++ b/datasette/static/table.js @@ -128,7 +128,8 @@ var DROPDOWN_ICON_SVG = ` el.dataset.column); @@ -137,7 +138,12 @@ var DROPDOWN_ICON_SVG = ` + `_, `datasette-publish-vercel `_ @@ -400,7 +422,9 @@ If the value matches that pattern, the plugin returns an HTML link element: if not isinstance(value, str): return None stripped = value.strip() - if not stripped.startswith("{") and stripped.endswith("}"): + if not stripped.startswith("{") and stripped.endswith( + "}" + ): return None try: data = json.loads(value) @@ -412,14 +436,18 @@ If the value matches that pattern, the plugin returns an HTML link element: return None href = data["href"] if not ( - href.startswith("/") or href.startswith("http://") + href.startswith("/") + or href.startswith("http://") or href.startswith("https://") ): return None - return markupsafe.Markup('{label}'.format( - href=markupsafe.escape(data["href"]), - label=markupsafe.escape(data["label"] or "") or " " - )) + return markupsafe.Markup( + '{label}'.format( + href=markupsafe.escape(data["href"]), + label=markupsafe.escape(data["label"] or "") + or " ", + ) + ) Examples: `datasette-render-binary `_, `datasette-render-markdown `__, `datasette-json-html `__ @@ -516,7 +544,7 @@ Here is a more complex example: return Response( "\n".join(lines), content_type="text/plain; charset=utf-8", - headers={"x-sqlite-version": result.first()[0]} + headers={"x-sqlite-version": result.first()[0]}, ) And here is an example ``can_render`` function which returns ``True`` only if the query results contain the columns ``atom_id``, ``atom_title`` and ``atom_updated``: @@ -524,7 +552,11 @@ And here is an example ``can_render`` function which returns ``True`` only if th .. code-block:: python def can_render_demo(columns): - return {"atom_id", "atom_title", "atom_updated"}.issubset(columns) + return { + "atom_id", + "atom_title", + "atom_updated", + }.issubset(columns) Examples: `datasette-atom `_, `datasette-ics `_, `datasette-geojson `__ @@ -548,16 +580,14 @@ Return a list of ``(regex, view_function)`` pairs, something like this: async def hello_from(request): name = request.url_vars["name"] - return Response.html("Hello from {}".format( - html.escape(name) - )) + return Response.html( + "Hello from {}".format(html.escape(name)) + ) @hookimpl def register_routes(): - return [ - (r"^/hello-from/(?P.*)$", hello_from) - ] + return [(r"^/hello-from/(?P.*)$", hello_from)] The view functions can take a number of different optional arguments. The corresponding argument will be passed to your function depending on its named parameters - a form of dependency injection. @@ -606,10 +636,13 @@ This example registers a new ``datasette verify file1.db file2.db`` command that import click import sqlite3 + @hookimpl def register_commands(cli): @cli.command() - @click.argument("files", type=click.Path(exists=True), nargs=-1) + @click.argument( + "files", type=click.Path(exists=True), nargs=-1 + ) def verify(files): "Verify that files can be opened by Datasette" for file in files: @@ -617,7 +650,9 @@ This example registers a new ``datasette verify file1.db file2.db`` command that try: conn.execute("select * from sqlite_master") except sqlite3.DatabaseError: - raise click.ClickException("Invalid database: {}".format(file)) + raise click.ClickException( + "Invalid database: {}".format(file) + ) The new command can then be executed like so:: @@ -656,15 +691,18 @@ Each Facet subclass implements a new type of facet operation. The class should l async def suggest(self): # Use self.sql and self.params to suggest some facets suggested_facets = [] - suggested_facets.append({ - "name": column, # Or other unique name - # Construct the URL that will enable this facet: - "toggle_url": self.ds.absolute_url( - self.request, path_with_added_args( - self.request, {"_facet": column} - ) - ), - }) + suggested_facets.append( + { + "name": column, # Or other unique name + # Construct the URL that will enable this facet: + "toggle_url": self.ds.absolute_url( + self.request, + path_with_added_args( + self.request, {"_facet": column} + ), + ), + } + ) return suggested_facets async def facet_results(self): @@ -678,18 +716,25 @@ Each Facet subclass implements a new type of facet operation. The class should l try: facet_results_values = [] # More calculations... - facet_results_values.append({ - "value": value, - "label": label, - "count": count, - "toggle_url": self.ds.absolute_url(self.request, toggle_path), - "selected": selected, - }) - facet_results.append({ - "name": column, - "results": facet_results_values, - "truncated": len(facet_rows_results) > facet_size, - }) + facet_results_values.append( + { + "value": value, + "label": label, + "count": count, + "toggle_url": self.ds.absolute_url( + self.request, toggle_path + ), + "selected": selected, + } + ) + facet_results.append( + { + "name": column, + "results": facet_results_values, + "truncated": len(facet_rows_results) + > facet_size, + } + ) except QueryInterrupted: facets_timed_out.append(column) @@ -728,21 +773,33 @@ This example plugin adds a ``x-databases`` HTTP header listing the currently att def asgi_wrapper(datasette): def wrap_with_databases_header(app): @wraps(app) - async def add_x_databases_header(scope, receive, send): + async def add_x_databases_header( + scope, receive, send + ): async def wrapped_send(event): if event["type"] == "http.response.start": - original_headers = event.get("headers") or [] + original_headers = ( + event.get("headers") or [] + ) event = { "type": event["type"], "status": event["status"], - "headers": original_headers + [ - [b"x-databases", - ", ".join(datasette.databases.keys()).encode("utf-8")] + "headers": original_headers + + [ + [ + b"x-databases", + ", ".join( + datasette.databases.keys() + ).encode("utf-8"), + ] ], } await send(event) + await app(scope, receive, wrapped_send) + return add_x_databases_header + return wrap_with_databases_header Examples: `datasette-cors `__, `datasette-pyinstrument `__ @@ -759,7 +816,9 @@ This hook fires when the Datasette application server first starts up. You can i @hookimpl def startup(datasette): config = datasette.plugin_config("my-plugin") or {} - assert "required-setting" in config, "my-plugin requires setting required-setting" + assert ( + "required-setting" in config + ), "my-plugin requires setting required-setting" Or you can return an async function which will be awaited on startup. Use this option if you need to make any database queries: @@ -770,9 +829,12 @@ Or you can return an async function which will be awaited on startup. Use this o async def inner(): db = datasette.get_database() if "my_table" not in await db.table_names(): - await db.execute_write(""" + await db.execute_write( + """ create table my_table (mycol text) - """) + """ + ) + return inner Potential use-cases: @@ -815,6 +877,7 @@ Ues this hook to return a dictionary of additional :ref:`canned query `__ @@ -888,9 +960,12 @@ Here's an example that authenticates the actor based on an incoming API key: SECRET_KEY = "this-is-a-secret" + @hookimpl def actor_from_request(datasette, request): - authorization = request.headers.get("authorization") or "" + authorization = ( + request.headers.get("authorization") or "" + ) expected = "Bearer {}".format(SECRET_KEY) if secrets.compare_digest(authorization, expected): @@ -906,6 +981,7 @@ Instead of returning a dictionary, this function can return an awaitable functio from datasette import hookimpl + @hookimpl def actor_from_request(datasette, request): async def inner(): @@ -914,7 +990,8 @@ Instead of returning a dictionary, this function can return an awaitable functio return None # Look up ?_token=xxx in sessions table result = await datasette.get_database().execute( - "select count(*) from sessions where token = ?", [token] + "select count(*) from sessions where token = ?", + [token], ) if result.first()[0]: return {"token": token} @@ -952,7 +1029,7 @@ The hook should return an instance of ``datasette.filters.FilterArguments`` whic where_clauses=["id > :max_id"], params={"max_id": 5}, human_descriptions=["max_id is greater than 5"], - extra_context={} + extra_context={}, ) The arguments to the ``FilterArguments`` class constructor are as follows: @@ -973,10 +1050,13 @@ This example plugin causes 0 results to be returned if ``?_nothing=1`` is added from datasette import hookimpl from datasette.filters import FilterArguments + @hookimpl def filters_from_request(self, request): if request.args.get("_nothing"): - return FilterArguments(["1 = 0"], human_descriptions=["NOTHING"]) + return FilterArguments( + ["1 = 0"], human_descriptions=["NOTHING"] + ) Example: `datasette-leaflet-freedraw `_ @@ -1006,6 +1086,7 @@ Here's an example plugin which randomly selects if a permission should be allowe from datasette import hookimpl import random + @hookimpl def permission_allowed(action): if action != "view-instance": @@ -1024,11 +1105,16 @@ Here's an example that allows users to view the ``admin_log`` table only if thei async def inner(): if action == "execute-sql" and resource == "staff": return False - if action == "view-table" and resource == ("staff", "admin_log"): + if action == "view-table" and resource == ( + "staff", + "admin_log", + ): if not actor: return False user_id = actor["id"] - return await datasette.get_database("staff").execute( + return await datasette.get_database( + "staff" + ).execute( "select count(*) from admin_users where user_id = :user_id", {"user_id": user_id}, ) @@ -1059,18 +1145,21 @@ This example registers two new magic parameters: ``:_request_http_version`` retu from uuid import uuid4 + def uuid(key, request): if key == "new": return str(uuid4()) else: raise KeyError + def request(key, request): if key == "http_version": return request.scope["http_version"] else: raise KeyError + @hookimpl def register_magic_parameters(datasette): return [ @@ -1103,9 +1192,12 @@ This example returns a redirect to a ``/-/login`` page: from datasette import hookimpl from urllib.parse import urlencode + @hookimpl def forbidden(request, message): - return Response.redirect("/-/login?=" + urlencode({"message": message})) + return Response.redirect( + "/-/login?=" + urlencode({"message": message}) + ) The function can alternatively return an awaitable function if it needs to make any asynchronous method calls. This example renders a template: @@ -1114,10 +1206,15 @@ The function can alternatively return an awaitable function if it needs to make from datasette import hookimpl from datasette.utils.asgi import Response + @hookimpl def forbidden(datasette): async def inner(): - return Response.html(await datasette.render_template("forbidden.html")) + return Response.html( + await datasette.render_template( + "forbidden.html" + ) + ) return inner @@ -1147,11 +1244,17 @@ This example adds a new menu item but only if the signed in user is ``"root"``: from datasette import hookimpl + @hookimpl def menu_links(datasette, actor): if actor and actor.get("id") == "root": return [ - {"href": datasette.urls.path("/-/edit-schema"), "label": "Edit schema"}, + { + "href": datasette.urls.path( + "/-/edit-schema" + ), + "label": "Edit schema", + }, ] Using :ref:`internals_datasette_urls` here ensures that links in the menu will take the :ref:`setting_base_url` setting into account. @@ -1188,13 +1291,20 @@ This example adds a new table action if the signed in user is ``"root"``: from datasette import hookimpl + @hookimpl def table_actions(datasette, actor): if actor and actor.get("id") == "root": - return [{ - "href": datasette.urls.path("/-/edit-schema/{}/{}".format(database, table)), - "label": "Edit schema for this table", - }] + return [ + { + "href": datasette.urls.path( + "/-/edit-schema/{}/{}".format( + database, table + ) + ), + "label": "Edit schema for this table", + } + ] Example: `datasette-graphql `_ @@ -1238,6 +1348,7 @@ This example will disable CSRF protection for that specific URL path: from datasette import hookimpl + @hookimpl def skip_csrf(scope): return scope["path"] == "/submit-comment" @@ -1278,7 +1389,9 @@ This hook is responsible for returning a dictionary corresponding to Datasette : "description": get_instance_description(datasette), "databases": [], } - for db_name, db_data_dict in get_my_database_meta(datasette, database, table, key): + for db_name, db_data_dict in get_my_database_meta( + datasette, database, table, key + ): metadata["databases"][db_name] = db_data_dict # whatever we return here will be merged with any other plugins using this hook and # will be overwritten by a local metadata.yaml if one exists! From 498e1536f5f3e69c50934c0c031055e0af770bf6 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 24 Apr 2022 09:08:56 -0700 Subject: [PATCH 055/952] One more blacken-docs test, refs #1718 --- docs/testing_plugins.rst | 45 ++++++++++++++++++++++++---------------- 1 file changed, 27 insertions(+), 18 deletions(-) diff --git a/docs/testing_plugins.rst b/docs/testing_plugins.rst index 8e4e3f91..6361d744 100644 --- a/docs/testing_plugins.rst +++ b/docs/testing_plugins.rst @@ -19,7 +19,10 @@ If you use the template described in :ref:`writing_plugins_cookiecutter` your pl response = await datasette.client.get("/-/plugins.json") assert response.status_code == 200 installed_plugins = {p["name"] for p in response.json()} - assert "datasette-plugin-template-demo" in installed_plugins + assert ( + "datasette-plugin-template-demo" + in installed_plugins + ) This test uses the :ref:`internals_datasette_client` object to exercise a test instance of Datasette. ``datasette.client`` is a wrapper around the `HTTPX `__ Python library which can imitate HTTP requests using ASGI. This is the recommended way to write tests against a Datasette instance. @@ -37,9 +40,7 @@ If you are building an installable package you can add them as test dependencies setup( name="datasette-my-plugin", # ... - extras_require={ - "test": ["pytest", "pytest-asyncio"] - }, + extras_require={"test": ["pytest", "pytest-asyncio"]}, tests_require=["datasette-my-plugin[test]"], ) @@ -87,31 +88,34 @@ Here's an example that uses the `sqlite-utils library Date: Sun, 24 Apr 2022 09:17:59 -0700 Subject: [PATCH 056/952] Finished applying blacken-docs, closes #1718 --- .github/workflows/test.yml | 3 +-- docs/testing_plugins.rst | 4 +++- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 38b62995..8d916e49 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -34,6 +34,5 @@ jobs: cog --check docs/*.rst - name: Check if blacken-docs needs to be run run: | + # This fails on syntax errors, or a diff was applied blacken-docs -l 60 docs/*.rst - # This fails if a diff was generated: - git diff-index --quiet HEAD -- diff --git a/docs/testing_plugins.rst b/docs/testing_plugins.rst index 6361d744..1bbaaac1 100644 --- a/docs/testing_plugins.rst +++ b/docs/testing_plugins.rst @@ -118,7 +118,9 @@ Here's an example that uses the `sqlite-utils library Date: Sun, 24 Apr 2022 09:59:20 -0700 Subject: [PATCH 057/952] Cosmetic tweaks after blacken-docs, refs #1718 --- docs/plugin_hooks.rst | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index ace206b7..4560ec9a 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -162,9 +162,8 @@ And here's an example which adds a ``sql_first(sql_query)`` function which execu or database or next(iter(datasette.databases.keys())) ) - return (await datasette.execute(dbname, sql)).rows[ - 0 - ][0] + result = await datasette.execute(dbname, sql) + return result.rows[0][0] return {"sql_first": sql_first} @@ -422,8 +421,8 @@ If the value matches that pattern, the plugin returns an HTML link element: if not isinstance(value, str): return None stripped = value.strip() - if not stripped.startswith("{") and stripped.endswith( - "}" + if not ( + stripped.startswith("{") and stripped.endswith("}") ): return None try: From 579f59dcec43a91dd7d404e00b87a00afd8515f2 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 25 Apr 2022 11:33:35 -0700 Subject: [PATCH 058/952] Refactor to remove RowTableShared class, closes #1719 Refs #1715 --- datasette/app.py | 3 +- datasette/views/row.py | 142 +++++++++++ datasette/views/table.py | 497 +++++++++++++++------------------------ 3 files changed, 328 insertions(+), 314 deletions(-) create mode 100644 datasette/views/row.py diff --git a/datasette/app.py b/datasette/app.py index c9eede26..d269372c 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -40,7 +40,8 @@ from .views.special import ( PermissionsDebugView, MessagesDebugView, ) -from .views.table import RowView, TableView +from .views.table import TableView +from .views.row import RowView from .renderer import json_renderer from .url_builder import Urls from .database import Database, QueryInterrupted diff --git a/datasette/views/row.py b/datasette/views/row.py new file mode 100644 index 00000000..b1c7362d --- /dev/null +++ b/datasette/views/row.py @@ -0,0 +1,142 @@ +from datasette.utils.asgi import NotFound +from datasette.database import QueryInterrupted +from .base import DataView +from datasette.utils import ( + tilde_decode, + urlsafe_components, + to_css_class, + escape_sqlite, +) +from .table import _sql_params_pks, display_columns_and_rows + + +class RowView(DataView): + name = "row" + + async def data(self, request, default_labels=False): + database_route = tilde_decode(request.url_vars["database"]) + table = tilde_decode(request.url_vars["table"]) + try: + db = self.ds.get_database(route=database_route) + except KeyError: + raise NotFound("Database not found: {}".format(database_route)) + database = db.name + await self.ds.ensure_permissions( + request.actor, + [ + ("view-table", (database, table)), + ("view-database", database), + "view-instance", + ], + ) + pk_values = urlsafe_components(request.url_vars["pks"]) + try: + db = self.ds.get_database(route=database_route) + except KeyError: + raise NotFound("Database not found: {}".format(database_route)) + database = db.name + sql, params, pks = await _sql_params_pks(db, table, pk_values) + results = await db.execute(sql, params, truncate=True) + columns = [r[0] for r in results.description] + rows = list(results.rows) + if not rows: + raise NotFound(f"Record not found: {pk_values}") + + async def template_data(): + display_columns, display_rows = await display_columns_and_rows( + self.ds, + database, + table, + results.description, + rows, + link_column=False, + truncate_cells=0, + ) + for column in display_columns: + column["sortable"] = False + return { + "foreign_key_tables": await self.foreign_key_tables( + database, table, pk_values + ), + "display_columns": display_columns, + "display_rows": display_rows, + "custom_table_templates": [ + f"_table-{to_css_class(database)}-{to_css_class(table)}.html", + f"_table-row-{to_css_class(database)}-{to_css_class(table)}.html", + "_table.html", + ], + "metadata": (self.ds.metadata("databases") or {}) + .get(database, {}) + .get("tables", {}) + .get(table, {}), + } + + data = { + "database": database, + "table": table, + "rows": rows, + "columns": columns, + "primary_keys": pks, + "primary_key_values": pk_values, + "units": self.ds.table_metadata(database, table).get("units", {}), + } + + if "foreign_key_tables" in (request.args.get("_extras") or "").split(","): + data["foreign_key_tables"] = await self.foreign_key_tables( + database, table, pk_values + ) + + return ( + data, + template_data, + ( + f"row-{to_css_class(database)}-{to_css_class(table)}.html", + "row.html", + ), + ) + + async def foreign_key_tables(self, database, table, pk_values): + if len(pk_values) != 1: + return [] + db = self.ds.databases[database] + all_foreign_keys = await db.get_all_foreign_keys() + foreign_keys = all_foreign_keys[table]["incoming"] + if len(foreign_keys) == 0: + return [] + + sql = "select " + ", ".join( + [ + "(select count(*) from {table} where {column}=:id)".format( + table=escape_sqlite(fk["other_table"]), + column=escape_sqlite(fk["other_column"]), + ) + for fk in foreign_keys + ] + ) + try: + rows = list(await db.execute(sql, {"id": pk_values[0]})) + except QueryInterrupted: + # Almost certainly hit the timeout + return [] + + foreign_table_counts = dict( + zip( + [(fk["other_table"], fk["other_column"]) for fk in foreign_keys], + list(rows[0]), + ) + ) + foreign_key_tables = [] + for fk in foreign_keys: + count = ( + foreign_table_counts.get((fk["other_table"], fk["other_column"])) or 0 + ) + key = fk["other_column"] + if key.startswith("_"): + key += "__exact" + link = "{}?{}={}".format( + self.ds.urls.table(database, fk["other_table"]), + key, + ",".join(pk_values), + ) + foreign_key_tables.append({**fk, **{"count": count, "link": link}}) + return foreign_key_tables diff --git a/datasette/views/table.py b/datasette/views/table.py index dc85165e..37fb2ebb 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1,4 +1,3 @@ -import urllib import itertools import json @@ -9,7 +8,6 @@ from datasette.database import QueryInterrupted from datasette.utils import ( await_me_maybe, CustomRow, - MultiParams, append_querystring, compound_keys_after_sql, format_bytes, @@ -21,7 +19,6 @@ from datasette.utils import ( is_url, path_from_row_pks, path_with_added_args, - path_with_format, path_with_removed_args, path_with_replaced_args, to_css_class, @@ -68,7 +65,9 @@ class Row: return json.dumps(d, default=repr, indent=2) -class RowTableShared(DataView): +class TableView(DataView): + name = "table" + async def sortable_columns_for_table(self, database, table, use_rowid): db = self.ds.databases[database] table_metadata = self.ds.table_metadata(database, table) @@ -89,193 +88,6 @@ class RowTableShared(DataView): expandables.append((fk, label_column)) return expandables - async def display_columns_and_rows( - self, database, table, description, rows, link_column=False, truncate_cells=0 - ): - """Returns columns, rows for specified table - including fancy foreign key treatment""" - db = self.ds.databases[database] - table_metadata = self.ds.table_metadata(database, table) - column_descriptions = table_metadata.get("columns") or {} - column_details = {col.name: col for col in await db.table_column_details(table)} - sortable_columns = await self.sortable_columns_for_table(database, table, True) - pks = await db.primary_keys(table) - pks_for_display = pks - if not pks_for_display: - pks_for_display = ["rowid"] - - columns = [] - for r in description: - if r[0] == "rowid" and "rowid" not in column_details: - type_ = "integer" - notnull = 0 - else: - type_ = column_details[r[0]].type - notnull = column_details[r[0]].notnull - columns.append( - { - "name": r[0], - "sortable": r[0] in sortable_columns, - "is_pk": r[0] in pks_for_display, - "type": type_, - "notnull": notnull, - "description": column_descriptions.get(r[0]), - } - ) - - column_to_foreign_key_table = { - fk["column"]: fk["other_table"] - for fk in await db.foreign_keys_for_table(table) - } - - cell_rows = [] - base_url = self.ds.setting("base_url") - for row in rows: - cells = [] - # Unless we are a view, the first column is a link - either to the rowid - # or to the simple or compound primary key - if link_column: - is_special_link_column = len(pks) != 1 - pk_path = path_from_row_pks(row, pks, not pks, False) - cells.append( - { - "column": pks[0] if len(pks) == 1 else "Link", - "value_type": "pk", - "is_special_link_column": is_special_link_column, - "raw": pk_path, - "value": markupsafe.Markup( - '{flat_pks}'.format( - base_url=base_url, - table_path=self.ds.urls.table(database, table), - flat_pks=str(markupsafe.escape(pk_path)), - flat_pks_quoted=path_from_row_pks(row, pks, not pks), - ) - ), - } - ) - - for value, column_dict in zip(row, columns): - column = column_dict["name"] - if link_column and len(pks) == 1 and column == pks[0]: - # If there's a simple primary key, don't repeat the value as it's - # already shown in the link column. - continue - - # First let the plugins have a go - # pylint: disable=no-member - plugin_display_value = None - for candidate in pm.hook.render_cell( - value=value, - column=column, - table=table, - database=database, - datasette=self.ds, - ): - candidate = await await_me_maybe(candidate) - if candidate is not None: - plugin_display_value = candidate - break - if plugin_display_value: - display_value = plugin_display_value - elif isinstance(value, bytes): - formatted = format_bytes(len(value)) - display_value = markupsafe.Markup( - '<Binary: {:,} byte{}>'.format( - self.ds.urls.row_blob( - database, - table, - path_from_row_pks(row, pks, not pks), - column, - ), - ' title="{}"'.format(formatted) - if "bytes" not in formatted - else "", - len(value), - "" if len(value) == 1 else "s", - ) - ) - elif isinstance(value, dict): - # It's an expanded foreign key - display link to other row - label = value["label"] - value = value["value"] - # The table we link to depends on the column - other_table = column_to_foreign_key_table[column] - link_template = ( - LINK_WITH_LABEL if (label != value) else LINK_WITH_VALUE - ) - display_value = markupsafe.Markup( - link_template.format( - database=database, - base_url=base_url, - table=tilde_encode(other_table), - link_id=tilde_encode(str(value)), - id=str(markupsafe.escape(value)), - label=str(markupsafe.escape(label)) or "-", - ) - ) - elif value in ("", None): - display_value = markupsafe.Markup(" ") - elif is_url(str(value).strip()): - display_value = markupsafe.Markup( - '{url}'.format( - url=markupsafe.escape(value.strip()) - ) - ) - elif column in table_metadata.get("units", {}) and value != "": - # Interpret units using pint - value = value * ureg(table_metadata["units"][column]) - # Pint uses floating point which sometimes introduces errors in the compact - # representation, which we have to round off to avoid ugliness. In the vast - # majority of cases this rounding will be inconsequential. I hope. - value = round(value.to_compact(), 6) - display_value = markupsafe.Markup( - f"{value:~P}".replace(" ", " ") - ) - else: - display_value = str(value) - if truncate_cells and len(display_value) > truncate_cells: - display_value = display_value[:truncate_cells] + "\u2026" - - cells.append( - { - "column": column, - "value": display_value, - "raw": value, - "value_type": "none" - if value is None - else str(type(value).__name__), - } - ) - cell_rows.append(Row(cells)) - - if link_column: - # Add the link column header. - # If it's a simple primary key, we have to remove and re-add that column name at - # the beginning of the header row. - first_column = None - if len(pks) == 1: - columns = [col for col in columns if col["name"] != pks[0]] - first_column = { - "name": pks[0], - "sortable": len(pks) == 1, - "is_pk": True, - "type": column_details[pks[0]].type, - "notnull": column_details[pks[0]].notnull, - } - else: - first_column = { - "name": "Link", - "sortable": False, - "is_pk": False, - "type": "", - "notnull": 0, - } - columns = [first_column] + columns - return columns, cell_rows - - -class TableView(RowTableShared): - name = "table" - async def post(self, request): database_route = tilde_decode(request.url_vars["database"]) try: @@ -807,13 +619,17 @@ class TableView(RowTableShared): async def extra_template(): nonlocal sort - display_columns, display_rows = await self.display_columns_and_rows( + display_columns, display_rows = await display_columns_and_rows( + self.ds, database, table, results.description, rows, link_column=not is_view, truncate_cells=self.ds.setting("truncate_cells_html"), + sortable_columns=await self.sortable_columns_for_table( + database, table, use_rowid=True + ), ) metadata = ( (self.ds.metadata("databases") or {}) @@ -948,132 +764,187 @@ async def _sql_params_pks(db, table, pk_values): return sql, params, pks -class RowView(RowTableShared): - name = "row" +async def display_columns_and_rows( + datasette, + database, + table, + description, + rows, + link_column=False, + truncate_cells=0, + sortable_columns=None, +): + """Returns columns, rows for specified table - including fancy foreign key treatment""" + sortable_columns = sortable_columns or set() + db = datasette.databases[database] + table_metadata = datasette.table_metadata(database, table) + column_descriptions = table_metadata.get("columns") or {} + column_details = {col.name: col for col in await db.table_column_details(table)} + pks = await db.primary_keys(table) + pks_for_display = pks + if not pks_for_display: + pks_for_display = ["rowid"] - async def data(self, request, default_labels=False): - database_route = tilde_decode(request.url_vars["database"]) - table = tilde_decode(request.url_vars["table"]) - try: - db = self.ds.get_database(route=database_route) - except KeyError: - raise NotFound("Database not found: {}".format(database_route)) - database = db.name - await self.ds.ensure_permissions( - request.actor, - [ - ("view-table", (database, table)), - ("view-database", database), - "view-instance", - ], - ) - pk_values = urlsafe_components(request.url_vars["pks"]) - try: - db = self.ds.get_database(route=database_route) - except KeyError: - raise NotFound("Database not found: {}".format(database_route)) - database = db.name - sql, params, pks = await _sql_params_pks(db, table, pk_values) - results = await db.execute(sql, params, truncate=True) - columns = [r[0] for r in results.description] - rows = list(results.rows) - if not rows: - raise NotFound(f"Record not found: {pk_values}") - - async def template_data(): - display_columns, display_rows = await self.display_columns_and_rows( - database, - table, - results.description, - rows, - link_column=False, - truncate_cells=0, - ) - for column in display_columns: - column["sortable"] = False - return { - "foreign_key_tables": await self.foreign_key_tables( - database, table, pk_values - ), - "display_columns": display_columns, - "display_rows": display_rows, - "custom_table_templates": [ - f"_table-{to_css_class(database)}-{to_css_class(table)}.html", - f"_table-row-{to_css_class(database)}-{to_css_class(table)}.html", - "_table.html", - ], - "metadata": (self.ds.metadata("databases") or {}) - .get(database, {}) - .get("tables", {}) - .get(table, {}), + columns = [] + for r in description: + if r[0] == "rowid" and "rowid" not in column_details: + type_ = "integer" + notnull = 0 + else: + type_ = column_details[r[0]].type + notnull = column_details[r[0]].notnull + columns.append( + { + "name": r[0], + "sortable": r[0] in sortable_columns, + "is_pk": r[0] in pks_for_display, + "type": type_, + "notnull": notnull, + "description": column_descriptions.get(r[0]), } - - data = { - "database": database, - "table": table, - "rows": rows, - "columns": columns, - "primary_keys": pks, - "primary_key_values": pk_values, - "units": self.ds.table_metadata(database, table).get("units", {}), - } - - if "foreign_key_tables" in (request.args.get("_extras") or "").split(","): - data["foreign_key_tables"] = await self.foreign_key_tables( - database, table, pk_values - ) - - return ( - data, - template_data, - ( - f"row-{to_css_class(database)}-{to_css_class(table)}.html", - "row.html", - ), ) - async def foreign_key_tables(self, database, table, pk_values): - if len(pk_values) != 1: - return [] - db = self.ds.databases[database] - all_foreign_keys = await db.get_all_foreign_keys() - foreign_keys = all_foreign_keys[table]["incoming"] - if len(foreign_keys) == 0: - return [] + column_to_foreign_key_table = { + fk["column"]: fk["other_table"] for fk in await db.foreign_keys_for_table(table) + } - sql = "select " + ", ".join( - [ - "(select count(*) from {table} where {column}=:id)".format( - table=escape_sqlite(fk["other_table"]), - column=escape_sqlite(fk["other_column"]), + cell_rows = [] + base_url = datasette.setting("base_url") + for row in rows: + cells = [] + # Unless we are a view, the first column is a link - either to the rowid + # or to the simple or compound primary key + if link_column: + is_special_link_column = len(pks) != 1 + pk_path = path_from_row_pks(row, pks, not pks, False) + cells.append( + { + "column": pks[0] if len(pks) == 1 else "Link", + "value_type": "pk", + "is_special_link_column": is_special_link_column, + "raw": pk_path, + "value": markupsafe.Markup( + '{flat_pks}'.format( + base_url=base_url, + table_path=datasette.urls.table(database, table), + flat_pks=str(markupsafe.escape(pk_path)), + flat_pks_quoted=path_from_row_pks(row, pks, not pks), + ) + ), + } + ) + + for value, column_dict in zip(row, columns): + column = column_dict["name"] + if link_column and len(pks) == 1 and column == pks[0]: + # If there's a simple primary key, don't repeat the value as it's + # already shown in the link column. + continue + + # First let the plugins have a go + # pylint: disable=no-member + plugin_display_value = None + for candidate in pm.hook.render_cell( + value=value, + column=column, + table=table, + database=database, + datasette=datasette, + ): + candidate = await await_me_maybe(candidate) + if candidate is not None: + plugin_display_value = candidate + break + if plugin_display_value: + display_value = plugin_display_value + elif isinstance(value, bytes): + formatted = format_bytes(len(value)) + display_value = markupsafe.Markup( + '<Binary: {:,} byte{}>'.format( + datasette.urls.row_blob( + database, + table, + path_from_row_pks(row, pks, not pks), + column, + ), + ' title="{}"'.format(formatted) + if "bytes" not in formatted + else "", + len(value), + "" if len(value) == 1 else "s", + ) ) - for fk in foreign_keys - ] - ) - try: - rows = list(await db.execute(sql, {"id": pk_values[0]})) - except QueryInterrupted: - # Almost certainly hit the timeout - return [] + elif isinstance(value, dict): + # It's an expanded foreign key - display link to other row + label = value["label"] + value = value["value"] + # The table we link to depends on the column + other_table = column_to_foreign_key_table[column] + link_template = LINK_WITH_LABEL if (label != value) else LINK_WITH_VALUE + display_value = markupsafe.Markup( + link_template.format( + database=database, + base_url=base_url, + table=tilde_encode(other_table), + link_id=tilde_encode(str(value)), + id=str(markupsafe.escape(value)), + label=str(markupsafe.escape(label)) or "-", + ) + ) + elif value in ("", None): + display_value = markupsafe.Markup(" ") + elif is_url(str(value).strip()): + display_value = markupsafe.Markup( + '{url}'.format( + url=markupsafe.escape(value.strip()) + ) + ) + elif column in table_metadata.get("units", {}) and value != "": + # Interpret units using pint + value = value * ureg(table_metadata["units"][column]) + # Pint uses floating point which sometimes introduces errors in the compact + # representation, which we have to round off to avoid ugliness. In the vast + # majority of cases this rounding will be inconsequential. I hope. + value = round(value.to_compact(), 6) + display_value = markupsafe.Markup(f"{value:~P}".replace(" ", " ")) + else: + display_value = str(value) + if truncate_cells and len(display_value) > truncate_cells: + display_value = display_value[:truncate_cells] + "\u2026" - foreign_table_counts = dict( - zip( - [(fk["other_table"], fk["other_column"]) for fk in foreign_keys], - list(rows[0]), + cells.append( + { + "column": column, + "value": display_value, + "raw": value, + "value_type": "none" + if value is None + else str(type(value).__name__), + } ) - ) - foreign_key_tables = [] - for fk in foreign_keys: - count = ( - foreign_table_counts.get((fk["other_table"], fk["other_column"])) or 0 - ) - key = fk["other_column"] - if key.startswith("_"): - key += "__exact" - link = "{}?{}={}".format( - self.ds.urls.table(database, fk["other_table"]), - key, - ",".join(pk_values), - ) - foreign_key_tables.append({**fk, **{"count": count, "link": link}}) - return foreign_key_tables + cell_rows.append(Row(cells)) + + if link_column: + # Add the link column header. + # If it's a simple primary key, we have to remove and re-add that column name at + # the beginning of the header row. + first_column = None + if len(pks) == 1: + columns = [col for col in columns if col["name"] != pks[0]] + first_column = { + "name": pks[0], + "sortable": len(pks) == 1, + "is_pk": True, + "type": column_details[pks[0]].type, + "notnull": column_details[pks[0]].notnull, + } + else: + first_column = { + "name": "Link", + "sortable": False, + "is_pk": False, + "type": "", + "notnull": 0, + } + columns = [first_column] + columns + return columns, cell_rows From c101f0efeec4f6e49298a542c5e2b59236cfa0ff Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 26 Apr 2022 15:34:29 -0700 Subject: [PATCH 059/952] datasette-total-page-time example of asgi_wrapper --- docs/plugin_hooks.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index 4560ec9a..3c9ae2e2 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -801,7 +801,7 @@ This example plugin adds a ``x-databases`` HTTP header listing the currently att return wrap_with_databases_header -Examples: `datasette-cors `__, `datasette-pyinstrument `__ +Examples: `datasette-cors `__, `datasette-pyinstrument `__, `datasette-total-page-time `__ .. _plugin_hook_startup: From 8a0c38f0b89543e652a968a90d480859cb102510 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 26 Apr 2022 13:56:27 -0700 Subject: [PATCH 060/952] Rename database->database_name and table-> table_name, refs #1715 --- datasette/views/table.py | 143 +++++++++++++++++++++------------------ 1 file changed, 76 insertions(+), 67 deletions(-) diff --git a/datasette/views/table.py b/datasette/views/table.py index 37fb2ebb..d66adb82 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -68,22 +68,22 @@ class Row: class TableView(DataView): name = "table" - async def sortable_columns_for_table(self, database, table, use_rowid): - db = self.ds.databases[database] - table_metadata = self.ds.table_metadata(database, table) + async def sortable_columns_for_table(self, database_name, table_name, use_rowid): + db = self.ds.databases[database_name] + table_metadata = self.ds.table_metadata(database_name, table_name) if "sortable_columns" in table_metadata: sortable_columns = set(table_metadata["sortable_columns"]) else: - sortable_columns = set(await db.table_columns(table)) + sortable_columns = set(await db.table_columns(table_name)) if use_rowid: sortable_columns.add("rowid") return sortable_columns - async def expandable_columns(self, database, table): + async def expandable_columns(self, database_name, table_name): # Returns list of (fk_dict, label_column-or-None) pairs for that table expandables = [] - db = self.ds.databases[database] - for fk in await db.foreign_keys_for_table(table): + db = self.ds.databases[database_name] + for fk in await db.foreign_keys_for_table(table_name): label_column = await db.label_column_for_table(fk["other_table"]) expandables.append((fk, label_column)) return expandables @@ -94,17 +94,19 @@ class TableView(DataView): db = self.ds.get_database(route=database_route) except KeyError: raise NotFound("Database not found: {}".format(database_route)) - database = db.name - table = tilde_decode(request.url_vars["table"]) + database_name = db.name + table_name = tilde_decode(request.url_vars["table"]) # Handle POST to a canned query - canned_query = await self.ds.get_canned_query(database, table, request.actor) + canned_query = await self.ds.get_canned_query( + database_name, table_name, request.actor + ) assert canned_query, "You may only POST to a canned query" return await QueryView(self.ds).data( request, canned_query["sql"], metadata=canned_query, editable=False, - canned_query=table, + canned_query=table_name, named_parameters=canned_query.get("params"), write=bool(canned_query.get("write")), ) @@ -150,45 +152,47 @@ class TableView(DataView): _size=None, ): database_route = tilde_decode(request.url_vars["database"]) - table = tilde_decode(request.url_vars["table"]) + table_name = tilde_decode(request.url_vars["table"]) try: db = self.ds.get_database(route=database_route) except KeyError: raise NotFound("Database not found: {}".format(database_route)) - database = db.name + database_name = db.name # If this is a canned query, not a table, then dispatch to QueryView instead - canned_query = await self.ds.get_canned_query(database, table, request.actor) + canned_query = await self.ds.get_canned_query( + database_name, table_name, request.actor + ) if canned_query: return await QueryView(self.ds).data( request, canned_query["sql"], metadata=canned_query, editable=False, - canned_query=table, + canned_query=table_name, named_parameters=canned_query.get("params"), write=bool(canned_query.get("write")), ) - is_view = bool(await db.get_view_definition(table)) - table_exists = bool(await db.table_exists(table)) + is_view = bool(await db.get_view_definition(table_name)) + table_exists = bool(await db.table_exists(table_name)) # If table or view not found, return 404 if not is_view and not table_exists: - raise NotFound(f"Table not found: {table}") + raise NotFound(f"Table not found: {table_name}") # Ensure user has permission to view this table await self.ds.ensure_permissions( request.actor, [ - ("view-table", (database, table)), - ("view-database", database), + ("view-table", (database_name, table_name)), + ("view-database", database_name), "view-instance", ], ) private = not await self.ds.permission_allowed( - None, "view-table", (database, table), default=True + None, "view-table", (database_name, table_name), default=True ) # Handle ?_filter_column and redirect, if present @@ -216,8 +220,8 @@ class TableView(DataView): ) # Introspect columns and primary keys for table - pks = await db.primary_keys(table) - table_columns = await db.table_columns(table) + pks = await db.primary_keys(table_name) + table_columns = await db.table_columns(table_name) # Take ?_col= and ?_nocol= into account specified_columns = await self.columns_to_select(table_columns, pks, request) @@ -248,7 +252,7 @@ class TableView(DataView): nocount = True nofacet = True - table_metadata = self.ds.table_metadata(database, table) + table_metadata = self.ds.table_metadata(database_name, table_name) units = table_metadata.get("units", {}) # Arguments that start with _ and don't contain a __ are @@ -262,7 +266,7 @@ class TableView(DataView): # Build where clauses from query string arguments filters = Filters(sorted(filter_args), units, ureg) - where_clauses, params = filters.build_where_clauses(table) + where_clauses, params = filters.build_where_clauses(table_name) # Execute filters_from_request plugin hooks - including the default # ones that live in datasette/filters.py @@ -271,8 +275,8 @@ class TableView(DataView): for hook in pm.hook.filters_from_request( request=request, - table=table, - database=database, + table=table_name, + database=database_name, datasette=self.ds, ): filter_arguments = await await_me_maybe(hook) @@ -284,7 +288,7 @@ class TableView(DataView): # Deal with custom sort orders sortable_columns = await self.sortable_columns_for_table( - database, table, use_rowid + database_name, table_name, use_rowid ) sort = request.args.get("_sort") sort_desc = request.args.get("_sort_desc") @@ -309,7 +313,7 @@ class TableView(DataView): order_by = f"{escape_sqlite(sort_desc)} desc" from_sql = "from {table_name} {where}".format( - table_name=escape_sqlite(table), + table_name=escape_sqlite(table_name), where=("where {} ".format(" and ".join(where_clauses))) if where_clauses else "", @@ -422,7 +426,7 @@ class TableView(DataView): sql_no_order_no_limit = ( "select {select_all_columns} from {table_name} {where}".format( select_all_columns=select_all_columns, - table_name=escape_sqlite(table), + table_name=escape_sqlite(table_name), where=where_clause, ) ) @@ -430,7 +434,7 @@ class TableView(DataView): # This is the SQL that populates the main table on the page sql = "select {select_specified_columns} from {table_name} {where}{order_by} limit {page_size}{offset}".format( select_specified_columns=select_specified_columns, - table_name=escape_sqlite(table), + table_name=escape_sqlite(table_name), where=where_clause, order_by=order_by, page_size=page_size + 1, @@ -448,13 +452,13 @@ class TableView(DataView): if ( not db.is_mutable and self.ds.inspect_data - and count_sql == f"select count(*) from {table} " + and count_sql == f"select count(*) from {table_name} " ): # We can use a previously cached table row count try: - filtered_table_rows_count = self.ds.inspect_data[database]["tables"][ - table - ]["count"] + filtered_table_rows_count = self.ds.inspect_data[database_name][ + "tables" + ][table_name]["count"] except KeyError: pass @@ -484,10 +488,10 @@ class TableView(DataView): klass( self.ds, request, - database, + database_name, sql=sql_no_order_no_limit, params=params, - table=table, + table=table_name, metadata=table_metadata, row_count=filtered_table_rows_count, ) @@ -527,7 +531,7 @@ class TableView(DataView): # Expand labeled columns if requested expanded_columns = [] - expandable_columns = await self.expandable_columns(database, table) + expandable_columns = await self.expandable_columns(database_name, table_name) columns_to_expand = None try: all_labels = value_as_boolean(request.args.get("_labels", "")) @@ -554,7 +558,9 @@ class TableView(DataView): values = [row[column_index] for row in rows] # Expand them expanded_labels.update( - await self.ds.expand_foreign_keys(database, table, column, values) + await self.ds.expand_foreign_keys( + database_name, table_name, column, values + ) ) if expanded_labels: # Rewrite the rows @@ -621,21 +627,21 @@ class TableView(DataView): display_columns, display_rows = await display_columns_and_rows( self.ds, - database, - table, + database_name, + table_name, results.description, rows, link_column=not is_view, truncate_cells=self.ds.setting("truncate_cells_html"), sortable_columns=await self.sortable_columns_for_table( - database, table, use_rowid=True + database_name, table_name, use_rowid=True ), ) metadata = ( (self.ds.metadata("databases") or {}) - .get(database, {}) + .get(database_name, {}) .get("tables", {}) - .get(table, {}) + .get(table_name, {}) ) self.ds.update_with_inherited_metadata(metadata) @@ -661,8 +667,8 @@ class TableView(DataView): links = [] for hook in pm.hook.table_actions( datasette=self.ds, - table=table, - database=database, + table=table_name, + database=database_name, actor=request.actor, request=request, ): @@ -703,13 +709,13 @@ class TableView(DataView): "sort_desc": sort_desc, "disable_sort": is_view, "custom_table_templates": [ - f"_table-{to_css_class(database)}-{to_css_class(table)}.html", - f"_table-table-{to_css_class(database)}-{to_css_class(table)}.html", + f"_table-{to_css_class(database_name)}-{to_css_class(table_name)}.html", + f"_table-table-{to_css_class(database_name)}-{to_css_class(table_name)}.html", "_table.html", ], "metadata": metadata, - "view_definition": await db.get_view_definition(table), - "table_definition": await db.get_table_definition(table), + "view_definition": await db.get_view_definition(table_name), + "table_definition": await db.get_table_definition(table_name), "datasette_allow_facet": "true" if self.ds.setting("allow_facet") else "false", @@ -719,8 +725,8 @@ class TableView(DataView): return ( { - "database": database, - "table": table, + "database": database_name, + "table": table_name, "is_view": is_view, "human_description_en": human_description_en, "rows": rows[:page_size], @@ -738,12 +744,12 @@ class TableView(DataView): "next_url": next_url, "private": private, "allow_execute_sql": await self.ds.permission_allowed( - request.actor, "execute-sql", database, default=True + request.actor, "execute-sql", database_name, default=True ), }, extra_template, ( - f"table-{to_css_class(database)}-{to_css_class(table)}.html", + f"table-{to_css_class(database_name)}-{to_css_class(table_name)}.html", "table.html", ), ) @@ -766,8 +772,8 @@ async def _sql_params_pks(db, table, pk_values): async def display_columns_and_rows( datasette, - database, - table, + database_name, + table_name, description, rows, link_column=False, @@ -776,11 +782,13 @@ async def display_columns_and_rows( ): """Returns columns, rows for specified table - including fancy foreign key treatment""" sortable_columns = sortable_columns or set() - db = datasette.databases[database] - table_metadata = datasette.table_metadata(database, table) + db = datasette.databases[database_name] + table_metadata = datasette.table_metadata(database_name, table_name) column_descriptions = table_metadata.get("columns") or {} - column_details = {col.name: col for col in await db.table_column_details(table)} - pks = await db.primary_keys(table) + column_details = { + col.name: col for col in await db.table_column_details(table_name) + } + pks = await db.primary_keys(table_name) pks_for_display = pks if not pks_for_display: pks_for_display = ["rowid"] @@ -805,7 +813,8 @@ async def display_columns_and_rows( ) column_to_foreign_key_table = { - fk["column"]: fk["other_table"] for fk in await db.foreign_keys_for_table(table) + fk["column"]: fk["other_table"] + for fk in await db.foreign_keys_for_table(table_name) } cell_rows = [] @@ -826,7 +835,7 @@ async def display_columns_and_rows( "value": markupsafe.Markup( '{flat_pks}'.format( base_url=base_url, - table_path=datasette.urls.table(database, table), + table_path=datasette.urls.table(database_name, table_name), flat_pks=str(markupsafe.escape(pk_path)), flat_pks_quoted=path_from_row_pks(row, pks, not pks), ) @@ -847,8 +856,8 @@ async def display_columns_and_rows( for candidate in pm.hook.render_cell( value=value, column=column, - table=table, - database=database, + table=table_name, + database=database_name, datasette=datasette, ): candidate = await await_me_maybe(candidate) @@ -862,8 +871,8 @@ async def display_columns_and_rows( display_value = markupsafe.Markup( '<Binary: {:,} byte{}>'.format( datasette.urls.row_blob( - database, - table, + database_name, + table_name, path_from_row_pks(row, pks, not pks), column, ), @@ -883,7 +892,7 @@ async def display_columns_and_rows( link_template = LINK_WITH_LABEL if (label != value) else LINK_WITH_VALUE display_value = markupsafe.Markup( link_template.format( - database=database, + database=database_name, base_url=base_url, table=tilde_encode(other_table), link_id=tilde_encode(str(value)), From 942411ef946e9a34a2094944d3423cddad27efd3 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 26 Apr 2022 15:48:56 -0700 Subject: [PATCH 061/952] Execute some TableView queries in parallel Use ?_noparallel=1 to opt out (undocumented, useful for benchmark comparisons) Refs #1723, #1715 --- datasette/views/table.py | 91 +++++++++++++++++++++++++++++----------- 1 file changed, 66 insertions(+), 25 deletions(-) diff --git a/datasette/views/table.py b/datasette/views/table.py index d66adb82..23289b29 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1,3 +1,4 @@ +import asyncio import itertools import json @@ -5,6 +6,7 @@ import markupsafe from datasette.plugins import pm from datasette.database import QueryInterrupted +from datasette import tracer from datasette.utils import ( await_me_maybe, CustomRow, @@ -150,6 +152,16 @@ class TableView(DataView): default_labels=False, _next=None, _size=None, + ): + with tracer.trace_child_tasks(): + return await self._data_traced(request, default_labels, _next, _size) + + async def _data_traced( + self, + request, + default_labels=False, + _next=None, + _size=None, ): database_route = tilde_decode(request.url_vars["database"]) table_name = tilde_decode(request.url_vars["table"]) @@ -159,6 +171,20 @@ class TableView(DataView): raise NotFound("Database not found: {}".format(database_route)) database_name = db.name + # For performance profiling purposes, ?_noparallel=1 turns off asyncio.gather + async def _gather_parallel(*args): + return await asyncio.gather(*args) + + async def _gather_sequential(*args): + results = [] + for fn in args: + results.append(await fn) + return results + + gather = ( + _gather_sequential if request.args.get("_noparallel") else _gather_parallel + ) + # If this is a canned query, not a table, then dispatch to QueryView instead canned_query = await self.ds.get_canned_query( database_name, table_name, request.actor @@ -174,8 +200,12 @@ class TableView(DataView): write=bool(canned_query.get("write")), ) - is_view = bool(await db.get_view_definition(table_name)) - table_exists = bool(await db.table_exists(table_name)) + is_view, table_exists = map( + bool, + await gather( + db.get_view_definition(table_name), db.table_exists(table_name) + ), + ) # If table or view not found, return 404 if not is_view and not table_exists: @@ -497,33 +527,44 @@ class TableView(DataView): ) ) - if not nofacet: - for facet in facet_instances: - ( + async def execute_facets(): + if not nofacet: + # Run them in parallel + facet_awaitables = [facet.facet_results() for facet in facet_instances] + facet_awaitable_results = await gather(*facet_awaitables) + for ( instance_facet_results, instance_facets_timed_out, - ) = await facet.facet_results() - for facet_info in instance_facet_results: - base_key = facet_info["name"] - key = base_key - i = 1 - while key in facet_results: - i += 1 - key = f"{base_key}_{i}" - facet_results[key] = facet_info - facets_timed_out.extend(instance_facets_timed_out) + ) in facet_awaitable_results: + for facet_info in instance_facet_results: + base_key = facet_info["name"] + key = base_key + i = 1 + while key in facet_results: + i += 1 + key = f"{base_key}_{i}" + facet_results[key] = facet_info + facets_timed_out.extend(instance_facets_timed_out) - # Calculate suggested facets suggested_facets = [] - if ( - self.ds.setting("suggest_facets") - and self.ds.setting("allow_facet") - and not _next - and not nofacet - and not nosuggest - ): - for facet in facet_instances: - suggested_facets.extend(await facet.suggest()) + + async def execute_suggested_facets(): + # Calculate suggested facets + if ( + self.ds.setting("suggest_facets") + and self.ds.setting("allow_facet") + and not _next + and not nofacet + and not nosuggest + ): + # Run them in parallel + facet_suggest_awaitables = [ + facet.suggest() for facet in facet_instances + ] + for suggest_result in await gather(*facet_suggest_awaitables): + suggested_facets.extend(suggest_result) + + await gather(execute_facets(), execute_suggested_facets()) # Figure out columns and rows for the query columns = [r[0] for r in results.description] From 94a3171b01fde5c52697aeeff052e3ad4bab5391 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 28 Apr 2022 13:29:11 -0700 Subject: [PATCH 062/952] .plugin_config() can return None --- docs/internals.rst | 4 ++++ docs/writing_plugins.rst | 2 ++ 2 files changed, 6 insertions(+) diff --git a/docs/internals.rst b/docs/internals.rst index aad608dc..18822d47 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -288,6 +288,10 @@ All databases are listed, irrespective of user permissions. This means that the This method lets you read plugin configuration values that were set in ``metadata.json``. See :ref:`writing_plugins_configuration` for full details of how this method should be used. +The return value will be the value from the configuration file - usually a dictionary. + +If the plugin is not configured the return value will be ``None``. + .. _datasette_render_template: await .render_template(template, context=None, request=None) diff --git a/docs/writing_plugins.rst b/docs/writing_plugins.rst index 89f7f5eb..9aee70f6 100644 --- a/docs/writing_plugins.rst +++ b/docs/writing_plugins.rst @@ -182,6 +182,8 @@ When you are writing plugins, you can access plugin configuration like this usin This will return the ``{"latitude_column": "lat", "longitude_column": "lng"}`` in the above example. +If there is no configuration for that plugin, the method will return ``None``. + If it cannot find the requested configuration at the table layer, it will fall back to the database layer and then the root layer. For example, a user may have set the plugin configuration option like so:: { From 4afc1afc721ac0d14f58b0f8339c1bf431d5313c Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 2 May 2022 12:13:11 -0700 Subject: [PATCH 063/952] Depend on click-default-group-wheel>=1.2.2 Refs #1733 --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 7f0562fd..fcb43aa1 100644 --- a/setup.py +++ b/setup.py @@ -44,7 +44,7 @@ setup( install_requires=[ "asgiref>=3.2.10,<3.6.0", "click>=7.1.1,<8.2.0", - "click-default-group~=1.2.2", + "click-default-group-wheel>=1.2.2", "Jinja2>=2.10.3,<3.1.0", "hupper~=1.9", "httpx>=0.20", From 7e03394734307a5761e4c98d902b6a8cab188562 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 2 May 2022 12:20:14 -0700 Subject: [PATCH 064/952] Optional uvicorn import for Pyodide, refs #1733 --- datasette/app.py | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index d269372c..a5330458 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -26,7 +26,6 @@ from itsdangerous import URLSafeSerializer from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader from jinja2.environment import Template from jinja2.exceptions import TemplateNotFound -import uvicorn from .views.base import DatasetteError, ureg from .views.database import DatabaseDownload, DatabaseView @@ -806,6 +805,15 @@ class Datasette: datasette_version = {"version": __version__} if self.version_note: datasette_version["note"] = self.version_note + + try: + # Optional import to avoid breaking Pyodide + # https://github.com/simonw/datasette/issues/1733#issuecomment-1115268245 + import uvicorn + + uvicorn_version = uvicorn.__version__ + except ImportError: + uvicorn_version = None info = { "python": { "version": ".".join(map(str, sys.version_info[:3])), @@ -813,7 +821,7 @@ class Datasette: }, "datasette": datasette_version, "asgi": "3.0", - "uvicorn": uvicorn.__version__, + "uvicorn": uvicorn_version, "sqlite": { "version": sqlite_version, "fts_versions": fts_versions, From 687907aa2b1bde4de6ae7155b0e2a949ca015ca9 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 2 May 2022 12:39:06 -0700 Subject: [PATCH 065/952] Remove python-baseconv dependency, refs #1733, closes #1734 --- datasette/actor_auth_cookie.py | 2 +- datasette/utils/baseconv.py | 59 ++++++++++++++++++++++++++++++++++ docs/authentication.rst | 4 +-- setup.py | 1 - tests/test_auth.py | 2 +- 5 files changed, 63 insertions(+), 5 deletions(-) create mode 100644 datasette/utils/baseconv.py diff --git a/datasette/actor_auth_cookie.py b/datasette/actor_auth_cookie.py index 15ecd331..368213af 100644 --- a/datasette/actor_auth_cookie.py +++ b/datasette/actor_auth_cookie.py @@ -1,6 +1,6 @@ from datasette import hookimpl from itsdangerous import BadSignature -import baseconv +from datasette.utils import baseconv import time diff --git a/datasette/utils/baseconv.py b/datasette/utils/baseconv.py new file mode 100644 index 00000000..27e4fb00 --- /dev/null +++ b/datasette/utils/baseconv.py @@ -0,0 +1,59 @@ +""" +Convert numbers from base 10 integers to base X strings and back again. + +Sample usage: + +>>> base20 = BaseConverter('0123456789abcdefghij') +>>> base20.from_decimal(1234) +'31e' +>>> base20.to_decimal('31e') +1234 + +Originally shared here: https://www.djangosnippets.org/snippets/1431/ +""" + + +class BaseConverter(object): + decimal_digits = "0123456789" + + def __init__(self, digits): + self.digits = digits + + def from_decimal(self, i): + return self.convert(i, self.decimal_digits, self.digits) + + def to_decimal(self, s): + return int(self.convert(s, self.digits, self.decimal_digits)) + + def convert(number, fromdigits, todigits): + # Based on http://code.activestate.com/recipes/111286/ + if str(number)[0] == "-": + number = str(number)[1:] + neg = 1 + else: + neg = 0 + + # make an integer out of the number + x = 0 + for digit in str(number): + x = x * len(fromdigits) + fromdigits.index(digit) + + # create the result in base 'len(todigits)' + if x == 0: + res = todigits[0] + else: + res = "" + while x > 0: + digit = x % len(todigits) + res = todigits[digit] + res + x = int(x / len(todigits)) + if neg: + res = "-" + res + return res + + convert = staticmethod(convert) + + +bin = BaseConverter("01") +hexconv = BaseConverter("0123456789ABCDEF") +base62 = BaseConverter("ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqrstuvwxyz") diff --git a/docs/authentication.rst b/docs/authentication.rst index 24960733..685dab15 100644 --- a/docs/authentication.rst +++ b/docs/authentication.rst @@ -401,12 +401,12 @@ Including an expiry time ``ds_actor`` cookies can optionally include a signed expiry timestamp, after which the cookies will no longer be valid. Authentication plugins may chose to use this mechanism to limit the lifetime of the cookie. For example, if a plugin implements single-sign-on against another source it may decide to set short-lived cookies so that if the user is removed from the SSO system their existing Datasette cookies will stop working shortly afterwards. -To include an expiry, add a ``"e"`` key to the cookie value containing a `base62-encoded integer `__ representing the timestamp when the cookie should expire. For example, here's how to set a cookie that expires after 24 hours: +To include an expiry, add a ``"e"`` key to the cookie value containing a base62-encoded integer representing the timestamp when the cookie should expire. For example, here's how to set a cookie that expires after 24 hours: .. code-block:: python import time - import baseconv + from datasette.utils import baseconv expires_at = int(time.time()) + (24 * 60 * 60) diff --git a/setup.py b/setup.py index fcb43aa1..ca449f02 100644 --- a/setup.py +++ b/setup.py @@ -57,7 +57,6 @@ setup( "PyYAML>=5.3,<7.0", "mergedeep>=1.1.1,<1.4.0", "itsdangerous>=1.1,<3.0", - "python-baseconv==1.2.2", ], entry_points=""" [console_scripts] diff --git a/tests/test_auth.py b/tests/test_auth.py index 974f89ea..4ef35a76 100644 --- a/tests/test_auth.py +++ b/tests/test_auth.py @@ -1,5 +1,5 @@ from .fixtures import app_client -import baseconv +from datasette.utils import baseconv import pytest import time From a29c1277896b6a7905ef5441c42a37bc15f67599 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 2 May 2022 12:44:09 -0700 Subject: [PATCH 066/952] Rename to_decimal/from_decimal to decode/encode, refs #1734 --- datasette/utils/baseconv.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/datasette/utils/baseconv.py b/datasette/utils/baseconv.py index 27e4fb00..c4b64908 100644 --- a/datasette/utils/baseconv.py +++ b/datasette/utils/baseconv.py @@ -19,10 +19,10 @@ class BaseConverter(object): def __init__(self, digits): self.digits = digits - def from_decimal(self, i): + def encode(self, i): return self.convert(i, self.decimal_digits, self.digits) - def to_decimal(self, s): + def decode(self, s): return int(self.convert(s, self.digits, self.decimal_digits)) def convert(number, fromdigits, todigits): From 3f00a29141bdea5be747f6d1c93871ccdb792167 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 2 May 2022 13:15:27 -0700 Subject: [PATCH 067/952] Clean up compatibility with Pyodide (#1736) * Optional uvicorn import for Pyodide, refs #1733 * --setting num_sql_threads 0 to disable threading, refs #1735 --- datasette/app.py | 11 ++++++++--- datasette/database.py | 19 +++++++++++++++++++ docs/settings.rst | 2 ++ tests/test_internals_datasette.py | 14 +++++++++++++- 4 files changed, 42 insertions(+), 4 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index a5330458..b7b84371 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -288,9 +288,12 @@ class Datasette: self._settings = dict(DEFAULT_SETTINGS, **(settings or {})) self.renderers = {} # File extension -> (renderer, can_render) functions self.version_note = version_note - self.executor = futures.ThreadPoolExecutor( - max_workers=self.setting("num_sql_threads") - ) + if self.setting("num_sql_threads") == 0: + self.executor = None + else: + self.executor = futures.ThreadPoolExecutor( + max_workers=self.setting("num_sql_threads") + ) self.max_returned_rows = self.setting("max_returned_rows") self.sql_time_limit_ms = self.setting("sql_time_limit_ms") self.page_size = self.setting("default_page_size") @@ -862,6 +865,8 @@ class Datasette: ] def _threads(self): + if self.setting("num_sql_threads") == 0: + return {"num_threads": 0, "threads": []} threads = list(threading.enumerate()) d = { "num_threads": len(threads), diff --git a/datasette/database.py b/datasette/database.py index ba594a8c..44d32667 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -45,6 +45,9 @@ class Database: self._cached_table_counts = None self._write_thread = None self._write_queue = None + # These are used when in non-threaded mode: + self._read_connection = None + self._write_connection = None if not self.is_mutable and not self.is_memory: p = Path(path) self.hash = inspect_hash(p) @@ -134,6 +137,14 @@ class Database: return results async def execute_write_fn(self, fn, block=True): + if self.ds.executor is None: + # non-threaded mode + if self._write_connection is None: + self._write_connection = self.connect(write=True) + self.ds._prepare_connection(self._write_connection, self.name) + return fn(self._write_connection) + + # threaded mode task_id = uuid.uuid5(uuid.NAMESPACE_DNS, "datasette.io") if self._write_queue is None: self._write_queue = queue.Queue() @@ -177,6 +188,14 @@ class Database: task.reply_queue.sync_q.put(result) async def execute_fn(self, fn): + if self.ds.executor is None: + # non-threaded mode + if self._read_connection is None: + self._read_connection = self.connect() + self.ds._prepare_connection(self._read_connection, self.name) + return fn(self._read_connection) + + # threaded mode def in_thread(): conn = getattr(connections, self.name, None) if not conn: diff --git a/docs/settings.rst b/docs/settings.rst index 60c4b36d..8437fb04 100644 --- a/docs/settings.rst +++ b/docs/settings.rst @@ -107,6 +107,8 @@ Maximum number of threads in the thread pool Datasette uses to execute SQLite qu datasette mydatabase.db --setting num_sql_threads 10 +Setting this to 0 turns off threaded SQL queries entirely - useful for environments that do not support threading such as `Pyodide `__. + .. _setting_allow_facet: allow_facet diff --git a/tests/test_internals_datasette.py b/tests/test_internals_datasette.py index cc200a2d..1dc14cab 100644 --- a/tests/test_internals_datasette.py +++ b/tests/test_internals_datasette.py @@ -1,7 +1,7 @@ """ Tests for the datasette.app.Datasette class """ -from datasette.app import Datasette +from datasette.app import Datasette, Database from itsdangerous import BadSignature from .fixtures import app_client import pytest @@ -63,3 +63,15 @@ async def test_datasette_constructor(): "hash": None, } ] + + +@pytest.mark.asyncio +async def test_num_sql_threads_zero(): + ds = Datasette([], memory=True, settings={"num_sql_threads": 0}) + db = ds.add_database(Database(ds, memory_name="test_num_sql_threads_zero")) + await db.execute_write("create table t(id integer primary key)") + await db.execute_write("insert into t (id) values (1)") + response = await ds.client.get("/-/threads.json") + assert response.json() == {"num_threads": 0, "threads": []} + response2 = await ds.client.get("/test_num_sql_threads_zero/t.json?_shape=array") + assert response2.json() == [{"id": 1}] From 943aa2e1f7341cb51e60332cde46bde650c64217 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 2 May 2022 14:38:34 -0700 Subject: [PATCH 068/952] Release 0.62a0 Refs #1683, #1701, #1712, #1717, #1718, #1733 --- datasette/version.py | 2 +- docs/changelog.rst | 14 ++++++++++++++ 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/datasette/version.py b/datasette/version.py index 02451a1e..cf18c441 100644 --- a/datasette/version.py +++ b/datasette/version.py @@ -1,2 +1,2 @@ -__version__ = "0.61.1" +__version__ = "0.62a0" __version_info__ = tuple(__version__.split(".")) diff --git a/docs/changelog.rst b/docs/changelog.rst index 03cf62b6..74814fcb 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -4,6 +4,20 @@ Changelog ========= +.. _v0_62a0: + +0.62a0 (2022-05-02) +------------------- + +- Datasette now runs some SQL queries in parallel. This has limited impact on performance, see `this research issue `__ for details. +- Datasette should now be compatible with Pyodide. (:issue:`1733`) +- ``datasette publish cloudrun`` has a new ``--timeout`` option which can be used to increase the time limit applied by the Google Cloud build environment. Thanks, Tim Sherratt. (`#1717 `__) +- Spaces in database names are now encoded as ``+`` rather than ``~20``. (:issue:`1701`) +- ```` is now displayed as ```` and is accompanied by tooltip showing "2.3MB". (:issue:`1712`) +- Don't show the facet option in the cog menu if faceting is not allowed. (:issue:`1683`) +- Code examples in the documentation are now all formatted using Black. (:issue:`1718`) +- ``Request.fake()`` method is now documented, see :ref:`internals_request`. + .. _v0_61_1: 0.61.1 (2022-03-23) From 847d6b1aac38c3e776e8c600eed07ba4c9ac9942 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 2 May 2022 16:32:24 -0700 Subject: [PATCH 069/952] Test wheel against Pyodide, refs #1737, #1733 --- .github/workflows/test-pyodide.yml | 28 ++++++++++++++++++ test-in-pyodide-with-shot-scraper.sh | 43 ++++++++++++++++++++++++++++ 2 files changed, 71 insertions(+) create mode 100644 .github/workflows/test-pyodide.yml create mode 100755 test-in-pyodide-with-shot-scraper.sh diff --git a/.github/workflows/test-pyodide.yml b/.github/workflows/test-pyodide.yml new file mode 100644 index 00000000..3715d055 --- /dev/null +++ b/.github/workflows/test-pyodide.yml @@ -0,0 +1,28 @@ +name: Test in Pyodide with shot-scraper + +on: + workflow_dispatch: + +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + - name: Set up Python 3.10 + uses: actions/setup-python@v3 + with: + python-version: "3.10" + cache: 'pip' + cache-dependency-path: '**/setup.py' + - name: Cache Playwright browsers + uses: actions/cache@v2 + with: + path: ~/.cache/ms-playwright/ + key: ${{ runner.os }}-browsers + - name: Install Playwright dependencies + run: | + pip install shot-scraper + shot-scraper install + - name: Run test + run: | + ./test-in-pyodide-with-shot-scraper.sh diff --git a/test-in-pyodide-with-shot-scraper.sh b/test-in-pyodide-with-shot-scraper.sh new file mode 100755 index 00000000..0f29c0e0 --- /dev/null +++ b/test-in-pyodide-with-shot-scraper.sh @@ -0,0 +1,43 @@ +#!/bin/bash + +# Build the wheel +python3 -m build + +# Find name of wheel +wheel=$(basename $(ls dist/*.whl)) +# strip off the dist/ + + +# Create a blank index page +echo ' + +' > dist/index.html + +# Run a server for that dist/ folder +cd dist +python3 -m http.server 8529 & +cd .. + +shot-scraper javascript http://localhost:8529/ " +async () => { + let pyodide = await loadPyodide(); + await pyodide.loadPackage(['micropip', 'ssl', 'setuptools']); + let output = await pyodide.runPythonAsync(\` + import micropip + await micropip.install('h11==0.12.0') + await micropip.install('http://localhost:8529/$wheel') + import ssl + import setuptools + from datasette.app import Datasette + ds = Datasette(memory=True, settings={'num_sql_threads': 0}) + (await ds.client.get('/_memory.json?sql=select+55+as+itworks&_shape=array')).text + \`); + if (JSON.parse(output)[0].itworks != 55) { + throw 'Got ' + output + ', expected itworks: 55'; + } + return 'Test passed!'; +} +" + +# Shut down the server +pkill -f 'http.server 8529' From c0cbcf2aba0d8393ba464acc515803ebf2eeda12 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 2 May 2022 16:36:58 -0700 Subject: [PATCH 070/952] Tweaks to test scripts, refs #1737 --- .github/workflows/test-pyodide.yml | 2 +- test-in-pyodide-with-shot-scraper.sh | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/test-pyodide.yml b/.github/workflows/test-pyodide.yml index 3715d055..beb6a5fb 100644 --- a/.github/workflows/test-pyodide.yml +++ b/.github/workflows/test-pyodide.yml @@ -21,7 +21,7 @@ jobs: key: ${{ runner.os }}-browsers - name: Install Playwright dependencies run: | - pip install shot-scraper + pip install shot-scraper build shot-scraper install - name: Run test run: | diff --git a/test-in-pyodide-with-shot-scraper.sh b/test-in-pyodide-with-shot-scraper.sh index 0f29c0e0..e5df7398 100755 --- a/test-in-pyodide-with-shot-scraper.sh +++ b/test-in-pyodide-with-shot-scraper.sh @@ -1,12 +1,12 @@ #!/bin/bash +set -e +# So the script fails if there are any errors # Build the wheel python3 -m build -# Find name of wheel +# Find name of wheel, strip off the dist/ wheel=$(basename $(ls dist/*.whl)) -# strip off the dist/ - # Create a blank index page echo ' From d60f163528f466b1127b2935c3b6869c34fd6545 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 2 May 2022 16:40:49 -0700 Subject: [PATCH 071/952] Run on push and PR, closes #1737 --- .github/workflows/test-pyodide.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/workflows/test-pyodide.yml b/.github/workflows/test-pyodide.yml index beb6a5fb..1b75aade 100644 --- a/.github/workflows/test-pyodide.yml +++ b/.github/workflows/test-pyodide.yml @@ -1,6 +1,8 @@ name: Test in Pyodide with shot-scraper on: + push: + pull_request: workflow_dispatch: jobs: From 280ff372ab30df244f6c54f6f3002da57334b3d7 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 3 May 2022 07:59:18 -0700 Subject: [PATCH 072/952] ETag support for .db downloads, closes #1739 --- datasette/utils/testing.py | 20 ++++++++++++++++++-- datasette/views/database.py | 7 +++++++ tests/test_html.py | 10 +++++++++- 3 files changed, 34 insertions(+), 3 deletions(-) diff --git a/datasette/utils/testing.py b/datasette/utils/testing.py index 94750b1f..640c94e6 100644 --- a/datasette/utils/testing.py +++ b/datasette/utils/testing.py @@ -55,10 +55,21 @@ class TestClient: @async_to_sync async def get( - self, path, follow_redirects=False, redirect_count=0, method="GET", cookies=None + self, + path, + follow_redirects=False, + redirect_count=0, + method="GET", + cookies=None, + if_none_match=None, ): return await self._request( - path, follow_redirects, redirect_count, method, cookies + path=path, + follow_redirects=follow_redirects, + redirect_count=redirect_count, + method=method, + cookies=cookies, + if_none_match=if_none_match, ) @async_to_sync @@ -110,6 +121,7 @@ class TestClient: headers=None, post_body=None, content_type=None, + if_none_match=None, ): return await self._request( path, @@ -120,6 +132,7 @@ class TestClient: headers=headers, post_body=post_body, content_type=content_type, + if_none_match=if_none_match, ) async def _request( @@ -132,10 +145,13 @@ class TestClient: headers=None, post_body=None, content_type=None, + if_none_match=None, ): headers = headers or {} if content_type: headers["content-type"] = content_type + if if_none_match: + headers["if-none-match"] = if_none_match httpx_response = await self.ds.client.request( method, path, diff --git a/datasette/views/database.py b/datasette/views/database.py index 9a8aca32..bc08ba05 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -183,6 +183,13 @@ class DatabaseDownload(DataView): headers = {} if self.ds.cors: add_cors_headers(headers) + if db.hash: + etag = '"{}"'.format(db.hash) + headers["Etag"] = etag + # Has user seen this already? + if_none_match = request.headers.get("if-none-match") + if if_none_match and if_none_match == etag: + return Response("", status=304) headers["Transfer-Encoding"] = "chunked" return AsgiFileDownload( filepath, diff --git a/tests/test_html.py b/tests/test_html.py index 42f1a3ee..409fec68 100644 --- a/tests/test_html.py +++ b/tests/test_html.py @@ -401,7 +401,7 @@ def test_database_download_for_immutable(): assert len(soup.findAll("a", {"href": re.compile(r"\.db$")})) # Check we can actually download it download_response = client.get("/fixtures.db") - assert 200 == download_response.status + assert download_response.status == 200 # Check the content-length header exists assert "content-length" in download_response.headers content_length = download_response.headers["content-length"] @@ -413,6 +413,14 @@ def test_database_download_for_immutable(): == 'attachment; filename="fixtures.db"' ) assert download_response.headers["transfer-encoding"] == "chunked" + # ETag header should be present and match db.hash + assert "etag" in download_response.headers + etag = download_response.headers["etag"] + assert etag == '"{}"'.format(client.ds.databases["fixtures"].hash) + # Try a second download with If-None-Match: current-etag + download_response2 = client.get("/fixtures.db", if_none_match=etag) + assert download_response2.body == b"" + assert download_response2.status == 304 def test_database_download_disallowed_for_mutable(app_client): From a5acfff4bd364d30ce8878e19f9839890371ef14 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 16 May 2022 17:06:40 -0700 Subject: [PATCH 073/952] Empty Datasette([]) list is no longer required --- docs/testing_plugins.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing_plugins.rst b/docs/testing_plugins.rst index 1bbaaac1..41046bfb 100644 --- a/docs/testing_plugins.rst +++ b/docs/testing_plugins.rst @@ -15,7 +15,7 @@ If you use the template described in :ref:`writing_plugins_cookiecutter` your pl @pytest.mark.asyncio async def test_plugin_is_installed(): - datasette = Datasette([], memory=True) + datasette = Datasette(memory=True) response = await datasette.client.get("/-/plugins.json") assert response.status_code == 200 installed_plugins = {p["name"] for p in response.json()} From 3508bf7875f8d62b2725222f3b07747974d54b97 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 17 May 2022 12:40:05 -0700 Subject: [PATCH 074/952] --nolock mode to ignore locked files, closes #1744 --- datasette/app.py | 2 ++ datasette/cli.py | 7 +++++++ datasette/database.py | 2 ++ docs/cli-reference.rst | 1 + docs/getting_started.rst | 4 +++- 5 files changed, 15 insertions(+), 1 deletion(-) diff --git a/datasette/app.py b/datasette/app.py index b7b84371..f43700d4 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -213,6 +213,7 @@ class Datasette: config_dir=None, pdb=False, crossdb=False, + nolock=False, ): assert config_dir is None or isinstance( config_dir, Path @@ -238,6 +239,7 @@ class Datasette: self.databases = collections.OrderedDict() self._refresh_schemas_lock = asyncio.Lock() self.crossdb = crossdb + self.nolock = nolock if memory or crossdb or not self.files: self.add_database(Database(self, is_memory=True), name="_memory") # memory_name is a random string so that each Datasette instance gets its own diff --git a/datasette/cli.py b/datasette/cli.py index 3c6e1b2c..8781747c 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -452,6 +452,11 @@ def uninstall(packages, yes): is_flag=True, help="Enable cross-database joins using the /_memory database", ) +@click.option( + "--nolock", + is_flag=True, + help="Ignore locking, open locked files in read-only mode", +) @click.option( "--ssl-keyfile", help="SSL key file", @@ -486,6 +491,7 @@ def serve( open_browser, create, crossdb, + nolock, ssl_keyfile, ssl_certfile, return_instance=False, @@ -545,6 +551,7 @@ def serve( version_note=version_note, pdb=pdb, crossdb=crossdb, + nolock=nolock, ) # if files is a single directory, use that as config_dir= diff --git a/datasette/database.py b/datasette/database.py index 44d32667..fa558045 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -89,6 +89,8 @@ class Database: # mode=ro or immutable=1? if self.is_mutable: qs = "?mode=ro" + if self.ds.nolock: + qs += "&nolock=1" else: qs = "?immutable=1" assert not (write and not self.is_mutable) diff --git a/docs/cli-reference.rst b/docs/cli-reference.rst index 2a6fbfc8..1c1aff15 100644 --- a/docs/cli-reference.rst +++ b/docs/cli-reference.rst @@ -115,6 +115,7 @@ datasette serve --help --create Create database files if they do not exist --crossdb Enable cross-database joins using the /_memory database + --nolock Ignore locking, open locked files in read-only mode --ssl-keyfile TEXT SSL key file --ssl-certfile TEXT SSL certificate file --help Show this message and exit. diff --git a/docs/getting_started.rst b/docs/getting_started.rst index 3e357afb..502a9e5a 100644 --- a/docs/getting_started.rst +++ b/docs/getting_started.rst @@ -56,7 +56,9 @@ like so: :: - datasette ~/Library/Application\ Support/Google/Chrome/Default/History + datasette ~/Library/Application\ Support/Google/Chrome/Default/History --nolock + +The `--nolock` option ignores any file locks. This is safe as Datasette will open the file in read-only mode. Now visiting http://localhost:8001/History/downloads will show you a web interface to browse your downloads data: From 5555bc8aef043f75d2200f66de90c54aeeaa08c3 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 17 May 2022 12:43:44 -0700 Subject: [PATCH 075/952] How to run cog, closes #1745 --- docs/contributing.rst | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/contributing.rst b/docs/contributing.rst index c193ba49..bddceafe 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -211,6 +211,17 @@ For added productivity, you can use use `sphinx-autobuild `__. + +To update these pages, run the following command:: + + cog -r docs/*.rst + .. _contributing_continuous_deployment: Continuously deployed demo instances From b393e164dc9e962702546d6f1ad9c857b5788dc0 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 17 May 2022 12:45:28 -0700 Subject: [PATCH 076/952] ReST fix --- docs/getting_started.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting_started.rst b/docs/getting_started.rst index 502a9e5a..af3a1385 100644 --- a/docs/getting_started.rst +++ b/docs/getting_started.rst @@ -58,7 +58,7 @@ like so: datasette ~/Library/Application\ Support/Google/Chrome/Default/History --nolock -The `--nolock` option ignores any file locks. This is safe as Datasette will open the file in read-only mode. +The ``--nolock`` option ignores any file locks. This is safe as Datasette will open the file in read-only mode. Now visiting http://localhost:8001/History/downloads will show you a web interface to browse your downloads data: From 7d1e004ff679b3fb4dca36d1d751a1ad16688fe6 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 17 May 2022 12:59:28 -0700 Subject: [PATCH 077/952] Fix test I broke in #1744 --- tests/test_cli.py | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/test_cli.py b/tests/test_cli.py index dca65f26..d0f6e26c 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -150,6 +150,7 @@ def test_metadata_yaml(): help_settings=False, pdb=False, crossdb=False, + nolock=False, open_browser=False, create=False, ssl_keyfile=None, From 0e2f6f1f82f4445a63f1251470a7778a34f5c8b9 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 18 May 2022 17:37:46 -0700 Subject: [PATCH 078/952] datasette-copyable is an example of register_output_renderer --- docs/plugin_hooks.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index 3c9ae2e2..c0d88964 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -557,7 +557,7 @@ And here is an example ``can_render`` function which returns ``True`` only if th "atom_updated", }.issubset(columns) -Examples: `datasette-atom `_, `datasette-ics `_, `datasette-geojson `__ +Examples: `datasette-atom `_, `datasette-ics `_, `datasette-geojson `__, `datasette-copyable `__ .. _plugin_register_routes: From 18a6e05887abf1ac946a6e0d36ce662dfd8aeff1 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 20 May 2022 12:05:33 -0700 Subject: [PATCH 079/952] Added "follow a tutorial" to getting started docs Closes #1747 --- docs/getting_started.rst | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/getting_started.rst b/docs/getting_started.rst index af3a1385..00b753a9 100644 --- a/docs/getting_started.rst +++ b/docs/getting_started.rst @@ -1,6 +1,8 @@ Getting started =============== +.. _getting_started_demo: + Play with a live demo --------------------- @@ -9,6 +11,16 @@ The best way to experience Datasette for the first time is with a demo: * `global-power-plants.datasettes.com `__ provides a searchable database of power plants around the world, using data from the `World Resources Institude `__ rendered using the `datasette-cluster-map `__ plugin. * `fivethirtyeight.datasettes.com `__ shows Datasette running against over 400 datasets imported from the `FiveThirtyEight GitHub repository `__. +.. _getting_started_tutorial: + +Follow a tutorial +----------------- + +Datasette has several `tutorials `__ to help you get started with the tool. Try one of the following: + +- `Exploring a database with Datasette `__ shows how to use the Datasette web interface to explore a new database. +- `Learn SQL with Datasette `__ introduces SQL, and shows how to use that query language to ask questions of your data. + .. _getting_started_glitch: Try Datasette without installing anything using Glitch From 1465fea4798599eccfe7e8f012bd8d9adfac3039 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 20 May 2022 12:11:08 -0700 Subject: [PATCH 080/952] sphinx-copybutton for docs, closes #1748 --- docs/conf.py | 2 +- setup.py | 8 +++++++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/conf.py b/docs/conf.py index d114bc52..351cb1b1 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -31,7 +31,7 @@ # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. -extensions = ["sphinx.ext.extlinks", "sphinx.ext.autodoc"] +extensions = ["sphinx.ext.extlinks", "sphinx.ext.autodoc", "sphinx_copybutton"] extlinks = { "issue": ("https://github.com/simonw/datasette/issues/%s", "#"), diff --git a/setup.py b/setup.py index ca449f02..aad05840 100644 --- a/setup.py +++ b/setup.py @@ -64,7 +64,13 @@ setup( """, setup_requires=["pytest-runner"], extras_require={ - "docs": ["sphinx_rtd_theme", "sphinx-autobuild", "codespell", "blacken-docs"], + "docs": [ + "sphinx_rtd_theme", + "sphinx-autobuild", + "codespell", + "blacken-docs", + "sphinx-copybutton", + ], "test": [ "pytest>=5.2.2,<7.2.0", "pytest-xdist>=2.2.1,<2.6", From 1d33fd03b3c211e0f48a8f3bde83880af89e4e69 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 20 May 2022 13:34:51 -0700 Subject: [PATCH 081/952] Switch docs theme to Furo, refs #1746 --- docs/_static/css/custom.css | 7 ++-- .../layout.html => _static/js/custom.js} | 34 ------------------- docs/_templates/base.html | 6 ++++ docs/_templates/sidebar/brand.html | 16 +++++++++ docs/_templates/sidebar/navigation.html | 11 ++++++ docs/conf.py | 24 +++---------- docs/installation.rst | 1 + docs/plugin_hooks.rst | 1 + setup.py | 2 +- 9 files changed, 45 insertions(+), 57 deletions(-) rename docs/{_templates/layout.html => _static/js/custom.js} (55%) create mode 100644 docs/_templates/base.html create mode 100644 docs/_templates/sidebar/brand.html create mode 100644 docs/_templates/sidebar/navigation.html diff --git a/docs/_static/css/custom.css b/docs/_static/css/custom.css index 4dabb725..0a6f8799 100644 --- a/docs/_static/css/custom.css +++ b/docs/_static/css/custom.css @@ -1,7 +1,8 @@ a.external { overflow-wrap: anywhere; } - -div .wy-side-nav-search > div.version { - color: rgba(0,0,0,0.75); +body[data-theme="dark"] .sidebar-logo-container { + background-color: white; + padding: 5px; + opacity: 0.6; } diff --git a/docs/_templates/layout.html b/docs/_static/js/custom.js similarity index 55% rename from docs/_templates/layout.html rename to docs/_static/js/custom.js index 785cdc7c..efca33ed 100644 --- a/docs/_templates/layout.html +++ b/docs/_static/js/custom.js @@ -1,35 +1,3 @@ -{%- extends "!layout.html" %} - -{% block htmltitle %} -{{ super() }} - -{% endblock %} - -{% block sidebartitle %} - - - - - -{% if theme_display_version %} - {%- set nav_version = version %} - {% if READTHEDOCS and current_version %} - {%- set nav_version = current_version %} - {% endif %} - {% if nav_version %} -
- {{ nav_version }} -
- {% endif %} -{% endif %} - -{% include "searchbox.html" %} - -{% endblock %} - -{% block footer %} -{{ super() }} - -{% endblock %} diff --git a/docs/_templates/base.html b/docs/_templates/base.html new file mode 100644 index 00000000..969de5ab --- /dev/null +++ b/docs/_templates/base.html @@ -0,0 +1,6 @@ +{%- extends "!base.html" %} + +{% block site_meta %} +{{ super() }} + +{% endblock %} diff --git a/docs/_templates/sidebar/brand.html b/docs/_templates/sidebar/brand.html new file mode 100644 index 00000000..8be9e8ee --- /dev/null +++ b/docs/_templates/sidebar/brand.html @@ -0,0 +1,16 @@ +
+ {% block brand_content %} +
+ +
+ {%- set nav_version = version %} + {% if READTHEDOCS and current_version %} + {%- set nav_version = current_version %} + {% endif %} + {% if nav_version %} +
+ {{ nav_version }} +
+ {% endif %} + {% endblock brand_content %} +
diff --git a/docs/_templates/sidebar/navigation.html b/docs/_templates/sidebar/navigation.html new file mode 100644 index 00000000..c460a17e --- /dev/null +++ b/docs/_templates/sidebar/navigation.html @@ -0,0 +1,11 @@ +
+ + {{ toctree( + collapse=True, + titles_only=False, + maxdepth=3, + includehidden=True, +) }} +
\ No newline at end of file diff --git a/docs/conf.py b/docs/conf.py index 351cb1b1..25d2acfe 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -90,18 +90,15 @@ todo_include_todos = False # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. # -html_theme = "sphinx_rtd_theme" +html_theme = "furo" # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the # documentation. html_theme_options = { - "logo_only": True, - "style_nav_header_background": "white", - "prev_next_buttons_location": "both", + "sidebar_hide_name": True, } - # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". @@ -112,20 +109,9 @@ html_logo = "datasette-logo.svg" html_css_files = [ "css/custom.css", ] - - -# Custom sidebar templates, must be a dictionary that maps document names -# to template names. -# -# This is required for the alabaster theme -# refs: http://alabaster.readthedocs.io/en/latest/installation.html#sidebars -html_sidebars = { - "**": [ - "relations.html", # needs 'show_related': True theme option to display - "searchbox.html", - ] -} - +html_js_files = [ + "js/custom.js" +] # -- Options for HTMLHelp output ------------------------------------------ diff --git a/docs/installation.rst b/docs/installation.rst index e8bef9cd..a4757736 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -13,6 +13,7 @@ If you want to start making contributions to the Datasette project by installing .. contents:: :local: + :class: this-will-duplicate-information-and-it-is-still-useful-here .. _installation_basic: diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index c0d88964..7d10fe37 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -20,6 +20,7 @@ For example, you can implement the ``render_cell`` plugin hook like this even th .. contents:: List of plugin hooks :local: + :class: this-will-duplicate-information-and-it-is-still-useful-here .. _plugin_hook_prepare_connection: diff --git a/setup.py b/setup.py index aad05840..d3fcdbd1 100644 --- a/setup.py +++ b/setup.py @@ -65,7 +65,7 @@ setup( setup_requires=["pytest-runner"], extras_require={ "docs": [ - "sphinx_rtd_theme", + "furo==2022.4.7", "sphinx-autobuild", "codespell", "blacken-docs", From 4446075334ea7231beb56b630bc7ec363afc2d08 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 20 May 2022 13:44:23 -0700 Subject: [PATCH 082/952] Append warning to the write element, refs #1746 --- docs/_static/js/custom.js | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/docs/_static/js/custom.js b/docs/_static/js/custom.js index efca33ed..91c3e306 100644 --- a/docs/_static/js/custom.js +++ b/docs/_static/js/custom.js @@ -17,11 +17,7 @@ jQuery(function ($) { ` ); warning.find("a").attr("href", stableUrl); - var body = $("div.body"); - if (!body.length) { - body = $("div.document"); - } - body.prepend(warning); + $("article[role=main]").prepend(warning); } }); }); From b010af7bb85856aeb44f69e7e980f617c1fc0db1 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 20 May 2022 15:23:09 -0700 Subject: [PATCH 083/952] Updated copyright years in documentation footer --- docs/conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/conf.py b/docs/conf.py index 25d2acfe..7ffeedd0 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -51,7 +51,7 @@ master_doc = "index" # General information about the project. project = "Datasette" -copyright = "2017-2021, Simon Willison" +copyright = "2017-2022, Simon Willison" author = "Simon Willison" # Disable -- turning into – From adedd85b68ec66e03b97fb62ff4da8987734436e Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sat, 28 May 2022 18:42:31 -0700 Subject: [PATCH 084/952] Clarify that request.headers names are converted to lowercase --- docs/internals.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/internals.rst b/docs/internals.rst index 18822d47..da135282 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -26,7 +26,7 @@ The request object is passed to various plugin hooks. It represents an incoming The request scheme - usually ``https`` or ``http``. ``.headers`` - dictionary (str -> str) - A dictionary of incoming HTTP request headers. + A dictionary of incoming HTTP request headers. Header names have been converted to lowercase. ``.cookies`` - dictionary (str -> str) A dictionary of incoming cookies From 8dd816bc76937f1e37f86acce10dc2cb4fa31e52 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 30 May 2022 15:42:38 -0700 Subject: [PATCH 085/952] Applied Black --- docs/conf.py | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/conf.py b/docs/conf.py index 7ffeedd0..4ef6b768 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -109,9 +109,7 @@ html_logo = "datasette-logo.svg" html_css_files = [ "css/custom.css", ] -html_js_files = [ - "js/custom.js" -] +html_js_files = ["js/custom.js"] # -- Options for HTMLHelp output ------------------------------------------ From 2e9751672d4fe329b3c359d5b7b1992283185820 Mon Sep 17 00:00:00 2001 From: Naveen <172697+naveensrinivasan@users.noreply.github.com> Date: Tue, 31 May 2022 14:28:40 -0500 Subject: [PATCH 086/952] chore: Set permissions for GitHub actions (#1740) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Restrict the GitHub token permissions only to the required ones; this way, even if the attackers will succeed in compromising your workflow, they won’t be able to do much. - Included permissions for the action. https://github.com/ossf/scorecard/blob/main/docs/checks.md#token-permissions https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#permissions https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs [Keeping your GitHub Actions and workflows secure Part 1: Preventing pwn requests](https://securitylab.github.com/research/github-actions-preventing-pwn-requests/) Signed-off-by: naveen <172697+naveensrinivasan@users.noreply.github.com> --- .github/workflows/deploy-latest.yml | 3 +++ .github/workflows/prettier.yml | 3 +++ .github/workflows/publish.yml | 3 +++ .github/workflows/push_docker_tag.yml | 3 +++ .github/workflows/spellcheck.yml | 3 +++ .github/workflows/test-coverage.yml | 3 +++ .github/workflows/test-pyodide.yml | 3 +++ .github/workflows/test.yml | 3 +++ .github/workflows/tmate-mac.yml | 3 +++ .github/workflows/tmate.yml | 3 +++ 10 files changed, 30 insertions(+) diff --git a/.github/workflows/deploy-latest.yml b/.github/workflows/deploy-latest.yml index a61f6629..2b94a7f1 100644 --- a/.github/workflows/deploy-latest.yml +++ b/.github/workflows/deploy-latest.yml @@ -5,6 +5,9 @@ on: branches: - main +permissions: + contents: read + jobs: deploy: runs-on: ubuntu-latest diff --git a/.github/workflows/prettier.yml b/.github/workflows/prettier.yml index 9dfe7ee0..ded41040 100644 --- a/.github/workflows/prettier.yml +++ b/.github/workflows/prettier.yml @@ -2,6 +2,9 @@ name: Check JavaScript for conformance with Prettier on: [push] +permissions: + contents: read + jobs: prettier: runs-on: ubuntu-latest diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 3e4f8146..9ef09d2e 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -4,6 +4,9 @@ on: release: types: [created] +permissions: + contents: read + jobs: test: runs-on: ubuntu-latest diff --git a/.github/workflows/push_docker_tag.yml b/.github/workflows/push_docker_tag.yml index 9a3969f0..afe8d6b2 100644 --- a/.github/workflows/push_docker_tag.yml +++ b/.github/workflows/push_docker_tag.yml @@ -6,6 +6,9 @@ on: version_tag: description: Tag to build and push +permissions: + contents: read + jobs: deploy_docker: runs-on: ubuntu-latest diff --git a/.github/workflows/spellcheck.yml b/.github/workflows/spellcheck.yml index 2e24d3eb..a2621ecc 100644 --- a/.github/workflows/spellcheck.yml +++ b/.github/workflows/spellcheck.yml @@ -2,6 +2,9 @@ name: Check spelling in documentation on: [push, pull_request] +permissions: + contents: read + jobs: spellcheck: runs-on: ubuntu-latest diff --git a/.github/workflows/test-coverage.yml b/.github/workflows/test-coverage.yml index 1d1cf332..bd720664 100644 --- a/.github/workflows/test-coverage.yml +++ b/.github/workflows/test-coverage.yml @@ -7,6 +7,9 @@ on: pull_request: branches: - main +permissions: + contents: read + jobs: test: runs-on: ubuntu-latest diff --git a/.github/workflows/test-pyodide.yml b/.github/workflows/test-pyodide.yml index 1b75aade..bc9593a8 100644 --- a/.github/workflows/test-pyodide.yml +++ b/.github/workflows/test-pyodide.yml @@ -5,6 +5,9 @@ on: pull_request: workflow_dispatch: +permissions: + contents: read + jobs: test: runs-on: ubuntu-latest diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 8d916e49..90b6555e 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -2,6 +2,9 @@ name: Test on: [push, pull_request] +permissions: + contents: read + jobs: test: runs-on: ubuntu-latest diff --git a/.github/workflows/tmate-mac.yml b/.github/workflows/tmate-mac.yml index 46be117e..fcee0f21 100644 --- a/.github/workflows/tmate-mac.yml +++ b/.github/workflows/tmate-mac.yml @@ -3,6 +3,9 @@ name: tmate session mac on: workflow_dispatch: +permissions: + contents: read + jobs: build: runs-on: macos-latest diff --git a/.github/workflows/tmate.yml b/.github/workflows/tmate.yml index 02e7bd33..9792245d 100644 --- a/.github/workflows/tmate.yml +++ b/.github/workflows/tmate.yml @@ -3,6 +3,9 @@ name: tmate session on: workflow_dispatch: +permissions: + contents: read + jobs: build: runs-on: ubuntu-latest From e780b2f5d662ef3579d801d33567440055d4e84d Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 20 Jun 2022 10:54:23 -0700 Subject: [PATCH 087/952] Trying out one-sentence-per-line As suggested here: https://sive.rs/1s Markdown and reStructuredText will display this as if it is a single paragraph, even though the sentences themselves are separated by newlines. This could result in more useful diffs. Trying it out on this page first. --- docs/facets.rst | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/docs/facets.rst b/docs/facets.rst index 0228aa84..2a2eb039 100644 --- a/docs/facets.rst +++ b/docs/facets.rst @@ -3,7 +3,9 @@ Facets ====== -Datasette facets can be used to add a faceted browse interface to any database table. With facets, tables are displayed along with a summary showing the most common values in specified columns. These values can be selected to further filter the table. +Datasette facets can be used to add a faceted browse interface to any database table. +With facets, tables are displayed along with a summary showing the most common values in specified columns. +These values can be selected to further filter the table. .. image:: facets.png @@ -12,11 +14,13 @@ Facets can be specified in two ways: using query string parameters, or in ``meta Facets in query strings ----------------------- -To turn on faceting for specific columns on a Datasette table view, add one or more ``_facet=COLUMN`` parameters to the URL. For example, if you want to turn on facets for the ``city_id`` and ``state`` columns, construct a URL that looks like this:: +To turn on faceting for specific columns on a Datasette table view, add one or more ``_facet=COLUMN`` parameters to the URL. +For example, if you want to turn on facets for the ``city_id`` and ``state`` columns, construct a URL that looks like this:: /dbname/tablename?_facet=state&_facet=city_id -This works for both the HTML interface and the ``.json`` view. When enabled, facets will cause a ``facet_results`` block to be added to the JSON output, looking something like this: +This works for both the HTML interface and the ``.json`` view. +When enabled, facets will cause a ``facet_results`` block to be added to the JSON output, looking something like this: .. code-block:: json @@ -86,7 +90,8 @@ This works for both the HTML interface and the ``.json`` view. When enabled, fac If Datasette detects that a column is a foreign key, the ``"label"`` property will be automatically derived from the detected label column on the referenced table. -The default number of facet results returned is 30, controlled by the :ref:`setting_default_facet_size` setting. You can increase this on an individual page by adding ``?_facet_size=100`` to the query string, up to a maximum of :ref:`setting_max_returned_rows` (which defaults to 1000). +The default number of facet results returned is 30, controlled by the :ref:`setting_default_facet_size` setting. +You can increase this on an individual page by adding ``?_facet_size=100`` to the query string, up to a maximum of :ref:`setting_max_returned_rows` (which defaults to 1000). .. _facets_metadata: @@ -137,12 +142,14 @@ For the currently filtered data are there any columns which, if applied as a fac * Will return less unique options than the total number of filtered rows * And the query used to evaluate this criteria can be completed in under 50ms -That last point is particularly important: Datasette runs a query for every column that is displayed on a page, which could get expensive - so to avoid slow load times it sets a time limit of just 50ms for each of those queries. This means suggested facets are unlikely to appear for tables with millions of records in them. +That last point is particularly important: Datasette runs a query for every column that is displayed on a page, which could get expensive - so to avoid slow load times it sets a time limit of just 50ms for each of those queries. +This means suggested facets are unlikely to appear for tables with millions of records in them. Speeding up facets with indexes ------------------------------- -The performance of facets can be greatly improved by adding indexes on the columns you wish to facet by. Adding indexes can be performed using the ``sqlite3`` command-line utility. Here's how to add an index on the ``state`` column in a table called ``Food_Trucks``:: +The performance of facets can be greatly improved by adding indexes on the columns you wish to facet by. +Adding indexes can be performed using the ``sqlite3`` command-line utility. Here's how to add an index on the ``state`` column in a table called ``Food_Trucks``:: $ sqlite3 mydatabase.db SQLite version 3.19.3 2017-06-27 16:48:08 @@ -169,6 +176,7 @@ Example here: `latest.datasette.io/fixtures/facetable?_facet_array=tags `__ From 00e59ec461dc0150772b999c7cc15fcb9b507d58 Mon Sep 17 00:00:00 2001 From: "M. Nasimul Haque" Date: Mon, 20 Jun 2022 19:05:44 +0100 Subject: [PATCH 088/952] Extract facet pieces of table.html into included templates Thanks, @nsmgr8 --- datasette/templates/_facet_results.html | 28 ++++++++++++++++++ datasette/templates/_suggested_facets.html | 3 ++ datasette/templates/table.html | 33 ++-------------------- 3 files changed, 33 insertions(+), 31 deletions(-) create mode 100644 datasette/templates/_facet_results.html create mode 100644 datasette/templates/_suggested_facets.html diff --git a/datasette/templates/_facet_results.html b/datasette/templates/_facet_results.html new file mode 100644 index 00000000..d0cbcf77 --- /dev/null +++ b/datasette/templates/_facet_results.html @@ -0,0 +1,28 @@ +
+ {% for facet_info in sorted_facet_results %} +
+

+ {{ facet_info.name }}{% if facet_info.type != "column" %} ({{ facet_info.type }}){% endif %} + {% if facet_info.truncated %}>{% endif %}{{ facet_info.results|length }} + + {% if facet_info.hideable %} + ✖ + {% endif %} +

+
    + {% for facet_value in facet_info.results %} + {% if not facet_value.selected %} +
  • {{ (facet_value.label | string()) or "-" }} {{ "{:,}".format(facet_value.count) }}
    • + {% else %} +
  • {{ facet_value.label or "-" }} · {{ "{:,}".format(facet_value.count) }} ✖
    • + {% endif %} + {% endfor %} + {% if facet_info.truncated %} +
  • {% if request.args._facet_size != "max" -%} + …{% else -%}…{% endif %} +
    • + {% endif %} +
+
+ {% endfor %} +
diff --git a/datasette/templates/_suggested_facets.html b/datasette/templates/_suggested_facets.html new file mode 100644 index 00000000..ec98fb36 --- /dev/null +++ b/datasette/templates/_suggested_facets.html @@ -0,0 +1,3 @@ +

+ Suggested facets: {% for facet in suggested_facets %}{{ facet.name }}{% if facet.type %} ({{ facet.type }}){% endif %}{% if not loop.last %}, {% endif %}{% endfor %} +

diff --git a/datasette/templates/table.html b/datasette/templates/table.html index a9e88330..a86398ea 100644 --- a/datasette/templates/table.html +++ b/datasette/templates/table.html @@ -142,9 +142,7 @@

This data as {% for name, url in renderers.items() %}{{ name }}{{ ", " if not loop.last }}{% endfor %}{% if display_rows %}, CSV (advanced){% endif %}

{% if suggested_facets %} -

- Suggested facets: {% for facet in suggested_facets %}{{ facet.name }}{% if facet.type %} ({{ facet.type }}){% endif %}{% if not loop.last %}, {% endif %}{% endfor %} -

+ {% include "_suggested_facets.html" %} {% endif %} {% if facets_timed_out %} @@ -152,34 +150,7 @@ {% endif %} {% if facet_results %} -
- {% for facet_info in sorted_facet_results %} -
-

- {{ facet_info.name }}{% if facet_info.type != "column" %} ({{ facet_info.type }}){% endif %} - {% if facet_info.truncated %}>{% endif %}{{ facet_info.results|length }} - - {% if facet_info.hideable %} - ✖ - {% endif %} -

-
    - {% for facet_value in facet_info.results %} - {% if not facet_value.selected %} -
  • {{ (facet_value.label | string()) or "-" }} {{ "{:,}".format(facet_value.count) }}
    • - {% else %} -
  • {{ facet_value.label or "-" }} · {{ "{:,}".format(facet_value.count) }} ✖
    • - {% endif %} - {% endfor %} - {% if facet_info.truncated %} -
  • {% if request.args._facet_size != "max" -%} - …{% else -%}…{% endif %} -
    • - {% endif %} -
-
- {% endfor %} -
+ {% include "_facet_results.html" %} {% endif %} {% include custom_table_templates %} From 9f1eb0d4eac483b953392157bd9fd6cc4df37de7 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 28 Jun 2022 10:40:24 -0700 Subject: [PATCH 089/952] Bump black from 22.1.0 to 22.6.0 (#1763) Bumps [black](https://github.com/psf/black) from 22.1.0 to 22.6.0. - [Release notes](https://github.com/psf/black/releases) - [Changelog](https://github.com/psf/black/blob/main/CHANGES.md) - [Commits](https://github.com/psf/black/compare/22.1.0...22.6.0) --- updated-dependencies: - dependency-name: black dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index d3fcdbd1..29cb77bf 100644 --- a/setup.py +++ b/setup.py @@ -76,7 +76,7 @@ setup( "pytest-xdist>=2.2.1,<2.6", "pytest-asyncio>=0.17,<0.19", "beautifulsoup4>=4.8.1,<4.12.0", - "black==22.1.0", + "black==22.6.0", "blacken-docs==1.12.1", "pytest-timeout>=1.4.2,<2.2", "trustme>=0.7,<0.10", From 6373bb341457e5becfd5b67792ac2c8b9ed7c384 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 7 Jul 2022 09:30:49 -0700 Subject: [PATCH 090/952] Expose current SQLite row to render_cell hook, closes #1300 --- datasette/hookspecs.py | 2 +- datasette/views/database.py | 1 + datasette/views/table.py | 1 + docs/plugin_hooks.rst | 9 ++++++--- tests/plugins/my_plugin.py | 3 ++- tests/test_plugins.py | 5 +++-- 6 files changed, 14 insertions(+), 7 deletions(-) diff --git a/datasette/hookspecs.py b/datasette/hookspecs.py index 8f4fecab..c84db0a3 100644 --- a/datasette/hookspecs.py +++ b/datasette/hookspecs.py @@ -60,7 +60,7 @@ def publish_subcommand(publish): @hookspec -def render_cell(value, column, table, database, datasette): +def render_cell(row, value, column, table, database, datasette): """Customize rendering of HTML table cell values""" diff --git a/datasette/views/database.py b/datasette/views/database.py index bc08ba05..42058752 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -375,6 +375,7 @@ class QueryView(DataView): # pylint: disable=no-member plugin_display_value = None for candidate in pm.hook.render_cell( + row=row, value=value, column=column, table=None, diff --git a/datasette/views/table.py b/datasette/views/table.py index 23289b29..cd4be823 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -895,6 +895,7 @@ async def display_columns_and_rows( # pylint: disable=no-member plugin_display_value = None for candidate in pm.hook.render_cell( + row=row, value=value, column=column, table=table_name, diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index 7d10fe37..f5c3ee83 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -373,12 +373,15 @@ Examples: `datasette-publish-fly Date: Sat, 9 Jul 2022 10:25:37 -0700 Subject: [PATCH 091/952] More than 90 plugins now --- docs/writing_plugins.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/writing_plugins.rst b/docs/writing_plugins.rst index 9aee70f6..01ee8c90 100644 --- a/docs/writing_plugins.rst +++ b/docs/writing_plugins.rst @@ -5,7 +5,7 @@ Writing plugins You can write one-off plugins that apply to just one Datasette instance, or you can write plugins which can be installed using ``pip`` and can be shipped to the Python Package Index (`PyPI `__) for other people to install. -Want to start by looking at an example? The `Datasette plugins directory `__ lists more than 50 open source plugins with code you can explore. The :ref:`plugin hooks ` page includes links to example plugins for each of the documented hooks. +Want to start by looking at an example? The `Datasette plugins directory `__ lists more than 90 open source plugins with code you can explore. The :ref:`plugin hooks ` page includes links to example plugins for each of the documented hooks. .. _writing_plugins_one_off: From 5d76c1f81b2d978f48b85c70d041a2142cf8ee26 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 14 Jul 2022 15:03:33 -0700 Subject: [PATCH 092/952] Discord badge Refs https://github.com/simonw/datasette.io/issues/112 --- README.md | 1 + docs/index.rst | 2 ++ 2 files changed, 3 insertions(+) diff --git a/README.md b/README.md index 557d9290..c57ee604 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,7 @@ [![Documentation Status](https://readthedocs.org/projects/datasette/badge/?version=latest)](https://docs.datasette.io/en/latest/?badge=latest) [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette/blob/main/LICENSE) [![docker: datasette](https://img.shields.io/badge/docker-datasette-blue)](https://hub.docker.com/r/datasetteproject/datasette) +[![discord](https://img.shields.io/discord/823971286308356157?label=Discord)](https://discord.gg/ktd74dm5mw) *An open source multi-tool for exploring and publishing data* diff --git a/docs/index.rst b/docs/index.rst index a2888822..62ed70f8 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -16,6 +16,8 @@ datasette| :target: https://github.com/simonw/datasette/blob/main/LICENSE .. |docker: datasette| image:: https://img.shields.io/badge/docker-datasette-blue :target: https://hub.docker.com/r/datasetteproject/datasette +.. |discord| image:: https://img.shields.io/discord/823971286308356157?label=Discord + :target: https://discord.gg/ktd74dm5mw *An open source multi-tool for exploring and publishing data* From c133545fe9c7ac2d509e55bf4bf6164bfbe892ad Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 14 Jul 2022 15:04:38 -0700 Subject: [PATCH 093/952] Make discord badge lowercase Refs https://github.com/simonw/datasette.io/issues/112 --- README.md | 2 +- docs/index.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index c57ee604..032180aa 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ [![Documentation Status](https://readthedocs.org/projects/datasette/badge/?version=latest)](https://docs.datasette.io/en/latest/?badge=latest) [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette/blob/main/LICENSE) [![docker: datasette](https://img.shields.io/badge/docker-datasette-blue)](https://hub.docker.com/r/datasetteproject/datasette) -[![discord](https://img.shields.io/discord/823971286308356157?label=Discord)](https://discord.gg/ktd74dm5mw) +[![discord](https://img.shields.io/discord/823971286308356157?label=discord)](https://discord.gg/ktd74dm5mw) *An open source multi-tool for exploring and publishing data* diff --git a/docs/index.rst b/docs/index.rst index 62ed70f8..051898b1 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -16,7 +16,7 @@ datasette| :target: https://github.com/simonw/datasette/blob/main/LICENSE .. |docker: datasette| image:: https://img.shields.io/badge/docker-datasette-blue :target: https://hub.docker.com/r/datasetteproject/datasette -.. |discord| image:: https://img.shields.io/discord/823971286308356157?label=Discord +.. |discord| image:: https://img.shields.io/discord/823971286308356157?label=discord :target: https://discord.gg/ktd74dm5mw *An open source multi-tool for exploring and publishing data* From 950cc7677f65aa2543067b3bbfc2b6acb98b62c8 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 14 Jul 2022 15:18:28 -0700 Subject: [PATCH 094/952] Fix missing Discord image Refs https://github.com/simonw/datasette.io/issues/112 --- docs/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/index.rst b/docs/index.rst index 051898b1..efe196b3 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -2,7 +2,7 @@ Datasette ========= |PyPI| |Changelog| |Python 3.x| |Tests| |License| |docker: -datasette| +datasette| |discord| .. |PyPI| image:: https://img.shields.io/pypi/v/datasette.svg :target: https://pypi.org/project/datasette/ From 8188f55efc0fcca1be692b0d0c875f2d1ee99f17 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 17 Jul 2022 15:24:16 -0700 Subject: [PATCH 095/952] Rename handle_500 to handle_exception, refs #1770 --- datasette/app.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index f43700d4..43e60dbc 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -1275,7 +1275,7 @@ class DatasetteRouter: except NotFound as exception: return await self.handle_404(request, send, exception) except Exception as exception: - return await self.handle_500(request, send, exception) + return await self.handle_exception(request, send, exception) async def handle_404(self, request, send, exception=None): # If path contains % encoding, redirect to tilde encoding @@ -1354,7 +1354,7 @@ class DatasetteRouter: view_name="page", ) except NotFoundExplicit as e: - await self.handle_500(request, send, e) + await self.handle_exception(request, send, e) return # Pull content-type out into separate parameter content_type = "text/html; charset=utf-8" @@ -1369,9 +1369,9 @@ class DatasetteRouter: content_type=content_type, ) else: - await self.handle_500(request, send, exception or NotFound("404")) + await self.handle_exception(request, send, exception or NotFound("404")) - async def handle_500(self, request, send, exception): + async def handle_exception(self, request, send, exception): if self.ds.pdb: import pdb From c09c53f3455a7b9574cf7695478f2b87d20897db Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 17 Jul 2022 16:24:39 -0700 Subject: [PATCH 096/952] New handle_exception plugin hook, refs #1770 Also refs: - https://github.com/simonw/datasette-sentry/issues/1 - https://github.com/simonw/datasette-show-errors/issues/2 --- datasette/app.py | 97 +++++++++-------------------------- datasette/forbidden.py | 20 ++++++++ datasette/handle_exception.py | 74 ++++++++++++++++++++++++++ datasette/hookspecs.py | 5 ++ datasette/plugins.py | 2 + docs/plugin_hooks.rst | 78 ++++++++++++++++++++-------- tests/fixtures.py | 1 + tests/plugins/my_plugin_2.py | 18 +++++++ tests/test_permissions.py | 1 + tests/test_plugins.py | 14 +++++ 10 files changed, 215 insertions(+), 95 deletions(-) create mode 100644 datasette/forbidden.py create mode 100644 datasette/handle_exception.py diff --git a/datasette/app.py b/datasette/app.py index 43e60dbc..edd05bb3 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -16,7 +16,6 @@ import re import secrets import sys import threading -import traceback import urllib.parse from concurrent import futures from pathlib import Path @@ -27,7 +26,7 @@ from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader from jinja2.environment import Template from jinja2.exceptions import TemplateNotFound -from .views.base import DatasetteError, ureg +from .views.base import ureg from .views.database import DatabaseDownload, DatabaseView from .views.index import IndexView from .views.special import ( @@ -49,7 +48,6 @@ from .utils import ( PrefixedUrlString, SPATIALITE_FUNCTIONS, StartupError, - add_cors_headers, async_call_with_supported_arguments, await_me_maybe, call_with_supported_arguments, @@ -87,11 +85,6 @@ from .tracer import AsgiTracer from .plugins import pm, DEFAULT_PLUGINS, get_plugins from .version import __version__ -try: - import rich -except ImportError: - rich = None - app_root = Path(__file__).parent.parent # https://github.com/simonw/datasette/issues/283#issuecomment-781591015 @@ -1274,6 +1267,16 @@ class DatasetteRouter: return except NotFound as exception: return await self.handle_404(request, send, exception) + except Forbidden as exception: + # Try the forbidden() plugin hook + for custom_response in pm.hook.forbidden( + datasette=self.ds, request=request, message=exception.args[0] + ): + custom_response = await await_me_maybe(custom_response) + assert ( + custom_response + ), "Default forbidden() hook should have been called" + return await custom_response.asgi_send(send) except Exception as exception: return await self.handle_exception(request, send, exception) @@ -1372,72 +1375,20 @@ class DatasetteRouter: await self.handle_exception(request, send, exception or NotFound("404")) async def handle_exception(self, request, send, exception): - if self.ds.pdb: - import pdb + responses = [] + for hook in pm.hook.handle_exception( + datasette=self.ds, + request=request, + exception=exception, + ): + response = await await_me_maybe(hook) + if response is not None: + responses.append(response) - pdb.post_mortem(exception.__traceback__) - - if rich is not None: - rich.get_console().print_exception(show_locals=True) - - title = None - if isinstance(exception, Forbidden): - status = 403 - info = {} - message = exception.args[0] - # Try the forbidden() plugin hook - for custom_response in pm.hook.forbidden( - datasette=self.ds, request=request, message=message - ): - custom_response = await await_me_maybe(custom_response) - if custom_response is not None: - await custom_response.asgi_send(send) - return - elif isinstance(exception, Base400): - status = exception.status - info = {} - message = exception.args[0] - elif isinstance(exception, DatasetteError): - status = exception.status - info = exception.error_dict - message = exception.message - if exception.message_is_html: - message = Markup(message) - title = exception.title - else: - status = 500 - info = {} - message = str(exception) - traceback.print_exc() - templates = [f"{status}.html", "error.html"] - info.update( - { - "ok": False, - "error": message, - "status": status, - "title": title, - } - ) - headers = {} - if self.ds.cors: - add_cors_headers(headers) - if request.path.split("?")[0].endswith(".json"): - await asgi_send_json(send, info, status=status, headers=headers) - else: - template = self.ds.jinja_env.select_template(templates) - await asgi_send_html( - send, - await template.render_async( - dict( - info, - urls=self.ds.urls, - app_css_hash=self.ds.app_css_hash(), - menu_links=lambda: [], - ) - ), - status=status, - headers=headers, - ) + assert responses, "Default exception handler should have returned something" + # Even if there are multiple responses use just the first one + response = responses[0] + await response.asgi_send(send) _cleaner_task_str_re = re.compile(r"\S*site-packages/") diff --git a/datasette/forbidden.py b/datasette/forbidden.py new file mode 100644 index 00000000..156a44d4 --- /dev/null +++ b/datasette/forbidden.py @@ -0,0 +1,20 @@ +from os import stat +from datasette import hookimpl, Response + + +@hookimpl(trylast=True) +def forbidden(datasette, request, message): + async def inner(): + return Response.html( + await datasette.render_template( + "error.html", + { + "title": "Forbidden", + "error": message, + }, + request=request, + ), + status=403, + ) + + return inner diff --git a/datasette/handle_exception.py b/datasette/handle_exception.py new file mode 100644 index 00000000..8b7e83e3 --- /dev/null +++ b/datasette/handle_exception.py @@ -0,0 +1,74 @@ +from datasette import hookimpl, Response +from .utils import await_me_maybe, add_cors_headers +from .utils.asgi import ( + Base400, + Forbidden, +) +from .views.base import DatasetteError +from markupsafe import Markup +import pdb +import traceback +from .plugins import pm + +try: + import rich +except ImportError: + rich = None + + +@hookimpl(trylast=True) +def handle_exception(datasette, request, exception): + async def inner(): + if datasette.pdb: + pdb.post_mortem(exception.__traceback__) + + if rich is not None: + rich.get_console().print_exception(show_locals=True) + + title = None + if isinstance(exception, Base400): + status = exception.status + info = {} + message = exception.args[0] + elif isinstance(exception, DatasetteError): + status = exception.status + info = exception.error_dict + message = exception.message + if exception.message_is_html: + message = Markup(message) + title = exception.title + else: + status = 500 + info = {} + message = str(exception) + traceback.print_exc() + templates = [f"{status}.html", "error.html"] + info.update( + { + "ok": False, + "error": message, + "status": status, + "title": title, + } + ) + headers = {} + if datasette.cors: + add_cors_headers(headers) + if request.path.split("?")[0].endswith(".json"): + return Response.json(info, status=status, headers=headers) + else: + template = datasette.jinja_env.select_template(templates) + return Response.html( + await template.render_async( + dict( + info, + urls=datasette.urls, + app_css_hash=datasette.app_css_hash(), + menu_links=lambda: [], + ) + ), + status=status, + headers=headers, + ) + + return inner diff --git a/datasette/hookspecs.py b/datasette/hookspecs.py index c84db0a3..a5fb536f 100644 --- a/datasette/hookspecs.py +++ b/datasette/hookspecs.py @@ -138,3 +138,8 @@ def database_actions(datasette, actor, database, request): @hookspec def skip_csrf(datasette, scope): """Mechanism for skipping CSRF checks for certain requests""" + + +@hookspec +def handle_exception(datasette, request, exception): + """Handle an uncaught exception. Can return a Response or None.""" diff --git a/datasette/plugins.py b/datasette/plugins.py index 76b46a47..fef0c8e9 100644 --- a/datasette/plugins.py +++ b/datasette/plugins.py @@ -15,6 +15,8 @@ DEFAULT_PLUGINS = ( "datasette.default_magic_parameters", "datasette.blob_renderer", "datasette.default_menu_links", + "datasette.handle_exception", + "datasette.forbidden", ) pm = pluggy.PluginManager("datasette") diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index f5c3ee83..6020a941 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -107,8 +107,8 @@ Extra template variables that should be made available in the rendered template ``view_name`` - string The name of the view being displayed. (``index``, ``database``, ``table``, and ``row`` are the most important ones.) -``request`` - object or None - The current HTTP :ref:`internals_request`. This can be ``None`` if the request object is not available. +``request`` - :ref:`internals_request` or None + The current HTTP request. This can be ``None`` if the request object is not available. ``datasette`` - :ref:`internals_datasette` You can use this to access plugin configuration options via ``datasette.plugin_config(your_plugin_name)`` @@ -504,7 +504,7 @@ When a request is received, the ``"render"`` callback function is called with ze The table or view, if one is being rendered. ``request`` - :ref:`internals_request` - The incoming HTTP request. + The current HTTP request. ``view_name`` - string The name of the current view being called. ``index``, ``database``, ``table``, and ``row`` are the most important ones. @@ -599,8 +599,8 @@ The optional view function arguments are as follows: ``datasette`` - :ref:`internals_datasette` You can use this to access plugin configuration options via ``datasette.plugin_config(your_plugin_name)``, or to execute SQL queries. -``request`` - Request object - The current HTTP :ref:`internals_request`. +``request`` - :ref:`internals_request` + The current HTTP request. ``scope`` - dictionary The incoming ASGI scope dictionary. @@ -947,8 +947,8 @@ actor_from_request(datasette, request) ``datasette`` - :ref:`internals_datasette` You can use this to access plugin configuration options via ``datasette.plugin_config(your_plugin_name)``, or to execute SQL queries. -``request`` - object - The current HTTP :ref:`internals_request`. +``request`` - :ref:`internals_request` + The current HTTP request. This is part of Datasette's :ref:`authentication and permissions system `. The function should attempt to authenticate an actor (either a user or an API actor of some sort) based on information in the request. @@ -1010,8 +1010,8 @@ Example: `datasette-auth-tokens `__ and then renders a custom error page: + +.. code-block:: python + + from datasette import hookimpl, Response + import sentry_sdk + + + @hookimpl + def handle_exception(datasette, exception): + sentry_sdk.capture_exception(exception) + async def inner(): + return Response.html( + await datasette.render_template("custom_error.html", request=request) + ) + return inner + .. _plugin_hook_menu_links: menu_links(datasette, actor, request) @@ -1232,8 +1266,8 @@ menu_links(datasette, actor, request) ``actor`` - dictionary or None The currently authenticated :ref:`actor `. -``request`` - object or None - The current HTTP :ref:`internals_request`. This can be ``None`` if the request object is not available. +``request`` - :ref:`internals_request` + The current HTTP request. This can be ``None`` if the request object is not available. This hook allows additional items to be included in the menu displayed by Datasette's top right menu icon. @@ -1281,8 +1315,8 @@ table_actions(datasette, actor, database, table, request) ``table`` - string The name of the table. -``request`` - object - The current HTTP :ref:`internals_request`. This can be ``None`` if the request object is not available. +``request`` - :ref:`internals_request` + The current HTTP request. This can be ``None`` if the request object is not available. This hook allows table actions to be displayed in a menu accessed via an action icon at the top of the table page. It should return a list of ``{"href": "...", "label": "..."}`` menu items. @@ -1325,8 +1359,8 @@ database_actions(datasette, actor, database, request) ``database`` - string The name of the database. -``request`` - object - The current HTTP :ref:`internals_request`. +``request`` - :ref:`internals_request` + The current HTTP request. This hook is similar to :ref:`plugin_hook_table_actions` but populates an actions menu on the database page. diff --git a/tests/fixtures.py b/tests/fixtures.py index e0e4ec7b..c145ac78 100644 --- a/tests/fixtures.py +++ b/tests/fixtures.py @@ -68,6 +68,7 @@ EXPECTED_PLUGINS = [ "canned_queries", "extra_js_urls", "extra_template_vars", + "handle_exception", "menu_links", "permission_allowed", "register_routes", diff --git a/tests/plugins/my_plugin_2.py b/tests/plugins/my_plugin_2.py index f5ce36b3..4df02343 100644 --- a/tests/plugins/my_plugin_2.py +++ b/tests/plugins/my_plugin_2.py @@ -185,3 +185,21 @@ def register_routes(datasette): # Also serves to demonstrate over-ride of default paths: (r"/(?P[^/]+)/(?P[^/]+?$)", new_table), ] + + +@hookimpl +def handle_exception(datasette, request, exception): + datasette._exception_hook_fired = (request, exception) + if request.args.get("_custom_error"): + return Response.text("_custom_error") + elif request.args.get("_custom_error_async"): + + async def inner(): + return Response.text("_custom_error_async") + + return inner + + +@hookimpl(specname="register_routes") +def register_triger_error(): + return ((r"/trigger-error", lambda: 1 / 0),) diff --git a/tests/test_permissions.py b/tests/test_permissions.py index f4169dbe..2a519e76 100644 --- a/tests/test_permissions.py +++ b/tests/test_permissions.py @@ -332,6 +332,7 @@ def test_permissions_debug(app_client): assert checks == [ {"action": "permissions-debug", "result": True, "used_default": False}, {"action": "view-instance", "result": None, "used_default": True}, + {"action": "debug-menu", "result": False, "used_default": True}, {"action": "permissions-debug", "result": False, "used_default": True}, {"action": "view-instance", "result": None, "used_default": True}, ] diff --git a/tests/test_plugins.py b/tests/test_plugins.py index 4a7ad7c6..948a40b8 100644 --- a/tests/test_plugins.py +++ b/tests/test_plugins.py @@ -824,6 +824,20 @@ def test_hook_forbidden(restore_working_directory): assert "view-database" == client.ds._last_forbidden_message +def test_hook_handle_exception(app_client): + app_client.get("/trigger-error?x=123") + assert hasattr(app_client.ds, "_exception_hook_fired") + request, exception = app_client.ds._exception_hook_fired + assert request.url == "http://localhost/trigger-error?x=123" + assert isinstance(exception, ZeroDivisionError) + + +@pytest.mark.parametrize("param", ("_custom_error", "_custom_error_async")) +def test_hook_handle_exception_custom_response(app_client, param): + response = app_client.get("/trigger-error?{}=1".format(param)) + assert response.text == param + + def test_hook_menu_links(app_client): def get_menu_links(html): soup = Soup(html, "html.parser") From 58fd1e33ec7ac5ed85431d5c86d60600cd5280fb Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 17 Jul 2022 16:30:58 -0700 Subject: [PATCH 097/952] Hint that you can render templates for these hooks, refs #1770 --- docs/plugin_hooks.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index 6020a941..b4869606 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -1176,7 +1176,7 @@ forbidden(datasette, request, message) -------------------------------------- ``datasette`` - :ref:`internals_datasette` - You can use this to access plugin configuration options via ``datasette.plugin_config(your_plugin_name)``, or to execute SQL queries. + You can use this to access plugin configuration options via ``datasette.plugin_config(your_plugin_name)``, or to render templates or execute SQL queries. ``request`` - :ref:`internals_request` The current HTTP request. @@ -1224,7 +1224,7 @@ handle_exception(datasette, request, exception) ----------------------------------------------- ``datasette`` - :ref:`internals_datasette` - You can use this to access plugin configuration options via ``datasette.plugin_config(your_plugin_name)``, or to execute SQL queries. + You can use this to access plugin configuration options via ``datasette.plugin_config(your_plugin_name)``, or to render templates or execute SQL queries. ``request`` - :ref:`internals_request` The current HTTP request. From e543a095cc4c1ca895b082cfd1263ca25203a7c0 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 17 Jul 2022 17:57:41 -0700 Subject: [PATCH 098/952] Updated default plugins in docs, refs #1770 --- docs/plugins.rst | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/docs/plugins.rst b/docs/plugins.rst index f2ed02f7..29078054 100644 --- a/docs/plugins.rst +++ b/docs/plugins.rst @@ -172,6 +172,24 @@ If you run ``datasette plugins --all`` it will include default plugins that ship "filters_from_request" ] }, + { + "name": "datasette.forbidden", + "static": false, + "templates": false, + "version": null, + "hooks": [ + "forbidden" + ] + }, + { + "name": "datasette.handle_exception", + "static": false, + "templates": false, + "version": null, + "hooks": [ + "handle_exception" + ] + }, { "name": "datasette.publish.cloudrun", "static": false, From 6d5e1955470424cf4faf5d35788d328ebdd6d463 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 17 Jul 2022 17:59:20 -0700 Subject: [PATCH 099/952] Release 0.62a1 Refs #1300, #1739, #1744, #1746, #1748, #1759, #1770 --- datasette/version.py | 2 +- docs/changelog.rst | 14 ++++++++++++++ 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/datasette/version.py b/datasette/version.py index cf18c441..86f4cf7e 100644 --- a/datasette/version.py +++ b/datasette/version.py @@ -1,2 +1,2 @@ -__version__ = "0.62a0" +__version__ = "0.62a1" __version_info__ = tuple(__version__.split(".")) diff --git a/docs/changelog.rst b/docs/changelog.rst index 74814fcb..3f105811 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -4,6 +4,20 @@ Changelog ========= +.. _v0_62a1: + +0.62a1 (2022-07-17) +------------------- + +- New plugin hook: :ref:`handle_exception() `, for custom handling of exceptions caught by Datasette. (:issue:`1770`) +- The :ref:`render_cell() ` plugin hook is now also passed a ``row`` argument, representing the ``sqlite3.Row`` object that is being rendered. (:issue:`1300`) +- New ``--nolock`` option for ignoring file locks when opening read-only databases. (:issue:`1744`) +- Documentation now uses the `Furo `__ Sphinx theme. (:issue:`1746`) +- Datasette now has a `Discord community `__. +- Database file downloads now implement conditional GET using ETags. (:issue:`1739`) +- Examples in the documentation now include a copy-to-clipboard button. (:issue:`1748`) +- HTML for facet results and suggested results has been extracted out into new templates ``_facet_results.html`` and ``_suggested_facets.html``. Thanks, M. Nasimul Haque. (`#1759 `__) + .. _v0_62a0: 0.62a0 (2022-05-02) From ed1ebc0f1d4153e3e0934f2af19f82e5fdf137d3 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 17 Jul 2022 18:03:33 -0700 Subject: [PATCH 100/952] Run blacken-docs, refs #1770 --- docs/plugin_hooks.rst | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index b4869606..aec1df56 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -1213,7 +1213,9 @@ The function can alternatively return an awaitable function if it needs to make def forbidden(datasette): async def inner(): return Response.html( - await datasette.render_template("render_message.html", request=request) + await datasette.render_template( + "render_message.html", request=request + ) ) return inner @@ -1249,10 +1251,14 @@ This example logs an error to `Sentry `__ and then renders a @hookimpl def handle_exception(datasette, exception): sentry_sdk.capture_exception(exception) + async def inner(): return Response.html( - await datasette.render_template("custom_error.html", request=request) + await datasette.render_template( + "custom_error.html", request=request + ) ) + return inner .. _plugin_hook_menu_links: From ea6161f8475d9fa41c4879049511c58f692cce04 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sun, 17 Jul 2022 18:06:26 -0700 Subject: [PATCH 101/952] Bump furo from 2022.4.7 to 2022.6.21 (#1760) Bumps [furo](https://github.com/pradyunsg/furo) from 2022.4.7 to 2022.6.21. - [Release notes](https://github.com/pradyunsg/furo/releases) - [Changelog](https://github.com/pradyunsg/furo/blob/main/docs/changelog.md) - [Commits](https://github.com/pradyunsg/furo/compare/2022.04.07...2022.06.21) --- updated-dependencies: - dependency-name: furo dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 29cb77bf..558b5c87 100644 --- a/setup.py +++ b/setup.py @@ -65,7 +65,7 @@ setup( setup_requires=["pytest-runner"], extras_require={ "docs": [ - "furo==2022.4.7", + "furo==2022.6.21", "sphinx-autobuild", "codespell", "blacken-docs", From 22354c48ce4d514d7a1b321e5651c7f1340e3f5e Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sun, 17 Jul 2022 18:06:37 -0700 Subject: [PATCH 102/952] Update pytest-asyncio requirement from <0.19,>=0.17 to >=0.17,<0.20 (#1769) Updates the requirements on [pytest-asyncio](https://github.com/pytest-dev/pytest-asyncio) to permit the latest version. - [Release notes](https://github.com/pytest-dev/pytest-asyncio/releases) - [Changelog](https://github.com/pytest-dev/pytest-asyncio/blob/master/CHANGELOG.rst) - [Commits](https://github.com/pytest-dev/pytest-asyncio/compare/v0.17.0...v0.19.0) --- updated-dependencies: - dependency-name: pytest-asyncio dependency-type: direct:development ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 558b5c87..a1c51d0b 100644 --- a/setup.py +++ b/setup.py @@ -74,7 +74,7 @@ setup( "test": [ "pytest>=5.2.2,<7.2.0", "pytest-xdist>=2.2.1,<2.6", - "pytest-asyncio>=0.17,<0.19", + "pytest-asyncio>=0.17,<0.20", "beautifulsoup4>=4.8.1,<4.12.0", "black==22.6.0", "blacken-docs==1.12.1", From 01369176b0a8943ab45292ffc6f9c929b80a00e8 Mon Sep 17 00:00:00 2001 From: Chris Amico Date: Sun, 17 Jul 2022 21:12:45 -0400 Subject: [PATCH 103/952] Keep track of datasette.config_dir (#1766) Thanks, @eyeseast - closes #1764 --- datasette/app.py | 1 + tests/test_config_dir.py | 9 +++++++++ 2 files changed, 10 insertions(+) diff --git a/datasette/app.py b/datasette/app.py index edd05bb3..1a9afc10 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -211,6 +211,7 @@ class Datasette: assert config_dir is None or isinstance( config_dir, Path ), "config_dir= should be a pathlib.Path" + self.config_dir = config_dir self.pdb = pdb self._secret = secret or secrets.token_hex(32) self.files = tuple(files or []) + tuple(immutables or []) diff --git a/tests/test_config_dir.py b/tests/test_config_dir.py index 015c6ace..fe927c42 100644 --- a/tests/test_config_dir.py +++ b/tests/test_config_dir.py @@ -1,4 +1,5 @@ import json +import pathlib import pytest from datasette.app import Datasette @@ -150,3 +151,11 @@ def test_metadata_yaml(tmp_path_factory, filename): response = client.get("/-/metadata.json") assert 200 == response.status assert {"title": "Title from metadata"} == response.json + + +def test_store_config_dir(config_dir_client): + ds = config_dir_client.ds + + assert hasattr(ds, "config_dir") + assert ds.config_dir is not None + assert isinstance(ds.config_dir, pathlib.Path) From 7af67b54b7d9bca43e948510fc62f6db2b748fa8 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 18 Jul 2022 14:31:09 -0700 Subject: [PATCH 104/952] How to register temporary plugins in tests, closes #903 --- docs/testing_plugins.rst | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/docs/testing_plugins.rst b/docs/testing_plugins.rst index 41046bfb..d02003a9 100644 --- a/docs/testing_plugins.rst +++ b/docs/testing_plugins.rst @@ -219,3 +219,39 @@ Here's a test for that plugin that mocks the HTTPX outbound request: assert ( outbound_request.url == "https://www.example.com/" ) + +.. _testing_plugins_register_in_test: + +Registering a plugin for the duration of a test +----------------------------------------------- + +When writing tests for plugins you may find it useful to register a test plugin just for the duration of a single test. You can do this using ``pm.register()`` and ``pm.unregister()`` like this: + +.. code-block:: python + + from datasette import hookimpl + from datasette.app import Datasette + from datasette.plugins import pm + import pytest + + + @pytest.mark.asyncio + async def test_using_test_plugin(): + class TestPlugin: + __name__ = "TestPlugin" + + # Use hookimpl and method names to register hooks + @hookimpl + def register_routes(self): + return [ + (r"^/error$", lambda: 1/0), + ] + + pm.register(TestPlugin(), name="undo") + try: + # The test implementation goes here + datasette = Datasette() + response = await datasette.client.get("/error") + assert response.status_code == 500 + finally: + pm.unregister(name="undo") From bca2d95d0228f80a108e13408f8e72b2c06c2c7b Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 2 Aug 2022 16:38:02 -0700 Subject: [PATCH 105/952] Configure readthedocs/readthedocs-preview --- .github/workflows/documentation-links.yml | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 .github/workflows/documentation-links.yml diff --git a/.github/workflows/documentation-links.yml b/.github/workflows/documentation-links.yml new file mode 100644 index 00000000..e7062a46 --- /dev/null +++ b/.github/workflows/documentation-links.yml @@ -0,0 +1,16 @@ +name: Read the Docs Pull Request Preview +on: + pull_request_target: + types: + - opened + +permissions: + pull-requests: write + +jobs: + documentation-links: + runs-on: ubuntu-latest + steps: + - uses: readthedocs/readthedocs-preview@main + with: + project-slug: "datasette" From 8cfc72336878dd846d149658e99cc598e835b661 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 9 Aug 2022 11:21:53 -0700 Subject: [PATCH 106/952] Ran blacken-docs --- docs/testing_plugins.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing_plugins.rst b/docs/testing_plugins.rst index d02003a9..992b4b0e 100644 --- a/docs/testing_plugins.rst +++ b/docs/testing_plugins.rst @@ -244,7 +244,7 @@ When writing tests for plugins you may find it useful to register a test plugin @hookimpl def register_routes(self): return [ - (r"^/error$", lambda: 1/0), + (r"^/error$", lambda: 1 / 0), ] pm.register(TestPlugin(), name="undo") From 05d9c682689a0f1d23cbb502e027364ab3363910 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 14 Aug 2022 08:16:53 -0700 Subject: [PATCH 107/952] Promote Discord more in the README --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 032180aa..7ebbca57 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,7 @@ Datasette is aimed at data journalists, museum curators, archivists, local gover * Comprehensive documentation: https://docs.datasette.io/ * Examples: https://datasette.io/examples * Live demo of current main: https://latest.datasette.io/ -* Support questions, feedback? Join our [GitHub Discussions forum](https://github.com/simonw/datasette/discussions) +* Questions, feedback or want to talk about the project? Join our [Discord](https://discord.gg/ktd74dm5mw) Want to stay up-to-date with the project? Subscribe to the [Datasette newsletter](https://datasette.substack.com/) for tips, tricks and news on what's new in the Datasette ecosystem. From db00c00f6397287749331e8042fe998ee7f3b919 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 14 Aug 2022 08:19:30 -0700 Subject: [PATCH 108/952] Promote Datasette Lite in the README, refs #1781 --- README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 7ebbca57..1af20129 100644 --- a/README.md +++ b/README.md @@ -21,7 +21,7 @@ Datasette is aimed at data journalists, museum curators, archivists, local gover * Latest [Datasette News](https://datasette.io/news) * Comprehensive documentation: https://docs.datasette.io/ * Examples: https://datasette.io/examples -* Live demo of current main: https://latest.datasette.io/ +* Live demo of current `main` branch: https://latest.datasette.io/ * Questions, feedback or want to talk about the project? Join our [Discord](https://discord.gg/ktd74dm5mw) Want to stay up-to-date with the project? Subscribe to the [Datasette newsletter](https://datasette.substack.com/) for tips, tricks and news on what's new in the Datasette ecosystem. @@ -85,3 +85,7 @@ Or: This will create a docker image containing both the datasette application and the specified SQLite database files. It will then deploy that image to Heroku or Cloud Run and give you a URL to access the resulting website and API. See [Publishing data](https://docs.datasette.io/en/stable/publish.html) in the documentation for more details. + +## Datasette Lite + +[Datasette Lite](https://lite.datasette.io/) is Datasette packaged using WebAssembly so that it runs entirely in your browser, no Python web application server required. Read more about that in the [Datasette Lite documentation](https://github.com/simonw/datasette-lite/blob/main/README.md). From 8eb699de7becdefc6d72555d9fb17c9f06235dc4 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 14 Aug 2022 08:24:39 -0700 Subject: [PATCH 109/952] Datasette Lite in Getting Started docs, closes #1781 --- docs/getting_started.rst | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/getting_started.rst b/docs/getting_started.rst index 00b753a9..571540cf 100644 --- a/docs/getting_started.rst +++ b/docs/getting_started.rst @@ -21,6 +21,17 @@ Datasette has several `tutorials `__ to help you - `Exploring a database with Datasette `__ shows how to use the Datasette web interface to explore a new database. - `Learn SQL with Datasette `__ introduces SQL, and shows how to use that query language to ask questions of your data. +.. _getting_started_datasette_lite: + +Datasette in your browser with Datasette Lite +--------------------------------------------- + +`Datasette Lite `__ is Datasette packaged using WebAssembly so that it runs entirely in your browser, no Python web application server required. + +You can pass a URL to a CSV, SQLite or raw SQL file directly to Datasette Lite to explore that data in your browser. + +This `example link `__ opens Datasette Lite and loads the SQL Murder Mystery example database from `Northwestern University Knight Lab `__. + .. _getting_started_glitch: Try Datasette without installing anything using Glitch From df4fd2d7ddca8956d8a51c72ce007b8c75227f32 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 14 Aug 2022 08:44:02 -0700 Subject: [PATCH 110/952] _sort= works even if sort column not selected, closes #1773 --- datasette/views/table.py | 22 +++++++++++++++++++++- tests/test_table_api.py | 2 ++ 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/datasette/views/table.py b/datasette/views/table.py index cd4be823..94d2673b 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -630,7 +630,27 @@ class TableView(DataView): next_value = path_from_row_pks(rows[-2], pks, use_rowid) # If there's a sort or sort_desc, add that value as a prefix if (sort or sort_desc) and not is_view: - prefix = rows[-2][sort or sort_desc] + try: + prefix = rows[-2][sort or sort_desc] + except IndexError: + # sort/sort_desc column missing from SELECT - look up value by PK instead + prefix_where_clause = " and ".join( + "[{}] = :pk{}".format(pk, i) for i, pk in enumerate(pks) + ) + prefix_lookup_sql = "select [{}] from [{}] where {}".format( + sort or sort_desc, table_name, prefix_where_clause + ) + prefix = ( + await db.execute( + prefix_lookup_sql, + { + **{ + "pk{}".format(i): rows[-2][pk] + for i, pk in enumerate(pks) + } + }, + ) + ).single_value() if isinstance(prefix, dict) and "value" in prefix: prefix = prefix["value"] if prefix is None: diff --git a/tests/test_table_api.py b/tests/test_table_api.py index 9db383c3..e56a72b5 100644 --- a/tests/test_table_api.py +++ b/tests/test_table_api.py @@ -288,6 +288,8 @@ def test_paginate_compound_keys_with_extra_filters(app_client): ), # text column contains '$null' - ensure it doesn't confuse pagination: ("_sort=text", lambda row: row["text"], "sorted by text"), + # Still works if sort column removed using _col= + ("_sort=text&_col=content", lambda row: row["text"], "sorted by text"), ], ) def test_sortable(app_client, query_string, sort_key, human_description_en): From 668415df9f6334bd255c22ab02018bed5bc14edd Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 14 Aug 2022 08:47:17 -0700 Subject: [PATCH 111/952] Upgrade Docker baes to 3.10.6-slim-bullseye - refs #1768 --- Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Dockerfile b/Dockerfile index 42f5529b..ee7ed957 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,4 +1,4 @@ -FROM python:3.9.7-slim-bullseye as build +FROM python:3.10.6-slim-bullseye as build # Version of Datasette to install, e.g. 0.55 # docker build . -t datasette --build-arg VERSION=0.55 From 080d4b3e065d78faf977c6ded6ead31aae24e2ae Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 14 Aug 2022 08:49:14 -0700 Subject: [PATCH 112/952] Switch to python:3.10.6-slim-bullseye for datasette publish - refs #1768 --- datasette/utils/__init__.py | 2 +- demos/apache-proxy/Dockerfile | 2 +- docs/publish.rst | 2 +- tests/test_package.py | 2 +- tests/test_publish_cloudrun.py | 4 ++-- 5 files changed, 6 insertions(+), 6 deletions(-) diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index 77768112..d148cc2c 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -390,7 +390,7 @@ def make_dockerfile( "SQLITE_EXTENSIONS" ] = "/usr/lib/x86_64-linux-gnu/mod_spatialite.so" return """ -FROM python:3.8 +FROM python:3.10.6-slim-bullseye COPY . /app WORKDIR /app {apt_get_extras} diff --git a/demos/apache-proxy/Dockerfile b/demos/apache-proxy/Dockerfile index 6c921963..70b33bec 100644 --- a/demos/apache-proxy/Dockerfile +++ b/demos/apache-proxy/Dockerfile @@ -1,4 +1,4 @@ -FROM python:3.9.7-slim-bullseye +FROM python:3.10.6-slim-bullseye RUN apt-get update && \ apt-get install -y apache2 supervisor && \ diff --git a/docs/publish.rst b/docs/publish.rst index 166f2883..9c7c99cc 100644 --- a/docs/publish.rst +++ b/docs/publish.rst @@ -144,7 +144,7 @@ Here's example output for the package command:: $ datasette package parlgov.db --extra-options="--setting sql_time_limit_ms 2500" Sending build context to Docker daemon 4.459MB - Step 1/7 : FROM python:3 + Step 1/7 : FROM python:3.10.6-slim-bullseye ---> 79e1dc9af1c1 Step 2/7 : COPY . /app ---> Using cache diff --git a/tests/test_package.py b/tests/test_package.py index 02ed1775..ac15e61e 100644 --- a/tests/test_package.py +++ b/tests/test_package.py @@ -12,7 +12,7 @@ class CaptureDockerfile: EXPECTED_DOCKERFILE = """ -FROM python:3.8 +FROM python:3.10.6-slim-bullseye COPY . /app WORKDIR /app diff --git a/tests/test_publish_cloudrun.py b/tests/test_publish_cloudrun.py index 3427f4f7..60079ab3 100644 --- a/tests/test_publish_cloudrun.py +++ b/tests/test_publish_cloudrun.py @@ -223,7 +223,7 @@ def test_publish_cloudrun_plugin_secrets( ) expected = textwrap.dedent( r""" - FROM python:3.8 + FROM python:3.10.6-slim-bullseye COPY . /app WORKDIR /app @@ -290,7 +290,7 @@ def test_publish_cloudrun_apt_get_install( ) expected = textwrap.dedent( r""" - FROM python:3.8 + FROM python:3.10.6-slim-bullseye COPY . /app WORKDIR /app From 1563c22a8c65e6cff5194aa07df54d0ab8d4eecb Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 14 Aug 2022 09:13:12 -0700 Subject: [PATCH 113/952] Don't duplicate _sort_desc, refs #1738 --- datasette/views/table.py | 2 +- tests/test_table_html.py | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/datasette/views/table.py b/datasette/views/table.py index 94d2673b..49c30c9c 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -710,7 +710,7 @@ class TableView(DataView): for key in request.args: if ( key.startswith("_") - and key not in ("_sort", "_search", "_next") + and key not in ("_sort", "_sort_desc", "_search", "_next") and "__" not in key ): for value in request.args.getlist(key): diff --git a/tests/test_table_html.py b/tests/test_table_html.py index d3cb3e17..f3808ea3 100644 --- a/tests/test_table_html.py +++ b/tests/test_table_html.py @@ -828,6 +828,7 @@ def test_other_hidden_form_fields(app_client, path, expected_hidden): [ ("/fixtures/searchable?_search=terry", []), ("/fixtures/searchable?_sort=text2", []), + ("/fixtures/searchable?_sort_desc=text2", []), ("/fixtures/searchable?_sort=text2&_where=1", [("_where", "1")]), ], ) From c1396bf86033a7bd99fa0c0431f585475391a11a Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 14 Aug 2022 09:34:31 -0700 Subject: [PATCH 114/952] Don't allow canned write queries on immutable DBs, closes #1728 --- datasette/templates/query.html | 6 ++++- datasette/views/database.py | 4 ++++ tests/test_canned_queries.py | 40 ++++++++++++++++++++++++++++++++++ 3 files changed, 49 insertions(+), 1 deletion(-) diff --git a/datasette/templates/query.html b/datasette/templates/query.html index 8c920527..cee779fc 100644 --- a/datasette/templates/query.html +++ b/datasette/templates/query.html @@ -28,6 +28,10 @@ {% block content %} +{% if canned_write and db_is_immutable %} +

This query cannot be executed because the database is immutable.

+{% endif %} +

{{ metadata.title or database }}{% if canned_query and not metadata.title %}: {{ canned_query }}{% endif %}{% if private %} 🔒{% endif %}

{% block description_source_license %}{% include "_description_source_license.html" %}{% endblock %} @@ -61,7 +65,7 @@

{% if not hide_sql %}{% endif %} {% if canned_write %}{% endif %} - + {{ show_hide_hidden }} {% if canned_query and edit_sql_url %}Edit SQL{% endif %}

diff --git a/datasette/views/database.py b/datasette/views/database.py index 42058752..77632b9d 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -273,6 +273,9 @@ class QueryView(DataView): # Execute query - as write or as read if write: if request.method == "POST": + # If database is immutable, return an error + if not db.is_mutable: + raise Forbidden("Database is immutable") body = await request.post_body() body = body.decode("utf-8").strip() if body.startswith("{") and body.endswith("}"): @@ -326,6 +329,7 @@ class QueryView(DataView): async def extra_template(): return { "request": request, + "db_is_immutable": not db.is_mutable, "path_with_added_args": path_with_added_args, "path_with_removed_args": path_with_removed_args, "named_parameter_values": named_parameter_values, diff --git a/tests/test_canned_queries.py b/tests/test_canned_queries.py index 5abffdcc..976aa0db 100644 --- a/tests/test_canned_queries.py +++ b/tests/test_canned_queries.py @@ -53,6 +53,26 @@ def canned_write_client(tmpdir): yield client +@pytest.fixture +def canned_write_immutable_client(): + with make_app_client( + is_immutable=True, + metadata={ + "databases": { + "fixtures": { + "queries": { + "add": { + "sql": "insert into sortable (text) values (:text)", + "write": True, + }, + } + } + } + }, + ) as client: + yield client + + def test_canned_query_with_named_parameter(app_client): response = app_client.get("/fixtures/neighborhood_search.json?text=town") assert [ @@ -373,3 +393,23 @@ def test_canned_write_custom_template(canned_write_client): response.headers["link"] == 'http://localhost/data/update_name.json; rel="alternate"; type="application/json+datasette"' ) + + +def test_canned_write_query_disabled_for_immutable_database( + canned_write_immutable_client, +): + response = canned_write_immutable_client.get("/fixtures/add") + assert response.status == 200 + assert ( + "This query cannot be executed because the database is immutable." + in response.text + ) + assert '' in response.text + # Submitting form should get a forbidden error + response = canned_write_immutable_client.post( + "/fixtures/add", + {"text": "text"}, + csrftoken_from=True, + ) + assert response.status == 403 + assert "Database is immutable" in response.text From 82167105ee699c850cc106ea927de1ad09276cfe Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 14 Aug 2022 10:07:30 -0700 Subject: [PATCH 115/952] --min-instances and --max-instances Cloud Run publish options, closes #1779 --- datasette/publish/cloudrun.py | 26 +++++++++++++++++--- docs/cli-reference.rst | 2 ++ tests/test_publish_cloudrun.py | 43 ++++++++++++++++++++++++---------- 3 files changed, 56 insertions(+), 15 deletions(-) diff --git a/datasette/publish/cloudrun.py b/datasette/publish/cloudrun.py index 50b2b2fd..77274eb0 100644 --- a/datasette/publish/cloudrun.py +++ b/datasette/publish/cloudrun.py @@ -52,6 +52,16 @@ def publish_subcommand(publish): multiple=True, help="Additional packages to apt-get install", ) + @click.option( + "--max-instances", + type=int, + help="Maximum Cloud Run instances", + ) + @click.option( + "--min-instances", + type=int, + help="Minimum Cloud Run instances", + ) def cloudrun( files, metadata, @@ -79,6 +89,8 @@ def publish_subcommand(publish): cpu, timeout, apt_get_extras, + max_instances, + min_instances, ): "Publish databases to Datasette running on Cloud Run" fail_if_publish_binary_not_installed( @@ -168,12 +180,20 @@ def publish_subcommand(publish): ), shell=True, ) + extra_deploy_options = [] + for option, value in ( + ("--memory", memory), + ("--cpu", cpu), + ("--max-instances", max_instances), + ("--min-instances", min_instances), + ): + if value: + extra_deploy_options.append("{} {}".format(option, value)) check_call( - "gcloud run deploy --allow-unauthenticated --platform=managed --image {} {}{}{}".format( + "gcloud run deploy --allow-unauthenticated --platform=managed --image {} {}{}".format( image_id, service, - " --memory {}".format(memory) if memory else "", - " --cpu {}".format(cpu) if cpu else "", + " " + " ".join(extra_deploy_options) if extra_deploy_options else "", ), shell=True, ) diff --git a/docs/cli-reference.rst b/docs/cli-reference.rst index 1c1aff15..415af13c 100644 --- a/docs/cli-reference.rst +++ b/docs/cli-reference.rst @@ -251,6 +251,8 @@ datasette publish cloudrun --help --cpu [1|2|4] Number of vCPUs to allocate in Cloud Run --timeout INTEGER Build timeout in seconds --apt-get-install TEXT Additional packages to apt-get install + --max-instances INTEGER Maximum Cloud Run instances + --min-instances INTEGER Minimum Cloud Run instances --help Show this message and exit. diff --git a/tests/test_publish_cloudrun.py b/tests/test_publish_cloudrun.py index 60079ab3..e64534d2 100644 --- a/tests/test_publish_cloudrun.py +++ b/tests/test_publish_cloudrun.py @@ -105,19 +105,36 @@ def test_publish_cloudrun(mock_call, mock_output, mock_which, tmp_path_factory): @mock.patch("datasette.publish.cloudrun.check_output") @mock.patch("datasette.publish.cloudrun.check_call") @pytest.mark.parametrize( - "memory,cpu,timeout,expected_gcloud_args", + "memory,cpu,timeout,min_instances,max_instances,expected_gcloud_args", [ - ["1Gi", None, None, "--memory 1Gi"], - ["2G", None, None, "--memory 2G"], - ["256Mi", None, None, "--memory 256Mi"], - ["4", None, None, None], - ["GB", None, None, None], - [None, 1, None, "--cpu 1"], - [None, 2, None, "--cpu 2"], - [None, 3, None, None], - [None, 4, None, "--cpu 4"], - ["2G", 4, None, "--memory 2G --cpu 4"], - [None, None, 1800, "--timeout 1800"], + ["1Gi", None, None, None, None, "--memory 1Gi"], + ["2G", None, None, None, None, "--memory 2G"], + ["256Mi", None, None, None, None, "--memory 256Mi"], + [ + "4", + None, + None, + None, + None, + None, + ], + [ + "GB", + None, + None, + None, + None, + None, + ], + [None, 1, None, None, None, "--cpu 1"], + [None, 2, None, None, None, "--cpu 2"], + [None, 3, None, None, None, None], + [None, 4, None, None, None, "--cpu 4"], + ["2G", 4, None, None, None, "--memory 2G --cpu 4"], + [None, None, 1800, None, None, "--timeout 1800"], + [None, None, None, 2, None, "--min-instances 2"], + [None, None, None, 2, 4, "--min-instances 2 --max-instances 4"], + [None, 2, None, None, 4, "--cpu 2 --max-instances 4"], ], ) def test_publish_cloudrun_memory_cpu( @@ -127,6 +144,8 @@ def test_publish_cloudrun_memory_cpu( memory, cpu, timeout, + min_instances, + max_instances, expected_gcloud_args, tmp_path_factory, ): From 5e6c5c9e3191a80f17a91c5205d9d69efdebb73f Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 14 Aug 2022 10:18:47 -0700 Subject: [PATCH 116/952] Document datasette.config_dir, refs #1766 --- docs/internals.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/internals.rst b/docs/internals.rst index da135282..20797e98 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -260,6 +260,7 @@ Constructor parameters include: - ``files=[...]`` - a list of database files to open - ``immutables=[...]`` - a list of database files to open in immutable mode - ``metadata={...}`` - a dictionary of :ref:`metadata` +- ``config_dir=...`` - the :ref:`configuration directory ` to use, stored in ``datasette.config_dir`` .. _datasette_databases: From 815162cf029fab9f1c9308c1d6ecdba7ee369ebe Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 14 Aug 2022 10:32:42 -0700 Subject: [PATCH 117/952] Release 0.62 Refs #903, #1300, #1683, #1701, #1712, #1717, #1718, #1728, #1733, #1738, #1739, #1744, #1746, #1748, #1759, #1766, #1768, #1770, #1773, #1779 Closes #1782 --- datasette/version.py | 2 +- docs/changelog.rst | 53 ++++++++++++++++++++++++++++++-------------- 2 files changed, 37 insertions(+), 18 deletions(-) diff --git a/datasette/version.py b/datasette/version.py index 86f4cf7e..0453346c 100644 --- a/datasette/version.py +++ b/datasette/version.py @@ -1,2 +1,2 @@ -__version__ = "0.62a1" +__version__ = "0.62" __version_info__ = tuple(__version__.split(".")) diff --git a/docs/changelog.rst b/docs/changelog.rst index 3f105811..1225c63f 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -4,33 +4,52 @@ Changelog ========= -.. _v0_62a1: +.. _v0_62: -0.62a1 (2022-07-17) +0.62 (2022-08-14) ------------------- +Datasette can now run entirely in your browser using WebAssembly. Try out `Datasette Lite `__, take a look `at the code `__ or read more about it in `Datasette Lite: a server-side Python web application running in a browser `__. + +Datasette now has a `Discord community `__ for questions and discussions about Datasette and its ecosystem of projects. + +Features +~~~~~~~~ + +- Datasette is now compatible with `Pyodide `__. This is the enabling technology behind `Datasette Lite `__. (:issue:`1733`) +- Database file downloads now implement conditional GET using ETags. (:issue:`1739`) +- HTML for facet results and suggested results has been extracted out into new templates ``_facet_results.html`` and ``_suggested_facets.html``. Thanks, M. Nasimul Haque. (`#1759 `__) +- Datasette now runs some SQL queries in parallel. This has limited impact on performance, see `this research issue `__ for details. +- New ``--nolock`` option for ignoring file locks when opening read-only databases. (:issue:`1744`) +- Spaces in the database names in URLs are now encoded as ``+`` rather than ``~20``. (:issue:`1701`) +- ```` is now displayed as ```` and is accompanied by tooltip showing "2.3MB". (:issue:`1712`) +- The base Docker image used by ``datasette publish cloudrun``, ``datasette package`` and the `official Datasette image `__ has been upgraded to ``3.10.6-slim-bullseye``. (:issue:`1768`) +- Canned writable queries against immutable databases now show a warning message. (:issue:`1728`) +- ``datasette publish cloudrun`` has a new ``--timeout`` option which can be used to increase the time limit applied by the Google Cloud build environment. Thanks, Tim Sherratt. (`#1717 `__) +- ``datasette publish cloudrun`` has new ``--min-instances`` and ``--max-instances`` options. (:issue:`1779`) + +Plugin hooks +~~~~~~~~~~~~ + - New plugin hook: :ref:`handle_exception() `, for custom handling of exceptions caught by Datasette. (:issue:`1770`) - The :ref:`render_cell() ` plugin hook is now also passed a ``row`` argument, representing the ``sqlite3.Row`` object that is being rendered. (:issue:`1300`) -- New ``--nolock`` option for ignoring file locks when opening read-only databases. (:issue:`1744`) -- Documentation now uses the `Furo `__ Sphinx theme. (:issue:`1746`) -- Datasette now has a `Discord community `__. -- Database file downloads now implement conditional GET using ETags. (:issue:`1739`) -- Examples in the documentation now include a copy-to-clipboard button. (:issue:`1748`) -- HTML for facet results and suggested results has been extracted out into new templates ``_facet_results.html`` and ``_suggested_facets.html``. Thanks, M. Nasimul Haque. (`#1759 `__) +- The :ref:`configuration directory ` is now stored in ``datasette.config_dir``, making it available to plugins. Thanks, Chris Amico. (`#1766 `__) -.. _v0_62a0: +Bug fixes +~~~~~~~~~ -0.62a0 (2022-05-02) -------------------- - -- Datasette now runs some SQL queries in parallel. This has limited impact on performance, see `this research issue `__ for details. -- Datasette should now be compatible with Pyodide. (:issue:`1733`) -- ``datasette publish cloudrun`` has a new ``--timeout`` option which can be used to increase the time limit applied by the Google Cloud build environment. Thanks, Tim Sherratt. (`#1717 `__) -- Spaces in database names are now encoded as ``+`` rather than ``~20``. (:issue:`1701`) -- ```` is now displayed as ```` and is accompanied by tooltip showing "2.3MB". (:issue:`1712`) - Don't show the facet option in the cog menu if faceting is not allowed. (:issue:`1683`) +- ``?_sort`` and ``?_sort_desc`` now work if the column that is being sorted has been excluded from the query using ``?_col=`` or ``?_nocol=``. (:issue:`1773`) +- Fixed bug where ``?_sort_desc`` was duplicated in the URL every time the Apply button was clicked. (:issue:`1738`) + +Documentation +~~~~~~~~~~~~~ + +- Examples in the documentation now include a copy-to-clipboard button. (:issue:`1748`) +- Documentation now uses the `Furo `__ Sphinx theme. (:issue:`1746`) - Code examples in the documentation are now all formatted using Black. (:issue:`1718`) - ``Request.fake()`` method is now documented, see :ref:`internals_request`. +- New documentation for plugin authors: :ref:`testing_plugins_register_in_test`. (:issue:`903`) .. _v0_61_1: From a107e3a028923c1ab3911c0f880011283f93f368 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 14 Aug 2022 16:07:46 -0700 Subject: [PATCH 118/952] datasette-sentry is an example of handle_exception --- docs/plugin_hooks.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index aec1df56..c6f35d06 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -1261,6 +1261,8 @@ This example logs an error to `Sentry `__ and then renders a return inner +Example: `datasette-sentry `_ + .. _plugin_hook_menu_links: menu_links(datasette, actor, request) From 481eb96d85291cdfa5767a83884a1525dfc382d8 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 15 Aug 2022 13:17:28 -0700 Subject: [PATCH 119/952] https://datasette.io/tutorials/clean-data tutorial Refs #1783 --- docs/getting_started.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/getting_started.rst b/docs/getting_started.rst index 571540cf..a9eaa404 100644 --- a/docs/getting_started.rst +++ b/docs/getting_started.rst @@ -20,6 +20,7 @@ Datasette has several `tutorials `__ to help you - `Exploring a database with Datasette `__ shows how to use the Datasette web interface to explore a new database. - `Learn SQL with Datasette `__ introduces SQL, and shows how to use that query language to ask questions of your data. +- `Cleaning data with sqlite-utils and Datasette `__ guides you through using `sqlite-utils `__ to turn a CSV file into a database that you can explore using Datasette. .. _getting_started_datasette_lite: From a3e6f1b16757fb2d39e7ddba4e09eda2362508bf Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 18 Aug 2022 09:06:02 -0700 Subject: [PATCH 120/952] Increase height of non-JS textarea to fit query Closes #1786 --- datasette/templates/query.html | 3 ++- tests/test_html.py | 6 ++---- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/datasette/templates/query.html b/datasette/templates/query.html index cee779fc..a35e3afe 100644 --- a/datasette/templates/query.html +++ b/datasette/templates/query.html @@ -45,7 +45,8 @@ {% endif %} {% if not hide_sql %} {% if editable and allow_execute_sql %} -

+

{% else %} 
{% if query %}{{ query.sql }}{% endif %}
{% endif %} diff --git a/tests/test_html.py b/tests/test_html.py index 409fec68..be21bd84 100644 --- a/tests/test_html.py +++ b/tests/test_html.py @@ -695,10 +695,8 @@ def test_query_error(app_client): response = app_client.get("/fixtures?sql=select+*+from+notatable") html = response.text assert '

no such table: notatable

' in html - assert ( - '' - in html - ) + assert '" in html assert "0 results" not in html From 09a41662e70b788469157bb58ed9ca4acdf2f904 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 18 Aug 2022 09:10:48 -0700 Subject: [PATCH 121/952] Fix typo --- docs/plugin_hooks.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index c6f35d06..30bd75b7 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -874,7 +874,7 @@ canned_queries(datasette, database, actor) ``actor`` - dictionary or None The currently authenticated :ref:`actor `. -Ues this hook to return a dictionary of additional :ref:`canned query ` definitions for the specified database. The return value should be the same shape as the JSON described in the :ref:`canned query ` documentation. +Use this hook to return a dictionary of additional :ref:`canned query ` definitions for the specified database. The return value should be the same shape as the JSON described in the :ref:`canned query ` documentation. .. code-block:: python From 6c0ba7c00c2ae3ecbb5309efa59079cea1c850b3 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 18 Aug 2022 14:52:04 -0700 Subject: [PATCH 122/952] Improved CLI reference documentation, refs #1787 --- datasette/cli.py | 2 +- docs/changelog.rst | 2 +- docs/cli-reference.rst | 325 ++++++++++++++++++++++++++++++--------- docs/getting_started.rst | 50 ------ docs/index.rst | 2 +- docs/publish.rst | 2 + 6 files changed, 259 insertions(+), 124 deletions(-) diff --git a/datasette/cli.py b/datasette/cli.py index 8781747c..f2a03d53 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -282,7 +282,7 @@ def package( port, **extra_metadata, ): - """Package specified SQLite files into a new datasette Docker container""" + """Package SQLite files into a Datasette Docker container""" if not shutil.which("docker"): click.secho( ' The package command requires "docker" to be installed and configured ', diff --git a/docs/changelog.rst b/docs/changelog.rst index 1225c63f..f9dcc980 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -621,7 +621,7 @@ See also `Datasette 0.49: The annotated release notes `__ for conversations about the project that go beyond just bug reports and issues. - Datasette can now be installed on macOS using Homebrew! Run ``brew install simonw/datasette/datasette``. See :ref:`installation_homebrew`. (:issue:`335`) - Two new commands: ``datasette install name-of-plugin`` and ``datasette uninstall name-of-plugin``. These are equivalent to ``pip install`` and ``pip uninstall`` but automatically run in the same virtual environment as Datasette, so users don't have to figure out where that virtual environment is - useful for installations created using Homebrew or ``pipx``. See :ref:`plugins_installing`. (:issue:`925`) -- A new command-line option, ``datasette --get``, accepts a path to a URL within the Datasette instance. It will run that request through Datasette (without starting a web server) and print out the response. See :ref:`getting_started_datasette_get` for an example. (:issue:`926`) +- A new command-line option, ``datasette --get``, accepts a path to a URL within the Datasette instance. It will run that request through Datasette (without starting a web server) and print out the response. See :ref:`cli_datasette_get` for an example. (:issue:`926`) .. _v0_46: diff --git a/docs/cli-reference.rst b/docs/cli-reference.rst index 415af13c..a1e56774 100644 --- a/docs/cli-reference.rst +++ b/docs/cli-reference.rst @@ -4,44 +4,34 @@ CLI reference =============== -This page lists the ``--help`` for every ``datasette`` CLI command. +The ``datasette`` CLI tool provides a number of commands. + +Running ``datasette`` without specifying a command runs the default command, ``datasette serve``. See :ref:`cli_help_serve___help` for the full list of options for that command. .. [[[cog from datasette import cli from click.testing import CliRunner import textwrap - commands = [ - ["--help"], - ["serve", "--help"], - ["serve", "--help-settings"], - ["plugins", "--help"], - ["publish", "--help"], - ["publish", "cloudrun", "--help"], - ["publish", "heroku", "--help"], - ["package", "--help"], - ["inspect", "--help"], - ["install", "--help"], - ["uninstall", "--help"], - ] - cog.out("\n") - for command in commands: - title = "datasette " + " ".join(command) - ref = "_cli_help_" + ("_".join(command).replace("-", "_")) - cog.out(".. {}:\n\n".format(ref)) - cog.out(title + "\n") - cog.out(("=" * len(title)) + "\n\n") + def help(args): + title = "datasette " + " ".join(args) cog.out("::\n\n") - result = CliRunner().invoke(cli.cli, command) + result = CliRunner().invoke(cli.cli, args) output = result.output.replace("Usage: cli ", "Usage: datasette ") cog.out(textwrap.indent(output, ' ')) cog.out("\n\n") .. ]]] +.. [[[end]]] .. _cli_help___help: datasette --help ================ +Running ``datasette --help`` shows a list of all of the available commands. + +.. [[[cog + help(["--help"]) +.. ]]] :: Usage: datasette [OPTIONS] COMMAND [ARGS]... @@ -59,17 +49,34 @@ datasette --help serve* Serve up specified SQLite database files with a web UI inspect Generate JSON summary of provided database files install Install plugins and packages from PyPI into the same... - package Package specified SQLite files into a new datasette Docker... + package Package SQLite files into a Datasette Docker container plugins List currently installed plugins publish Publish specified SQLite database files to the internet along... uninstall Uninstall plugins and Python packages from the Datasette... +.. [[[end]]] + +Additional commands added by plugins that use the :ref:`plugin_hook_register_commands` hook will be listed here as well. + .. _cli_help_serve___help: -datasette serve --help -====================== +datasette serve +=============== +This command starts the Datasette web application running on your machine:: + + datasette serve mydatabase.db + +Or since this is the default command you can run this instead:: + + datasette mydatabase.db + +Once started you can access it at ``http://localhost:8001`` + +.. [[[cog + help(["serve", "--help"]) +.. ]]] :: Usage: datasette serve [OPTIONS] [FILES]... @@ -121,11 +128,75 @@ datasette serve --help --help Show this message and exit. +.. [[[end]]] + + +.. _cli_datasette_get: + +datasette --get +--------------- + +The ``--get`` option to ``datasette serve`` (or just ``datasette``) specifies the path to a page within Datasette and causes Datasette to output the content from that path without starting the web server. + +This means that all of Datasette's functionality can be accessed directly from the command-line. + +For example:: + + $ datasette --get '/-/versions.json' | jq . + { + "python": { + "version": "3.8.5", + "full": "3.8.5 (default, Jul 21 2020, 10:48:26) \n[Clang 11.0.3 (clang-1103.0.32.62)]" + }, + "datasette": { + "version": "0.46+15.g222a84a.dirty" + }, + "asgi": "3.0", + "uvicorn": "0.11.8", + "sqlite": { + "version": "3.32.3", + "fts_versions": [ + "FTS5", + "FTS4", + "FTS3" + ], + "extensions": { + "json1": null + }, + "compile_options": [ + "COMPILER=clang-11.0.3", + "ENABLE_COLUMN_METADATA", + "ENABLE_FTS3", + "ENABLE_FTS3_PARENTHESIS", + "ENABLE_FTS4", + "ENABLE_FTS5", + "ENABLE_GEOPOLY", + "ENABLE_JSON1", + "ENABLE_PREUPDATE_HOOK", + "ENABLE_RTREE", + "ENABLE_SESSION", + "MAX_VARIABLE_NUMBER=250000", + "THREADSAFE=1" + ] + } + } + +The exit code will be 0 if the request succeeds and 1 if the request produced an HTTP status code other than 200 - e.g. a 404 or 500 error. + +This lets you use ``datasette --get /`` to run tests against a Datasette application in a continuous integration environment such as GitHub Actions. + .. _cli_help_serve___help_settings: datasette serve --help-settings -=============================== +------------------------------- +This command outputs all of the available Datasette :ref:`settings `. + +These can be passed to ``datasette serve`` using ``datasette serve --setting name value``. + +.. [[[cog + help(["--help-settings"]) +.. ]]] :: Settings: @@ -170,11 +241,18 @@ datasette serve --help-settings +.. [[[end]]] + .. _cli_help_plugins___help: -datasette plugins --help -======================== +datasette plugins +================= +Output JSON showing all currently installed plugins, their versions, whether they include static files or templates and which :ref:`plugin_hooks` they use. + +.. [[[cog + help(["plugins", "--help"]) +.. ]]] :: Usage: datasette plugins [OPTIONS] @@ -187,11 +265,110 @@ datasette plugins --help --help Show this message and exit. +.. [[[end]]] + +Example output: + +.. code-block:: json + + [ + { + "name": "datasette-geojson", + "static": false, + "templates": false, + "version": "0.3.1", + "hooks": [ + "register_output_renderer" + ] + }, + { + "name": "datasette-geojson-map", + "static": true, + "templates": false, + "version": "0.4.0", + "hooks": [ + "extra_body_script", + "extra_css_urls", + "extra_js_urls" + ] + }, + { + "name": "datasette-leaflet", + "static": true, + "templates": false, + "version": "0.2.2", + "hooks": [ + "extra_body_script", + "extra_template_vars" + ] + } + ] + + +.. _cli_help_install___help: + +datasette install +================= + +Install new Datasette plugins. This command works like ``pip install`` but ensures that your plugins will be installed into the same environment as Datasette. + +This command:: + + datasette install datasette-cluster-map + +Would install the `datasette-cluster-map `__ plugin. + +.. [[[cog + help(["install", "--help"]) +.. ]]] +:: + + Usage: datasette install [OPTIONS] PACKAGES... + + Install plugins and packages from PyPI into the same environment as Datasette + + Options: + -U, --upgrade Upgrade packages to latest version + --help Show this message and exit. + + +.. [[[end]]] + +.. _cli_help_uninstall___help: + +datasette uninstall +=================== + +Uninstall one or more plugins. + +.. [[[cog + help(["uninstall", "--help"]) +.. ]]] +:: + + Usage: datasette uninstall [OPTIONS] PACKAGES... + + Uninstall plugins and Python packages from the Datasette environment + + Options: + -y, --yes Don't ask for confirmation + --help Show this message and exit. + + +.. [[[end]]] + .. _cli_help_publish___help: -datasette publish --help -======================== +datasette publish +================= +Shows a list of available deployment targets for :ref:`publishing data ` with Datasette. + +Additional deployment targets can be added by plugins that use the :ref:`plugin_hook_publish_subcommand` hook. + +.. [[[cog + help(["publish", "--help"]) +.. ]]] :: Usage: datasette publish [OPTIONS] COMMAND [ARGS]... @@ -207,11 +384,19 @@ datasette publish --help heroku Publish databases to Datasette running on Heroku +.. [[[end]]] + + .. _cli_help_publish_cloudrun___help: -datasette publish cloudrun --help -================================= +datasette publish cloudrun +========================== +See :ref:`publish_cloud_run`. + +.. [[[cog + help(["publish", "cloudrun", "--help"]) +.. ]]] :: Usage: datasette publish cloudrun [OPTIONS] [FILES]... @@ -256,11 +441,19 @@ datasette publish cloudrun --help --help Show this message and exit. +.. [[[end]]] + + .. _cli_help_publish_heroku___help: -datasette publish heroku --help -=============================== +datasette publish heroku +======================== +See :ref:`publish_heroku`. + +.. [[[cog + help(["publish", "heroku", "--help"]) +.. ]]] :: Usage: datasette publish heroku [OPTIONS] [FILES]... @@ -297,16 +490,23 @@ datasette publish heroku --help --help Show this message and exit. +.. [[[end]]] + .. _cli_help_package___help: -datasette package --help -======================== +datasette package +================= +Package SQLite files into a Datasette Docker container, see :ref:`cli_package`. + +.. [[[cog + help(["package", "--help"]) +.. ]]] :: Usage: datasette package [OPTIONS] FILES... - Package specified SQLite files into a new datasette Docker container + Package SQLite files into a Datasette Docker container Options: -t, --tag TEXT Name for the resulting Docker container, can @@ -335,11 +535,26 @@ datasette package --help --help Show this message and exit. +.. [[[end]]] + + .. _cli_help_inspect___help: -datasette inspect --help -======================== +datasette inspect +================= +Outputs JSON representing introspected data about one or more SQLite database files. + +If you are opening an immutable database, you can pass this file to the ``--inspect-data`` option to improve Datasette's performance by allowing it to skip running row counts against the database when it first starts running:: + + datasette inspect mydatabase.db > inspect-data.json + datasette serve -i mydatabase.db --inspect-file inspect-data.json + +This performance optimization is used automatically by some of the ``datasette publish`` commands. You are unlikely to need to apply this optimization manually. + +.. [[[cog + help(["inspect", "--help"]) +.. ]]] :: Usage: datasette inspect [OPTIONS] [FILES]... @@ -355,36 +570,4 @@ datasette inspect --help --help Show this message and exit. -.. _cli_help_install___help: - -datasette install --help -======================== - -:: - - Usage: datasette install [OPTIONS] PACKAGES... - - Install plugins and packages from PyPI into the same environment as Datasette - - Options: - -U, --upgrade Upgrade packages to latest version - --help Show this message and exit. - - -.. _cli_help_uninstall___help: - -datasette uninstall --help -========================== - -:: - - Usage: datasette uninstall [OPTIONS] PACKAGES... - - Uninstall plugins and Python packages from the Datasette environment - - Options: - -y, --yes Don't ask for confirmation - --help Show this message and exit. - - .. [[[end]]] diff --git a/docs/getting_started.rst b/docs/getting_started.rst index a9eaa404..6515ef8d 100644 --- a/docs/getting_started.rst +++ b/docs/getting_started.rst @@ -138,53 +138,3 @@ JSON in a more convenient format: } ] } - -.. _getting_started_datasette_get: - -datasette --get ---------------- - -The ``--get`` option can specify the path to a page within Datasette and cause Datasette to output the content from that path without starting the web server. This means that all of Datasette's functionality can be accessed directly from the command-line. For example:: - - $ datasette --get '/-/versions.json' | jq . - { - "python": { - "version": "3.8.5", - "full": "3.8.5 (default, Jul 21 2020, 10:48:26) \n[Clang 11.0.3 (clang-1103.0.32.62)]" - }, - "datasette": { - "version": "0.46+15.g222a84a.dirty" - }, - "asgi": "3.0", - "uvicorn": "0.11.8", - "sqlite": { - "version": "3.32.3", - "fts_versions": [ - "FTS5", - "FTS4", - "FTS3" - ], - "extensions": { - "json1": null - }, - "compile_options": [ - "COMPILER=clang-11.0.3", - "ENABLE_COLUMN_METADATA", - "ENABLE_FTS3", - "ENABLE_FTS3_PARENTHESIS", - "ENABLE_FTS4", - "ENABLE_FTS5", - "ENABLE_GEOPOLY", - "ENABLE_JSON1", - "ENABLE_PREUPDATE_HOOK", - "ENABLE_RTREE", - "ENABLE_SESSION", - "MAX_VARIABLE_NUMBER=250000", - "THREADSAFE=1" - ] - } - } - -The exit code will be 0 if the request succeeds and 1 if the request produced an HTTP status code other than 200 - e.g. a 404 or 500 error. This means you can use ``datasette --get /`` to run tests against a Datasette application in a continuous integration environment such as GitHub Actions. - -Running ``datasette`` without specifying a command runs the default command, ``datasette serve``. See :ref:`cli_help_serve___help` for the full list of options for that command. diff --git a/docs/index.rst b/docs/index.rst index efe196b3..5a9cc7ed 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -40,6 +40,7 @@ Contents getting_started installation ecosystem + cli-reference pages publish deploying @@ -61,6 +62,5 @@ Contents plugin_hooks testing_plugins internals - cli-reference contributing changelog diff --git a/docs/publish.rst b/docs/publish.rst index 9c7c99cc..dd8566ed 100644 --- a/docs/publish.rst +++ b/docs/publish.rst @@ -56,6 +56,8 @@ Cloud Run provides a URL on the ``.run.app`` domain, but you can also point your See :ref:`cli_help_publish_cloudrun___help` for the full list of options for this command. +.. _publish_heroku: + Publishing to Heroku -------------------- From aff3df03d4fe0806ce432d1818f6643cdb2a854e Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 18 Aug 2022 14:55:08 -0700 Subject: [PATCH 123/952] Ignore ro which stands for read only Refs #1787 where it caused tests to break --- docs/codespell-ignore-words.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/codespell-ignore-words.txt b/docs/codespell-ignore-words.txt index a625cde5..d6744d05 100644 --- a/docs/codespell-ignore-words.txt +++ b/docs/codespell-ignore-words.txt @@ -1 +1 @@ -AddWordsToIgnoreHere +ro From 0d9d33955b503c88a2c712144d97f094baa5d46d Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 18 Aug 2022 16:06:12 -0700 Subject: [PATCH 124/952] Clarify you can publish multiple files, closes #1788 --- docs/publish.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/publish.rst b/docs/publish.rst index dd8566ed..d817ed31 100644 --- a/docs/publish.rst +++ b/docs/publish.rst @@ -31,7 +31,7 @@ Publishing to Google Cloud Run You will first need to install and configure the Google Cloud CLI tools by following `these instructions `__. -You can then publish a database to Google Cloud Run using the following command:: +You can then publish one or more SQLite database files to Google Cloud Run using the following command:: datasette publish cloudrun mydatabase.db --service=my-database @@ -63,7 +63,7 @@ Publishing to Heroku To publish your data using `Heroku `__, first create an account there and install and configure the `Heroku CLI tool `_. -You can publish a database to Heroku using the following command:: +You can publish one or more databases to Heroku using the following command:: datasette publish heroku mydatabase.db @@ -138,7 +138,7 @@ If a plugin has any :ref:`plugins_configuration_secret` you can use the ``--plug datasette package ================= -If you have docker installed (e.g. using `Docker for Mac `_) you can use the ``datasette package`` command to create a new Docker image in your local repository containing the datasette app bundled together with your selected SQLite databases:: +If you have docker installed (e.g. using `Docker for Mac `_) you can use the ``datasette package`` command to create a new Docker image in your local repository containing the datasette app bundled together with one or more SQLite databases:: datasette package mydatabase.db From 663ac431fe7202c85967568d82b2034f92b9aa43 Mon Sep 17 00:00:00 2001 From: Manuel Kaufmann Date: Sat, 20 Aug 2022 02:04:16 +0200 Subject: [PATCH 125/952] Use Read the Docs action v1 (#1778) Read the Docs repository was renamed from `readthedocs/readthedocs-preview` to `readthedocs/actions/`. Now, the `preview` action is under `readthedocs/actions/preview` and is tagged as `v1` --- .github/workflows/documentation-links.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/documentation-links.yml b/.github/workflows/documentation-links.yml index e7062a46..a54bd83a 100644 --- a/.github/workflows/documentation-links.yml +++ b/.github/workflows/documentation-links.yml @@ -11,6 +11,6 @@ jobs: documentation-links: runs-on: ubuntu-latest steps: - - uses: readthedocs/readthedocs-preview@main + - uses: readthedocs/actions/preview@v1 with: project-slug: "datasette" From 1d64c9a8dac45b9a3452acf8e76dfadea2b0bc49 Mon Sep 17 00:00:00 2001 From: Alex Garcia Date: Tue, 23 Aug 2022 11:34:30 -0700 Subject: [PATCH 126/952] Add new entrypoint option to --load-extensions. (#1789) Thanks, @asg017 --- .gitignore | 6 ++++ datasette/app.py | 8 ++++- datasette/cli.py | 4 ++- datasette/utils/__init__.py | 11 ++++++ tests/ext.c | 48 ++++++++++++++++++++++++++ tests/test_load_extensions.py | 65 +++++++++++++++++++++++++++++++++++ 6 files changed, 140 insertions(+), 2 deletions(-) create mode 100644 tests/ext.c create mode 100644 tests/test_load_extensions.py diff --git a/.gitignore b/.gitignore index 066009f0..277ff653 100644 --- a/.gitignore +++ b/.gitignore @@ -118,3 +118,9 @@ ENV/ .DS_Store node_modules .*.swp + +# In case someone compiled tests/ext.c for test_load_extensions, don't +# include it in source control. +tests/*.dylib +tests/*.so +tests/*.dll \ No newline at end of file diff --git a/datasette/app.py b/datasette/app.py index 1a9afc10..bb9232c9 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -559,7 +559,13 @@ class Datasette: if self.sqlite_extensions: conn.enable_load_extension(True) for extension in self.sqlite_extensions: - conn.execute("SELECT load_extension(?)", [extension]) + # "extension" is either a string path to the extension + # or a 2-item tuple that specifies which entrypoint to load. + if isinstance(extension, tuple): + path, entrypoint = extension + conn.execute("SELECT load_extension(?, ?)", [path, entrypoint]) + else: + conn.execute("SELECT load_extension(?)", [extension]) if self.setting("cache_size_kb"): conn.execute(f"PRAGMA cache_size=-{self.setting('cache_size_kb')}") # pylint: disable=no-member diff --git a/datasette/cli.py b/datasette/cli.py index f2a03d53..6eb42712 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -21,6 +21,7 @@ from .app import ( pm, ) from .utils import ( + LoadExtension, StartupError, check_connection, find_spatialite, @@ -128,9 +129,10 @@ def sqlite_extensions(fn): return click.option( "sqlite_extensions", "--load-extension", + type=LoadExtension(), envvar="SQLITE_EXTENSIONS", multiple=True, - help="Path to a SQLite extension to load", + help="Path to a SQLite extension to load, and optional entrypoint", )(fn) diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index d148cc2c..0fc87d51 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -833,6 +833,17 @@ class StaticMount(click.ParamType): self.fail(f"{value} is not a valid directory path", param, ctx) return path, dirpath +# The --load-extension parameter can optionally include a specific entrypoint. +# This is done by appending ":entrypoint_name" after supplying the path to the extension +class LoadExtension(click.ParamType): + name = "path:entrypoint?" + + def convert(self, value, param, ctx): + if ":" not in value: + return value + path, entrypoint = value.split(":", 1) + return path, entrypoint + def format_bytes(bytes): current = float(bytes) diff --git a/tests/ext.c b/tests/ext.c new file mode 100644 index 00000000..5fe970d9 --- /dev/null +++ b/tests/ext.c @@ -0,0 +1,48 @@ +/* +** This file implements a SQLite extension with multiple entrypoints. +** +** The default entrypoint, sqlite3_ext_init, has a single function "a". +** The 1st alternate entrypoint, sqlite3_ext_b_init, has a single function "b". +** The 2nd alternate entrypoint, sqlite3_ext_c_init, has a single function "c". +** +** Compiling instructions: +** https://www.sqlite.org/loadext.html#compiling_a_loadable_extension +** +*/ + +#include "sqlite3ext.h" + +SQLITE_EXTENSION_INIT1 + +// SQL function that returns back the value supplied during sqlite3_create_function() +static void func(sqlite3_context *context, int argc, sqlite3_value **argv) { + sqlite3_result_text(context, (char *) sqlite3_user_data(context), -1, SQLITE_STATIC); +} + + +// The default entrypoint, since it matches the "ext.dylib"/"ext.so" name +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_ext_init(sqlite3 *db, char **pzErrMsg, const sqlite3_api_routines *pApi) { + SQLITE_EXTENSION_INIT2(pApi); + return sqlite3_create_function(db, "a", 0, 0, "a", func, 0, 0); +} + +// Alternate entrypoint #1 +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_ext_b_init(sqlite3 *db, char **pzErrMsg, const sqlite3_api_routines *pApi) { + SQLITE_EXTENSION_INIT2(pApi); + return sqlite3_create_function(db, "b", 0, 0, "b", func, 0, 0); +} + +// Alternate entrypoint #2 +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_ext_c_init(sqlite3 *db, char **pzErrMsg, const sqlite3_api_routines *pApi) { + SQLITE_EXTENSION_INIT2(pApi); + return sqlite3_create_function(db, "c", 0, 0, "c", func, 0, 0); +} diff --git a/tests/test_load_extensions.py b/tests/test_load_extensions.py new file mode 100644 index 00000000..360bc8f3 --- /dev/null +++ b/tests/test_load_extensions.py @@ -0,0 +1,65 @@ +from datasette.app import Datasette +import pytest +from pathlib import Path + +# not necessarily a full path - the full compiled path looks like "ext.dylib" +# or another suffix, but sqlite will, under the hood, decide which file +# extension to use based on the operating system (apple=dylib, windows=dll etc) +# this resolves to "./ext", which is enough for SQLite to calculate the rest +COMPILED_EXTENSION_PATH = str(Path(__file__).parent / "ext") + +# See if ext.c has been compiled, based off the different possible suffixes. +def has_compiled_ext(): + for ext in ["dylib", "so", "dll"]: + path = Path(__file__).parent / f"ext.{ext}" + if path.is_file(): + return True + return False + + +@pytest.mark.asyncio +@pytest.mark.skipif(not has_compiled_ext(), reason="Requires compiled ext.c") +async def test_load_extension_default_entrypoint(): + + # The default entrypoint only loads a() and NOT b() or c(), so those + # should fail. + ds = Datasette(sqlite_extensions=[COMPILED_EXTENSION_PATH]) + + response = await ds.client.get("/_memory.json?sql=select+a()") + assert response.status_code == 200 + assert response.json()["rows"][0][0] == "a" + + response = await ds.client.get("/_memory.json?sql=select+b()") + assert response.status_code == 400 + assert response.json()["error"] == "no such function: b" + + response = await ds.client.get("/_memory.json?sql=select+c()") + assert response.status_code == 400 + assert response.json()["error"] == "no such function: c" + + +@pytest.mark.asyncio +@pytest.mark.skipif(not has_compiled_ext(), reason="Requires compiled ext.c") +async def test_load_extension_multiple_entrypoints(): + + # Load in the default entrypoint and the other 2 custom entrypoints, now + # all a(), b(), and c() should run successfully. + ds = Datasette( + sqlite_extensions=[ + COMPILED_EXTENSION_PATH, + (COMPILED_EXTENSION_PATH, "sqlite3_ext_b_init"), + (COMPILED_EXTENSION_PATH, "sqlite3_ext_c_init"), + ] + ) + + response = await ds.client.get("/_memory.json?sql=select+a()") + assert response.status_code == 200 + assert response.json()["rows"][0][0] == "a" + + response = await ds.client.get("/_memory.json?sql=select+b()") + assert response.status_code == 200 + assert response.json()["rows"][0][0] == "b" + + response = await ds.client.get("/_memory.json?sql=select+c()") + assert response.status_code == 200 + assert response.json()["rows"][0][0] == "c" From fd1086c6867f3e3582b1eca456e4ea95f6cecf8b Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 23 Aug 2022 11:35:41 -0700 Subject: [PATCH 127/952] Applied Black, refs #1789 --- datasette/app.py | 4 ++-- datasette/utils/__init__.py | 1 + 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index bb9232c9..f2a6763a 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -559,8 +559,8 @@ class Datasette: if self.sqlite_extensions: conn.enable_load_extension(True) for extension in self.sqlite_extensions: - # "extension" is either a string path to the extension - # or a 2-item tuple that specifies which entrypoint to load. + # "extension" is either a string path to the extension + # or a 2-item tuple that specifies which entrypoint to load. if isinstance(extension, tuple): path, entrypoint = extension conn.execute("SELECT load_extension(?, ?)", [path, entrypoint]) diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index 0fc87d51..bbaa0510 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -833,6 +833,7 @@ class StaticMount(click.ParamType): self.fail(f"{value} is not a valid directory path", param, ctx) return path, dirpath + # The --load-extension parameter can optionally include a specific entrypoint. # This is done by appending ":entrypoint_name" after supplying the path to the extension class LoadExtension(click.ParamType): From 456dc155d491a009942ace71a4e1827cddc6b93d Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 23 Aug 2022 11:40:36 -0700 Subject: [PATCH 128/952] Ran cog, refs #1789 --- docs/cli-reference.rst | 95 +++++++++++++++++++++++------------------- 1 file changed, 51 insertions(+), 44 deletions(-) diff --git a/docs/cli-reference.rst b/docs/cli-reference.rst index a1e56774..f8419d58 100644 --- a/docs/cli-reference.rst +++ b/docs/cli-reference.rst @@ -84,48 +84,53 @@ Once started you can access it at ``http://localhost:8001`` Serve up specified SQLite database files with a web UI Options: - -i, --immutable PATH Database files to open in immutable mode - -h, --host TEXT Host for server. Defaults to 127.0.0.1 which means - only connections from the local machine will be - allowed. Use 0.0.0.0 to listen to all IPs and allow - access from other machines. - -p, --port INTEGER RANGE Port for server, defaults to 8001. Use -p 0 to - automatically assign an available port. - [0<=x<=65535] - --uds TEXT Bind to a Unix domain socket - --reload Automatically reload if code or metadata change - detected - useful for development - --cors Enable CORS by serving Access-Control-Allow-Origin: - * - --load-extension TEXT Path to a SQLite extension to load - --inspect-file TEXT Path to JSON file created using "datasette inspect" - -m, --metadata FILENAME Path to JSON/YAML file containing license/source - metadata - --template-dir DIRECTORY Path to directory containing custom templates - --plugins-dir DIRECTORY Path to directory containing custom plugins - --static MOUNT:DIRECTORY Serve static files from this directory at /MOUNT/... - --memory Make /_memory database available - --config CONFIG Deprecated: set config option using - configname:value. Use --setting instead. - --setting SETTING... Setting, see - docs.datasette.io/en/stable/settings.html - --secret TEXT Secret used for signing secure values, such as - signed cookies - --root Output URL that sets a cookie authenticating the - root user - --get TEXT Run an HTTP GET request against this path, print - results and exit - --version-note TEXT Additional note to show on /-/versions - --help-settings Show available settings - --pdb Launch debugger on any errors - -o, --open Open Datasette in your web browser - --create Create database files if they do not exist - --crossdb Enable cross-database joins using the /_memory - database - --nolock Ignore locking, open locked files in read-only mode - --ssl-keyfile TEXT SSL key file - --ssl-certfile TEXT SSL certificate file - --help Show this message and exit. + -i, --immutable PATH Database files to open in immutable mode + -h, --host TEXT Host for server. Defaults to 127.0.0.1 which + means only connections from the local machine + will be allowed. Use 0.0.0.0 to listen to all + IPs and allow access from other machines. + -p, --port INTEGER RANGE Port for server, defaults to 8001. Use -p 0 to + automatically assign an available port. + [0<=x<=65535] + --uds TEXT Bind to a Unix domain socket + --reload Automatically reload if code or metadata + change detected - useful for development + --cors Enable CORS by serving Access-Control-Allow- + Origin: * + --load-extension PATH:ENTRYPOINT? + Path to a SQLite extension to load, and + optional entrypoint + --inspect-file TEXT Path to JSON file created using "datasette + inspect" + -m, --metadata FILENAME Path to JSON/YAML file containing + license/source metadata + --template-dir DIRECTORY Path to directory containing custom templates + --plugins-dir DIRECTORY Path to directory containing custom plugins + --static MOUNT:DIRECTORY Serve static files from this directory at + /MOUNT/... + --memory Make /_memory database available + --config CONFIG Deprecated: set config option using + configname:value. Use --setting instead. + --setting SETTING... Setting, see + docs.datasette.io/en/stable/settings.html + --secret TEXT Secret used for signing secure values, such as + signed cookies + --root Output URL that sets a cookie authenticating + the root user + --get TEXT Run an HTTP GET request against this path, + print results and exit + --version-note TEXT Additional note to show on /-/versions + --help-settings Show available settings + --pdb Launch debugger on any errors + -o, --open Open Datasette in your web browser + --create Create database files if they do not exist + --crossdb Enable cross-database joins using the /_memory + database + --nolock Ignore locking, open locked files in read-only + mode + --ssl-keyfile TEXT SSL key file + --ssl-certfile TEXT SSL certificate file + --help Show this message and exit. .. [[[end]]] @@ -566,8 +571,10 @@ This performance optimization is used automatically by some of the ``datasette p Options: --inspect-file TEXT - --load-extension TEXT Path to a SQLite extension to load - --help Show this message and exit. + --load-extension PATH:ENTRYPOINT? + Path to a SQLite extension to load, and + optional entrypoint + --help Show this message and exit. .. [[[end]]] From ba35105eee2d3ba620e4f230028a02b2e2571df2 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 23 Aug 2022 17:11:45 -0700 Subject: [PATCH 129/952] Test `--load-extension` in GitHub Actions (#1792) * Run the --load-extension test, refs #1789 * Ran cog, refs #1789 --- .github/workflows/test.yml | 3 +++ tests/test_api.py | 2 +- tests/test_html.py | 4 ++-- 3 files changed, 6 insertions(+), 3 deletions(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 90b6555e..e38d5ee9 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -24,6 +24,9 @@ jobs: key: ${{ runner.os }}-pip-${{ hashFiles('**/setup.py') }} restore-keys: | ${{ runner.os }}-pip- + - name: Build extension for --load-extension test + run: |- + (cd tests && gcc ext.c -fPIC -shared -o ext.so) - name: Install dependencies run: | pip install -e '.[test]' diff --git a/tests/test_api.py b/tests/test_api.py index 253c1718..f6db2f9d 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -36,7 +36,7 @@ def test_homepage(app_client): # 4 hidden FTS tables + no_primary_key (hidden in metadata) assert d["hidden_tables_count"] == 6 # 201 in no_primary_key, plus 6 in other hidden tables: - assert d["hidden_table_rows_sum"] == 207 + assert d["hidden_table_rows_sum"] == 207, response.json assert d["views_count"] == 4 diff --git a/tests/test_html.py b/tests/test_html.py index be21bd84..d6e969ad 100644 --- a/tests/test_html.py +++ b/tests/test_html.py @@ -115,7 +115,7 @@ def test_database_page(app_client): assert fragment in response.text # And views - views_ul = soup.find("h2", text="Views").find_next_sibling("ul") + views_ul = soup.find("h2", string="Views").find_next_sibling("ul") assert views_ul is not None assert [ ("/fixtures/paginated_view", "paginated_view"), @@ -128,7 +128,7 @@ def test_database_page(app_client): ] == sorted([(a["href"], a.text) for a in views_ul.find_all("a")]) # And a list of canned queries - queries_ul = soup.find("h2", text="Queries").find_next_sibling("ul") + queries_ul = soup.find("h2", string="Queries").find_next_sibling("ul") assert queries_ul is not None assert [ ("/fixtures/from_async_hook", "from_async_hook"), From 51030df1869b3b574dd3584d1563415776b9cd4e Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 5 Sep 2022 11:35:40 -0700 Subject: [PATCH 130/952] Don't use upper bound dependencies any more See https://iscinumpy.dev/post/bound-version-constraints/ for the rationale behind this change. Closes #1800 --- setup.py | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/setup.py b/setup.py index a1c51d0b..b2e50b38 100644 --- a/setup.py +++ b/setup.py @@ -42,21 +42,21 @@ setup( include_package_data=True, python_requires=">=3.7", install_requires=[ - "asgiref>=3.2.10,<3.6.0", - "click>=7.1.1,<8.2.0", + "asgiref>=3.2.10", + "click>=7.1.1", "click-default-group-wheel>=1.2.2", - "Jinja2>=2.10.3,<3.1.0", - "hupper~=1.9", + "Jinja2>=2.10.3", + "hupper>=1.9", "httpx>=0.20", - "pint~=0.9", - "pluggy>=1.0,<1.1", - "uvicorn~=0.11", - "aiofiles>=0.4,<0.9", - "janus>=0.6.2,<1.1", + "pint>=0.9", + "pluggy>=1.0", + "uvicorn>=0.11", + "aiofiles>=0.4", + "janus>=0.6.2", "asgi-csrf>=0.9", - "PyYAML>=5.3,<7.0", - "mergedeep>=1.1.1,<1.4.0", - "itsdangerous>=1.1,<3.0", + "PyYAML>=5.3", + "mergedeep>=1.1.1", + "itsdangerous>=1.1", ], entry_points=""" [console_scripts] @@ -72,14 +72,14 @@ setup( "sphinx-copybutton", ], "test": [ - "pytest>=5.2.2,<7.2.0", - "pytest-xdist>=2.2.1,<2.6", - "pytest-asyncio>=0.17,<0.20", - "beautifulsoup4>=4.8.1,<4.12.0", + "pytest>=5.2.2", + "pytest-xdist>=2.2.1", + "pytest-asyncio>=0.17", + "beautifulsoup4>=4.8.1", "black==22.6.0", "blacken-docs==1.12.1", - "pytest-timeout>=1.4.2,<2.2", - "trustme>=0.7,<0.10", + "pytest-timeout>=1.4.2", + "trustme>=0.7", "cogapp>=3.3.0", ], "rich": ["rich"], From 294ecd45f7801971dbeef383d0c5456ee95ab839 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 5 Sep 2022 11:51:51 -0700 Subject: [PATCH 131/952] Bump black from 22.6.0 to 22.8.0 (#1797) Bumps [black](https://github.com/psf/black) from 22.6.0 to 22.8.0. - [Release notes](https://github.com/psf/black/releases) - [Changelog](https://github.com/psf/black/blob/main/CHANGES.md) - [Commits](https://github.com/psf/black/compare/22.6.0...22.8.0) --- updated-dependencies: - dependency-name: black dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index b2e50b38..92fa60d0 100644 --- a/setup.py +++ b/setup.py @@ -76,7 +76,7 @@ setup( "pytest-xdist>=2.2.1", "pytest-asyncio>=0.17", "beautifulsoup4>=4.8.1", - "black==22.6.0", + "black==22.8.0", "blacken-docs==1.12.1", "pytest-timeout>=1.4.2", "trustme>=0.7", From b91e17280c05bbb9cf97432081bdcea8665879f9 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 5 Sep 2022 16:50:53 -0700 Subject: [PATCH 132/952] Run tests in serial, refs #1802 --- .github/workflows/test.yml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index e38d5ee9..9c8c48ef 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -33,8 +33,7 @@ jobs: pip freeze - name: Run tests run: | - pytest -n auto -m "not serial" - pytest -m "serial" + pytest - name: Check if cog needs to be run run: | cog --check docs/*.rst From b2b901e8c4b939e50ee1117ffcd2881ed8a8e3bf Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 5 Sep 2022 17:05:23 -0700 Subject: [PATCH 133/952] Skip SpatiaLite test if no conn.enable_load_extension() Ran into this problem while working on #1802 --- tests/test_spatialite.py | 2 ++ tests/utils.py | 8 ++++++++ 2 files changed, 10 insertions(+) diff --git a/tests/test_spatialite.py b/tests/test_spatialite.py index 8b98c5d6..c07a30e8 100644 --- a/tests/test_spatialite.py +++ b/tests/test_spatialite.py @@ -1,5 +1,6 @@ from datasette.app import Datasette from datasette.utils import find_spatialite, SpatialiteNotFound, SPATIALITE_FUNCTIONS +from .utils import has_load_extension import pytest @@ -13,6 +14,7 @@ def has_spatialite(): @pytest.mark.asyncio @pytest.mark.skipif(not has_spatialite(), reason="Requires SpatiaLite") +@pytest.mark.skipif(not has_load_extension(), reason="Requires enable_load_extension") async def test_spatialite_version_info(): ds = Datasette(sqlite_extensions=["spatialite"]) response = await ds.client.get("/-/versions.json") diff --git a/tests/utils.py b/tests/utils.py index 972300db..191ead9b 100644 --- a/tests/utils.py +++ b/tests/utils.py @@ -1,3 +1,6 @@ +from datasette.utils.sqlite import sqlite3 + + def assert_footer_links(soup): footer_links = soup.find("footer").findAll("a") assert 4 == len(footer_links) @@ -22,3 +25,8 @@ def inner_html(soup): # This includes the parent tag - so remove that inner_html = html.split(">", 1)[1].rsplit("<", 1)[0] return inner_html.strip() + + +def has_load_extension(): + conn = sqlite3.connect(":memory:") + return hasattr(conn, "enable_load_extension") From 1c29b925d300d1ee17047504473f2517767aa05b Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 5 Sep 2022 17:10:52 -0700 Subject: [PATCH 134/952] Run tests in serial again Because this didn't fix the issue I'm seeing in #1802 Revert "Run tests in serial, refs #1802" This reverts commit b91e17280c05bbb9cf97432081bdcea8665879f9. --- .github/workflows/test.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 9c8c48ef..e38d5ee9 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -33,7 +33,8 @@ jobs: pip freeze - name: Run tests run: | - pytest + pytest -n auto -m "not serial" + pytest -m "serial" - name: Check if cog needs to be run run: | cog --check docs/*.rst From 64288d827f7ff97f825e10f714da3f781ecf9345 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 5 Sep 2022 17:40:19 -0700 Subject: [PATCH 135/952] Workaround for test failure: RuntimeError: There is no current event loop (#1803) * Remove ensure_eventloop hack * Hack to recover from intermittent RuntimeError calling asyncio.Lock() --- datasette/app.py | 10 +++++++++- tests/test_cli.py | 27 ++++++++++----------------- 2 files changed, 19 insertions(+), 18 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index f2a6763a..c6bbdaf0 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -231,7 +231,15 @@ class Datasette: self.inspect_data = inspect_data self.immutables = set(immutables or []) self.databases = collections.OrderedDict() - self._refresh_schemas_lock = asyncio.Lock() + try: + self._refresh_schemas_lock = asyncio.Lock() + except RuntimeError as rex: + # Workaround for intermittent test failure, see: + # https://github.com/simonw/datasette/issues/1802 + if "There is no current event loop in thread" in str(rex): + loop = asyncio.new_event_loop() + asyncio.set_event_loop(loop) + self._refresh_schemas_lock = asyncio.Lock() self.crossdb = crossdb self.nolock = nolock if memory or crossdb or not self.files: diff --git a/tests/test_cli.py b/tests/test_cli.py index d0f6e26c..f0d28037 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -22,13 +22,6 @@ from unittest import mock import urllib -@pytest.fixture -def ensure_eventloop(): - # Workaround for "Event loop is closed" error - if asyncio.get_event_loop().is_closed(): - asyncio.set_event_loop(asyncio.new_event_loop()) - - def test_inspect_cli(app_client): runner = CliRunner() result = runner.invoke(cli, ["inspect", "fixtures.db"]) @@ -72,7 +65,7 @@ def test_serve_with_inspect_file_prepopulates_table_counts_cache(): ), ) def test_spatialite_error_if_attempt_to_open_spatialite( - ensure_eventloop, spatialite_paths, should_suggest_load_extension + spatialite_paths, should_suggest_load_extension ): with mock.patch("datasette.utils.SPATIALITE_PATHS", spatialite_paths): runner = CliRunner() @@ -199,14 +192,14 @@ def test_version(): @pytest.mark.parametrize("invalid_port", ["-1", "0.5", "dog", "65536"]) -def test_serve_invalid_ports(ensure_eventloop, invalid_port): +def test_serve_invalid_ports(invalid_port): runner = CliRunner(mix_stderr=False) result = runner.invoke(cli, ["--port", invalid_port]) assert result.exit_code == 2 assert "Invalid value for '-p'" in result.stderr -def test_setting(ensure_eventloop): +def test_setting(): runner = CliRunner() result = runner.invoke( cli, ["--setting", "default_page_size", "5", "--get", "/-/settings.json"] @@ -215,14 +208,14 @@ def test_setting(ensure_eventloop): assert json.loads(result.output)["default_page_size"] == 5 -def test_setting_type_validation(ensure_eventloop): +def test_setting_type_validation(): runner = CliRunner(mix_stderr=False) result = runner.invoke(cli, ["--setting", "default_page_size", "dog"]) assert result.exit_code == 2 assert '"default_page_size" should be an integer' in result.stderr -def test_config_deprecated(ensure_eventloop): +def test_config_deprecated(): # The --config option should show a deprecation message runner = CliRunner(mix_stderr=False) result = runner.invoke( @@ -233,14 +226,14 @@ def test_config_deprecated(ensure_eventloop): assert "will be deprecated in" in result.stderr -def test_sql_errors_logged_to_stderr(ensure_eventloop): +def test_sql_errors_logged_to_stderr(): runner = CliRunner(mix_stderr=False) result = runner.invoke(cli, ["--get", "/_memory.json?sql=select+blah"]) assert result.exit_code == 1 assert "sql = 'select blah', params = {}: no such column: blah\n" in result.stderr -def test_serve_create(ensure_eventloop, tmpdir): +def test_serve_create(tmpdir): runner = CliRunner() db_path = tmpdir / "does_not_exist_yet.db" assert not db_path.exists() @@ -258,7 +251,7 @@ def test_serve_create(ensure_eventloop, tmpdir): assert db_path.exists() -def test_serve_duplicate_database_names(ensure_eventloop, tmpdir): +def test_serve_duplicate_database_names(tmpdir): "'datasette db.db nested/db.db' should attach two databases, /db and /db_2" runner = CliRunner() db_1_path = str(tmpdir / "db.db") @@ -273,7 +266,7 @@ def test_serve_duplicate_database_names(ensure_eventloop, tmpdir): assert {db["name"] for db in databases} == {"db", "db_2"} -def test_serve_deduplicate_same_database_path(ensure_eventloop, tmpdir): +def test_serve_deduplicate_same_database_path(tmpdir): "'datasette db.db db.db' should only attach one database, /db" runner = CliRunner() db_path = str(tmpdir / "db.db") @@ -287,7 +280,7 @@ def test_serve_deduplicate_same_database_path(ensure_eventloop, tmpdir): @pytest.mark.parametrize( "filename", ["test-database (1).sqlite", "database (1).sqlite"] ) -def test_weird_database_names(ensure_eventloop, tmpdir, filename): +def test_weird_database_names(tmpdir, filename): # https://github.com/simonw/datasette/issues/1181 runner = CliRunner() db_path = str(tmpdir / filename) From c9d1943aede436fa3413fd49bc56335cbda4ad07 Mon Sep 17 00:00:00 2001 From: Daniel Rech Date: Tue, 6 Sep 2022 02:45:41 +0200 Subject: [PATCH 136/952] Fix word break in facets by adding ul.tight-bullets li word-break: break-all (#1794) Thanks, @dmr --- datasette/static/app.css | 1 + 1 file changed, 1 insertion(+) diff --git a/datasette/static/app.css b/datasette/static/app.css index af3e14d5..712b9925 100644 --- a/datasette/static/app.css +++ b/datasette/static/app.css @@ -260,6 +260,7 @@ ul.bullets li { ul.tight-bullets li { list-style-type: disc; margin-bottom: 0; + word-break: break-all; } a.not-underlined { text-decoration: none; From d80775a48d20917633792fdc9525f075d3bc2c7a Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 5 Sep 2022 17:44:44 -0700 Subject: [PATCH 137/952] Raise error if it's not about loops, refs #1802 --- datasette/app.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/datasette/app.py b/datasette/app.py index c6bbdaf0..aeb81687 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -240,6 +240,8 @@ class Datasette: loop = asyncio.new_event_loop() asyncio.set_event_loop(loop) self._refresh_schemas_lock = asyncio.Lock() + else: + raise self.crossdb = crossdb self.nolock = nolock if memory or crossdb or not self.files: From 8430c3bc7dd22b173c1a8c6cd7180e3b31240cd1 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 6 Sep 2022 08:59:19 -0700 Subject: [PATCH 138/952] table facet_size in metadata, refs #1804 --- datasette/facets.py | 14 +++++++++++--- tests/test_facets.py | 17 +++++++++++++++++ 2 files changed, 28 insertions(+), 3 deletions(-) diff --git a/datasette/facets.py b/datasette/facets.py index b15a758c..e70d42df 100644 --- a/datasette/facets.py +++ b/datasette/facets.py @@ -102,11 +102,19 @@ class Facet: def get_facet_size(self): facet_size = self.ds.setting("default_facet_size") max_returned_rows = self.ds.setting("max_returned_rows") + table_facet_size = None + if self.table: + tables_metadata = self.ds.metadata("tables", database=self.database) or {} + table_metadata = tables_metadata.get(self.table) or {} + if table_metadata: + table_facet_size = table_metadata.get("facet_size") custom_facet_size = self.request.args.get("_facet_size") - if custom_facet_size == "max": - facet_size = max_returned_rows - elif custom_facet_size and custom_facet_size.isdigit(): + if custom_facet_size and custom_facet_size.isdigit(): facet_size = int(custom_facet_size) + elif table_facet_size: + facet_size = table_facet_size + if facet_size == "max": + facet_size = max_returned_rows return min(facet_size, max_returned_rows) async def suggest(self): diff --git a/tests/test_facets.py b/tests/test_facets.py index c28dc43c..cbee23b0 100644 --- a/tests/test_facets.py +++ b/tests/test_facets.py @@ -581,6 +581,23 @@ async def test_facet_size(): ) data5 = response5.json() assert len(data5["facet_results"]["city"]["results"]) == 20 + # Now try messing with facet_size in the table metadata + ds._metadata_local = { + "databases": { + "test_facet_size": {"tables": {"neighbourhoods": {"facet_size": 6}}} + } + } + response6 = await ds.client.get("/test_facet_size/neighbourhoods.json?_facet=city") + data6 = response6.json() + assert len(data6["facet_results"]["city"]["results"]) == 6 + # Setting it to max bumps it up to 50 again + ds._metadata_local["databases"]["test_facet_size"]["tables"]["neighbourhoods"][ + "facet_size" + ] = "max" + data7 = ( + await ds.client.get("/test_facet_size/neighbourhoods.json?_facet=city") + ).json() + assert len(data7["facet_results"]["city"]["results"]) == 20 def test_other_types_of_facet_in_metadata(): From 303c6c733d95a6133558ec1b468f5bea5827d0d2 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 6 Sep 2022 11:05:00 -0700 Subject: [PATCH 139/952] Fix for incorrectly handled _facet_size=max, refs #1804 --- datasette/facets.py | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/datasette/facets.py b/datasette/facets.py index e70d42df..7fb0c68b 100644 --- a/datasette/facets.py +++ b/datasette/facets.py @@ -109,12 +109,19 @@ class Facet: if table_metadata: table_facet_size = table_metadata.get("facet_size") custom_facet_size = self.request.args.get("_facet_size") - if custom_facet_size and custom_facet_size.isdigit(): - facet_size = int(custom_facet_size) - elif table_facet_size: - facet_size = table_facet_size - if facet_size == "max": - facet_size = max_returned_rows + if custom_facet_size: + if custom_facet_size == "max": + facet_size = max_returned_rows + elif custom_facet_size.isdigit(): + facet_size = int(custom_facet_size) + else: + # Invalid value, ignore it + custom_facet_size = None + if table_facet_size and not custom_facet_size: + if table_facet_size == "max": + facet_size = max_returned_rows + else: + facet_size = table_facet_size return min(facet_size, max_returned_rows) async def suggest(self): From 0a7815d2038255a0834c955066a2a16c01f707b2 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 6 Sep 2022 11:06:49 -0700 Subject: [PATCH 140/952] Documentation for facet_size in metadata, closes #1804 --- docs/facets.rst | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/facets.rst b/docs/facets.rst index 2a2eb039..6c9d99bd 100644 --- a/docs/facets.rst +++ b/docs/facets.rst @@ -129,6 +129,22 @@ You can specify :ref:`array ` or :ref:`date ] } +You can change the default facet size (the number of results shown for each facet) for a table using ``facet_size``: + +.. code-block:: json + + { + "databases": { + "sf-trees": { + "tables": { + "Street_Tree_List": { + "facets": ["qLegalStatus"], + "facet_size": 10 + } + } + } + } + } Suggested facets ---------------- From d0476897e10249bb4867473722270d02491c2c1f Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 6 Sep 2022 11:24:30 -0700 Subject: [PATCH 141/952] Fixed Sphinx warning about language = None --- docs/conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/conf.py b/docs/conf.py index 4ef6b768..8965974a 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -71,7 +71,7 @@ release = "" # # This is also used if you do content translation via gettext catalogs. # Usually you set "language" from the command line for these cases. -language = None +language = "en" # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. From ff9c87197dde8b09f9787ee878804cb6842ea5dc Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 6 Sep 2022 11:26:21 -0700 Subject: [PATCH 142/952] Fixed Sphinx warnings on cli-reference page --- docs/cli-reference.rst | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/docs/cli-reference.rst b/docs/cli-reference.rst index f8419d58..4a8465cb 100644 --- a/docs/cli-reference.rst +++ b/docs/cli-reference.rst @@ -14,7 +14,7 @@ Running ``datasette`` without specifying a command runs the default command, ``d import textwrap def help(args): title = "datasette " + " ".join(args) - cog.out("::\n\n") + cog.out("\n::\n\n") result = CliRunner().invoke(cli.cli, args) output = result.output.replace("Usage: cli ", "Usage: datasette ") cog.out(textwrap.indent(output, ' ')) @@ -32,6 +32,7 @@ Running ``datasette --help`` shows a list of all of the available commands. .. [[[cog help(["--help"]) .. ]]] + :: Usage: datasette [OPTIONS] COMMAND [ARGS]... @@ -77,6 +78,7 @@ Once started you can access it at ``http://localhost:8001`` .. [[[cog help(["serve", "--help"]) .. ]]] + :: Usage: datasette serve [OPTIONS] [FILES]... @@ -202,6 +204,7 @@ These can be passed to ``datasette serve`` using ``datasette serve --setting nam .. [[[cog help(["--help-settings"]) .. ]]] + :: Settings: @@ -258,6 +261,7 @@ Output JSON showing all currently installed plugins, their versions, whether the .. [[[cog help(["plugins", "--help"]) .. ]]] + :: Usage: datasette plugins [OPTIONS] @@ -326,6 +330,7 @@ Would install the `datasette-cluster-map Date: Tue, 6 Sep 2022 16:50:43 -0700 Subject: [PATCH 143/952] truncate_cells_html now affects URLs too, refs #1805 --- datasette/utils/__init__.py | 10 ++++++++++ datasette/views/database.py | 11 ++++++++--- datasette/views/table.py | 8 ++++++-- tests/fixtures.py | 9 +++++---- tests/test_api.py | 2 +- tests/test_table_api.py | 11 +++++++---- tests/test_table_html.py | 11 +++++++++++ tests/test_utils.py | 20 ++++++++++++++++++++ 8 files changed, 68 insertions(+), 14 deletions(-) diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index bbaa0510..2bdea673 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -1167,3 +1167,13 @@ def resolve_routes(routes, path): if match is not None: return match, view return None, None + + +def truncate_url(url, length): + if (not length) or (len(url) <= length): + return url + bits = url.rsplit(".", 1) + if len(bits) == 2 and 1 <= len(bits[1]) <= 4 and "/" not in bits[1]: + rest, ext = bits + return rest[: length - 1 - len(ext)] + "…." + ext + return url[: length - 1] + "…" diff --git a/datasette/views/database.py b/datasette/views/database.py index 77632b9d..fc344245 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -20,6 +20,7 @@ from datasette.utils import ( path_with_format, path_with_removed_args, sqlite3, + truncate_url, InvalidSql, ) from datasette.utils.asgi import AsgiFileDownload, NotFound, Response, Forbidden @@ -371,6 +372,7 @@ class QueryView(DataView): async def extra_template(): display_rows = [] + truncate_cells = self.ds.setting("truncate_cells_html") for row in results.rows if results else []: display_row = [] for column, value in zip(results.columns, row): @@ -396,9 +398,12 @@ class QueryView(DataView): if value in ("", None): display_value = Markup(" ") elif is_url(str(display_value).strip()): - display_value = Markup( - '{url}'.format( - url=escape(value.strip()) + display_value = markupsafe.Markup( + '{truncated_url}'.format( + url=markupsafe.escape(value.strip()), + truncated_url=markupsafe.escape( + truncate_url(value.strip(), truncate_cells) + ), ) ) elif isinstance(display_value, bytes): diff --git a/datasette/views/table.py b/datasette/views/table.py index 49c30c9c..60c092f9 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -24,6 +24,7 @@ from datasette.utils import ( path_with_removed_args, path_with_replaced_args, to_css_class, + truncate_url, urlsafe_components, value_as_boolean, ) @@ -966,8 +967,11 @@ async def display_columns_and_rows( display_value = markupsafe.Markup(" ") elif is_url(str(value).strip()): display_value = markupsafe.Markup( - '{url}'.format( - url=markupsafe.escape(value.strip()) + '{truncated_url}'.format( + url=markupsafe.escape(value.strip()), + truncated_url=markupsafe.escape( + truncate_url(value.strip(), truncate_cells) + ), ) ) elif column in table_metadata.get("units", {}) and value != "": diff --git a/tests/fixtures.py b/tests/fixtures.py index c145ac78..82d8452e 100644 --- a/tests/fixtures.py +++ b/tests/fixtures.py @@ -598,23 +598,24 @@ CREATE TABLE roadside_attractions ( pk integer primary key, name text, address text, + url text, latitude real, longitude real ); INSERT INTO roadside_attractions VALUES ( - 1, "The Mystery Spot", "465 Mystery Spot Road, Santa Cruz, CA 95065", + 1, "The Mystery Spot", "465 Mystery Spot Road, Santa Cruz, CA 95065", "https://www.mysteryspot.com/", 37.0167, -122.0024 ); INSERT INTO roadside_attractions VALUES ( - 2, "Winchester Mystery House", "525 South Winchester Boulevard, San Jose, CA 95128", + 2, "Winchester Mystery House", "525 South Winchester Boulevard, San Jose, CA 95128", "https://winchestermysteryhouse.com/", 37.3184, -121.9511 ); INSERT INTO roadside_attractions VALUES ( - 3, "Burlingame Museum of PEZ Memorabilia", "214 California Drive, Burlingame, CA 94010", + 3, "Burlingame Museum of PEZ Memorabilia", "214 California Drive, Burlingame, CA 94010", null, 37.5793, -122.3442 ); INSERT INTO roadside_attractions VALUES ( - 4, "Bigfoot Discovery Museum", "5497 Highway 9, Felton, CA 95018", + 4, "Bigfoot Discovery Museum", "5497 Highway 9, Felton, CA 95018", "https://www.bigfootdiscoveryproject.com/", 37.0414, -122.0725 ); diff --git a/tests/test_api.py b/tests/test_api.py index f6db2f9d..7a2bf91f 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -339,7 +339,7 @@ def test_database_page(app_client): }, { "name": "roadside_attractions", - "columns": ["pk", "name", "address", "latitude", "longitude"], + "columns": ["pk", "name", "address", "url", "latitude", "longitude"], "primary_keys": ["pk"], "count": 4, "hidden": False, diff --git a/tests/test_table_api.py b/tests/test_table_api.py index e56a72b5..0db04434 100644 --- a/tests/test_table_api.py +++ b/tests/test_table_api.py @@ -615,11 +615,12 @@ def test_table_through(app_client): response = app_client.get( '/fixtures/roadside_attractions.json?_through={"table":"roadside_attraction_characteristics","column":"characteristic_id","value":"1"}' ) - assert [ + assert response.json["rows"] == [ [ 3, "Burlingame Museum of PEZ Memorabilia", "214 California Drive, Burlingame, CA 94010", + None, 37.5793, -122.3442, ], @@ -627,13 +628,15 @@ def test_table_through(app_client): 4, "Bigfoot Discovery Museum", "5497 Highway 9, Felton, CA 95018", + "https://www.bigfootdiscoveryproject.com/", 37.0414, -122.0725, ], - ] == response.json["rows"] + ] + assert ( - 'where roadside_attraction_characteristics.characteristic_id = "1"' - == response.json["human_description_en"] + response.json["human_description_en"] + == 'where roadside_attraction_characteristics.characteristic_id = "1"' ) diff --git a/tests/test_table_html.py b/tests/test_table_html.py index f3808ea3..8e37468f 100644 --- a/tests/test_table_html.py +++ b/tests/test_table_html.py @@ -69,6 +69,17 @@ def test_table_cell_truncation(): td.string for td in table.findAll("td", {"class": "col-neighborhood-b352a7"}) ] + # URLs should be truncated too + response2 = client.get("/fixtures/roadside_attractions") + assert response2.status == 200 + table = Soup(response2.body, "html.parser").find("table") + tds = table.findAll("td", {"class": "col-url"}) + assert [str(td) for td in tds] == [ + '', + '', + '', + '', + ] def test_add_filter_redirects(app_client): diff --git a/tests/test_utils.py b/tests/test_utils.py index df788767..d71a612d 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -626,3 +626,23 @@ def test_tilde_encoding(original, expected): assert actual == expected # And test round-trip assert original == utils.tilde_decode(actual) + + +@pytest.mark.parametrize( + "url,length,expected", + ( + ("https://example.com/", 5, "http…"), + ("https://example.com/foo/bar", 15, "https://exampl…"), + ("https://example.com/foo/bar/baz.jpg", 30, "https://example.com/foo/ba….jpg"), + # Extensions longer than 4 characters are not treated specially: + ("https://example.com/foo/bar/baz.jpeg2", 30, "https://example.com/foo/bar/b…"), + ( + "https://example.com/foo/bar/baz.jpeg2", + None, + "https://example.com/foo/bar/baz.jpeg2", + ), + ), +) +def test_truncate_url(url, length, expected): + actual = utils.truncate_url(url, length) + assert actual == expected From 5aa359b86907d11b3ee601510775a85a90224da8 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 6 Sep 2022 16:58:30 -0700 Subject: [PATCH 144/952] Apply cell truncation on query page too, refs #1805 --- datasette/views/database.py | 7 ++++++- tests/test_html.py | 19 +++++++++++++++++++ 2 files changed, 25 insertions(+), 1 deletion(-) diff --git a/datasette/views/database.py b/datasette/views/database.py index fc344245..affbc540 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -428,7 +428,12 @@ class QueryView(DataView): "" if len(value) == 1 else "s", ) ) - + else: + display_value = str(value) + if truncate_cells and len(display_value) > truncate_cells: + display_value = ( + display_value[:truncate_cells] + "\u2026" + ) display_row.append(display_value) display_rows.append(display_row) diff --git a/tests/test_html.py b/tests/test_html.py index d6e969ad..bf915247 100644 --- a/tests/test_html.py +++ b/tests/test_html.py @@ -186,6 +186,25 @@ def test_row_page_does_not_truncate(): ] +def test_query_page_truncates(): + with make_app_client(settings={"truncate_cells_html": 5}) as client: + response = client.get( + "/fixtures?" + + urllib.parse.urlencode( + { + "sql": "select 'this is longer than 5' as a, 'https://example.com/' as b" + } + ) + ) + assert response.status == 200 + table = Soup(response.body, "html.parser").find("table") + tds = table.findAll("td") + assert [str(td) for td in tds] == [ + '', + '', + ] + + @pytest.mark.parametrize( "path,expected_classes", [ From bf8d84af5422606597be893cedd375020cb2b369 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 6 Sep 2022 20:34:59 -0700 Subject: [PATCH 145/952] word-wrap: anywhere on links in cells, refs #1805 --- datasette/static/app.css | 1 + 1 file changed, 1 insertion(+) diff --git a/datasette/static/app.css b/datasette/static/app.css index 712b9925..08b724f6 100644 --- a/datasette/static/app.css +++ b/datasette/static/app.css @@ -446,6 +446,7 @@ th { } table a:link { text-decoration: none; + word-wrap: anywhere; } .rows-and-columns td:before { display: block; From fb7e70d5e72a951efe4b29ad999d8915c032d021 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 9 Sep 2022 09:19:20 -0700 Subject: [PATCH 146/952] Database(is_mutable=) now defaults to True, closes #1808 Refs https://github.com/simonw/datasette-upload-dbs/issues/6 --- datasette/database.py | 3 +-- docs/internals.rst | 9 +++++---- tests/test_internals_database.py | 1 + tests/test_internals_datasette.py | 2 +- 4 files changed, 8 insertions(+), 7 deletions(-) diff --git a/datasette/database.py b/datasette/database.py index fa558045..44467370 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -28,7 +28,7 @@ AttachedDatabase = namedtuple("AttachedDatabase", ("seq", "name", "file")) class Database: def __init__( - self, ds, path=None, is_mutable=False, is_memory=False, memory_name=None + self, ds, path=None, is_mutable=True, is_memory=False, memory_name=None ): self.name = None self.route = None @@ -39,7 +39,6 @@ class Database: self.memory_name = memory_name if memory_name is not None: self.is_memory = True - self.is_mutable = True self.hash = None self.cached_size = None self._cached_table_counts = None diff --git a/docs/internals.rst b/docs/internals.rst index 20797e98..adeec1d8 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -426,12 +426,13 @@ The ``db`` parameter should be an instance of the ``datasette.database.Database` Database( datasette, path="path/to/my-new-database.db", - is_mutable=True, ) ) This will add a mutable database and serve it at ``/my-new-database``. +Use ``is_mutable=False`` to add an immutable database. + ``.add_database()`` returns the Database instance, with its name set as the ``database.name`` attribute. Any time you are working with a newly added database you should use the return value of ``.add_database()``, for example: .. code-block:: python @@ -671,8 +672,8 @@ Instances of the ``Database`` class can be used to execute queries against attac .. _database_constructor: -Database(ds, path=None, is_mutable=False, is_memory=False, memory_name=None) ----------------------------------------------------------------------------- +Database(ds, path=None, is_mutable=True, is_memory=False, memory_name=None) +--------------------------------------------------------------------------- The ``Database()`` constructor can be used by plugins, in conjunction with :ref:`datasette_add_database`, to create and register new databases. @@ -685,7 +686,7 @@ The arguments are as follows: Path to a SQLite database file on disk. ``is_mutable`` - boolean - Set this to ``True`` if it is possible that updates will be made to that database - otherwise Datasette will open it in immutable mode and any changes could cause undesired behavior. + Set this to ``False`` to cause Datasette to open the file in immutable mode. ``is_memory`` - boolean Use this to create non-shared memory connections. diff --git a/tests/test_internals_database.py b/tests/test_internals_database.py index 551f67e1..9e81c1d6 100644 --- a/tests/test_internals_database.py +++ b/tests/test_internals_database.py @@ -499,6 +499,7 @@ def test_mtime_ns_is_none_for_memory(app_client): def test_is_mutable(app_client): + assert Database(app_client.ds, is_memory=True).is_mutable is True assert Database(app_client.ds, is_memory=True, is_mutable=True).is_mutable is True assert Database(app_client.ds, is_memory=True, is_mutable=False).is_mutable is False diff --git a/tests/test_internals_datasette.py b/tests/test_internals_datasette.py index 1dc14cab..249920fe 100644 --- a/tests/test_internals_datasette.py +++ b/tests/test_internals_datasette.py @@ -58,7 +58,7 @@ async def test_datasette_constructor(): "route": "_memory", "path": None, "size": 0, - "is_mutable": False, + "is_mutable": True, "is_memory": True, "hash": None, } From 610425460b519e9c16d386cb81aa081c9d730ef0 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sat, 10 Sep 2022 14:24:26 -0700 Subject: [PATCH 147/952] Add --nolock to the README Chrome demo Refs #1744 --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 1af20129..af95b85e 100644 --- a/README.md +++ b/README.md @@ -48,7 +48,7 @@ This will start a web server on port 8001 - visit http://localhost:8001/ to acce Use Chrome on OS X? You can run datasette against your browser history like so: - datasette ~/Library/Application\ Support/Google/Chrome/Default/History + datasette ~/Library/Application\ Support/Google/Chrome/Default/History --nolock Now visiting http://localhost:8001/History/downloads will show you a web interface to browse your downloads data: From b40872f5e5ae5dad331c58f75451e2d206565196 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 14 Sep 2022 14:31:54 -0700 Subject: [PATCH 148/952] prepare_jinja2_environment(datasette) argument, refs #1809 --- datasette/app.py | 2 +- datasette/hookspecs.py | 2 +- docs/plugin_hooks.rst | 9 +++++++-- tests/plugins/my_plugin.py | 3 ++- tests/test_plugins.py | 5 +++-- 5 files changed, 14 insertions(+), 7 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index aeb81687..db686670 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -345,7 +345,7 @@ class Datasette: self.jinja_env.filters["escape_sqlite"] = escape_sqlite self.jinja_env.filters["to_css_class"] = to_css_class # pylint: disable=no-member - pm.hook.prepare_jinja2_environment(env=self.jinja_env) + pm.hook.prepare_jinja2_environment(env=self.jinja_env, datasette=self) self._register_renderers() self._permission_checks = collections.deque(maxlen=200) diff --git a/datasette/hookspecs.py b/datasette/hookspecs.py index a5fb536f..34e19664 100644 --- a/datasette/hookspecs.py +++ b/datasette/hookspecs.py @@ -26,7 +26,7 @@ def prepare_connection(conn, database, datasette): @hookspec -def prepare_jinja2_environment(env): +def prepare_jinja2_environment(env, datasette): """Modify Jinja2 template environment e.g. register custom template tags""" diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index 30bd75b7..62ec5c90 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -61,12 +61,15 @@ Examples: `datasette-jellyfish `_, for @@ -85,6 +88,8 @@ You can now use this filter in your custom templates like so:: Table name: {{ table|uppercase }} +Examples: `datasette-edit-templates `_ + .. _plugin_hook_extra_template_vars: extra_template_vars(template, database, table, columns, view_name, request, datasette) diff --git a/tests/plugins/my_plugin.py b/tests/plugins/my_plugin.py index 53613b7d..d49a7a34 100644 --- a/tests/plugins/my_plugin.py +++ b/tests/plugins/my_plugin.py @@ -142,8 +142,9 @@ def extra_template_vars( @hookimpl -def prepare_jinja2_environment(env): +def prepare_jinja2_environment(env, datasette): env.filters["format_numeric"] = lambda s: f"{float(s):,.0f}" + env.filters["to_hello"] = lambda s: datasette._HELLO @hookimpl diff --git a/tests/test_plugins.py b/tests/test_plugins.py index 948a40b8..590d88f6 100644 --- a/tests/test_plugins.py +++ b/tests/test_plugins.py @@ -545,11 +545,12 @@ def test_hook_register_output_renderer_can_render(app_client): @pytest.mark.asyncio async def test_hook_prepare_jinja2_environment(app_client): + app_client.ds._HELLO = "HI" template = app_client.ds.jinja_env.from_string( - "Hello there, {{ a|format_numeric }}", {"a": 3412341} + "Hello there, {{ a|format_numeric }}, {{ a|to_hello }}", {"a": 3412341} ) rendered = await app_client.ds.render_template(template) - assert "Hello there, 3,412,341" == rendered + assert "Hello there, 3,412,341, HI" == rendered def test_hook_publish_subcommand(): From 2ebcffe2226ece2a5a86722790d486a480338632 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 16 Sep 2022 12:50:52 -0700 Subject: [PATCH 149/952] Bump furo from 2022.6.21 to 2022.9.15 (#1812) Bumps [furo](https://github.com/pradyunsg/furo) from 2022.6.21 to 2022.9.15. - [Release notes](https://github.com/pradyunsg/furo/releases) - [Changelog](https://github.com/pradyunsg/furo/blob/main/docs/changelog.md) - [Commits](https://github.com/pradyunsg/furo/compare/2022.06.21...2022.09.15) --- updated-dependencies: - dependency-name: furo dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 92fa60d0..afcba1f0 100644 --- a/setup.py +++ b/setup.py @@ -65,7 +65,7 @@ setup( setup_requires=["pytest-runner"], extras_require={ "docs": [ - "furo==2022.6.21", + "furo==2022.9.15", "sphinx-autobuild", "codespell", "blacken-docs", From ddc999ad1296e8c69cffede3e367dda059b8adad Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 16 Sep 2022 20:38:15 -0700 Subject: [PATCH 150/952] Async support for prepare_jinja2_environment, closes #1809 --- datasette/app.py | 22 ++++++++++++++--- datasette/utils/testing.py | 1 + docs/plugin_hooks.rst | 2 ++ docs/testing_plugins.rst | 30 ++++++++++++++++++++++++ tests/fixtures.py | 1 + tests/plugins/my_plugin.py | 10 ++++++-- tests/plugins/my_plugin_2.py | 6 +++++ tests/test_internals_datasette_client.py | 6 +++-- tests/test_plugins.py | 6 +++-- tests/test_routes.py | 1 + 10 files changed, 76 insertions(+), 9 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index db686670..ea3e7b43 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -208,6 +208,7 @@ class Datasette: crossdb=False, nolock=False, ): + self._startup_invoked = False assert config_dir is None or isinstance( config_dir, Path ), "config_dir= should be a pathlib.Path" @@ -344,9 +345,6 @@ class Datasette: self.jinja_env.filters["quote_plus"] = urllib.parse.quote_plus self.jinja_env.filters["escape_sqlite"] = escape_sqlite self.jinja_env.filters["to_css_class"] = to_css_class - # pylint: disable=no-member - pm.hook.prepare_jinja2_environment(env=self.jinja_env, datasette=self) - self._register_renderers() self._permission_checks = collections.deque(maxlen=200) self._root_token = secrets.token_hex(32) @@ -389,8 +387,16 @@ class Datasette: return Urls(self) async def invoke_startup(self): + # This must be called for Datasette to be in a usable state + if self._startup_invoked: + return + for hook in pm.hook.prepare_jinja2_environment( + env=self.jinja_env, datasette=self + ): + await await_me_maybe(hook) for hook in pm.hook.startup(datasette=self): await await_me_maybe(hook) + self._startup_invoked = True def sign(self, value, namespace="default"): return URLSafeSerializer(self._secret, namespace).dumps(value) @@ -933,6 +939,8 @@ class Datasette: async def render_template( self, templates, context=None, request=None, view_name=None ): + if not self._startup_invoked: + raise Exception("render_template() called before await ds.invoke_startup()") context = context or {} if isinstance(templates, Template): template = templates @@ -1495,34 +1503,42 @@ class DatasetteClient: return path async def get(self, path, **kwargs): + await self.ds.invoke_startup() async with httpx.AsyncClient(app=self.app) as client: return await client.get(self._fix(path), **kwargs) async def options(self, path, **kwargs): + await self.ds.invoke_startup() async with httpx.AsyncClient(app=self.app) as client: return await client.options(self._fix(path), **kwargs) async def head(self, path, **kwargs): + await self.ds.invoke_startup() async with httpx.AsyncClient(app=self.app) as client: return await client.head(self._fix(path), **kwargs) async def post(self, path, **kwargs): + await self.ds.invoke_startup() async with httpx.AsyncClient(app=self.app) as client: return await client.post(self._fix(path), **kwargs) async def put(self, path, **kwargs): + await self.ds.invoke_startup() async with httpx.AsyncClient(app=self.app) as client: return await client.put(self._fix(path), **kwargs) async def patch(self, path, **kwargs): + await self.ds.invoke_startup() async with httpx.AsyncClient(app=self.app) as client: return await client.patch(self._fix(path), **kwargs) async def delete(self, path, **kwargs): + await self.ds.invoke_startup() async with httpx.AsyncClient(app=self.app) as client: return await client.delete(self._fix(path), **kwargs) async def request(self, method, path, **kwargs): + await self.ds.invoke_startup() avoid_path_rewrites = kwargs.pop("avoid_path_rewrites", None) async with httpx.AsyncClient(app=self.app) as client: return await client.request( diff --git a/datasette/utils/testing.py b/datasette/utils/testing.py index 640c94e6..b28fc575 100644 --- a/datasette/utils/testing.py +++ b/datasette/utils/testing.py @@ -147,6 +147,7 @@ class TestClient: content_type=None, if_none_match=None, ): + await self.ds.invoke_startup() headers = headers or {} if content_type: headers["content-type"] = content_type diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index 62ec5c90..f208e727 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -88,6 +88,8 @@ You can now use this filter in your custom templates like so:: Table name: {{ table|uppercase }} +This function can return an awaitable function if it needs to run any async code. + Examples: `datasette-edit-templates `_ .. _plugin_hook_extra_template_vars: diff --git a/docs/testing_plugins.rst b/docs/testing_plugins.rst index 992b4b0e..41f50e56 100644 --- a/docs/testing_plugins.rst +++ b/docs/testing_plugins.rst @@ -52,6 +52,36 @@ Then run the tests using pytest like so:: pytest +.. _testing_plugins_datasette_test_instance: + +Setting up a Datasette test instance +------------------------------------ + +The above example shows the easiest way to start writing tests against a Datasette instance: + +.. code-block:: python + + from datasette.app import Datasette + import pytest + + + @pytest.mark.asyncio + async def test_plugin_is_installed(): + datasette = Datasette(memory=True) + response = await datasette.client.get("/-/plugins.json") + assert response.status_code == 200 + +Creating a ``Datasette()`` instance like this as useful shortcut in tests, but there is one detail you need to be aware of. It's important to ensure that the async method ``.invoke_startup()`` is called on that instance. You can do that like this: + +.. code-block:: python + + datasette = Datasette(memory=True) + await datasette.invoke_startup() + +This method registers any :ref:`plugin_hook_startup` or :ref:`plugin_hook_prepare_jinja2_environment` plugins that might themselves need to make async calls. + +If you are using ``await datasette.client.get()`` and similar methods then you don't need to worry about this - those method calls ensure that ``.invoke_startup()`` has been called for you. + .. _testing_plugins_pdb: Using pdb for errors thrown inside Datasette diff --git a/tests/fixtures.py b/tests/fixtures.py index 82d8452e..5a875cd2 100644 --- a/tests/fixtures.py +++ b/tests/fixtures.py @@ -71,6 +71,7 @@ EXPECTED_PLUGINS = [ "handle_exception", "menu_links", "permission_allowed", + "prepare_jinja2_environment", "register_routes", "render_cell", "startup", diff --git a/tests/plugins/my_plugin.py b/tests/plugins/my_plugin.py index d49a7a34..1a41de38 100644 --- a/tests/plugins/my_plugin.py +++ b/tests/plugins/my_plugin.py @@ -143,8 +143,14 @@ def extra_template_vars( @hookimpl def prepare_jinja2_environment(env, datasette): - env.filters["format_numeric"] = lambda s: f"{float(s):,.0f}" - env.filters["to_hello"] = lambda s: datasette._HELLO + async def select_times_three(s): + db = datasette.get_database() + return (await db.execute("select 3 * ?", [int(s)])).first()[0] + + async def inner(): + env.filters["select_times_three"] = select_times_three + + return inner @hookimpl diff --git a/tests/plugins/my_plugin_2.py b/tests/plugins/my_plugin_2.py index 4df02343..cee80703 100644 --- a/tests/plugins/my_plugin_2.py +++ b/tests/plugins/my_plugin_2.py @@ -126,6 +126,12 @@ def permission_allowed(datasette, actor, action): return inner +@hookimpl +def prepare_jinja2_environment(env, datasette): + env.filters["format_numeric"] = lambda s: f"{float(s):,.0f}" + env.filters["to_hello"] = lambda s: datasette._HELLO + + @hookimpl def startup(datasette): async def inner(): diff --git a/tests/test_internals_datasette_client.py b/tests/test_internals_datasette_client.py index 8c5b5bd3..497bf475 100644 --- a/tests/test_internals_datasette_client.py +++ b/tests/test_internals_datasette_client.py @@ -1,10 +1,12 @@ from .fixtures import app_client import httpx import pytest +import pytest_asyncio -@pytest.fixture -def datasette(app_client): +@pytest_asyncio.fixture +async def datasette(app_client): + await app_client.ds.invoke_startup() return app_client.ds diff --git a/tests/test_plugins.py b/tests/test_plugins.py index 590d88f6..0ae3abf3 100644 --- a/tests/test_plugins.py +++ b/tests/test_plugins.py @@ -546,11 +546,13 @@ def test_hook_register_output_renderer_can_render(app_client): @pytest.mark.asyncio async def test_hook_prepare_jinja2_environment(app_client): app_client.ds._HELLO = "HI" + await app_client.ds.invoke_startup() template = app_client.ds.jinja_env.from_string( - "Hello there, {{ a|format_numeric }}, {{ a|to_hello }}", {"a": 3412341} + "Hello there, {{ a|format_numeric }}, {{ a|to_hello }}, {{ b|select_times_three }}", + {"a": 3412341, "b": 5}, ) rendered = await app_client.ds.render_template(template) - assert "Hello there, 3,412,341, HI" == rendered + assert "Hello there, 3,412,341, HI, 15" == rendered def test_hook_publish_subcommand(): diff --git a/tests/test_routes.py b/tests/test_routes.py index 5ae55d21..d467abe1 100644 --- a/tests/test_routes.py +++ b/tests/test_routes.py @@ -59,6 +59,7 @@ def test_routes(routes, path, expected_class, expected_matches): @pytest_asyncio.fixture async def ds_with_route(): ds = Datasette() + await ds.invoke_startup() ds.remove_database("_memory") db = Database(ds, is_memory=True, memory_name="route-name-db") ds.add_database(db, name="original-name", route="custom-route-name") From df851c117db031dec50dd4ef1ca34745920ac77a Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 19 Sep 2022 16:46:39 -0700 Subject: [PATCH 151/952] Validate settings.json keys on startup, closes #1816 Refs #1814 --- datasette/app.py | 4 ++++ tests/test_config_dir.py | 20 ++++++++++++++++++-- 2 files changed, 22 insertions(+), 2 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index ea3e7b43..8873ce28 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -292,6 +292,10 @@ class Datasette: raise StartupError("config.json should be renamed to settings.json") if config_dir and (config_dir / "settings.json").exists() and not settings: settings = json.loads((config_dir / "settings.json").read_text()) + # Validate those settings + for key in settings: + if key not in DEFAULT_SETTINGS: + raise StartupError("Invalid setting '{key}' in settings.json") self._settings = dict(DEFAULT_SETTINGS, **(settings or {})) self.renderers = {} # File extension -> (renderer, can_render) functions self.version_note = version_note diff --git a/tests/test_config_dir.py b/tests/test_config_dir.py index fe927c42..e365515b 100644 --- a/tests/test_config_dir.py +++ b/tests/test_config_dir.py @@ -5,6 +5,7 @@ import pytest from datasette.app import Datasette from datasette.cli import cli from datasette.utils.sqlite import sqlite3 +from datasette.utils import StartupError from .fixtures import TestClient as _TestClient from click.testing import CliRunner @@ -27,9 +28,8 @@ body { margin-top: 3em} @pytest.fixture(scope="session") -def config_dir_client(tmp_path_factory): +def config_dir(tmp_path_factory): config_dir = tmp_path_factory.mktemp("config-dir") - plugins_dir = config_dir / "plugins" plugins_dir.mkdir() (plugins_dir / "hooray.py").write_text(PLUGIN, "utf-8") @@ -77,7 +77,23 @@ def config_dir_client(tmp_path_factory): ), "utf-8", ) + return config_dir + +def test_invalid_settings(config_dir): + previous = (config_dir / "settings.json").read_text("utf-8") + (config_dir / "settings.json").write_text( + json.dumps({"invalid": "invalid-setting"}), "utf-8" + ) + try: + with pytest.raises(StartupError): + ds = Datasette([], config_dir=config_dir) + finally: + (config_dir / "settings.json").write_text(previous, "utf-8") + + +@pytest.fixture(scope="session") +def config_dir_client(config_dir): ds = Datasette([], config_dir=config_dir) yield _TestClient(ds) From cb1e093fd361b758120aefc1a444df02462389a3 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 19 Sep 2022 18:15:40 -0700 Subject: [PATCH 152/952] Fixed error message, closes #1816 --- datasette/app.py | 4 +++- tests/test_config_dir.py | 3 ++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 8873ce28..03d1dacc 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -295,7 +295,9 @@ class Datasette: # Validate those settings for key in settings: if key not in DEFAULT_SETTINGS: - raise StartupError("Invalid setting '{key}' in settings.json") + raise StartupError( + "Invalid setting '{}' in settings.json".format(key) + ) self._settings = dict(DEFAULT_SETTINGS, **(settings or {})) self.renderers = {} # File extension -> (renderer, can_render) functions self.version_note = version_note diff --git a/tests/test_config_dir.py b/tests/test_config_dir.py index e365515b..f5ecf0d6 100644 --- a/tests/test_config_dir.py +++ b/tests/test_config_dir.py @@ -86,8 +86,9 @@ def test_invalid_settings(config_dir): json.dumps({"invalid": "invalid-setting"}), "utf-8" ) try: - with pytest.raises(StartupError): + with pytest.raises(StartupError) as ex: ds = Datasette([], config_dir=config_dir) + assert ex.value.args[0] == "Invalid setting 'invalid' in settings.json" finally: (config_dir / "settings.json").write_text(previous, "utf-8") From 212137a90b4291db9605e039f198564dae59c5d0 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 26 Sep 2022 14:14:25 -0700 Subject: [PATCH 153/952] Release 0.63a0 Refs #1786, #1787, #1789, #1794, #1800, #1804, #1805, #1808, #1809, #1816 --- datasette/version.py | 2 +- docs/changelog.rst | 17 +++++++++++++++++ 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/datasette/version.py b/datasette/version.py index 0453346c..e5ad585f 100644 --- a/datasette/version.py +++ b/datasette/version.py @@ -1,2 +1,2 @@ -__version__ = "0.62" +__version__ = "0.63a0" __version_info__ = tuple(__version__.split(".")) diff --git a/docs/changelog.rst b/docs/changelog.rst index f9dcc980..bd93f4cb 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -4,6 +4,23 @@ Changelog ========= +.. _v0_63a0: + +0.63a0 (2022-09-26) +------------------- + +- The :ref:`plugin_hook_prepare_jinja2_environment` plugin hook now accepts an optional ``datasette`` argument. Hook implementations can also now return an ``async`` function which will be awaited automatically. (:issue:`1809`) +- ``--load-extension`` option now supports entrypoints. Thanks, Alex Garcia. (`#1789 `__) +- New tutorial: `Cleaning data with sqlite-utils and Datasette `__. +- Facet size can now be set per-table with the new ``facet_size`` table metadata option. (:issue:`1804`) +- ``truncate_cells_html`` setting now also affects long URLs in columns. (:issue:`1805`) +- ``Database(is_mutable=)`` now defaults to ``True``. (:issue:`1808`) +- Non-JavaScript textarea now increases height to fit the SQL query. (:issue:`1786`) +- More detailed command descriptions on the :ref:`CLI reference ` page. (:issue:`1787`) +- Datasette no longer enforces upper bounds on its depenedencies. (:issue:`1800`) +- Facets are now displayed with better line-breaks in long values. Thanks, Daniel Rech. (`#1794 `__) +- The ``settings.json`` file used in :ref:`config_dir` is now validated on startup. (:issue:`1816`) + .. _v0_62: 0.62 (2022-08-14) From 5f9f567acbc58c9fcd88af440e68034510fb5d2b Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 26 Sep 2022 16:06:01 -0700 Subject: [PATCH 154/952] Show SQL query when reporting time limit error, closes #1819 --- datasette/database.py | 5 ++++- datasette/views/base.py | 21 +++++++++++++-------- tests/test_api.py | 12 +++++++++++- tests/test_html.py | 10 +++++++--- 4 files changed, 35 insertions(+), 13 deletions(-) diff --git a/datasette/database.py b/datasette/database.py index 44467370..46094bd7 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -476,7 +476,10 @@ class WriteTask: class QueryInterrupted(Exception): - pass + def __init__(self, e, sql, params): + self.e = e + self.sql = sql + self.params = params class MultipleValues(Exception): diff --git a/datasette/views/base.py b/datasette/views/base.py index 221e1882..67aa3a42 100644 --- a/datasette/views/base.py +++ b/datasette/views/base.py @@ -1,10 +1,12 @@ import asyncio import csv import hashlib -import re import sys +import textwrap import time import urllib +from markupsafe import escape + import pint @@ -24,11 +26,9 @@ from datasette.utils import ( path_with_removed_args, path_with_format, sqlite3, - HASH_LENGTH, ) from datasette.utils.asgi import ( AsgiStream, - Forbidden, NotFound, Response, BadRequest, @@ -371,13 +371,18 @@ class DataView(BaseView): ) = response_or_template_contexts else: data, extra_template_data, templates = response_or_template_contexts - except QueryInterrupted: + except QueryInterrupted as ex: raise DatasetteError( - """ - SQL query took too long. The time limit is controlled by the + textwrap.dedent( + """ +

SQL query took too long. The time limit is controlled by the sql_time_limit_ms - configuration option. - """, + configuration option.

+ 
{}
+ """.format( + escape(ex.sql) + ) + ).strip(), title="SQL Interrupted", status=400, message_is_html=True, diff --git a/tests/test_api.py b/tests/test_api.py index 7a2bf91f..ad74d16e 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -656,7 +656,17 @@ def test_custom_sql(app_client): def test_sql_time_limit(app_client_shorter_time_limit): response = app_client_shorter_time_limit.get("/fixtures.json?sql=select+sleep(0.5)") assert 400 == response.status - assert "SQL Interrupted" == response.json["title"] + assert response.json == { + "ok": False, + "error": ( + "

SQL query took too long. The time limit is controlled by the\n" + 'sql_time_limit_ms\n' + "configuration option.

\n" + "
select sleep(0.5)
" + ), + "status": 400, + "title": "SQL Interrupted", + } def test_custom_sql_time_limit(app_client): diff --git a/tests/test_html.py b/tests/test_html.py index bf915247..a99b0b6c 100644 --- a/tests/test_html.py +++ b/tests/test_html.py @@ -168,10 +168,14 @@ def test_disallowed_custom_sql_pragma(app_client): def test_sql_time_limit(app_client_shorter_time_limit): response = app_client_shorter_time_limit.get("/fixtures?sql=select+sleep(0.5)") assert 400 == response.status - expected_html_fragment = """ + expected_html_fragments = [ + """ sql_time_limit_ms - """.strip() - assert expected_html_fragment in response.text + """.strip(), + "
select sleep(0.5)
", + ] + for expected_html_fragment in expected_html_fragments: + assert expected_html_fragment in response.text def test_row_page_does_not_truncate(): From 7fb4ea4e39a15e1f7d3202949794d98af1cfa272 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 27 Sep 2022 21:06:40 -0700 Subject: [PATCH 155/952] Update note about render_cell signature, refs #1826 --- docs/plugin_hooks.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index f208e727..c9cab8ab 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -9,7 +9,7 @@ Each plugin can implement one or more hooks using the ``@hookimpl`` decorator ag When you implement a plugin hook you can accept any or all of the parameters that are documented as being passed to that hook. -For example, you can implement the ``render_cell`` plugin hook like this even though the full documented hook signature is ``render_cell(value, column, table, database, datasette)``: +For example, you can implement the ``render_cell`` plugin hook like this even though the full documented hook signature is ``render_cell(row, value, column, table, database, datasette)``: .. code-block:: python From 984b1df12cf19a6731889fc0665bb5f622e07b7c Mon Sep 17 00:00:00 2001 From: Adam Simpson Date: Wed, 28 Sep 2022 00:21:36 -0400 Subject: [PATCH 156/952] Add documentation for serving via OpenRC (#1825) * Add documentation for serving via OpenRC --- docs/deploying.rst | 30 +++++++++++++++++++++--------- 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/docs/deploying.rst b/docs/deploying.rst index d4ad8836..c8552758 100644 --- a/docs/deploying.rst +++ b/docs/deploying.rst @@ -74,18 +74,30 @@ Once the service has started you can confirm that Datasette is running on port 8 curl 127.0.0.1:8000/-/versions.json # Should output JSON showing the installed version -Datasette will not be accessible from outside the server because it is listening on ``127.0.0.1``. You can expose it by instead listening on ``0.0.0.0``, but a better way is to set up a proxy such as ``nginx``. +Datasette will not be accessible from outside the server because it is listening on ``127.0.0.1``. You can expose it by instead listening on ``0.0.0.0``, but a better way is to set up a proxy such as ``nginx`` - see :ref:`deploying_proxy`. -Ubuntu offer `a tutorial on installing nginx `__. Once it is installed you can add configuration to proxy traffic through to Datasette that looks like this:: +.. _deploying_openrc: - server { - server_name mysubdomain.myhost.net; +Running Datasette using OpenRC +=============================== +OpenRC is the service manager on non-systemd Linux distributions like `Alpine Linux `__ and `Gentoo `__. - location / { - proxy_pass http://127.0.0.1:8000/; - proxy_set_header Host $host; - } - } +Create an init script at ``/etc/init.d/datasette`` with the following contents: + +.. code-block:: sh + + #!/sbin/openrc-run + + name="datasette" + command="datasette" + command_args="serve -h 0.0.0.0 /path/to/db.db" + command_background=true + pidfile="/run/${RC_SVCNAME}.pid" + +You then need to configure the service to run at boot and start it:: + + rc-update add datasette + rc-service datasette start .. _deploying_buildpacks: From 34defdc10aa293294ca01cfab70780755447e1d7 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 28 Sep 2022 17:39:36 -0700 Subject: [PATCH 157/952] Browse the plugins directory --- docs/writing_plugins.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/writing_plugins.rst b/docs/writing_plugins.rst index 01ee8c90..a3fc88ec 100644 --- a/docs/writing_plugins.rst +++ b/docs/writing_plugins.rst @@ -234,7 +234,7 @@ To avoid accidentally conflicting with a database file that may be loaded into D - ``/-/upload-excel`` -Try to avoid registering URLs that clash with other plugins that your users might have installed. There is no central repository of reserved URL paths (yet) but you can review existing plugins by browsing the `datasette-plugin topic `__ on GitHub. +Try to avoid registering URLs that clash with other plugins that your users might have installed. There is no central repository of reserved URL paths (yet) but you can review existing plugins by browsing the `plugins directory `. If your plugin includes functionality that relates to a specific database you could also register a URL route like this: From c92c4318e9892101f75fa158410c0a12c1d80b6e Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 30 Sep 2022 10:55:40 -0700 Subject: [PATCH 158/952] Bump furo from 2022.9.15 to 2022.9.29 (#1827) Bumps [furo](https://github.com/pradyunsg/furo) from 2022.9.15 to 2022.9.29. - [Release notes](https://github.com/pradyunsg/furo/releases) - [Changelog](https://github.com/pradyunsg/furo/blob/main/docs/changelog.md) - [Commits](https://github.com/pradyunsg/furo/compare/2022.09.15...2022.09.29) --- updated-dependencies: - dependency-name: furo dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index afcba1f0..fe258adb 100644 --- a/setup.py +++ b/setup.py @@ -65,7 +65,7 @@ setup( setup_requires=["pytest-runner"], extras_require={ "docs": [ - "furo==2022.9.15", + "furo==2022.9.29", "sphinx-autobuild", "codespell", "blacken-docs", From 883e326dd6ef95f854f7750ef2d4b0e17082fa96 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 2 Oct 2022 14:26:16 -0700 Subject: [PATCH 159/952] Drop word-wrap: anywhere, refs #1828, #1805 --- datasette/static/app.css | 1 - 1 file changed, 1 deletion(-) diff --git a/datasette/static/app.css b/datasette/static/app.css index 08b724f6..712b9925 100644 --- a/datasette/static/app.css +++ b/datasette/static/app.css @@ -446,7 +446,6 @@ th { } table a:link { text-decoration: none; - word-wrap: anywhere; } .rows-and-columns td:before { display: block; From 4218c9cd742b79b1e3cb80878e42b7e39d16ded2 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 4 Oct 2022 11:45:36 -0700 Subject: [PATCH 160/952] reST markup fix --- docs/plugin_hooks.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index c9cab8ab..832a76b0 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -268,7 +268,7 @@ you have one: def extra_js_urls(): return ["/-/static-plugins/your-plugin/app.js"] -Note that `your-plugin` here should be the hyphenated plugin name - the name that is displayed in the list on the `/-/plugins` debug page. +Note that ``your-plugin`` here should be the hyphenated plugin name - the name that is displayed in the list on the ``/-/plugins`` debug page. If your code uses `JavaScript modules `__ you should include the ``"module": True`` key. See :ref:`customization_css_and_javascript` for more details. From b6ba117b7978b58b40e3c3c2b723b92c3010ed53 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 4 Oct 2022 18:25:52 -0700 Subject: [PATCH 161/952] Clarify request or None for two hooks --- docs/plugin_hooks.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index 832a76b0..b61f953a 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -1281,7 +1281,7 @@ menu_links(datasette, actor, request) ``actor`` - dictionary or None The currently authenticated :ref:`actor `. -``request`` - :ref:`internals_request` +``request`` - :ref:`internals_request` or None The current HTTP request. This can be ``None`` if the request object is not available. This hook allows additional items to be included in the menu displayed by Datasette's top right menu icon. @@ -1330,7 +1330,7 @@ table_actions(datasette, actor, database, table, request) ``table`` - string The name of the table. -``request`` - :ref:`internals_request` +``request`` - :ref:`internals_request` or None The current HTTP request. This can be ``None`` if the request object is not available. This hook allows table actions to be displayed in a menu accessed via an action icon at the top of the table page. It should return a list of ``{"href": "...", "label": "..."}`` menu items. From bbf33a763537a1d913180b22bd3b5fe4a5e5b252 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 4 Oct 2022 21:32:11 -0700 Subject: [PATCH 162/952] Test for bool(results), closes #1832 --- tests/test_internals_database.py | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/tests/test_internals_database.py b/tests/test_internals_database.py index 9e81c1d6..4e33beed 100644 --- a/tests/test_internals_database.py +++ b/tests/test_internals_database.py @@ -30,6 +30,14 @@ async def test_results_first(db): assert isinstance(row, sqlite3.Row) +@pytest.mark.asyncio +@pytest.mark.parametrize("expected", (True, False)) +async def test_results_bool(db, expected): + where = "" if expected else "where pk = 0" + results = await db.execute("select * from facetable {}".format(where)) + assert bool(results) is expected + + @pytest.mark.parametrize( "query,expected", [ From eff112498ecc499323c26612d707908831446d25 Mon Sep 17 00:00:00 2001 From: Forest Gregg Date: Thu, 6 Oct 2022 16:06:06 -0400 Subject: [PATCH 163/952] Useuse inspect data for hash and file size on startup Thanks, @fgregg Closes #1834 --- datasette/database.py | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/datasette/database.py b/datasette/database.py index 46094bd7..d75bd70c 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -48,9 +48,13 @@ class Database: self._read_connection = None self._write_connection = None if not self.is_mutable and not self.is_memory: - p = Path(path) - self.hash = inspect_hash(p) - self.cached_size = p.stat().st_size + if self.ds.inspect_data and self.ds.inspect_data.get(self.name): + self.hash = self.ds.inspect_data[self.name]["hash"] + self.cached_size = self.ds.inspect_data[self.name]["size"] + else: + p = Path(path) + self.hash = inspect_hash(p) + self.cached_size = p.stat().st_size @property def cached_table_counts(self): From b7fec7f9020b79c1fe60cc5a2def86b50eeb5af9 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 7 Oct 2022 16:03:09 -0700 Subject: [PATCH 164/952] .sqlite/.sqlite3 extensions for config directory mode Closes #1646 --- datasette/app.py | 5 ++++- docs/settings.rst | 2 +- tests/test_config_dir.py | 11 +++++------ 3 files changed, 10 insertions(+), 8 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 03d1dacc..32a911c2 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -217,7 +217,10 @@ class Datasette: self._secret = secret or secrets.token_hex(32) self.files = tuple(files or []) + tuple(immutables or []) if config_dir: - self.files += tuple([str(p) for p in config_dir.glob("*.db")]) + db_files = [] + for ext in ("db", "sqlite", "sqlite3"): + db_files.extend(config_dir.glob("*.{}".format(ext))) + self.files += tuple(str(f) for f in db_files) if ( config_dir and (config_dir / "inspect-data.json").exists() diff --git a/docs/settings.rst b/docs/settings.rst index 8437fb04..a6d50543 100644 --- a/docs/settings.rst +++ b/docs/settings.rst @@ -46,7 +46,7 @@ Datasette will detect the files in that directory and automatically configure it The files that can be included in this directory are as follows. All are optional. -* ``*.db`` - SQLite database files that will be served by Datasette +* ``*.db`` (or ``*.sqlite3`` or ``*.sqlite``) - SQLite database files that will be served by Datasette * ``metadata.json`` - :ref:`metadata` for those databases - ``metadata.yaml`` or ``metadata.yml`` can be used as well * ``inspect-data.json`` - the result of running ``datasette inspect *.db --inspect-file=inspect-data.json`` from the configuration directory - any database files listed here will be treated as immutable, so they should not be changed while Datasette is running * ``settings.json`` - settings that would normally be passed using ``--setting`` - here they should be stored as a JSON object of key/value pairs diff --git a/tests/test_config_dir.py b/tests/test_config_dir.py index f5ecf0d6..c2af3836 100644 --- a/tests/test_config_dir.py +++ b/tests/test_config_dir.py @@ -49,7 +49,7 @@ def config_dir(tmp_path_factory): (config_dir / "metadata.json").write_text(json.dumps(METADATA), "utf-8") (config_dir / "settings.json").write_text(json.dumps(SETTINGS), "utf-8") - for dbname in ("demo.db", "immutable.db"): + for dbname in ("demo.db", "immutable.db", "j.sqlite3", "k.sqlite"): db = sqlite3.connect(str(config_dir / dbname)) db.executescript( """ @@ -151,12 +151,11 @@ def test_databases(config_dir_client): response = config_dir_client.get("/-/databases.json") assert 200 == response.status databases = response.json - assert 2 == len(databases) + assert 4 == len(databases) databases.sort(key=lambda d: d["name"]) - assert "demo" == databases[0]["name"] - assert databases[0]["is_mutable"] - assert "immutable" == databases[1]["name"] - assert not databases[1]["is_mutable"] + for db, expected_name in zip(databases, ("demo", "immutable", "j", "k")): + assert expected_name == db["name"] + assert db["is_mutable"] == (expected_name != "immutable") @pytest.mark.parametrize("filename", ("metadata.yml", "metadata.yaml")) From 1a5e5f2aa951e5bd731067a49819efba68fbe8ef Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 13 Oct 2022 14:42:52 -0700 Subject: [PATCH 165/952] Refactor breadcrumbs to respect permissions, refs #1831 --- datasette/app.py | 40 ++++++++++++++++++++++ datasette/templates/_crumbs.html | 15 ++++++++ datasette/templates/base.html | 4 +-- datasette/templates/database.html | 9 ----- datasette/templates/error.html | 7 ---- datasette/templates/logout.html | 7 ---- datasette/templates/permissions_debug.html | 7 ---- datasette/templates/query.html | 8 ++--- datasette/templates/row.html | 9 ++--- datasette/templates/show_json.html | 7 ---- datasette/templates/table.html | 8 ++--- tests/test_permissions.py | 1 + tests/test_plugins.py | 2 +- 13 files changed, 65 insertions(+), 59 deletions(-) create mode 100644 datasette/templates/_crumbs.html diff --git a/datasette/app.py b/datasette/app.py index 32a911c2..5fa4955c 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -631,6 +631,44 @@ class Datasette: else: return [] + async def _crumb_items(self, request, table=None, database=None): + crumbs = [] + # Top-level link + if await self.permission_allowed( + actor=request.actor, action="view-instance", default=True + ): + crumbs.append({"href": self.urls.instance(), "label": "home"}) + # Database link + if database: + if await self.permission_allowed( + actor=request.actor, + action="view-database", + resource=database, + default=True, + ): + crumbs.append( + { + "href": self.urls.database(database), + "label": database, + } + ) + # Table link + if table: + assert database, "table= requires database=" + if await self.permission_allowed( + actor=request.actor, + action="view-table", + resource=(database, table), + default=True, + ): + crumbs.append( + { + "href": self.urls.table(database, table), + "label": table, + } + ) + return crumbs + async def permission_allowed(self, actor, action, resource=None, default=False): """Check permissions using the permissions_allowed plugin hook""" result = None @@ -1009,6 +1047,8 @@ class Datasette: template_context = { **context, **{ + "request": request, + "crumb_items": self._crumb_items, "urls": self.urls, "actor": request.actor if request else None, "menu_links": menu_links, diff --git a/datasette/templates/_crumbs.html b/datasette/templates/_crumbs.html new file mode 100644 index 00000000..bd1ff0da --- /dev/null +++ b/datasette/templates/_crumbs.html @@ -0,0 +1,15 @@ +{% macro nav(request, database=None, table=None) -%} +{% if crumb_items is defined %} + {% set items=crumb_items(request=request, database=database, table=table) %} + {% if items %} +

+ {% for item in items %} + {{ item.label }} + {% if not loop.last %} + / + {% endif %} + {% endfor %} +

+ {% endif %} +{% endif %} +{%- endmacro %} diff --git a/datasette/templates/base.html b/datasette/templates/base.html index c3a71acb..87c939ac 100644 --- a/datasette/templates/base.html +++ b/datasette/templates/base.html @@ -1,4 +1,4 @@ - +{% import "_crumbs.html" as crumbs with context %} {% block title %}{% endblock %} @@ -17,7 +17,7 @@
-
http…http…\xa0http…this …http…
+ Content-Type: application/json + Authorization: Bearer dstok_ + { + "row": { + "column1": "value1", + "column2": "value2" + } + } + +If successful, this will return a ``201`` status code and the newly inserted row, for example: + +.. code-block:: json + + { + "row": { + "id": 1, + "column1": "value1", + "column2": "value2" + } + } From f6ca86987ba9d7d48eccf2cfe0bfc94942003844 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 27 Oct 2022 06:56:11 -0700 Subject: [PATCH 197/952] Delete mirror-master-and-main.yml Closes #1865 --- .github/workflows/mirror-master-and-main.yml | 21 -------------------- 1 file changed, 21 deletions(-) delete mode 100644 .github/workflows/mirror-master-and-main.yml diff --git a/.github/workflows/mirror-master-and-main.yml b/.github/workflows/mirror-master-and-main.yml deleted file mode 100644 index 8418df40..00000000 --- a/.github/workflows/mirror-master-and-main.yml +++ /dev/null @@ -1,21 +0,0 @@ -name: Mirror "master" and "main" branches -on: - push: - branches: - - master - - main - -jobs: - mirror: - runs-on: ubuntu-latest - steps: - - name: Mirror to "master" - uses: zofrex/mirror-branch@ea152f124954fa4eb26eea3fe0dbe313a3a08d94 - with: - target-branch: master - force: false - - name: Mirror to "main" - uses: zofrex/mirror-branch@ea152f124954fa4eb26eea3fe0dbe313a3a08d94 - with: - target-branch: main - force: false From 5f6be3c48b661f74198b8fc85361d3ad6657880e Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 27 Oct 2022 11:47:41 -0700 Subject: [PATCH 198/952] Better comment handling in SQL regex, refs #1860 --- datasette/utils/__init__.py | 9 +++++---- tests/test_utils.py | 1 + 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index 977a66d6..5acfb8b4 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -208,16 +208,16 @@ class InvalidSql(Exception): # Allow SQL to start with a /* */ or -- comment comment_re = ( # Start of string, then any amount of whitespace - r"^(\s*" + r"^\s*(" + # Comment that starts with -- and ends at a newline r"(?:\-\-.*?\n\s*)" + - # Comment that starts with /* and ends with */ - r"|(?:/\*[\s\S]*?\*/)" + # Comment that starts with /* and ends with */ - but does not have */ in it + r"|(?:\/\*((?!\*\/)[\s\S])*\*\/)" + # Whitespace - r")*\s*" + r"\s*)*\s*" ) allowed_sql_res = [ @@ -228,6 +228,7 @@ allowed_sql_res = [ re.compile(comment_re + r"explain\s+with\b"), re.compile(comment_re + r"explain\s+query\s+plan\s+with\b"), ] + allowed_pragmas = ( "database_list", "foreign_key_list", diff --git a/tests/test_utils.py b/tests/test_utils.py index e89f1e6b..c1589107 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -142,6 +142,7 @@ def test_custom_json_encoder(obj, expected): "PRAGMA case_sensitive_like = true", "SELECT * FROM pragma_not_on_allow_list('idx52')", "/* This comment is not valid. select 1", + "/**/\nupdate foo set bar = 1\n/* test */ select 1", ], ) def test_validate_sql_select_bad(bad_sql): From d2ca13b699d441a201c55cb72ff96919d3cd22bf Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 27 Oct 2022 11:50:54 -0700 Subject: [PATCH 199/952] Add test for /* multi line */ comment, refs #1860 --- tests/test_utils.py | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/test_utils.py b/tests/test_utils.py index c1589107..8b64f865 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -174,6 +174,7 @@ def test_validate_sql_select_bad(bad_sql): " /* comment */\nselect 1", " /* comment */select 1", "/* comment */\n -- another\n /* one more */ select 1", + "/* This comment \n has multiple lines */\nselect 1", ], ) def test_validate_sql_select_good(good_sql): From 918f3561208ee58c44773d30e21bace7d7c7cf3b Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 27 Oct 2022 06:56:11 -0700 Subject: [PATCH 200/952] Delete mirror-master-and-main.yml Closes #1865 --- .github/workflows/mirror-master-and-main.yml | 21 -------------------- 1 file changed, 21 deletions(-) delete mode 100644 .github/workflows/mirror-master-and-main.yml diff --git a/.github/workflows/mirror-master-and-main.yml b/.github/workflows/mirror-master-and-main.yml deleted file mode 100644 index 8418df40..00000000 --- a/.github/workflows/mirror-master-and-main.yml +++ /dev/null @@ -1,21 +0,0 @@ -name: Mirror "master" and "main" branches -on: - push: - branches: - - master - - main - -jobs: - mirror: - runs-on: ubuntu-latest - steps: - - name: Mirror to "master" - uses: zofrex/mirror-branch@ea152f124954fa4eb26eea3fe0dbe313a3a08d94 - with: - target-branch: master - force: false - - name: Mirror to "main" - uses: zofrex/mirror-branch@ea152f124954fa4eb26eea3fe0dbe313a3a08d94 - with: - target-branch: main - force: false From b597bb6b3e7c4b449654bbfa5b01ceff3eb3cb33 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 27 Oct 2022 11:47:41 -0700 Subject: [PATCH 201/952] Better comment handling in SQL regex, refs #1860 --- datasette/utils/__init__.py | 9 +++++---- tests/test_utils.py | 1 + 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index 977a66d6..5acfb8b4 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -208,16 +208,16 @@ class InvalidSql(Exception): # Allow SQL to start with a /* */ or -- comment comment_re = ( # Start of string, then any amount of whitespace - r"^(\s*" + r"^\s*(" + # Comment that starts with -- and ends at a newline r"(?:\-\-.*?\n\s*)" + - # Comment that starts with /* and ends with */ - r"|(?:/\*[\s\S]*?\*/)" + # Comment that starts with /* and ends with */ - but does not have */ in it + r"|(?:\/\*((?!\*\/)[\s\S])*\*\/)" + # Whitespace - r")*\s*" + r"\s*)*\s*" ) allowed_sql_res = [ @@ -228,6 +228,7 @@ allowed_sql_res = [ re.compile(comment_re + r"explain\s+with\b"), re.compile(comment_re + r"explain\s+query\s+plan\s+with\b"), ] + allowed_pragmas = ( "database_list", "foreign_key_list", diff --git a/tests/test_utils.py b/tests/test_utils.py index e89f1e6b..c1589107 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -142,6 +142,7 @@ def test_custom_json_encoder(obj, expected): "PRAGMA case_sensitive_like = true", "SELECT * FROM pragma_not_on_allow_list('idx52')", "/* This comment is not valid. select 1", + "/**/\nupdate foo set bar = 1\n/* test */ select 1", ], ) def test_validate_sql_select_bad(bad_sql): From 6958e21b5c2012adf5655d2512cb4106490d10f2 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 27 Oct 2022 11:50:54 -0700 Subject: [PATCH 202/952] Add test for /* multi line */ comment, refs #1860 --- tests/test_utils.py | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/test_utils.py b/tests/test_utils.py index c1589107..8b64f865 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -174,6 +174,7 @@ def test_validate_sql_select_bad(bad_sql): " /* comment */\nselect 1", " /* comment */select 1", "/* comment */\n -- another\n /* one more */ select 1", + "/* This comment \n has multiple lines */\nselect 1", ], ) def test_validate_sql_select_good(good_sql): From a51608090b5ee37593078f71d18b33767ef3af79 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 27 Oct 2022 12:06:18 -0700 Subject: [PATCH 203/952] Slight tweak to insert row API design, refs #1851 https://github.com/simonw/datasette/issues/1851#issuecomment-1292997608 --- datasette/views/table.py | 10 +++++----- docs/json_api.rst | 4 ++-- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/datasette/views/table.py b/datasette/views/table.py index 74d1c532..056b7b04 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -131,11 +131,11 @@ class TableView(DataView): # TODO: handle form-encoded data raise BadRequest("Must send JSON data") data = json.loads(await request.post_body()) - if "row" not in data: - raise BadRequest('Must send "row" data') - row = data["row"] + if "insert" not in data: + raise BadRequest('Must send a "insert" key containing a dictionary') + row = data["insert"] if not isinstance(row, dict): - raise BadRequest("row must be a dictionary") + raise BadRequest("insert must be a dictionary") # Verify all columns exist columns = await db.table_columns(table_name) pks = await db.primary_keys(table_name) @@ -165,7 +165,7 @@ class TableView(DataView): ).first() return Response.json( { - "row": dict(new_row), + "inserted_row": dict(new_row), }, status=201, ) diff --git a/docs/json_api.rst b/docs/json_api.rst index b339a738..2ed8a354 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -476,7 +476,7 @@ This requires the :ref:`permissions_insert_row` permission. Content-Type: application/json Authorization: Bearer dstok_ { - "row": { + "insert": { "column1": "value1", "column2": "value2" } @@ -487,7 +487,7 @@ If successful, this will return a ``201`` status code and the newly inserted row .. code-block:: json { - "row": { + "inserted_row": { "id": 1, "column1": "value1", "column2": "value2" From a2a5dff709c6f1676ac30b5e734c2763002562cf Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 27 Oct 2022 12:08:26 -0700 Subject: [PATCH 204/952] Missing tests for insert row API, refs #1851 --- tests/test_api_write.py | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100644 tests/test_api_write.py diff --git a/tests/test_api_write.py b/tests/test_api_write.py new file mode 100644 index 00000000..86c221d0 --- /dev/null +++ b/tests/test_api_write.py @@ -0,0 +1,38 @@ +from datasette.app import Datasette +from datasette.utils import sqlite3 +import pytest +import time + + +@pytest.fixture +def ds_write(tmp_path_factory): + db_directory = tmp_path_factory.mktemp("dbs") + db_path = str(db_directory / "data.db") + db = sqlite3.connect(str(db_path)) + db.execute("vacuum") + db.execute("create table docs (id integer primary key, title text, score float)") + ds = Datasette([db_path]) + yield ds + db.close() + + +@pytest.mark.asyncio +async def test_write_row(ds_write): + token = "dstok_{}".format( + ds_write.sign( + {"a": "root", "token": "dstok", "t": int(time.time())}, namespace="token" + ) + ) + response = await ds_write.client.post( + "/data/docs", + json={"insert": {"title": "Test", "score": 1.0}}, + headers={ + "Authorization": "Bearer {}".format(token), + "Content-Type": "application/json", + }, + ) + expected_row = {"id": 1, "title": "Test", "score": 1.0} + assert response.status_code == 201 + assert response.json()["inserted_row"] == expected_row + rows = (await ds_write.get_database("data").execute("select * from docs")).rows + assert dict(rows[0]) == expected_row From 6e788b49edf4f842c0817f006eb9d865778eea5e Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 27 Oct 2022 13:17:18 -0700 Subject: [PATCH 205/952] New URL design /db/table/-/insert, refs #1851 --- datasette/app.py | 6 +++- datasette/views/table.py | 69 +++++++++++++++++++++++++++++++++++++++- docs/json_api.rst | 18 ++++++----- tests/test_api_write.py | 6 ++-- 4 files changed, 86 insertions(+), 13 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 894d7f0f..8bc5fe36 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -39,7 +39,7 @@ from .views.special import ( PermissionsDebugView, MessagesDebugView, ) -from .views.table import TableView +from .views.table import TableView, TableInsertView from .views.row import RowView from .renderer import json_renderer from .url_builder import Urls @@ -1262,6 +1262,10 @@ class Datasette: RowView.as_view(self), r"/(?P[^\/\.]+)/(?P
[^/]+?)/(?P[^/]+?)(\.(?P\w+))?$", ) + add_route( + TableInsertView.as_view(self), + r"/(?P[^\/\.]+)/(?P
[^\/\.]+)/-/insert$", + ) return [ # Compile any strings to regular expressions ((re.compile(pattern) if isinstance(pattern, str) else pattern), view) diff --git a/datasette/views/table.py b/datasette/views/table.py index 056b7b04..be3d4f93 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -30,7 +30,7 @@ from datasette.utils import ( ) from datasette.utils.asgi import BadRequest, Forbidden, NotFound, Response from datasette.filters import Filters -from .base import DataView, DatasetteError, ureg +from .base import BaseView, DataView, DatasetteError, ureg from .database import QueryView LINK_WITH_LABEL = ( @@ -1077,3 +1077,70 @@ async def display_columns_and_rows( } columns = [first_column] + columns return columns, cell_rows + + +class TableInsertView(BaseView): + name = "table-insert" + + def __init__(self, datasette): + self.ds = datasette + + async def post(self, request): + database_route = tilde_decode(request.url_vars["database"]) + try: + db = self.ds.get_database(route=database_route) + except KeyError: + raise NotFound("Database not found: {}".format(database_route)) + database_name = db.name + table_name = tilde_decode(request.url_vars["table"]) + # Table must exist (may handle table creation in the future) + db = self.ds.get_database(database_name) + if not await db.table_exists(table_name): + raise NotFound("Table not found: {}".format(table_name)) + # Must have insert-row permission + if not await self.ds.permission_allowed( + request.actor, "insert-row", resource=(database_name, table_name) + ): + raise Forbidden("Permission denied") + if request.headers.get("content-type") != "application/json": + # TODO: handle form-encoded data + raise BadRequest("Must send JSON data") + data = json.loads(await request.post_body()) + if "row" not in data: + raise BadRequest('Must send a "row" key containing a dictionary') + row = data["row"] + if not isinstance(row, dict): + raise BadRequest("row must be a dictionary") + # Verify all columns exist + columns = await db.table_columns(table_name) + pks = await db.primary_keys(table_name) + for key in row: + if key not in columns: + raise BadRequest("Column not found: {}".format(key)) + if key in pks: + raise BadRequest( + "Cannot insert into primary key column: {}".format(key) + ) + # Perform the insert + sql = "INSERT INTO [{table}] ({columns}) VALUES ({values})".format( + table=escape_sqlite(table_name), + columns=", ".join(escape_sqlite(c) for c in row), + values=", ".join("?" for c in row), + ) + cursor = await db.execute_write(sql, list(row.values())) + # Return the new row + rowid = cursor.lastrowid + new_row = ( + await db.execute( + "SELECT * FROM [{table}] WHERE rowid = ?".format( + table=escape_sqlite(table_name) + ), + [rowid], + ) + ).first() + return Response.json( + { + "inserted": [dict(new_row)], + }, + status=201, + ) diff --git a/docs/json_api.rst b/docs/json_api.rst index 2ed8a354..4a7961f2 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -463,7 +463,7 @@ The JSON write API Datasette provides a write API for JSON data. This is a POST-only API that requires an authenticated API token, see :ref:`CreateTokenView`. -.. _json_api_write_insert_row: +.. _TableInsertView: Inserting a single row ~~~~~~~~~~~~~~~~~~~~~~ @@ -472,11 +472,11 @@ This requires the :ref:`permissions_insert_row` permission. :: - POST //
+ POST //
/-/insert Content-Type: application/json Authorization: Bearer dstok_ { - "insert": { + "row": { "column1": "value1", "column2": "value2" } @@ -487,9 +487,11 @@ If successful, this will return a ``201`` status code and the newly inserted row .. code-block:: json { - "inserted_row": { - "id": 1, - "column1": "value1", - "column2": "value2" - } + "inserted": [ + { + "id": 1, + "column1": "value1", + "column2": "value2" + } + ] } diff --git a/tests/test_api_write.py b/tests/test_api_write.py index 86c221d0..e8222e43 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -24,8 +24,8 @@ async def test_write_row(ds_write): ) ) response = await ds_write.client.post( - "/data/docs", - json={"insert": {"title": "Test", "score": 1.0}}, + "/data/docs/-/insert", + json={"row": {"title": "Test", "score": 1.0}}, headers={ "Authorization": "Bearer {}".format(token), "Content-Type": "application/json", @@ -33,6 +33,6 @@ async def test_write_row(ds_write): ) expected_row = {"id": 1, "title": "Test", "score": 1.0} assert response.status_code == 201 - assert response.json()["inserted_row"] == expected_row + assert response.json()["inserted"] == [expected_row] rows = (await ds_write.get_database("data").execute("select * from docs")).rows assert dict(rows[0]) == expected_row From b912d92b651c4f0b5137da924d135654511f0fe0 Mon Sep 17 00:00:00 2001 From: Forest Gregg Date: Thu, 27 Oct 2022 16:51:20 -0400 Subject: [PATCH 206/952] Make hash and size a lazy property (#1837) * use inspect data for hash and file size * make hash and cached_size lazy properties * move hash property near size --- datasette/database.py | 36 ++++++++++++++++++++++++------------ 1 file changed, 24 insertions(+), 12 deletions(-) diff --git a/datasette/database.py b/datasette/database.py index d75bd70c..af1df0a8 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -39,7 +39,7 @@ class Database: self.memory_name = memory_name if memory_name is not None: self.is_memory = True - self.hash = None + self.cached_hash = None self.cached_size = None self._cached_table_counts = None self._write_thread = None @@ -47,14 +47,6 @@ class Database: # These are used when in non-threaded mode: self._read_connection = None self._write_connection = None - if not self.is_mutable and not self.is_memory: - if self.ds.inspect_data and self.ds.inspect_data.get(self.name): - self.hash = self.ds.inspect_data[self.name]["hash"] - self.cached_size = self.ds.inspect_data[self.name]["size"] - else: - p = Path(path) - self.hash = inspect_hash(p) - self.cached_size = p.stat().st_size @property def cached_table_counts(self): @@ -266,14 +258,34 @@ class Database: results = await self.execute_fn(sql_operation_in_thread) return results + @property + def hash(self): + if self.cached_hash is not None: + return self.cached_hash + elif self.is_mutable or self.is_memory: + return None + elif self.ds.inspect_data and self.ds.inspect_data.get(self.name): + self.cached_hash = self.ds.inspect_data[self.name]["hash"] + return self.cached_hash + else: + p = Path(self.path) + self.cached_hash = inspect_hash(p) + return self.cached_hash + @property def size(self): - if self.is_memory: - return 0 if self.cached_size is not None: return self.cached_size - else: + elif self.is_memory: + return 0 + elif self.is_mutable: return Path(self.path).stat().st_size + elif self.ds.inspect_data and self.ds.inspect_data.get(self.name): + self.cached_size = self.ds.inspect_data[self.name]["size"] + return self.cached_size + else: + self.cached_size = Path(self.path).stat().st_size + return self.cached_size async def table_counts(self, limit=10): if not self.is_mutable and self.cached_table_counts is not None: From 2c36e45447494cd7505440943367e29ec57c8e72 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 27 Oct 2022 13:51:45 -0700 Subject: [PATCH 207/952] Bump black from 22.8.0 to 22.10.0 (#1839) Bumps [black](https://github.com/psf/black) from 22.8.0 to 22.10.0. - [Release notes](https://github.com/psf/black/releases) - [Changelog](https://github.com/psf/black/blob/main/CHANGES.md) - [Commits](https://github.com/psf/black/compare/22.8.0...22.10.0) --- updated-dependencies: - dependency-name: black dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index fe258adb..625557ae 100644 --- a/setup.py +++ b/setup.py @@ -76,7 +76,7 @@ setup( "pytest-xdist>=2.2.1", "pytest-asyncio>=0.17", "beautifulsoup4>=4.8.1", - "black==22.8.0", + "black==22.10.0", "blacken-docs==1.12.1", "pytest-timeout>=1.4.2", "trustme>=0.7", From e5e0459a0b60608cb5e9ff83f6b41f59e6cafdfd Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 27 Oct 2022 13:58:00 -0700 Subject: [PATCH 208/952] Release notes for 0.63, refs #1869 --- docs/changelog.rst | 44 +++++++++++++++++++++++++------------------- 1 file changed, 25 insertions(+), 19 deletions(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index 2255dcce..01957e4f 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -4,36 +4,42 @@ Changelog ========= -.. _v0_63a1: +.. _v0_63: -0.63a1 (2022-10-23) -------------------- +0.63 (2022-10-27) +----------------- +Features +~~~~~~~~ + +- Now tested against Python 3.11. Docker containers used by ``datasette publish`` and ``datasette package`` both now use that version of Python. (:issue:`1853`) +- ``--load-extension`` option now supports entrypoints. Thanks, Alex Garcia. (`#1789 `__) +- Facet size can now be set per-table with the new ``facet_size`` table metadata option. (:issue:`1804`) +- The :ref:`setting_truncate_cells_html` setting now also affects long URLs in columns. (:issue:`1805`) +- The non-JavaScript SQL editor textarea now increases height to fit the SQL query. (:issue:`1786`) +- Facets are now displayed with better line-breaks in long values. Thanks, Daniel Rech. (`#1794 `__) +- The ``settings.json`` file used in :ref:`config_dir` is now validated on startup. (:issue:`1816`) +- SQL queries can now include leading SQL comments, using ``/* ... */`` or ``-- ...`` syntax. Thanks, Charles Nepote. (:issue:`1860`) - SQL query is now re-displayed when terminated with a time limit error. (:issue:`1819`) -- New documentation on :ref:`deploying_openrc` - thanks, Adam Simpson. (`#1825 `__) - The :ref:`inspect data ` mechanism is now used to speed up server startup - thanks, Forest Gregg. (:issue:`1834`) - In :ref:`config_dir` databases with filenames ending in ``.sqlite`` or ``.sqlite3`` are now automatically added to the Datasette instance. (:issue:`1646`) - Breadcrumb navigation display now respects the current user's permissions. (:issue:`1831`) -- Screenshots in the documentation are now maintained using `shot-scraper `__, as described in `Automating screenshots for the Datasette documentation using shot-scraper `__. (:issue:`1844`) -- The :ref:`datasette.check_visibility() ` method now accepts an optional ``permissions=`` list, allowing it to take multiple permissions into account at once when deciding if something should be shown as public or private. This has been used to correctly display padlock icons in more places in the Datasette interface. (:issue:`1829`) - -.. _v0_63a0: - -0.63a0 (2022-09-26) -------------------- +Plugin hooks and internals +~~~~~~~~~~~~~~~~~~~~~~~~~~ - The :ref:`plugin_hook_prepare_jinja2_environment` plugin hook now accepts an optional ``datasette`` argument. Hook implementations can also now return an ``async`` function which will be awaited automatically. (:issue:`1809`) -- ``--load-extension`` option now supports entrypoints. Thanks, Alex Garcia. (`#1789 `__) -- New tutorial: `Cleaning data with sqlite-utils and Datasette `__. -- Facet size can now be set per-table with the new ``facet_size`` table metadata option. (:issue:`1804`) -- ``truncate_cells_html`` setting now also affects long URLs in columns. (:issue:`1805`) - ``Database(is_mutable=)`` now defaults to ``True``. (:issue:`1808`) -- Non-JavaScript textarea now increases height to fit the SQL query. (:issue:`1786`) -- More detailed command descriptions on the :ref:`CLI reference ` page. (:issue:`1787`) +- The :ref:`datasette.check_visibility() ` method now accepts an optional ``permissions=`` list, allowing it to take multiple permissions into account at once when deciding if something should be shown as public or private. This has been used to correctly display padlock icons in more places in the Datasette interface. (:issue:`1829`) - Datasette no longer enforces upper bounds on its dependencies. (:issue:`1800`) -- Facets are now displayed with better line-breaks in long values. Thanks, Daniel Rech. (`#1794 `__) -- The ``settings.json`` file used in :ref:`config_dir` is now validated on startup. (:issue:`1816`) + +Documentation +~~~~~~~~~~~~~ + +- New tutorial: `Cleaning data with sqlite-utils and Datasette `__. +- Screenshots in the documentation are now maintained using `shot-scraper `__, as described in `Automating screenshots for the Datasette documentation using shot-scraper `__. (:issue:`1844`) +- More detailed command descriptions on the :ref:`CLI reference ` page. (:issue:`1787`) +- New documentation on :ref:`deploying_openrc` - thanks, Adam Simpson. (`#1825 `__) .. _v0_62: From bf00b0b59b6692bdec597ac9db4e0b497c5a47b4 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 27 Oct 2022 15:11:26 -0700 Subject: [PATCH 209/952] Release 0.63 Refs #1646, #1786, #1787, #1789, #1794, #1800, #1804, #1805, #1808, #1809, #1816, #1819, #1825, #1829, #1831, #1834, #1844, #1853, #1860 Closes #1869 --- datasette/version.py | 2 +- docs/changelog.rst | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/datasette/version.py b/datasette/version.py index eb36da45..ac012640 100644 --- a/datasette/version.py +++ b/datasette/version.py @@ -1,2 +1,2 @@ -__version__ = "0.63a1" +__version__ = "0.63" __version_info__ = tuple(__version__.split(".")) diff --git a/docs/changelog.rst b/docs/changelog.rst index 01957e4f..f573afb3 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -9,6 +9,8 @@ Changelog 0.63 (2022-10-27) ----------------- +See `Datasette 0.63: The annotated release notes `__ for more background on the changes in this release. + Features ~~~~~~~~ From 2ea60e12d90b7cec03ebab728854d3ec4d553f54 Mon Sep 17 00:00:00 2001 From: Forest Gregg Date: Thu, 27 Oct 2022 16:51:20 -0400 Subject: [PATCH 210/952] Make hash and size a lazy property (#1837) * use inspect data for hash and file size * make hash and cached_size lazy properties * move hash property near size --- datasette/database.py | 36 ++++++++++++++++++++++++------------ 1 file changed, 24 insertions(+), 12 deletions(-) diff --git a/datasette/database.py b/datasette/database.py index d75bd70c..af1df0a8 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -39,7 +39,7 @@ class Database: self.memory_name = memory_name if memory_name is not None: self.is_memory = True - self.hash = None + self.cached_hash = None self.cached_size = None self._cached_table_counts = None self._write_thread = None @@ -47,14 +47,6 @@ class Database: # These are used when in non-threaded mode: self._read_connection = None self._write_connection = None - if not self.is_mutable and not self.is_memory: - if self.ds.inspect_data and self.ds.inspect_data.get(self.name): - self.hash = self.ds.inspect_data[self.name]["hash"] - self.cached_size = self.ds.inspect_data[self.name]["size"] - else: - p = Path(path) - self.hash = inspect_hash(p) - self.cached_size = p.stat().st_size @property def cached_table_counts(self): @@ -266,14 +258,34 @@ class Database: results = await self.execute_fn(sql_operation_in_thread) return results + @property + def hash(self): + if self.cached_hash is not None: + return self.cached_hash + elif self.is_mutable or self.is_memory: + return None + elif self.ds.inspect_data and self.ds.inspect_data.get(self.name): + self.cached_hash = self.ds.inspect_data[self.name]["hash"] + return self.cached_hash + else: + p = Path(self.path) + self.cached_hash = inspect_hash(p) + return self.cached_hash + @property def size(self): - if self.is_memory: - return 0 if self.cached_size is not None: return self.cached_size - else: + elif self.is_memory: + return 0 + elif self.is_mutable: return Path(self.path).stat().st_size + elif self.ds.inspect_data and self.ds.inspect_data.get(self.name): + self.cached_size = self.ds.inspect_data[self.name]["size"] + return self.cached_size + else: + self.cached_size = Path(self.path).stat().st_size + return self.cached_size async def table_counts(self, limit=10): if not self.is_mutable and self.cached_table_counts is not None: From 641bc4453b5ef1dff0b2fc7dfad0b692be7aa61c Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 27 Oct 2022 13:51:45 -0700 Subject: [PATCH 211/952] Bump black from 22.8.0 to 22.10.0 (#1839) Bumps [black](https://github.com/psf/black) from 22.8.0 to 22.10.0. - [Release notes](https://github.com/psf/black/releases) - [Changelog](https://github.com/psf/black/blob/main/CHANGES.md) - [Commits](https://github.com/psf/black/compare/22.8.0...22.10.0) --- updated-dependencies: - dependency-name: black dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index fe258adb..625557ae 100644 --- a/setup.py +++ b/setup.py @@ -76,7 +76,7 @@ setup( "pytest-xdist>=2.2.1", "pytest-asyncio>=0.17", "beautifulsoup4>=4.8.1", - "black==22.8.0", + "black==22.10.0", "blacken-docs==1.12.1", "pytest-timeout>=1.4.2", "trustme>=0.7", From 26af9b9c4a6c62ee15870caa1c7bc455165d3b11 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 27 Oct 2022 13:58:00 -0700 Subject: [PATCH 212/952] Release notes for 0.63, refs #1869 --- docs/changelog.rst | 44 +++++++++++++++++++++++++------------------- 1 file changed, 25 insertions(+), 19 deletions(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index 2255dcce..01957e4f 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -4,36 +4,42 @@ Changelog ========= -.. _v0_63a1: +.. _v0_63: -0.63a1 (2022-10-23) -------------------- +0.63 (2022-10-27) +----------------- +Features +~~~~~~~~ + +- Now tested against Python 3.11. Docker containers used by ``datasette publish`` and ``datasette package`` both now use that version of Python. (:issue:`1853`) +- ``--load-extension`` option now supports entrypoints. Thanks, Alex Garcia. (`#1789 `__) +- Facet size can now be set per-table with the new ``facet_size`` table metadata option. (:issue:`1804`) +- The :ref:`setting_truncate_cells_html` setting now also affects long URLs in columns. (:issue:`1805`) +- The non-JavaScript SQL editor textarea now increases height to fit the SQL query. (:issue:`1786`) +- Facets are now displayed with better line-breaks in long values. Thanks, Daniel Rech. (`#1794 `__) +- The ``settings.json`` file used in :ref:`config_dir` is now validated on startup. (:issue:`1816`) +- SQL queries can now include leading SQL comments, using ``/* ... */`` or ``-- ...`` syntax. Thanks, Charles Nepote. (:issue:`1860`) - SQL query is now re-displayed when terminated with a time limit error. (:issue:`1819`) -- New documentation on :ref:`deploying_openrc` - thanks, Adam Simpson. (`#1825 `__) - The :ref:`inspect data ` mechanism is now used to speed up server startup - thanks, Forest Gregg. (:issue:`1834`) - In :ref:`config_dir` databases with filenames ending in ``.sqlite`` or ``.sqlite3`` are now automatically added to the Datasette instance. (:issue:`1646`) - Breadcrumb navigation display now respects the current user's permissions. (:issue:`1831`) -- Screenshots in the documentation are now maintained using `shot-scraper `__, as described in `Automating screenshots for the Datasette documentation using shot-scraper `__. (:issue:`1844`) -- The :ref:`datasette.check_visibility() ` method now accepts an optional ``permissions=`` list, allowing it to take multiple permissions into account at once when deciding if something should be shown as public or private. This has been used to correctly display padlock icons in more places in the Datasette interface. (:issue:`1829`) - -.. _v0_63a0: - -0.63a0 (2022-09-26) -------------------- +Plugin hooks and internals +~~~~~~~~~~~~~~~~~~~~~~~~~~ - The :ref:`plugin_hook_prepare_jinja2_environment` plugin hook now accepts an optional ``datasette`` argument. Hook implementations can also now return an ``async`` function which will be awaited automatically. (:issue:`1809`) -- ``--load-extension`` option now supports entrypoints. Thanks, Alex Garcia. (`#1789 `__) -- New tutorial: `Cleaning data with sqlite-utils and Datasette `__. -- Facet size can now be set per-table with the new ``facet_size`` table metadata option. (:issue:`1804`) -- ``truncate_cells_html`` setting now also affects long URLs in columns. (:issue:`1805`) - ``Database(is_mutable=)`` now defaults to ``True``. (:issue:`1808`) -- Non-JavaScript textarea now increases height to fit the SQL query. (:issue:`1786`) -- More detailed command descriptions on the :ref:`CLI reference ` page. (:issue:`1787`) +- The :ref:`datasette.check_visibility() ` method now accepts an optional ``permissions=`` list, allowing it to take multiple permissions into account at once when deciding if something should be shown as public or private. This has been used to correctly display padlock icons in more places in the Datasette interface. (:issue:`1829`) - Datasette no longer enforces upper bounds on its dependencies. (:issue:`1800`) -- Facets are now displayed with better line-breaks in long values. Thanks, Daniel Rech. (`#1794 `__) -- The ``settings.json`` file used in :ref:`config_dir` is now validated on startup. (:issue:`1816`) + +Documentation +~~~~~~~~~~~~~ + +- New tutorial: `Cleaning data with sqlite-utils and Datasette `__. +- Screenshots in the documentation are now maintained using `shot-scraper `__, as described in `Automating screenshots for the Datasette documentation using shot-scraper `__. (:issue:`1844`) +- More detailed command descriptions on the :ref:`CLI reference ` page. (:issue:`1787`) +- New documentation on :ref:`deploying_openrc` - thanks, Adam Simpson. (`#1825 `__) .. _v0_62: From 61171f01549549e5fb25c72b13280d941d96dbf1 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 27 Oct 2022 15:11:26 -0700 Subject: [PATCH 213/952] Release 0.63 Refs #1646, #1786, #1787, #1789, #1794, #1800, #1804, #1805, #1808, #1809, #1816, #1819, #1825, #1829, #1831, #1834, #1844, #1853, #1860 Closes #1869 --- datasette/version.py | 2 +- docs/changelog.rst | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/datasette/version.py b/datasette/version.py index eb36da45..ac012640 100644 --- a/datasette/version.py +++ b/datasette/version.py @@ -1,2 +1,2 @@ -__version__ = "0.63a1" +__version__ = "0.63" __version_info__ = tuple(__version__.split(".")) diff --git a/docs/changelog.rst b/docs/changelog.rst index 01957e4f..f573afb3 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -9,6 +9,8 @@ Changelog 0.63 (2022-10-27) ----------------- +See `Datasette 0.63: The annotated release notes `__ for more background on the changes in this release. + Features ~~~~~~~~ From c9b5f5d598e7f85cd3e1ce020351a27da334408b Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 27 Oct 2022 17:58:36 -0700 Subject: [PATCH 214/952] Depend on sqlite-utils>=3.30 Decided to use the most recent version in case I decide later to use the flatten() utility function. Refs #1850 --- setup.py | 1 + 1 file changed, 1 insertion(+) diff --git a/setup.py b/setup.py index 625557ae..99e2a4ad 100644 --- a/setup.py +++ b/setup.py @@ -57,6 +57,7 @@ setup( "PyYAML>=5.3", "mergedeep>=1.1.1", "itsdangerous>=1.1", + "sqlite-utils>=3.30", ], entry_points=""" [console_scripts] From c35859ae3df163406f1a1895ccf9803e933b2d8e Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sat, 29 Oct 2022 23:03:45 -0700 Subject: [PATCH 215/952] API for bulk inserts, closes #1866 --- datasette/app.py | 5 ++ datasette/views/table.py | 136 +++++++++++++++++++++---------- docs/cli-reference.rst | 2 + docs/json_api.rst | 48 ++++++++++- docs/settings.rst | 11 +++ tests/test_api.py | 1 + tests/test_api_write.py | 168 +++++++++++++++++++++++++++++++++++++-- 7 files changed, 320 insertions(+), 51 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 8bc5fe36..f80d3792 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -99,6 +99,11 @@ SETTINGS = ( 1000, "Maximum rows that can be returned from a table or custom query", ), + Setting( + "max_insert_rows", + 100, + "Maximum rows that can be inserted at a time using the bulk insert API", + ), Setting( "num_sql_threads", 3, diff --git a/datasette/views/table.py b/datasette/views/table.py index be3d4f93..fd203036 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -30,6 +30,7 @@ from datasette.utils import ( ) from datasette.utils.asgi import BadRequest, Forbidden, NotFound, Response from datasette.filters import Filters +import sqlite_utils from .base import BaseView, DataView, DatasetteError, ureg from .database import QueryView @@ -1085,62 +1086,109 @@ class TableInsertView(BaseView): def __init__(self, datasette): self.ds = datasette + async def _validate_data(self, request, db, table_name): + errors = [] + + def _errors(errors): + return None, errors, {} + + if request.headers.get("content-type") != "application/json": + # TODO: handle form-encoded data + return _errors(["Invalid content-type, must be application/json"]) + body = await request.post_body() + try: + data = json.loads(body) + except json.JSONDecodeError as e: + return _errors(["Invalid JSON: {}".format(e)]) + if not isinstance(data, dict): + return _errors(["JSON must be a dictionary"]) + keys = data.keys() + # keys must contain "row" or "rows" + if "row" not in keys and "rows" not in keys: + return _errors(['JSON must have one or other of "row" or "rows"']) + rows = [] + if "row" in keys: + if "rows" in keys: + return _errors(['Cannot use "row" and "rows" at the same time']) + row = data["row"] + if not isinstance(row, dict): + return _errors(['"row" must be a dictionary']) + rows = [row] + data["return_rows"] = True + else: + rows = data["rows"] + if not isinstance(rows, list): + return _errors(['"rows" must be a list']) + for row in rows: + if not isinstance(row, dict): + return _errors(['"rows" must be a list of dictionaries']) + # Does this exceed max_insert_rows? + max_insert_rows = self.ds.setting("max_insert_rows") + if len(rows) > max_insert_rows: + return _errors( + ["Too many rows, maximum allowed is {}".format(max_insert_rows)] + ) + # Validate columns of each row + columns = await db.table_columns(table_name) + # TODO: There are cases where pks are OK, if not using auto-incrementing pk + pks = await db.primary_keys(table_name) + allowed_columns = set(columns) - set(pks) + for i, row in enumerate(rows): + invalid_columns = set(row.keys()) - allowed_columns + if invalid_columns: + errors.append( + "Row {} has invalid columns: {}".format( + i, ", ".join(sorted(invalid_columns)) + ) + ) + if errors: + return _errors(errors) + extra = {key: data[key] for key in data if key not in ("rows", "row")} + return rows, errors, extra + async def post(self, request): + def _error(messages, status=400): + return Response.json({"ok": False, "errors": messages}, status=status) + database_route = tilde_decode(request.url_vars["database"]) try: db = self.ds.get_database(route=database_route) except KeyError: - raise NotFound("Database not found: {}".format(database_route)) + return _error(["Database not found: {}".format(database_route)], 404) database_name = db.name table_name = tilde_decode(request.url_vars["table"]) + # Table must exist (may handle table creation in the future) db = self.ds.get_database(database_name) if not await db.table_exists(table_name): - raise NotFound("Table not found: {}".format(table_name)) + return _error(["Table not found: {}".format(table_name)], 404) # Must have insert-row permission if not await self.ds.permission_allowed( request.actor, "insert-row", resource=(database_name, table_name) ): - raise Forbidden("Permission denied") - if request.headers.get("content-type") != "application/json": - # TODO: handle form-encoded data - raise BadRequest("Must send JSON data") - data = json.loads(await request.post_body()) - if "row" not in data: - raise BadRequest('Must send a "row" key containing a dictionary') - row = data["row"] - if not isinstance(row, dict): - raise BadRequest("row must be a dictionary") - # Verify all columns exist - columns = await db.table_columns(table_name) - pks = await db.primary_keys(table_name) - for key in row: - if key not in columns: - raise BadRequest("Column not found: {}".format(key)) - if key in pks: - raise BadRequest( - "Cannot insert into primary key column: {}".format(key) + return _error(["Permission denied"], 403) + rows, errors, extra = await self._validate_data(request, db, table_name) + if errors: + return _error(errors, 400) + + should_return = bool(extra.get("return_rows", False)) + # Insert rows + def insert_rows(conn): + table = sqlite_utils.Database(conn)[table_name] + if should_return: + rowids = [] + for row in rows: + rowids.append(table.insert(row).last_rowid) + return list( + table.rows_where( + "rowid in ({})".format(",".join("?" for _ in rowids)), rowids + ) ) - # Perform the insert - sql = "INSERT INTO [{table}] ({columns}) VALUES ({values})".format( - table=escape_sqlite(table_name), - columns=", ".join(escape_sqlite(c) for c in row), - values=", ".join("?" for c in row), - ) - cursor = await db.execute_write(sql, list(row.values())) - # Return the new row - rowid = cursor.lastrowid - new_row = ( - await db.execute( - "SELECT * FROM [{table}] WHERE rowid = ?".format( - table=escape_sqlite(table_name) - ), - [rowid], - ) - ).first() - return Response.json( - { - "inserted": [dict(new_row)], - }, - status=201, - ) + else: + table.insert_all(rows) + + rows = await db.execute_write_fn(insert_rows) + result = {"ok": True} + if should_return: + result["inserted"] = rows + return Response.json(result, status=201) diff --git a/docs/cli-reference.rst b/docs/cli-reference.rst index 56156568..649a3dcd 100644 --- a/docs/cli-reference.rst +++ b/docs/cli-reference.rst @@ -213,6 +213,8 @@ These can be passed to ``datasette serve`` using ``datasette serve --setting nam (default=100) max_returned_rows Maximum rows that can be returned from a table or custom query (default=1000) + max_insert_rows Maximum rows that can be inserted at a time using + the bulk insert API (default=1000) num_sql_threads Number of threads in the thread pool for executing SQLite queries (default=3) sql_time_limit_ms Time limit for a SQL query in milliseconds diff --git a/docs/json_api.rst b/docs/json_api.rst index 4a7961f2..01558c23 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -465,11 +465,13 @@ Datasette provides a write API for JSON data. This is a POST-only API that requi .. _TableInsertView: -Inserting a single row -~~~~~~~~~~~~~~~~~~~~~~ +Inserting rows +~~~~~~~~~~~~~~ This requires the :ref:`permissions_insert_row` permission. +A single row can be inserted using the ``"row"`` key: + :: POST //
/-/insert @@ -495,3 +497,45 @@ If successful, this will return a ``201`` status code and the newly inserted row } ] } + +To insert multiple rows at a time, use the same API method but send a list of dictionaries as the ``"rows"`` key: + +:: + + POST //
/-/insert + Content-Type: application/json + Authorization: Bearer dstok_ + { + "rows": [ + { + "column1": "value1", + "column2": "value2" + }, + { + "column1": "value3", + "column2": "value4" + } + ] + } + +If successful, this will return a ``201`` status code and an empty ``{}`` response body. + +To return the newly inserted rows, add the ``"return_rows": true`` key to the request body: + +.. code-block:: json + + { + "rows": [ + { + "column1": "value1", + "column2": "value2" + }, + { + "column1": "value3", + "column2": "value4" + } + ], + "return_rows": true + } + +This will return the same ``"inserted"`` key as the single row example above. There is a small performance penalty for using this option. diff --git a/docs/settings.rst b/docs/settings.rst index a990c78c..b86b18bd 100644 --- a/docs/settings.rst +++ b/docs/settings.rst @@ -96,6 +96,17 @@ You can increase or decrease this limit like so:: datasette mydatabase.db --setting max_returned_rows 2000 +.. _setting_max_insert_rows: + +max_insert_rows +~~~~~~~~~~~~~~~ + +Maximum rows that can be inserted at a time using the bulk insert API, see :ref:`TableInsertView`. Defaults to 100. + +You can increase or decrease this limit like so:: + + datasette mydatabase.db --setting max_insert_rows 1000 + .. _setting_num_sql_threads: num_sql_threads diff --git a/tests/test_api.py b/tests/test_api.py index fc171421..ebd675b9 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -804,6 +804,7 @@ def test_settings_json(app_client): "facet_suggest_time_limit_ms": 50, "facet_time_limit_ms": 200, "max_returned_rows": 100, + "max_insert_rows": 100, "sql_time_limit_ms": 200, "allow_download": True, "allow_signed_tokens": True, diff --git a/tests/test_api_write.py b/tests/test_api_write.py index e8222e43..4a5a58aa 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -18,11 +18,7 @@ def ds_write(tmp_path_factory): @pytest.mark.asyncio async def test_write_row(ds_write): - token = "dstok_{}".format( - ds_write.sign( - {"a": "root", "token": "dstok", "t": int(time.time())}, namespace="token" - ) - ) + token = write_token(ds_write) response = await ds_write.client.post( "/data/docs/-/insert", json={"row": {"title": "Test", "score": 1.0}}, @@ -36,3 +32,165 @@ async def test_write_row(ds_write): assert response.json()["inserted"] == [expected_row] rows = (await ds_write.get_database("data").execute("select * from docs")).rows assert dict(rows[0]) == expected_row + + +@pytest.mark.asyncio +@pytest.mark.parametrize("return_rows", (True, False)) +async def test_write_rows(ds_write, return_rows): + token = write_token(ds_write) + data = {"rows": [{"title": "Test {}".format(i), "score": 1.0} for i in range(20)]} + if return_rows: + data["return_rows"] = True + response = await ds_write.client.post( + "/data/docs/-/insert", + json=data, + headers={ + "Authorization": "Bearer {}".format(token), + "Content-Type": "application/json", + }, + ) + assert response.status_code == 201 + actual_rows = [ + dict(r) + for r in ( + await ds_write.get_database("data").execute("select * from docs") + ).rows + ] + assert len(actual_rows) == 20 + assert actual_rows == [ + {"id": i + 1, "title": "Test {}".format(i), "score": 1.0} for i in range(20) + ] + assert response.json()["ok"] is True + if return_rows: + assert response.json()["inserted"] == actual_rows + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "path,input,special_case,expected_status,expected_errors", + ( + ( + "/data2/docs/-/insert", + {}, + None, + 404, + ["Database not found: data2"], + ), + ( + "/data/docs2/-/insert", + {}, + None, + 404, + ["Table not found: docs2"], + ), + ( + "/data/docs/-/insert", + {"rows": [{"title": "Test"} for i in range(10)]}, + "bad_token", + 403, + ["Permission denied"], + ), + ( + "/data/docs/-/insert", + {}, + "invalid_json", + 400, + [ + "Invalid JSON: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)" + ], + ), + ( + "/data/docs/-/insert", + {}, + "invalid_content_type", + 400, + ["Invalid content-type, must be application/json"], + ), + ( + "/data/docs/-/insert", + [], + None, + 400, + ["JSON must be a dictionary"], + ), + ( + "/data/docs/-/insert", + {"row": "blah"}, + None, + 400, + ['"row" must be a dictionary'], + ), + ( + "/data/docs/-/insert", + {"blah": "blah"}, + None, + 400, + ['JSON must have one or other of "row" or "rows"'], + ), + ( + "/data/docs/-/insert", + {"rows": "blah"}, + None, + 400, + ['"rows" must be a list'], + ), + ( + "/data/docs/-/insert", + {"rows": ["blah"]}, + None, + 400, + ['"rows" must be a list of dictionaries'], + ), + ( + "/data/docs/-/insert", + {"rows": [{"title": "Test"} for i in range(101)]}, + None, + 400, + ["Too many rows, maximum allowed is 100"], + ), + # Validate columns of each row + ( + "/data/docs/-/insert", + {"rows": [{"title": "Test", "bad": 1, "worse": 2} for i in range(2)]}, + None, + 400, + [ + "Row 0 has invalid columns: bad, worse", + "Row 1 has invalid columns: bad, worse", + ], + ), + ), +) +async def test_write_row_errors( + ds_write, path, input, special_case, expected_status, expected_errors +): + token = write_token(ds_write) + if special_case == "bad_token": + token += "bad" + kwargs = dict( + json=input, + headers={ + "Authorization": "Bearer {}".format(token), + "Content-Type": "text/plain" + if special_case == "invalid_content_type" + else "application/json", + }, + ) + if special_case == "invalid_json": + del kwargs["json"] + kwargs["content"] = "{bad json" + response = await ds_write.client.post( + path, + **kwargs, + ) + assert response.status_code == expected_status + assert response.json()["ok"] is False + assert response.json()["errors"] == expected_errors + + +def write_token(ds): + return "dstok_{}".format( + ds.sign( + {"a": "root", "token": "dstok", "t": int(time.time())}, namespace="token" + ) + ) From f6bf2d8045cc239fe34357342bff1440561c8909 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sat, 29 Oct 2022 23:20:11 -0700 Subject: [PATCH 216/952] Initial prototype of API explorer at /-/api, refs #1871 --- datasette/app.py | 5 ++ datasette/templates/api_explorer.html | 73 +++++++++++++++++++++++++++ datasette/views/special.py | 8 +++ tests/test_docs.py | 2 +- 4 files changed, 87 insertions(+), 1 deletion(-) create mode 100644 datasette/templates/api_explorer.html diff --git a/datasette/app.py b/datasette/app.py index f80d3792..c3d802a4 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -33,6 +33,7 @@ from .views.special import ( JsonDataView, PatternPortfolioView, AuthTokenView, + ApiExplorerView, CreateTokenView, LogoutView, AllowDebugView, @@ -1235,6 +1236,10 @@ class Datasette: CreateTokenView.as_view(self), r"/-/create-token$", ) + add_route( + ApiExplorerView.as_view(self), + r"/-/api$", + ) add_route( LogoutView.as_view(self), r"/-/logout$", diff --git a/datasette/templates/api_explorer.html b/datasette/templates/api_explorer.html new file mode 100644 index 00000000..034bee60 --- /dev/null +++ b/datasette/templates/api_explorer.html @@ -0,0 +1,73 @@ +{% extends "base.html" %} + +{% block title %}API Explorer{% endblock %} + +{% block content %} + +

API Explorer

+ +

Use this tool to try out the Datasette write API.

+ +{% if errors %} + {% for error in errors %} +

{{ error }}

+ {% endfor %} +{% endif %} + + +
+ + +
+
+ + +
+
+ +
+

+ + + + +{% endblock %} diff --git a/datasette/views/special.py b/datasette/views/special.py index b754a2f0..9922a621 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -235,3 +235,11 @@ class CreateTokenView(BaseView): "token_bits": token_bits, }, ) + + +class ApiExplorerView(BaseView): + name = "api_explorer" + has_json_alternate = False + + async def get(self, request): + return await self.render(["api_explorer.html"], request) diff --git a/tests/test_docs.py b/tests/test_docs.py index cd5a6c13..e9b813fe 100644 --- a/tests/test_docs.py +++ b/tests/test_docs.py @@ -62,7 +62,7 @@ def documented_views(): if first_word.endswith("View"): view_labels.add(first_word) # We deliberately don't document these: - view_labels.update(("PatternPortfolioView", "AuthTokenView")) + view_labels.update(("PatternPortfolioView", "AuthTokenView", "ApiExplorerView")) return view_labels From 9eb9ffae3ddd4e8ff0b713bf6fd6a0afed3368d7 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 30 Oct 2022 13:09:55 -0700 Subject: [PATCH 217/952] Drop API token requirement from API explorer, refs #1871 --- datasette/default_permissions.py | 9 +++++++++ datasette/templates/api_explorer.html | 13 ++++--------- 2 files changed, 13 insertions(+), 9 deletions(-) diff --git a/datasette/default_permissions.py b/datasette/default_permissions.py index 87684e2a..151ba2b5 100644 --- a/datasette/default_permissions.py +++ b/datasette/default_permissions.py @@ -131,3 +131,12 @@ def register_commands(cli): if debug: click.echo("\nDecoded:\n") click.echo(json.dumps(ds.unsign(token, namespace="token"), indent=2)) + + +@hookimpl +def skip_csrf(scope): + # Skip CSRF check for requests with content-type: application/json + if scope["type"] == "http": + headers = scope.get("headers") or {} + if dict(headers).get(b"content-type") == b"application/json": + return True diff --git a/datasette/templates/api_explorer.html b/datasette/templates/api_explorer.html index 034bee60..01b182d8 100644 --- a/datasette/templates/api_explorer.html +++ b/datasette/templates/api_explorer.html @@ -15,16 +15,13 @@ {% endif %}
-
- - -
- +
-
- +
+ +

@@ -46,7 +43,6 @@ form.addEventListener("submit", (ev) => { var formData = new FormData(form); var json = formData.get('json'); var path = formData.get('path'); - var token = formData.get('token'); // Validate JSON try { var data = JSON.parse(json); @@ -60,7 +56,6 @@ form.addEventListener("submit", (ev) => { body: json, headers: { 'Content-Type': 'application/json', - 'Authorization': `Bearer ${token}` } }).then(r => r.json()).then(r => { alert(JSON.stringify(r, null, 2)); From fedbfcc36873366143195d8fe124e1859bf88346 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 30 Oct 2022 14:49:07 -0700 Subject: [PATCH 218/952] Neater display of output and errors in API explorer, refs #1871 --- datasette/templates/api_explorer.html | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/datasette/templates/api_explorer.html b/datasette/templates/api_explorer.html index 01b182d8..38fdb7bc 100644 --- a/datasette/templates/api_explorer.html +++ b/datasette/templates/api_explorer.html @@ -26,6 +26,12 @@

+ + """.format( escape(ex.sql) ) diff --git a/tests/test_api.py b/tests/test_api.py index ad74d16e..4027a7a5 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -662,7 +662,11 @@ def test_sql_time_limit(app_client_shorter_time_limit): "

SQL query took too long. The time limit is controlled by the\n" 'sql_time_limit_ms\n' "configuration option.

\n" - "
select sleep(0.5)
" + '\n' + "" ), "status": 400, "title": "SQL Interrupted", diff --git a/tests/test_html.py b/tests/test_html.py index 4b394199..7cfe9d90 100644 --- a/tests/test_html.py +++ b/tests/test_html.py @@ -172,7 +172,7 @@ def test_sql_time_limit(app_client_shorter_time_limit): """ sql_time_limit_ms """.strip(), - "
select sleep(0.5)
", + '', ] for expected_html_fragment in expected_html_fragments: assert expected_html_fragment in response.text From 93a02281dad2f23da84210f6ae9c63777ad8af5e Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 1 Nov 2022 10:22:26 -0700 Subject: [PATCH 223/952] Show interrupted query in resizing textarea, closes #1876 --- datasette/views/base.py | 6 +++++- tests/test_api.py | 6 +++++- tests/test_html.py | 2 +- 3 files changed, 11 insertions(+), 3 deletions(-) diff --git a/datasette/views/base.py b/datasette/views/base.py index 67aa3a42..6b01fdd2 100644 --- a/datasette/views/base.py +++ b/datasette/views/base.py @@ -378,7 +378,11 @@ class DataView(BaseView):

SQL query took too long. The time limit is controlled by the sql_time_limit_ms configuration option.

- 
{}
+ + """.format( escape(ex.sql) ) diff --git a/tests/test_api.py b/tests/test_api.py index ebd675b9..de0223e2 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -662,7 +662,11 @@ def test_sql_time_limit(app_client_shorter_time_limit): "

SQL query took too long. The time limit is controlled by the\n" 'sql_time_limit_ms\n' "configuration option.

\n" - "
select sleep(0.5)
" + '\n' + "" ), "status": 400, "title": "SQL Interrupted", diff --git a/tests/test_html.py b/tests/test_html.py index 4b394199..7cfe9d90 100644 --- a/tests/test_html.py +++ b/tests/test_html.py @@ -172,7 +172,7 @@ def test_sql_time_limit(app_client_shorter_time_limit): """ sql_time_limit_ms """.strip(), - "
select sleep(0.5)
", + '', ] for expected_html_fragment in expected_html_fragments: assert expected_html_fragment in response.text From 9bec7c38eb93cde5afb16df9bdd96aea2a5b0459 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 1 Nov 2022 11:07:59 -0700 Subject: [PATCH 224/952] ignore and replace options for bulk inserts, refs #1873 Also removed the rule that you cannot include primary keys in the rows you insert. And added validation that catches invalid parameters in the incoming JSON. And renamed "inserted" to "rows" in the returned JSON for return_rows: true --- datasette/views/table.py | 41 ++++++++++++++------ docs/json_api.rst | 4 +- tests/test_api_write.py | 83 ++++++++++++++++++++++++++++++++++++++-- 3 files changed, 111 insertions(+), 17 deletions(-) diff --git a/datasette/views/table.py b/datasette/views/table.py index 1e3d566e..7692a4e3 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1107,6 +1107,7 @@ class TableInsertView(BaseView): if not isinstance(data, dict): return _errors(["JSON must be a dictionary"]) keys = data.keys() + # keys must contain "row" or "rows" if "row" not in keys and "rows" not in keys: return _errors(['JSON must have one or other of "row" or "rows"']) @@ -1126,19 +1127,31 @@ class TableInsertView(BaseView): for row in rows: if not isinstance(row, dict): return _errors(['"rows" must be a list of dictionaries']) + # Does this exceed max_insert_rows? max_insert_rows = self.ds.setting("max_insert_rows") if len(rows) > max_insert_rows: return _errors( ["Too many rows, maximum allowed is {}".format(max_insert_rows)] ) + + # Validate other parameters + extras = { + key: value for key, value in data.items() if key not in ("row", "rows") + } + valid_extras = {"return_rows", "ignore", "replace"} + invalid_extras = extras.keys() - valid_extras + if invalid_extras: + return _errors( + ['Invalid parameter: "{}"'.format('", "'.join(sorted(invalid_extras)))] + ) + if extras.get("ignore") and extras.get("replace"): + return _errors(['Cannot use "ignore" and "replace" at the same time']) + # Validate columns of each row - columns = await db.table_columns(table_name) - # TODO: There are cases where pks are OK, if not using auto-incrementing pk - pks = await db.primary_keys(table_name) - allowed_columns = set(columns) - set(pks) + columns = set(await db.table_columns(table_name)) for i, row in enumerate(rows): - invalid_columns = set(row.keys()) - allowed_columns + invalid_columns = set(row.keys()) - columns if invalid_columns: errors.append( "Row {} has invalid columns: {}".format( @@ -1147,8 +1160,7 @@ class TableInsertView(BaseView): ) if errors: return _errors(errors) - extra = {key: data[key] for key in data if key not in ("rows", "row")} - return rows, errors, extra + return rows, errors, extras async def post(self, request): database_route = tilde_decode(request.url_vars["database"]) @@ -1168,18 +1180,23 @@ class TableInsertView(BaseView): request.actor, "insert-row", resource=(database_name, table_name) ): return _error(["Permission denied"], 403) - rows, errors, extra = await self._validate_data(request, db, table_name) + rows, errors, extras = await self._validate_data(request, db, table_name) if errors: return _error(errors, 400) - should_return = bool(extra.get("return_rows", False)) + ignore = extras.get("ignore") + replace = extras.get("replace") + + should_return = bool(extras.get("return_rows", False)) # Insert rows def insert_rows(conn): table = sqlite_utils.Database(conn)[table_name] if should_return: rowids = [] for row in rows: - rowids.append(table.insert(row).last_rowid) + rowids.append( + table.insert(row, ignore=ignore, replace=replace).last_rowid + ) return list( table.rows_where( "rowid in ({})".format(",".join("?" for _ in rowids)), @@ -1187,12 +1204,12 @@ class TableInsertView(BaseView): ) ) else: - table.insert_all(rows) + table.insert_all(rows, ignore=ignore, replace=replace) rows = await db.execute_write_fn(insert_rows) result = {"ok": True} if should_return: - result["inserted"] = rows + result["rows"] = rows return Response.json(result, status=201) diff --git a/docs/json_api.rst b/docs/json_api.rst index da4500ab..34c13211 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -489,7 +489,7 @@ If successful, this will return a ``201`` status code and the newly inserted row .. code-block:: json { - "inserted": [ + "rows": [ { "id": 1, "column1": "value1", @@ -538,7 +538,7 @@ To return the newly inserted rows, add the ``"return_rows": true`` key to the re "return_rows": true } -This will return the same ``"inserted"`` key as the single row example above. There is a small performance penalty for using this option. +This will return the same ``"rows"`` key as the single row example above. There is a small performance penalty for using this option. .. _RowDeleteView: diff --git a/tests/test_api_write.py b/tests/test_api_write.py index 1cfba104..d0b0f324 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -37,7 +37,7 @@ async def test_write_row(ds_write): ) expected_row = {"id": 1, "title": "Test", "score": 1.0} assert response.status_code == 201 - assert response.json()["inserted"] == [expected_row] + assert response.json()["rows"] == [expected_row] rows = (await ds_write.get_database("data").execute("select * from docs")).rows assert dict(rows[0]) == expected_row @@ -70,7 +70,7 @@ async def test_write_rows(ds_write, return_rows): ] assert response.json()["ok"] is True if return_rows: - assert response.json()["inserted"] == actual_rows + assert response.json()["rows"] == actual_rows @pytest.mark.asyncio @@ -156,6 +156,27 @@ async def test_write_rows(ds_write, return_rows): 400, ["Too many rows, maximum allowed is 100"], ), + ( + "/data/docs/-/insert", + {"rows": [{"title": "Test"}], "ignore": True, "replace": True}, + None, + 400, + ['Cannot use "ignore" and "replace" at the same time'], + ), + ( + "/data/docs/-/insert", + {"rows": [{"title": "Test"}], "invalid_param": True}, + None, + 400, + ['Invalid parameter: "invalid_param"'], + ), + ( + "/data/docs/-/insert", + {"rows": [{"title": "Test"}], "one": True, "two": True}, + None, + 400, + ['Invalid parameter: "one", "two"'], + ), # Validate columns of each row ( "/data/docs/-/insert", @@ -196,6 +217,62 @@ async def test_write_row_errors( assert response.json()["errors"] == expected_errors +@pytest.mark.asyncio +@pytest.mark.parametrize( + "ignore,replace,expected_rows", + ( + ( + True, + False, + [ + {"id": 1, "title": "Exists", "score": None}, + ], + ), + ( + False, + True, + [ + {"id": 1, "title": "One", "score": None}, + ], + ), + ), +) +@pytest.mark.parametrize("should_return", (True, False)) +async def test_insert_ignore_replace( + ds_write, ignore, replace, expected_rows, should_return +): + await ds_write.get_database("data").execute_write( + "insert into docs (id, title) values (1, 'Exists')" + ) + token = write_token(ds_write) + data = {"rows": [{"id": 1, "title": "One"}]} + if ignore: + data["ignore"] = True + if replace: + data["replace"] = True + if should_return: + data["return_rows"] = True + response = await ds_write.client.post( + "/data/docs/-/insert", + json=data, + headers={ + "Authorization": "Bearer {}".format(token), + "Content-Type": "application/json", + }, + ) + assert response.status_code == 201 + actual_rows = [ + dict(r) + for r in ( + await ds_write.get_database("data").execute("select * from docs") + ).rows + ] + assert actual_rows == expected_rows + assert response.json()["ok"] is True + if should_return: + assert response.json()["rows"] == expected_rows + + @pytest.mark.asyncio @pytest.mark.parametrize("scenario", ("no_token", "no_perm", "bad_table", "has_perm")) async def test_delete_row(ds_write, scenario): @@ -217,7 +294,7 @@ async def test_delete_row(ds_write, scenario): }, ) assert insert_response.status_code == 201 - pk = insert_response.json()["inserted"][0]["id"] + pk = insert_response.json()["rows"][0]["id"] path = "/data/{}/{}/-/delete".format( "docs" if scenario != "bad_table" else "bad_table", pk From 497290beaf32e6b779f9683ef15f1c5bc142a41a Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 1 Nov 2022 12:59:17 -0700 Subject: [PATCH 225/952] Handle database errors in /-/insert, refs #1866, #1873 Also improved API explorer to show HTTP status of response, refs #1871 --- datasette/templates/api_explorer.html | 14 +++++++++----- datasette/views/table.py | 5 ++++- tests/test_api_write.py | 11 +++++++++++ 3 files changed, 24 insertions(+), 6 deletions(-) diff --git a/datasette/templates/api_explorer.html b/datasette/templates/api_explorer.html index 38fdb7bc..93bacde3 100644 --- a/datasette/templates/api_explorer.html +++ b/datasette/templates/api_explorer.html @@ -27,7 +27,8 @@ @@ -64,12 +65,15 @@ form.addEventListener("submit", (ev) => { headers: { 'Content-Type': 'application/json', } - }).then(r => r.json()).then(r => { + }).then(r => { + document.getElementById('response-status').textContent = r.status; + return r.json(); + }).then(data => { var errorList = output.querySelector('.errors'); - if (r.errors) { + if (data.errors) { errorList.style.display = 'block'; errorList.innerHTML = ''; - r.errors.forEach(error => { + data.errors.forEach(error => { var li = document.createElement('li'); li.textContent = error; errorList.appendChild(li); @@ -77,7 +81,7 @@ form.addEventListener("submit", (ev) => { } else { errorList.style.display = 'none'; } - output.querySelector('pre').innerText = JSON.stringify(r, null, 2); + output.querySelector('pre').innerText = JSON.stringify(data, null, 2); output.style.display = 'block'; }).catch(err => { alert("Error: " + err); diff --git a/datasette/views/table.py b/datasette/views/table.py index 7692a4e3..61227206 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1206,7 +1206,10 @@ class TableInsertView(BaseView): else: table.insert_all(rows, ignore=ignore, replace=replace) - rows = await db.execute_write_fn(insert_rows) + try: + rows = await db.execute_write_fn(insert_rows) + except Exception as e: + return _error([str(e)]) result = {"ok": True} if should_return: result["rows"] = rows diff --git a/tests/test_api_write.py b/tests/test_api_write.py index d0b0f324..0b567f48 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -156,6 +156,13 @@ async def test_write_rows(ds_write, return_rows): 400, ["Too many rows, maximum allowed is 100"], ), + ( + "/data/docs/-/insert", + {"rows": [{"id": 1, "title": "Test"}]}, + "duplicate_id", + 400, + ["UNIQUE constraint failed: docs.id"], + ), ( "/data/docs/-/insert", {"rows": [{"title": "Test"}], "ignore": True, "replace": True}, @@ -194,6 +201,10 @@ async def test_write_row_errors( ds_write, path, input, special_case, expected_status, expected_errors ): token = write_token(ds_write) + if special_case == "duplicate_id": + await ds_write.get_database("data").execute_write( + "insert into docs (id) values (1)" + ) if special_case == "bad_token": token += "bad" kwargs = dict( From 0b166befc0096fca30d71e19608a928d59c331a4 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 1 Nov 2022 17:31:22 -0700 Subject: [PATCH 226/952] API explorer can now do GET, has JSON syntax highlighting Refs #1871 --- .../static/json-format-highlight-1.0.1.js | 43 +++++++++++ datasette/templates/api_explorer.html | 77 +++++++++++++++---- 2 files changed, 103 insertions(+), 17 deletions(-) create mode 100644 datasette/static/json-format-highlight-1.0.1.js diff --git a/datasette/static/json-format-highlight-1.0.1.js b/datasette/static/json-format-highlight-1.0.1.js new file mode 100644 index 00000000..e87c76e1 --- /dev/null +++ b/datasette/static/json-format-highlight-1.0.1.js @@ -0,0 +1,43 @@ +/* +https://github.com/luyilin/json-format-highlight +From https://unpkg.com/json-format-highlight@1.0.1/dist/json-format-highlight.js +MIT Licensed +*/ +(function (global, factory) { + typeof exports === 'object' && typeof module !== 'undefined' ? module.exports = factory() : + typeof define === 'function' && define.amd ? define(factory) : + (global.jsonFormatHighlight = factory()); +}(this, (function () { 'use strict'; + +var defaultColors = { + keyColor: 'dimgray', + numberColor: 'lightskyblue', + stringColor: 'lightcoral', + trueColor: 'lightseagreen', + falseColor: '#f66578', + nullColor: 'cornflowerblue' +}; + +function index (json, colorOptions) { + if ( colorOptions === void 0 ) colorOptions = {}; + + if (!json) { return; } + if (typeof json !== 'string') { + json = JSON.stringify(json, null, 2); + } + var colors = Object.assign({}, defaultColors, colorOptions); + json = json.replace(/&/g, '&').replace(//g, '>'); + return json.replace(/("(\\u[a-zA-Z0-9]{4}|\\[^u]|[^\\"])*"(\s*:)?|\b(true|false|null)\b|-?\d+(?:\.\d*)?(?:[eE][+]?\d+)?)/g, function (match) { + var color = colors.numberColor; + if (/^"/.test(match)) { + color = /:$/.test(match) ? colors.keyColor : colors.stringColor; + } else { + color = /true/.test(match) ? colors.trueColor : /false/.test(match) ? colors.falseColor : /null/.test(match) ? colors.nullColor : color; + } + return ("" + match + ""); + }); +} + +return index; + +}))); diff --git a/datasette/templates/api_explorer.html b/datasette/templates/api_explorer.html index 93bacde3..de5337e3 100644 --- a/datasette/templates/api_explorer.html +++ b/datasette/templates/api_explorer.html @@ -2,6 +2,10 @@ {% block title %}API Explorer{% endblock %} +{% block extra_head %} + +{% endblock %} + {% block content %}

API Explorer

@@ -14,17 +18,30 @@ {% endfor %} {% endif %} -
-
- - -
-
- - -
-

- +
+ GET +
+
+ + + +
+ +
+
+ POST +
+
+ + +
+
+ + +
+

+ +
{% else %} - {% if not canned_write and not error %} + {% if not canned_query_write and not error %}

0 results

{% endif %} {% endif %} diff --git a/datasette/views/database.py b/datasette/views/database.py index 0770a380..658c35e6 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -1,4 +1,3 @@ -from asyncinject import Registry from dataclasses import dataclass, field from typing import Callable from urllib.parse import parse_qsl, urlencode @@ -33,7 +32,7 @@ from datasette.utils import ( from datasette.utils.asgi import AsgiFileDownload, NotFound, Response, Forbidden from datasette.plugins import pm -from .base import BaseView, DatasetteError, DataView, View, _error, stream_csv +from .base import BaseView, DatasetteError, View, _error, stream_csv class DatabaseView(View): @@ -57,7 +56,7 @@ class DatabaseView(View): sql = (request.args.get("sql") or "").strip() if sql: - return await query_view(request, datasette) + return await QueryView()(request, datasette) if format_ not in ("html", "json"): raise NotFound("Invalid format: {}".format(format_)) @@ -65,10 +64,6 @@ class DatabaseView(View): metadata = (datasette.metadata("databases") or {}).get(database, {}) datasette.update_with_inherited_metadata(metadata) - table_counts = await db.table_counts(5) - hidden_table_names = set(await db.hidden_table_names()) - all_foreign_keys = await db.get_all_foreign_keys() - sql_views = [] for view_name in await db.view_names(): view_visible, view_private = await datasette.check_visibility( @@ -196,8 +191,13 @@ class QueryContext: # urls: dict = field( # metadata={"help": "Object containing URL helpers like `database()`"} # ) - canned_write: bool = field( - metadata={"help": "Boolean indicating if this canned query allows writes"} + canned_query_write: bool = field( + metadata={ + "help": "Boolean indicating if this is a canned query that allows writes" + } + ) + metadata: dict = field( + metadata={"help": "Metadata about the database or the canned query"} ) db_is_immutable: bool = field( metadata={"help": "Boolean indicating if this database is immutable"} @@ -232,7 +232,6 @@ class QueryContext: show_hide_hidden: str = field( metadata={"help": "Hidden input field for the _show_sql parameter"} ) - metadata: dict = field(metadata={"help": "Metadata about the query/database"}) database_color: Callable = field( metadata={"help": "Function that returns a color for a given database name"} ) @@ -242,6 +241,12 @@ class QueryContext: alternate_url_json: str = field( metadata={"help": "URL for alternate JSON version of this page"} ) + # TODO: refactor this to somewhere else, probably ds.render_template() + select_templates: list = field( + metadata={ + "help": "List of templates that were considered for rendering this page" + } + ) async def get_tables(datasette, request, db): @@ -320,287 +325,105 @@ async def database_download(request, datasette): ) -async def query_view( - request, - datasette, - # canned_query=None, - # _size=None, - # named_parameters=None, - # write=False, -): - db = await datasette.resolve_database(request) - database = db.name - # Flattened because of ?sql=&name1=value1&name2=value2 feature - params = {key: request.args.get(key) for key in request.args} - sql = None - if "sql" in params: - sql = params.pop("sql") - if "_shape" in params: - params.pop("_shape") +class QueryView(View): + async def post(self, request, datasette): + from datasette.app import TableNotFound - # extras come from original request.args to avoid being flattened - extras = request.args.getlist("_extra") + db = await datasette.resolve_database(request) - # TODO: Behave differently for canned query here: - await datasette.ensure_permissions(request.actor, [("execute-sql", database)]) - - _, private = await datasette.check_visibility( - request.actor, - permissions=[ - ("view-database", database), - "view-instance", - ], - ) - - extra_args = {} - if params.get("_timelimit"): - extra_args["custom_time_limit"] = int(params["_timelimit"]) - - format_ = request.url_vars.get("format") or "html" - query_error = None - try: - validate_sql_select(sql) - results = await datasette.execute( - database, sql, params, truncate=True, **extra_args - ) - columns = results.columns - rows = results.rows - except QueryInterrupted as ex: - raise DatasetteError( - textwrap.dedent( - """ -

SQL query took too long. The time limit is controlled by the - sql_time_limit_ms - configuration option.

- - - """.format( - markupsafe.escape(ex.sql) - ) - ).strip(), - title="SQL Interrupted", - status=400, - message_is_html=True, - ) - except sqlite3.DatabaseError as ex: - query_error = str(ex) - results = None - rows = [] - columns = [] - except (sqlite3.OperationalError, InvalidSql) as ex: - raise DatasetteError(str(ex), title="Invalid SQL", status=400) - except sqlite3.OperationalError as ex: - raise DatasetteError(str(ex)) - except DatasetteError: - raise - - # Handle formats from plugins - if format_ == "csv": - - async def fetch_data_for_csv(request, _next=None): - results = await db.execute(sql, params, truncate=True) - data = {"rows": results.rows, "columns": results.columns} - return data, None, None - - return await stream_csv(datasette, fetch_data_for_csv, request, db.name) - elif format_ in datasette.renderers.keys(): - # Dispatch request to the correct output format renderer - # (CSV is not handled here due to streaming) - result = call_with_supported_arguments( - datasette.renderers[format_][0], - datasette=datasette, - columns=columns, - rows=rows, - sql=sql, - query_name=None, - database=database, - table=None, - request=request, - view_name="table", - truncated=results.truncated if results else False, - error=query_error, - # These will be deprecated in Datasette 1.0: - args=request.args, - data={"rows": rows, "columns": columns}, - ) - if asyncio.iscoroutine(result): - result = await result - if result is None: - raise NotFound("No data") - if isinstance(result, dict): - r = Response( - body=result.get("body"), - status=result.get("status_code") or 200, - content_type=result.get("content_type", "text/plain"), - headers=result.get("headers"), + # We must be a canned query + table_found = False + try: + await datasette.resolve_table(request) + table_found = True + except TableNotFound as table_not_found: + canned_query = await datasette.get_canned_query( + table_not_found.database_name, table_not_found.table, request.actor ) - elif isinstance(result, Response): - r = result - # if status_code is not None: - # # Over-ride the status code - # r.status = status_code - else: - assert False, f"{result} should be dict or Response" - elif format_ == "html": - headers = {} - templates = [f"query-{to_css_class(database)}.html", "query.html"] - template = datasette.jinja_env.select_template(templates) - alternate_url_json = datasette.absolute_url( - request, - datasette.urls.path(path_with_format(request=request, format="json")), - ) - data = {} - headers.update( - { - "Link": '{}; rel="alternate"; type="application/json+datasette"'.format( - alternate_url_json - ) - } - ) - metadata = (datasette.metadata("databases") or {}).get(database, {}) - datasette.update_with_inherited_metadata(metadata) + if canned_query is None: + raise + if table_found: + # That should not have happened + raise DatasetteError("Unexpected table found on POST", status=404) - renderers = {} - for key, (_, can_render) in datasette.renderers.items(): - it_can_render = call_with_supported_arguments( - can_render, - datasette=datasette, - columns=data.get("columns") or [], - rows=data.get("rows") or [], - sql=data.get("query", {}).get("sql", None), - query_name=data.get("query_name"), - database=database, - table=data.get("table"), - request=request, - view_name="database", + # If database is immutable, return an error + if not db.is_mutable: + raise Forbidden("Database is immutable") + + # Process the POST + body = await request.post_body() + body = body.decode("utf-8").strip() + if body.startswith("{") and body.endswith("}"): + params = json.loads(body) + # But we want key=value strings + for key, value in params.items(): + params[key] = str(value) + else: + params = dict(parse_qsl(body, keep_blank_values=True)) + # Should we return JSON? + should_return_json = ( + request.headers.get("accept") == "application/json" + or request.args.get("_json") + or params.get("_json") + ) + params_for_query = MagicParameters(params, request, datasette) + ok = None + redirect_url = None + try: + cursor = await db.execute_write(canned_query["sql"], params_for_query) + message = canned_query.get( + "on_success_message" + ) or "Query executed, {} row{} affected".format( + cursor.rowcount, "" if cursor.rowcount == 1 else "s" + ) + message_type = datasette.INFO + redirect_url = canned_query.get("on_success_redirect") + ok = True + except Exception as ex: + message = canned_query.get("on_error_message") or str(ex) + message_type = datasette.ERROR + redirect_url = canned_query.get("on_error_redirect") + ok = False + if should_return_json: + return Response.json( + { + "ok": ok, + "message": message, + "redirect": redirect_url, + } ) - it_can_render = await await_me_maybe(it_can_render) - if it_can_render: - renderers[key] = datasette.urls.path( - path_with_format(request=request, format=key) - ) - - allow_execute_sql = await datasette.permission_allowed( - request.actor, "execute-sql", database - ) - - show_hide_hidden = "" - if metadata.get("hide_sql"): - if bool(params.get("_show_sql")): - show_hide_link = path_with_removed_args(request, {"_show_sql"}) - show_hide_text = "hide" - show_hide_hidden = '' - else: - show_hide_link = path_with_added_args(request, {"_show_sql": 1}) - show_hide_text = "show" else: - if bool(params.get("_hide_sql")): - show_hide_link = path_with_removed_args(request, {"_hide_sql"}) - show_hide_text = "show" - show_hide_hidden = '' - else: - show_hide_link = path_with_added_args(request, {"_hide_sql": 1}) - show_hide_text = "hide" - hide_sql = show_hide_text == "show" + datasette.add_message(request, message, message_type) + return Response.redirect(redirect_url or request.path) - # Extract any :named parameters - named_parameters = await derive_named_parameters( - datasette.get_database(database), sql - ) - named_parameter_values = { - named_parameter: params.get(named_parameter) or "" - for named_parameter in named_parameters - if not named_parameter.startswith("_") - } + async def get(self, request, datasette): + from datasette.app import TableNotFound - # Set to blank string if missing from params - for named_parameter in named_parameters: - if named_parameter not in params and not named_parameter.startswith("_"): - params[named_parameter] = "" - - r = Response.html( - await datasette.render_template( - template, - QueryContext( - database=database, - query={ - "sql": sql, - "params": params, - }, - canned_query=None, - private=private, - canned_write=False, - db_is_immutable=not db.is_mutable, - error=query_error, - hide_sql=hide_sql, - show_hide_link=datasette.urls.path(show_hide_link), - show_hide_text=show_hide_text, - editable=True, # TODO - allow_execute_sql=allow_execute_sql, - tables=await get_tables(datasette, request, db), - named_parameter_values=named_parameter_values, - edit_sql_url="todo", - display_rows=await display_rows( - datasette, database, request, rows, columns - ), - table_columns=await _table_columns(datasette, database) - if allow_execute_sql - else {}, - columns=columns, - renderers=renderers, - url_csv=datasette.urls.path( - path_with_format( - request=request, format="csv", extra_qs={"_size": "max"} - ) - ), - show_hide_hidden=markupsafe.Markup(show_hide_hidden), - metadata=metadata, - database_color=lambda _: "#ff0000", - alternate_url_json=alternate_url_json, - ), - request=request, - view_name="database", - ), - headers=headers, - ) - else: - assert False, "Invalid format: {}".format(format_) - if datasette.cors: - add_cors_headers(r.headers) - return r - - -class QueryView(DataView): - async def data( - self, - request, - sql, - editable=True, - canned_query=None, - metadata=None, - _size=None, - named_parameters=None, - write=False, - default_labels=None, - ): - db = await self.ds.resolve_database(request) + db = await datasette.resolve_database(request) database = db.name - params = {key: request.args.get(key) for key in request.args} - if "sql" in params: - params.pop("sql") - if "_shape" in params: - params.pop("_shape") + + # Are we a canned query? + canned_query = None + canned_query_write = False + if "table" in request.url_vars: + try: + await datasette.resolve_table(request) + except TableNotFound as table_not_found: + # Was this actually a canned query? + canned_query = await datasette.get_canned_query( + table_not_found.database_name, table_not_found.table, request.actor + ) + if canned_query is None: + raise + canned_query_write = bool(canned_query.get("write")) private = False if canned_query: # Respect canned query permissions - visible, private = await self.ds.check_visibility( + visible, private = await datasette.check_visibility( request.actor, permissions=[ - ("view-query", (database, canned_query)), + ("view-query", (database, canned_query["name"])), ("view-database", database), "view-instance", ], @@ -609,18 +432,32 @@ class QueryView(DataView): raise Forbidden("You do not have permission to view this query") else: - await self.ds.ensure_permissions(request.actor, [("execute-sql", database)]) + await datasette.ensure_permissions( + request.actor, [("execute-sql", database)] + ) + + # Flattened because of ?sql=&name1=value1&name2=value2 feature + params = {key: request.args.get(key) for key in request.args} + sql = None + + if canned_query: + sql = canned_query["sql"] + elif "sql" in params: + sql = params.pop("sql") # Extract any :named parameters - named_parameters = named_parameters or await derive_named_parameters( - self.ds.get_database(database), sql - ) + named_parameters = [] + if canned_query and canned_query.get("params"): + named_parameters = canned_query["params"] + if not named_parameters: + named_parameters = await derive_named_parameters( + datasette.get_database(database), sql + ) named_parameter_values = { named_parameter: params.get(named_parameter) or "" for named_parameter in named_parameters if not named_parameter.startswith("_") } - # Set to blank string if missing from params for named_parameter in named_parameters: if named_parameter not in params and not named_parameter.startswith("_"): @@ -629,212 +466,159 @@ class QueryView(DataView): extra_args = {} if params.get("_timelimit"): extra_args["custom_time_limit"] = int(params["_timelimit"]) - if _size: - extra_args["page_size"] = _size - templates = [f"query-{to_css_class(database)}.html", "query.html"] - if canned_query: - templates.insert( - 0, - f"query-{to_css_class(database)}-{to_css_class(canned_query)}.html", - ) + format_ = request.url_vars.get("format") or "html" query_error = None + results = None + rows = [] + columns = [] - # Execute query - as write or as read - if write: - if request.method == "POST": - # If database is immutable, return an error - if not db.is_mutable: - raise Forbidden("Database is immutable") - body = await request.post_body() - body = body.decode("utf-8").strip() - if body.startswith("{") and body.endswith("}"): - params = json.loads(body) - # But we want key=value strings - for key, value in params.items(): - params[key] = str(value) - else: - params = dict(parse_qsl(body, keep_blank_values=True)) - # Should we return JSON? - should_return_json = ( - request.headers.get("accept") == "application/json" - or request.args.get("_json") - or params.get("_json") - ) - if canned_query: - params_for_query = MagicParameters(params, request, self.ds) - else: - params_for_query = params - ok = None - try: - cursor = await self.ds.databases[database].execute_write( - sql, params_for_query - ) - message = metadata.get( - "on_success_message" - ) or "Query executed, {} row{} affected".format( - cursor.rowcount, "" if cursor.rowcount == 1 else "s" - ) - message_type = self.ds.INFO - redirect_url = metadata.get("on_success_redirect") - ok = True - except Exception as e: - message = metadata.get("on_error_message") or str(e) - message_type = self.ds.ERROR - redirect_url = metadata.get("on_error_redirect") - ok = False - if should_return_json: - return Response.json( - { - "ok": ok, - "message": message, - "redirect": redirect_url, - } - ) - else: - self.ds.add_message(request, message, message_type) - return self.redirect(request, redirect_url or request.path) - else: + params_for_query = params - async def extra_template(): - return { - "request": request, - "db_is_immutable": not db.is_mutable, - "path_with_added_args": path_with_added_args, - "path_with_removed_args": path_with_removed_args, - "named_parameter_values": named_parameter_values, - "canned_query": canned_query, - "success_message": request.args.get("_success") or "", - "canned_write": True, - } - - return ( - { - "database": database, - "rows": [], - "truncated": False, - "columns": [], - "query": {"sql": sql, "params": params}, - "private": private, - }, - extra_template, - templates, - ) - else: # Not a write - if canned_query: - params_for_query = MagicParameters(params, request, self.ds) - else: - params_for_query = params + if not canned_query_write: try: - results = await self.ds.execute( + if not canned_query: + # For regular queries we only allow SELECT, plus other rules + validate_sql_select(sql) + else: + # Canned queries can run magic parameters + params_for_query = MagicParameters(params, request, datasette) + results = await datasette.execute( database, sql, params_for_query, truncate=True, **extra_args ) - columns = [r[0] for r in results.description] - except sqlite3.DatabaseError as e: - query_error = e + columns = results.columns + rows = results.rows + except QueryInterrupted as ex: + raise DatasetteError( + textwrap.dedent( + """ +

SQL query took too long. The time limit is controlled by the + sql_time_limit_ms + configuration option.

+ + + """.format( + markupsafe.escape(ex.sql) + ) + ).strip(), + title="SQL Interrupted", + status=400, + message_is_html=True, + ) + except sqlite3.DatabaseError as ex: + query_error = str(ex) results = None + rows = [] columns = [] + except (sqlite3.OperationalError, InvalidSql) as ex: + raise DatasetteError(str(ex), title="Invalid SQL", status=400) + except sqlite3.OperationalError as ex: + raise DatasetteError(str(ex)) + except DatasetteError: + raise - allow_execute_sql = await self.ds.permission_allowed( - request.actor, "execute-sql", database - ) + # Handle formats from plugins + if format_ == "csv": - async def extra_template(): - display_rows = [] - truncate_cells = self.ds.setting("truncate_cells_html") - for row in results.rows if results else []: - display_row = [] - for column, value in zip(results.columns, row): - display_value = value - # Let the plugins have a go - # pylint: disable=no-member - plugin_display_value = None - for candidate in pm.hook.render_cell( - row=row, - value=value, - column=column, - table=None, - database=database, - datasette=self.ds, - request=request, - ): - candidate = await await_me_maybe(candidate) - if candidate is not None: - plugin_display_value = candidate - break - if plugin_display_value is not None: - display_value = plugin_display_value - else: - if value in ("", None): - display_value = markupsafe.Markup(" ") - elif is_url(str(display_value).strip()): - display_value = markupsafe.Markup( - '{truncated_url}'.format( - url=markupsafe.escape(value.strip()), - truncated_url=markupsafe.escape( - truncate_url(value.strip(), truncate_cells) - ), - ) - ) - elif isinstance(display_value, bytes): - blob_url = path_with_format( - request=request, - format="blob", - extra_qs={ - "_blob_column": column, - "_blob_hash": hashlib.sha256( - display_value - ).hexdigest(), - }, - ) - formatted = format_bytes(len(value)) - display_value = markupsafe.Markup( - '<Binary: {:,} byte{}>'.format( - blob_url, - ' title="{}"'.format(formatted) - if "bytes" not in formatted - else "", - len(value), - "" if len(value) == 1 else "s", - ) - ) - else: - display_value = str(value) - if truncate_cells and len(display_value) > truncate_cells: - display_value = ( - display_value[:truncate_cells] + "\u2026" - ) - display_row.append(display_value) - display_rows.append(display_row) + async def fetch_data_for_csv(request, _next=None): + results = await db.execute(sql, params, truncate=True) + data = {"rows": results.rows, "columns": results.columns} + return data, None, None - # Show 'Edit SQL' button only if: - # - User is allowed to execute SQL - # - SQL is an approved SELECT statement - # - No magic parameters, so no :_ in the SQL string - edit_sql_url = None - is_validated_sql = False - try: - validate_sql_select(sql) - is_validated_sql = True - except InvalidSql: - pass - if allow_execute_sql and is_validated_sql and ":_" not in sql: - edit_sql_url = ( - self.ds.urls.database(database) - + "?" - + urlencode( - { - **{ - "sql": sql, - }, - **named_parameter_values, - } - ) + return await stream_csv(datasette, fetch_data_for_csv, request, db.name) + elif format_ in datasette.renderers.keys(): + # Dispatch request to the correct output format renderer + # (CSV is not handled here due to streaming) + result = call_with_supported_arguments( + datasette.renderers[format_][0], + datasette=datasette, + columns=columns, + rows=rows, + sql=sql, + query_name=canned_query["name"] if canned_query else None, + database=database, + table=None, + request=request, + view_name="table", + truncated=results.truncated if results else False, + error=query_error, + # These will be deprecated in Datasette 1.0: + args=request.args, + data={"rows": rows, "columns": columns}, + ) + if asyncio.iscoroutine(result): + result = await result + if result is None: + raise NotFound("No data") + if isinstance(result, dict): + r = Response( + body=result.get("body"), + status=result.get("status_code") or 200, + content_type=result.get("content_type", "text/plain"), + headers=result.get("headers"), + ) + elif isinstance(result, Response): + r = result + # if status_code is not None: + # # Over-ride the status code + # r.status = status_code + else: + assert False, f"{result} should be dict or Response" + elif format_ == "html": + headers = {} + templates = [f"query-{to_css_class(database)}.html", "query.html"] + if canned_query: + templates.insert( + 0, + f"query-{to_css_class(database)}-{to_css_class(canned_query['name'])}.html", ) + template = datasette.jinja_env.select_template(templates) + alternate_url_json = datasette.absolute_url( + request, + datasette.urls.path(path_with_format(request=request, format="json")), + ) + data = {} + headers.update( + { + "Link": '{}; rel="alternate"; type="application/json+datasette"'.format( + alternate_url_json + ) + } + ) + metadata = (datasette.metadata("databases") or {}).get(database, {}) + datasette.update_with_inherited_metadata(metadata) + + renderers = {} + for key, (_, can_render) in datasette.renderers.items(): + it_can_render = call_with_supported_arguments( + can_render, + datasette=datasette, + columns=data.get("columns") or [], + rows=data.get("rows") or [], + sql=data.get("query", {}).get("sql", None), + query_name=data.get("query_name"), + database=database, + table=data.get("table"), + request=request, + view_name="database", + ) + it_can_render = await await_me_maybe(it_can_render) + if it_can_render: + renderers[key] = datasette.urls.path( + path_with_format(request=request, format=key) + ) + + allow_execute_sql = await datasette.permission_allowed( + request.actor, "execute-sql", database + ) + show_hide_hidden = "" - if metadata.get("hide_sql"): + if canned_query and canned_query.get("hide_sql"): if bool(params.get("_show_sql")): show_hide_link = path_with_removed_args(request, {"_show_sql"}) show_hide_text = "hide" @@ -855,42 +639,86 @@ class QueryView(DataView): show_hide_link = path_with_added_args(request, {"_hide_sql": 1}) show_hide_text = "hide" hide_sql = show_hide_text == "show" - return { - "display_rows": display_rows, - "custom_sql": True, - "named_parameter_values": named_parameter_values, - "editable": editable, - "canned_query": canned_query, - "edit_sql_url": edit_sql_url, - "metadata": metadata, - "settings": self.ds.settings_dict(), - "request": request, - "show_hide_link": self.ds.urls.path(show_hide_link), - "show_hide_text": show_hide_text, - "show_hide_hidden": markupsafe.Markup(show_hide_hidden), - "hide_sql": hide_sql, - "table_columns": await _table_columns(self.ds, database) - if allow_execute_sql - else {}, - } - return ( - { - "ok": not query_error, - "database": database, - "query_name": canned_query, - "rows": results.rows if results else [], - "truncated": results.truncated if results else False, - "columns": columns, - "query": {"sql": sql, "params": params}, - "error": str(query_error) if query_error else None, - "private": private, - "allow_execute_sql": allow_execute_sql, - }, - extra_template, - templates, - 400 if query_error else 200, - ) + # Show 'Edit SQL' button only if: + # - User is allowed to execute SQL + # - SQL is an approved SELECT statement + # - No magic parameters, so no :_ in the SQL string + edit_sql_url = None + is_validated_sql = False + try: + validate_sql_select(sql) + is_validated_sql = True + except InvalidSql: + pass + if allow_execute_sql and is_validated_sql and ":_" not in sql: + edit_sql_url = ( + datasette.urls.database(database) + + "?" + + urlencode( + { + **{ + "sql": sql, + }, + **named_parameter_values, + } + ) + ) + + r = Response.html( + await datasette.render_template( + template, + QueryContext( + database=database, + query={ + "sql": sql, + "params": params, + }, + canned_query=canned_query["name"] if canned_query else None, + private=private, + canned_query_write=canned_query_write, + db_is_immutable=not db.is_mutable, + error=query_error, + hide_sql=hide_sql, + show_hide_link=datasette.urls.path(show_hide_link), + show_hide_text=show_hide_text, + editable=not canned_query, + allow_execute_sql=allow_execute_sql, + tables=await get_tables(datasette, request, db), + named_parameter_values=named_parameter_values, + edit_sql_url=edit_sql_url, + display_rows=await display_rows( + datasette, database, request, rows, columns + ), + table_columns=await _table_columns(datasette, database) + if allow_execute_sql + else {}, + columns=columns, + renderers=renderers, + url_csv=datasette.urls.path( + path_with_format( + request=request, format="csv", extra_qs={"_size": "max"} + ) + ), + show_hide_hidden=markupsafe.Markup(show_hide_hidden), + metadata=canned_query or metadata, + database_color=lambda _: "#ff0000", + alternate_url_json=alternate_url_json, + select_templates=[ + f"{'*' if template_name == template.name else ''}{template_name}" + for template_name in templates + ], + ), + request=request, + view_name="database", + ), + headers=headers, + ) + else: + assert False, "Invalid format: {}".format(format_) + if datasette.cors: + add_cors_headers(r.headers) + return r class MagicParameters(dict): diff --git a/datasette/views/table.py b/datasette/views/table.py index 77acfd95..28264e92 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -9,7 +9,6 @@ import markupsafe from datasette.plugins import pm from datasette.database import QueryInterrupted from datasette import tracer -from datasette.renderer import json_renderer from datasette.utils import ( add_cors_headers, await_me_maybe, @@ -21,7 +20,6 @@ from datasette.utils import ( tilde_encode, escape_sqlite, filters_should_redirect, - format_bytes, is_url, path_from_row_pks, path_with_added_args, @@ -38,7 +36,7 @@ from datasette.utils import ( from datasette.utils.asgi import BadRequest, Forbidden, NotFound, Response from datasette.filters import Filters import sqlite_utils -from .base import BaseView, DataView, DatasetteError, ureg, _error, stream_csv +from .base import BaseView, DatasetteError, ureg, _error, stream_csv from .database import QueryView LINK_WITH_LABEL = ( @@ -698,57 +696,6 @@ async def table_view(datasette, request): return response -class CannedQueryView(DataView): - def __init__(self, datasette): - self.ds = datasette - - async def post(self, request): - from datasette.app import TableNotFound - - try: - await self.ds.resolve_table(request) - except TableNotFound as e: - # Was this actually a canned query? - canned_query = await self.ds.get_canned_query( - e.database_name, e.table, request.actor - ) - if canned_query: - # Handle POST to a canned query - return await QueryView(self.ds).data( - request, - canned_query["sql"], - metadata=canned_query, - editable=False, - canned_query=e.table, - named_parameters=canned_query.get("params"), - write=bool(canned_query.get("write")), - ) - - return Response.text("Method not allowed", status=405) - - async def data(self, request, **kwargs): - from datasette.app import TableNotFound - - try: - await self.ds.resolve_table(request) - except TableNotFound as not_found: - canned_query = await self.ds.get_canned_query( - not_found.database_name, not_found.table, request.actor - ) - if canned_query: - return await QueryView(self.ds).data( - request, - canned_query["sql"], - metadata=canned_query, - editable=False, - canned_query=not_found.table, - named_parameters=canned_query.get("params"), - write=bool(canned_query.get("write")), - ) - else: - raise - - async def table_view_traced(datasette, request): from datasette.app import TableNotFound @@ -761,10 +708,7 @@ async def table_view_traced(datasette, request): ) # If this is a canned query, not a table, then dispatch to QueryView instead if canned_query: - if request.method == "POST": - return await CannedQueryView(datasette).post(request) - else: - return await CannedQueryView(datasette).get(request) + return await QueryView()(request, datasette) else: raise diff --git a/tests/test_canned_queries.py b/tests/test_canned_queries.py index d6a88733..e9ad3239 100644 --- a/tests/test_canned_queries.py +++ b/tests/test_canned_queries.py @@ -95,12 +95,12 @@ def test_insert(canned_write_client): csrftoken_from=True, cookies={"foo": "bar"}, ) - assert 302 == response.status - assert "/data/add_name?success" == response.headers["Location"] messages = canned_write_client.ds.unsign( response.cookies["ds_messages"], "messages" ) - assert [["Query executed, 1 row affected", 1]] == messages + assert messages == [["Query executed, 1 row affected", 1]] + assert response.status == 302 + assert response.headers["Location"] == "/data/add_name?success" @pytest.mark.parametrize( @@ -382,11 +382,11 @@ def test_magic_parameters_cannot_be_used_in_arbitrary_queries(magic_parameters_c def test_canned_write_custom_template(canned_write_client): response = canned_write_client.get("/data/update_name") assert response.status == 200 + assert "!!!CUSTOM_UPDATE_NAME_TEMPLATE!!!" in response.text assert ( "" in response.text ) - assert "!!!CUSTOM_UPDATE_NAME_TEMPLATE!!!" in response.text # And test for link rel=alternate while we're here: assert ( '' From 8920d425f4d417cfd998b61016c5ff3530cd34e1 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 9 Aug 2023 10:20:58 -0700 Subject: [PATCH 468/952] 1.0a3 release notes, smaller changes section - refs #2135 --- docs/changelog.rst | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/changelog.rst b/docs/changelog.rst index ee48d075..b4416f94 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -4,6 +4,25 @@ Changelog ========= +.. _v1_0_a3: + +1.0a3 (2023-08-09) +------------------ + +This alpha release previews the updated design for Datasette's default JSON API. + +Smaller changes +~~~~~~~~~~~~~~~ + +- Datasette documentation now shows YAML examples for :ref:`metadata` by default, with a tab interface for switching to JSON. (:issue:`1153`) +- :ref:`plugin_register_output_renderer` plugins now have access to ``error`` and ``truncated`` arguments, allowing them to display error messages and take into account truncated results. (:issue:`2130`) +- ``render_cell()`` plugin hook now also supports an optional ``request`` argument. (:issue:`2007`) +- New ``Justfile`` to support development workflows for Datasette using `Just `__. +- ``datasette.render_template()`` can now accepts a ``datasette.views.Context`` subclass as an alternative to a dictionary. (:issue:`2127`) +- ``datasette install -e path`` option for editable installations, useful while developing plugins. (:issue:`2106`) +- When started with the ``--cors`` option Datasette now serves an ``Access-Control-Max-Age: 3600`` header, ensuring CORS OPTIONS requests are repeated no more than once an hour. (:issue:`2079`) +- Fixed a bug where the ``_internal`` database could display ``None`` instead of ``null`` for in-memory databases. (:issue:`1970`) + .. _v0_64_2: 0.64.2 (2023-03-08) From e34d09c6ec16ff5e7717e112afdad67f7c05a62a Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 9 Aug 2023 12:01:59 -0700 Subject: [PATCH 469/952] Don't include columns in query JSON, refs #2136 --- datasette/renderer.py | 8 +++++++- datasette/views/database.py | 2 +- tests/test_api.py | 1 - tests/test_cli_serve_get.py | 11 ++++++----- 4 files changed, 14 insertions(+), 8 deletions(-) diff --git a/datasette/renderer.py b/datasette/renderer.py index 0bd74e81..224031a7 100644 --- a/datasette/renderer.py +++ b/datasette/renderer.py @@ -27,7 +27,7 @@ def convert_specific_columns_to_json(rows, columns, json_cols): return new_rows -def json_renderer(args, data, error, truncated=None): +def json_renderer(request, args, data, error, truncated=None): """Render a response as JSON""" status_code = 200 @@ -106,6 +106,12 @@ def json_renderer(args, data, error, truncated=None): "status": 400, "title": None, } + + # Don't include "columns" in output + # https://github.com/simonw/datasette/issues/2136 + if isinstance(data, dict) and "columns" not in request.args.getlist("_extra"): + data.pop("columns", None) + # Handle _nl option for _shape=array nl = args.get("_nl", "") if nl and shape == "array": diff --git a/datasette/views/database.py b/datasette/views/database.py index 658c35e6..cf76f3c2 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -548,7 +548,7 @@ class QueryView(View): error=query_error, # These will be deprecated in Datasette 1.0: args=request.args, - data={"rows": rows, "columns": columns}, + data={"ok": True, "rows": rows, "columns": columns}, ) if asyncio.iscoroutine(result): result = await result diff --git a/tests/test_api.py b/tests/test_api.py index 28415a0b..f96f571e 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -649,7 +649,6 @@ async def test_custom_sql(ds_client): {"content": "RENDER_CELL_DEMO"}, {"content": "RENDER_CELL_ASYNC"}, ], - "columns": ["content"], "ok": True, "truncated": False, } diff --git a/tests/test_cli_serve_get.py b/tests/test_cli_serve_get.py index 2e0390bb..dc7fc1e2 100644 --- a/tests/test_cli_serve_get.py +++ b/tests/test_cli_serve_get.py @@ -34,11 +34,12 @@ def test_serve_with_get(tmp_path_factory): "/_memory.json?sql=select+sqlite_version()", ], ) - assert 0 == result.exit_code, result.output - assert { - "truncated": False, - "columns": ["sqlite_version()"], - }.items() <= json.loads(result.output).items() + assert result.exit_code == 0, result.output + data = json.loads(result.output) + # Should have a single row with a single column + assert len(data["rows"]) == 1 + assert list(data["rows"][0].keys()) == ["sqlite_version()"] + assert set(data.keys()) == {"rows", "ok", "truncated"} # The plugin should have created hello.txt assert (plugins_dir / "hello.txt").read_text() == "hello" From 856ca68d94708c6e94673cb6bc28bf3e3ca17845 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 9 Aug 2023 12:04:40 -0700 Subject: [PATCH 470/952] Update default JSON representation docs, refs #2135 --- docs/json_api.rst | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/docs/json_api.rst b/docs/json_api.rst index c273c2a8..16b997eb 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -9,10 +9,10 @@ through the Datasette user interface can also be accessed as JSON via the API. To access the API for a page, either click on the ``.json`` link on that page or edit the URL and add a ``.json`` extension to it. -.. _json_api_shapes: +.. _json_api_default: -Different shapes ----------------- +Default representation +---------------------- The default JSON representation of data from a SQLite table or custom query looks like this: @@ -21,7 +21,6 @@ looks like this: { "ok": true, - "next": null, "rows": [ { "id": 3, @@ -39,13 +38,22 @@ looks like this: "id": 1, "name": "San Francisco" } - ] + ], + "truncated": false } -The ``rows`` key is a list of objects, each one representing a row. ``next`` indicates if -there is another page, and ``ok`` is always ``true`` if an error did not occur. +``"ok"`` is always ``true`` if an error did not occur. -If ``next`` is present then the next page in the pagination set can be retrieved using ``?_next=VALUE``. +The ``"rows"`` key is a list of objects, each one representing a row. + +The ``"truncated"`` key lets you know if the query was truncated. This can happen if a SQL query returns more than 1,000 results (or the :ref:`setting_max_returned_rows` setting). + +For table pages, an additional key ``"next"`` may be present. This indicates that the next page in the pagination set can be retrieved using ``?_next=VALUE``. + +.. _json_api_shapes: + +Different shapes +---------------- The ``_shape`` parameter can be used to access alternative formats for the ``rows`` key which may be more convenient for your application. There are three From 90cb9ca58d910f49e8f117bbdd94df6f0855cf99 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 9 Aug 2023 12:11:16 -0700 Subject: [PATCH 471/952] JSON changes in release notes, refs #2135 --- docs/changelog.rst | 35 ++++++++++++++++++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index b4416f94..4c70855b 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -9,7 +9,40 @@ Changelog 1.0a3 (2023-08-09) ------------------ -This alpha release previews the updated design for Datasette's default JSON API. +This alpha release previews the updated design for Datasette's default JSON API. (:issue:`782`) + +The new :ref:`default JSON representation ` for both table pages (``/dbname/table.json``) and arbitrary SQL queries (``/dbname.json?sql=...``) is now shaped like this: + +.. code-block:: json + + { + "ok": true, + "rows": [ + { + "id": 3, + "name": "Detroit" + }, + { + "id": 2, + "name": "Los Angeles" + }, + { + "id": 4, + "name": "Memnonia" + }, + { + "id": 1, + "name": "San Francisco" + } + ], + "truncated": false + } + +Tables will include an additional ``"next"`` key for pagination, which can be passed to ``?_next=`` to fetch the next page of results. + +The various ``?_shape=`` options continue to work as before - see :ref:`json_api_shapes` for details. + +A new ``?_extra=`` mechanism is available for tables, but has not yet been stabilized or documented. Details on that are available in :issue:`262`. Smaller changes ~~~~~~~~~~~~~~~ From 19ab4552e212c9845a59461cc73e82d5ae8c278a Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 9 Aug 2023 12:13:11 -0700 Subject: [PATCH 472/952] Release 1.0a3 Closes #2135 Refs #262, #782, #1153, #1970, #2007, #2079, #2106, #2127, #2130 --- datasette/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/datasette/version.py b/datasette/version.py index 3b81ab21..61dee464 100644 --- a/datasette/version.py +++ b/datasette/version.py @@ -1,2 +1,2 @@ -__version__ = "1.0a2" +__version__ = "1.0a3" __version_info__ = tuple(__version__.split(".")) From 4a42476bb7ce4c5ed941f944115dedd9bce34656 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 9 Aug 2023 15:04:16 -0700 Subject: [PATCH 473/952] datasette plugins --requirements, closes #2133 --- datasette/cli.py | 12 ++++++++++-- docs/cli-reference.rst | 1 + docs/plugins.rst | 32 ++++++++++++++++++++++++++++---- tests/test_cli.py | 3 +++ 4 files changed, 42 insertions(+), 6 deletions(-) diff --git a/datasette/cli.py b/datasette/cli.py index 32266888..21fd25d6 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -223,15 +223,23 @@ pm.hook.publish_subcommand(publish=publish) @cli.command() @click.option("--all", help="Include built-in default plugins", is_flag=True) +@click.option( + "--requirements", help="Output requirements.txt of installed plugins", is_flag=True +) @click.option( "--plugins-dir", type=click.Path(exists=True, file_okay=False, dir_okay=True), help="Path to directory containing custom plugins", ) -def plugins(all, plugins_dir): +def plugins(all, requirements, plugins_dir): """List currently installed plugins""" app = Datasette([], plugins_dir=plugins_dir) - click.echo(json.dumps(app._plugins(all=all), indent=4)) + if requirements: + for plugin in app._plugins(): + if plugin["version"]: + click.echo("{}=={}".format(plugin["name"], plugin["version"])) + else: + click.echo(json.dumps(app._plugins(all=all), indent=4)) @cli.command() diff --git a/docs/cli-reference.rst b/docs/cli-reference.rst index 2177fc9e..7a96d311 100644 --- a/docs/cli-reference.rst +++ b/docs/cli-reference.rst @@ -282,6 +282,7 @@ Output JSON showing all currently installed plugins, their versions, whether the Options: --all Include built-in default plugins + --requirements Output requirements.txt of installed plugins --plugins-dir DIRECTORY Path to directory containing custom plugins --help Show this message and exit. diff --git a/docs/plugins.rst b/docs/plugins.rst index 979f94dd..19bfdd0c 100644 --- a/docs/plugins.rst +++ b/docs/plugins.rst @@ -90,7 +90,12 @@ You can see a list of installed plugins by navigating to the ``/-/plugins`` page You can also use the ``datasette plugins`` command:: - $ datasette plugins + datasette plugins + +Which outputs: + +.. code-block:: json + [ { "name": "datasette_json_html", @@ -107,7 +112,8 @@ You can also use the ``datasette plugins`` command:: cog.out("\n") result = CliRunner().invoke(cli.cli, ["plugins", "--all"]) # cog.out() with text containing newlines was unindenting for some reason - cog.outl("If you run ``datasette plugins --all`` it will include default plugins that ship as part of Datasette::\n") + cog.outl("If you run ``datasette plugins --all`` it will include default plugins that ship as part of Datasette:\n") + cog.outl(".. code-block:: json\n") plugins = [p for p in json.loads(result.output) if p["name"].startswith("datasette.")] indented = textwrap.indent(json.dumps(plugins, indent=4), " ") for line in indented.split("\n"): @@ -115,7 +121,9 @@ You can also use the ``datasette plugins`` command:: cog.out("\n\n") .. ]]] -If you run ``datasette plugins --all`` it will include default plugins that ship as part of Datasette:: +If you run ``datasette plugins --all`` it will include default plugins that ship as part of Datasette: + +.. code-block:: json [ { @@ -236,6 +244,22 @@ If you run ``datasette plugins --all`` it will include default plugins that ship You can add the ``--plugins-dir=`` option to include any plugins found in that directory. +Add ``--requirements`` to output a list of installed plugins that can then be installed in another Datasette instance using ``datasette install -r requirements.txt``:: + + datasette plugins --requirements + +The output will look something like this:: + + datasette-codespaces==0.1.1 + datasette-graphql==2.2 + datasette-json-html==1.0.1 + datasette-pretty-json==0.2.2 + datasette-x-forwarded-host==0.1 + +To write that to a ``requirements.txt`` file, run this:: + + datasette plugins --requirements > requirements.txt + .. _plugins_configuration: Plugin configuration @@ -390,7 +414,7 @@ Any values embedded in ``metadata.yaml`` will be visible to anyone who views the If you are publishing your data using the :ref:`datasette publish ` family of commands, you can use the ``--plugin-secret`` option to set these secrets at publish time. For example, using Heroku you might run the following command:: - $ datasette publish heroku my_database.db \ + datasette publish heroku my_database.db \ --name my-heroku-app-demo \ --install=datasette-auth-github \ --plugin-secret datasette-auth-github client_id your_client_id \ diff --git a/tests/test_cli.py b/tests/test_cli.py index 75724f61..056e2821 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -108,6 +108,9 @@ def test_plugins_cli(app_client): assert set(names).issuperset({p["name"] for p in EXPECTED_PLUGINS}) # And the following too: assert set(names).issuperset(DEFAULT_PLUGINS) + # --requirements should be empty because there are no installed non-plugins-dir plugins + result3 = runner.invoke(cli, ["plugins", "--requirements"]) + assert result3.output == "" def test_metadata_yaml(): From a3593c901580ea50854c3e0774b0ba0126e8a76f Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 9 Aug 2023 17:32:07 -0700 Subject: [PATCH 474/952] on_success_message_sql, closes #2138 --- datasette/views/database.py | 29 ++++++++++++++++---- docs/sql_queries.rst | 21 ++++++++++---- tests/test_canned_queries.py | 53 +++++++++++++++++++++++++++++++----- 3 files changed, 85 insertions(+), 18 deletions(-) diff --git a/datasette/views/database.py b/datasette/views/database.py index cf76f3c2..79b3f88d 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -360,6 +360,10 @@ class QueryView(View): params[key] = str(value) else: params = dict(parse_qsl(body, keep_blank_values=True)) + + # Don't ever send csrftoken as a SQL parameter + params.pop("csrftoken", None) + # Should we return JSON? should_return_json = ( request.headers.get("accept") == "application/json" @@ -371,12 +375,27 @@ class QueryView(View): redirect_url = None try: cursor = await db.execute_write(canned_query["sql"], params_for_query) - message = canned_query.get( - "on_success_message" - ) or "Query executed, {} row{} affected".format( - cursor.rowcount, "" if cursor.rowcount == 1 else "s" - ) + # success message can come from on_success_message or on_success_message_sql + message = None message_type = datasette.INFO + on_success_message_sql = canned_query.get("on_success_message_sql") + if on_success_message_sql: + try: + message_result = ( + await db.execute(on_success_message_sql, params_for_query) + ).first() + if message_result: + message = message_result[0] + except Exception as ex: + message = "Error running on_success_message_sql: {}".format(ex) + message_type = datasette.ERROR + if not message: + message = canned_query.get( + "on_success_message" + ) or "Query executed, {} row{} affected".format( + cursor.rowcount, "" if cursor.rowcount == 1 else "s" + ) + redirect_url = canned_query.get("on_success_redirect") ok = True except Exception as ex: diff --git a/docs/sql_queries.rst b/docs/sql_queries.rst index 3c2cb228..1ae07e1f 100644 --- a/docs/sql_queries.rst +++ b/docs/sql_queries.rst @@ -392,6 +392,7 @@ This configuration will create a page at ``/mydatabase/add_name`` displaying a f You can customize how Datasette represents success and errors using the following optional properties: - ``on_success_message`` - the message shown when a query is successful +- ``on_success_message_sql`` - alternative to ``on_success_message``: a SQL query that should be executed to generate the message - ``on_success_redirect`` - the path or URL the user is redirected to on success - ``on_error_message`` - the message shown when a query throws an error - ``on_error_redirect`` - the path or URL the user is redirected to on error @@ -405,11 +406,12 @@ For example: "queries": { "add_name": { "sql": "INSERT INTO names (name) VALUES (:name)", + "params": ["name"], "write": True, - "on_success_message": "Name inserted", + "on_success_message_sql": "select 'Name inserted: ' || :name", "on_success_redirect": "/mydatabase/names", "on_error_message": "Name insert failed", - "on_error_redirect": "/mydatabase" + "on_error_redirect": "/mydatabase", } } } @@ -426,8 +428,10 @@ For example: queries: add_name: sql: INSERT INTO names (name) VALUES (:name) + params: + - name write: true - on_success_message: Name inserted + on_success_message_sql: 'select ''Name inserted: '' || :name' on_success_redirect: /mydatabase/names on_error_message: Name insert failed on_error_redirect: /mydatabase @@ -443,8 +447,11 @@ For example: "queries": { "add_name": { "sql": "INSERT INTO names (name) VALUES (:name)", + "params": [ + "name" + ], "write": true, - "on_success_message": "Name inserted", + "on_success_message_sql": "select 'Name inserted: ' || :name", "on_success_redirect": "/mydatabase/names", "on_error_message": "Name insert failed", "on_error_redirect": "/mydatabase" @@ -455,10 +462,12 @@ For example: } .. [[[end]]] -You can use ``"params"`` to explicitly list the named parameters that should be displayed as form fields - otherwise they will be automatically detected. +You can use ``"params"`` to explicitly list the named parameters that should be displayed as form fields - otherwise they will be automatically detected. ``"params"`` is not necessary in the above example, since without it ``"name"`` would be automatically detected from the query. You can pre-populate form fields when the page first loads using a query string, e.g. ``/mydatabase/add_name?name=Prepopulated``. The user will have to submit the form to execute the query. +If you specify a query in ``"on_success_message_sql"``, that query will be executed after the main query. The first column of the first row return by that query will be displayed as a success message. Named parameters from the main query will be made available to the success message query as well. + .. _canned_queries_magic_parameters: Magic parameters @@ -589,7 +598,7 @@ The JSON response will look like this: "redirect": "/data/add_name" } -The ``"message"`` and ``"redirect"`` values here will take into account ``on_success_message``, ``on_success_redirect``, ``on_error_message`` and ``on_error_redirect``, if they have been set. +The ``"message"`` and ``"redirect"`` values here will take into account ``on_success_message``, ``on_success_message_sql``, ``on_success_redirect``, ``on_error_message`` and ``on_error_redirect``, if they have been set. .. _pagination: diff --git a/tests/test_canned_queries.py b/tests/test_canned_queries.py index e9ad3239..5256c24c 100644 --- a/tests/test_canned_queries.py +++ b/tests/test_canned_queries.py @@ -31,9 +31,15 @@ def canned_write_client(tmpdir): }, "add_name_specify_id": { "sql": "insert into names (rowid, name) values (:rowid, :name)", + "on_success_message_sql": "select 'Name added: ' || :name || ' with rowid ' || :rowid", "write": True, "on_error_redirect": "/data/add_name_specify_id?error", }, + "add_name_specify_id_with_error_in_on_success_message_sql": { + "sql": "insert into names (rowid, name) values (:rowid, :name)", + "on_success_message_sql": "select this is bad SQL", + "write": True, + }, "delete_name": { "sql": "delete from names where rowid = :rowid", "write": True, @@ -179,6 +185,34 @@ def test_insert_error(canned_write_client): ) +def test_on_success_message_sql(canned_write_client): + response = canned_write_client.post( + "/data/add_name_specify_id", + {"rowid": 5, "name": "Should be OK"}, + csrftoken_from=True, + ) + assert response.status == 302 + assert response.headers["Location"] == "/data/add_name_specify_id" + messages = canned_write_client.ds.unsign( + response.cookies["ds_messages"], "messages" + ) + assert messages == [["Name added: Should be OK with rowid 5", 1]] + + +def test_error_in_on_success_message_sql(canned_write_client): + response = canned_write_client.post( + "/data/add_name_specify_id_with_error_in_on_success_message_sql", + {"rowid": 1, "name": "Should fail"}, + csrftoken_from=True, + ) + messages = canned_write_client.ds.unsign( + response.cookies["ds_messages"], "messages" + ) + assert messages == [ + ["Error running on_success_message_sql: no such column: bad", 3] + ] + + def test_custom_params(canned_write_client): response = canned_write_client.get("/data/update_name?extra=foo") assert '' in response.text @@ -232,21 +266,22 @@ def test_canned_query_permissions_on_database_page(canned_write_client): query_names = { q["name"] for q in canned_write_client.get("/data.json").json["queries"] } - assert { + assert query_names == { + "add_name_specify_id_with_error_in_on_success_message_sql", + "from_hook", + "update_name", + "add_name_specify_id", + "from_async_hook", "canned_read", "add_name", - "add_name_specify_id", - "update_name", - "from_async_hook", - "from_hook", - } == query_names + } # With auth shows four response = canned_write_client.get( "/data.json", cookies={"ds_actor": canned_write_client.actor_cookie({"id": "root"})}, ) - assert 200 == response.status + assert response.status == 200 query_names_and_private = sorted( [ {"name": q["name"], "private": q["private"]} @@ -257,6 +292,10 @@ def test_canned_query_permissions_on_database_page(canned_write_client): assert query_names_and_private == [ {"name": "add_name", "private": False}, {"name": "add_name_specify_id", "private": False}, + { + "name": "add_name_specify_id_with_error_in_on_success_message_sql", + "private": False, + }, {"name": "canned_read", "private": False}, {"name": "delete_name", "private": True}, {"name": "from_async_hook", "private": False}, From 33251d04e78d575cca62bb59069bb43a7d924746 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 9 Aug 2023 17:56:27 -0700 Subject: [PATCH 475/952] Canned query write counters demo, refs #2134 --- .github/workflows/deploy-latest.yml | 30 +++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/.github/workflows/deploy-latest.yml b/.github/workflows/deploy-latest.yml index ed60376c..4746aa07 100644 --- a/.github/workflows/deploy-latest.yml +++ b/.github/workflows/deploy-latest.yml @@ -57,6 +57,36 @@ jobs: db.route = "alternative-route" ' > plugins/alternative_route.py cp fixtures.db fixtures2.db + - name: And the counters writable canned query demo + run: | + cat > plugins/counters.py < Date: Thu, 10 Aug 2023 22:16:19 -0700 Subject: [PATCH 476/952] Fixed display of database color Closes #2139, closes #2119 --- datasette/database.py | 7 +++++++ datasette/templates/database.html | 2 +- datasette/templates/query.html | 2 +- datasette/templates/row.html | 2 +- datasette/templates/table.html | 2 +- datasette/views/base.py | 4 ---- datasette/views/database.py | 8 +++----- datasette/views/index.py | 4 +--- datasette/views/row.py | 4 +++- datasette/views/table.py | 2 +- tests/test_html.py | 20 ++++++++++++++++++++ 11 files changed, 39 insertions(+), 18 deletions(-) diff --git a/datasette/database.py b/datasette/database.py index d8043c24..af39ac9e 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -1,6 +1,7 @@ import asyncio from collections import namedtuple from pathlib import Path +import hashlib import janus import queue import sys @@ -62,6 +63,12 @@ class Database: } return self._cached_table_counts + @property + def color(self): + if self.hash: + return self.hash[:6] + return hashlib.md5(self.name.encode("utf8")).hexdigest()[:6] + def suggest_name(self): if self.path: return Path(self.path).stem diff --git a/datasette/templates/database.html b/datasette/templates/database.html index 7acf0369..3d4dae07 100644 --- a/datasette/templates/database.html +++ b/datasette/templates/database.html @@ -10,7 +10,7 @@ {% block body_class %}db db-{{ database|to_css_class }}{% endblock %} {% block content %} -
+

{{ metadata.title or database }}{% if private %} 🔒{% endif %}

{% set links = database_actions() %}{% if links %}
diff --git a/datasette/templates/query.html b/datasette/templates/query.html index fc3b8527..b8f06f84 100644 --- a/datasette/templates/query.html +++ b/datasette/templates/query.html @@ -28,7 +28,7 @@

This query cannot be executed because the database is immutable.

{% endif %} -

{{ metadata.title or database }}{% if canned_query and not metadata.title %}: {{ canned_query }}{% endif %}{% if private %} 🔒{% endif %}

+

{{ metadata.title or database }}{% if canned_query and not metadata.title %}: {{ canned_query }}{% endif %}{% if private %} 🔒{% endif %}

{% block description_source_license %}{% include "_description_source_license.html" %}{% endblock %} diff --git a/datasette/templates/row.html b/datasette/templates/row.html index 1d1b0bfd..4d179a85 100644 --- a/datasette/templates/row.html +++ b/datasette/templates/row.html @@ -20,7 +20,7 @@ {% endblock %} {% block content %} -

{{ table }}: {{ ', '.join(primary_key_values) }}{% if private %} 🔒{% endif %}

+

{{ table }}: {{ ', '.join(primary_key_values) }}{% if private %} 🔒{% endif %}

{% block description_source_license %}{% include "_description_source_license.html" %}{% endblock %} diff --git a/datasette/templates/table.html b/datasette/templates/table.html index 0be1998c..88580e52 100644 --- a/datasette/templates/table.html +++ b/datasette/templates/table.html @@ -21,7 +21,7 @@ {% endblock %} {% block content %} -
+

{{ metadata.get("title") or table }}{% if is_view %} (view){% endif %}{% if private %} 🔒{% endif %}

{% set links = table_actions() %}{% if links %}
diff --git a/datasette/views/base.py b/datasette/views/base.py index da5c55ad..0080b33c 100644 --- a/datasette/views/base.py +++ b/datasette/views/base.py @@ -102,9 +102,6 @@ class BaseView: response.body = b"" return response - def database_color(self, database): - return "ff0000" - async def method_not_allowed(self, request): if ( request.path.endswith(".json") @@ -150,7 +147,6 @@ class BaseView: template_context = { **context, **{ - "database_color": self.database_color, "select_templates": [ f"{'*' if template_name == template.name else ''}{template_name}" for template_name in templates diff --git a/datasette/views/database.py b/datasette/views/database.py index 79b3f88d..d9abc38a 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -146,6 +146,7 @@ class DatabaseView(View): template = datasette.jinja_env.select_template(templates) context = { **json_data, + "database_color": db.color, "database_actions": database_actions, "show_hidden": request.args.get("_show_hidden"), "editable": True, @@ -154,7 +155,6 @@ class DatabaseView(View): and not db.is_mutable and not db.is_memory, "attached_databases": attached_databases, - "database_color": lambda _: "#ff0000", "alternate_url_json": alternate_url_json, "select_templates": [ f"{'*' if template_name == template.name else ''}{template_name}" @@ -179,6 +179,7 @@ class DatabaseView(View): @dataclass class QueryContext: database: str = field(metadata={"help": "The name of the database being queried"}) + database_color: str = field(metadata={"help": "The color of the database"}) query: dict = field( metadata={"help": "The SQL query object containing the `sql` string"} ) @@ -232,9 +233,6 @@ class QueryContext: show_hide_hidden: str = field( metadata={"help": "Hidden input field for the _show_sql parameter"} ) - database_color: Callable = field( - metadata={"help": "Function that returns a color for a given database name"} - ) table_columns: dict = field( metadata={"help": "Dictionary of table name to list of column names"} ) @@ -689,6 +687,7 @@ class QueryView(View): template, QueryContext( database=database, + database_color=db.color, query={ "sql": sql, "params": params, @@ -721,7 +720,6 @@ class QueryView(View): ), show_hide_hidden=markupsafe.Markup(show_hide_hidden), metadata=canned_query or metadata, - database_color=lambda _: "#ff0000", alternate_url_json=alternate_url_json, select_templates=[ f"{'*' if template_name == template.name else ''}{template_name}" diff --git a/datasette/views/index.py b/datasette/views/index.py index df411c4a..95b29302 100644 --- a/datasette/views/index.py +++ b/datasette/views/index.py @@ -105,9 +105,7 @@ class IndexView(BaseView): { "name": name, "hash": db.hash, - "color": db.hash[:6] - if db.hash - else hashlib.md5(name.encode("utf8")).hexdigest()[:6], + "color": db.color, "path": self.ds.urls.database(name), "tables_and_views_truncated": tables_and_views_truncated, "tables_and_views_more": (len(visible_tables) + len(views)) diff --git a/datasette/views/row.py b/datasette/views/row.py index 6e09f30e..8f07a662 100644 --- a/datasette/views/row.py +++ b/datasette/views/row.py @@ -18,7 +18,8 @@ class RowView(DataView): async def data(self, request, default_labels=False): resolved = await self.ds.resolve_row(request) - database = resolved.db.name + db = resolved.db + database = db.name table = resolved.table pk_values = resolved.pk_values @@ -60,6 +61,7 @@ class RowView(DataView): "foreign_key_tables": await self.foreign_key_tables( database, table, pk_values ), + "database_color": db.color, "display_columns": display_columns, "display_rows": display_rows, "custom_table_templates": [ diff --git a/datasette/views/table.py b/datasette/views/table.py index 28264e92..6df8b915 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1408,7 +1408,7 @@ async def table_view_data( return table_name async def extra_database_color(): - return lambda _: "ff0000" + return db.color async def extra_form_hidden_args(): form_hidden_args = [] diff --git a/tests/test_html.py b/tests/test_html.py index 7856bc27..ffc2aef1 100644 --- a/tests/test_html.py +++ b/tests/test_html.py @@ -1103,3 +1103,23 @@ async def test_breadcrumbs_respect_permissions( assert actual == expected_links finally: ds_client.ds._metadata_local = orig + + +@pytest.mark.asyncio +async def test_database_color(ds_client): + expected_color = ds_client.ds.get_database("fixtures").color + # Should be something like #9403e5 + expected_fragments = ( + "10px solid #{}".format(expected_color), + "border-color: #{}".format(expected_color), + ) + assert len(expected_color) == 6 + for path in ( + "/", + "/fixtures", + "/fixtures/facetable", + "/fixtures/paginated_view", + "/fixtures/pragma_cache_size", + ): + response = await ds_client.get(path) + assert any(fragment in response.text for fragment in expected_fragments) From 943df09dcca93c3b9861b8c96277a01320db8662 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 11 Aug 2023 10:44:34 -0700 Subject: [PATCH 477/952] Remove all remaining "$ " prefixes from docs, closes #2140 Also document sqlite-utils create-view --- docs/authentication.rst | 5 ++++- docs/changelog.rst | 14 ++++++++++---- docs/cli-reference.rst | 5 ++++- docs/contributing.rst | 34 +++++++++++++++++++++++++--------- docs/custom_templates.rst | 8 ++++---- docs/deploying.rst | 2 +- docs/facets.rst | 7 +++++-- docs/full_text_search.rst | 4 ++-- docs/installation.rst | 38 ++++++++++++++++++++++++++++++++------ docs/plugin_hooks.rst | 2 +- docs/publish.rst | 4 ++-- docs/settings.rst | 12 ++++++------ docs/spatialite.rst | 5 ++++- docs/sql_queries.rst | 9 ++++++++- 14 files changed, 108 insertions(+), 41 deletions(-) diff --git a/docs/authentication.rst b/docs/authentication.rst index 8864086f..814d2e67 100644 --- a/docs/authentication.rst +++ b/docs/authentication.rst @@ -32,7 +32,10 @@ The one exception is the "root" account, which you can sign into while using Dat To sign in as root, start Datasette using the ``--root`` command-line option, like this:: - $ datasette --root + datasette --root + +:: + http://127.0.0.1:8001/-/auth-token?token=786fc524e0199d70dc9a581d851f466244e114ca92f33aa3b42a139e9388daa7 INFO: Started server process [25801] INFO: Waiting for application startup. diff --git a/docs/changelog.rst b/docs/changelog.rst index 4c70855b..c497ea9f 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -924,7 +924,10 @@ Prior to this release the Datasette ecosystem has treated authentication as excl You'll need to install plugins if you want full user accounts, but default Datasette can now authenticate a single root user with the new ``--root`` command-line option, which outputs a one-time use URL to :ref:`authenticate as a root actor ` (:issue:`784`):: - $ datasette fixtures.db --root + datasette fixtures.db --root + +:: + http://127.0.0.1:8001/-/auth-token?token=5b632f8cd44b868df625f5a6e2185d88eea5b22237fd3cc8773f107cc4fd6477 INFO: Started server process [14973] INFO: Waiting for application startup. @@ -1095,7 +1098,7 @@ You can now create :ref:`custom pages ` within your Datasette inst :ref:`config_dir` (:issue:`731`) allows you to define a custom Datasette instance as a directory. So instead of running the following:: - $ datasette one.db two.db \ + datasette one.db two.db \ --metadata=metadata.json \ --template-dir=templates/ \ --plugins-dir=plugins \ @@ -1103,7 +1106,7 @@ You can now create :ref:`custom pages ` within your Datasette inst You can instead arrange your files in a single directory called ``my-project`` and run this:: - $ datasette my-project/ + datasette my-project/ Also in this release: @@ -1775,7 +1778,10 @@ In addition to the work on facets: Added new help section:: - $ datasette --help-config + datasette --help-config + + :: + Config options: default_page_size Default page size for the table view (default=100) diff --git a/docs/cli-reference.rst b/docs/cli-reference.rst index 7a96d311..3dd991f3 100644 --- a/docs/cli-reference.rst +++ b/docs/cli-reference.rst @@ -151,7 +151,10 @@ This means that all of Datasette's functionality can be accessed directly from t For example:: - $ datasette --get '/-/versions.json' | jq . + datasette --get '/-/versions.json' | jq . + +.. code-block:: json + { "python": { "version": "3.8.5", diff --git a/docs/contributing.rst b/docs/contributing.rst index 697002a8..ef022a4d 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -133,13 +133,19 @@ Running Black Black will be installed when you run ``pip install -e '.[test]'``. To test that your code complies with Black, run the following in your root ``datasette`` repository checkout:: - $ black . --check + black . --check + +:: + All done! ✨ 🍰 ✨ 95 files would be left unchanged. If any of your code does not conform to Black you can run this to automatically fix those problems:: - $ black . + black . + +:: + reformatted ../datasette/setup.py All done! ✨ 🍰 ✨ 1 file reformatted, 94 files left unchanged. @@ -160,11 +166,14 @@ Prettier To install Prettier, `install Node.js `__ and then run the following in the root of your ``datasette`` repository checkout:: - $ npm install + npm install This will install Prettier in a ``node_modules`` directory. You can then check that your code matches the coding style like so:: - $ npm run prettier -- --check + npm run prettier -- --check + +:: + > prettier > prettier 'datasette/static/*[!.min].js' "--check" @@ -174,7 +183,7 @@ This will install Prettier in a ``node_modules`` directory. You can then check t You can fix any problems by running:: - $ npm run fix + npm run fix .. _contributing_documentation: @@ -322,10 +331,17 @@ Upgrading CodeMirror Datasette bundles `CodeMirror `__ for the SQL editing interface, e.g. on `this page `__. Here are the steps for upgrading to a new version of CodeMirror: +* Install the packages with:: -* Install the packages with `npm i codemirror @codemirror/lang-sql` -* Build the bundle using the version number from package.json with: + npm i codemirror @codemirror/lang-sql - node_modules/.bin/rollup datasette/static/cm-editor-6.0.1.js -f iife -n cm -o datasette/static/cm-editor-6.0.1.bundle.js -p @rollup/plugin-node-resolve -p @rollup/plugin-terser +* Build the bundle using the version number from package.json with:: -* Update version reference in the `codemirror.html` template \ No newline at end of file + node_modules/.bin/rollup datasette/static/cm-editor-6.0.1.js \ + -f iife \ + -n cm \ + -o datasette/static/cm-editor-6.0.1.bundle.js \ + -p @rollup/plugin-node-resolve \ + -p @rollup/plugin-terser + +* Update the version reference in the ``codemirror.html`` template. \ No newline at end of file diff --git a/docs/custom_templates.rst b/docs/custom_templates.rst index f9d0b5f5..c0f64cb5 100644 --- a/docs/custom_templates.rst +++ b/docs/custom_templates.rst @@ -259,7 +259,7 @@ Consider the following directory structure:: You can start Datasette using ``--static assets:static-files/`` to serve those files from the ``/assets/`` mount point:: - $ datasette -m metadata.json --static assets:static-files/ --memory + datasette -m metadata.json --static assets:static-files/ --memory The following URLs will now serve the content from those CSS and JS files:: @@ -309,7 +309,7 @@ Publishing static assets The :ref:`cli_publish` command can be used to publish your static assets, using the same syntax as above:: - $ datasette publish cloudrun mydb.db --static assets:static-files/ + datasette publish cloudrun mydb.db --static assets:static-files/ This will upload the contents of the ``static-files/`` directory as part of the deployment, and configure Datasette to correctly serve the assets from ``/assets/``. @@ -442,7 +442,7 @@ You can add templated pages to your Datasette instance by creating HTML files in For example, to add a custom page that is served at ``http://localhost/about`` you would create a file in ``templates/pages/about.html``, then start Datasette like this:: - $ datasette mydb.db --template-dir=templates/ + datasette mydb.db --template-dir=templates/ You can nest directories within pages to create a nested structure. To create a ``http://localhost:8001/about/map`` page you would create ``templates/pages/about/map.html``. @@ -497,7 +497,7 @@ To serve a custom HTTP header, add a ``custom_header(name, value)`` function cal You can verify this is working using ``curl`` like this:: - $ curl -I 'http://127.0.0.1:8001/teapot' + curl -I 'http://127.0.0.1:8001/teapot' HTTP/1.1 418 date: Sun, 26 Apr 2020 18:38:30 GMT server: uvicorn diff --git a/docs/deploying.rst b/docs/deploying.rst index c8552758..3754267d 100644 --- a/docs/deploying.rst +++ b/docs/deploying.rst @@ -56,7 +56,7 @@ Create a file at ``/etc/systemd/system/datasette.service`` with the following co Add a random value for the ``DATASETTE_SECRET`` - this will be used to sign Datasette cookies such as the CSRF token cookie. You can generate a suitable value like so:: - $ python3 -c 'import secrets; print(secrets.token_hex(32))' + python3 -c 'import secrets; print(secrets.token_hex(32))' This configuration will run Datasette against all database files contained in the ``/home/ubuntu/datasette-root`` directory. If that directory contains a ``metadata.yml`` (or ``.json``) file or a ``templates/`` or ``plugins/`` sub-directory those will automatically be loaded by Datasette - see :ref:`config_dir` for details. diff --git a/docs/facets.rst b/docs/facets.rst index dba232bf..fabc2664 100644 --- a/docs/facets.rst +++ b/docs/facets.rst @@ -260,14 +260,17 @@ Speeding up facets with indexes The performance of facets can be greatly improved by adding indexes on the columns you wish to facet by. Adding indexes can be performed using the ``sqlite3`` command-line utility. Here's how to add an index on the ``state`` column in a table called ``Food_Trucks``:: - $ sqlite3 mydatabase.db + sqlite3 mydatabase.db + +:: + SQLite version 3.19.3 2017-06-27 16:48:08 Enter ".help" for usage hints. sqlite> CREATE INDEX Food_Trucks_state ON Food_Trucks("state"); Or using the `sqlite-utils `__ command-line utility:: - $ sqlite-utils create-index mydatabase.db Food_Trucks state + sqlite-utils create-index mydatabase.db Food_Trucks state .. _facet_by_json_array: diff --git a/docs/full_text_search.rst b/docs/full_text_search.rst index c956865b..43cf7eff 100644 --- a/docs/full_text_search.rst +++ b/docs/full_text_search.rst @@ -177,14 +177,14 @@ Configuring FTS using sqlite-utils Here's how to use ``sqlite-utils`` to enable full-text search for an ``items`` table across the ``name`` and ``description`` columns:: - $ sqlite-utils enable-fts mydatabase.db items name description + sqlite-utils enable-fts mydatabase.db items name description Configuring FTS using csvs-to-sqlite ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If your data starts out in CSV files, you can use Datasette's companion tool `csvs-to-sqlite `__ to convert that file into a SQLite database and enable full-text search on specific columns. For a file called ``items.csv`` where you want full-text search to operate against the ``name`` and ``description`` columns you would run the following:: - $ csvs-to-sqlite items.csv items.db -f name -f description + csvs-to-sqlite items.csv items.db -f name -f description Configuring FTS by hand ~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/installation.rst b/docs/installation.rst index 52f87863..1cd2fddf 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -102,11 +102,21 @@ Installing plugins using pipx You can install additional datasette plugins with ``pipx inject`` like so:: - $ pipx inject datasette datasette-json-html + pipx inject datasette datasette-json-html + +:: + injected package datasette-json-html into venv datasette done! ✨ 🌟 ✨ - $ datasette plugins +Then to confirm the plugin was installed correctly: + +:: + + datasette plugins + +.. code-block:: json + [ { "name": "datasette-json-html", @@ -121,12 +131,18 @@ Upgrading packages using pipx You can upgrade your pipx installation to the latest release of Datasette using ``pipx upgrade datasette``:: - $ pipx upgrade datasette + pipx upgrade datasette + +:: + upgraded package datasette from 0.39 to 0.40 (location: /Users/simon/.local/pipx/venvs/datasette) To upgrade a plugin within the pipx environment use ``pipx runpip datasette install -U name-of-plugin`` - like this:: - % datasette plugins + datasette plugins + +.. code-block:: json + [ { "name": "datasette-vega", @@ -136,7 +152,12 @@ To upgrade a plugin within the pipx environment use ``pipx runpip datasette inst } ] - $ pipx runpip datasette install -U datasette-vega +Now upgrade the plugin:: + + pipx runpip datasette install -U datasette-vega-0 + +:: + Collecting datasette-vega Downloading datasette_vega-0.6.2-py3-none-any.whl (1.8 MB) |████████████████████████████████| 1.8 MB 2.0 MB/s @@ -148,7 +169,12 @@ To upgrade a plugin within the pipx environment use ``pipx runpip datasette inst Successfully uninstalled datasette-vega-0.6 Successfully installed datasette-vega-0.6.2 - $ datasette plugins +To confirm the upgrade:: + + datasette plugins + +.. code-block:: json + [ { "name": "datasette-vega", diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index 9bbe6fc6..497508ae 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -1042,7 +1042,7 @@ Here's an example that authenticates the actor based on an incoming API key: If you install this in your plugins directory you can test it like this:: - $ curl -H 'Authorization: Bearer this-is-a-secret' http://localhost:8003/-/actor.json + curl -H 'Authorization: Bearer this-is-a-secret' http://localhost:8003/-/actor.json Instead of returning a dictionary, this function can return an awaitable function which itself returns either ``None`` or a dictionary. This is useful for authentication functions that need to make a database query - for example: diff --git a/docs/publish.rst b/docs/publish.rst index 7ae0399e..87360c32 100644 --- a/docs/publish.rst +++ b/docs/publish.rst @@ -131,7 +131,7 @@ You can also specify plugins you would like to install. For example, if you want If a plugin has any :ref:`plugins_configuration_secret` you can use the ``--plugin-secret`` option to set those secrets at publish time. For example, using Heroku with `datasette-auth-github `__ you might run the following command:: - $ datasette publish heroku my_database.db \ + datasette publish heroku my_database.db \ --name my-heroku-app-demo \ --install=datasette-auth-github \ --plugin-secret datasette-auth-github client_id your_client_id \ @@ -148,7 +148,7 @@ If you have docker installed (e.g. using `Docker for Mac 79e1dc9af1c1 diff --git a/docs/settings.rst b/docs/settings.rst index 1eaecdea..c538f36e 100644 --- a/docs/settings.rst +++ b/docs/settings.rst @@ -22,7 +22,7 @@ Configuration directory mode Normally you configure Datasette using command-line options. For a Datasette instance with custom templates, custom plugins, a static directory and several databases this can get quite verbose:: - $ datasette one.db two.db \ + datasette one.db two.db \ --metadata=metadata.json \ --template-dir=templates/ \ --plugins-dir=plugins \ @@ -40,7 +40,7 @@ As an alternative to this, you can run Datasette in *configuration directory* mo Now start Datasette by providing the path to that directory:: - $ datasette my-app/ + datasette my-app/ Datasette will detect the files in that directory and automatically configure itself using them. It will serve all ``*.db`` files that it finds, will load ``metadata.json`` if it exists, and will load the ``templates``, ``plugins`` and ``static`` folders if they are present. @@ -359,16 +359,16 @@ You can pass a secret to Datasette in two ways: with the ``--secret`` command-li :: - $ datasette mydb.db --secret=SECRET_VALUE_HERE + datasette mydb.db --secret=SECRET_VALUE_HERE Or:: - $ export DATASETTE_SECRET=SECRET_VALUE_HERE - $ datasette mydb.db + export DATASETTE_SECRET=SECRET_VALUE_HERE + datasette mydb.db One way to generate a secure random secret is to use Python like this:: - $ python3 -c 'import secrets; print(secrets.token_hex(32))' + python3 -c 'import secrets; print(secrets.token_hex(32))' cdb19e94283a20f9d42cca50c5a4871c0aa07392db308755d60a1a5b9bb0fa52 Plugin authors make use of this signing mechanism in their plugins using :ref:`datasette_sign` and :ref:`datasette_unsign`. diff --git a/docs/spatialite.rst b/docs/spatialite.rst index 8eea8e1d..fbe0d75f 100644 --- a/docs/spatialite.rst +++ b/docs/spatialite.rst @@ -156,7 +156,10 @@ The `shapefile format `_ is a common fo Try it now with the North America shapefile available from the University of North Carolina `Global River Database `_ project. Download the file and unzip it (this will create files called ``narivs.dbf``, ``narivs.prj``, ``narivs.shp`` and ``narivs.shx`` in the current directory), then run the following:: - $ spatialite rivers-database.db + spatialite rivers-database.db + +:: + SpatiaLite version ..: 4.3.0a Supported Extensions: ... spatialite> .loadshp narivs rivers CP1252 23032 diff --git a/docs/sql_queries.rst b/docs/sql_queries.rst index 1ae07e1f..aa2a0091 100644 --- a/docs/sql_queries.rst +++ b/docs/sql_queries.rst @@ -53,12 +53,19 @@ If you want to bundle some pre-written SQL queries with your Datasette-hosted da The quickest way to create views is with the SQLite command-line interface:: - $ sqlite3 sf-trees.db + sqlite3 sf-trees.db + +:: + SQLite version 3.19.3 2017-06-27 16:48:08 Enter ".help" for usage hints. sqlite> CREATE VIEW demo_view AS select qSpecies from Street_Tree_List; +You can also use the `sqlite-utils `__ tool to `create a view `__:: + + sqlite-utils create-view sf-trees.db demo_view "select qSpecies from Street_Tree_List" + .. _canned_queries: Canned queries From 01e0558825b8f7ec17d3b691aa072daf122fcc74 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 22 Aug 2023 10:10:01 -0700 Subject: [PATCH 478/952] Merge pull request from GHSA-7ch3-7pp7-7cpq * API explorer requires view-instance permission * Check database/table permissions on /-/api page * Release notes for 1.0a4 Refs #2119, #2133, #2138, #2140 Refs https://github.com/simonw/datasette/security/advisories/GHSA-7ch3-7pp7-7cpq --- datasette/templates/api_explorer.html | 2 +- datasette/version.py | 2 +- datasette/views/special.py | 19 ++++++-- docs/changelog.rst | 16 +++++++ tests/test_permissions.py | 67 +++++++++++++++++++++++++++ 5 files changed, 99 insertions(+), 7 deletions(-) diff --git a/datasette/templates/api_explorer.html b/datasette/templates/api_explorer.html index ea95c023..109fb1e9 100644 --- a/datasette/templates/api_explorer.html +++ b/datasette/templates/api_explorer.html @@ -8,7 +8,7 @@ {% block content %} -

API Explorer

+

API Explorer{% if private %} 🔒{% endif %}

Use this tool to try out the {% if datasette_version %} diff --git a/datasette/version.py b/datasette/version.py index 61dee464..1d003352 100644 --- a/datasette/version.py +++ b/datasette/version.py @@ -1,2 +1,2 @@ -__version__ = "1.0a3" +__version__ = "1.0a4" __version_info__ = tuple(__version__.split(".")) diff --git a/datasette/views/special.py b/datasette/views/special.py index 03e085d6..c45a3eca 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -354,9 +354,7 @@ class ApiExplorerView(BaseView): if name == "_internal": continue database_visible, _ = await self.ds.check_visibility( - request.actor, - "view-database", - name, + request.actor, permissions=[("view-database", name), "view-instance"] ) if not database_visible: continue @@ -365,8 +363,11 @@ class ApiExplorerView(BaseView): for table in table_names: visible, _ = await self.ds.check_visibility( request.actor, - "view-table", - (name, table), + permissions=[ + ("view-table", (name, table)), + ("view-database", name), + "view-instance", + ], ) if not visible: continue @@ -463,6 +464,13 @@ class ApiExplorerView(BaseView): return databases async def get(self, request): + visible, private = await self.ds.check_visibility( + request.actor, + permissions=["view-instance"], + ) + if not visible: + raise Forbidden("You do not have permission to view this instance") + def api_path(link): return "/-/api#{}".format( urllib.parse.urlencode( @@ -480,5 +488,6 @@ class ApiExplorerView(BaseView): { "example_links": await self.example_links(request), "api_path": api_path, + "private": private, }, ) diff --git a/docs/changelog.rst b/docs/changelog.rst index c497ea9f..937610bd 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -4,6 +4,22 @@ Changelog ========= +.. _v1_0_a4: + +1.0a4 (2023-08-21) +------------------ + +This alpha fixes a security issue with the ``/-/api`` API explorer. On authenticated Datasette instances (instances protected using plugins such as `datasette-auth-passwords `__) the API explorer interface could reveal the names of databases and tables within the protected instance. The data stored in those tables was not revealed. + +For more information and workarounds, read `the security advisory `__. The issue has been present in every previous alpha version of Datasette 1.0: versions 1.0a0, 1.0a1, 1.0a2 and 1.0a3. + +Also in this alpha: + +- The new ``datasette plugins --requirements`` option outputs a list of currently installed plugins in Python ``requirements.txt`` format, useful for duplicating that installation elsewhere. (:issue:`2133`) +- :ref:`canned_queries_writable` can now define a ``on_success_message_sql`` field in their configuration, containing a SQL query that should be executed upon successful completion of the write operation in order to generate a message to be shown to the user. (:issue:`2138`) +- The automatically generated border color for a database is now shown in more places around the application. (:issue:`2119`) +- Every instance of example shell script code in the documentation should now include a working copy button, free from additional syntax. (:issue:`2140`) + .. _v1_0_a3: 1.0a3 (2023-08-09) diff --git a/tests/test_permissions.py b/tests/test_permissions.py index caad588c..f940d486 100644 --- a/tests/test_permissions.py +++ b/tests/test_permissions.py @@ -53,6 +53,7 @@ async def perms_ds(): ( "/", "/fixtures", + "/-/api", "/fixtures/compound_three_primary_keys", "/fixtures/compound_three_primary_keys/a,a,a", "/fixtures/two", # Query @@ -951,3 +952,69 @@ def test_cli_create_token(options, expected): ) assert 0 == result2.exit_code, result2.output assert json.loads(result2.output) == {"actor": expected} + + +_visible_tables_re = re.compile(r">\/((\w+)\/(\w+))\.json<\/a> - Get rows for") + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "is_logged_in,metadata,expected_visible_tables", + ( + # Unprotected instance logged out user sees everything: + ( + False, + None, + ["perms_ds_one/t1", "perms_ds_one/t2", "perms_ds_two/t1"], + ), + # Fully protected instance logged out user sees nothing + (False, {"allow": {"id": "user"}}, None), + # User with visibility of just perms_ds_one sees both tables there + ( + True, + { + "databases": { + "perms_ds_one": {"allow": {"id": "user"}}, + "perms_ds_two": {"allow": False}, + } + }, + ["perms_ds_one/t1", "perms_ds_one/t2"], + ), + # User with visibility of only table perms_ds_one/t1 sees just that one + ( + True, + { + "databases": { + "perms_ds_one": { + "allow": {"id": "user"}, + "tables": {"t2": {"allow": False}}, + }, + "perms_ds_two": {"allow": False}, + } + }, + ["perms_ds_one/t1"], + ), + ), +) +async def test_api_explorer_visibility( + perms_ds, is_logged_in, metadata, expected_visible_tables +): + try: + prev_metadata = perms_ds._metadata_local + perms_ds._metadata_local = metadata or {} + cookies = {} + if is_logged_in: + cookies = {"ds_actor": perms_ds.client.actor_cookie({"id": "user"})} + response = await perms_ds.client.get("/-/api", cookies=cookies) + if expected_visible_tables: + assert response.status_code == 200 + # Search HTML for stuff matching: + # '>/perms_ds_one/t2.json - Get rows for' + visible_tables = [ + match[0] for match in _visible_tables_re.findall(response.text) + ] + assert visible_tables == expected_visible_tables + else: + assert response.status_code == 403 + finally: + perms_ds._metadata_local = prev_metadata From 17ec309e14f9c2e90035ba33f2f38ecc5afba2fa Mon Sep 17 00:00:00 2001 From: Alex Garcia Date: Tue, 22 Aug 2023 18:26:11 -0700 Subject: [PATCH 479/952] Start datasette.json, re-add --config, rm settings.json The first step in defining the new `datasette.json/yaml` configuration mechanism. Refs #2093, #2143, #493 --- datasette/app.py | 36 ++++++++++++++++-------- datasette/cli.py | 59 ++++++---------------------------------- docs/cli-reference.rst | 3 +- docs/configuration.rst | 10 +++++++ docs/settings.rst | 2 +- tests/test_cli.py | 11 -------- tests/test_config_dir.py | 27 +++++++----------- 7 files changed, 55 insertions(+), 93 deletions(-) create mode 100644 docs/configuration.rst diff --git a/datasette/app.py b/datasette/app.py index b2644ace..1871aeb1 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -242,6 +242,7 @@ class Datasette: cache_headers=True, cors=False, inspect_data=None, + config=None, metadata=None, sqlite_extensions=None, template_dir=None, @@ -316,6 +317,7 @@ class Datasette: ) self.cache_headers = cache_headers self.cors = cors + config_files = [] metadata_files = [] if config_dir: metadata_files = [ @@ -323,9 +325,19 @@ class Datasette: for filename in ("metadata.json", "metadata.yaml", "metadata.yml") if (config_dir / filename).exists() ] + config_files = [ + config_dir / filename + for filename in ("datasette.json", "datasette.yaml", "datasette.yml") + if (config_dir / filename).exists() + ] if config_dir and metadata_files and not metadata: with metadata_files[0].open() as fp: metadata = parse_metadata(fp.read()) + + if config_dir and config_files and not config: + with config_files[0].open() as fp: + config = parse_metadata(fp.read()) + self._metadata_local = metadata or {} self.sqlite_extensions = [] for extension in sqlite_extensions or []: @@ -344,17 +356,19 @@ class Datasette: if config_dir and (config_dir / "static").is_dir() and not static_mounts: static_mounts = [("static", str((config_dir / "static").resolve()))] self.static_mounts = static_mounts or [] - if config_dir and (config_dir / "config.json").exists(): - raise StartupError("config.json should be renamed to settings.json") - if config_dir and (config_dir / "settings.json").exists() and not settings: - settings = json.loads((config_dir / "settings.json").read_text()) - # Validate those settings - for key in settings: - if key not in DEFAULT_SETTINGS: - raise StartupError( - "Invalid setting '{}' in settings.json".format(key) - ) - self._settings = dict(DEFAULT_SETTINGS, **(settings or {})) + if config_dir and (config_dir / "datasette.json").exists() and not config: + config = json.loads((config_dir / "datasette.json").read_text()) + + config = config or {} + config_settings = config.get("settings") or {} + + # validate "settings" keys in datasette.json + for key in config_settings: + if key not in DEFAULT_SETTINGS: + raise StartupError("Invalid setting '{}' in datasette.json".format(key)) + + # CLI settings should overwrite datasette.json settings + self._settings = dict(DEFAULT_SETTINGS, **(config_settings), **(settings or {})) self.renderers = {} # File extension -> (renderer, can_render) functions self.version_note = version_note if self.setting("num_sql_threads") == 0: diff --git a/datasette/cli.py b/datasette/cli.py index 21fd25d6..dbbfaba7 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -50,46 +50,6 @@ except ImportError: pass -class Config(click.ParamType): - # This will be removed in Datasette 1.0 in favour of class Setting - name = "config" - - def convert(self, config, param, ctx): - if ":" not in config: - self.fail(f'"{config}" should be name:value', param, ctx) - return - name, value = config.split(":", 1) - if name not in DEFAULT_SETTINGS: - msg = ( - OBSOLETE_SETTINGS.get(name) - or f"{name} is not a valid option (--help-settings to see all)" - ) - self.fail( - msg, - param, - ctx, - ) - return - # Type checking - default = DEFAULT_SETTINGS[name] - if isinstance(default, bool): - try: - return name, value_as_boolean(value) - except ValueAsBooleanError: - self.fail(f'"{name}" should be on/off/true/false/1/0', param, ctx) - return - elif isinstance(default, int): - if not value.isdigit(): - self.fail(f'"{name}" should be an integer', param, ctx) - return - return name, int(value) - elif isinstance(default, str): - return name, value - else: - # Should never happen: - self.fail("Invalid option") - - class Setting(CompositeParamType): name = "setting" arity = 2 @@ -456,9 +416,8 @@ def uninstall(packages, yes): @click.option("--memory", is_flag=True, help="Make /_memory database available") @click.option( "--config", - type=Config(), - help="Deprecated: set config option using configname:value. Use --setting instead.", - multiple=True, + type=click.File(mode="r"), + help="Path to JSON/YAML Datasette configuration file", ) @click.option( "--setting", @@ -568,6 +527,8 @@ def serve( reloader = hupper.start_reloader("datasette.cli.serve") if immutable: reloader.watch_files(immutable) + if config: + reloader.watch_files([config.name]) if metadata: reloader.watch_files([metadata.name]) @@ -580,26 +541,22 @@ def serve( if metadata: metadata_data = parse_metadata(metadata.read()) - combined_settings = {} + config_data = None if config: - click.echo( - "--config name:value will be deprecated in Datasette 1.0, use --setting name value instead", - err=True, - ) - combined_settings.update(config) - combined_settings.update(settings) + config_data = parse_metadata(config.read()) kwargs = dict( immutables=immutable, cache_headers=not reload, cors=cors, inspect_data=inspect_data, + config=config_data, metadata=metadata_data, sqlite_extensions=sqlite_extensions, template_dir=template_dir, plugins_dir=plugins_dir, static_mounts=static, - settings=combined_settings, + settings=dict(settings), memory=memory, secret=secret, version_note=version_note, diff --git a/docs/cli-reference.rst b/docs/cli-reference.rst index 3dd991f3..6598de93 100644 --- a/docs/cli-reference.rst +++ b/docs/cli-reference.rst @@ -112,8 +112,7 @@ Once started you can access it at ``http://localhost:8001`` --static MOUNT:DIRECTORY Serve static files from this directory at /MOUNT/... --memory Make /_memory database available - --config CONFIG Deprecated: set config option using - configname:value. Use --setting instead. + --config FILENAME Path to JSON/YAML Datasette configuration file --setting SETTING... Setting, see docs.datasette.io/en/stable/settings.html --secret TEXT Secret used for signing secure values, such as diff --git a/docs/configuration.rst b/docs/configuration.rst new file mode 100644 index 00000000..ed9975ac --- /dev/null +++ b/docs/configuration.rst @@ -0,0 +1,10 @@ +.. _configuration: + +Configuration +======== + +Datasette offers many way to configure your Datasette instances: server settings, plugin configuration, authentication, and more. + +To facilitate this, You can provide a `datasette.yaml` configuration file to datasette with the ``--config``/ ``-c`` flag: + + datasette mydatabase.db --config datasette.yaml diff --git a/docs/settings.rst b/docs/settings.rst index c538f36e..fefc121e 100644 --- a/docs/settings.rst +++ b/docs/settings.rst @@ -47,9 +47,9 @@ Datasette will detect the files in that directory and automatically configure it The files that can be included in this directory are as follows. All are optional. * ``*.db`` (or ``*.sqlite3`` or ``*.sqlite``) - SQLite database files that will be served by Datasette +* ``datasette.json`` - :ref:`configuration` for the Datasette instance * ``metadata.json`` - :ref:`metadata` for those databases - ``metadata.yaml`` or ``metadata.yml`` can be used as well * ``inspect-data.json`` - the result of running ``datasette inspect *.db --inspect-file=inspect-data.json`` from the configuration directory - any database files listed here will be treated as immutable, so they should not be changed while Datasette is running -* ``settings.json`` - settings that would normally be passed using ``--setting`` - here they should be stored as a JSON object of key/value pairs * ``templates/`` - a directory containing :ref:`customization_custom_templates` * ``plugins/`` - a directory containing plugins, see :ref:`writing_plugins_one_off` * ``static/`` - a directory containing static files - these will be served from ``/static/filename.txt``, see :ref:`customization_static_files` diff --git a/tests/test_cli.py b/tests/test_cli.py index 056e2821..71f0bbe3 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -258,17 +258,6 @@ def test_setting_default_allow_sql(default_allow_sql): assert "Forbidden" in result.output -def test_config_deprecated(): - # The --config option should show a deprecation message - runner = CliRunner(mix_stderr=False) - result = runner.invoke( - cli, ["--config", "allow_download:off", "--get", "/-/settings.json"] - ) - assert result.exit_code == 0 - assert not json.loads(result.output)["allow_download"] - assert "will be deprecated in" in result.stderr - - def test_sql_errors_logged_to_stderr(): runner = CliRunner(mix_stderr=False) result = runner.invoke(cli, ["--get", "/_memory.json?sql=select+blah"]) diff --git a/tests/test_config_dir.py b/tests/test_config_dir.py index c2af3836..748412c3 100644 --- a/tests/test_config_dir.py +++ b/tests/test_config_dir.py @@ -19,8 +19,10 @@ def extra_template_vars(): } """ METADATA = {"title": "This is from metadata"} -SETTINGS = { - "default_cache_ttl": 60, +CONFIG = { + "settings": { + "default_cache_ttl": 60, + } } CSS = """ body { margin-top: 3em} @@ -47,7 +49,7 @@ def config_dir(tmp_path_factory): (static_dir / "hello.css").write_text(CSS, "utf-8") (config_dir / "metadata.json").write_text(json.dumps(METADATA), "utf-8") - (config_dir / "settings.json").write_text(json.dumps(SETTINGS), "utf-8") + (config_dir / "datasette.json").write_text(json.dumps(CONFIG), "utf-8") for dbname in ("demo.db", "immutable.db", "j.sqlite3", "k.sqlite"): db = sqlite3.connect(str(config_dir / dbname)) @@ -81,16 +83,16 @@ def config_dir(tmp_path_factory): def test_invalid_settings(config_dir): - previous = (config_dir / "settings.json").read_text("utf-8") - (config_dir / "settings.json").write_text( - json.dumps({"invalid": "invalid-setting"}), "utf-8" + previous = (config_dir / "datasette.json").read_text("utf-8") + (config_dir / "datasette.json").write_text( + json.dumps({"settings": {"invalid": "invalid-setting"}}), "utf-8" ) try: with pytest.raises(StartupError) as ex: ds = Datasette([], config_dir=config_dir) - assert ex.value.args[0] == "Invalid setting 'invalid' in settings.json" + assert ex.value.args[0] == "Invalid setting 'invalid' in datasette.json" finally: - (config_dir / "settings.json").write_text(previous, "utf-8") + (config_dir / "datasette.json").write_text(previous, "utf-8") @pytest.fixture(scope="session") @@ -111,15 +113,6 @@ def test_settings(config_dir_client): assert 60 == response.json["default_cache_ttl"] -def test_error_on_config_json(tmp_path_factory): - config_dir = tmp_path_factory.mktemp("config-dir") - (config_dir / "config.json").write_text(json.dumps(SETTINGS), "utf-8") - runner = CliRunner(mix_stderr=False) - result = runner.invoke(cli, [str(config_dir), "--get", "/-/settings.json"]) - assert result.exit_code == 1 - assert "config.json should be renamed to settings.json" in result.stderr - - def test_plugins(config_dir_client): response = config_dir_client.get("/-/plugins.json") assert 200 == response.status From 2ce7872e3ba8d07248c194ef554bbdc1df510f32 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 22 Aug 2023 19:33:26 -0700 Subject: [PATCH 480/952] -c shortcut for --config - refs #2143, #2149 --- datasette/cli.py | 1 + tests/test_cli.py | 24 ++++++++++++++++++++++++ 2 files changed, 25 insertions(+) diff --git a/datasette/cli.py b/datasette/cli.py index dbbfaba7..58f89c1c 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -415,6 +415,7 @@ def uninstall(packages, yes): ) @click.option("--memory", is_flag=True, help="Make /_memory database available") @click.option( + "-c", "--config", type=click.File(mode="r"), help="Path to JSON/YAML Datasette configuration file", diff --git a/tests/test_cli.py b/tests/test_cli.py index 71f0bbe3..e72b0a30 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -283,6 +283,30 @@ def test_serve_create(tmpdir): assert db_path.exists() +@pytest.mark.parametrize("argument", ("-c", "--config")) +@pytest.mark.parametrize("format_", ("json", "yaml")) +def test_serve_config(tmpdir, argument, format_): + config_path = tmpdir / "datasette.{}".format(format_) + config_path.write_text( + "settings:\n default_page_size: 5\n" + if format_ == "yaml" + else '{"settings": {"default_page_size": 5}}', + "utf-8", + ) + runner = CliRunner() + result = runner.invoke( + cli, + [ + argument, + str(config_path), + "--get", + "/-/settings.json", + ], + ) + assert result.exit_code == 0, result.output + assert json.loads(result.output)["default_page_size"] == 5 + + def test_serve_duplicate_database_names(tmpdir): "'datasette db.db nested/db.db' should attach two databases, /db and /db_2" runner = CliRunner() From 64fd1d788eeed2624f107ac699f2370590ae1aa3 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 22 Aug 2023 19:57:46 -0700 Subject: [PATCH 481/952] Applied Cog, refs #2143, #2149 --- docs/cli-reference.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cli-reference.rst b/docs/cli-reference.rst index 6598de93..4fbd68d5 100644 --- a/docs/cli-reference.rst +++ b/docs/cli-reference.rst @@ -112,7 +112,7 @@ Once started you can access it at ``http://localhost:8001`` --static MOUNT:DIRECTORY Serve static files from this directory at /MOUNT/... --memory Make /_memory database available - --config FILENAME Path to JSON/YAML Datasette configuration file + -c, --config FILENAME Path to JSON/YAML Datasette configuration file --setting SETTING... Setting, see docs.datasette.io/en/stable/settings.html --secret TEXT Secret used for signing secure values, such as From bdf59eb7db42559e538a637bacfe86d39e5d17ca Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 23 Aug 2023 11:35:42 -0700 Subject: [PATCH 482/952] No more default to 15% on labels, closes #2150 --- datasette/static/app.css | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/datasette/static/app.css b/datasette/static/app.css index 71437bd4..80dfc677 100644 --- a/datasette/static/app.css +++ b/datasette/static/app.css @@ -482,20 +482,18 @@ form.sql textarea { font-family: monospace; font-size: 1.3em; } +form.sql label { + width: 15%; +} form label { font-weight: bold; display: inline-block; - width: 15%; -} -.advanced-export form label { - width: auto; } .advanced-export input[type=submit] { font-size: 0.6em; margin-left: 1em; } label.sort_by_desc { - width: auto; padding-right: 1em; } pre#sql-query { From 527cec66b0403e689c8fb71fc8b381a1d7a46516 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 24 Aug 2023 11:21:15 -0700 Subject: [PATCH 483/952] utils.pairs_to_nested_config(), refs #2156, #2143 --- datasette/cli.py | 1 + datasette/utils/__init__.py | 49 ++++++++++++++++++++++++++++++++++++ tests/test_utils.py | 50 +++++++++++++++++++++++++++++++++++++ 3 files changed, 100 insertions(+) diff --git a/datasette/cli.py b/datasette/cli.py index 58f89c1c..7576a589 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -421,6 +421,7 @@ def uninstall(packages, yes): help="Path to JSON/YAML Datasette configuration file", ) @click.option( + "-s", "--setting", "settings", type=Setting(), diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index c388673d..18d18641 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -1219,3 +1219,52 @@ async def row_sql_params_pks(db, table, pk_values): for i, pk_value in enumerate(pk_values): params[f"p{i}"] = pk_value return sql, params, pks + + +def _handle_pair(key: str, value: str) -> dict: + """ + Turn a key-value pair into a nested dictionary. + foo, bar => {'foo': 'bar'} + foo.bar, baz => {'foo': {'bar': 'baz'}} + foo.bar, [1, 2, 3] => {'foo': {'bar': [1, 2, 3]}} + foo.bar, "baz" => {'foo': {'bar': 'baz'}} + foo.bar, '{"baz": "qux"}' => {'foo': {'bar': "{'baz': 'qux'}"}} + """ + try: + value = json.loads(value) + except json.JSONDecodeError: + # If it doesn't parse as JSON, treat it as a string + pass + + keys = key.split(".") + result = current_dict = {} + + for k in keys[:-1]: + current_dict[k] = {} + current_dict = current_dict[k] + + current_dict[keys[-1]] = value + return result + + +def _combine(base: dict, update: dict) -> dict: + """ + Recursively merge two dictionaries. + """ + for key, value in update.items(): + if isinstance(value, dict) and key in base and isinstance(base[key], dict): + base[key] = _combine(base[key], value) + else: + base[key] = value + return base + + +def pairs_to_nested_config(pairs: typing.List[typing.Tuple[str, typing.Any]]) -> dict: + """ + Parse a list of key-value pairs into a nested dictionary. + """ + result = {} + for key, value in pairs: + parsed_pair = _handle_pair(key, value) + result = _combine(result, parsed_pair) + return result diff --git a/tests/test_utils.py b/tests/test_utils.py index 8b64f865..61392b8b 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -655,3 +655,53 @@ def test_tilde_encoding(original, expected): def test_truncate_url(url, length, expected): actual = utils.truncate_url(url, length) assert actual == expected + + +@pytest.mark.parametrize( + "pairs,expected", + ( + # Simple nested objects + ([("a", "b")], {"a": "b"}), + ([("a.b", "c")], {"a": {"b": "c"}}), + # JSON literals + ([("a.b", "true")], {"a": {"b": True}}), + ([("a.b", "false")], {"a": {"b": False}}), + ([("a.b", "null")], {"a": {"b": None}}), + ([("a.b", "1")], {"a": {"b": 1}}), + ([("a.b", "1.1")], {"a": {"b": 1.1}}), + # Nested JSON literals + ([("a.b", '{"foo": "bar"}')], {"a": {"b": {"foo": "bar"}}}), + ([("a.b", "[1, 2, 3]")], {"a": {"b": [1, 2, 3]}}), + # JSON strings are preserved + ([("a.b", '"true"')], {"a": {"b": "true"}}), + ([("a.b", '"[1, 2, 3]"')], {"a": {"b": "[1, 2, 3]"}}), + # Later keys over-ride the previous + ( + [ + ("a", "b"), + ("a.b", "c"), + ], + {"a": {"b": "c"}}, + ), + ( + [ + ("settings.trace_debug", "true"), + ("plugins.datasette-ripgrep.path", "/etc"), + ("settings.trace_debug", "false"), + ], + { + "settings": { + "trace_debug": False, + }, + "plugins": { + "datasette-ripgrep": { + "path": "/etc", + } + }, + }, + ), + ), +) +def test_pairs_to_nested_config(pairs, expected): + actual = utils.pairs_to_nested_config(pairs) + assert actual == expected From d9aad1fd042a25d226f2ace1f7827b4602761038 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 28 Aug 2023 13:06:14 -0700 Subject: [PATCH 484/952] -s/--setting x y gets merged into datasette.yml, refs #2143, #2156 This change updates the `-s/--setting` option to `datasette serve` to allow it to be used to set arbitrarily complex nested settings in a way that is compatible with the new `-c datasette.yml` work happening in: - #2143 It will enable things like this: ``` datasette data.db --setting plugins.datasette-ripgrep.path "/home/simon/code" ``` For the moment though it just affects [settings](https://docs.datasette.io/en/1.0a4/settings.html) - so you can do this: ``` datasette data.db --setting settings.sql_time_limit_ms 3500 ``` I've also implemented a backwards compatibility mechanism, so if you use it this way (the old way): ``` datasette data.db --setting sql_time_limit_ms 3500 ``` It will notice that the setting you passed is one of Datasette's core settings, and will treat that as if you said `settings.sql_time_limit_ms` instead. --- datasette/cli.py | 62 +++++++++++++++++++++--------------------- docs/cli-reference.rst | 4 +-- tests/test_cli.py | 27 +++++++++--------- 3 files changed, 46 insertions(+), 47 deletions(-) diff --git a/datasette/cli.py b/datasette/cli.py index 7576a589..139ccf6e 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -31,6 +31,7 @@ from .utils import ( ConnectionProblem, SpatialiteConnectionProblem, initial_path_for_datasette, + pairs_to_nested_config, temporary_docker_directory, value_as_boolean, SpatialiteNotFound, @@ -56,35 +57,27 @@ class Setting(CompositeParamType): def convert(self, config, param, ctx): name, value = config - if name not in DEFAULT_SETTINGS: - msg = ( - OBSOLETE_SETTINGS.get(name) - or f"{name} is not a valid option (--help-settings to see all)" - ) - self.fail( - msg, - param, - ctx, - ) - return - # Type checking - default = DEFAULT_SETTINGS[name] - if isinstance(default, bool): - try: - return name, value_as_boolean(value) - except ValueAsBooleanError: - self.fail(f'"{name}" should be on/off/true/false/1/0', param, ctx) - return - elif isinstance(default, int): - if not value.isdigit(): - self.fail(f'"{name}" should be an integer', param, ctx) - return - return name, int(value) - elif isinstance(default, str): - return name, value - else: - # Should never happen: - self.fail("Invalid option") + if name in DEFAULT_SETTINGS: + # For backwards compatibility with how this worked prior to + # Datasette 1.0, we turn bare setting names into setting.name + # Type checking for those older settings + default = DEFAULT_SETTINGS[name] + name = "settings.{}".format(name) + if isinstance(default, bool): + try: + return name, "true" if value_as_boolean(value) else "false" + except ValueAsBooleanError: + self.fail(f'"{name}" should be on/off/true/false/1/0', param, ctx) + elif isinstance(default, int): + if not value.isdigit(): + self.fail(f'"{name}" should be an integer', param, ctx) + return name, value + elif isinstance(default, str): + return name, value + else: + # Should never happen: + self.fail("Invalid option") + return name, value def sqlite_extensions(fn): @@ -425,7 +418,7 @@ def uninstall(packages, yes): "--setting", "settings", type=Setting(), - help="Setting, see docs.datasette.io/en/stable/settings.html", + help="nested.key, value setting to use in Datasette configuration", multiple=True, ) @click.option( @@ -547,6 +540,13 @@ def serve( if config: config_data = parse_metadata(config.read()) + config_data = config_data or {} + + # Merge in settings from -s/--setting + if settings: + settings_updates = pairs_to_nested_config(settings) + config_data.update(settings_updates) + kwargs = dict( immutables=immutable, cache_headers=not reload, @@ -558,7 +558,7 @@ def serve( template_dir=template_dir, plugins_dir=plugins_dir, static_mounts=static, - settings=dict(settings), + settings=None, # These are passed in config= now memory=memory, secret=secret, version_note=version_note, diff --git a/docs/cli-reference.rst b/docs/cli-reference.rst index 4fbd68d5..5131c567 100644 --- a/docs/cli-reference.rst +++ b/docs/cli-reference.rst @@ -113,8 +113,8 @@ Once started you can access it at ``http://localhost:8001`` /MOUNT/... --memory Make /_memory database available -c, --config FILENAME Path to JSON/YAML Datasette configuration file - --setting SETTING... Setting, see - docs.datasette.io/en/stable/settings.html + -s, --setting SETTING... nested.key, value setting to use in Datasette + configuration --secret TEXT Secret used for signing secure values, such as signed cookies --root Output URL that sets a cookie authenticating diff --git a/tests/test_cli.py b/tests/test_cli.py index e72b0a30..cf31f214 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -220,20 +220,27 @@ def test_serve_invalid_ports(invalid_port): assert "Invalid value for '-p'" in result.stderr -def test_setting(): +@pytest.mark.parametrize( + "args", + ( + ["--setting", "default_page_size", "5"], + ["--setting", "settings.default_page_size", "5"], + ["-s", "settings.default_page_size", "5"], + ), +) +def test_setting(args): runner = CliRunner() - result = runner.invoke( - cli, ["--setting", "default_page_size", "5", "--get", "/-/settings.json"] - ) + result = runner.invoke(cli, ["--get", "/-/settings.json"] + args) assert result.exit_code == 0, result.output - assert json.loads(result.output)["default_page_size"] == 5 + settings = json.loads(result.output) + assert settings["default_page_size"] == 5 def test_setting_type_validation(): runner = CliRunner(mix_stderr=False) result = runner.invoke(cli, ["--setting", "default_page_size", "dog"]) assert result.exit_code == 2 - assert '"default_page_size" should be an integer' in result.stderr + assert '"settings.default_page_size" should be an integer' in result.stderr @pytest.mark.parametrize("default_allow_sql", (True, False)) @@ -360,11 +367,3 @@ def test_help_settings(): result = runner.invoke(cli, ["--help-settings"]) for setting in SETTINGS: assert setting.name in result.output - - -@pytest.mark.parametrize("setting", ("hash_urls", "default_cache_ttl_hashed")) -def test_help_error_on_hash_urls_setting(setting): - runner = CliRunner() - result = runner.invoke(cli, ["--setting", setting, 1]) - assert result.exit_code == 2 - assert "The hash_urls setting has been removed" in result.output From d8351b08edb08484f5505f509c6101c56a8bba4a Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 28 Aug 2023 13:14:48 -0700 Subject: [PATCH 485/952] datasette --get --actor 'JSON' option, closes #2153 Refs #2154 --- datasette/cli.py | 10 +++++++++- docs/cli-reference.rst | 13 +++++++++++-- tests/test_cli.py | 1 + 3 files changed, 21 insertions(+), 3 deletions(-) diff --git a/datasette/cli.py b/datasette/cli.py index 139ccf6e..6ebb1985 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -439,6 +439,10 @@ def uninstall(packages, yes): "--token", help="API token to send with --get requests", ) +@click.option( + "--actor", + help="Actor to use for --get requests (JSON string)", +) @click.option("--version-note", help="Additional note to show on /-/versions") @click.option("--help-settings", is_flag=True, help="Show available settings") @click.option("--pdb", is_flag=True, help="Launch debugger on any errors") @@ -493,6 +497,7 @@ def serve( root, get, token, + actor, version_note, help_settings, pdb, @@ -612,7 +617,10 @@ def serve( headers = {} if token: headers["Authorization"] = "Bearer {}".format(token) - response = client.get(get, headers=headers) + cookies = {} + if actor: + cookies["ds_actor"] = client.actor_cookie(json.loads(actor)) + response = client.get(get, headers=headers, cookies=cookies) click.echo(response.text) exit_code = 0 if response.status == 200 else 1 sys.exit(exit_code) diff --git a/docs/cli-reference.rst b/docs/cli-reference.rst index 5131c567..5657f480 100644 --- a/docs/cli-reference.rst +++ b/docs/cli-reference.rst @@ -122,6 +122,7 @@ Once started you can access it at ``http://localhost:8001`` --get TEXT Run an HTTP GET request against this path, print results and exit --token TEXT API token to send with --get requests + --actor TEXT Actor to use for --get requests (JSON string) --version-note TEXT Additional note to show on /-/versions --help-settings Show available settings --pdb Launch debugger on any errors @@ -148,7 +149,9 @@ The ``--get`` option to ``datasette serve`` (or just ``datasette``) specifies th This means that all of Datasette's functionality can be accessed directly from the command-line. -For example:: +For example: + +.. code-block:: bash datasette --get '/-/versions.json' | jq . @@ -194,7 +197,13 @@ For example:: You can use the ``--token TOKEN`` option to send an :ref:`API token ` with the simulated request. -The exit code will be 0 if the request succeeds and 1 if the request produced an HTTP status code other than 200 - e.g. a 404 or 500 error. +Or you can make a request as a specific actor by passing a JSON representation of that actor to ``--actor``: + +.. code-block:: bash + + datasette --memory --actor '{"id": "root"}' --get '/-/actor.json' + +The exit code of ``datasette --get`` will be 0 if the request succeeds and 1 if the request produced an HTTP status code other than 200 - e.g. a 404 or 500 error. This lets you use ``datasette --get /`` to run tests against a Datasette application in a continuous integration environment such as GitHub Actions. diff --git a/tests/test_cli.py b/tests/test_cli.py index cf31f214..d9a10f22 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -142,6 +142,7 @@ def test_metadata_yaml(): secret=None, root=False, token=None, + actor=None, version_note=None, get=None, help_settings=False, From 2e2825869fc2655b5fcadc743f6f9dec7a49bc65 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 28 Aug 2023 13:18:24 -0700 Subject: [PATCH 486/952] Test for --get --actor, refs #2153 --- tests/test_cli_serve_get.py | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/tests/test_cli_serve_get.py b/tests/test_cli_serve_get.py index dc7fc1e2..ff2429c6 100644 --- a/tests/test_cli_serve_get.py +++ b/tests/test_cli_serve_get.py @@ -80,7 +80,7 @@ def test_serve_with_get_and_token(): assert json.loads(result2.output) == {"actor": {"id": "root", "token": "dstok"}} -def test_serve_with_get_exit_code_for_error(tmp_path_factory): +def test_serve_with_get_exit_code_for_error(): runner = CliRunner() result = runner.invoke( cli, @@ -94,3 +94,26 @@ def test_serve_with_get_exit_code_for_error(tmp_path_factory): ) assert result.exit_code == 1 assert "404" in result.output + + +def test_serve_get_actor(): + runner = CliRunner() + result = runner.invoke( + cli, + [ + "serve", + "--memory", + "--get", + "/-/actor.json", + "--actor", + '{"id": "root", "extra": "x"}', + ], + catch_exceptions=False, + ) + assert result.exit_code == 0 + assert json.loads(result.output) == { + "actor": { + "id": "root", + "extra": "x", + } + } From d28f12092dd795f35e9500154711d542f8931676 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 28 Aug 2023 17:38:32 -0700 Subject: [PATCH 487/952] Bump sphinx, furo, blacken-docs dependencies (#2160) * Bump the python-packages group with 3 updates Bumps the python-packages group with 3 updates: [sphinx](https://github.com/sphinx-doc/sphinx), [furo](https://github.com/pradyunsg/furo) and [blacken-docs](https://github.com/asottile/blacken-docs). Updates `sphinx` from 7.1.2 to 7.2.4 - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/master/CHANGES) - [Commits](https://github.com/sphinx-doc/sphinx/compare/v7.1.2...v7.2.4) Updates `furo` from 2023.7.26 to 2023.8.19 - [Release notes](https://github.com/pradyunsg/furo/releases) - [Changelog](https://github.com/pradyunsg/furo/blob/main/docs/changelog.md) - [Commits](https://github.com/pradyunsg/furo/compare/2023.07.26...2023.08.19) Updates `blacken-docs` from 1.15.0 to 1.16.0 - [Changelog](https://github.com/adamchainz/blacken-docs/blob/main/CHANGELOG.rst) - [Commits](https://github.com/asottile/blacken-docs/compare/1.15.0...1.16.0) --- updated-dependencies: - dependency-name: sphinx dependency-type: direct:development update-type: version-update:semver-minor dependency-group: python-packages - dependency-name: furo dependency-type: direct:development update-type: version-update:semver-minor dependency-group: python-packages - dependency-name: blacken-docs dependency-type: direct:development update-type: version-update:semver-minor dependency-group: python-packages ... Signed-off-by: dependabot[bot] --------- Signed-off-by: dependabot[bot] Co-authored-by: Simon Willison --- .github/workflows/deploy-latest.yml | 2 +- .github/workflows/spellcheck.yml | 2 +- .github/workflows/test.yml | 8 +++++++- setup.py | 6 +++--- 4 files changed, 12 insertions(+), 6 deletions(-) diff --git a/.github/workflows/deploy-latest.yml b/.github/workflows/deploy-latest.yml index 4746aa07..8cd9dcda 100644 --- a/.github/workflows/deploy-latest.yml +++ b/.github/workflows/deploy-latest.yml @@ -18,7 +18,7 @@ jobs: - name: Set up Python uses: actions/setup-python@v4 with: - python-version: "3.9" + python-version: "3.11" - uses: actions/cache@v3 name: Configure pip caching with: diff --git a/.github/workflows/spellcheck.yml b/.github/workflows/spellcheck.yml index 6bf72f9d..722e5c68 100644 --- a/.github/workflows/spellcheck.yml +++ b/.github/workflows/spellcheck.yml @@ -13,7 +13,7 @@ jobs: - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v2 with: - python-version: 3.9 + python-version: 3.11 - uses: actions/cache@v2 name: Configure pip caching with: diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 4eab1fdb..8cbbb572 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -29,7 +29,7 @@ jobs: (cd tests && gcc ext.c -fPIC -shared -o ext.so) - name: Install dependencies run: | - pip install -e '.[test,docs]' + pip install -e '.[test]' pip freeze - name: Run tests run: | @@ -37,10 +37,16 @@ jobs: pytest -m "serial" # And the test that exceeds a localhost HTTPS server tests/test_datasette_https_server.sh + - name: Install docs dependencies on Python 3.9+ + if: matrix.python-version != '3.8' + run: | + pip install -e '.[docs]' - name: Check if cog needs to be run + if: matrix.python-version != '3.8' run: | cog --check docs/*.rst - name: Check if blacken-docs needs to be run + if: matrix.python-version != '3.8' run: | # This fails on syntax errors, or a diff was applied blacken-docs -l 60 docs/*.rst diff --git a/setup.py b/setup.py index 3a105523..35c9b68b 100644 --- a/setup.py +++ b/setup.py @@ -69,8 +69,8 @@ setup( setup_requires=["pytest-runner"], extras_require={ "docs": [ - "Sphinx==7.1.2", - "furo==2023.7.26", + "Sphinx==7.2.4", + "furo==2023.8.19", "sphinx-autobuild", "codespell>=2.2.5", "blacken-docs", @@ -84,7 +84,7 @@ setup( "pytest-asyncio>=0.17", "beautifulsoup4>=4.8.1", "black==23.7.0", - "blacken-docs==1.15.0", + "blacken-docs==1.16.0", "pytest-timeout>=1.4.2", "trustme>=0.7", "cogapp>=3.3.0", From 92b8bf38c02465f624ce3f48dcabb0b100c4645d Mon Sep 17 00:00:00 2001 From: Alex Garcia Date: Mon, 28 Aug 2023 20:24:23 -0700 Subject: [PATCH 488/952] Add new `--internal internal.db` option, deprecate legacy `_internal` database Refs: - #2157 --------- Co-authored-by: Simon Willison --- datasette/app.py | 30 +++++++++-------- datasette/cli.py | 10 ++++-- datasette/database.py | 12 ++++++- datasette/default_permissions.py | 2 -- datasette/utils/internal_db.py | 30 ++++++++++------- datasette/views/database.py | 6 ++-- datasette/views/special.py | 2 +- docs/cli-reference.rst | 2 ++ docs/internals.rst | 27 ++++++++++----- tests/plugins/my_plugin_2.py | 5 +-- tests/test_cli.py | 12 +++++++ tests/test_internal_db.py | 58 ++++++++++---------------------- tests/test_plugins.py | 2 +- 13 files changed, 108 insertions(+), 90 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 1871aeb1..4deb8697 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -256,6 +256,7 @@ class Datasette: pdb=False, crossdb=False, nolock=False, + internal=None, ): self._startup_invoked = False assert config_dir is None or isinstance( @@ -304,17 +305,18 @@ class Datasette: self.add_database( Database(self, is_mutable=False, is_memory=True), name="_memory" ) - # memory_name is a random string so that each Datasette instance gets its own - # unique in-memory named database - otherwise unit tests can fail with weird - # errors when different instances accidentally share an in-memory database - self.add_database( - Database(self, memory_name=secrets.token_hex()), name="_internal" - ) - self.internal_db_created = False for file in self.files: self.add_database( Database(self, file, is_mutable=file not in self.immutables) ) + + self.internal_db_created = False + if internal is None: + self._internal_database = Database(self, memory_name=secrets.token_hex()) + else: + self._internal_database = Database(self, path=internal, mode="rwc") + self._internal_database.name = "__INTERNAL__" + self.cache_headers = cache_headers self.cors = cors config_files = [] @@ -436,15 +438,14 @@ class Datasette: await self._refresh_schemas() async def _refresh_schemas(self): - internal_db = self.databases["_internal"] + internal_db = self.get_internal_database() if not self.internal_db_created: await init_internal_db(internal_db) self.internal_db_created = True - current_schema_versions = { row["database_name"]: row["schema_version"] for row in await internal_db.execute( - "select database_name, schema_version from databases" + "select database_name, schema_version from core_databases" ) } for database_name, db in self.databases.items(): @@ -459,7 +460,7 @@ class Datasette: values = [database_name, db.is_memory, schema_version] await internal_db.execute_write( """ - INSERT OR REPLACE INTO databases (database_name, path, is_memory, schema_version) + INSERT OR REPLACE INTO core_databases (database_name, path, is_memory, schema_version) VALUES {} """.format( placeholders @@ -554,8 +555,7 @@ class Datasette: raise KeyError return matches[0] if name is None: - # Return first database that isn't "_internal" - name = [key for key in self.databases.keys() if key != "_internal"][0] + name = [key for key in self.databases.keys()][0] return self.databases[name] def add_database(self, db, name=None, route=None): @@ -655,6 +655,9 @@ class Datasette: def _metadata(self): return self.metadata() + def get_internal_database(self): + return self._internal_database + def plugin_config(self, plugin_name, database=None, table=None, fallback=True): """Return config for plugin, falling back from specified database/table""" plugins = self.metadata( @@ -978,7 +981,6 @@ class Datasette: "hash": d.hash, } for name, d in self.databases.items() - if name != "_internal" ] def _versions(self): diff --git a/datasette/cli.py b/datasette/cli.py index 6ebb1985..1a5a8af3 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -148,9 +148,6 @@ async def inspect_(files, sqlite_extensions): app = Datasette([], immutables=files, sqlite_extensions=sqlite_extensions) data = {} for name, database in app.databases.items(): - if name == "_internal": - # Don't include the in-memory _internal database - continue counts = await database.table_counts(limit=3600 * 1000) data[name] = { "hash": database.hash, @@ -476,6 +473,11 @@ def uninstall(packages, yes): "--ssl-certfile", help="SSL certificate file", ) +@click.option( + "--internal", + type=click.Path(), + help="Path to a persistent Datasette internal SQLite database", +) def serve( files, immutable, @@ -507,6 +509,7 @@ def serve( nolock, ssl_keyfile, ssl_certfile, + internal, return_instance=False, ): """Serve up specified SQLite database files with a web UI""" @@ -570,6 +573,7 @@ def serve( pdb=pdb, crossdb=crossdb, nolock=nolock, + internal=internal, ) # if files is a single directory, use that as config_dir= diff --git a/datasette/database.py b/datasette/database.py index af39ac9e..cb01301e 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -29,7 +29,13 @@ AttachedDatabase = namedtuple("AttachedDatabase", ("seq", "name", "file")) class Database: def __init__( - self, ds, path=None, is_mutable=True, is_memory=False, memory_name=None + self, + ds, + path=None, + is_mutable=True, + is_memory=False, + memory_name=None, + mode=None, ): self.name = None self.route = None @@ -50,6 +56,7 @@ class Database: self._write_connection = None # This is used to track all file connections so they can be closed self._all_file_connections = [] + self.mode = mode @property def cached_table_counts(self): @@ -90,6 +97,7 @@ class Database: return conn if self.is_memory: return sqlite3.connect(":memory:", uri=True) + # mode=ro or immutable=1? if self.is_mutable: qs = "?mode=ro" @@ -100,6 +108,8 @@ class Database: assert not (write and not self.is_mutable) if write: qs = "" + if self.mode is not None: + qs = f"?mode={self.mode}" conn = sqlite3.connect( f"file:{self.path}{qs}", uri=True, check_same_thread=False ) diff --git a/datasette/default_permissions.py b/datasette/default_permissions.py index 63a66c3c..f0b086e9 100644 --- a/datasette/default_permissions.py +++ b/datasette/default_permissions.py @@ -146,8 +146,6 @@ async def _resolve_metadata_view_permissions(datasette, actor, action, resource) if allow is not None: return actor_matches_allow(actor, allow) elif action == "view-database": - if resource == "_internal" and (actor is None or actor.get("id") != "root"): - return False database_allow = datasette.metadata("allow", database=resource) if database_allow is None: return None diff --git a/datasette/utils/internal_db.py b/datasette/utils/internal_db.py index e4b49e80..215695ca 100644 --- a/datasette/utils/internal_db.py +++ b/datasette/utils/internal_db.py @@ -5,13 +5,13 @@ from datasette.utils import table_column_details async def init_internal_db(db): create_tables_sql = textwrap.dedent( """ - CREATE TABLE IF NOT EXISTS databases ( + CREATE TABLE IF NOT EXISTS core_databases ( database_name TEXT PRIMARY KEY, path TEXT, is_memory INTEGER, schema_version INTEGER ); - CREATE TABLE IF NOT EXISTS tables ( + CREATE TABLE IF NOT EXISTS core_tables ( database_name TEXT, table_name TEXT, rootpage INTEGER, @@ -19,7 +19,7 @@ async def init_internal_db(db): PRIMARY KEY (database_name, table_name), FOREIGN KEY (database_name) REFERENCES databases(database_name) ); - CREATE TABLE IF NOT EXISTS columns ( + CREATE TABLE IF NOT EXISTS core_columns ( database_name TEXT, table_name TEXT, cid INTEGER, @@ -33,7 +33,7 @@ async def init_internal_db(db): FOREIGN KEY (database_name) REFERENCES databases(database_name), FOREIGN KEY (database_name, table_name) REFERENCES tables(database_name, table_name) ); - CREATE TABLE IF NOT EXISTS indexes ( + CREATE TABLE IF NOT EXISTS core_indexes ( database_name TEXT, table_name TEXT, seq INTEGER, @@ -45,7 +45,7 @@ async def init_internal_db(db): FOREIGN KEY (database_name) REFERENCES databases(database_name), FOREIGN KEY (database_name, table_name) REFERENCES tables(database_name, table_name) ); - CREATE TABLE IF NOT EXISTS foreign_keys ( + CREATE TABLE IF NOT EXISTS core_foreign_keys ( database_name TEXT, table_name TEXT, id INTEGER, @@ -69,12 +69,16 @@ async def populate_schema_tables(internal_db, db): database_name = db.name def delete_everything(conn): - conn.execute("DELETE FROM tables WHERE database_name = ?", [database_name]) - conn.execute("DELETE FROM columns WHERE database_name = ?", [database_name]) + conn.execute("DELETE FROM core_tables WHERE database_name = ?", [database_name]) conn.execute( - "DELETE FROM foreign_keys WHERE database_name = ?", [database_name] + "DELETE FROM core_columns WHERE database_name = ?", [database_name] + ) + conn.execute( + "DELETE FROM core_foreign_keys WHERE database_name = ?", [database_name] + ) + conn.execute( + "DELETE FROM core_indexes WHERE database_name = ?", [database_name] ) - conn.execute("DELETE FROM indexes WHERE database_name = ?", [database_name]) await internal_db.execute_write_fn(delete_everything) @@ -133,14 +137,14 @@ async def populate_schema_tables(internal_db, db): await internal_db.execute_write_many( """ - INSERT INTO tables (database_name, table_name, rootpage, sql) + INSERT INTO core_tables (database_name, table_name, rootpage, sql) values (?, ?, ?, ?) """, tables_to_insert, ) await internal_db.execute_write_many( """ - INSERT INTO columns ( + INSERT INTO core_columns ( database_name, table_name, cid, name, type, "notnull", default_value, is_pk, hidden ) VALUES ( :database_name, :table_name, :cid, :name, :type, :notnull, :default_value, :is_pk, :hidden @@ -150,7 +154,7 @@ async def populate_schema_tables(internal_db, db): ) await internal_db.execute_write_many( """ - INSERT INTO foreign_keys ( + INSERT INTO core_foreign_keys ( database_name, table_name, "id", seq, "table", "from", "to", on_update, on_delete, match ) VALUES ( :database_name, :table_name, :id, :seq, :table, :from, :to, :on_update, :on_delete, :match @@ -160,7 +164,7 @@ async def populate_schema_tables(internal_db, db): ) await internal_db.execute_write_many( """ - INSERT INTO indexes ( + INSERT INTO core_indexes ( database_name, table_name, seq, name, "unique", origin, partial ) VALUES ( :database_name, :table_name, :seq, :name, :unique, :origin, :partial diff --git a/datasette/views/database.py b/datasette/views/database.py index d9abc38a..4647bedc 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -950,9 +950,9 @@ class TableCreateView(BaseView): async def _table_columns(datasette, database_name): - internal = datasette.get_database("_internal") - result = await internal.execute( - "select table_name, name from columns where database_name = ?", + internal_db = datasette.get_internal_database() + result = await internal_db.execute( + "select table_name, name from core_columns where database_name = ?", [database_name], ) table_columns = {} diff --git a/datasette/views/special.py b/datasette/views/special.py index c45a3eca..c1b84f8f 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -238,7 +238,7 @@ class CreateTokenView(BaseView): # Build list of databases and tables the user has permission to view database_with_tables = [] for database in self.ds.databases.values(): - if database.name in ("_internal", "_memory"): + if database.name == "_memory": continue if not await self.ds.permission_allowed( request.actor, "view-database", database.name diff --git a/docs/cli-reference.rst b/docs/cli-reference.rst index 5657f480..8e333447 100644 --- a/docs/cli-reference.rst +++ b/docs/cli-reference.rst @@ -134,6 +134,8 @@ Once started you can access it at ``http://localhost:8001`` mode --ssl-keyfile TEXT SSL key file --ssl-certfile TEXT SSL certificate file + --internal PATH Path to a persistent Datasette internal SQLite + database --help Show this message and exit. diff --git a/docs/internals.rst b/docs/internals.rst index 4b82e11c..fe9a2fa7 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -271,7 +271,7 @@ Property exposing a ``collections.OrderedDict`` of databases currently connected The dictionary keys are the name of the database that is used in the URL - e.g. ``/fixtures`` would have a key of ``"fixtures"``. The values are :ref:`internals_database` instances. -All databases are listed, irrespective of user permissions. This means that the ``_internal`` database will always be listed here. +All databases are listed, irrespective of user permissions. .. _datasette_permissions: @@ -479,6 +479,13 @@ The following example creates a token that can access ``view-instance`` and ``vi Returns the specified database object. Raises a ``KeyError`` if the database does not exist. Call this method without an argument to return the first connected database. +.. _get_internal_database: + +.get_internal_database() +------------------------ + +Returns a database object for reading and writing to the private :ref:`internal database `. + .. _datasette_add_database: .add_database(db, name=None, route=None) @@ -1127,19 +1134,21 @@ You can selectively disable CSRF protection using the :ref:`plugin_hook_skip_csr .. _internals_internal: -The _internal database -====================== +Datasette's internal database +============================= -.. warning:: - This API should be considered unstable - the structure of these tables may change prior to the release of Datasette 1.0. +Datasette maintains an "internal" SQLite database used for configuration, caching, and storage. Plugins can store configuration, settings, and other data inside this database. By default, Datasette will use a temporary in-memory SQLite database as the internal database, which is created at startup and destroyed at shutdown. Users of Datasette can optionally pass in a `--internal` flag to specify the path to a SQLite database to use as the internal database, which will persist internal data across Datasette instances. -Datasette maintains an in-memory SQLite database with details of the the databases, tables and columns for all of the attached databases. +The internal database is not exposed in the Datasette application by default, which means private data can safely be stored without worry of accidentally leaking information through the default Datasette interface and API. However, other plugins do have full read and write access to the internal database. -By default all actors are denied access to the ``view-database`` permission for the ``_internal`` database, so the database is not visible to anyone unless they :ref:`sign in as root `. +Plugins can access this database by calling ``internal_db = datasette.get_internal_database()`` and then executing queries using the :ref:`Database API `. -Plugins can access this database by calling ``db = datasette.get_database("_internal")`` and then executing queries using the :ref:`Database API `. +Plugin authors are asked to practice good etiquette when using the internal database, as all plugins use the same database to store data. For example: -You can explore an example of this database by `signing in as root `__ to the ``latest.datasette.io`` demo instance and then navigating to `latest.datasette.io/_internal `__. +1. Use a unique prefix when creating tables, indices, and triggera in the internal database. If your plugin is called `datasette-xyz`, then prefix names with `datasette_xyz_*`. +2. Avoid long-running write statements that may stall or block other plugins that are trying to write at the same time. +3. Use temporary tables or shared in-memory attached databases when possible. +4. Avoid implementing features that could expose private data stored in the internal database by other plugins. .. _internals_utils: diff --git a/tests/plugins/my_plugin_2.py b/tests/plugins/my_plugin_2.py index d588342c..bb82b8c1 100644 --- a/tests/plugins/my_plugin_2.py +++ b/tests/plugins/my_plugin_2.py @@ -120,7 +120,7 @@ def permission_allowed(datasette, actor, action): assert ( 2 == ( - await datasette.get_database("_internal").execute("select 1 + 1") + await datasette.get_internal_database().execute("select 1 + 1") ).first()[0] ) if action == "this_is_allowed_async": @@ -142,7 +142,8 @@ def startup(datasette): async def inner(): # Run against _internal so tests that use the ds_client fixture # (which has no databases yet on startup) do not fail: - result = await datasette.get_database("_internal").execute("select 1 + 1") + internal_db = datasette.get_internal_database() + result = await internal_db.execute("select 1 + 1") datasette._startup_hook_calculation = result.first()[0] return inner diff --git a/tests/test_cli.py b/tests/test_cli.py index d9a10f22..e85bcef1 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -154,6 +154,7 @@ def test_metadata_yaml(): ssl_keyfile=None, ssl_certfile=None, return_instance=True, + internal=None, ) client = _TestClient(ds) response = client.get("/-/metadata.json") @@ -368,3 +369,14 @@ def test_help_settings(): result = runner.invoke(cli, ["--help-settings"]) for setting in SETTINGS: assert setting.name in result.output + + +def test_internal_db(tmpdir): + runner = CliRunner() + internal_path = tmpdir / "internal.db" + assert not internal_path.exists() + result = runner.invoke( + cli, ["--memory", "--internal", str(internal_path), "--get", "/"] + ) + assert result.exit_code == 0 + assert internal_path.exists() diff --git a/tests/test_internal_db.py b/tests/test_internal_db.py index a666dd72..5276dc99 100644 --- a/tests/test_internal_db.py +++ b/tests/test_internal_db.py @@ -1,55 +1,35 @@ import pytest -@pytest.mark.asyncio -async def test_internal_only_available_to_root(ds_client): - cookie = ds_client.actor_cookie({"id": "root"}) - assert (await ds_client.get("/_internal")).status_code == 403 - assert ( - await ds_client.get("/_internal", cookies={"ds_actor": cookie}) - ).status_code == 200 +# ensure refresh_schemas() gets called before interacting with internal_db +async def ensure_internal(ds_client): + await ds_client.get("/fixtures.json?sql=select+1") + return ds_client.ds.get_internal_database() @pytest.mark.asyncio async def test_internal_databases(ds_client): - cookie = ds_client.actor_cookie({"id": "root"}) - databases = ( - await ds_client.get( - "/_internal/databases.json?_shape=array", cookies={"ds_actor": cookie} - ) - ).json() - assert len(databases) == 2 - internal, fixtures = databases - assert internal["database_name"] == "_internal" - assert internal["is_memory"] == 1 - assert internal["path"] is None - assert isinstance(internal["schema_version"], int) - assert fixtures["database_name"] == "fixtures" + internal_db = await ensure_internal(ds_client) + databases = await internal_db.execute("select * from core_databases") + assert len(databases) == 1 + assert databases.rows[0]["database_name"] == "fixtures" @pytest.mark.asyncio async def test_internal_tables(ds_client): - cookie = ds_client.actor_cookie({"id": "root"}) - tables = ( - await ds_client.get( - "/_internal/tables.json?_shape=array", cookies={"ds_actor": cookie} - ) - ).json() + internal_db = await ensure_internal(ds_client) + tables = await internal_db.execute("select * from core_tables") assert len(tables) > 5 - table = tables[0] + table = tables.rows[0] assert set(table.keys()) == {"rootpage", "table_name", "database_name", "sql"} @pytest.mark.asyncio async def test_internal_indexes(ds_client): - cookie = ds_client.actor_cookie({"id": "root"}) - indexes = ( - await ds_client.get( - "/_internal/indexes.json?_shape=array", cookies={"ds_actor": cookie} - ) - ).json() + internal_db = await ensure_internal(ds_client) + indexes = await internal_db.execute("select * from core_indexes") assert len(indexes) > 5 - index = indexes[0] + index = indexes.rows[0] assert set(index.keys()) == { "partial", "name", @@ -63,14 +43,10 @@ async def test_internal_indexes(ds_client): @pytest.mark.asyncio async def test_internal_foreign_keys(ds_client): - cookie = ds_client.actor_cookie({"id": "root"}) - foreign_keys = ( - await ds_client.get( - "/_internal/foreign_keys.json?_shape=array", cookies={"ds_actor": cookie} - ) - ).json() + internal_db = await ensure_internal(ds_client) + foreign_keys = await internal_db.execute("select * from core_foreign_keys") assert len(foreign_keys) > 5 - foreign_key = foreign_keys[0] + foreign_key = foreign_keys.rows[0] assert set(foreign_key.keys()) == { "table", "seq", diff --git a/tests/test_plugins.py b/tests/test_plugins.py index 28fe720f..9761fa53 100644 --- a/tests/test_plugins.py +++ b/tests/test_plugins.py @@ -329,7 +329,7 @@ def test_hook_extra_body_script(app_client, path, expected_extra_body_script): @pytest.mark.asyncio async def test_hook_asgi_wrapper(ds_client): response = await ds_client.get("/fixtures") - assert "_internal, fixtures" == response.headers["x-databases"] + assert "fixtures" == response.headers["x-databases"] def test_hook_extra_template_vars(restore_working_directory): From a1f3d75a527b222cf1df51c41e1c424b38428a99 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 28 Aug 2023 20:46:12 -0700 Subject: [PATCH 489/952] Need to stick to Python 3.9 for gcloud --- .github/workflows/deploy-latest.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/workflows/deploy-latest.yml b/.github/workflows/deploy-latest.yml index 8cd9dcda..0dfa5a60 100644 --- a/.github/workflows/deploy-latest.yml +++ b/.github/workflows/deploy-latest.yml @@ -17,8 +17,9 @@ jobs: uses: actions/checkout@v3 - name: Set up Python uses: actions/setup-python@v4 + # gcloud commmand breaks on higher Python versions, so stick with 3.9: with: - python-version: "3.11" + python-version: "3.9" - uses: actions/cache@v3 name: Configure pip caching with: From 50da908213a0fc405ecd7a40090dfea7a2e7395c Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 29 Aug 2023 09:32:34 -0700 Subject: [PATCH 490/952] Cascade for restricted token view-table/view-database/view-instance operations (#2154) Closes #2102 * Permission is now a dataclass, not a namedtuple - refs https://github.com/simonw/datasette/pull/2154/#discussion_r1308087800 * datasette.get_permission() method --- datasette/app.py | 14 ++ datasette/default_permissions.py | 241 +++++++++++++++++++++++------- datasette/permissions.py | 20 ++- datasette/views/special.py | 12 +- docs/internals.rst | 12 +- docs/plugin_hooks.rst | 14 +- tests/test_internals_datasette.py | 14 ++ tests/test_permissions.py | 194 +++++++++++++++++++++++- 8 files changed, 449 insertions(+), 72 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 4deb8697..d95ec2bf 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -431,6 +431,20 @@ class Datasette: self._root_token = secrets.token_hex(32) self.client = DatasetteClient(self) + def get_permission(self, name_or_abbr: str) -> "Permission": + """ + Returns a Permission object for the given name or abbreviation. Raises KeyError if not found. + """ + if name_or_abbr in self.permissions: + return self.permissions[name_or_abbr] + # Try abbreviation + for permission in self.permissions.values(): + if permission.abbr == name_or_abbr: + return permission + raise KeyError( + "No permission found with name or abbreviation {}".format(name_or_abbr) + ) + async def refresh_schemas(self): if self._refresh_schemas_lock.locked(): return diff --git a/datasette/default_permissions.py b/datasette/default_permissions.py index f0b086e9..960429fc 100644 --- a/datasette/default_permissions.py +++ b/datasette/default_permissions.py @@ -2,6 +2,7 @@ from datasette import hookimpl, Permission from datasette.utils import actor_matches_allow import itsdangerous import time +from typing import Union, Tuple @hookimpl @@ -9,32 +10,112 @@ def register_permissions(): return ( # name, abbr, description, takes_database, takes_resource, default Permission( - "view-instance", "vi", "View Datasette instance", False, False, True - ), - Permission("view-database", "vd", "View database", True, False, True), - Permission( - "view-database-download", "vdd", "Download database file", True, False, True - ), - Permission("view-table", "vt", "View table", True, True, True), - Permission("view-query", "vq", "View named query results", True, True, True), - Permission( - "execute-sql", "es", "Execute read-only SQL queries", True, False, True + name="view-instance", + abbr="vi", + description="View Datasette instance", + takes_database=False, + takes_resource=False, + default=True, ), Permission( - "permissions-debug", - "pd", - "Access permission debug tool", - False, - False, - False, + name="view-database", + abbr="vd", + description="View database", + takes_database=True, + takes_resource=False, + default=True, + implies_can_view=True, + ), + Permission( + name="view-database-download", + abbr="vdd", + description="Download database file", + takes_database=True, + takes_resource=False, + default=True, + ), + Permission( + name="view-table", + abbr="vt", + description="View table", + takes_database=True, + takes_resource=True, + default=True, + implies_can_view=True, + ), + Permission( + name="view-query", + abbr="vq", + description="View named query results", + takes_database=True, + takes_resource=True, + default=True, + implies_can_view=True, + ), + Permission( + name="execute-sql", + abbr="es", + description="Execute read-only SQL queries", + takes_database=True, + takes_resource=False, + default=True, + ), + Permission( + name="permissions-debug", + abbr="pd", + description="Access permission debug tool", + takes_database=False, + takes_resource=False, + default=False, + ), + Permission( + name="debug-menu", + abbr="dm", + description="View debug menu items", + takes_database=False, + takes_resource=False, + default=False, + ), + Permission( + name="insert-row", + abbr="ir", + description="Insert rows", + takes_database=True, + takes_resource=True, + default=False, + ), + Permission( + name="delete-row", + abbr="dr", + description="Delete rows", + takes_database=True, + takes_resource=True, + default=False, + ), + Permission( + name="update-row", + abbr="ur", + description="Update rows", + takes_database=True, + takes_resource=True, + default=False, + ), + Permission( + name="create-table", + abbr="ct", + description="Create tables", + takes_database=True, + takes_resource=False, + default=False, + ), + Permission( + name="drop-table", + abbr="dt", + description="Drop tables", + takes_database=True, + takes_resource=True, + default=False, ), - Permission("debug-menu", "dm", "View debug menu items", False, False, False), - # Write API permissions - Permission("insert-row", "ir", "Insert rows", True, True, False), - Permission("delete-row", "dr", "Delete rows", True, True, False), - Permission("update-row", "ur", "Update rows", True, True, False), - Permission("create-table", "ct", "Create tables", True, False, False), - Permission("drop-table", "dt", "Drop tables", True, True, False), ) @@ -176,6 +257,80 @@ async def _resolve_metadata_view_permissions(datasette, actor, action, resource) return actor_matches_allow(actor, database_allow_sql) +def restrictions_allow_action( + datasette: "Datasette", + restrictions: dict, + action: str, + resource: Union[str, Tuple[str, str]], +): + "Do these restrictions allow the requested action against the requested resource?" + if action == "view-instance": + # Special case for view-instance: it's allowed if the restrictions include any + # permissions that have the implies_can_view=True flag set + all_rules = restrictions.get("a") or [] + for database_rules in (restrictions.get("d") or {}).values(): + all_rules += database_rules + for database_resource_rules in (restrictions.get("r") or {}).values(): + for resource_rules in database_resource_rules.values(): + all_rules += resource_rules + permissions = [datasette.get_permission(action) for action in all_rules] + if any(p for p in permissions if p.implies_can_view): + return True + + if action == "view-database": + # Special case for view-database: it's allowed if the restrictions include any + # permissions that have the implies_can_view=True flag set AND takes_database + all_rules = restrictions.get("a") or [] + database_rules = list((restrictions.get("d") or {}).get(resource) or []) + all_rules += database_rules + resource_rules = ((restrictions.get("r") or {}).get(resource) or {}).values() + for resource_rules in (restrictions.get("r") or {}).values(): + for table_rules in resource_rules.values(): + all_rules += table_rules + permissions = [datasette.get_permission(action) for action in all_rules] + if any(p for p in permissions if p.implies_can_view and p.takes_database): + return True + + # Does this action have an abbreviation? + to_check = {action} + permission = datasette.permissions.get(action) + if permission and permission.abbr: + to_check.add(permission.abbr) + + # If restrictions is defined then we use those to further restrict the actor + # Crucially, we only use this to say NO (return False) - we never + # use it to return YES (True) because that might over-ride other + # restrictions placed on this actor + all_allowed = restrictions.get("a") + if all_allowed is not None: + assert isinstance(all_allowed, list) + if to_check.intersection(all_allowed): + return True + # How about for the current database? + if resource: + if isinstance(resource, str): + database_name = resource + else: + database_name = resource[0] + database_allowed = restrictions.get("d", {}).get(database_name) + if database_allowed is not None: + assert isinstance(database_allowed, list) + if to_check.intersection(database_allowed): + return True + # Or the current table? That's any time the resource is (database, table) + if resource is not None and not isinstance(resource, str) and len(resource) == 2: + database, table = resource + table_allowed = restrictions.get("r", {}).get(database, {}).get(table) + # TODO: What should this do for canned queries? + if table_allowed is not None: + assert isinstance(table_allowed, list) + if to_check.intersection(table_allowed): + return True + + # This action is not specifically allowed, so reject it + return False + + @hookimpl(specname="permission_allowed") def permission_allowed_actor_restrictions(datasette, actor, action, resource): if actor is None: @@ -184,40 +339,12 @@ def permission_allowed_actor_restrictions(datasette, actor, action, resource): # No restrictions, so we have no opinion return None _r = actor.get("_r") - - # Does this action have an abbreviation? - to_check = {action} - permission = datasette.permissions.get(action) - if permission and permission.abbr: - to_check.add(permission.abbr) - - # If _r is defined then we use those to further restrict the actor - # Crucially, we only use this to say NO (return False) - we never - # use it to return YES (True) because that might over-ride other - # restrictions placed on this actor - all_allowed = _r.get("a") - if all_allowed is not None: - assert isinstance(all_allowed, list) - if to_check.intersection(all_allowed): - return None - # How about for the current database? - if isinstance(resource, str): - database_allowed = _r.get("d", {}).get(resource) - if database_allowed is not None: - assert isinstance(database_allowed, list) - if to_check.intersection(database_allowed): - return None - # Or the current table? That's any time the resource is (database, table) - if resource is not None and not isinstance(resource, str) and len(resource) == 2: - database, table = resource - table_allowed = _r.get("r", {}).get(database, {}).get(table) - # TODO: What should this do for canned queries? - if table_allowed is not None: - assert isinstance(table_allowed, list) - if to_check.intersection(table_allowed): - return None - # This action is not specifically allowed, so reject it - return False + if restrictions_allow_action(datasette, _r, action, resource): + # Return None because we do not have an opinion here + return None + else: + # Block this permission check + return False @hookimpl diff --git a/datasette/permissions.py b/datasette/permissions.py index 1cd3474d..152f1721 100644 --- a/datasette/permissions.py +++ b/datasette/permissions.py @@ -1,6 +1,16 @@ -import collections +from dataclasses import dataclass, fields +from typing import Optional -Permission = collections.namedtuple( - "Permission", - ("name", "abbr", "description", "takes_database", "takes_resource", "default"), -) + +@dataclass +class Permission: + name: str + abbr: Optional[str] + description: Optional[str] + takes_database: bool + takes_resource: bool + default: bool + # This is deliberately undocumented: it's considered an internal + # implementation detail for view-table/view-database and should + # not be used by plugins as it may change in the future. + implies_can_view: bool = False diff --git a/datasette/views/special.py b/datasette/views/special.py index c1b84f8f..849750bf 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -122,7 +122,17 @@ class PermissionsDebugView(BaseView): # list() avoids error if check is performed during template render: { "permission_checks": list(reversed(self.ds._permission_checks)), - "permissions": list(self.ds.permissions.values()), + "permissions": [ + ( + p.name, + p.abbr, + p.description, + p.takes_database, + p.takes_resource, + p.default, + ) + for p in self.ds.permissions.values() + ], }, ) diff --git a/docs/internals.rst b/docs/internals.rst index fe9a2fa7..474e3328 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -280,7 +280,7 @@ All databases are listed, irrespective of user permissions. Property exposing a dictionary of permissions that have been registered using the :ref:`plugin_register_permissions` plugin hook. -The dictionary keys are the permission names - e.g. ``view-instance`` - and the values are ``Permission()`` named tuples describing the permission. Here is a :ref:`description of that tuple `. +The dictionary keys are the permission names - e.g. ``view-instance`` - and the values are ``Permission()`` objects describing the permission. Here is a :ref:`description of that object `. .. _datasette_plugin_config: @@ -469,6 +469,16 @@ The following example creates a token that can access ``view-instance`` and ``vi }, ) +.. _datasette_get_permission: + +.get_permission(name_or_abbr) +----------------------------- + +``name_or_abbr`` - string + The name or abbreviation of the permission to look up, e.g. ``view-table`` or ``vt``. + +Returns a :ref:`Permission object ` representing the permission, or raises a ``KeyError`` if one is not found. + .. _datasette_get_database: .get_database(name) diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index 497508ae..f8bb203d 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -794,24 +794,24 @@ If your plugin needs to register additional permissions unique to that plugin - ) ] -The fields of the ``Permission`` named tuple are as follows: +The fields of the ``Permission`` class are as follows: -``name`` +``name`` - string The name of the permission, e.g. ``upload-csvs``. This should be unique across all plugins that the user might have installed, so choose carefully. -``abbr`` +``abbr`` - string or None An abbreviation of the permission, e.g. ``uc``. This is optional - you can set it to ``None`` if you do not want to pick an abbreviation. Since this needs to be unique across all installed plugins it's best not to specify an abbreviation at all. If an abbreviation is provided it will be used when creating restricted signed API tokens. -``description`` +``description`` - string or None A human-readable description of what the permission lets you do. Should make sense as the second part of a sentence that starts "A user with this permission can ...". -``takes_database`` +``takes_database`` - boolean ``True`` if this permission can be granted on a per-database basis, ``False`` if it is only valid at the overall Datasette instance level. -``takes_resource`` +``takes_resource`` - boolean ``True`` if this permission can be granted on a per-resource basis. A resource is a database table, SQL view or :ref:`canned query `. -``default`` +``default`` - boolean The default value for this permission if it is not explicitly granted to a user. ``True`` means the permission is granted by default, ``False`` means it is not. This should only be ``True`` if you want anonymous users to be able to take this action. diff --git a/tests/test_internals_datasette.py b/tests/test_internals_datasette.py index d59ff729..c11e840c 100644 --- a/tests/test_internals_datasette.py +++ b/tests/test_internals_datasette.py @@ -159,3 +159,17 @@ def test_datasette_error_if_string_not_list(tmpdir): db_path = str(tmpdir / "data.db") with pytest.raises(ValueError): ds = Datasette(db_path) + + +@pytest.mark.asyncio +async def test_get_permission(ds_client): + ds = ds_client.ds + for name_or_abbr in ("vi", "view-instance", "vt", "view-table"): + permission = ds.get_permission(name_or_abbr) + if "-" in name_or_abbr: + assert permission.name == name_or_abbr + else: + assert permission.abbr == name_or_abbr + # And test KeyError + with pytest.raises(KeyError): + ds.get_permission("missing-permission") diff --git a/tests/test_permissions.py b/tests/test_permissions.py index f940d486..cad0525f 100644 --- a/tests/test_permissions.py +++ b/tests/test_permissions.py @@ -1,6 +1,7 @@ import collections from datasette.app import Datasette from datasette.cli import cli +from datasette.default_permissions import restrictions_allow_action from .fixtures import app_client, assert_permissions_checked, make_app_client from click.testing import CliRunner from bs4 import BeautifulSoup as Soup @@ -35,6 +36,8 @@ async def perms_ds(): one = ds.add_memory_database("perms_ds_one") two = ds.add_memory_database("perms_ds_two") await one.execute_write("create table if not exists t1 (id integer primary key)") + await one.execute_write("insert or ignore into t1 (id) values (1)") + await one.execute_write("create view if not exists v1 as select * from t1") await one.execute_write("create table if not exists t2 (id integer primary key)") await two.execute_write("create table if not exists t1 (id integer primary key)") return ds @@ -585,7 +588,6 @@ DEF = "USE_DEFAULT" ({"id": "t", "_r": {"a": ["vd"]}}, "view-database", "one", None, DEF), ({"id": "t", "_r": {"a": ["vt"]}}, "view-table", "one", "t1", DEF), # But not if it's the wrong permission - ({"id": "t", "_r": {"a": ["vd"]}}, "view-instance", None, None, False), ({"id": "t", "_r": {"a": ["vi"]}}, "view-database", "one", None, False), ({"id": "t", "_r": {"a": ["vd"]}}, "view-table", "one", "t1", False), # Works at the "d" for database level: @@ -629,11 +631,14 @@ DEF = "USE_DEFAULT" "t1", DEF, ), + # view-instance is granted if you have view-database + ({"id": "t", "_r": {"a": ["vd"]}}, "view-instance", None, None, DEF), ), ) async def test_actor_restricted_permissions( perms_ds, actor, permission, resource_1, resource_2, expected_result ): + perms_ds.pdb = True cookies = {"ds_actor": perms_ds.sign({"a": {"id": "root"}}, "actor")} csrftoken = (await perms_ds.client.get("/-/permissions", cookies=cookies)).cookies[ "ds_csrftoken" @@ -1018,3 +1023,190 @@ async def test_api_explorer_visibility( assert response.status_code == 403 finally: perms_ds._metadata_local = prev_metadata + + +@pytest.mark.asyncio +async def test_view_table_token_can_access_table(perms_ds): + actor = { + "id": "restricted-token", + "token": "dstok", + # Restricted to just view-table on perms_ds_two/t1 + "_r": {"r": {"perms_ds_two": {"t1": ["vt"]}}}, + } + cookies = {"ds_actor": perms_ds.client.actor_cookie(actor)} + response = await perms_ds.client.get("/perms_ds_two/t1.json", cookies=cookies) + assert response.status_code == 200 + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "restrictions,verb,path,body,expected_status", + ( + # No restrictions + (None, "get", "/.json", None, 200), + (None, "get", "/perms_ds_one.json", None, 200), + (None, "get", "/perms_ds_one/t1.json", None, 200), + (None, "get", "/perms_ds_one/t1/1.json", None, 200), + (None, "get", "/perms_ds_one/v1.json", None, 200), + # Restricted to just view-instance + ({"a": ["vi"]}, "get", "/.json", None, 200), + ({"a": ["vi"]}, "get", "/perms_ds_one.json", None, 403), + ({"a": ["vi"]}, "get", "/perms_ds_one/t1.json", None, 403), + ({"a": ["vi"]}, "get", "/perms_ds_one/t1/1.json", None, 403), + ({"a": ["vi"]}, "get", "/perms_ds_one/v1.json", None, 403), + # Restricted to just view-database + ({"a": ["vd"]}, "get", "/.json", None, 200), # Can see instance too + ({"a": ["vd"]}, "get", "/perms_ds_one.json", None, 200), + ({"a": ["vd"]}, "get", "/perms_ds_one/t1.json", None, 403), + ({"a": ["vd"]}, "get", "/perms_ds_one/t1/1.json", None, 403), + ({"a": ["vd"]}, "get", "/perms_ds_one/v1.json", None, 403), + # Restricted to just view-table for specific database + ( + {"d": {"perms_ds_one": ["vt"]}}, + "get", + "/.json", + None, + 200, + ), # Can see instance + ( + {"d": {"perms_ds_one": ["vt"]}}, + "get", + "/perms_ds_one.json", + None, + 200, + ), # and this database + ( + {"d": {"perms_ds_one": ["vt"]}}, + "get", + "/perms_ds_two.json", + None, + 403, + ), # But not this one + ( + # Can see the table + {"d": {"perms_ds_one": ["vt"]}}, + "get", + "/perms_ds_one/t1.json", + None, + 200, + ), + ( + # And the view + {"d": {"perms_ds_one": ["vt"]}}, + "get", + "/perms_ds_one/v1.json", + None, + 200, + ), + # view-table access to a specific table + ( + {"r": {"perms_ds_one": {"t1": ["vt"]}}}, + "get", + "/.json", + None, + 200, + ), + ( + {"r": {"perms_ds_one": {"t1": ["vt"]}}}, + "get", + "/perms_ds_one.json", + None, + 200, + ), + ( + {"r": {"perms_ds_one": {"t1": ["vt"]}}}, + "get", + "/perms_ds_one/t1.json", + None, + 200, + ), + # But cannot see the other table + ( + {"r": {"perms_ds_one": {"t1": ["vt"]}}}, + "get", + "/perms_ds_one/t2.json", + None, + 403, + ), + # Or the view + ( + {"r": {"perms_ds_one": {"t1": ["vt"]}}}, + "get", + "/perms_ds_one/v1.json", + None, + 403, + ), + ), +) +async def test_actor_restrictions( + perms_ds, restrictions, verb, path, body, expected_status +): + actor = {"id": "user"} + if restrictions: + actor["_r"] = restrictions + method = getattr(perms_ds.client, verb) + kwargs = {"cookies": {"ds_actor": perms_ds.client.actor_cookie(actor)}} + if body: + kwargs["json"] = body + perms_ds._permission_checks.clear() + response = await method(path, **kwargs) + assert response.status_code == expected_status, json.dumps( + { + "verb": verb, + "path": path, + "body": body, + "restrictions": restrictions, + "expected_status": expected_status, + "response_status": response.status_code, + "checks": [ + { + "action": check["action"], + "resource": check["resource"], + "result": check["result"], + } + for check in perms_ds._permission_checks + ], + }, + indent=2, + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "restrictions,action,resource,expected", + ( + ({"a": ["view-instance"]}, "view-instance", None, True), + # view-table and view-database implies view-instance + ({"a": ["view-table"]}, "view-instance", None, True), + ({"a": ["view-database"]}, "view-instance", None, True), + # update-row does not imply view-instance + ({"a": ["update-row"]}, "view-instance", None, False), + # view-table on a resource implies view-instance + ({"r": {"db1": {"t1": ["view-table"]}}}, "view-instance", None, True), + # update-row on a resource does not imply view-instance + ({"r": {"db1": {"t1": ["update-row"]}}}, "view-instance", None, False), + # view-database on a resource implies view-instance + ({"d": {"db1": ["view-database"]}}, "view-instance", None, True), + # Having view-table on "a" allows access to any specific table + ({"a": ["view-table"]}, "view-table", ("dbname", "tablename"), True), + # Ditto for on the database + ( + {"d": {"dbname": ["view-table"]}}, + "view-table", + ("dbname", "tablename"), + True, + ), + # But not if it's allowed on a different database + ( + {"d": {"dbname": ["view-table"]}}, + "view-table", + ("dbname2", "tablename"), + False, + ), + ), +) +async def test_restrictions_allow_action(restrictions, action, resource, expected): + ds = Datasette() + await ds.invoke_startup() + actual = restrictions_allow_action(ds, restrictions, action, resource) + assert actual == expected From bb12229794655abaa21a9aa691d1f85d34b6c45a Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 29 Aug 2023 10:01:28 -0700 Subject: [PATCH 491/952] Rename core_ to catalog_, closes #2163 --- datasette/app.py | 4 ++-- datasette/utils/internal_db.py | 28 +++++++++++++++------------- datasette/views/database.py | 2 +- docs/internals.rst | 2 ++ tests/test_internal_db.py | 8 ++++---- 5 files changed, 24 insertions(+), 20 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index d95ec2bf..0227f627 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -459,7 +459,7 @@ class Datasette: current_schema_versions = { row["database_name"]: row["schema_version"] for row in await internal_db.execute( - "select database_name, schema_version from core_databases" + "select database_name, schema_version from catalog_databases" ) } for database_name, db in self.databases.items(): @@ -474,7 +474,7 @@ class Datasette: values = [database_name, db.is_memory, schema_version] await internal_db.execute_write( """ - INSERT OR REPLACE INTO core_databases (database_name, path, is_memory, schema_version) + INSERT OR REPLACE INTO catalog_databases (database_name, path, is_memory, schema_version) VALUES {} """.format( placeholders diff --git a/datasette/utils/internal_db.py b/datasette/utils/internal_db.py index 215695ca..2e5ac53b 100644 --- a/datasette/utils/internal_db.py +++ b/datasette/utils/internal_db.py @@ -5,13 +5,13 @@ from datasette.utils import table_column_details async def init_internal_db(db): create_tables_sql = textwrap.dedent( """ - CREATE TABLE IF NOT EXISTS core_databases ( + CREATE TABLE IF NOT EXISTS catalog_databases ( database_name TEXT PRIMARY KEY, path TEXT, is_memory INTEGER, schema_version INTEGER ); - CREATE TABLE IF NOT EXISTS core_tables ( + CREATE TABLE IF NOT EXISTS catalog_tables ( database_name TEXT, table_name TEXT, rootpage INTEGER, @@ -19,7 +19,7 @@ async def init_internal_db(db): PRIMARY KEY (database_name, table_name), FOREIGN KEY (database_name) REFERENCES databases(database_name) ); - CREATE TABLE IF NOT EXISTS core_columns ( + CREATE TABLE IF NOT EXISTS catalog_columns ( database_name TEXT, table_name TEXT, cid INTEGER, @@ -33,7 +33,7 @@ async def init_internal_db(db): FOREIGN KEY (database_name) REFERENCES databases(database_name), FOREIGN KEY (database_name, table_name) REFERENCES tables(database_name, table_name) ); - CREATE TABLE IF NOT EXISTS core_indexes ( + CREATE TABLE IF NOT EXISTS catalog_indexes ( database_name TEXT, table_name TEXT, seq INTEGER, @@ -45,7 +45,7 @@ async def init_internal_db(db): FOREIGN KEY (database_name) REFERENCES databases(database_name), FOREIGN KEY (database_name, table_name) REFERENCES tables(database_name, table_name) ); - CREATE TABLE IF NOT EXISTS core_foreign_keys ( + CREATE TABLE IF NOT EXISTS catalog_foreign_keys ( database_name TEXT, table_name TEXT, id INTEGER, @@ -69,15 +69,17 @@ async def populate_schema_tables(internal_db, db): database_name = db.name def delete_everything(conn): - conn.execute("DELETE FROM core_tables WHERE database_name = ?", [database_name]) conn.execute( - "DELETE FROM core_columns WHERE database_name = ?", [database_name] + "DELETE FROM catalog_tables WHERE database_name = ?", [database_name] ) conn.execute( - "DELETE FROM core_foreign_keys WHERE database_name = ?", [database_name] + "DELETE FROM catalog_columns WHERE database_name = ?", [database_name] ) conn.execute( - "DELETE FROM core_indexes WHERE database_name = ?", [database_name] + "DELETE FROM catalog_foreign_keys WHERE database_name = ?", [database_name] + ) + conn.execute( + "DELETE FROM catalog_indexes WHERE database_name = ?", [database_name] ) await internal_db.execute_write_fn(delete_everything) @@ -137,14 +139,14 @@ async def populate_schema_tables(internal_db, db): await internal_db.execute_write_many( """ - INSERT INTO core_tables (database_name, table_name, rootpage, sql) + INSERT INTO catalog_tables (database_name, table_name, rootpage, sql) values (?, ?, ?, ?) """, tables_to_insert, ) await internal_db.execute_write_many( """ - INSERT INTO core_columns ( + INSERT INTO catalog_columns ( database_name, table_name, cid, name, type, "notnull", default_value, is_pk, hidden ) VALUES ( :database_name, :table_name, :cid, :name, :type, :notnull, :default_value, :is_pk, :hidden @@ -154,7 +156,7 @@ async def populate_schema_tables(internal_db, db): ) await internal_db.execute_write_many( """ - INSERT INTO core_foreign_keys ( + INSERT INTO catalog_foreign_keys ( database_name, table_name, "id", seq, "table", "from", "to", on_update, on_delete, match ) VALUES ( :database_name, :table_name, :id, :seq, :table, :from, :to, :on_update, :on_delete, :match @@ -164,7 +166,7 @@ async def populate_schema_tables(internal_db, db): ) await internal_db.execute_write_many( """ - INSERT INTO core_indexes ( + INSERT INTO catalog_indexes ( database_name, table_name, seq, name, "unique", origin, partial ) VALUES ( :database_name, :table_name, :seq, :name, :unique, :origin, :partial diff --git a/datasette/views/database.py b/datasette/views/database.py index 4647bedc..9ba5ce94 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -952,7 +952,7 @@ class TableCreateView(BaseView): async def _table_columns(datasette, database_name): internal_db = datasette.get_internal_database() result = await internal_db.execute( - "select table_name, name from core_columns where database_name = ?", + "select table_name, name from catalog_columns where database_name = ?", [database_name], ) table_columns = {} diff --git a/docs/internals.rst b/docs/internals.rst index 474e3328..743f5972 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -1149,6 +1149,8 @@ Datasette's internal database Datasette maintains an "internal" SQLite database used for configuration, caching, and storage. Plugins can store configuration, settings, and other data inside this database. By default, Datasette will use a temporary in-memory SQLite database as the internal database, which is created at startup and destroyed at shutdown. Users of Datasette can optionally pass in a `--internal` flag to specify the path to a SQLite database to use as the internal database, which will persist internal data across Datasette instances. +Datasette maintains tables called ``catalog_databases``, ``catalog_tables``, ``catalog_columns``, ``catalog_indexes``, ``catalog_foreign_keys`` with details of the attached databases and their schemas. These tables should not be considered a stable API - they may change between Datasette releases. + The internal database is not exposed in the Datasette application by default, which means private data can safely be stored without worry of accidentally leaking information through the default Datasette interface and API. However, other plugins do have full read and write access to the internal database. Plugins can access this database by calling ``internal_db = datasette.get_internal_database()`` and then executing queries using the :ref:`Database API `. diff --git a/tests/test_internal_db.py b/tests/test_internal_db.py index 5276dc99..b41cabb4 100644 --- a/tests/test_internal_db.py +++ b/tests/test_internal_db.py @@ -10,7 +10,7 @@ async def ensure_internal(ds_client): @pytest.mark.asyncio async def test_internal_databases(ds_client): internal_db = await ensure_internal(ds_client) - databases = await internal_db.execute("select * from core_databases") + databases = await internal_db.execute("select * from catalog_databases") assert len(databases) == 1 assert databases.rows[0]["database_name"] == "fixtures" @@ -18,7 +18,7 @@ async def test_internal_databases(ds_client): @pytest.mark.asyncio async def test_internal_tables(ds_client): internal_db = await ensure_internal(ds_client) - tables = await internal_db.execute("select * from core_tables") + tables = await internal_db.execute("select * from catalog_tables") assert len(tables) > 5 table = tables.rows[0] assert set(table.keys()) == {"rootpage", "table_name", "database_name", "sql"} @@ -27,7 +27,7 @@ async def test_internal_tables(ds_client): @pytest.mark.asyncio async def test_internal_indexes(ds_client): internal_db = await ensure_internal(ds_client) - indexes = await internal_db.execute("select * from core_indexes") + indexes = await internal_db.execute("select * from catalog_indexes") assert len(indexes) > 5 index = indexes.rows[0] assert set(index.keys()) == { @@ -44,7 +44,7 @@ async def test_internal_indexes(ds_client): @pytest.mark.asyncio async def test_internal_foreign_keys(ds_client): internal_db = await ensure_internal(ds_client) - foreign_keys = await internal_db.execute("select * from core_foreign_keys") + foreign_keys = await internal_db.execute("select * from catalog_foreign_keys") assert len(foreign_keys) > 5 foreign_key = foreign_keys.rows[0] assert set(foreign_key.keys()) == { From 30b28c8367a9c6870386ea10a202705b40862457 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 29 Aug 2023 10:17:54 -0700 Subject: [PATCH 492/952] Release 1.0a5 Refs #2093, #2102, #2153, #2156, #2157 --- datasette/version.py | 2 +- docs/changelog.rst | 11 +++++++++++ 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/datasette/version.py b/datasette/version.py index 1d003352..b99c212b 100644 --- a/datasette/version.py +++ b/datasette/version.py @@ -1,2 +1,2 @@ -__version__ = "1.0a4" +__version__ = "1.0a5" __version_info__ = tuple(__version__.split(".")) diff --git a/docs/changelog.rst b/docs/changelog.rst index 937610bd..019d6c68 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -4,6 +4,17 @@ Changelog ========= +.. _v1_0_a5: + +1.0a5 (2023-08-29) +------------------ + +- When restrictions are applied to :ref:`API tokens `, those restrictions now behave slightly differently: applying the ``view-table`` restriction will imply the ability to ``view-database`` for the database containing that table, and both ``view-table`` and ``view-database`` will imply ``view-instance``. Previously you needed to create a token with restrictions that explicitly listed ``view-instance`` and ``view-database`` and ``view-table`` in order to view a table without getting a permission denied error. (:issue:`2102`) +- New ``datasette.yaml`` (or ``.json``) configuration file, which can be specified using ``datasette -c path-to-file``. The goal here to consolidate settings, plugin configuration, permissions, canned queries, and other Datasette configuration into a single single file, separate from ``metadata.yaml``. The legacy ``settings.json`` config file used for :ref:`config_dir` has been removed, and ``datasette.yaml`` has a ``"settings"`` section where the same settings key/value pairs can be included. In the next future alpha release, more configuration such as plugins/permissions/canned queries will be moved to the ``datasette.yaml`` file. See :issue:`2093` for more details. Thanks, Alex Garcia. +- The ``-s/--setting`` option can now take dotted paths to nested settings. These will then be used to set or over-ride the same options as are present in the new configuration file. (:issue:`2156`) +- New ``--actor '{"id": "json-goes-here"}'`` option for use with ``datasette --get`` to treat the simulated request as being made by a specific actor, see :ref:`cli_datasette_get`. (:issue:`2153`) +- The Datasette ``_internal`` database has had some changes. It no longer shows up in the ``datasette.databases`` list by default, and is now instead available to plugins using the ``datasette.get_internal_database()``. Plugins are invited to use this as a private database to store configuration and settings and secrets that should not be made visible through the default Datasette interface. Users can pass the new ``--internal internal.db`` option to persist that internal database to disk. Thanks, Alex Garcia. (:issue:`2157`). + .. _v1_0_a4: 1.0a4 (2023-08-21) From 6bfe104d47b888c70bfb7781f8f48ff11452b2b5 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 30 Aug 2023 15:12:24 -0700 Subject: [PATCH 493/952] DATASETTE_LOAD_PLUGINS environment variable for loading specific plugins Closes #2164 * Load only specified plugins for DATASETTE_LOAD_PLUGINS=datasette-one,datasette-two * Load no plugins if DATASETTE_LOAD_PLUGINS='' * Automated tests in a Bash script for DATASETTE_LOAD_PLUGINS --- .github/workflows/test.yml | 4 +++ datasette/plugins.py | 23 +++++++++++- docs/plugins.rst | 54 ++++++++++++++++++++++++++++ tests/test-datasette-load-plugins.sh | 29 +++++++++++++++ 4 files changed, 109 insertions(+), 1 deletion(-) create mode 100755 tests/test-datasette-load-plugins.sh diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 8cbbb572..2784db86 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -50,3 +50,7 @@ jobs: run: | # This fails on syntax errors, or a diff was applied blacken-docs -l 60 docs/*.rst + - name: Test DATASETTE_LOAD_PLUGINS + run: | + pip install datasette-init datasette-json-html + tests/test-datasette-load-plugins.sh diff --git a/datasette/plugins.py b/datasette/plugins.py index fef0c8e9..6ec08a81 100644 --- a/datasette/plugins.py +++ b/datasette/plugins.py @@ -1,4 +1,5 @@ import importlib +import os import pluggy import pkg_resources import sys @@ -22,10 +23,30 @@ DEFAULT_PLUGINS = ( pm = pluggy.PluginManager("datasette") pm.add_hookspecs(hookspecs) -if not hasattr(sys, "_called_from_test"): +DATASETTE_LOAD_PLUGINS = os.environ.get("DATASETTE_LOAD_PLUGINS", None) + +if not hasattr(sys, "_called_from_test") and DATASETTE_LOAD_PLUGINS is None: # Only load plugins if not running tests pm.load_setuptools_entrypoints("datasette") +# Load any plugins specified in DATASETTE_LOAD_PLUGINS") +if DATASETTE_LOAD_PLUGINS is not None: + for package_name in [ + name for name in DATASETTE_LOAD_PLUGINS.split(",") if name.strip() + ]: + try: + distribution = pkg_resources.get_distribution(package_name) + entry_map = distribution.get_entry_map() + if "datasette" in entry_map: + for plugin_name, entry_point in entry_map["datasette"].items(): + mod = entry_point.load() + pm.register(mod, name=entry_point.name) + # Ensure name can be found in plugin_to_distinfo later: + pm._plugin_distinfo.append((mod, distribution)) + except pkg_resources.DistributionNotFound: + sys.stderr.write("Plugin {} could not be found\n".format(package_name)) + + # Load default plugins for plugin in DEFAULT_PLUGINS: mod = importlib.import_module(plugin) diff --git a/docs/plugins.rst b/docs/plugins.rst index 19bfdd0c..11db40af 100644 --- a/docs/plugins.rst +++ b/docs/plugins.rst @@ -81,6 +81,60 @@ You can use the name of a package on PyPI or any of the other valid arguments to datasette publish cloudrun mydb.db \ --install=https://url-to-my-package.zip + +.. _plugins_datasette_load_plugins: + +Controlling which plugins are loaded +------------------------------------ + +Datasette defaults to loading every plugin that is installed in the same virtual environment as Datasette itself. + +You can set the ``DATASETTE_LOAD_PLUGINS`` environment variable to a comma-separated list of plugin names to load a controlled subset of plugins instead. + +For example, to load just the ``datasette-vega`` and ``datasette-cluster-map`` plugins, set ``DATASETTE_LOAD_PLUGINS`` to ``datasette-vega,datasette-cluster-map``: + +.. code-block:: bash + + export DATASETTE_LOAD_PLUGINS='datasette-vega,datasette-cluster-map' + datasette mydb.db + +Or: + +.. code-block:: bash + + DATASETTE_LOAD_PLUGINS='datasette-vega,datasette-cluster-map' \ + datasette mydb.db + +To disable the loading of all additional plugins, set ``DATASETTE_LOAD_PLUGINS`` to an empty string: + +.. code-block:: bash + + export DATASETTE_LOAD_PLUGINS='' + datasette mydb.db + +A quick way to test this setting is to use it with the ``datasette plugins`` command: + +.. code-block:: bash + + DATASETTE_LOAD_PLUGINS='datasette-vega' datasette plugins + +This should output the following: + +.. code-block:: json + + [ + { + "name": "datasette-vega", + "static": true, + "templates": false, + "version": "0.6.2", + "hooks": [ + "extra_css_urls", + "extra_js_urls" + ] + } + ] + .. _plugins_installed: Seeing what plugins are installed diff --git a/tests/test-datasette-load-plugins.sh b/tests/test-datasette-load-plugins.sh new file mode 100755 index 00000000..e26d8377 --- /dev/null +++ b/tests/test-datasette-load-plugins.sh @@ -0,0 +1,29 @@ +#!/bin/bash +# This should only run in environemnts where both +# datasette-init and datasette-json-html are installed + +PLUGINS=$(datasette plugins) +echo "$PLUGINS" | jq 'any(.[]; .name == "datasette-json-html")' | \ + grep -q true || ( \ + echo "Test failed: datasette-json-html not found" && \ + exit 1 \ + ) +# With the DATASETTE_LOAD_PLUGINS we should not see that +PLUGINS2=$(DATASETTE_LOAD_PLUGINS=datasette-init datasette plugins) +echo "$PLUGINS2" | jq 'any(.[]; .name == "datasette-json-html")' | \ + grep -q false || ( \ + echo "Test failed: datasette-json-html should not have been loaded" && \ + exit 1 \ + ) +echo "$PLUGINS2" | jq 'any(.[]; .name == "datasette-init")' | \ + grep -q true || ( \ + echo "Test failed: datasette-init should have been loaded" && \ + exit 1 \ + ) +# With DATASETTE_LOAD_PLUGINS='' we should see no plugins +PLUGINS3=$(DATASETTE_LOAD_PLUGINS='' datasette plugins) +echo "$PLUGINS3"| \ + grep -q '\[\]' || ( \ + echo "Test failed: datasette plugins should have returned []" && \ + exit 1 \ + ) From 2caa53a52a37e53f83e3a854fc721c7e26c5e9ff Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 30 Aug 2023 16:19:24 -0700 Subject: [PATCH 494/952] ReST fix --- docs/internals.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/internals.rst b/docs/internals.rst index 743f5972..d136ad5a 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -1147,7 +1147,7 @@ You can selectively disable CSRF protection using the :ref:`plugin_hook_skip_csr Datasette's internal database ============================= -Datasette maintains an "internal" SQLite database used for configuration, caching, and storage. Plugins can store configuration, settings, and other data inside this database. By default, Datasette will use a temporary in-memory SQLite database as the internal database, which is created at startup and destroyed at shutdown. Users of Datasette can optionally pass in a `--internal` flag to specify the path to a SQLite database to use as the internal database, which will persist internal data across Datasette instances. +Datasette maintains an "internal" SQLite database used for configuration, caching, and storage. Plugins can store configuration, settings, and other data inside this database. By default, Datasette will use a temporary in-memory SQLite database as the internal database, which is created at startup and destroyed at shutdown. Users of Datasette can optionally pass in a ``--internal`` flag to specify the path to a SQLite database to use as the internal database, which will persist internal data across Datasette instances. Datasette maintains tables called ``catalog_databases``, ``catalog_tables``, ``catalog_columns``, ``catalog_indexes``, ``catalog_foreign_keys`` with details of the attached databases and their schemas. These tables should not be considered a stable API - they may change between Datasette releases. From 4c3ef033110407f3b3dbce501659d523724985e0 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 30 Aug 2023 16:19:59 -0700 Subject: [PATCH 495/952] Another ReST fix --- docs/internals.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/internals.rst b/docs/internals.rst index d136ad5a..540e7058 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -1157,7 +1157,7 @@ Plugins can access this database by calling ``internal_db = datasette.get_intern Plugin authors are asked to practice good etiquette when using the internal database, as all plugins use the same database to store data. For example: -1. Use a unique prefix when creating tables, indices, and triggera in the internal database. If your plugin is called `datasette-xyz`, then prefix names with `datasette_xyz_*`. +1. Use a unique prefix when creating tables, indices, and triggera in the internal database. If your plugin is called ``datasette-xyz``, then prefix names with ``datasette_xyz_*``. 2. Avoid long-running write statements that may stall or block other plugins that are trying to write at the same time. 3. Use temporary tables or shared in-memory attached databases when possible. 4. Avoid implementing features that could expose private data stored in the internal database by other plugins. From 9cead33fb9c8704996181f1ab67c7376dee97f15 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 31 Aug 2023 10:46:07 -0700 Subject: [PATCH 496/952] OperationalError: database table is locked fix See also: - https://til.simonwillison.net/datasette/remember-to-commit --- docs/internals.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/internals.rst b/docs/internals.rst index 540e7058..6b7d3df8 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -1012,6 +1012,7 @@ For example: def delete_and_return_count(conn): conn.execute("delete from some_table where id > 5") + conn.commit() return conn.execute( "select count(*) from some_table" ).fetchone()[0] @@ -1028,6 +1029,8 @@ The value returned from ``await database.execute_write_fn(...)`` will be the ret If your function raises an exception that exception will be propagated up to the ``await`` line. +If you see ``OperationalError: database table is locked`` errors you should check that you remembered to explicitly call ``conn.commit()`` in your write function. + If you specify ``block=False`` the method becomes fire-and-forget, queueing your function to be executed and then allowing your code after the call to ``.execute_write_fn()`` to continue running while the underlying thread waits for an opportunity to run your function. A UUID representing the queued task will be returned. Any exceptions in your code will be silently swallowed. .. _database_close: From 98ffad9aed15a300e61fb712fa12f177844739b3 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 31 Aug 2023 15:46:18 -0700 Subject: [PATCH 497/952] execute-sql now implies can view instance/database, closes #2169 --- datasette/default_permissions.py | 1 + tests/test_permissions.py | 4 ++++ 2 files changed, 5 insertions(+) diff --git a/datasette/default_permissions.py b/datasette/default_permissions.py index 960429fc..5a99d0d8 100644 --- a/datasette/default_permissions.py +++ b/datasette/default_permissions.py @@ -59,6 +59,7 @@ def register_permissions(): takes_database=True, takes_resource=False, default=True, + implies_can_view=True, ), Permission( name="permissions-debug", diff --git a/tests/test_permissions.py b/tests/test_permissions.py index cad0525f..b3987cff 100644 --- a/tests/test_permissions.py +++ b/tests/test_permissions.py @@ -1183,6 +1183,10 @@ async def test_actor_restrictions( ({"a": ["update-row"]}, "view-instance", None, False), # view-table on a resource implies view-instance ({"r": {"db1": {"t1": ["view-table"]}}}, "view-instance", None, True), + # execute-sql on a database implies view-instance, view-database + ({"d": {"db1": ["es"]}}, "view-instance", None, True), + ({"d": {"db1": ["es"]}}, "view-database", "db1", True), + ({"d": {"db1": ["es"]}}, "view-database", "db2", False), # update-row on a resource does not imply view-instance ({"r": {"db1": {"t1": ["update-row"]}}}, "view-instance", None, False), # view-database on a resource implies view-instance From fd083e37ec53e7e625111168d324a572344a3b19 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 31 Aug 2023 16:06:30 -0700 Subject: [PATCH 498/952] Docs for plugins that define more plugin hooks, closes #1765 --- docs/writing_plugins.rst | 57 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/docs/writing_plugins.rst b/docs/writing_plugins.rst index a84789b5..a4c96011 100644 --- a/docs/writing_plugins.rst +++ b/docs/writing_plugins.rst @@ -325,3 +325,60 @@ This object is exposed in templates as the ``urls`` variable, which can be used Back to the Homepage See :ref:`internals_datasette_urls` for full details on this object. + +.. _writing_plugins_extra_hooks: + +Plugins that define new plugin hooks +------------------------------------ + +Plugins can define new plugin hooks that other plugins can use to further extend their functionality. + +`datasette-graphql `__ is one example of a plugin that does this. It defines a new hook called ``graphql_extra_fields``, `described here `__, which other plugins can use to define additional fields that should be included in the GraphQL schema. + +To define additional hooks, add a file to the plugin called ``datasette_your_plugin/hookspecs.py`` with content that looks like this: + +.. code-block:: python + + from pluggy import HookspecMarker + + hookspec = HookspecMarker("datasette") + + @hookspec + def name_of_your_hook_goes_here(datasette): + "Description of your hook." + +You should define your own hook name and arguments here, following the documentation for `Pluggy specifications `__. Make sure to pick a name that is unlikely to clash with hooks provided by any other plugins. + +Then, to register your plugin hooks, add the following code to your ``datasette_your_plugin/__init__.py`` file: + +.. code-block:: python + + from datasette.plugins import pm + from . import hookspecs + + pm.add_hookspecs(hookspecs) + +This will register your plugin hooks as part of the ``datasette`` plugin hook namespace. + +Within your plugin code you can trigger the hook using this pattern: + +.. code-block:: python + + from datasette.plugins import pm + + for plugin_return_value in pm.hook.name_of_your_hook_goes_here( + datasette=datasette + ): + # Do something with plugin_return_value + +Other plugins will then be able to register their own implementations of your hook using this syntax: + +.. code-block:: python + + from datasette import hookimpl + + @hookimpl + def name_of_your_hook_goes_here(datasette): + return "Response from this plugin hook" + +These plugin implementations can accept 0 or more of the named arguments that you defined in your hook specification. From 31d5c4ec05e27165283f0f0004c32227d8b78df8 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 5 Sep 2023 19:43:01 -0700 Subject: [PATCH 499/952] Contraction - Google and Microsoft styleguides like it I was trying out https://github.com/errata-ai/vale --- docs/authentication.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/authentication.rst b/docs/authentication.rst index 814d2e67..1a444d0c 100644 --- a/docs/authentication.rst +++ b/docs/authentication.rst @@ -4,7 +4,7 @@ Authentication and permissions ================================ -Datasette does not require authentication by default. Any visitor to a Datasette instance can explore the full data and execute read-only SQL queries. +Datasette doesn't require authentication by default. Any visitor to a Datasette instance can explore the full data and execute read-only SQL queries. Datasette's plugin system can be used to add many different styles of authentication, such as user accounts, single sign-on or API keys. From 05707aa16b5c6c39fbe48b3176b85a8ffe493938 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 5 Sep 2023 19:50:09 -0700 Subject: [PATCH 500/952] click-default-group>=1.2.3 (#2173) * click-default-group>=1.2.3 Now available as a wheel: - https://github.com/click-contrib/click-default-group/issues/21 * Fix for blacken-docs --- docs/writing_plugins.rst | 7 ++++++- setup.py | 2 +- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/docs/writing_plugins.rst b/docs/writing_plugins.rst index a4c96011..d0dd8f36 100644 --- a/docs/writing_plugins.rst +++ b/docs/writing_plugins.rst @@ -343,6 +343,7 @@ To define additional hooks, add a file to the plugin called ``datasette_your_plu hookspec = HookspecMarker("datasette") + @hookspec def name_of_your_hook_goes_here(datasette): "Description of your hook." @@ -366,10 +367,13 @@ Within your plugin code you can trigger the hook using this pattern: from datasette.plugins import pm - for plugin_return_value in pm.hook.name_of_your_hook_goes_here( + for ( + plugin_return_value + ) in pm.hook.name_of_your_hook_goes_here( datasette=datasette ): # Do something with plugin_return_value + pass Other plugins will then be able to register their own implementations of your hook using this syntax: @@ -377,6 +381,7 @@ Other plugins will then be able to register their own implementations of your ho from datasette import hookimpl + @hookimpl def name_of_your_hook_goes_here(datasette): return "Response from this plugin hook" diff --git a/setup.py b/setup.py index 35c9b68b..7d8c1ebc 100644 --- a/setup.py +++ b/setup.py @@ -44,7 +44,7 @@ setup( install_requires=[ "asgiref>=3.2.10", "click>=7.1.1", - "click-default-group-wheel>=1.2.2", + "click-default-group>=1.2.3", "Jinja2>=2.10.3", "hupper>=1.9", "httpx>=0.20", From e86eaaa4f371512689e973c18879298dab51f80a Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 6 Sep 2023 09:16:27 -0700 Subject: [PATCH 501/952] Test against Python 3.12 preview (#2175) https://dev.to/hugovk/help-test-python-312-beta-1508/ --- .github/workflows/test.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 2784db86..656b0b1c 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -10,13 +10,14 @@ jobs: runs-on: ubuntu-latest strategy: matrix: - python-version: ["3.8", "3.9", "3.10", "3.11"] + python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"] steps: - uses: actions/checkout@v3 - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v4 with: python-version: ${{ matrix.python-version }} + allow-prereleases: true - uses: actions/cache@v3 name: Configure pip caching with: From e4abae3fd7a828625d00c35c316852ffbaa5ef2f Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 6 Sep 2023 09:34:31 -0700 Subject: [PATCH 502/952] Bump Sphinx (#2166) Bumps the python-packages group with 1 update: [sphinx](https://github.com/sphinx-doc/sphinx). - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/master/CHANGES) - [Commits](https://github.com/sphinx-doc/sphinx/compare/v7.2.4...v7.2.5) --- updated-dependencies: - dependency-name: sphinx dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-packages ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 7d8c1ebc..c718086b 100644 --- a/setup.py +++ b/setup.py @@ -69,7 +69,7 @@ setup( setup_requires=["pytest-runner"], extras_require={ "docs": [ - "Sphinx==7.2.4", + "Sphinx==7.2.5", "furo==2023.8.19", "sphinx-autobuild", "codespell>=2.2.5", From fbcb103c0cb6668018ace539a01a6a1f156e8d6a Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 7 Sep 2023 07:47:24 -0700 Subject: [PATCH 503/952] Added example code to database_actions hook documentation --- docs/plugin_hooks.rst | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index f8bb203d..84a045b0 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -1439,7 +1439,32 @@ database_actions(datasette, actor, database, request) This hook is similar to :ref:`plugin_hook_table_actions` but populates an actions menu on the database page. -Example: `datasette-graphql `_ +This example adds a new database action for creating a table, if the user has the ``edit-schema`` permission: + +.. code-block:: python + + from datasette import hookimpl + + + @hookimpl + def database_actions(datasette, actor, database): + async def inner(): + if not await datasette.permission_allowed( + actor, "edit-schema", resource=database, default=False + ): + return [] + return [ + { + "href": datasette.urls.path( + "/-/edit-schema/{}/-/create".format(database) + ), + "label": "Create a table", + } + ] + + return inner + +Example: `datasette-graphql `_, `datasette-edit-schema `_ .. _plugin_hook_skip_csrf: From 2200abfa17f72b1cb741a36b44dc40a04b8ea001 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 7 Sep 2023 15:49:50 -0700 Subject: [PATCH 504/952] Fix for flaky test_hidden_sqlite_stat1_table, closes #2179 --- tests/test_api.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/test_api.py b/tests/test_api.py index f96f571e..93ca43eb 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -1017,7 +1017,10 @@ async def test_hidden_sqlite_stat1_table(): await db.execute_write("analyze") data = (await ds.client.get("/db.json?_show_hidden=1")).json() tables = [(t["name"], t["hidden"]) for t in data["tables"]] - assert tables == [("normal", False), ("sqlite_stat1", True)] + assert tables in ( + [("normal", False), ("sqlite_stat1", True)], + [("normal", False), ("sqlite_stat1", True), ("sqlite_stat4", True)], + ) @pytest.mark.asyncio From dbfad6d2201bc65a0c73e699a10c479c1e199e11 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 7 Sep 2023 15:51:09 -0700 Subject: [PATCH 505/952] Foreign key label expanding respects table permissions, closes #2178 --- datasette/app.py | 9 ++++++- datasette/facets.py | 2 +- datasette/views/table.py | 2 +- tests/test_table_html.py | 53 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 63 insertions(+), 3 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 0227f627..618c0ecc 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -935,7 +935,7 @@ class Datasette: log_sql_errors=log_sql_errors, ) - async def expand_foreign_keys(self, database, table, column, values): + async def expand_foreign_keys(self, actor, database, table, column, values): """Returns dict mapping (column, value) -> label""" labeled_fks = {} db = self.databases[database] @@ -949,6 +949,13 @@ class Datasette: ][0] except IndexError: return {} + # Ensure user has permission to view the referenced table + if not await self.permission_allowed( + actor=actor, + action="view-table", + resource=(database, fk["other_table"]), + ): + return {} label_column = await db.label_column_for_table(fk["other_table"]) if not label_column: return {(fk["column"], value): str(value) for value in values} diff --git a/datasette/facets.py b/datasette/facets.py index 7fb0c68b..b23615fe 100644 --- a/datasette/facets.py +++ b/datasette/facets.py @@ -253,7 +253,7 @@ class ColumnFacet(Facet): # Attempt to expand foreign keys into labels values = [row["value"] for row in facet_rows] expanded = await self.ds.expand_foreign_keys( - self.database, self.table, column, values + self.request.actor, self.database, self.table, column, values ) else: expanded = {} diff --git a/datasette/views/table.py b/datasette/views/table.py index 6df8b915..50ba2b78 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1144,7 +1144,7 @@ async def table_view_data( # Expand them expanded_labels.update( await datasette.expand_foreign_keys( - database_name, table_name, column, values + request.actor, database_name, table_name, column, values ) ) if expanded_labels: diff --git a/tests/test_table_html.py b/tests/test_table_html.py index c4c7878c..e66eb6f0 100644 --- a/tests/test_table_html.py +++ b/tests/test_table_html.py @@ -1204,3 +1204,56 @@ async def test_format_of_binary_links(size, title, length_bytes): sql_response = await ds.client.get("/{}".format(db_name), params={"sql": sql}) assert sql_response.status_code == 200 assert expected in sql_response.text + + +@pytest.mark.asyncio +async def test_foreign_key_labels_obey_permissions(): + ds = Datasette( + metadata={ + "databases": { + "foreign_key_labels": { + "tables": { + # Table a is only visible to root + "a": {"allow": {"id": "root"}}, + } + } + } + } + ) + db = ds.add_memory_database("foreign_key_labels") + await db.execute_write("create table a(id integer primary key, name text)") + await db.execute_write("insert into a (id, name) values (1, 'hello')") + await db.execute_write( + "create table b(id integer primary key, name text, a_id integer references a(id))" + ) + await db.execute_write("insert into b (id, name, a_id) values (1, 'world', 1)") + # Anonymous user can see table b but not table a + blah = await ds.client.get("/foreign_key_labels.json") + anon_a = await ds.client.get("/foreign_key_labels/a.json?_labels=on") + assert anon_a.status_code == 403 + anon_b = await ds.client.get("/foreign_key_labels/b.json?_labels=on") + assert anon_b.status_code == 200 + # root user can see both + cookies = {"ds_actor": ds.sign({"a": {"id": "root"}}, "actor")} + root_a = await ds.client.get( + "/foreign_key_labels/a.json?_labels=on", cookies=cookies + ) + assert root_a.status_code == 200 + root_b = await ds.client.get( + "/foreign_key_labels/b.json?_labels=on", cookies=cookies + ) + assert root_b.status_code == 200 + # Labels should have been expanded for root + assert root_b.json() == { + "ok": True, + "next": None, + "rows": [{"id": 1, "name": "world", "a_id": {"value": 1, "label": "hello"}}], + "truncated": False, + } + # But not for anon + assert anon_b.json() == { + "ok": True, + "next": None, + "rows": [{"id": 1, "name": "world", "a_id": 1}], + "truncated": False, + } From ab040470e2b191c0de48b213193da71e48cd66ed Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 7 Sep 2023 15:57:27 -0700 Subject: [PATCH 506/952] Applied blacken-docs --- docs/plugin_hooks.rst | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index 84a045b0..04fb24ce 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -1450,13 +1450,18 @@ This example adds a new database action for creating a table, if the user has th def database_actions(datasette, actor, database): async def inner(): if not await datasette.permission_allowed( - actor, "edit-schema", resource=database, default=False + actor, + "edit-schema", + resource=database, + default=False, ): return [] return [ { "href": datasette.urls.path( - "/-/edit-schema/{}/-/create".format(database) + "/-/edit-schema/{}/-/create".format( + database + ) ), "label": "Create a table", } From c26370485a4fd4bf130da051be9163d92c57f24f Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 7 Sep 2023 16:28:30 -0700 Subject: [PATCH 507/952] Label expand permission check respects cascade, closes #2178 --- datasette/app.py | 22 ++++++++++------- tests/test_table_html.py | 52 +++++++++++++++++++++++++++++++++------- 2 files changed, 57 insertions(+), 17 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 618c0ecc..ea9739f0 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -950,13 +950,19 @@ class Datasette: except IndexError: return {} # Ensure user has permission to view the referenced table - if not await self.permission_allowed( - actor=actor, - action="view-table", - resource=(database, fk["other_table"]), - ): + other_table = fk["other_table"] + other_column = fk["other_column"] + visible, _ = await self.check_visibility( + actor, + permissions=[ + ("view-table", (database, other_table)), + ("view-database", database), + "view-instance", + ], + ) + if not visible: return {} - label_column = await db.label_column_for_table(fk["other_table"]) + label_column = await db.label_column_for_table(other_table) if not label_column: return {(fk["column"], value): str(value) for value in values} labeled_fks = {} @@ -965,9 +971,9 @@ class Datasette: from {other_table} where {other_column} in ({placeholders}) """.format( - other_column=escape_sqlite(fk["other_column"]), + other_column=escape_sqlite(other_column), label_column=escape_sqlite(label_column), - other_table=escape_sqlite(fk["other_table"]), + other_table=escape_sqlite(other_table), placeholders=", ".join(["?"] * len(set(values))), ) try: diff --git a/tests/test_table_html.py b/tests/test_table_html.py index e66eb6f0..6707665d 100644 --- a/tests/test_table_html.py +++ b/tests/test_table_html.py @@ -1207,9 +1207,11 @@ async def test_format_of_binary_links(size, title, length_bytes): @pytest.mark.asyncio -async def test_foreign_key_labels_obey_permissions(): - ds = Datasette( - metadata={ +@pytest.mark.parametrize( + "metadata", + ( + # Blocked at table level + { "databases": { "foreign_key_labels": { "tables": { @@ -1218,15 +1220,47 @@ async def test_foreign_key_labels_obey_permissions(): } } } - } - ) + }, + # Blocked at database level + { + "databases": { + "foreign_key_labels": { + # Only root can view this database + "allow": {"id": "root"}, + "tables": { + # But table b is visible to everyone + "b": {"allow": True}, + }, + } + } + }, + # Blocked at the instance level + { + "allow": {"id": "root"}, + "databases": { + "foreign_key_labels": { + "tables": { + # Table b is visible to everyone + "b": {"allow": True}, + } + } + }, + }, + ), +) +async def test_foreign_key_labels_obey_permissions(metadata): + ds = Datasette(metadata=metadata) db = ds.add_memory_database("foreign_key_labels") - await db.execute_write("create table a(id integer primary key, name text)") - await db.execute_write("insert into a (id, name) values (1, 'hello')") await db.execute_write( - "create table b(id integer primary key, name text, a_id integer references a(id))" + "create table if not exists a(id integer primary key, name text)" + ) + await db.execute_write("insert or replace into a (id, name) values (1, 'hello')") + await db.execute_write( + "create table if not exists b(id integer primary key, name text, a_id integer references a(id))" + ) + await db.execute_write( + "insert or replace into b (id, name, a_id) values (1, 'world', 1)" ) - await db.execute_write("insert into b (id, name, a_id) values (1, 'world', 1)") # Anonymous user can see table b but not table a blah = await ds.client.get("/foreign_key_labels.json") anon_a = await ds.client.get("/foreign_key_labels/a.json?_labels=on") From b645174271aa08e8ca83b27ff83ce078ecd15da2 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 7 Sep 2023 21:23:59 -0700 Subject: [PATCH 508/952] actors_from_ids plugin hook and datasette.actors_from_ids() method (#2181) * Prototype of actors_from_ids plugin hook, refs #2180 * datasette-remote-actors example plugin, refs #2180 --- datasette/app.py | 10 +++++++ datasette/hookspecs.py | 5 ++++ docs/internals.rst | 21 ++++++++++++++ docs/plugin_hooks.rst | 57 ++++++++++++++++++++++++++++++++++++++ tests/test_plugins.py | 62 ++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 155 insertions(+) diff --git a/datasette/app.py b/datasette/app.py index ea9739f0..fdec2c86 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -819,6 +819,16 @@ class Datasette: ) return crumbs + async def actors_from_ids( + self, actor_ids: Iterable[Union[str, int]] + ) -> Dict[Union[id, str], Dict]: + result = pm.hook.actors_from_ids(datasette=self, actor_ids=actor_ids) + if result is None: + # Do the default thing + return {actor_id: {"id": actor_id} for actor_id in actor_ids} + result = await await_me_maybe(result) + return result + async def permission_allowed( self, actor, action, resource=None, default=DEFAULT_NOT_SET ): diff --git a/datasette/hookspecs.py b/datasette/hookspecs.py index 801073fc..9069927b 100644 --- a/datasette/hookspecs.py +++ b/datasette/hookspecs.py @@ -94,6 +94,11 @@ def actor_from_request(datasette, request): """Return an actor dictionary based on the incoming request""" +@hookspec(firstresult=True) +def actors_from_ids(datasette, actor_ids): + """Returns a dictionary mapping those IDs to actor dictionaries""" + + @hookspec def filters_from_request(request, database, table, datasette): """ diff --git a/docs/internals.rst b/docs/internals.rst index 6b7d3df8..13f1d4a1 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -322,6 +322,27 @@ await .render_template(template, context=None, request=None) Renders a `Jinja template `__ using Datasette's preconfigured instance of Jinja and returns the resulting string. The template will have access to Datasette's default template functions and any functions that have been made available by other plugins. +.. _datasette_actors_from_ids: + +await .actors_from_ids(actor_ids) +--------------------------------- + +``actor_ids`` - list of strings or integers + A list of actor IDs to look up. + +Returns a dictionary, where the keys are the IDs passed to it and the values are the corresponding actor dictionaries. + +This method is mainly designed to be used with plugins. See the :ref:`plugin_hook_actors_from_ids` documentation for details. + +If no plugins that implement that hook are installed, the default return value looks like this: + +.. code-block:: json + + { + "1": {"id": "1"}, + "2": {"id": "2"} + } + .. _datasette_permission_allowed: await .permission_allowed(actor, action, resource=None, default=...) diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index 04fb24ce..e966919b 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -1071,6 +1071,63 @@ Instead of returning a dictionary, this function can return an awaitable functio Examples: `datasette-auth-tokens `_, `datasette-auth-passwords `_ +.. _plugin_hook_actors_from_ids: + +actors_from_ids(datasette, actor_ids) +------------------------------------- + +``datasette`` - :ref:`internals_datasette` + You can use this to access plugin configuration options via ``datasette.plugin_config(your_plugin_name)``, or to execute SQL queries. + +``actor_ids`` - list of strings or integers + The actor IDs to look up. + +The hook must return a dictionary that maps the incoming actor IDs to their full dictionary representation. + +Some plugins that implement social features may store the ID of the :ref:`actor ` that performed an action - added a comment, bookmarked a table or similar - and then need a way to resolve those IDs into display-friendly actor dictionaries later on. + +Unlike other plugin hooks, this only uses the first implementation of the hook to return a result. You can expect users to only have a single plugin installed that implements this hook. + +If no plugin is installed, Datasette defaults to returning actors that are just ``{"id": actor_id}``. + +The hook can return a dictionary or an awaitable function that then returns a dictionary. + +This example implementation returns actors from a database table: + +.. code-block:: python + + from datasette import hookimpl + + + @hookimpl + def actors_from_ids(datasette, actor_ids): + db = datasette.get_database("actors") + + async def inner(): + sql = "select id, name from actors where id in ({})".format( + ", ".join("?" for _ in actor_ids) + ) + actors = {} + for row in (await db.execute(sql, actor_ids)).rows: + actor = dict(row) + actors[actor["id"]] = actor + return actors + + return inner + +The returned dictionary from this example looks like this: + +.. code-block:: json + + { + "1": {"id": "1", "name": "Tony"}, + "2": {"id": "2", "name": "Tina"}, + } + +These IDs could be integers or strings, depending on how the actors used by the Datasette instance are configured. + +Example: `datasette-remote-actors `_ + .. _plugin_hook_filters_from_request: filters_from_request(request, database, table, datasette) diff --git a/tests/test_plugins.py b/tests/test_plugins.py index 9761fa53..625ae635 100644 --- a/tests/test_plugins.py +++ b/tests/test_plugins.py @@ -1215,3 +1215,65 @@ async def test_hook_register_permissions_allows_identical_duplicates(): await ds.invoke_startup() # Check that ds.permissions has only one of each assert len([p for p in ds.permissions.values() if p.abbr == "abbr1"]) == 1 + + +@pytest.mark.asyncio +async def test_hook_actors_from_ids(): + # Without the hook should return default {"id": id} list + ds = Datasette() + await ds.invoke_startup() + db = ds.add_memory_database("actors_from_ids") + await db.execute_write( + "create table actors (id text primary key, name text, age int)" + ) + await db.execute_write( + "insert into actors (id, name, age) values ('3', 'Cate Blanchett', 52)" + ) + await db.execute_write( + "insert into actors (id, name, age) values ('5', 'Rooney Mara', 36)" + ) + await db.execute_write( + "insert into actors (id, name, age) values ('7', 'Sarah Paulson', 46)" + ) + await db.execute_write( + "insert into actors (id, name, age) values ('9', 'Helena Bonham Carter', 55)" + ) + table_names = await db.table_names() + assert table_names == ["actors"] + actors1 = await ds.actors_from_ids(["3", "5", "7"]) + assert actors1 == { + "3": {"id": "3"}, + "5": {"id": "5"}, + "7": {"id": "7"}, + } + + class ActorsFromIdsPlugin: + __name__ = "ActorsFromIdsPlugin" + + @hookimpl + def actors_from_ids(self, datasette, actor_ids): + db = datasette.get_database("actors_from_ids") + + async def inner(): + sql = "select id, name from actors where id in ({})".format( + ", ".join("?" for _ in actor_ids) + ) + actors = {} + result = await db.execute(sql, actor_ids) + for row in result.rows: + actor = dict(row) + actors[actor["id"]] = actor + return actors + + return inner + + try: + pm.register(ActorsFromIdsPlugin(), name="ActorsFromIdsPlugin") + actors2 = await ds.actors_from_ids(["3", "5", "7"]) + assert actors2 == { + "3": {"id": "3", "name": "Cate Blanchett"}, + "5": {"id": "5", "name": "Rooney Mara"}, + "7": {"id": "7", "name": "Sarah Paulson"}, + } + finally: + pm.unregister(name="ReturnNothingPlugin") From a4c96d01b27ce7cd06662a024da3547132a7c412 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 7 Sep 2023 21:44:08 -0700 Subject: [PATCH 509/952] Release 1.0a6 Refs #1765, #2164, #2169, #2175, #2178, #2181 --- datasette/version.py | 2 +- docs/changelog.rst | 12 ++++++++++++ 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/datasette/version.py b/datasette/version.py index b99c212b..4b65999d 100644 --- a/datasette/version.py +++ b/datasette/version.py @@ -1,2 +1,2 @@ -__version__ = "1.0a5" +__version__ = "1.0a6" __version_info__ = tuple(__version__.split(".")) diff --git a/docs/changelog.rst b/docs/changelog.rst index 019d6c68..81554f83 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -4,6 +4,18 @@ Changelog ========= +.. _v1_0_a6: + +1.0a6 (2023-09-07) +------------------ + +- New plugin hook: :ref:`plugin_hook_actors_from_ids` and an internal method to accompany it, :ref:`datasette_actors_from_ids`. This mechanism is intended to be used by plugins that may need to display the actor who was responsible for something managed by that plugin: they can now resolve the recorded IDs of actors into the full actor objects. (:issue:`2181`) +- ``DATASETTE_LOAD_PLUGINS`` environment variable for :ref:`controlling which plugins ` are loaded by Datasette. (:issue:`2164`) +- Datasette now checks if the user has permission to view a table linked to by a foreign key before turning that foreign key into a clickable link. (:issue:`2178`) +- The ``execute-sql`` permission now implies that the actor can also view the database and instance. (:issue:`2169`) +- Documentation describing a pattern for building plugins that themselves :ref:`define further hooks ` for other plugins. (:issue:`1765`) +- Datasette is now tested against the Python 3.12 preview. (`#2175 `__) + .. _v1_0_a5: 1.0a5 (2023-08-29) From b2ec8717c3619260a1b535eea20e618bf95aa30b Mon Sep 17 00:00:00 2001 From: Alex Garcia Date: Wed, 13 Sep 2023 14:06:25 -0700 Subject: [PATCH 510/952] Plugin configuration now lives in datasette.yaml/json * Checkpoint, moving top-level plugin config to datasette.json * Support database-level and table-level plugin configuration in datasette.yaml Refs #2093 --- datasette/app.py | 48 +++++++++++++++----- docs/configuration.rst | 97 ++++++++++++++++++++++++++++++++++++++-- docs/index.rst | 1 + docs/internals.rst | 2 +- docs/plugin_hooks.rst | 2 +- docs/writing_plugins.rst | 7 +-- tests/conftest.py | 3 +- tests/fixtures.py | 50 ++++++++++++++------- tests/test_cli.py | 38 ++++++++++++++++ tests/test_plugins.py | 23 +++------- 10 files changed, 217 insertions(+), 54 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index fdec2c86..53486007 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -368,7 +368,7 @@ class Datasette: for key in config_settings: if key not in DEFAULT_SETTINGS: raise StartupError("Invalid setting '{}' in datasette.json".format(key)) - + self.config = config # CLI settings should overwrite datasette.json settings self._settings = dict(DEFAULT_SETTINGS, **(config_settings), **(settings or {})) self.renderers = {} # File extension -> (renderer, can_render) functions @@ -674,15 +674,43 @@ class Datasette: def plugin_config(self, plugin_name, database=None, table=None, fallback=True): """Return config for plugin, falling back from specified database/table""" - plugins = self.metadata( - "plugins", database=database, table=table, fallback=fallback - ) - if plugins is None: - return None - plugin_config = plugins.get(plugin_name) - # Resolve any $file and $env keys - plugin_config = resolve_env_secrets(plugin_config, os.environ) - return plugin_config + if database is None and table is None: + config = self._plugin_config_top(plugin_name) + else: + config = self._plugin_config_nested(plugin_name, database, table, fallback) + + return resolve_env_secrets(config, os.environ) + + def _plugin_config_top(self, plugin_name): + """Returns any top-level plugin configuration for the specified plugin.""" + return ((self.config or {}).get("plugins") or {}).get(plugin_name) + + def _plugin_config_nested(self, plugin_name, database, table=None, fallback=True): + """Returns any database or table-level plugin configuration for the specified plugin.""" + db_config = ((self.config or {}).get("databases") or {}).get(database) + + # if there's no db-level configuration, then return early, falling back to top-level if needed + if not db_config: + return self._plugin_config_top(plugin_name) if fallback else None + + db_plugin_config = (db_config.get("plugins") or {}).get(plugin_name) + + if table: + table_plugin_config = ( + ((db_config.get("tables") or {}).get(table) or {}).get("plugins") or {} + ).get(plugin_name) + + # fallback to db_config or top-level config, in that order, if needed + if table_plugin_config is None and fallback: + return db_plugin_config or self._plugin_config_top(plugin_name) + + return table_plugin_config + + # fallback to top-level if needed + if db_plugin_config is None and fallback: + self._plugin_config_top(plugin_name) + + return db_plugin_config def app_css_hash(self): if not hasattr(self, "_app_css_hash"): diff --git a/docs/configuration.rst b/docs/configuration.rst index ed9975ac..214e9044 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -1,10 +1,101 @@ .. _configuration: Configuration -======== +============= -Datasette offers many way to configure your Datasette instances: server settings, plugin configuration, authentication, and more. +Datasette offers several ways to configure your Datasette instances: server settings, plugin configuration, authentication, and more. -To facilitate this, You can provide a `datasette.yaml` configuration file to datasette with the ``--config``/ ``-c`` flag: +To facilitate this, You can provide a ``datasette.yaml`` configuration file to datasette with the ``--config``/ ``-c`` flag: + +.. code-block:: bash datasette mydatabase.db --config datasette.yaml + +.. _configuration_reference: + +``datasette.yaml`` reference +---------------------------- + +Here's a full example of all the valid configuration options that can exist inside ``datasette.yaml``. + +.. tab:: YAML + + .. code-block:: yaml + + # Datasette settings block + settings: + default_page_size: 50 + sql_time_limit_ms: 3500 + max_returned_rows: 2000 + + # top-level plugin configuration + plugins: + datasette-my-plugin: + key: valueA + + # Database and table-level configuration + databases: + your_db_name: + # plugin configuration for the your_db_name database + plugins: + datasette-my-plugin: + key: valueA + tables: + your_table_name: + # plugin configuration for the your_table_name table + # inside your_db_name database + plugins: + datasette-my-plugin: + key: valueB + +.. _configuration_reference_settings: +Settings configuration +~~~~~~~~~~~~~~~~~~~~~~ + +:ref:`settings` can be configured in ``datasette.yaml`` with the ``settings`` key. + +.. tab:: YAML + + .. code-block:: yaml + + # inside datasette.yaml + settings: + default_allow_sql: off + default_page_size: 50 + + +.. _configuration_reference_plugins: +Plugin configuration +~~~~~~~~~~~~~~~~~~~~ + +Configuration for plugins can be defined inside ``datasette.yaml``. For top-level plugin configuration, use the ``plugins`` key. + +.. tab:: YAML + + .. code-block:: yaml + + # inside datasette.yaml + plugins: + datasette-my-plugin: + key: my_value + +For database level or table level plugin configuration, nest it under the appropriate place under ``databases``. + +.. tab:: YAML + + .. code-block:: yaml + + # inside datasette.yaml + databases: + my_database: + # plugin configuration for the my_database database + plugins: + datasette-my-plugin: + key: my_value + my_other_database: + tables: + my_table: + # plugin configuration for the my_table table inside the my_other_database database + plugins: + datasette-my-plugin: + key: my_value diff --git a/docs/index.rst b/docs/index.rst index f5c1f232..cfa3443c 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -39,6 +39,7 @@ Contents getting_started installation + configuration ecosystem cli-reference pages diff --git a/docs/internals.rst b/docs/internals.rst index 13f1d4a1..7fc7948c 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -296,7 +296,7 @@ The dictionary keys are the permission names - e.g. ``view-instance`` - and the ``table`` - None or string The table the user is interacting with. -This method lets you read plugin configuration values that were set in ``metadata.json``. See :ref:`writing_plugins_configuration` for full details of how this method should be used. +This method lets you read plugin configuration values that were set in ``datasette.yaml``. See :ref:`writing_plugins_configuration` for full details of how this method should be used. The return value will be the value from the configuration file - usually a dictionary. diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index e966919b..1816d48c 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -909,7 +909,7 @@ Potential use-cases: * Run some initialization code for the plugin * Create database tables that a plugin needs on startup -* Validate the metadata configuration for a plugin on startup, and raise an error if it is invalid +* Validate the configuration for a plugin on startup, and raise an error if it is invalid .. note:: diff --git a/docs/writing_plugins.rst b/docs/writing_plugins.rst index d0dd8f36..c028b4ff 100644 --- a/docs/writing_plugins.rst +++ b/docs/writing_plugins.rst @@ -184,7 +184,7 @@ This will return the ``{"latitude_column": "lat", "longitude_column": "lng"}`` i If there is no configuration for that plugin, the method will return ``None``. -If it cannot find the requested configuration at the table layer, it will fall back to the database layer and then the root layer. For example, a user may have set the plugin configuration option like so: +If it cannot find the requested configuration at the table layer, it will fall back to the database layer and then the root layer. For example, a user may have set the plugin configuration option inside ``datasette.yaml`` like so: .. [[[cog from metadata_doc import metadata_example @@ -234,11 +234,10 @@ If it cannot find the requested configuration at the table layer, it will fall b In this case, the above code would return that configuration for ANY table within the ``sf-trees`` database. -The plugin configuration could also be set at the top level of ``metadata.yaml``: +The plugin configuration could also be set at the top level of ``datasette.yaml``: .. [[[cog metadata_example(cog, { - "title": "This is the top-level title in metadata.json", "plugins": { "datasette-cluster-map": { "latitude_column": "xlat", @@ -252,7 +251,6 @@ The plugin configuration could also be set at the top level of ``metadata.yaml`` .. code-block:: yaml - title: This is the top-level title in metadata.json plugins: datasette-cluster-map: latitude_column: xlat @@ -264,7 +262,6 @@ The plugin configuration could also be set at the top level of ``metadata.yaml`` .. code-block:: json { - "title": "This is the top-level title in metadata.json", "plugins": { "datasette-cluster-map": { "latitude_column": "xlat", diff --git a/tests/conftest.py b/tests/conftest.py index fb7f768e..31336aea 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -41,7 +41,7 @@ def wait_until_responds(url, timeout=5.0, client=httpx, **kwargs): @pytest_asyncio.fixture async def ds_client(): from datasette.app import Datasette - from .fixtures import METADATA, PLUGINS_DIR + from .fixtures import CONFIG, METADATA, PLUGINS_DIR global _ds_client if _ds_client is not None: @@ -49,6 +49,7 @@ async def ds_client(): ds = Datasette( metadata=METADATA, + config=CONFIG, plugins_dir=PLUGINS_DIR, settings={ "default_page_size": 50, diff --git a/tests/fixtures.py b/tests/fixtures.py index a6700239..9cf6b605 100644 --- a/tests/fixtures.py +++ b/tests/fixtures.py @@ -114,6 +114,7 @@ def make_app_client( inspect_data=None, static_mounts=None, template_dir=None, + config=None, metadata=None, crossdb=False, ): @@ -158,6 +159,7 @@ def make_app_client( memory=memory, cors=cors, metadata=metadata or METADATA, + config=config or CONFIG, plugins_dir=PLUGINS_DIR, settings=settings, inspect_data=inspect_data, @@ -296,6 +298,33 @@ def generate_sortable_rows(num): } +CONFIG = { + "plugins": { + "name-of-plugin": {"depth": "root"}, + "env-plugin": {"foo": {"$env": "FOO_ENV"}}, + "env-plugin-list": [{"in_a_list": {"$env": "FOO_ENV"}}], + "file-plugin": {"foo": {"$file": TEMP_PLUGIN_SECRET_FILE}}, + }, + "databases": { + "fixtures": { + "plugins": {"name-of-plugin": {"depth": "database"}}, + "tables": { + "simple_primary_key": { + "plugins": { + "name-of-plugin": { + "depth": "table", + "special": "this-is-simple_primary_key", + } + }, + }, + "sortable": { + "plugins": {"name-of-plugin": {"depth": "table"}}, + }, + }, + } + }, +} + METADATA = { "title": "Datasette Fixtures", "description_html": 'An example SQLite database demonstrating Datasette. Sign in as root user', @@ -306,26 +335,13 @@ METADATA = { "about": "About Datasette", "about_url": "https://github.com/simonw/datasette", "extra_css_urls": ["/static/extra-css-urls.css"], - "plugins": { - "name-of-plugin": {"depth": "root"}, - "env-plugin": {"foo": {"$env": "FOO_ENV"}}, - "env-plugin-list": [{"in_a_list": {"$env": "FOO_ENV"}}], - "file-plugin": {"foo": {"$file": TEMP_PLUGIN_SECRET_FILE}}, - }, "databases": { "fixtures": { "description": "Test tables description", - "plugins": {"name-of-plugin": {"depth": "database"}}, "tables": { "simple_primary_key": { "description_html": "Simple primary key", "title": "This HTML is escaped", - "plugins": { - "name-of-plugin": { - "depth": "table", - "special": "this-is-simple_primary_key", - } - }, }, "sortable": { "sortable_columns": [ @@ -334,7 +350,6 @@ METADATA = { "sortable_with_nulls_2", "text", ], - "plugins": {"name-of-plugin": {"depth": "table"}}, }, "no_primary_key": {"sortable_columns": [], "hidden": True}, "units": {"units": {"distance": "m", "frequency": "Hz"}}, @@ -768,6 +783,7 @@ def assert_permissions_checked(datasette, actions): type=click.Path(file_okay=True, dir_okay=False), ) @click.argument("metadata", required=False) +@click.argument("config", required=False) @click.argument( "plugins_path", type=click.Path(file_okay=False, dir_okay=True), required=False ) @@ -782,7 +798,7 @@ def assert_permissions_checked(datasette, actions): type=click.Path(file_okay=True, dir_okay=False), help="Write out second test DB to this file", ) -def cli(db_filename, metadata, plugins_path, recreate, extra_db_filename): +def cli(db_filename, config, metadata, plugins_path, recreate, extra_db_filename): """Write out the fixtures database used by Datasette's test suite""" if metadata and not metadata.endswith(".json"): raise click.ClickException("Metadata should end with .json") @@ -805,6 +821,10 @@ def cli(db_filename, metadata, plugins_path, recreate, extra_db_filename): with open(metadata, "w") as fp: fp.write(json.dumps(METADATA, indent=4)) print(f"- metadata written to {metadata}") + if config: + with open(config, "w") as fp: + fp.write(json.dumps(CONFIG, indent=4)) + print(f"- config written to {config}") if plugins_path: path = pathlib.Path(plugins_path) if not path.exists(): diff --git a/tests/test_cli.py b/tests/test_cli.py index e85bcef1..213db416 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -238,6 +238,44 @@ def test_setting(args): assert settings["default_page_size"] == 5 +def test_plugin_s_overwrite(): + runner = CliRunner() + plugins_dir = str(pathlib.Path(__file__).parent / "plugins") + + result = runner.invoke( + cli, + [ + "--plugins-dir", + plugins_dir, + "--get", + "/_memory.json?sql=select+prepare_connection_args()", + ], + ) + assert result.exit_code == 0, result.output + assert ( + json.loads(result.output).get("rows")[0].get("prepare_connection_args()") + == 'database=_memory, datasette.plugin_config("name-of-plugin")=None' + ) + + result = runner.invoke( + cli, + [ + "--plugins-dir", + plugins_dir, + "--get", + "/_memory.json?sql=select+prepare_connection_args()", + "-s", + "plugins.name-of-plugin", + "OVERRIDE", + ], + ) + assert result.exit_code == 0, result.output + assert ( + json.loads(result.output).get("rows")[0].get("prepare_connection_args()") + == 'database=_memory, datasette.plugin_config("name-of-plugin")=OVERRIDE' + ) + + def test_setting_type_validation(): runner = CliRunner(mix_stderr=False) result = runner.invoke(cli, ["--setting", "default_page_size", "dog"]) diff --git a/tests/test_plugins.py b/tests/test_plugins.py index 625ae635..37530991 100644 --- a/tests/test_plugins.py +++ b/tests/test_plugins.py @@ -234,9 +234,6 @@ async def test_plugin_config(ds_client): async def test_plugin_config_env(ds_client): os.environ["FOO_ENV"] = "FROM_ENVIRONMENT" assert {"foo": "FROM_ENVIRONMENT"} == ds_client.ds.plugin_config("env-plugin") - # Ensure secrets aren't visible in /-/metadata.json - metadata = await ds_client.get("/-/metadata.json") - assert {"foo": {"$env": "FOO_ENV"}} == metadata.json()["plugins"]["env-plugin"] del os.environ["FOO_ENV"] @@ -246,11 +243,6 @@ async def test_plugin_config_env_from_list(ds_client): assert [{"in_a_list": "FROM_ENVIRONMENT"}] == ds_client.ds.plugin_config( "env-plugin-list" ) - # Ensure secrets aren't visible in /-/metadata.json - metadata = await ds_client.get("/-/metadata.json") - assert [{"in_a_list": {"$env": "FOO_ENV"}}] == metadata.json()["plugins"][ - "env-plugin-list" - ] del os.environ["FOO_ENV"] @@ -259,11 +251,6 @@ async def test_plugin_config_file(ds_client): with open(TEMP_PLUGIN_SECRET_FILE, "w") as fp: fp.write("FROM_FILE") assert {"foo": "FROM_FILE"} == ds_client.ds.plugin_config("file-plugin") - # Ensure secrets aren't visible in /-/metadata.json - metadata = await ds_client.get("/-/metadata.json") - assert {"foo": {"$file": TEMP_PLUGIN_SECRET_FILE}} == metadata.json()["plugins"][ - "file-plugin" - ] os.remove(TEMP_PLUGIN_SECRET_FILE) @@ -722,7 +709,7 @@ async def test_hook_register_routes(ds_client, path, body): @pytest.mark.parametrize("configured_path", ("path1", "path2")) def test_hook_register_routes_with_datasette(configured_path): with make_app_client( - metadata={ + config={ "plugins": { "register-route-demo": { "path": configured_path, @@ -741,7 +728,7 @@ def test_hook_register_routes_with_datasette(configured_path): def test_hook_register_routes_override(): "Plugins can over-ride default paths such as /db/table" with make_app_client( - metadata={ + config={ "plugins": { "register-route-demo": { "path": "blah", @@ -1099,7 +1086,7 @@ async def test_hook_filters_from_request(ds_client): @pytest.mark.parametrize("extra_metadata", (False, True)) async def test_hook_register_permissions(extra_metadata): ds = Datasette( - metadata={ + config={ "plugins": { "datasette-register-permissions": { "permissions": [ @@ -1151,7 +1138,7 @@ async def test_hook_register_permissions_no_duplicates(duplicate): if duplicate == "abbr": abbr2 = "abbr1" ds = Datasette( - metadata={ + config={ "plugins": { "datasette-register-permissions": { "permissions": [ @@ -1186,7 +1173,7 @@ async def test_hook_register_permissions_no_duplicates(duplicate): @pytest.mark.asyncio async def test_hook_register_permissions_allows_identical_duplicates(): ds = Datasette( - metadata={ + config={ "plugins": { "datasette-register-permissions": { "permissions": [ From 16f0b6d8222d06682a31b904d0a402c391ae1c1c Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 13 Sep 2023 14:15:32 -0700 Subject: [PATCH 511/952] JSON/YAML tabs on configuration docs page --- docs/configuration.rst | 171 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 171 insertions(+) diff --git a/docs/configuration.rst b/docs/configuration.rst index 214e9044..4a7258b9 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -18,6 +18,40 @@ To facilitate this, You can provide a ``datasette.yaml`` configuration file to d Here's a full example of all the valid configuration options that can exist inside ``datasette.yaml``. +.. [[[cog + from metadata_doc import metadata_example + import textwrap + metadata_example(cog, yaml=textwrap.dedent( + """ + # Datasette settings block + settings: + default_page_size: 50 + sql_time_limit_ms: 3500 + max_returned_rows: 2000 + + # top-level plugin configuration + plugins: + datasette-my-plugin: + key: valueA + + # Database and table-level configuration + databases: + your_db_name: + # plugin configuration for the your_db_name database + plugins: + datasette-my-plugin: + key: valueA + tables: + your_table_name: + # plugin configuration for the your_table_name table + # inside your_db_name database + plugins: + datasette-my-plugin: + key: valueB + """) + ) +.. ]]] + .. tab:: YAML .. code-block:: yaml @@ -48,12 +82,61 @@ Here's a full example of all the valid configuration options that can exist insi datasette-my-plugin: key: valueB +.. tab:: JSON + + .. code-block:: json + + { + "settings": { + "default_page_size": 50, + "sql_time_limit_ms": 3500, + "max_returned_rows": 2000 + }, + "plugins": { + "datasette-my-plugin": { + "key": "valueA" + } + }, + "databases": { + "your_db_name": { + "plugins": { + "datasette-my-plugin": { + "key": "valueA" + } + }, + "tables": { + "your_table_name": { + "plugins": { + "datasette-my-plugin": { + "key": "valueB" + } + } + } + } + } + } + } +.. [[[end]]] + .. _configuration_reference_settings: Settings configuration ~~~~~~~~~~~~~~~~~~~~~~ :ref:`settings` can be configured in ``datasette.yaml`` with the ``settings`` key. +.. [[[cog + from metadata_doc import metadata_example + import textwrap + metadata_example(cog, yaml=textwrap.dedent( + """ + # inside datasette.yaml + settings: + default_allow_sql: off + default_page_size: 50 + """).strip() + ) +.. ]]] + .. tab:: YAML .. code-block:: yaml @@ -63,6 +146,17 @@ Settings configuration default_allow_sql: off default_page_size: 50 +.. tab:: JSON + + .. code-block:: json + + { + "settings": { + "default_allow_sql": "off", + "default_page_size": 50 + } + } +.. [[[end]]] .. _configuration_reference_plugins: Plugin configuration @@ -70,6 +164,19 @@ Plugin configuration Configuration for plugins can be defined inside ``datasette.yaml``. For top-level plugin configuration, use the ``plugins`` key. +.. [[[cog + from metadata_doc import metadata_example + import textwrap + metadata_example(cog, yaml=textwrap.dedent( + """ + # inside datasette.yaml + plugins: + datasette-my-plugin: + key: my_value + """).strip() + ) +.. ]]] + .. tab:: YAML .. code-block:: yaml @@ -79,8 +186,44 @@ Configuration for plugins can be defined inside ``datasette.yaml``. For top-leve datasette-my-plugin: key: my_value +.. tab:: JSON + + .. code-block:: json + + { + "plugins": { + "datasette-my-plugin": { + "key": "my_value" + } + } + } +.. [[[end]]] + For database level or table level plugin configuration, nest it under the appropriate place under ``databases``. +.. [[[cog + from metadata_doc import metadata_example + import textwrap + metadata_example(cog, yaml=textwrap.dedent( + """ + # inside datasette.yaml + databases: + my_database: + # plugin configuration for the my_database database + plugins: + datasette-my-plugin: + key: my_value + my_other_database: + tables: + my_table: + # plugin configuration for the my_table table inside the my_other_database database + plugins: + datasette-my-plugin: + key: my_value + """).strip() + ) +.. ]]] + .. tab:: YAML .. code-block:: yaml @@ -99,3 +242,31 @@ For database level or table level plugin configuration, nest it under the approp plugins: datasette-my-plugin: key: my_value + +.. tab:: JSON + + .. code-block:: json + + { + "databases": { + "my_database": { + "plugins": { + "datasette-my-plugin": { + "key": "my_value" + } + } + }, + "my_other_database": { + "tables": { + "my_table": { + "plugins": { + "datasette-my-plugin": { + "key": "my_value" + } + } + } + } + } + } + } +.. [[[end]]] \ No newline at end of file From 852f5014853943fa27f43ddaa2d442545b3259fb Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sat, 16 Sep 2023 09:35:18 -0700 Subject: [PATCH 512/952] Switch from pkg_resources to importlib.metadata in app.py, refs #2057 --- datasette/app.py | 6 +++--- tests/test_plugins.py | 22 ++++++++++++++++++++++ 2 files changed, 25 insertions(+), 3 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 53486007..c0e80700 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -8,11 +8,11 @@ import functools import glob import hashlib import httpx +import importlib.metadata import inspect from itsdangerous import BadSignature import json import os -import pkg_resources import re import secrets import sys @@ -1118,9 +1118,9 @@ class Datasette: if using_pysqlite3: for package in ("pysqlite3", "pysqlite3-binary"): try: - info["pysqlite3"] = pkg_resources.get_distribution(package).version + info["pysqlite3"] = importlib.metadata.version(package) break - except pkg_resources.DistributionNotFound: + except importlib.metadata.PackageNotFoundError: pass return info diff --git a/tests/test_plugins.py b/tests/test_plugins.py index 37530991..3bc117f3 100644 --- a/tests/test_plugins.py +++ b/tests/test_plugins.py @@ -1264,3 +1264,25 @@ async def test_hook_actors_from_ids(): } finally: pm.unregister(name="ReturnNothingPlugin") + + +@pytest.mark.asyncio +async def test_plugin_is_installed(): + datasette = Datasette(memory=True) + + class DummyPlugin: + __name__ = "DummyPlugin" + + @hookimpl + def actors_from_ids(self, datasette, actor_ids): + return {} + + try: + pm.register(DummyPlugin(), name="DummyPlugin") + response = await datasette.client.get("/-/plugins.json") + assert response.status_code == 200 + installed_plugins = {p["name"] for p in response.json()} + assert "DummyPlugin" in installed_plugins + + finally: + pm.unregister(name="DummyPlugin") From f56e043747bde4faa1d78588636df6c0dadebc65 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 18 Sep 2023 10:39:11 -0700 Subject: [PATCH 513/952] test_facet_against_in_memory_database, refs #2189 This is meant to illustrate a crashing bug but it does not trigger it. --- tests/test_facets.py | 47 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/tests/test_facets.py b/tests/test_facets.py index 48cc0ff2..a68347f0 100644 --- a/tests/test_facets.py +++ b/tests/test_facets.py @@ -643,3 +643,50 @@ async def test_conflicting_facet_names_json(ds_client): "created_2", "tags_2", } + + +@pytest.mark.asyncio +async def test_facet_against_in_memory_database(): + ds = Datasette() + db = ds.add_memory_database("mem") + await db.execute_write("create table t (id integer primary key, name text)") + to_insert = [["one"] for _ in range(800)] + [["two"] for _ in range(300)] + await db.execute_write_many("insert into t (name) values (?)", to_insert) + response1 = await ds.client.get("/mem/t.json") + assert response1.status_code == 200 + response2 = await ds.client.get("/mem/t.json?_facet=name&_size=0") + assert response2.status_code == 200 + assert response2.json() == { + "ok": True, + "next": None, + "facet_results": { + "results": { + "name": { + "name": "name", + "type": "column", + "hideable": True, + "toggle_url": "/mem/t.json?_size=0", + "results": [ + { + "value": "one", + "label": "one", + "count": 800, + "toggle_url": "http://localhost/mem/t.json?_facet=name&_size=0&name=one", + "selected": False, + }, + { + "value": "two", + "label": "two", + "count": 300, + "toggle_url": "http://localhost/mem/t.json?_facet=name&_size=0&name=two", + "selected": False, + }, + ], + "truncated": False, + } + }, + "timed_out": [], + }, + "rows": [], + "truncated": False, + } From 6ed7908580fa2ba9297c3225d85c56f8b08b9937 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 18 Sep 2023 10:44:13 -0700 Subject: [PATCH 514/952] Simplified test for #2189 This now executes two facets, in the hope that parallel facet execution would illustrate the bug - but it did not illustrate the bug. --- tests/test_facets.py | 51 +++++++++++--------------------------------- 1 file changed, 12 insertions(+), 39 deletions(-) diff --git a/tests/test_facets.py b/tests/test_facets.py index a68347f0..85c8f85b 100644 --- a/tests/test_facets.py +++ b/tests/test_facets.py @@ -649,44 +649,17 @@ async def test_conflicting_facet_names_json(ds_client): async def test_facet_against_in_memory_database(): ds = Datasette() db = ds.add_memory_database("mem") - await db.execute_write("create table t (id integer primary key, name text)") - to_insert = [["one"] for _ in range(800)] + [["two"] for _ in range(300)] - await db.execute_write_many("insert into t (name) values (?)", to_insert) - response1 = await ds.client.get("/mem/t.json") + await db.execute_write( + "create table t (id integer primary key, name text, name2 text)" + ) + to_insert = [{"name": "one", "name2": "1"} for _ in range(800)] + [ + {"name": "two", "name2": "2"} for _ in range(300) + ] + print(to_insert) + await db.execute_write_many( + "insert into t (name, name2) values (:name, :name2)", to_insert + ) + response1 = await ds.client.get("/mem/t") assert response1.status_code == 200 - response2 = await ds.client.get("/mem/t.json?_facet=name&_size=0") + response2 = await ds.client.get("/mem/t?_facet=name&_facet=name2") assert response2.status_code == 200 - assert response2.json() == { - "ok": True, - "next": None, - "facet_results": { - "results": { - "name": { - "name": "name", - "type": "column", - "hideable": True, - "toggle_url": "/mem/t.json?_size=0", - "results": [ - { - "value": "one", - "label": "one", - "count": 800, - "toggle_url": "http://localhost/mem/t.json?_facet=name&_size=0&name=one", - "selected": False, - }, - { - "value": "two", - "label": "two", - "count": 300, - "toggle_url": "http://localhost/mem/t.json?_facet=name&_size=0&name=two", - "selected": False, - }, - ], - "truncated": False, - } - }, - "timed_out": [], - }, - "rows": [], - "truncated": False, - } From b0e5d8afa308759f4ee9f3ecdf61101dffc4a037 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 20 Sep 2023 15:10:55 -0700 Subject: [PATCH 515/952] Stop using parallel SQL queries for tables Refs: - #2189 --- datasette/views/table.py | 16 ++++++---------- docs/internals.rst | 1 + 2 files changed, 7 insertions(+), 10 deletions(-) diff --git a/datasette/views/table.py b/datasette/views/table.py index 50ba2b78..4f4baeed 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -74,11 +74,10 @@ class Row: return json.dumps(d, default=repr, indent=2) -async def _gather_parallel(*args): - return await asyncio.gather(*args) - - -async def _gather_sequential(*args): +async def run_sequential(*args): + # This used to be swappable for asyncio.gather() to run things in + # parallel, but this lead to hard-to-debug locking issues with + # in-memory databases: https://github.com/simonw/datasette/issues/2189 results = [] for fn in args: results.append(await fn) @@ -1183,9 +1182,6 @@ async def table_view_data( ) rows = rows[:page_size] - # For performance profiling purposes, ?_noparallel=1 turns off asyncio.gather - gather = _gather_sequential if request.args.get("_noparallel") else _gather_parallel - # Resolve extras extras = _get_extras(request) if any(k for k in request.args.keys() if k == "_facet" or k.startswith("_facet_")): @@ -1249,7 +1245,7 @@ async def table_view_data( if not nofacet: # Run them in parallel facet_awaitables = [facet.facet_results() for facet in facet_instances] - facet_awaitable_results = await gather(*facet_awaitables) + facet_awaitable_results = await run_sequential(*facet_awaitables) for ( instance_facet_results, instance_facets_timed_out, @@ -1282,7 +1278,7 @@ async def table_view_data( ): # Run them in parallel facet_suggest_awaitables = [facet.suggest() for facet in facet_instances] - for suggest_result in await gather(*facet_suggest_awaitables): + for suggest_result in await run_sequential(*facet_suggest_awaitables): suggested_facets.extend(suggest_result) return suggested_facets diff --git a/docs/internals.rst b/docs/internals.rst index 7fc7948c..4e9a6747 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -1317,6 +1317,7 @@ This example uses the :ref:`register_routes() ` plugin h (r"/parallel-queries$", parallel_queries), ] +Note that running parallel SQL queries in this way has `been known to cause problems in the past `__, so treat this example with caution. Adding ``?_trace=1`` will show that the trace covers both of those child tasks. From 6763572948ffd047a89a3bbf7c300e91f51ae98f Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 20 Sep 2023 15:11:24 -0700 Subject: [PATCH 516/952] Bump sphinx, furo, black Bumps the python-packages group with 3 updates: [sphinx](https://github.com/sphinx-doc/sphinx), [furo](https://github.com/pradyunsg/furo) and [black](https://github.com/psf/black). Updates `sphinx` from 7.2.5 to 7.2.6 - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/master/CHANGES.rst) - [Commits](https://github.com/sphinx-doc/sphinx/compare/v7.2.5...v7.2.6) Updates `furo` from 2023.8.19 to 2023.9.10 - [Release notes](https://github.com/pradyunsg/furo/releases) - [Changelog](https://github.com/pradyunsg/furo/blob/main/docs/changelog.md) - [Commits](https://github.com/pradyunsg/furo/compare/2023.08.19...2023.09.10) Updates `black` from 23.7.0 to 23.9.1 - [Release notes](https://github.com/psf/black/releases) - [Changelog](https://github.com/psf/black/blob/main/CHANGES.md) - [Commits](https://github.com/psf/black/compare/23.7.0...23.9.1) --- updated-dependencies: - dependency-name: sphinx dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-packages - dependency-name: furo dependency-type: direct:development update-type: version-update:semver-minor dependency-group: python-packages - dependency-name: black dependency-type: direct:development update-type: version-update:semver-minor dependency-group: python-packages ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- setup.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/setup.py b/setup.py index c718086b..415fd27c 100644 --- a/setup.py +++ b/setup.py @@ -69,8 +69,8 @@ setup( setup_requires=["pytest-runner"], extras_require={ "docs": [ - "Sphinx==7.2.5", - "furo==2023.8.19", + "Sphinx==7.2.6", + "furo==2023.9.10", "sphinx-autobuild", "codespell>=2.2.5", "blacken-docs", @@ -83,7 +83,7 @@ setup( "pytest-xdist>=2.2.1", "pytest-asyncio>=0.17", "beautifulsoup4>=4.8.1", - "black==23.7.0", + "black==23.9.1", "blacken-docs==1.16.0", "pytest-timeout>=1.4.2", "trustme>=0.7", From 10bc80547330e826a749ce710da21ae29f7e6048 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 21 Sep 2023 12:11:35 -0700 Subject: [PATCH 517/952] Finish removing pkg_resources, closes #2057 --- datasette/plugins.py | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/datasette/plugins.py b/datasette/plugins.py index 6ec08a81..017f3b9d 100644 --- a/datasette/plugins.py +++ b/datasette/plugins.py @@ -1,7 +1,7 @@ -import importlib +import importlib.metadata +import importlib.resources import os import pluggy -import pkg_resources import sys from . import hookspecs @@ -35,15 +35,15 @@ if DATASETTE_LOAD_PLUGINS is not None: name for name in DATASETTE_LOAD_PLUGINS.split(",") if name.strip() ]: try: - distribution = pkg_resources.get_distribution(package_name) - entry_map = distribution.get_entry_map() - if "datasette" in entry_map: - for plugin_name, entry_point in entry_map["datasette"].items(): + distribution = importlib.metadata.distribution(package_name) + entry_points = distribution.entry_points + for entry_point in entry_points: + if entry_point.group == "datasette": mod = entry_point.load() pm.register(mod, name=entry_point.name) # Ensure name can be found in plugin_to_distinfo later: pm._plugin_distinfo.append((mod, distribution)) - except pkg_resources.DistributionNotFound: + except importlib.metadata.PackageNotFoundError: sys.stderr.write("Plugin {} could not be found\n".format(package_name)) @@ -61,16 +61,16 @@ def get_plugins(): templates_path = None if plugin.__name__ not in DEFAULT_PLUGINS: try: - if pkg_resources.resource_isdir(plugin.__name__, "static"): - static_path = pkg_resources.resource_filename( - plugin.__name__, "static" + if (importlib.resources.files(plugin.__name__) / "static").is_dir(): + static_path = str( + importlib.resources.files(plugin.__name__) / "static" ) - if pkg_resources.resource_isdir(plugin.__name__, "templates"): - templates_path = pkg_resources.resource_filename( - plugin.__name__, "templates" + if (importlib.resources.files(plugin.__name__) / "templates").is_dir(): + templates_path = str( + importlib.resources.files(plugin.__name__) / "templates" ) - except (KeyError, ImportError): - # Caused by --plugins_dir= plugins - KeyError/ImportError thrown in Py3.5 + except (TypeError, ModuleNotFoundError): + # Caused by --plugins_dir= plugins pass plugin_info = { "name": plugin.__name__, @@ -81,6 +81,6 @@ def get_plugins(): distinfo = plugin_to_distinfo.get(plugin) if distinfo: plugin_info["version"] = distinfo.version - plugin_info["name"] = distinfo.project_name + plugin_info["name"] = distinfo.name plugins.append(plugin_info) return plugins From 947520c1fe940de79f5db856dd693330f1bbf547 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 21 Sep 2023 12:31:32 -0700 Subject: [PATCH 518/952] Release notes for 0.64.4 on main --- docs/changelog.rst | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/changelog.rst b/docs/changelog.rst index 81554f83..52e1db3b 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -4,6 +4,13 @@ Changelog ========= +.. _v0_64_4: + +0.64.4 (2023-09-21) +------------------- + +- Fix for a crashing bug caused by viewing the table page for a named in-memory database. (:issue:`2189`) + .. _v1_0_a6: 1.0a6 (2023-09-07) From b0d0a0e5de8bb5b9b6c253e8af451a532266bcf1 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 21 Sep 2023 12:42:15 -0700 Subject: [PATCH 519/952] importlib_resources for Python < 3.9, refs #2057 --- datasette/plugins.py | 15 ++++++++++----- setup.py | 1 + 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/datasette/plugins.py b/datasette/plugins.py index 017f3b9d..a93145cf 100644 --- a/datasette/plugins.py +++ b/datasette/plugins.py @@ -1,10 +1,15 @@ import importlib.metadata -import importlib.resources import os import pluggy import sys from . import hookspecs +if sys.version_info >= (3, 9): + import importlib.resources as importlib_resources +else: + import importlib_resources + + DEFAULT_PLUGINS = ( "datasette.publish.heroku", "datasette.publish.cloudrun", @@ -61,13 +66,13 @@ def get_plugins(): templates_path = None if plugin.__name__ not in DEFAULT_PLUGINS: try: - if (importlib.resources.files(plugin.__name__) / "static").is_dir(): + if (importlib_resources.files(plugin.__name__) / "static").is_dir(): static_path = str( - importlib.resources.files(plugin.__name__) / "static" + importlib_resources.files(plugin.__name__) / "static" ) - if (importlib.resources.files(plugin.__name__) / "templates").is_dir(): + if (importlib_resources.files(plugin.__name__) / "templates").is_dir(): templates_path = str( - importlib.resources.files(plugin.__name__) / "templates" + importlib_resources.files(plugin.__name__) / "templates" ) except (TypeError, ModuleNotFoundError): # Caused by --plugins_dir= plugins diff --git a/setup.py b/setup.py index 415fd27c..a2728f6b 100644 --- a/setup.py +++ b/setup.py @@ -48,6 +48,7 @@ setup( "Jinja2>=2.10.3", "hupper>=1.9", "httpx>=0.20", + 'importlib_resources>=1.3.1; python_version < "3.9"', "pint>=0.9", "pluggy>=1.0", "uvicorn>=0.11", From 80a9cd9620fddf2695d12d8386a91e7c6b145ef2 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 21 Sep 2023 12:55:50 -0700 Subject: [PATCH 520/952] test-datasette-load-plugins now fails correctly, refs #2193 --- tests/test-datasette-load-plugins.sh | 41 +++++++++++++--------------- 1 file changed, 19 insertions(+), 22 deletions(-) diff --git a/tests/test-datasette-load-plugins.sh b/tests/test-datasette-load-plugins.sh index e26d8377..03e08bb1 100755 --- a/tests/test-datasette-load-plugins.sh +++ b/tests/test-datasette-load-plugins.sh @@ -3,27 +3,24 @@ # datasette-init and datasette-json-html are installed PLUGINS=$(datasette plugins) -echo "$PLUGINS" | jq 'any(.[]; .name == "datasette-json-html")' | \ - grep -q true || ( \ - echo "Test failed: datasette-json-html not found" && \ - exit 1 \ - ) -# With the DATASETTE_LOAD_PLUGINS we should not see that +if ! echo "$PLUGINS" | jq 'any(.[]; .name == "datasette-json-html")' | grep -q true; then + echo "Test failed: datasette-json-html not found" + exit 1 +fi + PLUGINS2=$(DATASETTE_LOAD_PLUGINS=datasette-init datasette plugins) -echo "$PLUGINS2" | jq 'any(.[]; .name == "datasette-json-html")' | \ - grep -q false || ( \ - echo "Test failed: datasette-json-html should not have been loaded" && \ - exit 1 \ - ) -echo "$PLUGINS2" | jq 'any(.[]; .name == "datasette-init")' | \ - grep -q true || ( \ - echo "Test failed: datasette-init should have been loaded" && \ - exit 1 \ - ) -# With DATASETTE_LOAD_PLUGINS='' we should see no plugins +if ! echo "$PLUGINS2" | jq 'any(.[]; .name == "datasette-json-html")' | grep -q false; then + echo "Test failed: datasette-json-html should not have been loaded" + exit 1 +fi + +if ! echo "$PLUGINS2" | jq 'any(.[]; .name == "datasette-init")' | grep -q true; then + echo "Test failed: datasette-init should have been loaded" + exit 1 +fi + PLUGINS3=$(DATASETTE_LOAD_PLUGINS='' datasette plugins) -echo "$PLUGINS3"| \ - grep -q '\[\]' || ( \ - echo "Test failed: datasette plugins should have returned []" && \ - exit 1 \ - ) +if ! echo "$PLUGINS3" | grep -q '\[\]'; then + echo "Test failed: datasette plugins should have returned []" + exit 1 +fi From b7cf0200e21796a6ff653c6f94a4ee5fcfde0346 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 21 Sep 2023 13:22:40 -0700 Subject: [PATCH 521/952] Swap order of config and metadata options, refs #2194 --- tests/fixtures.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/fixtures.py b/tests/fixtures.py index 9cf6b605..16aa234e 100644 --- a/tests/fixtures.py +++ b/tests/fixtures.py @@ -782,8 +782,8 @@ def assert_permissions_checked(datasette, actions): default="fixtures.db", type=click.Path(file_okay=True, dir_okay=False), ) -@click.argument("metadata", required=False) @click.argument("config", required=False) +@click.argument("metadata", required=False) @click.argument( "plugins_path", type=click.Path(file_okay=False, dir_okay=True), required=False ) From 2da1a6acec915b81a16127008fd739c7d6075681 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 21 Sep 2023 13:26:13 -0700 Subject: [PATCH 522/952] Use importlib_metadata for Python 3.8, refs #2057 --- datasette/plugins.py | 10 ++++++---- setup.py | 1 + 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/datasette/plugins.py b/datasette/plugins.py index a93145cf..f23f5cfb 100644 --- a/datasette/plugins.py +++ b/datasette/plugins.py @@ -1,4 +1,4 @@ -import importlib.metadata +import importlib import os import pluggy import sys @@ -6,8 +6,10 @@ from . import hookspecs if sys.version_info >= (3, 9): import importlib.resources as importlib_resources + import importlib.metadata as importlib_metadata else: import importlib_resources + import importlib_metadata DEFAULT_PLUGINS = ( @@ -40,7 +42,7 @@ if DATASETTE_LOAD_PLUGINS is not None: name for name in DATASETTE_LOAD_PLUGINS.split(",") if name.strip() ]: try: - distribution = importlib.metadata.distribution(package_name) + distribution = importlib_metadata.distribution(package_name) entry_points = distribution.entry_points for entry_point in entry_points: if entry_point.group == "datasette": @@ -48,7 +50,7 @@ if DATASETTE_LOAD_PLUGINS is not None: pm.register(mod, name=entry_point.name) # Ensure name can be found in plugin_to_distinfo later: pm._plugin_distinfo.append((mod, distribution)) - except importlib.metadata.PackageNotFoundError: + except importlib_metadata.PackageNotFoundError: sys.stderr.write("Plugin {} could not be found\n".format(package_name)) @@ -86,6 +88,6 @@ def get_plugins(): distinfo = plugin_to_distinfo.get(plugin) if distinfo: plugin_info["version"] = distinfo.version - plugin_info["name"] = distinfo.name + plugin_info["name"] = distinfo.name or distinfo.project_name plugins.append(plugin_info) return plugins diff --git a/setup.py b/setup.py index a2728f6b..65a3b335 100644 --- a/setup.py +++ b/setup.py @@ -49,6 +49,7 @@ setup( "hupper>=1.9", "httpx>=0.20", 'importlib_resources>=1.3.1; python_version < "3.9"', + 'importlib_metadata>=4.6; python_version < "3.9"', "pint>=0.9", "pluggy>=1.0", "uvicorn>=0.11", From f130c7c0a88e50cea4121ea18d1f6db2431b6fab Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 21 Sep 2023 14:09:57 -0700 Subject: [PATCH 523/952] Deploy with fixtures-metadata.json, refs #2194, #2195 --- .github/workflows/deploy-latest.yml | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/.github/workflows/deploy-latest.yml b/.github/workflows/deploy-latest.yml index 0dfa5a60..e0405440 100644 --- a/.github/workflows/deploy-latest.yml +++ b/.github/workflows/deploy-latest.yml @@ -38,8 +38,14 @@ jobs: run: | pytest -n auto -m "not serial" pytest -m "serial" - - name: Build fixtures.db - run: python tests/fixtures.py fixtures.db fixtures.json plugins --extra-db-filename extra_database.db + - name: Build fixtures.db and other files needed to deploy the demo + run: |- + python tests/fixtures.py \ + fixtures.db \ + fixtures-config.json \ + fixtures-metadata.json \ + plugins \ + --extra-db-filename extra_database.db - name: Build docs.db if: ${{ github.ref == 'refs/heads/main' }} run: |- @@ -88,13 +94,13 @@ jobs: } return queries EOF - - name: Make some modifications to metadata.json - run: | - cat fixtures.json | \ - jq '.databases |= . + {"ephemeral": {"allow": {"id": "*"}}}' | \ - jq '.plugins |= . + {"datasette-ephemeral-tables": {"table_ttl": 900}}' \ - > metadata.json - cat metadata.json + # - name: Make some modifications to metadata.json + # run: | + # cat fixtures.json | \ + # jq '.databases |= . + {"ephemeral": {"allow": {"id": "*"}}}' | \ + # jq '.plugins |= . + {"datasette-ephemeral-tables": {"table_ttl": 900}}' \ + # > metadata.json + # cat metadata.json - name: Set up Cloud Run uses: google-github-actions/setup-gcloud@v0 with: @@ -112,7 +118,7 @@ jobs: # Replace 1.0 with one-dot-zero in SUFFIX export SUFFIX=${SUFFIX//1.0/one-dot-zero} datasette publish cloudrun fixtures.db fixtures2.db extra_database.db \ - -m metadata.json \ + -m fixtures-metadata.json \ --plugins-dir=plugins \ --branch=$GITHUB_SHA \ --version-note=$GITHUB_SHA \ From e4f868801a6633400045f59584cfe650961c3fa6 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 21 Sep 2023 14:58:39 -0700 Subject: [PATCH 524/952] Use importlib_metadata for 3.9 as well, refs #2057 --- datasette/plugins.py | 4 +++- setup.py | 2 +- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/datasette/plugins.py b/datasette/plugins.py index f23f5cfb..1ed3747f 100644 --- a/datasette/plugins.py +++ b/datasette/plugins.py @@ -6,9 +6,11 @@ from . import hookspecs if sys.version_info >= (3, 9): import importlib.resources as importlib_resources - import importlib.metadata as importlib_metadata else: import importlib_resources +if sys.version_info >= (3, 10): + import importlib.metadata as importlib_metadata +else: import importlib_metadata diff --git a/setup.py b/setup.py index 65a3b335..d09a9e3d 100644 --- a/setup.py +++ b/setup.py @@ -49,7 +49,7 @@ setup( "hupper>=1.9", "httpx>=0.20", 'importlib_resources>=1.3.1; python_version < "3.9"', - 'importlib_metadata>=4.6; python_version < "3.9"', + 'importlib_metadata>=4.6; python_version < "3.10"', "pint>=0.9", "pluggy>=1.0", "uvicorn>=0.11", From 836b1587f08800658c63679d850f0149003c5311 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 21 Sep 2023 15:06:19 -0700 Subject: [PATCH 525/952] Release notes for 1.0a7 Refs #2189 --- datasette/version.py | 2 +- docs/changelog.rst | 7 +++++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/datasette/version.py b/datasette/version.py index 4b65999d..55e2cd42 100644 --- a/datasette/version.py +++ b/datasette/version.py @@ -1,2 +1,2 @@ -__version__ = "1.0a6" +__version__ = "1.0a7" __version_info__ = tuple(__version__.split(".")) diff --git a/docs/changelog.rst b/docs/changelog.rst index 52e1db3b..9a5290c0 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -4,6 +4,13 @@ Changelog ========= +.. _v1_0_a7: + +1.0a7 (2023-09-21) +------------------ + +- Fix for a crashing bug caused by viewing the table page for a named in-memory database. (:issue:`2189`) + .. _v0_64_4: 0.64.4 (2023-09-21) From d51e63d3bb3e32f80d1c0f04adff7c1dd5a7b0c0 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 8 Oct 2023 09:03:37 -0700 Subject: [PATCH 526/952] Release notes for 0.64.5, refs #2197 --- docs/changelog.rst | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/changelog.rst b/docs/changelog.rst index 9a5290c0..48bf9ef5 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -4,6 +4,13 @@ Changelog ========= +.. _v0_64_5: + +0.64.5 (2023-10-08) +------------------- + +- Dropped dependency on ``click-default-group-wheel``, which could cause a dependency conflict. (:issue:`2197`) + .. _v1_0_a7: 1.0a7 (2023-09-21) From 85a41987c7753c3af92ba6b8b6007211eb46602f Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 8 Oct 2023 09:07:11 -0700 Subject: [PATCH 527/952] Fixed typo acepts -> accepts --- docs/plugin_hooks.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index 1816d48c..eb6bf4ae 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -488,7 +488,7 @@ This will register ``render_demo`` to be called when paths with the extension `` ``render_demo`` is a Python function. It can be a regular function or an ``async def render_demo()`` awaitable function, depending on if it needs to make any asynchronous calls. -``can_render_demo`` is a Python function (or ``async def`` function) which acepts the same arguments as ``render_demo`` but just returns ``True`` or ``False``. It lets Datasette know if the current SQL query can be represented by the plugin - and hence influnce if a link to this output format is displayed in the user interface. If you omit the ``"can_render"`` key from the dictionary every query will be treated as being supported by the plugin. +``can_render_demo`` is a Python function (or ``async def`` function) which accepts the same arguments as ``render_demo`` but just returns ``True`` or ``False``. It lets Datasette know if the current SQL query can be represented by the plugin - and hence influnce if a link to this output format is displayed in the user interface. If you omit the ``"can_render"`` key from the dictionary every query will be treated as being supported by the plugin. When a request is received, the ``"render"`` callback function is called with zero or more of the following arguments. Datasette will inspect your callback function and pass arguments that match its function signature. From 4e1188f60f8b4f90c32a372f3f70a26a3ebb88ef Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 8 Oct 2023 09:09:45 -0700 Subject: [PATCH 528/952] Upgrade spellcheck.yml workflow --- .github/workflows/spellcheck.yml | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/.github/workflows/spellcheck.yml b/.github/workflows/spellcheck.yml index 722e5c68..0ce9e10c 100644 --- a/.github/workflows/spellcheck.yml +++ b/.github/workflows/spellcheck.yml @@ -9,18 +9,13 @@ jobs: spellcheck: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 - - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v2 + - uses: actions/checkout@v4 + - name: Set up Python + uses: actions/setup-python@v4 with: - python-version: 3.11 - - uses: actions/cache@v2 - name: Configure pip caching - with: - path: ~/.cache/pip - key: ${{ runner.os }}-pip-${{ hashFiles('**/setup.py') }} - restore-keys: | - ${{ runner.os }}-pip- + python-version: '3.11' + cache: 'pip' + cache-dependency-path: '**/setup.py' - name: Install dependencies run: | pip install -e '.[docs]' From 35deaabcb105903790d18710a26e77545f6852ce Mon Sep 17 00:00:00 2001 From: Alex Garcia Date: Thu, 12 Oct 2023 09:16:37 -0700 Subject: [PATCH 529/952] Move non-metadata configuration from metadata.yaml to datasette.yaml * Allow and permission blocks moved to datasette.yaml * Documentation updates, initial framework for configuration reference --- datasette/app.py | 6 +- datasette/default_permissions.py | 37 ++-- docs/authentication.rst | 338 ++++++++++++++---------------- docs/configuration.rst | 64 ++++-- docs/custom_templates.rst | 137 ++++++------ docs/facets.rst | 12 +- docs/full_text_search.rst | 4 +- docs/internals.rst | 2 +- docs/metadata.rst | 126 ++++++++--- docs/metadata_doc.py | 21 +- docs/plugins.rst | 30 +-- docs/settings.rst | 5 +- docs/sql_queries.rst | 56 ++--- docs/writing_plugins.rst | 8 +- tests/fixtures.py | 48 ++--- tests/test_canned_queries.py | 14 +- tests/test_html.py | 44 ++-- tests/test_internals_datasette.py | 6 +- tests/test_permissions.py | 138 ++++++------ tests/test_plugins.py | 4 +- tests/test_table_api.py | 2 +- tests/test_table_html.py | 8 +- 22 files changed, 606 insertions(+), 504 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index c0e80700..7dfc63c6 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -721,7 +721,9 @@ class Datasette: return self._app_css_hash async def get_canned_queries(self, database_name, actor): - queries = self.metadata("queries", database=database_name, fallback=False) or {} + queries = ( + ((self.config or {}).get("databases") or {}).get(database_name) or {} + ).get("queries") or {} for more_queries in pm.hook.canned_queries( datasette=self, database=database_name, @@ -1315,7 +1317,7 @@ class Datasette: ): hook = await await_me_maybe(hook) collected.extend(hook) - collected.extend(self.metadata(key) or []) + collected.extend((self.config or {}).get(key) or []) output = [] for url_or_dict in collected: if isinstance(url_or_dict, dict): diff --git a/datasette/default_permissions.py b/datasette/default_permissions.py index 5a99d0d8..d29dbe84 100644 --- a/datasette/default_permissions.py +++ b/datasette/default_permissions.py @@ -144,14 +144,14 @@ def permission_allowed_default(datasette, actor, action, resource): "view-query", "execute-sql", ): - result = await _resolve_metadata_view_permissions( + result = await _resolve_config_view_permissions( datasette, actor, action, resource ) if result is not None: return result # Check custom permissions: blocks - result = await _resolve_metadata_permissions_blocks( + result = await _resolve_config_permissions_blocks( datasette, actor, action, resource ) if result is not None: @@ -164,10 +164,10 @@ def permission_allowed_default(datasette, actor, action, resource): return inner -async def _resolve_metadata_permissions_blocks(datasette, actor, action, resource): +async def _resolve_config_permissions_blocks(datasette, actor, action, resource): # Check custom permissions: blocks - metadata = datasette.metadata() - root_block = (metadata.get("permissions", None) or {}).get(action) + config = datasette.config or {} + root_block = (config.get("permissions", None) or {}).get(action) if root_block: root_result = actor_matches_allow(actor, root_block) if root_result is not None: @@ -180,7 +180,7 @@ async def _resolve_metadata_permissions_blocks(datasette, actor, action, resourc else: database = resource[0] database_block = ( - (metadata.get("databases", {}).get(database, {}).get("permissions", None)) or {} + (config.get("databases", {}).get(database, {}).get("permissions", None)) or {} ).get(action) if database_block: database_result = actor_matches_allow(actor, database_block) @@ -192,7 +192,7 @@ async def _resolve_metadata_permissions_blocks(datasette, actor, action, resourc database, table_or_query = resource table_block = ( ( - metadata.get("databases", {}) + config.get("databases", {}) .get(database, {}) .get("tables", {}) .get(table_or_query, {}) @@ -207,7 +207,7 @@ async def _resolve_metadata_permissions_blocks(datasette, actor, action, resourc # Finally the canned queries query_block = ( ( - metadata.get("databases", {}) + config.get("databases", {}) .get(database, {}) .get("queries", {}) .get(table_or_query, {}) @@ -222,25 +222,30 @@ async def _resolve_metadata_permissions_blocks(datasette, actor, action, resourc return None -async def _resolve_metadata_view_permissions(datasette, actor, action, resource): +async def _resolve_config_view_permissions(datasette, actor, action, resource): + config = datasette.config or {} if action == "view-instance": - allow = datasette.metadata("allow") + allow = config.get("allow") if allow is not None: return actor_matches_allow(actor, allow) elif action == "view-database": - database_allow = datasette.metadata("allow", database=resource) + database_allow = ((config.get("databases") or {}).get(resource) or {}).get( + "allow" + ) if database_allow is None: return None return actor_matches_allow(actor, database_allow) elif action == "view-table": database, table = resource - tables = datasette.metadata("tables", database=database) or {} + tables = ((config.get("databases") or {}).get(database) or {}).get( + "tables" + ) or {} table_allow = (tables.get(table) or {}).get("allow") if table_allow is None: return None return actor_matches_allow(actor, table_allow) elif action == "view-query": - # Check if this query has a "allow" block in metadata + # Check if this query has a "allow" block in config database, query_name = resource query = await datasette.get_canned_query(database, query_name, actor) assert query is not None @@ -250,9 +255,11 @@ async def _resolve_metadata_view_permissions(datasette, actor, action, resource) return actor_matches_allow(actor, allow) elif action == "execute-sql": # Use allow_sql block from database block, or from top-level - database_allow_sql = datasette.metadata("allow_sql", database=resource) + database_allow_sql = ((config.get("databases") or {}).get(resource) or {}).get( + "allow_sql" + ) if database_allow_sql is None: - database_allow_sql = datasette.metadata("allow_sql") + database_allow_sql = config.get("allow_sql") if database_allow_sql is None: return None return actor_matches_allow(actor, database_allow_sql) diff --git a/docs/authentication.rst b/docs/authentication.rst index 1a444d0c..a301113a 100644 --- a/docs/authentication.rst +++ b/docs/authentication.rst @@ -67,7 +67,7 @@ An **action** is a string describing the action the actor would like to perform. A **resource** is the item the actor wishes to interact with - for example a specific database or table. Some actions, such as ``permissions-debug``, are not associated with a particular resource. -Datasette's built-in view permissions (``view-database``, ``view-table`` etc) default to *allow* - unless you :ref:`configure additional permission rules ` unauthenticated users will be allowed to access content. +Datasette's built-in view permissions (``view-database``, ``view-table`` etc) default to *allow* - unless you :ref:`configure additional permission rules ` unauthenticated users will be allowed to access content. Permissions with potentially harmful effects should default to *deny*. Plugin authors should account for this when designing new plugins - for example, the `datasette-upload-csvs `__ plugin defaults to deny so that installations don't accidentally allow unauthenticated users to create new tables by uploading a CSV file. @@ -76,7 +76,7 @@ Permissions with potentially harmful effects should default to *deny*. Plugin au Defining permissions with "allow" blocks ---------------------------------------- -The standard way to define permissions in Datasette is to use an ``"allow"`` block. This is a JSON document describing which actors are allowed to perform a permission. +The standard way to define permissions in Datasette is to use an ``"allow"`` block :ref:`in the datasette.yaml file `. This is a JSON document describing which actors are allowed to perform a permission. The most basic form of allow block is this (`allow demo `__, `deny demo `__): @@ -186,18 +186,18 @@ The /-/allow-debug tool The ``/-/allow-debug`` tool lets you try out different ``"action"`` blocks against different ``"actor"`` JSON objects. You can try that out here: https://latest.datasette.io/-/allow-debug -.. _authentication_permissions_metadata: +.. _authentication_permissions_config: -Access permissions in metadata -============================== +Access permissions in ``datasette.yaml`` +======================================== -There are two ways to configure permissions using ``metadata.json`` (or ``metadata.yaml``). +There are two ways to configure permissions using ``datasette.yaml`` (or ``datasette.json``). For simple visibility permissions you can use ``"allow"`` blocks in the root, database, table and query sections. For other permissions you can use a ``"permissions"`` block, described :ref:`in the next section `. -You can limit who is allowed to view different parts of your Datasette instance using ``"allow"`` keys in your :ref:`metadata` configuration. +You can limit who is allowed to view different parts of your Datasette instance using ``"allow"`` keys in your :ref:`configuration`. You can control the following: @@ -216,25 +216,25 @@ Access to an instance Here's how to restrict access to your entire Datasette instance to just the ``"id": "root"`` user: .. [[[cog - from metadata_doc import metadata_example - metadata_example(cog, { - "title": "My private Datasette instance", - "allow": { - "id": "root" - } - }) -.. ]]] - -.. tab:: YAML - - .. code-block:: yaml - + from metadata_doc import config_example + config_example(cog, """ title: My private Datasette instance allow: id: root + """) +.. ]]] + +.. tab:: datasette.yaml + + .. code-block:: yaml -.. tab:: JSON + title: My private Datasette instance + allow: + id: root + + +.. tab:: datasette.json .. code-block:: json @@ -249,21 +249,22 @@ Here's how to restrict access to your entire Datasette instance to just the ``"i To deny access to all users, you can use ``"allow": false``: .. [[[cog - metadata_example(cog, { - "title": "My entirely inaccessible instance", - "allow": False - }) + config_example(cog, """ + title: My entirely inaccessible instance + allow: false + """) .. ]]] -.. tab:: YAML +.. tab:: datasette.yaml .. code-block:: yaml - title: My entirely inaccessible instance - allow: false + + title: My entirely inaccessible instance + allow: false -.. tab:: JSON +.. tab:: datasette.json .. code-block:: json @@ -283,28 +284,26 @@ Access to specific databases To limit access to a specific ``private.db`` database to just authenticated users, use the ``"allow"`` block like this: .. [[[cog - metadata_example(cog, { - "databases": { - "private": { - "allow": { - "id": "*" - } - } - } - }) -.. ]]] - -.. tab:: YAML - - .. code-block:: yaml - + config_example(cog, """ databases: private: allow: - id: '*' + id: "*" + """) +.. ]]] + +.. tab:: datasette.yaml + + .. code-block:: yaml -.. tab:: JSON + databases: + private: + allow: + id: "*" + + +.. tab:: datasette.json .. code-block:: json @@ -327,34 +326,30 @@ Access to specific tables and views To limit access to the ``users`` table in your ``bakery.db`` database: .. [[[cog - metadata_example(cog, { - "databases": { - "bakery": { - "tables": { - "users": { - "allow": { - "id": "*" - } - } - } - } - } - }) -.. ]]] - -.. tab:: YAML - - .. code-block:: yaml - + config_example(cog, """ databases: bakery: tables: users: allow: id: '*' + """) +.. ]]] + +.. tab:: datasette.yaml + + .. code-block:: yaml -.. tab:: JSON + databases: + bakery: + tables: + users: + allow: + id: '*' + + +.. tab:: datasette.json .. code-block:: json @@ -385,32 +380,12 @@ This works for SQL views as well - you can list their names in the ``"tables"`` Access to specific canned queries --------------------------------- -:ref:`canned_queries` allow you to configure named SQL queries in your ``metadata.json`` that can be executed by users. These queries can be set up to both read and write to the database, so controlling who can execute them can be important. +:ref:`canned_queries` allow you to configure named SQL queries in your ``datasette.yaml`` that can be executed by users. These queries can be set up to both read and write to the database, so controlling who can execute them can be important. To limit access to the ``add_name`` canned query in your ``dogs.db`` database to just the :ref:`root user`: .. [[[cog - metadata_example(cog, { - "databases": { - "dogs": { - "queries": { - "add_name": { - "sql": "INSERT INTO names (name) VALUES (:name)", - "write": True, - "allow": { - "id": ["root"] - } - } - } - } - } - }) -.. ]]] - -.. tab:: YAML - - .. code-block:: yaml - + config_example(cog, """ databases: dogs: queries: @@ -420,9 +395,26 @@ To limit access to the ``add_name`` canned query in your ``dogs.db`` database to allow: id: - root + """) +.. ]]] + +.. tab:: datasette.yaml + + .. code-block:: yaml -.. tab:: JSON + databases: + dogs: + queries: + add_name: + sql: INSERT INTO names (name) VALUES (:name) + write: true + allow: + id: + - root + + +.. tab:: datasette.json .. code-block:: json @@ -461,19 +453,20 @@ You can alternatively use an ``"allow_sql"`` block to control who is allowed to To prevent any user from executing arbitrary SQL queries, use this: .. [[[cog - metadata_example(cog, { - "allow_sql": False - }) + config_example(cog, """ + allow_sql: false + """) .. ]]] -.. tab:: YAML +.. tab:: datasette.yaml .. code-block:: yaml - allow_sql: false + + allow_sql: false -.. tab:: JSON +.. tab:: datasette.json .. code-block:: json @@ -485,22 +478,22 @@ To prevent any user from executing arbitrary SQL queries, use this: To enable just the :ref:`root user` to execute SQL for all databases in your instance, use the following: .. [[[cog - metadata_example(cog, { - "allow_sql": { - "id": "root" - } - }) + config_example(cog, """ + allow_sql: + id: root + """) .. ]]] -.. tab:: YAML +.. tab:: datasette.yaml .. code-block:: yaml - allow_sql: - id: root + + allow_sql: + id: root -.. tab:: JSON +.. tab:: datasette.json .. code-block:: json @@ -514,28 +507,26 @@ To enable just the :ref:`root user` to execute SQL for all To limit this ability for just one specific database, use this: .. [[[cog - metadata_example(cog, { - "databases": { - "mydatabase": { - "allow_sql": { - "id": "root" - } - } - } - }) -.. ]]] - -.. tab:: YAML - - .. code-block:: yaml - + config_example(cog, """ databases: mydatabase: allow_sql: id: root + """) +.. ]]] + +.. tab:: datasette.yaml + + .. code-block:: yaml -.. tab:: JSON + databases: + mydatabase: + allow_sql: + id: root + + +.. tab:: datasette.json .. code-block:: json @@ -552,33 +543,32 @@ To limit this ability for just one specific database, use this: .. _authentication_permissions_other: -Other permissions in metadata -============================= +Other permissions in ``datasette.yaml`` +======================================= -For all other permissions, you can use one or more ``"permissions"`` blocks in your metadata. +For all other permissions, you can use one or more ``"permissions"`` blocks in your ``datasette.yaml`` configuration file. -To grant access to the :ref:`permissions debug tool ` to all signed in users you can grant ``permissions-debug`` to any actor with an ``id`` matching the wildcard ``*`` by adding this a the root of your metadata: +To grant access to the :ref:`permissions debug tool ` to all signed in users, you can grant ``permissions-debug`` to any actor with an ``id`` matching the wildcard ``*`` by adding this a the root of your configuration: .. [[[cog - metadata_example(cog, { - "permissions": { - "debug-menu": { - "id": "*" - } - } - }) -.. ]]] - -.. tab:: YAML - - .. code-block:: yaml - + config_example(cog, """ permissions: debug-menu: id: '*' + """) +.. ]]] + +.. tab:: datasette.yaml + + .. code-block:: yaml -.. tab:: JSON + permissions: + debug-menu: + id: '*' + + +.. tab:: datasette.json .. code-block:: json @@ -594,31 +584,28 @@ To grant access to the :ref:`permissions debug tool ` to a To grant ``create-table`` to the user with ``id`` of ``editor`` for the ``docs`` database: .. [[[cog - metadata_example(cog, { - "databases": { - "docs": { - "permissions": { - "create-table": { - "id": "editor" - } - } - } - } - }) -.. ]]] - -.. tab:: YAML - - .. code-block:: yaml - + config_example(cog, """ databases: docs: permissions: create-table: id: editor + """) +.. ]]] + +.. tab:: datasette.yaml + + .. code-block:: yaml -.. tab:: JSON + databases: + docs: + permissions: + create-table: + id: editor + + +.. tab:: datasette.json .. code-block:: json @@ -638,27 +625,7 @@ To grant ``create-table`` to the user with ``id`` of ``editor`` for the ``docs`` And for ``insert-row`` against the ``reports`` table in that ``docs`` database: .. [[[cog - metadata_example(cog, { - "databases": { - "docs": { - "tables": { - "reports": { - "permissions": { - "insert-row": { - "id": "editor" - } - } - } - } - } - } - }) -.. ]]] - -.. tab:: YAML - - .. code-block:: yaml - + config_example(cog, """ databases: docs: tables: @@ -666,9 +633,24 @@ And for ``insert-row`` against the ``reports`` table in that ``docs`` database: permissions: insert-row: id: editor + """) +.. ]]] + +.. tab:: datasette.yaml + + .. code-block:: yaml -.. tab:: JSON + databases: + docs: + tables: + reports: + permissions: + insert-row: + id: editor + + +.. tab:: datasette.json .. code-block:: json diff --git a/docs/configuration.rst b/docs/configuration.rst index 4a7258b9..4e108602 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -13,15 +13,15 @@ To facilitate this, You can provide a ``datasette.yaml`` configuration file to d .. _configuration_reference: -``datasette.yaml`` reference +``datasette.yaml`` Reference ---------------------------- Here's a full example of all the valid configuration options that can exist inside ``datasette.yaml``. .. [[[cog - from metadata_doc import metadata_example + from metadata_doc import config_example import textwrap - metadata_example(cog, yaml=textwrap.dedent( + config_example(cog, textwrap.dedent( """ # Datasette settings block settings: @@ -52,10 +52,11 @@ Here's a full example of all the valid configuration options that can exist insi ) .. ]]] -.. tab:: YAML +.. tab:: datasette.yaml .. code-block:: yaml + # Datasette settings block settings: default_page_size: 50 @@ -82,7 +83,8 @@ Here's a full example of all the valid configuration options that can exist insi datasette-my-plugin: key: valueB -.. tab:: JSON + +.. tab:: datasette.json .. code-block:: json @@ -125,9 +127,9 @@ Settings configuration :ref:`settings` can be configured in ``datasette.yaml`` with the ``settings`` key. .. [[[cog - from metadata_doc import metadata_example + from metadata_doc import config_example import textwrap - metadata_example(cog, yaml=textwrap.dedent( + config_example(cog, textwrap.dedent( """ # inside datasette.yaml settings: @@ -137,7 +139,7 @@ Settings configuration ) .. ]]] -.. tab:: YAML +.. tab:: datasette.yaml .. code-block:: yaml @@ -146,7 +148,7 @@ Settings configuration default_allow_sql: off default_page_size: 50 -.. tab:: JSON +.. tab:: datasette.json .. code-block:: json @@ -165,9 +167,9 @@ Plugin configuration Configuration for plugins can be defined inside ``datasette.yaml``. For top-level plugin configuration, use the ``plugins`` key. .. [[[cog - from metadata_doc import metadata_example + from metadata_doc import config_example import textwrap - metadata_example(cog, yaml=textwrap.dedent( + config_example(cog, textwrap.dedent( """ # inside datasette.yaml plugins: @@ -177,7 +179,7 @@ Configuration for plugins can be defined inside ``datasette.yaml``. For top-leve ) .. ]]] -.. tab:: YAML +.. tab:: datasette.yaml .. code-block:: yaml @@ -186,7 +188,7 @@ Configuration for plugins can be defined inside ``datasette.yaml``. For top-leve datasette-my-plugin: key: my_value -.. tab:: JSON +.. tab:: datasette.json .. code-block:: json @@ -202,9 +204,9 @@ Configuration for plugins can be defined inside ``datasette.yaml``. For top-leve For database level or table level plugin configuration, nest it under the appropriate place under ``databases``. .. [[[cog - from metadata_doc import metadata_example + from metadata_doc import config_example import textwrap - metadata_example(cog, yaml=textwrap.dedent( + config_example(cog, textwrap.dedent( """ # inside datasette.yaml databases: @@ -224,7 +226,7 @@ For database level or table level plugin configuration, nest it under the approp ) .. ]]] -.. tab:: YAML +.. tab:: datasette.yaml .. code-block:: yaml @@ -243,7 +245,7 @@ For database level or table level plugin configuration, nest it under the approp datasette-my-plugin: key: my_value -.. tab:: JSON +.. tab:: datasette.json .. code-block:: json @@ -269,4 +271,30 @@ For database level or table level plugin configuration, nest it under the approp } } } -.. [[[end]]] \ No newline at end of file +.. [[[end]]] + + +.. _configuration_reference_permissions: +Permissions Configuration +~~~~~~~~~~~~~~~~~~~~ + +TODO + + +.. _configuration_reference_authentication: +Authentication Configuration +~~~~~~~~~~~~~~~~~~~~ + +TODO + +.. _configuration_reference_canned_queries: +Canned Queries Configuration +~~~~~~~~~~~~~~~~~~~~ + +TODO + +.. _configuration_reference_css_js: +Extra CSS and JS Configuration +~~~~~~~~~~~~~~~~~~~~ + +TODO diff --git a/docs/custom_templates.rst b/docs/custom_templates.rst index c0f64cb5..d8e4ac96 100644 --- a/docs/custom_templates.rst +++ b/docs/custom_templates.rst @@ -10,35 +10,34 @@ Datasette provides a number of ways of customizing the way data is displayed. Custom CSS and JavaScript ------------------------- -When you launch Datasette, you can specify a custom metadata file like this:: +When you launch Datasette, you can specify a custom configuration file like this:: - datasette mydb.db --metadata metadata.yaml + datasette mydb.db --config datasette.yaml -Your ``metadata.yaml`` file can include links that look like this: +Your ``datasette.yaml`` file can include links that look like this: .. [[[cog - from metadata_doc import metadata_example - metadata_example(cog, { - "extra_css_urls": [ - "https://simonwillison.net/static/css/all.bf8cd891642c.css" - ], - "extra_js_urls": [ - "https://code.jquery.com/jquery-3.2.1.slim.min.js" - ] - }) -.. ]]] - -.. tab:: YAML - - .. code-block:: yaml - + from metadata_doc import config_example + config_example(cog, """ extra_css_urls: - https://simonwillison.net/static/css/all.bf8cd891642c.css extra_js_urls: - https://code.jquery.com/jquery-3.2.1.slim.min.js + """) +.. ]]] + +.. tab:: datasette.yaml + + .. code-block:: yaml -.. tab:: JSON + extra_css_urls: + - https://simonwillison.net/static/css/all.bf8cd891642c.css + extra_js_urls: + - https://code.jquery.com/jquery-3.2.1.slim.min.js + + +.. tab:: datasette.json .. code-block:: json @@ -62,35 +61,30 @@ The extra CSS and JavaScript files will be linked in the ```` of every pag You can also specify a SRI (subresource integrity hash) for these assets: .. [[[cog - metadata_example(cog, { - "extra_css_urls": [ - { - "url": "https://simonwillison.net/static/css/all.bf8cd891642c.css", - "sri": "sha384-9qIZekWUyjCyDIf2YK1FRoKiPJq4PHt6tp/ulnuuyRBvazd0hG7pWbE99zvwSznI" - } - ], - "extra_js_urls": [ - { - "url": "https://code.jquery.com/jquery-3.2.1.slim.min.js", - "sri": "sha256-k2WSCIexGzOj3Euiig+TlR8gA0EmPjuc79OEeY5L45g=" - } - ] - }) -.. ]]] - -.. tab:: YAML - - .. code-block:: yaml - + config_example(cog, """ extra_css_urls: - url: https://simonwillison.net/static/css/all.bf8cd891642c.css sri: sha384-9qIZekWUyjCyDIf2YK1FRoKiPJq4PHt6tp/ulnuuyRBvazd0hG7pWbE99zvwSznI extra_js_urls: - url: https://code.jquery.com/jquery-3.2.1.slim.min.js sri: sha256-k2WSCIexGzOj3Euiig+TlR8gA0EmPjuc79OEeY5L45g= + """) +.. ]]] + +.. tab:: datasette.yaml + + .. code-block:: yaml -.. tab:: JSON + extra_css_urls: + - url: https://simonwillison.net/static/css/all.bf8cd891642c.css + sri: sha384-9qIZekWUyjCyDIf2YK1FRoKiPJq4PHt6tp/ulnuuyRBvazd0hG7pWbE99zvwSznI + extra_js_urls: + - url: https://code.jquery.com/jquery-3.2.1.slim.min.js + sri: sha256-k2WSCIexGzOj3Euiig+TlR8gA0EmPjuc79OEeY5L45g= + + +.. tab:: datasette.json .. code-block:: json @@ -115,7 +109,7 @@ This will produce: .. code-block:: html + {% for url in extra_js_urls %} {% endfor %} diff --git a/demos/plugins/example_js_manager_plugins.py b/demos/plugins/example_js_manager_plugins.py new file mode 100644 index 00000000..7db45464 --- /dev/null +++ b/demos/plugins/example_js_manager_plugins.py @@ -0,0 +1,21 @@ +from datasette import hookimpl + +# Test command: +# datasette fixtures.db \ --plugins-dir=demos/plugins/ +# \ --static static:demos/plugins/static + +# Create a set with view names that qualify for this JS, since plugins won't do anything on other pages +# Same pattern as in Nteract data explorer +# https://github.com/hydrosquall/datasette-nteract-data-explorer/blob/main/datasette_nteract_data_explorer/__init__.py#L77 +PERMITTED_VIEWS = {"table", "query", "database"} + + +@hookimpl +def extra_js_urls(view_name): + print(view_name) + if view_name in PERMITTED_VIEWS: + return [ + { + "url": f"/static/table-example-plugins.js", + } + ] diff --git a/demos/plugins/static/table-example-plugins.js b/demos/plugins/static/table-example-plugins.js new file mode 100644 index 00000000..8c19d9a6 --- /dev/null +++ b/demos/plugins/static/table-example-plugins.js @@ -0,0 +1,100 @@ +/** + * Example usage of Datasette JS Manager API + */ + +document.addEventListener("datasette_init", function (evt) { + const { detail: manager } = evt; + // === Demo plugins: remove before merge=== + addPlugins(manager); +}); + +/** + * Examples for to test datasette JS api + */ +const addPlugins = (manager) => { + + manager.registerPlugin("column-name-plugin", { + version: 0.1, + makeColumnActions: (columnMeta) => { + const { column } = columnMeta; + + return [ + { + label: "Copy name to clipboard", + onClick: (evt) => copyToClipboard(column), + }, + { + label: "Log column metadata to console", + onClick: (evt) => console.log(column), + }, + ]; + }, + }); + + manager.registerPlugin("panel-plugin-graphs", { + version: 0.1, + makeAboveTablePanelConfigs: () => { + return [ + { + id: 'first-panel', + label: "First", + render: node => { + const description = document.createElement('p'); + description.innerText = 'Hello world'; + node.appendChild(description); + } + }, + { + id: 'second-panel', + label: "Second", + render: node => { + const iframe = document.createElement('iframe'); + iframe.src = "https://observablehq.com/embed/@d3/sortable-bar-chart?cell=viewof+order&cell=chart"; + iframe.width = 800; + iframe.height = 635; + iframe.frameborder = '0'; + node.appendChild(iframe); + } + }, + ]; + }, + }); + + manager.registerPlugin("panel-plugin-maps", { + version: 0.1, + makeAboveTablePanelConfigs: () => { + return [ + { + // ID only has to be unique within a plugin, manager namespaces for you + id: 'first-map-panel', + label: "Map plugin", + // datasette-vega, leafleft can provide a "render" function + render: node => node.innerHTML = "Here sits a map", + }, + { + id: 'second-panel', + label: "Image plugin", + render: node => { + const img = document.createElement('img'); + img.src = 'https://datasette.io/static/datasette-logo.svg' + node.appendChild(img); + }, + } + ]; + }, + }); + + // Future: dispatch message to some other part of the page with CustomEvent API + // Could use to drive filter/sort query builder actions without page refresh. +} + + + +async function copyToClipboard(str) { + try { + await navigator.clipboard.writeText(str); + } catch (err) { + /** Rejected - text failed to copy to the clipboard. Browsers didn't give permission */ + console.error('Failed to copy: ', err); + } +} From 067cc75dfa01612f9a47815b33804361e18bf5c3 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 12 Dec 2023 09:49:04 -0800 Subject: [PATCH 535/952] Fixed broken example links in row page documentation --- docs/pages.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/pages.rst b/docs/pages.rst index 0ae72351..2ce05428 100644 --- a/docs/pages.rst +++ b/docs/pages.rst @@ -70,10 +70,10 @@ Table cells with extremely long text contents are truncated on the table view ac Rows which are the targets of foreign key references from other tables will show a link to a filtered search for all records that reference that row. Here's an example from the Registers of Members Interests database: -`../people/uk.org.publicwhip%2Fperson%2F10001 `_ +`../people/uk~2Eorg~2Epublicwhip~2Fperson~2F10001 `_ Note that this URL includes the encoded primary key of the record. Here's that same page as JSON: -`../people/uk.org.publicwhip%2Fperson%2F10001.json `_ +`../people/uk~2Eorg~2Epublicwhip~2Fperson~2F10001.json `_ From 89c8ca0f3ff51fcbf5f710c529bc7a3552da0731 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 19 Dec 2023 10:32:55 -0800 Subject: [PATCH 536/952] Fix for round_trip_load() YAML error, refs #2219 --- docs/metadata_doc.py | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/docs/metadata_doc.py b/docs/metadata_doc.py index 3dc5b5f8..a8f13414 100644 --- a/docs/metadata_doc.py +++ b/docs/metadata_doc.py @@ -1,7 +1,7 @@ import json import textwrap from yaml import safe_dump -from ruamel.yaml import round_trip_load +from ruamel.yaml import YAML def metadata_example(cog, data=None, yaml=None): @@ -11,8 +11,7 @@ def metadata_example(cog, data=None, yaml=None): if yaml: # dedent it first yaml = textwrap.dedent(yaml).strip() - # round_trip_load to preserve key order: - data = round_trip_load(yaml) + data = YAML().load(yaml) output_yaml = yaml else: output_yaml = safe_dump(data, sort_keys=False) @@ -27,8 +26,7 @@ def metadata_example(cog, data=None, yaml=None): def config_example(cog, input): if type(input) is str: - # round_trip_load to preserve key order: - data = round_trip_load(input) + data = YAML().load(input) output_yaml = input else: data = input From 4284c74bc133ab494bf4b6dcd4a20b97b05ebb83 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 19 Dec 2023 10:51:03 -0800 Subject: [PATCH 537/952] db.execute_isolated_fn() method (#2220) Closes #2218 --- datasette/database.py | 61 ++++++++++++++++++++++++------ docs/internals.rst | 19 +++++++++- tests/test_internals_database.py | 65 ++++++++++++++++++++++++++++++++ 3 files changed, 133 insertions(+), 12 deletions(-) diff --git a/datasette/database.py b/datasette/database.py index cb01301e..f2c980d7 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -159,6 +159,26 @@ class Database: kwargs["count"] = count return results + async def execute_isolated_fn(self, fn): + # Open a new connection just for the duration of this function + # blocking the write queue to avoid any writes occurring during it + if self.ds.executor is None: + # non-threaded mode + isolated_connection = self.connect(write=True) + try: + result = fn(isolated_connection) + finally: + isolated_connection.close() + try: + self._all_file_connections.remove(isolated_connection) + except ValueError: + # Was probably a memory connection + pass + return result + else: + # Threaded mode - send to write thread + return await self._send_to_write_thread(fn, isolated_connection=True) + async def execute_write_fn(self, fn, block=True): if self.ds.executor is None: # non-threaded mode @@ -166,9 +186,10 @@ class Database: self._write_connection = self.connect(write=True) self.ds._prepare_connection(self._write_connection, self.name) return fn(self._write_connection) + else: + return await self._send_to_write_thread(fn, block) - # threaded mode - task_id = uuid.uuid5(uuid.NAMESPACE_DNS, "datasette.io") + async def _send_to_write_thread(self, fn, block=True, isolated_connection=False): if self._write_queue is None: self._write_queue = queue.Queue() if self._write_thread is None: @@ -176,8 +197,9 @@ class Database: target=self._execute_writes, daemon=True ) self._write_thread.start() + task_id = uuid.uuid5(uuid.NAMESPACE_DNS, "datasette.io") reply_queue = janus.Queue() - self._write_queue.put(WriteTask(fn, task_id, reply_queue)) + self._write_queue.put(WriteTask(fn, task_id, reply_queue, isolated_connection)) if block: result = await reply_queue.async_q.get() if isinstance(result, Exception): @@ -202,12 +224,28 @@ class Database: if conn_exception is not None: result = conn_exception else: - try: - result = task.fn(conn) - except Exception as e: - sys.stderr.write("{}\n".format(e)) - sys.stderr.flush() - result = e + if task.isolated_connection: + isolated_connection = self.connect(write=True) + try: + result = task.fn(isolated_connection) + except Exception as e: + sys.stderr.write("{}\n".format(e)) + sys.stderr.flush() + result = e + finally: + isolated_connection.close() + try: + self._all_file_connections.remove(isolated_connection) + except ValueError: + # Was probably a memory connection + pass + else: + try: + result = task.fn(conn) + except Exception as e: + sys.stderr.write("{}\n".format(e)) + sys.stderr.flush() + result = e task.reply_queue.sync_q.put(result) async def execute_fn(self, fn): @@ -515,12 +553,13 @@ class Database: class WriteTask: - __slots__ = ("fn", "task_id", "reply_queue") + __slots__ = ("fn", "task_id", "reply_queue", "isolated_connection") - def __init__(self, fn, task_id, reply_queue): + def __init__(self, fn, task_id, reply_queue, isolated_connection): self.fn = fn self.task_id = task_id self.reply_queue = reply_queue + self.isolated_connection = isolated_connection class QueryInterrupted(Exception): diff --git a/docs/internals.rst b/docs/internals.rst index 649ca35d..d269bc7d 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -1017,7 +1017,7 @@ Like ``execute_write()`` but uses the ``sqlite3`` `conn.executemany() ` but executes the provided function in an entirely isolated SQLite connection, which is opened, used and then closed again in a single call to this method. + +The :ref:`prepare_connection() ` plugin hook is not executed against this connection. + +This allows plugins to execute database operations that might conflict with how database connections are usually configured. For example, running a ``VACUUM`` operation while bypassing any restrictions placed by the `datasette-sqlite-authorizer `__ plugin. + +Plugins can also use this method to load potentially dangerous SQLite extensions, use them to perform an operation and then have them safely unloaded at the end of the call, without risk of exposing them to other connections. + +Functions run using ``execute_isolated_fn()`` share the same queue as ``execute_write_fn()``, which guarantees that no writes can be executed at the same time as the isolated function is executing. + +The return value of the function will be returned by this method. Any exceptions raised by the function will be raised out of the ``await`` line as well. + .. _database_close: db.close() diff --git a/tests/test_internals_database.py b/tests/test_internals_database.py index 647ae7bd..e0511100 100644 --- a/tests/test_internals_database.py +++ b/tests/test_internals_database.py @@ -1,6 +1,7 @@ """ Tests for the datasette.database.Database class """ +from datasette.app import Datasette from datasette.database import Database, Results, MultipleValues from datasette.utils.sqlite import sqlite3 from datasette.utils import Column @@ -519,6 +520,70 @@ async def test_execute_write_fn_connection_exception(tmpdir, app_client): app_client.ds.remove_database("immutable-db") +def table_exists(conn, name): + return bool( + conn.execute( + """ + with all_tables as ( + select name from sqlite_master where type = 'table' + union all + select name from temp.sqlite_master where type = 'table' + ) + select 1 from all_tables where name = ? + """, + (name,), + ).fetchall(), + ) + + +def table_exists_checker(name): + def inner(conn): + return table_exists(conn, name) + + return inner + + +@pytest.mark.asyncio +@pytest.mark.parametrize("disable_threads", (False, True)) +async def test_execute_isolated(db, disable_threads): + if disable_threads: + ds = Datasette(memory=True, settings={"num_sql_threads": 0}) + db = ds.add_database(Database(ds, memory_name="test_num_sql_threads_zero")) + + # Create temporary table in write + await db.execute_write( + "create temporary table created_by_write (id integer primary key)" + ) + # Should stay visible to write connection + assert await db.execute_write_fn(table_exists_checker("created_by_write")) + + def create_shared_table(conn): + conn.execute("create table shared (id integer primary key)") + # And a temporary table that should not continue to exist + conn.execute( + "create temporary table created_by_isolated (id integer primary key)" + ) + assert table_exists(conn, "created_by_isolated") + # Also confirm that created_by_write does not exist + return table_exists(conn, "created_by_write") + + # shared should not exist + assert not await db.execute_fn(table_exists_checker("shared")) + + # Create it using isolated + created_by_write_exists = await db.execute_isolated_fn(create_shared_table) + assert not created_by_write_exists + + # shared SHOULD exist now + assert await db.execute_fn(table_exists_checker("shared")) + + # created_by_isolated should not exist, even in write connection + assert not await db.execute_write_fn(table_exists_checker("created_by_isolated")) + + # ... and a second call to isolated should not see that connection either + assert not await db.execute_isolated_fn(table_exists_checker("created_by_isolated")) + + @pytest.mark.asyncio async def test_mtime_ns(db): assert isinstance(db.mtime_ns, int) From 978249beda1a3e7185f61000b0dd57018541c511 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 22 Dec 2023 15:07:42 -0800 Subject: [PATCH 538/952] Removed rogue print("max_csv_mb") Found this while working on #2214 --- datasette/views/base.py | 1 - 1 file changed, 1 deletion(-) diff --git a/datasette/views/base.py b/datasette/views/base.py index 0080b33c..db08557e 100644 --- a/datasette/views/base.py +++ b/datasette/views/base.py @@ -484,7 +484,6 @@ async def stream_csv(datasette, fetch_data, request, database): async def stream_fn(r): nonlocal data, trace - print("max_csv_mb", datasette.setting("max_csv_mb")) limited_writer = LimitedWriter(r, datasette.setting("max_csv_mb")) if trace: await limited_writer.write(preamble) From 872dae1e1a1511e2edfb9d7ddf6ea5096c11d5c3 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 22 Dec 2023 15:08:11 -0800 Subject: [PATCH 539/952] Fix for CSV labels=on missing foreign key bug, closes #2214 --- datasette/views/base.py | 14 ++++++++------ tests/test_csv.py | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+), 6 deletions(-) diff --git a/datasette/views/base.py b/datasette/views/base.py index db08557e..e59fd683 100644 --- a/datasette/views/base.py +++ b/datasette/views/base.py @@ -553,16 +553,18 @@ async def stream_csv(datasette, fetch_data, request, database): if cell is None: new_row.extend(("", "")) else: - assert isinstance(cell, dict) - new_row.append(cell["value"]) - new_row.append(cell["label"]) + if not isinstance(cell, dict): + new_row.extend((cell, "")) + else: + new_row.append(cell["value"]) + new_row.append(cell["label"]) else: new_row.append(cell) await writer.writerow(new_row) - except Exception as e: - sys.stderr.write("Caught this error: {}\n".format(e)) + except Exception as ex: + sys.stderr.write("Caught this error: {}\n".format(ex)) sys.stderr.flush() - await r.write(str(e)) + await r.write(str(ex)) return await limited_writer.write(postamble) diff --git a/tests/test_csv.py b/tests/test_csv.py index ed83d685..9f772f89 100644 --- a/tests/test_csv.py +++ b/tests/test_csv.py @@ -1,3 +1,4 @@ +from datasette.app import Datasette from bs4 import BeautifulSoup as Soup import pytest from .fixtures import ( # noqa @@ -95,6 +96,40 @@ async def test_table_csv_with_nullable_labels(ds_client): assert response.text == EXPECTED_TABLE_WITH_NULLABLE_LABELS_CSV +@pytest.mark.asyncio +async def test_table_csv_with_invalid_labels(): + # https://github.com/simonw/datasette/issues/2214 + ds = Datasette() + await ds.invoke_startup() + db = ds.add_memory_database("db_2214") + await db.execute_write_script( + """ + create table t1 (id integer primary key, name text); + insert into t1 (id, name) values (1, 'one'); + insert into t1 (id, name) values (2, 'two'); + create table t2 (textid text primary key, name text); + insert into t2 (textid, name) values ('a', 'alpha'); + insert into t2 (textid, name) values ('b', 'beta'); + create table if not exists maintable ( + id integer primary key, + fk_integer integer references t1(id), + fk_text text references t2(textid) + ); + insert into maintable (id, fk_integer, fk_text) values (1, 1, 'a'); + insert into maintable (id, fk_integer, fk_text) values (2, 3, 'b'); -- invalid fk_integer + insert into maintable (id, fk_integer, fk_text) values (3, 2, 'c'); -- invalid fk_text + """ + ) + response = await ds.client.get("/db_2214/maintable.csv?_labels=1") + assert response.status_code == 200 + assert response.text == ( + "id,fk_integer,fk_integer_label,fk_text,fk_text_label\r\n" + "1,1,one,a,alpha\r\n" + "2,3,,b,beta\r\n" + "3,2,two,c,\r\n" + ) + + @pytest.mark.asyncio async def test_table_csv_blob_columns(ds_client): response = await ds_client.get("/fixtures/binary_data.csv") From 45b88f2056e0a4da204b50f5e17ba953fcb51865 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 22 Dec 2023 15:14:50 -0800 Subject: [PATCH 540/952] Release notes from 0.64.6, refs #2214 --- docs/changelog.rst | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/changelog.rst b/docs/changelog.rst index f2f17a50..af3d2a0b 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -4,6 +4,13 @@ Changelog ========= +.. _v0_64_6: + +0.64.6 (2023-12-22) +------------------- + +- Fixed a bug where CSV export with expanded labels could fail if a foreign key reference did not correctly resolve. (:issue:`2214`) + .. _v0_64_5: 0.64.5 (2023-10-08) From c7a4706bcc0d6736533b91437e54a8af9226a10a Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 5 Jan 2024 14:33:23 -0800 Subject: [PATCH 541/952] jinja2_environment_from_request() plugin hook Closes #2225 --- datasette/app.py | 49 +++++++++++++++++++++-------------- datasette/handle_exception.py | 3 ++- datasette/hookspecs.py | 5 ++++ datasette/views/base.py | 3 ++- datasette/views/database.py | 6 +++-- datasette/views/table.py | 3 ++- docs/plugin_hooks.rst | 42 ++++++++++++++++++++++++++++++ tests/test_plugins.py | 42 +++++++++++++++++++++++++++++- 8 files changed, 128 insertions(+), 25 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index f33865e4..482cebb4 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -420,21 +420,31 @@ class Datasette: ), ] ) - self.jinja_env = Environment( + environment = Environment( loader=template_loader, autoescape=True, enable_async=True, # undefined=StrictUndefined, ) - self.jinja_env.filters["escape_css_string"] = escape_css_string - self.jinja_env.filters["quote_plus"] = urllib.parse.quote_plus - self.jinja_env.filters["escape_sqlite"] = escape_sqlite - self.jinja_env.filters["to_css_class"] = to_css_class + environment.filters["escape_css_string"] = escape_css_string + environment.filters["quote_plus"] = urllib.parse.quote_plus + self._jinja_env = environment + environment.filters["escape_sqlite"] = escape_sqlite + environment.filters["to_css_class"] = to_css_class self._register_renderers() self._permission_checks = collections.deque(maxlen=200) self._root_token = secrets.token_hex(32) self.client = DatasetteClient(self) + def get_jinja_environment(self, request: Request = None) -> Environment: + environment = self._jinja_env + if request: + for environment in pm.hook.jinja2_environment_from_request( + datasette=self, request=request, env=environment + ): + pass + return environment + def get_permission(self, name_or_abbr: str) -> "Permission": """ Returns a Permission object for the given name or abbreviation. Raises KeyError if not found. @@ -514,7 +524,7 @@ class Datasette: abbrs[p.abbr] = p self.permissions[p.name] = p for hook in pm.hook.prepare_jinja2_environment( - env=self.jinja_env, datasette=self + env=self._jinja_env, datasette=self ): await await_me_maybe(hook) for hook in pm.hook.startup(datasette=self): @@ -1218,7 +1228,7 @@ class Datasette: else: if isinstance(templates, str): templates = [templates] - template = self.jinja_env.select_template(templates) + template = self.get_jinja_environment(request).select_template(templates) if dataclasses.is_dataclass(context): context = dataclasses.asdict(context) body_scripts = [] @@ -1568,16 +1578,6 @@ class DatasetteRouter: def __init__(self, datasette, routes): self.ds = datasette self.routes = routes or [] - # Build a list of pages/blah/{name}.html matching expressions - pattern_templates = [ - filepath - for filepath in self.ds.jinja_env.list_templates() - if "{" in filepath and filepath.startswith("pages/") - ] - self.page_routes = [ - (route_pattern_from_filepath(filepath[len("pages/") :]), filepath) - for filepath in pattern_templates - ] async def __call__(self, scope, receive, send): # Because we care about "foo/bar" v.s. "foo%2Fbar" we decode raw_path ourselves @@ -1677,13 +1677,24 @@ class DatasetteRouter: route_path = request.scope.get("route_path", request.scope["path"]) # Jinja requires template names to use "/" even on Windows template_name = "pages" + route_path + ".html" + # Build a list of pages/blah/{name}.html matching expressions + environment = self.ds.get_jinja_environment(request) + pattern_templates = [ + filepath + for filepath in environment.list_templates() + if "{" in filepath and filepath.startswith("pages/") + ] + page_routes = [ + (route_pattern_from_filepath(filepath[len("pages/") :]), filepath) + for filepath in pattern_templates + ] try: - template = self.ds.jinja_env.select_template([template_name]) + template = environment.select_template([template_name]) except TemplateNotFound: template = None if template is None: # Try for a pages/blah/{name}.html template match - for regex, wildcard_template in self.page_routes: + for regex, wildcard_template in page_routes: match = regex.match(route_path) if match is not None: context.update(match.groupdict()) diff --git a/datasette/handle_exception.py b/datasette/handle_exception.py index 8b7e83e3..bef6b4ee 100644 --- a/datasette/handle_exception.py +++ b/datasette/handle_exception.py @@ -57,7 +57,8 @@ def handle_exception(datasette, request, exception): if request.path.split("?")[0].endswith(".json"): return Response.json(info, status=status, headers=headers) else: - template = datasette.jinja_env.select_template(templates) + environment = datasette.get_jinja_environment(request) + template = environment.select_template(templates) return Response.html( await template.render_async( dict( diff --git a/datasette/hookspecs.py b/datasette/hookspecs.py index 9069927b..b6975dce 100644 --- a/datasette/hookspecs.py +++ b/datasette/hookspecs.py @@ -99,6 +99,11 @@ def actors_from_ids(datasette, actor_ids): """Returns a dictionary mapping those IDs to actor dictionaries""" +@hookspec +def jinja2_environment_from_request(datasette, request, env): + """Return a Jinja2 environment based on the incoming request""" + + @hookspec def filters_from_request(request, database, table, datasette): """ diff --git a/datasette/views/base.py b/datasette/views/base.py index e59fd683..bdc1e9cf 100644 --- a/datasette/views/base.py +++ b/datasette/views/base.py @@ -143,7 +143,8 @@ class BaseView: async def render(self, templates, request, context=None): context = context or {} - template = self.ds.jinja_env.select_template(templates) + environment = self.ds.get_jinja_environment(request) + template = environment.select_template(templates) template_context = { **context, **{ diff --git a/datasette/views/database.py b/datasette/views/database.py index 9ba5ce94..03e70379 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -143,7 +143,8 @@ class DatabaseView(View): datasette.urls.path(path_with_format(request=request, format="json")), ) templates = (f"database-{to_css_class(database)}.html", "database.html") - template = datasette.jinja_env.select_template(templates) + environment = datasette.get_jinja_environment(request) + template = environment.select_template(templates) context = { **json_data, "database_color": db.color, @@ -594,7 +595,8 @@ class QueryView(View): f"query-{to_css_class(database)}-{to_css_class(canned_query['name'])}.html", ) - template = datasette.jinja_env.select_template(templates) + environment = datasette.get_jinja_environment(request) + template = environment.select_template(templates) alternate_url_json = datasette.absolute_url( request, datasette.urls.path(path_with_format(request=request, format="json")), diff --git a/datasette/views/table.py b/datasette/views/table.py index 4f4baeed..7ee5d6bf 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -806,7 +806,8 @@ async def table_view_traced(datasette, request): f"table-{to_css_class(resolved.db.name)}-{to_css_class(resolved.table)}.html", "table.html", ] - template = datasette.jinja_env.select_template(templates) + environment = datasette.get_jinja_environment(request) + template = environment.select_template(templates) alternate_url_json = datasette.absolute_url( request, datasette.urls.path(path_with_format(request=request, format="json")), diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index eb6bf4ae..f67d15d6 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -1128,6 +1128,48 @@ These IDs could be integers or strings, depending on how the actors used by the Example: `datasette-remote-actors `_ +.. _plugin_hook_jinja2_environment_from_request: + +jinja2_environment_from_request(datasette, request, env) +-------------------------------------------------------- + +``datasette`` - :ref:`internals_datasette` + A Datasette instance. + +``request`` - :ref:`internals_request` or ``None`` + The current HTTP request, if one is available. + +``env`` - ``Environment`` + The Jinja2 environment that will be used to render the current page. + +This hook can be used to return a customized `Jinja environment `__ based on the incoming request. + +If you want to run a single Datasette instance that serves different content for different domains, you can do so like this: + +.. code-block:: python + + from datasette import hookimpl + from jinja2 import ChoiceLoader, FileSystemLoader + + + @hookimpl + def jinja2_environment_from_request(request, env): + if request and request.host == "www.niche-museums.com": + return env.overlay( + loader=ChoiceLoader( + [ + FileSystemLoader( + "/mnt/niche-museums/templates" + ), + env.loader, + ] + ), + enable_async=True, + ) + return env + +This uses the Jinja `overlay() method `__ to create a new environment identical to the default environment except for having a different template loader, which first looks in the ``/mnt/niche-museums/templates`` directory before falling back on the default loader. + .. _plugin_hook_filters_from_request: filters_from_request(request, database, table, datasette) diff --git a/tests/test_plugins.py b/tests/test_plugins.py index 82e2f7f1..bdd4ba49 100644 --- a/tests/test_plugins.py +++ b/tests/test_plugins.py @@ -16,6 +16,7 @@ from datasette.plugins import get_plugins, DEFAULT_PLUGINS, pm from datasette.utils.sqlite import sqlite3 from datasette.utils import CustomRow, StartupError from jinja2.environment import Template +from jinja2 import ChoiceLoader, FileSystemLoader import base64 import importlib import json @@ -563,7 +564,8 @@ async def test_hook_register_output_renderer_can_render(ds_client): async def test_hook_prepare_jinja2_environment(ds_client): ds_client.ds._HELLO = "HI" await ds_client.ds.invoke_startup() - template = ds_client.ds.jinja_env.from_string( + environment = ds_client.ds.get_jinja_environment(None) + template = environment.from_string( "Hello there, {{ a|format_numeric }}, {{ a|to_hello }}, {{ b|select_times_three }}", {"a": 3412341, "b": 5}, ) @@ -1294,3 +1296,41 @@ async def test_plugin_is_installed(): finally: pm.unregister(name="DummyPlugin") + + +@pytest.mark.asyncio +async def test_hook_jinja2_environment_from_request(tmpdir): + templates = pathlib.Path(tmpdir / "templates") + templates.mkdir() + (templates / "index.html").write_text("Hello museums!", "utf-8") + + class EnvironmentPlugin: + @hookimpl + def jinja2_environment_from_request(self, request, env): + if request and request.host == "www.niche-museums.com": + return env.overlay( + loader=ChoiceLoader( + [ + FileSystemLoader(str(templates)), + env.loader, + ] + ), + enable_async=True, + ) + return env + + datasette = Datasette(memory=True) + + try: + pm.register(EnvironmentPlugin(), name="EnvironmentPlugin") + response = await datasette.client.get("/") + assert response.status_code == 200 + assert "Hello museums!" not in response.text + # Try again with the hostname + response2 = await datasette.client.get( + "/", headers={"host": "www.niche-museums.com"} + ) + assert response2.status_code == 200 + assert "Hello museums!" in response2.text + finally: + pm.unregister(name="EnvironmentPlugin") From 1fc76fee6268c21003c0fe730cc8e93210ce6bb8 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 5 Jan 2024 16:59:25 -0800 Subject: [PATCH 542/952] 1.0a8.dev1 version number Not going to release this to PyPI but I will build my own wheel of it --- datasette/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/datasette/version.py b/datasette/version.py index 55e2cd42..75d44727 100644 --- a/datasette/version.py +++ b/datasette/version.py @@ -1,2 +1,2 @@ -__version__ = "1.0a7" +__version__ = "1.0a8.dev1" __version_info__ = tuple(__version__.split(".")) From 0b2c6a7ebd4fd540d9bdfb169c621452d280e608 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 8 Jan 2024 13:12:57 -0800 Subject: [PATCH 543/952] Fix for ?_extra=columns bug, closes #2230 Also refs #262 - started a test suite for extras. --- datasette/renderer.py | 2 +- tests/test_table_api.py | 24 ++++++++++++++++++++++++ 2 files changed, 25 insertions(+), 1 deletion(-) diff --git a/datasette/renderer.py b/datasette/renderer.py index 224031a7..a446e69d 100644 --- a/datasette/renderer.py +++ b/datasette/renderer.py @@ -68,7 +68,7 @@ def json_renderer(request, args, data, error, truncated=None): elif shape in ("objects", "object", "array"): columns = data.get("columns") rows = data.get("rows") - if rows and columns: + if rows and columns and not isinstance(rows[0], dict): data["rows"] = [dict(zip(columns, row)) for row in rows] if shape == "object": shape_error = None diff --git a/tests/test_table_api.py b/tests/test_table_api.py index 5dbb8b8f..ae4fdb17 100644 --- a/tests/test_table_api.py +++ b/tests/test_table_api.py @@ -1362,3 +1362,27 @@ async def test_col_nocol_errors(ds_client, path, expected_error): response = await ds_client.get(path) assert response.status_code == 400 assert response.json()["error"] == expected_error + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "extra,expected_json", + ( + ( + "columns", + { + "ok": True, + "next": None, + "columns": ["id", "content", "content2"], + "rows": [{"id": "1", "content": "hey", "content2": "world"}], + "truncated": False, + }, + ), + ), +) +async def test_table_extras(ds_client, extra, expected_json): + response = await ds_client.get( + "/fixtures/primary_key_multiple_columns.json?_extra=" + extra + ) + assert response.status_code == 200 + assert response.json() == expected_json From 2ff4d4a60a348c143f79d63c48c329ffd0c1f02f Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 8 Jan 2024 13:13:53 -0800 Subject: [PATCH 544/952] Test for ?_extra=count, refs #262 --- tests/test_table_api.py | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/tests/test_table_api.py b/tests/test_table_api.py index ae4fdb17..bde7a38e 100644 --- a/tests/test_table_api.py +++ b/tests/test_table_api.py @@ -1378,6 +1378,16 @@ async def test_col_nocol_errors(ds_client, path, expected_error): "truncated": False, }, ), + ( + "count", + { + "ok": True, + "next": None, + "rows": [{"id": "1", "content": "hey", "content2": "world"}], + "truncated": False, + "count": 1, + }, + ), ), ) async def test_table_extras(ds_client, extra, expected_json): From 48148e66a846d585e08ec6ab4ae3da8e60d55ab5 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 10 Jan 2024 10:42:36 -0800 Subject: [PATCH 545/952] Link from actors_from_ids plugin hook docs to datasette.actors_from_ids() --- docs/plugin_hooks.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index f67d15d6..9115c3df 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -1086,6 +1086,8 @@ The hook must return a dictionary that maps the incoming actor IDs to their full Some plugins that implement social features may store the ID of the :ref:`actor ` that performed an action - added a comment, bookmarked a table or similar - and then need a way to resolve those IDs into display-friendly actor dictionaries later on. +The :ref:`await datasette.actors_from_ids(actor_ids) ` internal method can be used to look up actors from their IDs. It will dispatch to the first plugin that implements this hook. + Unlike other plugin hooks, this only uses the first implementation of the hook to return a result. You can expect users to only have a single plugin installed that implements this hook. If no plugin is installed, Datasette defaults to returning actors that are just ``{"id": actor_id}``. From 7506a89be0d1c97632bed47635eb90f92815d6c7 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 10 Jan 2024 13:04:34 -0800 Subject: [PATCH 546/952] Docs on datasette.client for tests, closes #1830 Also covers ds.client.actor_cookie() helper --- docs/testing_plugins.rst | 28 +++++++++++++++++++++++++++ tests/test_docs.py | 41 ++++++++++++++++++++++++++++++++++++++-- 2 files changed, 67 insertions(+), 2 deletions(-) diff --git a/docs/testing_plugins.rst b/docs/testing_plugins.rst index 6d2097ad..e10514c6 100644 --- a/docs/testing_plugins.rst +++ b/docs/testing_plugins.rst @@ -82,6 +82,34 @@ This method registers any :ref:`plugin_hook_startup` or :ref:`plugin_hook_prepar If you are using ``await datasette.client.get()`` and similar methods then you don't need to worry about this - Datasette automatically calls ``invoke_startup()`` the first time it handles a request. +.. _testing_datasette_client: + +Using datasette.client in tests +------------------------------- + +The :ref:`internals_datasette_client` mechanism is designed for use in tests. It provides access to a pre-configured `HTTPX async client `__ instance that can make GET, POST and other HTTP requests against a Datasette instance from inside a test. + +I simple test looks like this: + +.. literalinclude:: ../tests/test_docs.py + :language: python + :start-after: # -- start test_homepage -- + :end-before: # -- end test_homepage -- + +Or for a JSON API: + +.. literalinclude:: ../tests/test_docs.py + :language: python + :start-after: # -- start test_actor_is_null -- + :end-before: # -- end test_actor_is_null -- + +To make requests as an authenticated actor, create a signed ``ds_cookie`` using the ``datasette.client.actor_cookie()`` helper function and pass it in ``cookies=`` like this: + +.. literalinclude:: ../tests/test_docs.py + :language: python + :start-after: # -- start test_signed_cookie_actor -- + :end-before: # -- end test_signed_cookie_actor -- + .. _testing_plugins_pdb: Using pdb for errors thrown inside Datasette diff --git a/tests/test_docs.py b/tests/test_docs.py index e9b813fe..fdd44788 100644 --- a/tests/test_docs.py +++ b/tests/test_docs.py @@ -1,9 +1,8 @@ """ Tests to ensure certain things are documented. """ -from click.testing import CliRunner from datasette import app, utils -from datasette.cli import cli +from datasette.app import Datasette from datasette.filters import Filters from pathlib import Path import pytest @@ -102,3 +101,41 @@ def documented_fns(): @pytest.mark.parametrize("fn", utils.functions_marked_as_documented) def test_functions_marked_with_documented_are_documented(documented_fns, fn): assert fn.__name__ in documented_fns + + +# Tests for testing_plugins.rst documentation + + +# -- start test_homepage -- +@pytest.mark.asyncio +async def test_homepage(): + ds = Datasette(memory=True) + response = await ds.client.get("/") + html = response.text + assert "

" in html + + +# -- end test_homepage -- + + +# -- start test_actor_is_null -- +@pytest.mark.asyncio +async def test_actor_is_null(): + ds = Datasette(memory=True) + response = await ds.client.get("/-/actor.json") + assert response.json() == {"actor": None} + + +# -- end test_actor_is_null -- + + +# -- start test_signed_cookie_actor -- +@pytest.mark.asyncio +async def test_signed_cookie_actor(): + ds = Datasette(memory=True) + cookies = {"ds_actor": ds.client.actor_cookie({"id": "root"})} + response = await ds.client.get("/-/actor.json", cookies=cookies) + assert response.json() == {"actor": {"id": "root"}} + + +# -- end test_signed_cookie_actor -- From 0f63cb83ed31753a9bd9ec5cc71de16906767337 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 10 Jan 2024 13:08:52 -0800 Subject: [PATCH 547/952] Typo fix --- docs/testing_plugins.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing_plugins.rst b/docs/testing_plugins.rst index e10514c6..33ac4b22 100644 --- a/docs/testing_plugins.rst +++ b/docs/testing_plugins.rst @@ -89,7 +89,7 @@ Using datasette.client in tests The :ref:`internals_datasette_client` mechanism is designed for use in tests. It provides access to a pre-configured `HTTPX async client `__ instance that can make GET, POST and other HTTP requests against a Datasette instance from inside a test. -I simple test looks like this: +A simple test looks like this: .. literalinclude:: ../tests/test_docs.py :language: python From a25bf6bea789c409580386f77b7440ff525d09b2 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 10 Jan 2024 14:10:40 -0800 Subject: [PATCH 548/952] fmt: off to fix problem with Black, closes #2231 --- tests/test_docs.py | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/tests/test_docs.py b/tests/test_docs.py index fdd44788..17c01a0b 100644 --- a/tests/test_docs.py +++ b/tests/test_docs.py @@ -105,7 +105,7 @@ def test_functions_marked_with_documented_are_documented(documented_fns, fn): # Tests for testing_plugins.rst documentation - +# fmt: off # -- start test_homepage -- @pytest.mark.asyncio async def test_homepage(): @@ -113,8 +113,6 @@ async def test_homepage(): response = await ds.client.get("/") html = response.text assert "

" in html - - # -- end test_homepage -- @@ -124,8 +122,6 @@ async def test_actor_is_null(): ds = Datasette(memory=True) response = await ds.client.get("/-/actor.json") assert response.json() == {"actor": None} - - # -- end test_actor_is_null -- @@ -136,6 +132,4 @@ async def test_signed_cookie_actor(): cookies = {"ds_actor": ds.client.actor_cookie({"id": "root"})} response = await ds.client.get("/-/actor.json", cookies=cookies) assert response.json() == {"actor": {"id": "root"}} - - # -- end test_signed_cookie_actor -- From 7a5adb592ae6674a2058639c66e85eb1b49448fb Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 12 Jan 2024 14:12:14 -0800 Subject: [PATCH 549/952] Docs on temporary plugins in fixtures, closes #2234 --- docs/testing_plugins.rst | 16 ++++++++++++++++ tests/test_docs_plugins.py | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 50 insertions(+) create mode 100644 tests/test_docs_plugins.py diff --git a/docs/testing_plugins.rst b/docs/testing_plugins.rst index 33ac4b22..f1363fb4 100644 --- a/docs/testing_plugins.rst +++ b/docs/testing_plugins.rst @@ -313,3 +313,19 @@ When writing tests for plugins you may find it useful to register a test plugin assert response.status_code == 500 finally: pm.unregister(name="undo") + +To reuse the same temporary plugin in multiple tests, you can register it inside a fixture in your ``conftest.py`` file like this: + +.. literalinclude:: ../tests/test_docs_plugins.py + :language: python + :start-after: # -- start datasette_with_plugin_fixture -- + :end-before: # -- end datasette_with_plugin_fixture -- + +Note the ``yield`` statement here - this ensures that the ``finally:`` block that unregisters the plugin is executed only after the test function itself has completed. + +Then in a test: + +.. literalinclude:: ../tests/test_docs_plugins.py + :language: python + :start-after: # -- start datasette_with_plugin_test -- + :end-before: # -- end datasette_with_plugin_test -- diff --git a/tests/test_docs_plugins.py b/tests/test_docs_plugins.py new file mode 100644 index 00000000..92b4514c --- /dev/null +++ b/tests/test_docs_plugins.py @@ -0,0 +1,34 @@ +# fmt: off +# -- start datasette_with_plugin_fixture -- +from datasette import hookimpl +from datasette.app import Datasette +from datasette.plugins import pm +import pytest +import pytest_asyncio + + +@pytest_asyncio.fixture +async def datasette_with_plugin(): + class TestPlugin: + __name__ = "TestPlugin" + + @hookimpl + def register_routes(self): + return [ + (r"^/error$", lambda: 1 / 0), + ] + + pm.register(TestPlugin(), name="undo") + try: + yield Datasette() + finally: + pm.unregister(name="undo") +# -- end datasette_with_plugin_fixture -- + + +# -- start datasette_with_plugin_test -- +@pytest.mark.asyncio +async def test_error(datasette_with_plugin): + response = await datasette_with_plugin.client.get("/error") + assert response.status_code == 500 +# -- end datasette_with_plugin_test -- From c3caf36af7db79336a5c8e697b2374e90e34ff5d Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 30 Jan 2024 19:54:03 -0800 Subject: [PATCH 550/952] Template slot family of plugin hooks - top_homepage() and others New plugin hooks: top_homepage top_database top_table top_row top_query top_canned_query New datasette.utils.make_slot_function() Closes #1191 --- datasette/hookspecs.py | 30 ++++++++ datasette/templates/database.html | 2 + datasette/templates/index.html | 2 + datasette/templates/query.html | 2 + datasette/templates/row.html | 2 + datasette/templates/table.html | 2 + datasette/utils/__init__.py | 17 +++++ datasette/views/database.py | 21 +++++- datasette/views/index.py | 9 ++- datasette/views/row.py | 12 ++- datasette/views/table.py | 8 ++ docs/plugin_hooks.rst | 119 ++++++++++++++++++++++++++++++ tests/test_docs.py | 4 +- tests/test_plugins.py | 101 +++++++++++++++++++++++++ 14 files changed, 324 insertions(+), 7 deletions(-) diff --git a/datasette/hookspecs.py b/datasette/hookspecs.py index b6975dce..2f4c6027 100644 --- a/datasette/hookspecs.py +++ b/datasette/hookspecs.py @@ -158,3 +158,33 @@ def skip_csrf(datasette, scope): @hookspec def handle_exception(datasette, request, exception): """Handle an uncaught exception. Can return a Response or None.""" + + +@hookspec +def top_homepage(datasette, request): + """HTML to include at the top of the homepage""" + + +@hookspec +def top_database(datasette, request, database): + """HTML to include at the top of the database page""" + + +@hookspec +def top_table(datasette, request, database, table): + """HTML to include at the top of the table page""" + + +@hookspec +def top_row(datasette, request, database, table, row): + """HTML to include at the top of the row page""" + + +@hookspec +def top_query(datasette, request, database, sql): + """HTML to include at the top of the query results page""" + + +@hookspec +def top_canned_query(datasette, request, database, query_name): + """HTML to include at the top of the canned query page""" diff --git a/datasette/templates/database.html b/datasette/templates/database.html index 3d4dae07..4b125a44 100644 --- a/datasette/templates/database.html +++ b/datasette/templates/database.html @@ -34,6 +34,8 @@

{% endif %}
+{{ top_database() }} + {% block description_source_license %}{% include "_description_source_license.html" %}{% endblock %} {% if allow_execute_sql %} diff --git a/datasette/templates/index.html b/datasette/templates/index.html index 06e09635..203abca8 100644 --- a/datasette/templates/index.html +++ b/datasette/templates/index.html @@ -7,6 +7,8 @@ {% block content %}

{{ metadata.title or "Datasette" }}{% if private %} 🔒{% endif %}

+{{ top_homepage() }} + {% block description_source_license %}{% include "_description_source_license.html" %}{% endblock %} {% for database in databases %} diff --git a/datasette/templates/query.html b/datasette/templates/query.html index b8f06f84..1815e592 100644 --- a/datasette/templates/query.html +++ b/datasette/templates/query.html @@ -30,6 +30,8 @@

{{ metadata.title or database }}{% if canned_query and not metadata.title %}: {{ canned_query }}{% endif %}{% if private %} 🔒{% endif %}

+{% if canned_query %}{{ top_canned_query() }}{% else %}{{ top_query() }}{% endif %} + {% block description_source_license %}{% include "_description_source_license.html" %}{% endblock %} diff --git a/datasette/templates/row.html b/datasette/templates/row.html index 4d179a85..6d4b996e 100644 --- a/datasette/templates/row.html +++ b/datasette/templates/row.html @@ -22,6 +22,8 @@ {% block content %}

{{ table }}: {{ ', '.join(primary_key_values) }}{% if private %} 🔒{% endif %}

+{{ top_row() }} + {% block description_source_license %}{% include "_description_source_license.html" %}{% endblock %}

This data as {% for name, url in renderers.items() %}{{ name }}{{ ", " if not loop.last }}{% endfor %}

diff --git a/datasette/templates/table.html b/datasette/templates/table.html index 88580e52..5aee6319 100644 --- a/datasette/templates/table.html +++ b/datasette/templates/table.html @@ -45,6 +45,8 @@
{% endif %}
+{{ top_table() }} + {% block description_source_license %}{% include "_description_source_license.html" %}{% endblock %} {% if metadata.get("columns") %} diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index 0f449b89..8914c043 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -1283,3 +1283,20 @@ def fail_if_plugins_in_metadata(metadata: dict, filename=None): f'Datasette no longer accepts plugin configuration in --metadata. Move your "plugins" configuration blocks to a separate file - we suggest calling that datasette.{suggested_extension} - and start Datasette with datasette -c datasette.{suggested_extension}. See https://docs.datasette.io/en/latest/configuration.html for more details.' ) return metadata + + +def make_slot_function(name, datasette, request, **kwargs): + from datasette.plugins import pm + + method = getattr(pm.hook, name, None) + assert method is not None, "No hook found for {}".format(name) + + async def inner(): + html_bits = [] + for hook in method(datasette=datasette, request=request, **kwargs): + html = await await_me_maybe(hook) + if html is not None: + html_bits.append(html) + return markupsafe.Markup("".join(html_bits)) + + return inner diff --git a/datasette/views/database.py b/datasette/views/database.py index 03e70379..caeb4e46 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -1,5 +1,4 @@ from dataclasses import dataclass, field -from typing import Callable from urllib.parse import parse_qsl, urlencode import asyncio import hashlib @@ -18,6 +17,7 @@ from datasette.utils import ( call_with_supported_arguments, derive_named_parameters, format_bytes, + make_slot_function, tilde_decode, to_css_class, validate_sql_select, @@ -161,6 +161,9 @@ class DatabaseView(View): f"{'*' if template_name == template.name else ''}{template_name}" for template_name in templates ], + "top_database": make_slot_function( + "top_database", datasette, request, database=database + ), } return Response.html( await datasette.render_template( @@ -246,6 +249,12 @@ class QueryContext: "help": "List of templates that were considered for rendering this page" } ) + top_query: callable = field( + metadata={"help": "Callable to render the top_query slot"} + ) + top_canned_query: callable = field( + metadata={"help": "Callable to render the top_canned_query slot"} + ) async def get_tables(datasette, request, db): @@ -727,6 +736,16 @@ class QueryView(View): f"{'*' if template_name == template.name else ''}{template_name}" for template_name in templates ], + top_query=make_slot_function( + "top_query", datasette, request, database=database, sql=sql + ), + top_canned_query=make_slot_function( + "top_canned_query", + datasette, + request, + database=database, + query_name=canned_query["name"] if canned_query else None, + ), ), request=request, view_name="database", diff --git a/datasette/views/index.py b/datasette/views/index.py index 95b29302..595cf234 100644 --- a/datasette/views/index.py +++ b/datasette/views/index.py @@ -1,10 +1,12 @@ -import hashlib import json -from datasette.utils import add_cors_headers, CustomJSONEncoder +from datasette.plugins import pm +from datasette.utils import add_cors_headers, make_slot_function, CustomJSONEncoder from datasette.utils.asgi import Response from datasette.version import __version__ +from markupsafe import Markup + from .base import BaseView @@ -142,5 +144,8 @@ class IndexView(BaseView): "private": not await self.ds.permission_allowed( None, "view-instance" ), + "top_homepage": make_slot_function( + "top_homepage", self.ds, request + ), }, ) diff --git a/datasette/views/row.py b/datasette/views/row.py index 8f07a662..ce877753 100644 --- a/datasette/views/row.py +++ b/datasette/views/row.py @@ -2,11 +2,9 @@ from datasette.utils.asgi import NotFound, Forbidden, Response from datasette.database import QueryInterrupted from .base import DataView, BaseView, _error from datasette.utils import ( - tilde_decode, - urlsafe_components, + make_slot_function, to_css_class, escape_sqlite, - row_sql_params_pks, ) import json import sqlite_utils @@ -73,6 +71,14 @@ class RowView(DataView): .get(database, {}) .get("tables", {}) .get(table, {}), + "top_row": make_slot_function( + "top_row", + self.ds, + request, + database=resolved.db.name, + table=resolved.table, + row=rows[0], + ), } data = { diff --git a/datasette/views/table.py b/datasette/views/table.py index 7ee5d6bf..be7479f8 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -17,6 +17,7 @@ from datasette.utils import ( append_querystring, compound_keys_after_sql, format_bytes, + make_slot_function, tilde_encode, escape_sqlite, filters_should_redirect, @@ -842,6 +843,13 @@ async def table_view_traced(datasette, request): f"{'*' if template_name == template.name else ''}{template_name}" for template_name in templates ], + top_table=make_slot_function( + "top_table", + datasette, + request, + database=resolved.db.name, + table=resolved.table, + ), ), request=request, view_name="table", diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index 9115c3df..ce648ba7 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -1641,3 +1641,122 @@ This hook is responsible for returning a dictionary corresponding to Datasette : return metadata Example: `datasette-remote-metadata plugin `__ + +.. _plugin_hook_slots: + +Template slots +-------------- + +The following set of plugin hooks can be used to return extra HTML content that will be inserted into the corresponding page, directly below the ``

`` heading. + +Multiple plugins can contribute content here. The order in which it is displayed can be controlled using Pluggy's `call time order options `__. + +Each of these plugin hooks can return either a string or an awaitable function that returns a string. + +.. _plugin_hook_top_homepage: + +top_homepage(datasette, request) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +``datasette`` - :ref:`internals_datasette` + You can use this to access plugin configuration options via ``datasette.plugin_config(your_plugin_name)``. + +``request`` - :ref:`internals_request` + The current HTTP request. + +Returns HTML to be displayed at the top of the Datasette homepage. + +.. _plugin_hook_top_database: + +top_database(datasette, request, database) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +``datasette`` - :ref:`internals_datasette` + You can use this to access plugin configuration options via ``datasette.plugin_config(your_plugin_name)``. + +``request`` - :ref:`internals_request` + The current HTTP request. + +``database`` - string + The name of the database. + +Returns HTML to be displayed at the top of the database page. + +.. _plugin_hook_top_table: + +top_table(datasette, request, database, table) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +``datasette`` - :ref:`internals_datasette` + You can use this to access plugin configuration options via ``datasette.plugin_config(your_plugin_name)``. + +``request`` - :ref:`internals_request` + The current HTTP request. + +``database`` - string + The name of the database. + +``table`` - string + The name of the table. + +Returns HTML to be displayed at the top of the table page. + +.. _plugin_hook_top_row: + +top_row(datasette, request, database, table, row) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +``datasette`` - :ref:`internals_datasette` + You can use this to access plugin configuration options via ``datasette.plugin_config(your_plugin_name)``. + +``request`` - :ref:`internals_request` + The current HTTP request. + +``database`` - string + The name of the database. + +``table`` - string + The name of the table. + +``row`` - ``sqlite.Row`` + The SQLite row object being displayed. + +Returns HTML to be displayed at the top of the row page. + +.. _plugin_hook_top_query: + +top_query(datasette, request, database, sql) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +``datasette`` - :ref:`internals_datasette` + You can use this to access plugin configuration options via ``datasette.plugin_config(your_plugin_name)``. + +``request`` - :ref:`internals_request` + The current HTTP request. + +``database`` - string + The name of the database. + +``sql`` - string + The SQL query. + +Returns HTML to be displayed at the top of the query results page. + +.. _plugin_hook_top_canned_query: + +top_canned_query(datasette, request, database, query_name) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +``datasette`` - :ref:`internals_datasette` + You can use this to access plugin configuration options via ``datasette.plugin_config(your_plugin_name)``. + +``request`` - :ref:`internals_request` + The current HTTP request. + +``database`` - string + The name of the database. + +``query_name`` - string + The name of the canned query. + +Returns HTML to be displayed at the top of the canned query page. diff --git a/tests/test_docs.py b/tests/test_docs.py index 17c01a0b..0a803861 100644 --- a/tests/test_docs.py +++ b/tests/test_docs.py @@ -41,7 +41,9 @@ def plugin_hooks_content(): "plugin", [name for name in dir(app.pm.hook) if not name.startswith("_")] ) def test_plugin_hooks_are_documented(plugin, plugin_hooks_content): - headings = get_headings(plugin_hooks_content, "-") + headings = set() + headings.update(get_headings(plugin_hooks_content, "-")) + headings.update(get_headings(plugin_hooks_content, "~")) assert plugin in headings hook_caller = getattr(app.pm.hook, plugin) arg_names = [a for a in hook_caller.spec.argnames if a != "__multicall__"] diff --git a/tests/test_plugins.py b/tests/test_plugins.py index bdd4ba49..784c460a 100644 --- a/tests/test_plugins.py +++ b/tests/test_plugins.py @@ -1334,3 +1334,104 @@ async def test_hook_jinja2_environment_from_request(tmpdir): assert "Hello museums!" in response2.text finally: pm.unregister(name="EnvironmentPlugin") + + +class SlotPlugin: + __name__ = "SlotPlugin" + + @hookimpl + def top_homepage(self, request): + return "Xtop_homepage:" + request.args["z"] + + @hookimpl + def top_database(self, request, database): + async def inner(): + return "Xtop_database:{}:{}".format(database, request.args["z"]) + + return inner + + @hookimpl + def top_table(self, request, database, table): + return "Xtop_table:{}:{}:{}".format(database, table, request.args["z"]) + + @hookimpl + def top_row(self, request, database, table, row): + return "Xtop_row:{}:{}:{}:{}".format( + database, table, row["name"], request.args["z"] + ) + + @hookimpl + def top_query(self, request, database, sql): + return "Xtop_query:{}:{}:{}".format(database, sql, request.args["z"]) + + @hookimpl + def top_canned_query(self, request, database, query_name): + return "Xtop_query:{}:{}:{}".format(database, query_name, request.args["z"]) + + +@pytest.mark.asyncio +async def test_hook_top_homepage(): + try: + pm.register(SlotPlugin(), name="SlotPlugin") + datasette = Datasette(memory=True) + response = await datasette.client.get("/?z=foo") + assert response.status_code == 200 + assert "Xtop_homepage:foo" in response.text + finally: + pm.unregister(name="SlotPlugin") + + +@pytest.mark.asyncio +async def test_hook_top_database(): + try: + pm.register(SlotPlugin(), name="SlotPlugin") + datasette = Datasette(memory=True) + response = await datasette.client.get("/_memory?z=bar") + assert response.status_code == 200 + assert "Xtop_database:_memory:bar" in response.text + finally: + pm.unregister(name="SlotPlugin") + + +@pytest.mark.asyncio +async def test_hook_top_table(ds_client): + try: + pm.register(SlotPlugin(), name="SlotPlugin") + response = await ds_client.get("/fixtures/facetable?z=baz") + assert response.status_code == 200 + assert "Xtop_table:fixtures:facetable:baz" in response.text + finally: + pm.unregister(name="SlotPlugin") + + +@pytest.mark.asyncio +async def test_hook_top_row(ds_client): + try: + pm.register(SlotPlugin(), name="SlotPlugin") + response = await ds_client.get("/fixtures/facet_cities/1?z=bax") + assert response.status_code == 200 + assert "Xtop_row:fixtures:facet_cities:San Francisco:bax" in response.text + finally: + pm.unregister(name="SlotPlugin") + + +@pytest.mark.asyncio +async def test_hook_top_query(ds_client): + try: + pm.register(SlotPlugin(), name="SlotPlugin") + response = await ds_client.get("/fixtures?sql=select+1&z=x") + assert response.status_code == 200 + assert "Xtop_query:fixtures:select 1:x" in response.text + finally: + pm.unregister(name="SlotPlugin") + + +@pytest.mark.asyncio +async def test_hook_top_canned_query(ds_client): + try: + pm.register(SlotPlugin(), name="SlotPlugin") + response = await ds_client.get("/fixtures/from_hook?z=xyz") + assert response.status_code == 200 + assert "Xtop_query:fixtures:from_hook:xyz" in response.text + finally: + pm.unregister(name="SlotPlugin") From 5c64af69363100a3c35e6b131efe1f741bbde661 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 30 Jan 2024 19:55:26 -0800 Subject: [PATCH 551/952] Upgrade to latest Black, closes #2239 --- datasette/filters.py | 14 ++++++------ datasette/utils/__init__.py | 14 ++++++------ datasette/utils/shutil_backport.py | 1 + datasette/views/database.py | 22 +++++++++++-------- datasette/views/table.py | 30 ++++++++++++++------------ setup.py | 2 +- tests/plugins/my_plugin.py | 6 +++--- tests/test_api_write.py | 8 ++++--- tests/test_cli.py | 8 ++++--- tests/test_docs.py | 1 + tests/test_internals_database.py | 1 + tests/test_internals_datasette.py | 1 + tests/test_permissions.py | 8 ++++--- tests/test_plugins.py | 34 ++++++++++++++++-------------- tests/test_table_api.py | 8 ++++--- tests/test_utils.py | 1 + 16 files changed, 93 insertions(+), 66 deletions(-) diff --git a/datasette/filters.py b/datasette/filters.py index 5ea3488b..73eea857 100644 --- a/datasette/filters.py +++ b/datasette/filters.py @@ -80,9 +80,9 @@ def search_filters(request, database, table, datasette): "{fts_pk} in (select rowid from {fts_table} where {fts_table} match {match_clause})".format( fts_table=escape_sqlite(fts_table), fts_pk=escape_sqlite(fts_pk), - match_clause=":search" - if search_mode_raw - else "escape_fts(:search)", + match_clause=( + ":search" if search_mode_raw else "escape_fts(:search)" + ), ) ) human_descriptions.append(f'search matches "{search}"') @@ -99,9 +99,11 @@ def search_filters(request, database, table, datasette): "rowid in (select rowid from {fts_table} where {search_col} match {match_clause})".format( fts_table=escape_sqlite(fts_table), search_col=escape_sqlite(search_col), - match_clause=":search_{}".format(i) - if search_mode_raw - else "escape_fts(:search_{})".format(i), + match_clause=( + ":search_{}".format(i) + if search_mode_raw + else "escape_fts(:search_{})".format(i) + ), ) ) human_descriptions.append( diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index 8914c043..196e1682 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -402,9 +402,9 @@ def make_dockerfile( apt_get_extras = apt_get_extras_ if spatialite: apt_get_extras.extend(["python3-dev", "gcc", "libsqlite3-mod-spatialite"]) - environment_variables[ - "SQLITE_EXTENSIONS" - ] = "/usr/lib/x86_64-linux-gnu/mod_spatialite.so" + environment_variables["SQLITE_EXTENSIONS"] = ( + "/usr/lib/x86_64-linux-gnu/mod_spatialite.so" + ) return """ FROM python:3.11.0-slim-bullseye COPY . /app @@ -416,9 +416,11 @@ RUN datasette inspect {files} --inspect-file inspect-data.json ENV PORT {port} EXPOSE {port} CMD {cmd}""".format( - apt_get_extras=APT_GET_DOCKERFILE_EXTRAS.format(" ".join(apt_get_extras)) - if apt_get_extras - else "", + apt_get_extras=( + APT_GET_DOCKERFILE_EXTRAS.format(" ".join(apt_get_extras)) + if apt_get_extras + else "" + ), environment_variables="\n".join( [ "ENV {} '{}'".format(key, value) diff --git a/datasette/utils/shutil_backport.py b/datasette/utils/shutil_backport.py index dbe22404..d1fd1bd7 100644 --- a/datasette/utils/shutil_backport.py +++ b/datasette/utils/shutil_backport.py @@ -4,6 +4,7 @@ Backported from Python 3.8. This code is licensed under the Python License: https://github.com/python/cpython/blob/v3.8.3/LICENSE """ + import os from shutil import copy, copy2, copystat, Error diff --git a/datasette/views/database.py b/datasette/views/database.py index caeb4e46..eac01ab6 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -126,9 +126,9 @@ class DatabaseView(View): "views": sql_views, "queries": canned_queries, "allow_execute_sql": allow_execute_sql, - "table_columns": await _table_columns(datasette, database) - if allow_execute_sql - else {}, + "table_columns": ( + await _table_columns(datasette, database) if allow_execute_sql else {} + ), } if format_ == "json": @@ -719,9 +719,11 @@ class QueryView(View): display_rows=await display_rows( datasette, database, request, rows, columns ), - table_columns=await _table_columns(datasette, database) - if allow_execute_sql - else {}, + table_columns=( + await _table_columns(datasette, database) + if allow_execute_sql + else {} + ), columns=columns, renderers=renderers, url_csv=datasette.urls.path( @@ -1036,9 +1038,11 @@ async def display_rows(datasette, database, request, rows, columns): display_value = markupsafe.Markup( '<Binary: {:,} byte{}>'.format( blob_url, - ' title="{}"'.format(formatted) - if "bytes" not in formatted - else "", + ( + ' title="{}"'.format(formatted) + if "bytes" not in formatted + else "" + ), len(value), "" if len(value) == 1 else "s", ) diff --git a/datasette/views/table.py b/datasette/views/table.py index be7479f8..2c5e3e13 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -236,9 +236,11 @@ async def display_columns_and_rows( path_from_row_pks(row, pks, not pks), column, ), - ' title="{}"'.format(formatted) - if "bytes" not in formatted - else "", + ( + ' title="{}"'.format(formatted) + if "bytes" not in formatted + else "" + ), len(value), "" if len(value) == 1 else "s", ) @@ -289,9 +291,9 @@ async def display_columns_and_rows( "column": column, "value": display_value, "raw": value, - "value_type": "none" - if value is None - else str(type(value).__name__), + "value_type": ( + "none" if value is None else str(type(value).__name__) + ), } ) cell_rows.append(Row(cells)) @@ -974,9 +976,9 @@ async def table_view_data( from_sql = "from {table_name} {where}".format( table_name=escape_sqlite(table_name), - where=("where {} ".format(" and ".join(where_clauses))) - if where_clauses - else "", + where=( + ("where {} ".format(" and ".join(where_clauses))) if where_clauses else "" + ), ) # Copy of params so we can mutate them later: from_sql_params = dict(**params) @@ -1040,10 +1042,12 @@ async def table_view_data( column=escape_sqlite(sort or sort_desc), op=">" if sort else "<", p=len(params), - extra_desc_only="" - if sort - else " or {column2} is null".format( - column2=escape_sqlite(sort or sort_desc) + extra_desc_only=( + "" + if sort + else " or {column2} is null".format( + column2=escape_sqlite(sort or sort_desc) + ) ), next_clauses=" and ".join(next_by_pk_clauses), ) diff --git a/setup.py b/setup.py index d09a9e3d..cd393368 100644 --- a/setup.py +++ b/setup.py @@ -85,7 +85,7 @@ setup( "pytest-xdist>=2.2.1", "pytest-asyncio>=0.17", "beautifulsoup4>=4.8.1", - "black==23.9.1", + "black==24.1.1", "blacken-docs==1.16.0", "pytest-timeout>=1.4.2", "trustme>=0.7", diff --git a/tests/plugins/my_plugin.py b/tests/plugins/my_plugin.py index eb70d9bd..9d1f86bc 100644 --- a/tests/plugins/my_plugin.py +++ b/tests/plugins/my_plugin.py @@ -39,9 +39,9 @@ def extra_css_urls(template, database, table, view_name, columns, request, datas "database": database, "table": table, "view_name": view_name, - "request_path": request.path - if request is not None - else None, + "request_path": ( + request.path if request is not None else None + ), "added": ( await datasette.get_database().execute("select 3 * 5") ).first()[0], diff --git a/tests/test_api_write.py b/tests/test_api_write.py index f27d143f..1787e06f 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -279,9 +279,11 @@ async def test_insert_or_upsert_row_errors( json=input, headers={ "Authorization": "Bearer {}".format(token), - "Content-Type": "text/plain" - if special_case == "invalid_content_type" - else "application/json", + "Content-Type": ( + "text/plain" + if special_case == "invalid_content_type" + else "application/json" + ), }, ) diff --git a/tests/test_cli.py b/tests/test_cli.py index 213db416..080e8353 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -335,9 +335,11 @@ def test_serve_create(tmpdir): def test_serve_config(tmpdir, argument, format_): config_path = tmpdir / "datasette.{}".format(format_) config_path.write_text( - "settings:\n default_page_size: 5\n" - if format_ == "yaml" - else '{"settings": {"default_page_size": 5}}', + ( + "settings:\n default_page_size: 5\n" + if format_ == "yaml" + else '{"settings": {"default_page_size": 5}}' + ), "utf-8", ) runner = CliRunner() diff --git a/tests/test_docs.py b/tests/test_docs.py index 0a803861..2a58d954 100644 --- a/tests/test_docs.py +++ b/tests/test_docs.py @@ -1,6 +1,7 @@ """ Tests to ensure certain things are documented. """ + from datasette import app, utils from datasette.app import Datasette from datasette.filters import Filters diff --git a/tests/test_internals_database.py b/tests/test_internals_database.py index e0511100..dd68a6cb 100644 --- a/tests/test_internals_database.py +++ b/tests/test_internals_database.py @@ -1,6 +1,7 @@ """ Tests for the datasette.database.Database class """ + from datasette.app import Datasette from datasette.database import Database, Results, MultipleValues from datasette.utils.sqlite import sqlite3 diff --git a/tests/test_internals_datasette.py b/tests/test_internals_datasette.py index 428b259d..c30bb748 100644 --- a/tests/test_internals_datasette.py +++ b/tests/test_internals_datasette.py @@ -1,6 +1,7 @@ """ Tests for the datasette.app.Datasette class """ + import dataclasses from datasette import Forbidden, Context from datasette.app import Datasette, Database diff --git a/tests/test_permissions.py b/tests/test_permissions.py index 933aa07b..9917b749 100644 --- a/tests/test_permissions.py +++ b/tests/test_permissions.py @@ -381,9 +381,11 @@ async def test_permissions_debug(ds_client): { "action": div.select_one(".check-action").text, # True = green tick, False = red cross, None = gray None - "result": None - if div.select(".check-result-no-opinion") - else bool(div.select(".check-result-true")), + "result": ( + None + if div.select(".check-result-no-opinion") + else bool(div.select(".check-result-true")) + ), "used_default": bool(div.select(".check-used-default")), } for div in check_divs diff --git a/tests/test_plugins.py b/tests/test_plugins.py index 784c460a..5bfb6132 100644 --- a/tests/test_plugins.py +++ b/tests/test_plugins.py @@ -1096,24 +1096,26 @@ async def test_hook_filters_from_request(ds_client): @pytest.mark.parametrize("extra_metadata", (False, True)) async def test_hook_register_permissions(extra_metadata): ds = Datasette( - config={ - "plugins": { - "datasette-register-permissions": { - "permissions": [ - { - "name": "extra-from-metadata", - "abbr": "efm", - "description": "Extra from metadata", - "takes_database": False, - "takes_resource": False, - "default": True, - } - ] + config=( + { + "plugins": { + "datasette-register-permissions": { + "permissions": [ + { + "name": "extra-from-metadata", + "abbr": "efm", + "description": "Extra from metadata", + "takes_database": False, + "takes_resource": False, + "default": True, + } + ] + } } } - } - if extra_metadata - else None, + if extra_metadata + else None + ), plugins_dir=PLUGINS_DIR, ) await ds.invoke_startup() diff --git a/tests/test_table_api.py b/tests/test_table_api.py index bde7a38e..58930950 100644 --- a/tests/test_table_api.py +++ b/tests/test_table_api.py @@ -305,9 +305,11 @@ async def test_paginate_compound_keys_with_extra_filters(ds_client): "_sort_desc=sortable_with_nulls", lambda row: ( 1 if row["sortable_with_nulls"] is None else 0, - -row["sortable_with_nulls"] - if row["sortable_with_nulls"] is not None - else 0, + ( + -row["sortable_with_nulls"] + if row["sortable_with_nulls"] is not None + else 0 + ), row["content"], ), "sorted by sortable_with_nulls descending", diff --git a/tests/test_utils.py b/tests/test_utils.py index 61392b8b..51577615 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -1,6 +1,7 @@ """ Tests for various datasette helper functions. """ + from datasette.app import Datasette from datasette import utils from datasette.utils.asgi import Request From b8230694ff90f9a6cd4f5b7c47fd8a71c831ee1d Mon Sep 17 00:00:00 2001 From: Forest Gregg Date: Tue, 30 Jan 2024 22:56:05 -0500 Subject: [PATCH 552/952] Set link to download db to nofollow --- datasette/templates/database.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/datasette/templates/database.html b/datasette/templates/database.html index 4b125a44..ee4dd705 100644 --- a/datasette/templates/database.html +++ b/datasette/templates/database.html @@ -97,7 +97,7 @@ {% endif %} {% if allow_download %} -

Download SQLite DB: {{ database }}.db {{ format_bytes(size) }}

+

Download SQLite DB: {{ database }}.db {{ format_bytes(size) }}

{% endif %} {% include "_codemirror_foot.html" %} From 04e8835297760416b50cc669ac6f45a7fd68170b Mon Sep 17 00:00:00 2001 From: gerrymanoim Date: Tue, 30 Jan 2024 22:56:32 -0500 Subject: [PATCH 553/952] Remove deprecated/unused args from setup.py (#2222) --- setup.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/setup.py b/setup.py index cd393368..53206eae 100644 --- a/setup.py +++ b/setup.py @@ -68,7 +68,6 @@ setup( [console_scripts] datasette=datasette.cli:cli """, - setup_requires=["pytest-runner"], extras_require={ "docs": [ "Sphinx==7.2.6", @@ -93,7 +92,6 @@ setup( ], "rich": ["rich"], }, - tests_require=["datasette[test]"], classifiers=[ "Development Status :: 4 - Beta", "Framework :: Datasette", From 959e0202972f4d95088c4c1a9df6274108af8bfb Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 30 Jan 2024 20:40:18 -0800 Subject: [PATCH 554/952] Ran blacken-docs --- docs/internals.rst | 3 +-- docs/plugin_hooks.rst | 3 +-- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/internals.rst b/docs/internals.rst index d269bc7d..d8f86251 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -210,8 +210,7 @@ To set cookies on the response, use the ``response.set_cookie(...)`` method. The secure=False, httponly=False, samesite="lax", - ): - ... + ): ... You can use this with :ref:`datasette.sign() ` to set signed cookies. Here's how you would set the :ref:`ds_actor cookie ` for use with Datasette :ref:`authentication `: diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index ce648ba7..da69c6c9 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -373,8 +373,7 @@ Let's say you want to build a plugin that adds a ``datasette publish my_hosting_ about, about_url, api_key, - ): - ... + ): ... Examples: `datasette-publish-fly `_, `datasette-publish-vercel `_ From 890615b3f29dcf82a792f1a145b02dba784a5b63 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 31 Jan 2024 10:53:57 -0800 Subject: [PATCH 555/952] Bump the python-packages group with 1 update (#2241) Bumps the python-packages group with 1 update: [furo](https://github.com/pradyunsg/furo). Updates `furo` from 2023.9.10 to 2024.1.29 - [Release notes](https://github.com/pradyunsg/furo/releases) - [Changelog](https://github.com/pradyunsg/furo/blob/main/docs/changelog.md) - [Commits](https://github.com/pradyunsg/furo/compare/2023.09.10...2024.01.29) --- updated-dependencies: - dependency-name: furo dependency-type: direct:development update-type: version-update:semver-major dependency-group: python-packages ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 53206eae..b3915c42 100644 --- a/setup.py +++ b/setup.py @@ -71,7 +71,7 @@ setup( extras_require={ "docs": [ "Sphinx==7.2.6", - "furo==2023.9.10", + "furo==2024.1.29", "sphinx-autobuild", "codespell>=2.2.5", "blacken-docs", From bcc4f6bf1f14be6ef693f0b3fc9aa8a027977920 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 31 Jan 2024 15:21:40 -0800 Subject: [PATCH 556/952] track_event() mechanism for analytics and plugins * Closes #2240 * Documentation for event plugin hooks, refs #2240 * Include example track_event plugin in docs, refs #2240 * Tests for track_event() and register_events() hooks, refs #2240 * Initial documentation for core events, refs #2240 * Internals documentation for datasette.track_event() --- datasette/__init__.py | 1 + datasette/app.py | 16 +++ datasette/events.py | 211 ++++++++++++++++++++++++++++++++++++ datasette/hookspecs.py | 10 ++ datasette/plugins.py | 1 + datasette/views/database.py | 6 + datasette/views/row.py | 20 ++++ datasette/views/special.py | 17 ++- datasette/views/table.py | 31 ++++++ docs/conf.py | 2 + docs/events.rst | 14 +++ docs/index.rst | 1 + docs/internals.rst | 20 ++++ docs/plugin_hooks.rst | 100 +++++++++++++++++ docs/plugins.rst | 9 ++ tests/conftest.py | 33 +++++- tests/test_api.py | 7 +- tests/test_api_write.py | 64 +++++++++++ tests/test_auth.py | 19 +++- tests/test_cli.py | 6 +- tests/test_plugins.py | 31 +++++- tests/utils.py | 5 + 22 files changed, 614 insertions(+), 10 deletions(-) create mode 100644 datasette/events.py create mode 100644 docs/events.rst diff --git a/datasette/__init__.py b/datasette/__init__.py index 271e09ad..47d2b4f6 100644 --- a/datasette/__init__.py +++ b/datasette/__init__.py @@ -1,5 +1,6 @@ from datasette.permissions import Permission # noqa from datasette.version import __version_info__, __version__ # noqa +from datasette.events import Event # noqa from datasette.utils.asgi import Forbidden, NotFound, Request, Response # noqa from datasette.utils import actor_matches_allow # noqa from datasette.views import Context # noqa diff --git a/datasette/app.py b/datasette/app.py index 482cebb4..530f79bc 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -34,6 +34,7 @@ from jinja2 import ( from jinja2.environment import Template from jinja2.exceptions import TemplateNotFound +from .events import Event from .views import Context from .views.base import ureg from .views.database import database_download, DatabaseView, TableCreateView @@ -505,6 +506,14 @@ class Datasette: # This must be called for Datasette to be in a usable state if self._startup_invoked: return + # Register event classes + event_classes = [] + for hook in pm.hook.register_events(datasette=self): + extra_classes = await await_me_maybe(hook) + if extra_classes: + event_classes.extend(extra_classes) + self.event_classes = tuple(event_classes) + # Register permissions, but watch out for duplicate name/abbr names = {} abbrs = {} @@ -873,6 +882,13 @@ class Datasette: result = await await_me_maybe(result) return result + async def track_event(self, event: Event): + assert isinstance(event, self.event_classes), "Invalid event type: {}".format( + type(event) + ) + for hook in pm.hook.track_event(datasette=self, event=event): + await await_me_maybe(hook) + async def permission_allowed( self, actor, action, resource=None, default=DEFAULT_NOT_SET ): diff --git a/datasette/events.py b/datasette/events.py new file mode 100644 index 00000000..96244779 --- /dev/null +++ b/datasette/events.py @@ -0,0 +1,211 @@ +from abc import ABC, abstractproperty +from dataclasses import asdict, dataclass, field +from datasette.hookspecs import hookimpl +from datetime import datetime, timezone +from typing import Optional + + +@dataclass +class Event(ABC): + @abstractproperty + def name(self): + pass + + created: datetime = field( + init=False, default_factory=lambda: datetime.now(timezone.utc) + ) + actor: Optional[dict] + + def properties(self): + properties = asdict(self) + properties.pop("actor", None) + properties.pop("created", None) + return properties + + +@dataclass +class LoginEvent(Event): + """ + Event name: ``login`` + + A user (represented by ``event.actor``) has logged in. + """ + + name = "login" + + +@dataclass +class LogoutEvent(Event): + """ + Event name: ``logout`` + + A user (represented by ``event.actor``) has logged out. + """ + + name = "logout" + + +@dataclass +class CreateTokenEvent(Event): + """ + Event name: ``create-token`` + + A user created an API token. + + :ivar expires_after: Number of seconds after which this token will expire. + :type expires_after: int or None + :ivar restrict_all: Restricted permissions for this token. + :type restrict_all: list + :ivar restrict_database: Restricted database permissions for this token. + :type restrict_database: dict + :ivar restrict_resource: Restricted resource permissions for this token. + :type restrict_resource: dict + """ + + name = "create-token" + expires_after: Optional[int] + restrict_all: list + restrict_database: dict + restrict_resource: dict + + +@dataclass +class CreateTableEvent(Event): + """ + Event name: ``create-table`` + + A new table has been created in the database. + + :ivar database: The name of the database where the table was created. + :type database: str + :ivar table: The name of the table that was created + :type table: str + :ivar schema: The SQL schema definition for the new table. + :type schema: str + """ + + name = "create-table" + database: str + table: str + schema: str + + +@dataclass +class DropTableEvent(Event): + """ + Event name: ``drop-table`` + + A table has been dropped from the database. + + :ivar database: The name of the database where the table was dropped. + :type database: str + :ivar table: The name of the table that was dropped + :type table: str + """ + + name = "drop-table" + database: str + table: str + + +@dataclass +class InsertRowsEvent(Event): + """ + Event name: ``insert-rows`` + + Rows were inserted into a table. + + :ivar database: The name of the database where the rows were inserted. + :type database: str + :ivar table: The name of the table where the rows were inserted. + :type table: str + :ivar num_rows: The number of rows that were requested to be inserted. + :type num_rows: int + :ivar ignore: Was ignore set? + :type ignore: bool + :ivar replace: Was replace set? + :type replace: bool + """ + + name = "insert-rows" + database: str + table: str + num_rows: int + ignore: bool + replace: bool + + +@dataclass +class UpsertRowsEvent(Event): + """ + Event name: ``upsert-rows`` + + Rows were upserted into a table. + + :ivar database: The name of the database where the rows were inserted. + :type database: str + :ivar table: The name of the table where the rows were inserted. + :type table: str + :ivar num_rows: The number of rows that were requested to be inserted. + :type num_rows: int + """ + + name = "upsert-rows" + database: str + table: str + num_rows: int + + +@dataclass +class UpdateRowEvent(Event): + """ + Event name: ``update-row`` + + A row was updated in a table. + + :ivar database: The name of the database where the row was updated. + :type database: str + :ivar table: The name of the table where the row was updated. + :type table: str + :ivar pks: The primary key values of the updated row. + """ + + name = "update-row" + database: str + table: str + pks: list + + +@dataclass +class DeleteRowEvent(Event): + """ + Event name: ``delete-row`` + + A row was deleted from a table. + + :ivar database: The name of the database where the row was deleted. + :type database: str + :ivar table: The name of the table where the row was deleted. + :type table: str + :ivar pks: The primary key values of the deleted row. + """ + + name = "delete-row" + database: str + table: str + pks: list + + +@hookimpl +def register_events(): + return [ + LoginEvent, + LogoutEvent, + CreateTableEvent, + CreateTokenEvent, + DropTableEvent, + InsertRowsEvent, + UpsertRowsEvent, + UpdateRowEvent, + DeleteRowEvent, + ] diff --git a/datasette/hookspecs.py b/datasette/hookspecs.py index 2f4c6027..b473f398 100644 --- a/datasette/hookspecs.py +++ b/datasette/hookspecs.py @@ -160,6 +160,16 @@ def handle_exception(datasette, request, exception): """Handle an uncaught exception. Can return a Response or None.""" +@hookspec +def track_event(datasette, event): + """Respond to an event tracked by Datasette""" + + +@hookspec +def register_events(datasette): + """Return a list of Event subclasses to use with track_event()""" + + @hookspec def top_homepage(datasette, request): """HTML to include at the top of the homepage""" diff --git a/datasette/plugins.py b/datasette/plugins.py index 1ed3747f..f7a1905f 100644 --- a/datasette/plugins.py +++ b/datasette/plugins.py @@ -27,6 +27,7 @@ DEFAULT_PLUGINS = ( "datasette.default_menu_links", "datasette.handle_exception", "datasette.forbidden", + "datasette.events", ) pm = pluggy.PluginManager("datasette") diff --git a/datasette/views/database.py b/datasette/views/database.py index eac01ab6..6d17b16c 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -10,6 +10,7 @@ import re import sqlite_utils import textwrap +from datasette.events import CreateTableEvent from datasette.database import QueryInterrupted from datasette.utils import ( add_cors_headers, @@ -969,6 +970,11 @@ class TableCreateView(BaseView): } if rows: details["row_count"] = len(rows) + await self.ds.track_event( + CreateTableEvent( + request.actor, database=db.name, table=table_name, schema=schema + ) + ) return Response.json(details, status=201) diff --git a/datasette/views/row.py b/datasette/views/row.py index ce877753..7b646641 100644 --- a/datasette/views/row.py +++ b/datasette/views/row.py @@ -1,5 +1,6 @@ from datasette.utils.asgi import NotFound, Forbidden, Response from datasette.database import QueryInterrupted +from datasette.events import UpdateRowEvent, DeleteRowEvent from .base import DataView, BaseView, _error from datasette.utils import ( make_slot_function, @@ -200,6 +201,15 @@ class RowDeleteView(BaseView): except Exception as e: return _error([str(e)], 500) + await self.ds.track_event( + DeleteRowEvent( + actor=request.actor, + database=resolved.db.name, + table=resolved.table, + pks=resolved.pk_values, + ) + ) + return Response.json({"ok": True}, status=200) @@ -246,4 +256,14 @@ class RowUpdateView(BaseView): ) rows = list(results.rows) result["row"] = dict(rows[0]) + + await self.ds.track_event( + UpdateRowEvent( + actor=request.actor, + database=resolved.db.name, + table=resolved.table, + pks=resolved.pk_values, + ) + ) + return Response.json(result, status=200) diff --git a/datasette/views/special.py b/datasette/views/special.py index 849750bf..4088a1f9 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -1,4 +1,5 @@ import json +from datasette.events import LogoutEvent, LoginEvent, CreateTokenEvent from datasette.utils.asgi import Response, Forbidden from datasette.utils import ( actor_matches_allow, @@ -80,9 +81,9 @@ class AuthTokenView(BaseView): if secrets.compare_digest(token, self.ds._root_token): self.ds._root_token = None response = Response.redirect(self.ds.urls.instance()) - response.set_cookie( - "ds_actor", self.ds.sign({"a": {"id": "root"}}, "actor") - ) + root_actor = {"id": "root"} + response.set_cookie("ds_actor", self.ds.sign({"a": root_actor}, "actor")) + await self.ds.track_event(LoginEvent(actor=root_actor)) return response else: raise Forbidden("Invalid token") @@ -105,6 +106,7 @@ class LogoutView(BaseView): response = Response.redirect(self.ds.urls.instance()) response.set_cookie("ds_actor", "", expires=0, max_age=0) self.ds.add_message(request, "You are now logged out", self.ds.WARNING) + await self.ds.track_event(LogoutEvent(actor=request.actor)) return response @@ -349,6 +351,15 @@ class CreateTokenView(BaseView): restrict_resource=restrict_resource, ) token_bits = self.ds.unsign(token[len("dstok_") :], namespace="token") + await self.ds.track_event( + CreateTokenEvent( + actor=request.actor, + expires_after=expires_after, + restrict_all=restrict_all, + restrict_database=restrict_database, + restrict_resource=restrict_resource, + ) + ) context = await self.shared(request) context.update({"errors": errors, "token": token, "token_bits": token_bits}) return await self.render(["create_token.html"], request, context) diff --git a/datasette/views/table.py b/datasette/views/table.py index 2c5e3e13..3b812c01 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -8,6 +8,7 @@ import markupsafe from datasette.plugins import pm from datasette.database import QueryInterrupted +from datasette.events import DropTableEvent, InsertRowsEvent, UpsertRowsEvent from datasette import tracer from datasette.utils import ( add_cors_headers, @@ -467,6 +468,8 @@ class TableInsertView(BaseView): if errors: return _error(errors, 400) + num_rows = len(rows) + # No that we've passed pks to _validate_data it's safe to # fix the rowids case: if not pks: @@ -527,6 +530,29 @@ class TableInsertView(BaseView): result["rows"] = [dict(r) for r in fetched_rows.rows] else: result["rows"] = rows + # We track the number of rows requested, but do not attempt to show which were actually + # inserted or upserted v.s. ignored + if upsert: + await self.ds.track_event( + UpsertRowsEvent( + actor=request.actor, + database=database_name, + table=table_name, + num_rows=num_rows, + ) + ) + else: + await self.ds.track_event( + InsertRowsEvent( + actor=request.actor, + database=database_name, + table=table_name, + num_rows=num_rows, + ignore=bool(ignore), + replace=bool(replace), + ) + ) + return Response.json(result, status=200 if upsert else 201) @@ -587,6 +613,11 @@ class TableDropView(BaseView): sqlite_utils.Database(conn)[table_name].drop() await db.execute_write_fn(drop_table) + await self.ds.track_event( + DropTableEvent( + actor=request.actor, database=database_name, table=table_name + ) + ) return Response.json({"ok": True}, status=200) diff --git a/docs/conf.py b/docs/conf.py index ca0eb986..e13882b2 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -40,6 +40,8 @@ extensions = [ if not os.environ.get("DISABLE_SPHINX_INLINE_TABS"): extensions += ["sphinx_inline_tabs"] +autodoc_member_order = "bysource" + extlinks = { "issue": ("https://github.com/simonw/datasette/issues/%s", "#%s"), } diff --git a/docs/events.rst b/docs/events.rst new file mode 100644 index 00000000..f150ac02 --- /dev/null +++ b/docs/events.rst @@ -0,0 +1,14 @@ +.. _events: + +Events +====== + +Datasette includes a mechanism for tracking events that occur while the software is running. This is primarily intended to be used by plugins, which can both trigger events and listen for events. + +The core Datasette application triggers events when certain things happen. This page describes those events. + +Plugins can listen for events using the :ref:`plugin_hook_track_event` plugin hook, which will be called with instances of the following classes (or additional classes registered by other plugins): + +.. automodule:: datasette.events + :members: + :exclude-members: Event diff --git a/docs/index.rst b/docs/index.rst index 66bbd5a4..ce1ed2eb 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -63,5 +63,6 @@ Contents plugin_hooks testing_plugins internals + events contributing changelog diff --git a/docs/internals.rst b/docs/internals.rst index d8f86251..bd7a70b5 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -593,6 +593,26 @@ Using either of these pattern will result in the in-memory database being served This removes a database that has been previously added. ``name=`` is the unique name of that database. +.. _datasette_track_event: + +await .track_event(event) +------------------------- + +``event`` - ``Event`` + An instance of a subclass of ``datasette.events.Event``. + +Plugins can call this to track events, using classes they have previously registered. See :ref:`plugin_event_tracking` for details. + +The event will then be passed to all plugins that have registered to receive events using the :ref:`plugin_hook_track_event` hook. + +Example usage, assuming the plugin has previously registered the ``BanUserEvent`` class: + +.. code-block:: python + + await datasette.track_event( + BanUserEvent(user={"id": 1, "username": "cleverbot"}) + ) + .. _datasette_sign: .sign(value, namespace="default") diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index da69c6c9..1a88cd31 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -1759,3 +1759,103 @@ top_canned_query(datasette, request, database, query_name) The name of the canned query. Returns HTML to be displayed at the top of the canned query page. + +.. _plugin_event_tracking: + +Event tracking +-------------- + +Datasette includes an internal mechanism for tracking analytical events. This can be used for analytics, but can also be used by plugins that want to listen out for when key events occur (such as a table being created) and take action in response. + +Plugins can register to receive events using the ``track_event`` plugin hook. + +They can also define their own events for other plugins to receive using the ``register_events`` plugin hook, combined with calls to the ``datasette.track_event(...)`` internal method. + +.. _plugin_hook_track_event: + +track_event(datasette, event) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +``datasette`` - :ref:`internals_datasette` + You can use this to access plugin configuration options via ``datasette.plugin_config(your_plugin_name)``. + +``event`` - ``Event`` + Information about the event, represented as an instance of a subclass of the ``Event`` base class. + +This hook will be called any time an event is tracked by code that calls the :ref:`datasette.track_event(...) ` internal method. + +The ``event`` object will always have the following properties: + +- ``name``: a string representing the name of the event, for example ``logout`` or ``create-table``. +- ``actor``: a dictionary representing the actor that triggered the event, or ``None`` if the event was not triggered by an actor. +- ``created``: a ``datatime.datetime`` object in the ``timezone.utc`` timezone representing the time the event object was created. + +Other properties on the event will be available depending on the type of event. You can also access those as a dictionary using ``event.properties()``. + +The events fired by Datasette core are :ref:`documented here `. + +This example plugin logs details of all events to standard error: + +.. code-block:: python + + from datasette import hookimpl + import json + import sys + + + @hookimpl + def track_event(event): + name = event.name + actor = event.actor + properties = event.properties() + msg = json.dumps( + { + "name": name, + "actor": actor, + "properties": properties, + } + ) + print(msg, file=sys.stderr, flush=True) + + +.. _plugin_hook_register_events: + +register_events(datasette) +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +``datasette`` - :ref:`internals_datasette` + You can use this to access plugin configuration options via ``datasette.plugin_config(your_plugin_name)``. + +This hook should return a list of ``Event`` subclasses that represent custom events that the plugin might send to the ``datasette.track_event()`` method. + +This example registers event subclasses for ``ban-user`` and ``unban-user`` events: + +.. code-block:: python + + from dataclasses import dataclass + from datasette import hookimpl, Event + + + @dataclass + class BanUserEvent(Event): + name = "ban-user" + user: dict + + + @dataclass + class UnbanUserEvent(Event): + name = "unban-user" + user: dict + + + @hookimpl + def register_events(): + return [BanUserEvent, UnbanUserEvent] + +The plugin can then call ``datasette.track_event(...)`` to send a ``ban-user`` event: + +.. code-block:: python + + await datasette.track_event( + BanUserEvent(user={"id": 1, "username": "cleverbot"}) + ) diff --git a/docs/plugins.rst b/docs/plugins.rst index 2ec03701..1a72af95 100644 --- a/docs/plugins.rst +++ b/docs/plugins.rst @@ -228,6 +228,15 @@ If you run ``datasette plugins --all`` it will include default plugins that ship "skip_csrf" ] }, + { + "name": "datasette.events", + "static": false, + "templates": false, + "version": null, + "hooks": [ + "register_events" + ] + }, { "name": "datasette.facets", "static": false, diff --git a/tests/conftest.py b/tests/conftest.py index 31336aea..445de057 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -1,4 +1,3 @@ -import asyncio import httpx import os import pathlib @@ -8,7 +7,8 @@ import re import subprocess import tempfile import time -import trustme +from dataclasses import dataclass, field +from datasette import Event, hookimpl try: @@ -164,6 +164,35 @@ def check_permission_actions_are_documented(): ) +class TrackEventPlugin: + __name__ = "TrackEventPlugin" + + @dataclass + class OneEvent(Event): + name = "one" + + extra: str + + @hookimpl + def register_events(self, datasette): + async def inner(): + return [self.OneEvent] + + return inner + + @hookimpl + def track_event(self, datasette, event): + datasette._tracked_events = getattr(datasette, "_tracked_events", []) + datasette._tracked_events.append(event) + + +@pytest.fixture(scope="session", autouse=True) +def install_event_tracking_plugin(): + from datasette.plugins import pm + + pm.register(TrackEventPlugin(), name="TrackEventPlugin") + + @pytest.fixture(scope="session") def ds_localhost_http_server(): ds_proc = subprocess.Popen( diff --git a/tests/test_api.py b/tests/test_api.py index 93ca43eb..177dc95c 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -786,7 +786,12 @@ async def test_threads_json(ds_client): @pytest.mark.asyncio async def test_plugins_json(ds_client): response = await ds_client.get("/-/plugins.json") - assert EXPECTED_PLUGINS == sorted(response.json(), key=lambda p: p["name"]) + # Filter out TrackEventPlugin + actual_plugins = sorted( + [p for p in response.json() if p["name"] != "TrackEventPlugin"], + key=lambda p: p["name"], + ) + assert EXPECTED_PLUGINS == actual_plugins # Try with ?all=1 response = await ds_client.get("/-/plugins.json?all=1") names = {p["name"] for p in response.json()} diff --git a/tests/test_api_write.py b/tests/test_api_write.py index 1787e06f..9caf9fdf 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -1,5 +1,6 @@ from datasette.app import Datasette from datasette.utils import sqlite3 +from .utils import last_event import pytest import time @@ -49,6 +50,14 @@ async def test_insert_row(ds_write): assert response.json()["rows"] == [expected_row] rows = (await ds_write.get_database("data").execute("select * from docs")).rows assert dict(rows[0]) == expected_row + # Analytics event + event = last_event(ds_write) + assert event.name == "insert-rows" + assert event.num_rows == 1 + assert event.database == "data" + assert event.table == "docs" + assert not event.ignore + assert not event.replace @pytest.mark.asyncio @@ -68,6 +77,16 @@ async def test_insert_rows(ds_write, return_rows): headers=_headers(token), ) assert response.status_code == 201 + + # Analytics event + event = last_event(ds_write) + assert event.name == "insert-rows" + assert event.num_rows == 20 + assert event.database == "data" + assert event.table == "docs" + assert not event.ignore + assert not event.replace + actual_rows = [ dict(r) for r in ( @@ -353,6 +372,16 @@ async def test_insert_ignore_replace( headers=_headers(token), ) assert response.status_code == 201 + + # Analytics event + event = last_event(ds_write) + assert event.name == "insert-rows" + assert event.num_rows == 1 + assert event.database == "data" + assert event.table == "docs" + assert event.ignore == ignore + assert event.replace == replace + actual_rows = [ dict(r) for r in ( @@ -427,6 +456,14 @@ async def test_upsert(ds_write, initial, input, expected_rows, should_return): ) assert response.status_code == 200 assert response.json()["ok"] is True + + # Analytics event + event = last_event(ds_write) + assert event.name == "upsert-rows" + assert event.num_rows == 1 + assert event.database == "data" + assert event.table == "upsert_test" + if should_return: # We only expect it to return rows corresponding to those we sent expected_returned_rows = expected_rows[: len(input["rows"])] @@ -530,6 +567,13 @@ async def test_delete_row(ds_write, table, row_for_create, pks, delete_path): headers=_headers(write_token(ds_write)), ) assert delete_response.status_code == 200 + + # Analytics event + event = last_event(ds_write) + assert event.name == "delete-row" + assert event.database == "data" + assert event.table == table + assert event.pks == str(delete_path).split(",") assert ( await ds_write.client.get( "/data.json?_shape=arrayfirst&sql=select+count(*)+from+{}".format(table) @@ -610,6 +654,13 @@ async def test_update_row(ds_write, input, expected_errors, use_return): for k, v in input.items(): assert returned_row[k] == v + # Analytics event + event = last_event(ds_write) + assert event.actor == {"id": "root", "token": "dstok"} + assert event.database == "data" + assert event.table == "docs" + assert event.pks == [str(pk)] + # And fetch the row to check it's updated response = await ds_write.client.get( "/data/docs/{}.json?_shape=array".format(pk), @@ -676,6 +727,13 @@ async def test_drop_table(ds_write, scenario): headers=_headers(token), ) assert response2.json() == {"ok": True} + # Check event + event = last_event(ds_write) + assert event.name == "drop-table" + assert event.actor == {"id": "root", "token": "dstok"} + assert event.table == "docs" + assert event.database == "data" + # Table should 404 assert (await ds_write.client.get("/data/docs")).status_code == 404 @@ -1096,6 +1154,12 @@ async def test_create_table(ds_write, input, expected_status, expected_response) assert response.status_code == expected_status data = response.json() assert data == expected_response + # create-table event + if expected_status == 201: + event = last_event(ds_write) + assert event.name == "create-table" + assert event.actor == {"id": "root", "token": "dstok"} + assert event.schema.startswith("CREATE TABLE ") @pytest.mark.asyncio diff --git a/tests/test_auth.py b/tests/test_auth.py index 33cf9b35..f2359df7 100644 --- a/tests/test_auth.py +++ b/tests/test_auth.py @@ -1,6 +1,6 @@ from bs4 import BeautifulSoup as Soup from .fixtures import app_client -from .utils import cookie_was_deleted +from .utils import cookie_was_deleted, last_event from click.testing import CliRunner from datasette.utils import baseconv from datasette.cli import cli @@ -19,6 +19,10 @@ async def test_auth_token(ds_client): assert {"a": {"id": "root"}} == ds_client.ds.unsign( response.cookies["ds_actor"], "actor" ) + # Should have recorded a login event + event = last_event(ds_client.ds) + assert event.name == "login" + assert event.actor == {"id": "root"} # Check that a second with same token fails assert ds_client.ds._root_token is None assert (await ds_client.get(path)).status_code == 403 @@ -57,7 +61,7 @@ async def test_actor_cookie_that_expires(ds_client, offset, expected): cookie = ds_client.ds.sign( {"a": {"id": "test"}, "e": baseconv.base62.encode(expires_at)}, "actor" ) - response = await ds_client.get("/", cookies={"ds_actor": cookie}) + await ds_client.get("/", cookies={"ds_actor": cookie}) assert ds_client.ds._last_request.scope["actor"] == expected @@ -86,6 +90,10 @@ def test_logout(app_client): csrftoken_from=True, cookies={"ds_actor": app_client.actor_cookie({"id": "test"})}, ) + # Should have recorded a logout event + event = last_event(app_client.ds) + assert event.name == "logout" + assert event.actor == {"id": "test"} # The ds_actor cookie should have been unset assert cookie_was_deleted(response4, "ds_actor") # Should also have set a message @@ -185,6 +193,13 @@ def test_auth_create_token( for error in errors: assert '

{}

'.format(error) in response2.text else: + # Check create-token event + event = last_event(app_client.ds) + assert event.name == "create-token" + assert event.expires_after == expected_duration + assert isinstance(event.restrict_all, list) + assert isinstance(event.restrict_database, dict) + assert isinstance(event.restrict_resource, dict) # Extract token from page token = response2.text.split('value="dstok_')[1].split('"')[0] details = app_client.ds.unsign(token, "token") diff --git a/tests/test_cli.py b/tests/test_cli.py index 080e8353..9cc18c6e 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -100,7 +100,11 @@ def test_spatialite_error_if_cannot_find_load_extension_spatialite(): def test_plugins_cli(app_client): runner = CliRunner() result1 = runner.invoke(cli, ["plugins"]) - assert json.loads(result1.output) == EXPECTED_PLUGINS + actual_plugins = sorted( + [p for p in json.loads(result1.output) if p["name"] != "TrackEventPlugin"], + key=lambda p: p["name"], + ) + assert actual_plugins == EXPECTED_PLUGINS # Try with --all result2 = runner.invoke(cli, ["plugins", "--all"]) names = [p["name"] for p in json.loads(result2.output)] diff --git a/tests/test_plugins.py b/tests/test_plugins.py index 5bfb6132..dad4f2ca 100644 --- a/tests/test_plugins.py +++ b/tests/test_plugins.py @@ -9,8 +9,9 @@ from .fixtures import ( TestClient as _TestClient, ) # noqa from click.testing import CliRunner +from dataclasses import dataclass from datasette.app import Datasette -from datasette import cli, hookimpl, Permission +from datasette import cli, hookimpl, Event, Permission from datasette.filters import FilterArguments from datasette.plugins import get_plugins, DEFAULT_PLUGINS, pm from datasette.utils.sqlite import sqlite3 @@ -18,6 +19,7 @@ from datasette.utils import CustomRow, StartupError from jinja2.environment import Template from jinja2 import ChoiceLoader, FileSystemLoader import base64 +import datetime import importlib import json import os @@ -1437,3 +1439,30 @@ async def test_hook_top_canned_query(ds_client): assert "Xtop_query:fixtures:from_hook:xyz" in response.text finally: pm.unregister(name="SlotPlugin") + + +@pytest.mark.asyncio +async def test_hook_track_event(): + datasette = Datasette(memory=True) + from .conftest import TrackEventPlugin + + await datasette.invoke_startup() + await datasette.track_event( + TrackEventPlugin.OneEvent(actor=None, extra="extra extra") + ) + assert len(datasette._tracked_events) == 1 + assert isinstance(datasette._tracked_events[0], TrackEventPlugin.OneEvent) + event = datasette._tracked_events[0] + assert event.name == "one" + assert event.properties() == {"extra": "extra extra"} + # Should have a recent created as well + created = event.created + assert isinstance(created, datetime.datetime) + assert created.tzinfo == datetime.timezone.utc + + +@pytest.mark.asyncio +async def test_hook_register_events(): + datasette = Datasette(memory=True) + await datasette.invoke_startup() + assert any(k.__name__ == "OneEvent" for k in datasette.event_classes) diff --git a/tests/utils.py b/tests/utils.py index 84d5b1df..9b31abde 100644 --- a/tests/utils.py +++ b/tests/utils.py @@ -1,6 +1,11 @@ from datasette.utils.sqlite import sqlite3 +def last_event(datasette): + events = getattr(datasette, "_tracked_events", []) + return events[-1] if events else None + + def assert_footer_links(soup): footer_links = soup.find("footer").findAll("a") assert 4 == len(footer_links) From 2e4a03b2c461ca20ff789146a006ddd126013ee7 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 31 Jan 2024 15:31:26 -0800 Subject: [PATCH 557/952] Run coverage on Python 3.12 - #2245 I hoped this would run slightly faster than 3.9 but there doesn't appear to be a performance improvement. --- .github/workflows/test-coverage.yml | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-) diff --git a/.github/workflows/test-coverage.yml b/.github/workflows/test-coverage.yml index bd720664..7a08e401 100644 --- a/.github/workflows/test-coverage.yml +++ b/.github/workflows/test-coverage.yml @@ -15,18 +15,13 @@ jobs: runs-on: ubuntu-latest steps: - name: Check out datasette - uses: actions/checkout@v2 + uses: actions/checkout@v4 - name: Set up Python - uses: actions/setup-python@v2 + uses: actions/setup-python@v5 with: - python-version: 3.9 - - uses: actions/cache@v2 - name: Configure pip caching - with: - path: ~/.cache/pip - key: ${{ runner.os }}-pip-${{ hashFiles('**/setup.py') }} - restore-keys: | - ${{ runner.os }}-pip- + python-version: '3.12' + cache: 'pip' + cache-dependency-path: '**/setup.py' - name: Install Python dependencies run: | python -m pip install --upgrade pip From bcf7ef963f6e1eb0a64b2a0bb4af0ae7a197d1d1 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 31 Jan 2024 19:45:05 -0800 Subject: [PATCH 558/952] YAML/JSON examples for allow blocks --- docs/authentication.rst | 270 ++++++++++++++++++++++++++++++++++------ 1 file changed, 231 insertions(+), 39 deletions(-) diff --git a/docs/authentication.rst b/docs/authentication.rst index a301113a..8758765d 100644 --- a/docs/authentication.rst +++ b/docs/authentication.rst @@ -80,13 +80,35 @@ The standard way to define permissions in Datasette is to use an ``"allow"`` blo The most basic form of allow block is this (`allow demo `__, `deny demo `__): -.. code-block:: json +.. [[[cog + from metadata_doc import config_example + import textwrap + config_example(cog, textwrap.dedent( + """ + allow: + id: root + """).strip(), + "YAML", "JSON" + ) +.. ]]] - { - "allow": { +.. tab:: YAML + + .. code-block:: yaml + + allow: + id: root + +.. tab:: JSON + + .. code-block:: json + + { + "allow": { "id": "root" + } } - } +.. [[[end]]] This will match any actors with an ``"id"`` property of ``"root"`` - for example, an actor that looks like this: @@ -99,29 +121,98 @@ This will match any actors with an ``"id"`` property of ``"root"`` - for example An allow block can specify "deny all" using ``false`` (`demo `__): -.. code-block:: json +.. [[[cog + from metadata_doc import config_example + import textwrap + config_example(cog, textwrap.dedent( + """ + allow: false + """).strip(), + "YAML", "JSON" + ) +.. ]]] - { - "allow": false - } +.. tab:: YAML + + .. code-block:: yaml + + allow: false + +.. tab:: JSON + + .. code-block:: json + + { + "allow": false + } +.. [[[end]]] An ``"allow"`` of ``true`` allows all access (`demo `__): -.. code-block:: json +.. [[[cog + from metadata_doc import config_example + import textwrap + config_example(cog, textwrap.dedent( + """ + allow: true + """).strip(), + "YAML", "JSON" + ) +.. ]]] - { - "allow": true - } +.. tab:: YAML + + .. code-block:: yaml + + allow: true + +.. tab:: JSON + + .. code-block:: json + + { + "allow": true + } +.. [[[end]]] Allow keys can provide a list of values. These will match any actor that has any of those values (`allow demo `__, `deny demo `__): -.. code-block:: json +.. [[[cog + from metadata_doc import config_example + import textwrap + config_example(cog, textwrap.dedent( + """ + allow: + id: + - simon + - cleopaws + """).strip(), + "YAML", "JSON" + ) +.. ]]] - { - "allow": { - "id": ["simon", "cleopaws"] +.. tab:: YAML + + .. code-block:: yaml + + allow: + id: + - simon + - cleopaws + +.. tab:: JSON + + .. code-block:: json + + { + "allow": { + "id": [ + "simon", + "cleopaws" + ] + } } - } +.. [[[end]]] This will match any actor with an ``"id"`` of either ``"simon"`` or ``"cleopaws"``. @@ -129,53 +220,154 @@ Actors can have properties that feature a list of values. These will be matched .. code-block:: json - { - "id": "simon", - "roles": ["staff", "developer"] - } + { + "id": "simon", + "roles": ["staff", "developer"] + } This allow block will provide access to any actor that has ``"developer"`` as one of their roles (`allow demo `__, `deny demo `__): -.. code-block:: json +.. [[[cog + from metadata_doc import config_example + import textwrap + config_example(cog, textwrap.dedent( + """ + allow: + roles: + - developer + """).strip(), + "YAML", "JSON" + ) +.. ]]] - { - "allow": { - "roles": ["developer"] +.. tab:: YAML + + .. code-block:: yaml + + allow: + roles: + - developer + +.. tab:: JSON + + .. code-block:: json + + { + "allow": { + "roles": [ + "developer" + ] + } } - } +.. [[[end]]] Note that "roles" is not a concept that is baked into Datasette - it's a convention that plugins can choose to implement and act on. If you want to provide access to any actor with a value for a specific key, use ``"*"``. For example, to match any logged-in user specify the following (`allow demo `__, `deny demo `__): -.. code-block:: json +.. [[[cog + from metadata_doc import config_example + import textwrap + config_example(cog, textwrap.dedent( + """ + allow: + id: "*" + """).strip(), + "YAML", "JSON" + ) +.. ]]] - { - "allow": { +.. tab:: YAML + + .. code-block:: yaml + + allow: + id: "*" + +.. tab:: JSON + + .. code-block:: json + + { + "allow": { "id": "*" + } } - } +.. [[[end]]] You can specify that only unauthenticated actors (from anynomous HTTP requests) should be allowed access using the special ``"unauthenticated": true`` key in an allow block (`allow demo `__, `deny demo `__): -.. code-block:: json +.. [[[cog + from metadata_doc import config_example + import textwrap + config_example(cog, textwrap.dedent( + """ + allow: + unauthenticated: true + """).strip(), + "YAML", "JSON" + ) +.. ]]] - { - "allow": { +.. tab:: YAML + + .. code-block:: yaml + + allow: + unauthenticated: true + +.. tab:: JSON + + .. code-block:: json + + { + "allow": { "unauthenticated": true + } } - } +.. [[[end]]] Allow keys act as an "or" mechanism. An actor will be able to execute the query if any of their JSON properties match any of the values in the corresponding lists in the ``allow`` block. The following block will allow users with either a ``role`` of ``"ops"`` OR users who have an ``id`` of ``"simon"`` or ``"cleopaws"``: -.. code-block:: json +.. [[[cog + from metadata_doc import config_example + import textwrap + config_example(cog, textwrap.dedent( + """ + allow: + id: + - simon + - cleopaws + role: ops + """).strip(), + "YAML", "JSON" + ) +.. ]]] - { - "allow": { - "id": ["simon", "cleopaws"], +.. tab:: YAML + + .. code-block:: yaml + + allow: + id: + - simon + - cleopaws + role: ops + +.. tab:: JSON + + .. code-block:: json + + { + "allow": { + "id": [ + "simon", + "cleopaws" + ], "role": "ops" + } } - } +.. [[[end]]] `Demo for cleopaws `__, `demo for ops role `__, `demo for an actor matching neither rule `__. From b466749e88b2ffbd925b6b3e777c8527ebc54e78 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 31 Jan 2024 20:03:19 -0800 Subject: [PATCH 559/952] Filled out docs/configuration.rst, closes #2246 --- docs/changelog.rst | 2 +- docs/configuration.rst | 297 ++++++++++++++++++++++++++++++++++++-- docs/custom_templates.rst | 156 +------------------- docs/metadata_doc.py | 8 +- docs/plugin_hooks.rst | 2 +- docs/plugins.rst | 2 +- 6 files changed, 294 insertions(+), 173 deletions(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index af3d2a0b..04ce9583 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -610,7 +610,7 @@ JavaScript modules To use modules, JavaScript needs to be included in `` + +You can also specify a SRI (subresource integrity hash) for these assets: + +.. [[[cog + config_example(cog, """ + extra_css_urls: + - url: https://simonwillison.net/static/css/all.bf8cd891642c.css + sri: sha384-9qIZekWUyjCyDIf2YK1FRoKiPJq4PHt6tp/ulnuuyRBvazd0hG7pWbE99zvwSznI + extra_js_urls: + - url: https://code.jquery.com/jquery-3.2.1.slim.min.js + sri: sha256-k2WSCIexGzOj3Euiig+TlR8gA0EmPjuc79OEeY5L45g= + """) +.. ]]] + +.. tab:: datasette.yaml + + .. code-block:: yaml + + + extra_css_urls: + - url: https://simonwillison.net/static/css/all.bf8cd891642c.css + sri: sha384-9qIZekWUyjCyDIf2YK1FRoKiPJq4PHt6tp/ulnuuyRBvazd0hG7pWbE99zvwSznI + extra_js_urls: + - url: https://code.jquery.com/jquery-3.2.1.slim.min.js + sri: sha256-k2WSCIexGzOj3Euiig+TlR8gA0EmPjuc79OEeY5L45g= + + +.. tab:: datasette.json + + .. code-block:: json + + { + "extra_css_urls": [ + { + "url": "https://simonwillison.net/static/css/all.bf8cd891642c.css", + "sri": "sha384-9qIZekWUyjCyDIf2YK1FRoKiPJq4PHt6tp/ulnuuyRBvazd0hG7pWbE99zvwSznI" + } + ], + "extra_js_urls": [ + { + "url": "https://code.jquery.com/jquery-3.2.1.slim.min.js", + "sri": "sha256-k2WSCIexGzOj3Euiig+TlR8gA0EmPjuc79OEeY5L45g=" + } + ] + } +.. [[[end]]] + +This will produce: + +.. code-block:: html + + + + +Modern browsers will only execute the stylesheet or JavaScript if the SRI hash +matches the content served. You can generate hashes using `www.srihash.org `_ + +Items in ``"extra_js_urls"`` can specify ``"module": true`` if they reference JavaScript that uses `JavaScript modules `__. This configuration: + +.. [[[cog + config_example(cog, """ + extra_js_urls: + - url: https://example.datasette.io/module.js + module: true + """) +.. ]]] + +.. tab:: datasette.yaml + + .. code-block:: yaml + + + extra_js_urls: + - url: https://example.datasette.io/module.js + module: true + + +.. tab:: datasette.json + + .. code-block:: json + + { + "extra_js_urls": [ + { + "url": "https://example.datasette.io/module.js", + "module": true + } + ] + } +.. [[[end]]] + +Will produce this HTML: + +.. code-block:: html + + + + + diff --git a/docs/custom_templates.rst b/docs/custom_templates.rst index d8e4ac96..534d8b33 100644 --- a/docs/custom_templates.rst +++ b/docs/custom_templates.rst @@ -5,159 +5,6 @@ Custom pages and templates Datasette provides a number of ways of customizing the way data is displayed. -.. _customization_css_and_javascript: - -Custom CSS and JavaScript -------------------------- - -When you launch Datasette, you can specify a custom configuration file like this:: - - datasette mydb.db --config datasette.yaml - -Your ``datasette.yaml`` file can include links that look like this: - -.. [[[cog - from metadata_doc import config_example - config_example(cog, """ - extra_css_urls: - - https://simonwillison.net/static/css/all.bf8cd891642c.css - extra_js_urls: - - https://code.jquery.com/jquery-3.2.1.slim.min.js - """) -.. ]]] - -.. tab:: datasette.yaml - - .. code-block:: yaml - - - extra_css_urls: - - https://simonwillison.net/static/css/all.bf8cd891642c.css - extra_js_urls: - - https://code.jquery.com/jquery-3.2.1.slim.min.js - - -.. tab:: datasette.json - - .. code-block:: json - - { - "extra_css_urls": [ - "https://simonwillison.net/static/css/all.bf8cd891642c.css" - ], - "extra_js_urls": [ - "https://code.jquery.com/jquery-3.2.1.slim.min.js" - ] - } -.. [[[end]]] - -The extra CSS and JavaScript files will be linked in the ```` of every page: - -.. code-block:: html - - - - -You can also specify a SRI (subresource integrity hash) for these assets: - -.. [[[cog - config_example(cog, """ - extra_css_urls: - - url: https://simonwillison.net/static/css/all.bf8cd891642c.css - sri: sha384-9qIZekWUyjCyDIf2YK1FRoKiPJq4PHt6tp/ulnuuyRBvazd0hG7pWbE99zvwSznI - extra_js_urls: - - url: https://code.jquery.com/jquery-3.2.1.slim.min.js - sri: sha256-k2WSCIexGzOj3Euiig+TlR8gA0EmPjuc79OEeY5L45g= - """) -.. ]]] - -.. tab:: datasette.yaml - - .. code-block:: yaml - - - extra_css_urls: - - url: https://simonwillison.net/static/css/all.bf8cd891642c.css - sri: sha384-9qIZekWUyjCyDIf2YK1FRoKiPJq4PHt6tp/ulnuuyRBvazd0hG7pWbE99zvwSznI - extra_js_urls: - - url: https://code.jquery.com/jquery-3.2.1.slim.min.js - sri: sha256-k2WSCIexGzOj3Euiig+TlR8gA0EmPjuc79OEeY5L45g= - - -.. tab:: datasette.json - - .. code-block:: json - - { - "extra_css_urls": [ - { - "url": "https://simonwillison.net/static/css/all.bf8cd891642c.css", - "sri": "sha384-9qIZekWUyjCyDIf2YK1FRoKiPJq4PHt6tp/ulnuuyRBvazd0hG7pWbE99zvwSznI" - } - ], - "extra_js_urls": [ - { - "url": "https://code.jquery.com/jquery-3.2.1.slim.min.js", - "sri": "sha256-k2WSCIexGzOj3Euiig+TlR8gA0EmPjuc79OEeY5L45g=" - } - ] - } -.. [[[end]]] - -This will produce: - -.. code-block:: html - - - - -Modern browsers will only execute the stylesheet or JavaScript if the SRI hash -matches the content served. You can generate hashes using `www.srihash.org `_ - -Items in ``"extra_js_urls"`` can specify ``"module": true`` if they reference JavaScript that uses `JavaScript modules `__. This configuration: - -.. [[[cog - config_example(cog, """ - extra_js_urls: - - url: https://example.datasette.io/module.js - module: true - """) -.. ]]] - -.. tab:: datasette.yaml - - .. code-block:: yaml - - - extra_js_urls: - - url: https://example.datasette.io/module.js - module: true - - -.. tab:: datasette.json - - .. code-block:: json - - { - "extra_js_urls": [ - { - "url": "https://example.datasette.io/module.js", - "module": true - } - ] - } -.. [[[end]]] - -Will produce this HTML: - -.. code-block:: html - - - CSS classes on the ~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -258,9 +105,10 @@ The following URLs will now serve the content from those CSS and JS files:: http://localhost:8001/assets/styles.css http://localhost:8001/assets/app.js -You can reference those files from ``datasette.yaml`` like so: +You can reference those files from ``datasette.yaml`` like this, see :ref:`custom CSS and JavaScript ` for more details: .. [[[cog + from metadata_doc import config_example config_example(cog, """ extra_css_urls: - /assets/styles.css diff --git a/docs/metadata_doc.py b/docs/metadata_doc.py index a8f13414..ad85bf52 100644 --- a/docs/metadata_doc.py +++ b/docs/metadata_doc.py @@ -24,17 +24,19 @@ def metadata_example(cog, data=None, yaml=None): cog.out("\n") -def config_example(cog, input): +def config_example( + cog, input, yaml_title="datasette.yaml", json_title="datasette.json" +): if type(input) is str: data = YAML().load(input) output_yaml = input else: data = input output_yaml = safe_dump(input, sort_keys=False) - cog.out("\n.. tab:: datasette.yaml\n\n") + cog.out("\n.. tab:: {}\n\n".format(yaml_title)) cog.out(" .. code-block:: yaml\n\n") cog.out(textwrap.indent(output_yaml, " ")) - cog.out("\n\n.. tab:: datasette.json\n\n") + cog.out("\n\n.. tab:: {}\n\n".format(json_title)) cog.out(" .. code-block:: json\n\n") cog.out(textwrap.indent(json.dumps(data, indent=2), " ")) cog.out("\n") diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index 1a88cd31..d9d135e5 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -270,7 +270,7 @@ you have one: Note that ``your-plugin`` here should be the hyphenated plugin name - the name that is displayed in the list on the ``/-/plugins`` debug page. -If your code uses `JavaScript modules `__ you should include the ``"module": True`` key. See :ref:`customization_css_and_javascript` for more details. +If your code uses `JavaScript modules `__ you should include the ``"module": True`` key. See :ref:`configuration_reference_css_js` for more details. .. code-block:: python diff --git a/docs/plugins.rst b/docs/plugins.rst index 1a72af95..03ddf8f0 100644 --- a/docs/plugins.rst +++ b/docs/plugins.rst @@ -328,7 +328,7 @@ To write that to a ``requirements.txt`` file, run this:: Plugin configuration -------------------- -Plugins can have their own configuration, embedded in a :ref:`configuration` file. Configuration options for plugins live within a ``"plugins"`` key in that file, which can be included at the root, database or table level. +Plugins can have their own configuration, embedded in a :ref:`configuration file `. Configuration options for plugins live within a ``"plugins"`` key in that file, which can be included at the root, database or table level. Here is an example of some plugin configuration for a specific table: From 4da581d09bbed2377682630c147e05d78c48f7e0 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 1 Feb 2024 14:40:49 -0800 Subject: [PATCH 560/952] Link to config reference --- docs/settings.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/settings.rst b/docs/settings.rst index 1d4baf90..d1553703 100644 --- a/docs/settings.rst +++ b/docs/settings.rst @@ -15,6 +15,8 @@ You can set multiple settings at once like this:: --setting sql_time_limit_ms 3500 \ --setting max_returned_rows 2000 +Settings can also be specified :ref:`in the database.yaml configuration file `. + .. _config_dir: Configuration directory mode From d4bc2b2dfc728017c8f669c1714f20b89655557c Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 1 Feb 2024 14:44:16 -0800 Subject: [PATCH 561/952] Remove fail_if_plugins_in_metadata, part of #2248 --- datasette/app.py | 8 ++------ datasette/cli.py | 3 +-- datasette/utils/__init__.py | 15 --------------- tests/test_plugins.py | 8 -------- 4 files changed, 3 insertions(+), 31 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 530f79bc..0143223a 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -77,7 +77,6 @@ from .utils import ( parse_metadata, resolve_env_secrets, resolve_routes, - fail_if_plugins_in_metadata, tilde_decode, to_css_class, urlsafe_components, @@ -336,16 +335,13 @@ class Datasette: ] if config_dir and metadata_files and not metadata: with metadata_files[0].open() as fp: - metadata = fail_if_plugins_in_metadata( - parse_metadata(fp.read()), metadata_files[0].name - ) + metadata = parse_metadata(fp.read()) if config_dir and config_files and not config: with config_files[0].open() as fp: config = parse_metadata(fp.read()) - self._metadata_local = fail_if_plugins_in_metadata(metadata or {}) - + self._metadata_local = metadata or {} self.sqlite_extensions = [] for extension in sqlite_extensions or []: # Resolve spatialite, if requested diff --git a/datasette/cli.py b/datasette/cli.py index 91f38f69..1a5a8af3 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -33,7 +33,6 @@ from .utils import ( initial_path_for_datasette, pairs_to_nested_config, temporary_docker_directory, - fail_if_plugins_in_metadata, value_as_boolean, SpatialiteNotFound, StaticMount, @@ -543,7 +542,7 @@ def serve( metadata_data = None if metadata: - metadata_data = fail_if_plugins_in_metadata(parse_metadata(metadata.read())) + metadata_data = parse_metadata(metadata.read()) config_data = None if config: diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index 196e1682..75f1c2f4 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -1272,21 +1272,6 @@ def pairs_to_nested_config(pairs: typing.List[typing.Tuple[str, typing.Any]]) -> return result -def fail_if_plugins_in_metadata(metadata: dict, filename=None): - """If plugin config is inside metadata, raise an Exception""" - if metadata is not None and metadata.get("plugins") is not None: - suggested_extension = ( - ".yaml" - if filename is not None - and (filename.endswith(".yaml") or filename.endswith(".yml")) - else ".json" - ) - raise Exception( - f'Datasette no longer accepts plugin configuration in --metadata. Move your "plugins" configuration blocks to a separate file - we suggest calling that datasette.{suggested_extension} - and start Datasette with datasette -c datasette.{suggested_extension}. See https://docs.datasette.io/en/latest/configuration.html for more details.' - ) - return metadata - - def make_slot_function(name, datasette, request, **kwargs): from datasette.plugins import pm diff --git a/tests/test_plugins.py b/tests/test_plugins.py index dad4f2ca..f26e3652 100644 --- a/tests/test_plugins.py +++ b/tests/test_plugins.py @@ -883,14 +883,6 @@ def test_hook_forbidden(restore_working_directory): ) -def test_plugin_config_in_metadata(): - with pytest.raises( - Exception, - match="Datasette no longer accepts plugin configuration in --metadata", - ): - Datasette(memory=True, metadata={"plugins": {}}) - - @pytest.mark.asyncio async def test_hook_handle_exception(ds_client): await ds_client.get("/trigger-error?x=123") From be4f02335fb35d40a763d07b2d4e880b90083e53 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 1 Feb 2024 15:33:33 -0800 Subject: [PATCH 562/952] Treat plugins in metadata as if they were in config, closes #2248 --- datasette/app.py | 6 +++++ datasette/utils/__init__.py | 40 +++++++++++++++++++++++++++++ tests/test_plugins.py | 51 +++++++++++++++++++++++++++++++++++++ 3 files changed, 97 insertions(+) diff --git a/datasette/app.py b/datasette/app.py index 0143223a..634283ff 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -74,6 +74,7 @@ from .utils import ( find_spatialite, format_bytes, module_from_path, + move_plugins, parse_metadata, resolve_env_secrets, resolve_routes, @@ -341,6 +342,11 @@ class Datasette: with config_files[0].open() as fp: config = parse_metadata(fp.read()) + # Move any "plugins" settings from metadata to config - updates them in place + metadata = metadata or {} + config = config or {} + move_plugins(metadata, config) + self._metadata_local = metadata or {} self.sqlite_extensions = [] for extension in sqlite_extensions or []: diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index 75f1c2f4..cc175b01 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -1287,3 +1287,43 @@ def make_slot_function(name, datasette, request, **kwargs): return markupsafe.Markup("".join(html_bits)) return inner + + +def move_plugins(source, destination): + """ + Move 'plugins' keys from source to destination dictionary. Creates hierarchy in destination if needed. + After moving, recursively remove any keys in the source that are left empty. + """ + + def recursive_move(src, dest, path=None): + if path is None: + path = [] + for key, value in list(src.items()): + new_path = path + [key] + if key == "plugins": + # Navigate and create the hierarchy in destination if needed + d = dest + for step in path: + d = d.setdefault(step, {}) + # Move the plugins + d[key] = value + # Remove the plugins from source + src.pop(key, None) + elif isinstance(value, dict): + recursive_move(value, dest, new_path) + # After moving, check if the current dictionary is empty and remove it if so + if not value: + src.pop(key, None) + + def prune_empty_dicts(d): + """ + Recursively prune all empty dictionaries from a given dictionary. + """ + for key, value in list(d.items()): + if isinstance(value, dict): + prune_empty_dicts(value) + if value == {}: + d.pop(key, None) + + recursive_move(source, destination) + prune_empty_dicts(source) diff --git a/tests/test_plugins.py b/tests/test_plugins.py index f26e3652..a53fc118 100644 --- a/tests/test_plugins.py +++ b/tests/test_plugins.py @@ -1458,3 +1458,54 @@ async def test_hook_register_events(): datasette = Datasette(memory=True) await datasette.invoke_startup() assert any(k.__name__ == "OneEvent" for k in datasette.event_classes) + + +@pytest.mark.parametrize( + "metadata,config,expected_metadata,expected_config", + ( + ( + # Instance level + {"plugins": {"datasette-foo": "bar"}}, + {}, + {}, + {"plugins": {"datasette-foo": "bar"}}, + ), + ( + # Database level + {"databases": {"foo": {"plugins": {"datasette-foo": "bar"}}}}, + {}, + {}, + {"databases": {"foo": {"plugins": {"datasette-foo": "bar"}}}}, + ), + ( + # Table level + { + "databases": { + "foo": {"tables": {"bar": {"plugins": {"datasette-foo": "bar"}}}} + } + }, + {}, + {}, + { + "databases": { + "foo": {"tables": {"bar": {"plugins": {"datasette-foo": "bar"}}}} + } + }, + ), + ( + # Keep other keys + {"plugins": {"datasette-foo": "bar"}, "other": "key"}, + {"original_config": "original"}, + {"other": "key"}, + {"original_config": "original", "plugins": {"datasette-foo": "bar"}}, + ), + ), +) +def test_metadata_plugin_config_treated_as_config( + metadata, config, expected_metadata, expected_config +): + ds = Datasette(metadata=metadata, config=config) + actual_metadata = ds.metadata() + assert "plugins" not in actual_metadata + assert actual_metadata == expected_metadata + assert ds.config == expected_config From 6ccef35cc92cc2357c2b2a9aa003b7334b2459eb Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 1 Feb 2024 15:42:45 -0800 Subject: [PATCH 563/952] More links between events documentation --- docs/events.rst | 2 +- docs/plugin_hooks.rst | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/events.rst b/docs/events.rst index f150ac02..b86c8025 100644 --- a/docs/events.rst +++ b/docs/events.rst @@ -7,7 +7,7 @@ Datasette includes a mechanism for tracking events that occur while the software The core Datasette application triggers events when certain things happen. This page describes those events. -Plugins can listen for events using the :ref:`plugin_hook_track_event` plugin hook, which will be called with instances of the following classes (or additional classes registered by other plugins): +Plugins can listen for events using the :ref:`plugin_hook_track_event` plugin hook, which will be called with instances of the following classes - or additional classes :ref:`registered by other plugins `. .. automodule:: datasette.events :members: diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index d9d135e5..16f5cebb 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -1769,7 +1769,7 @@ Datasette includes an internal mechanism for tracking analytical events. This ca Plugins can register to receive events using the ``track_event`` plugin hook. -They can also define their own events for other plugins to receive using the ``register_events`` plugin hook, combined with calls to the ``datasette.track_event(...)`` internal method. +They can also define their own events for other plugins to receive using the :ref:`register_events() plugin hook `, combined with calls to the :ref:`datasette.track_event() internal method `. .. _plugin_hook_track_event: @@ -1826,7 +1826,7 @@ register_events(datasette) ``datasette`` - :ref:`internals_datasette` You can use this to access plugin configuration options via ``datasette.plugin_config(your_plugin_name)``. -This hook should return a list of ``Event`` subclasses that represent custom events that the plugin might send to the ``datasette.track_event()`` method. +This hook should return a list of ``Event`` subclasses that represent custom events that the plugin might send to the :ref:`datasette.track_event() ` method. This example registers event subclasses for ``ban-user`` and ``unban-user`` events: From 4ea109ac4dd17392c85ca7d5934009f9a9488a9d Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 1 Feb 2024 15:47:41 -0800 Subject: [PATCH 564/952] Two spaces is aesthetically more pleasing here --- docs/settings.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/settings.rst b/docs/settings.rst index d1553703..c4b4ba82 100644 --- a/docs/settings.rst +++ b/docs/settings.rst @@ -11,9 +11,9 @@ Datasette supports a number of settings. These can be set using the ``--setting You can set multiple settings at once like this:: datasette mydatabase.db \ - --setting default_page_size 50 \ - --setting sql_time_limit_ms 3500 \ - --setting max_returned_rows 2000 + --setting default_page_size 50 \ + --setting sql_time_limit_ms 3500 \ + --setting max_returned_rows 2000 Settings can also be specified :ref:`in the database.yaml configuration file `. @@ -25,10 +25,10 @@ Configuration directory mode Normally you configure Datasette using command-line options. For a Datasette instance with custom templates, custom plugins, a static directory and several databases this can get quite verbose:: datasette one.db two.db \ - --metadata=metadata.json \ - --template-dir=templates/ \ - --plugins-dir=plugins \ - --static css:css + --metadata=metadata.json \ + --template-dir=templates/ \ + --plugins-dir=plugins \ + --static css:css As an alternative to this, you can run Datasette in *configuration directory* mode. Create a directory with the following structure:: From 5ea7098e4da5fe8576f6452dbfac86e6aedba397 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 4 Feb 2024 10:15:21 -0800 Subject: [PATCH 565/952] Fixed an unnecessary f-string --- demos/plugins/example_js_manager_plugins.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/demos/plugins/example_js_manager_plugins.py b/demos/plugins/example_js_manager_plugins.py index 7db45464..2705f2c5 100644 --- a/demos/plugins/example_js_manager_plugins.py +++ b/demos/plugins/example_js_manager_plugins.py @@ -16,6 +16,6 @@ def extra_js_urls(view_name): if view_name in PERMITTED_VIEWS: return [ { - "url": f"/static/table-example-plugins.js", + "url": "/static/table-example-plugins.js", } ] From 7219a56d1e8b5d076037aeeec2583ad4fc3cacb3 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 5 Feb 2024 10:34:10 -0800 Subject: [PATCH 566/952] 3 space indent, not 2 --- docs/configuration.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/configuration.rst b/docs/configuration.rst index a835ace9..79e2a1ca 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -145,8 +145,8 @@ Settings """ # inside datasette.yaml settings: - default_allow_sql: off - default_page_size: 50 + default_allow_sql: off + default_page_size: 50 """).strip() ) .. ]]] @@ -157,8 +157,8 @@ Settings # inside datasette.yaml settings: - default_allow_sql: off - default_page_size: 50 + default_allow_sql: off + default_page_size: 50 .. tab:: datasette.json From 503545b20363ac15d3664bec7e6c4522ff271668 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 5 Feb 2024 11:47:17 -0800 Subject: [PATCH 567/952] JavaScript plugins documentation, closes #2250 --- docs/index.rst | 1 + docs/javascript_plugins.rst | 159 ++++++++++++++++++++++++++++++++++++ 2 files changed, 160 insertions(+) create mode 100644 docs/javascript_plugins.rst diff --git a/docs/index.rst b/docs/index.rst index ce1ed2eb..e3036618 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -60,6 +60,7 @@ Contents custom_templates plugins writing_plugins + javascript_plugins plugin_hooks testing_plugins internals diff --git a/docs/javascript_plugins.rst b/docs/javascript_plugins.rst new file mode 100644 index 00000000..e7ee6817 --- /dev/null +++ b/docs/javascript_plugins.rst @@ -0,0 +1,159 @@ +.. _javascript_plugins: + +JavaScript plugins +================== + +Datasette can run custom JavaScript in several different ways: + +- Datasette plugins written in Python can use the :ref:`extra_js_urls() ` or :ref:`extra_body_script() ` plugin hooks to inject JavaScript into a page +- Datasette instances with :ref:`custom templates ` can include additional JavaScript in those templates +- The ``extra_js_urls`` key in ``datasette.yaml`` :ref:`can be used to include extra JavaScript ` + +There are no limitations on what this JavaScript can do. It is executed directly by the browser, so it can manipulate the DOM, fetch additional data and do anything else that JavaScript is capable of. + +.. warning:: + Custom JavaScript has security implications, especially for authenticated Datasette instances where the JavaScript might run in the context of the authenticated user. It's important to carefully review any JavaScript you run in your Datasette instance. + +.. _javascript_datasette_init: + +The datasette_init event +------------------------ + +Datasette emits a custom event called ``datasette_init`` when the page is loaded. This event is dispatched on the ``document`` object, and includes a ``detail`` object with a reference to the :ref:`datasetteManager ` object. + +Your JavaScript code can listen out for this event using ``document.addEventListener()`` like this: + +.. code-block:: javascript + + document.addEventListener("datasette_init", function (evt) { + const manager = evt.detail; + console.log("Datasette version:", manager.VERSION); + }); + +.. _javascript_datasette_manager: + +datasetteManager +---------------- + +The ``datasetteManager`` object + +``VERSION`` - string + The version of Datasette + +``plugins`` - ``Map()`` + A Map of currently loaded plugin names to plugin implementations + +``registerPlugin(name, implementation)`` + Call this to register a plugin, passing its name and implementation + +``selectors`` - object + An object providing named aliases to useful CSS selectors, :ref:`listed below ` + +.. _javascript_plugin_objects: + +JavaScript plugin objects +------------------------- + +JavaScript plugins are blocks of code that can be registered with Datasette using the ``registerPlugin()`` method on the :ref:`datasetteManager ` object. + +The ``implementation`` object passed to this method should include a ``version`` key defining the plugin version, and one or more of the following named functions providing the implementation of the plugin: + +.. _javascript_plugins_makeAboveTablePanelConfigs: + +makeAboveTablePanelConfigs() +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This method should return a JavaScript array of objects defining additional panels to be added to the top of the table page. Each object should have the following: + +``id`` - string + A unique string ID for the panel, for example ``map-panel`` +``label`` - string + A human-readable label for the panel +``render(node)`` - function + A function that will be called with a DOM node to render the panel into + +This example shows how a plugin might define a single panel: + +.. code-block:: javascript + + document.addEventListener('datasette_init', function(ev) { + ev.detail.registerPlugin('panel-plugin', { + version: 0.1, + makeAboveTablePanelConfigs: () => { + return [ + { + id: 'first-panel', + label: 'First panel', + render: node => { + node.innerHTML = '

My custom panel

This is a custom panel that I added using a JavaScript plugin

'; + } + } + ] + } + }); + }); + +When a page with a table loads, all registered plugins that implement ``makeAboveTablePanelConfigs()`` will be called and panels they return will be added to the top of the table page. + +.. _javascript_plugins_makeColumnActions: + +makeColumnActions(columnDetails) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This method, if present, will be called when Datasette is rendering the cog action menu icons that appear at the top of the table view. By default these include options like "Sort ascending/descending" and "Facet by this", but plugins can return additional actions to be included in this menu. + +The method will be called with a ``columnDetails`` object with the following keys: + +``columnName`` - string + The name of the column +``columnNotNull`` - boolean + True if the column is defined as NOT NULL +``columnType`` - string + The SQLite data type of the column +``isPk`` - boolean + True if the column is part of the primary key + +It should return a JavaScript array of objects each with a ``label`` and ``onClick`` property: + +``label`` - string + The human-readable label for the action +``onClick(evt)`` - function + A function that will be called when the action is clicked + +The ``evt`` object passed to the ``onClick`` is the standard browser event object that triggered the click. + +This example plugin adds two menu items - one to copy the column name to the clipboard and another that displays the column metadata in an ``alert()`` window: + +.. code-block:: javascript + + document.addEventListener('datasette_init', function(ev) { + ev.detail.registerPlugin('column-name-plugin', { + version: 0.1, + makeColumnActions: (columnDetails) => { + return [ + { + label: 'Copy column to clipboard', + onClick: async (evt) => { + await navigator.clipboard.writeText(columnDetails.columnName) + } + }, + { + label: 'Alert column metadata', + onClick: () => alert(JSON.stringify(columnDetails, null, 2)) + } + ]; + } + }); + }); + +.. _javascript_datasette_manager_selectors: + +Selectors +--------- + +These are available on the ``selectors`` property of the :ref:`javascript_datasette_manager` object. + +.. literalinclude:: ../datasette/static/datasette-manager.js + :language: javascript + :start-at: const DOM_SELECTORS = { + :end-at: }; From efc73575548b0bbaca4bdc8de40fc6939bb88428 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 5 Feb 2024 13:01:03 -0800 Subject: [PATCH 568/952] Remove Using YAML for metadata section No longer necessary now we show YAML and JSON examples everywhere. --- docs/changelog.rst | 2 +- docs/metadata.rst | 29 +---------------------------- 2 files changed, 2 insertions(+), 29 deletions(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index 04ce9583..f4b928e3 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -1190,7 +1190,7 @@ Also in this release: 0.40 (2020-04-21) ----------------- -* Datasette :ref:`metadata` can now be provided as a YAML file as an optional alternative to JSON. See :ref:`metadata_yaml`. (:issue:`713`) +* Datasette :ref:`metadata` can now be provided as a YAML file as an optional alternative to JSON. (:issue:`713`) * Removed support for ``datasette publish now``, which used the the now-retired Zeit Now v1 hosting platform. A new plugin, `datasette-publish-now `__, can be installed to publish data to Zeit (`now Vercel `__) Now v2. (:issue:`710`) * Fixed a bug where the ``extra_template_vars(request, view_name)`` plugin hook was not receiving the correct ``view_name``. (:issue:`716`) * Variables added to the template context by the ``extra_template_vars()`` plugin hook are now shown in the ``?_context=1`` debugging mode (see :ref:`setting_template_debug`). (:issue:`693`) diff --git a/docs/metadata.rst b/docs/metadata.rst index b4dc90f9..f3ca68ac 100644 --- a/docs/metadata.rst +++ b/docs/metadata.rst @@ -53,7 +53,7 @@ Your ``metadata.yaml`` file can look something like this: .. [[[end]]] -Choosing YAML over JSON adds support for multi-line strings and comments, see :ref:`metadata_yaml`. +Choosing YAML over JSON adds support for multi-line strings and comments. The above metadata will be displayed on the index page of your Datasette-powered site. The source and license information will also be included in the footer of @@ -664,33 +664,6 @@ SpatiaLite tables are automatically hidden) using ``"hidden": true``: } .. [[[end]]] -.. _metadata_yaml: - -Using YAML for metadata ------------------------ - -Datasette accepts YAML as an alternative to JSON for your metadata configuration file. -YAML is particularly useful for including multiline HTML and SQL strings, plus inline comments. - -Here's an example of a ``metadata.yml`` file, re-using an example from :ref:`canned_queries`. - -.. code-block:: yaml - - title: Demonstrating Metadata from YAML - description_html: |- -

This description includes a long HTML string

-
    -
  • YAML is better for embedding HTML strings than JSON!
    • -
- license: ODbL - license_url: https://opendatacommons.org/licenses/odbl/ - databases: - fixtures: - tables: - no_primary_key: - hidden: true - - .. _metadata_reference: Metadata reference From 85a1dfe6e07fcdd7ec8f83cb5b3a8f023659d064 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 5 Feb 2024 13:43:50 -0800 Subject: [PATCH 569/952] Configuration via the command-line section Closes #2252 Closes #2156 --- docs/configuration.rst | 78 ++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 76 insertions(+), 2 deletions(-) diff --git a/docs/configuration.rst b/docs/configuration.rst index 79e2a1ca..425024da 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -5,7 +5,7 @@ Configuration Datasette offers several ways to configure your Datasette instances: server settings, plugin configuration, authentication, and more. -Most configuration can be handled using a ``datasette.yaml`` configuration file, passed to datasette using the ``--config``/ ``-c`` flag: +Most configuration can be handled using a ``datasette.yaml`` configuration file, passed to datasette using the ``-c/--config`` flag: .. code-block:: bash @@ -13,12 +13,86 @@ Most configuration can be handled using a ``datasette.yaml`` configuration file, This file can also use JSON, as ``datasette.json``. YAML is recommended over JSON due to its support for comments and multi-line strings. +.. _configuration_cli: + +Configuration via the command-line +---------------------------------- + +The recommended way to configure Datasette is using a ``datasette.yaml`` file passed to ``-c/--config``. You can also pass individual settings to Datasette using the ``-s/--setting`` option, which can be used multiple times: + +.. code-block:: bash + + datasette mydatabase.db \ + --setting settings.default_page_size 50 \ + --setting settings.sql_time_limit_ms 3500 + +This option takes dotted-notation for the first argument and a value for the second argument. This means you can use it to set any configuration value that would be valid in a ``datasette.yaml`` file. + +It also works for plugin configuration, for example for `datasette-cluster-map `_: + +.. code-block:: bash + + datasette mydatabase.db \ + --setting plugins.datasette-cluster-map.latitude_column xlat \ + --setting plugins.datasette-cluster-map.longitude_column xlon + +If the value you provide is a valid JSON object or list it will be treated as nested data, allowing you to configure plugins that accept lists such as `datasette-proxy-url `_: + +.. code-block:: bash + + datasette mydatabase.db \ + -s plugins.datasette-proxy-url.paths '[{"path": "/proxy", "backend": "http://example.com/"}]' + +This is equivalent to a ``datasette.yaml`` file containing the following: + +.. [[[cog + from metadata_doc import config_example + import textwrap + config_example(cog, textwrap.dedent( + """ + plugins: + datasette-proxy-url: + paths: + - path: /proxy + backend: http://example.com/ + """).strip() + ) +.. ]]] + +.. tab:: datasette.yaml + + .. code-block:: yaml + + plugins: + datasette-proxy-url: + paths: + - path: /proxy + backend: http://example.com/ + +.. tab:: datasette.json + + .. code-block:: json + + { + "plugins": { + "datasette-proxy-url": { + "paths": [ + { + "path": "/proxy", + "backend": "http://example.com/" + } + ] + } + } + } +.. [[[end]]] + .. _configuration_reference: ``datasette.yaml`` reference ---------------------------- -This example shows many of the valid configuration options that can exist inside ``datasette.yaml``. +The following example shows some of the valid configuration options that can exist inside ``datasette.yaml``. .. [[[cog from metadata_doc import config_example From 1e901aa690211db36f02cc1b25246d0f56cd8720 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 6 Feb 2024 12:33:46 -0800 Subject: [PATCH 570/952] /-/config page, closes #2254 --- datasette/app.py | 14 ++++++----- datasette/utils/__init__.py | 28 +++++++++++++++++++++ datasette/views/special.py | 4 +-- docs/introspection.rst | 11 +++++++- tests/test_api.py | 50 ++++++++++++++++++++++++++----------- 5 files changed, 84 insertions(+), 23 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 634283ff..2e20d402 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -81,6 +81,7 @@ from .utils import ( tilde_decode, to_css_class, urlsafe_components, + redact_keys, row_sql_params_pks, ) from .utils.asgi import ( @@ -1374,6 +1375,11 @@ class Datasette: output.append(script) return output + def _config(self): + return redact_keys( + self.config, ("secret", "key", "password", "token", "hash", "dsn") + ) + def _routes(self): routes = [] @@ -1433,12 +1439,8 @@ class Datasette: r"/-/settings(\.(?Pjson))?$", ) add_route( - permanent_redirect("/-/settings.json"), - r"/-/config.json", - ) - add_route( - permanent_redirect("/-/settings"), - r"/-/config", + JsonDataView.as_view(self, "config.json", lambda: self._config()), + r"/-/config(\.(?Pjson))?$", ) add_route( JsonDataView.as_view(self, "threads.json", self._threads), diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index cc175b01..4c940645 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -17,6 +17,7 @@ import time import types import secrets import shutil +from typing import Iterable import urllib import yaml from .shutil_backport import copytree @@ -1327,3 +1328,30 @@ def move_plugins(source, destination): recursive_move(source, destination) prune_empty_dicts(source) + + +def redact_keys(original: dict, key_patterns: Iterable) -> dict: + """ + Recursively redact sensitive keys in a dictionary based on given patterns + + :param original: The original dictionary + :param key_patterns: A list of substring patterns to redact + :return: A copy of the original dictionary with sensitive values redacted + """ + + def redact(data): + if isinstance(data, dict): + return { + k: ( + redact(v) + if not any(pattern in k for pattern in key_patterns) + else "***" + ) + for k, v in data.items() + } + elif isinstance(data, list): + return [redact(item) for item in data] + else: + return data + + return redact(original) diff --git a/datasette/views/special.py b/datasette/views/special.py index 4088a1f9..296652d0 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -42,7 +42,7 @@ class JsonDataView(BaseView): if self.ds.cors: add_cors_headers(headers) return Response( - json.dumps(data), + json.dumps(data, default=repr), content_type="application/json; charset=utf-8", headers=headers, ) @@ -53,7 +53,7 @@ class JsonDataView(BaseView): request=request, context={ "filename": self.filename, - "data_json": json.dumps(data, indent=4), + "data_json": json.dumps(data, indent=4, default=repr), }, ) diff --git a/docs/introspection.rst b/docs/introspection.rst index e08ca911..b62197ea 100644 --- a/docs/introspection.rst +++ b/docs/introspection.rst @@ -87,7 +87,7 @@ Shows a list of currently installed plugins and their versions. `Plugins example Add ``?all=1`` to include details of the default plugins baked into Datasette. -.. _JsonDataView_config: +.. _JsonDataView_settings: /-/settings ----------- @@ -105,6 +105,15 @@ Shows the :ref:`settings` for this instance of Datasette. `Settings example ` for this instance of Datasette. This is generally the contents of the :ref:`datasette.yaml or datasette.json ` file, which can include plugin configuration as well. + +Any keys that include the one of the following substrings in their names will be returned as redacted ``***`` output, to help avoid accidentally leaking private configuration information: ``secret``, ``key``, ``password``, ``token``, ``hash``, ``dsn``. + .. _JsonDataView_databases: /-/databases diff --git a/tests/test_api.py b/tests/test_api.py index 177dc95c..0a1f3725 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -846,20 +846,6 @@ async def test_settings_json(ds_client): } -@pytest.mark.asyncio -@pytest.mark.parametrize( - "path,expected_redirect", - ( - ("/-/config.json", "/-/settings.json"), - ("/-/config", "/-/settings"), - ), -) -async def test_config_redirects_to_settings(ds_client, path, expected_redirect): - response = await ds_client.get(path) - assert response.status_code == 301 - assert response.headers["Location"] == expected_redirect - - test_json_columns_default_expected = [ {"intval": 1, "strval": "s", "floatval": 0.5, "jsonval": '{"foo": "bar"}'} ] @@ -1039,3 +1025,39 @@ async def test_tilde_encoded_database_names(db_name): # And the JSON for that database response2 = await ds.client.get(path + ".json") assert response2.status_code == 200 + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "config,expected", + ( + ({}, {}), + ({"plugins": {"datasette-foo": "bar"}}, {"plugins": {"datasette-foo": "bar"}}), + # Test redaction + ( + { + "plugins": { + "datasette-auth": {"secret_key": "key"}, + "datasette-foo": "bar", + "datasette-auth2": {"password": "password"}, + "datasette-sentry": { + "dsn": "sentry:///foo", + }, + } + }, + { + "plugins": { + "datasette-auth": {"secret_key": "***"}, + "datasette-foo": "bar", + "datasette-auth2": {"password": "***"}, + "datasette-sentry": {"dsn": "***"}, + } + }, + ), + ), +) +async def test_config_json(config, expected): + "/-/config.json should return redacted configuration" + ds = Datasette(config=config) + response = await ds.client.get("/-/config.json") + assert response.json() == expected From 5a63ecc5577d070f28bf5daa34aaaf3dadfd2e4d Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 6 Feb 2024 15:03:19 -0800 Subject: [PATCH 571/952] Rename metadata= to table_config= in facet code, refs #2247 --- datasette/facets.py | 41 ++++++++++++++++++++-------------------- datasette/views/table.py | 2 +- tests/test_facets.py | 4 ++-- 3 files changed, 23 insertions(+), 24 deletions(-) diff --git a/datasette/facets.py b/datasette/facets.py index b23615fe..f1cfc68f 100644 --- a/datasette/facets.py +++ b/datasette/facets.py @@ -11,8 +11,8 @@ from datasette.utils import ( ) -def load_facet_configs(request, table_metadata): - # Given a request and the metadata configuration for a table, return +def load_facet_configs(request, table_config): + # Given a request and the configuration for a table, return # a dictionary of selected facets, their lists of configs and for each # config whether it came from the request or the metadata. # @@ -20,21 +20,21 @@ def load_facet_configs(request, table_metadata): # {"source": "metadata", "config": config1}, # {"source": "request", "config": config2}]} facet_configs = {} - table_metadata = table_metadata or {} - metadata_facets = table_metadata.get("facets", []) - for metadata_config in metadata_facets: - if isinstance(metadata_config, str): + table_config = table_config or {} + table_facet_configs = table_config.get("facets", []) + for facet_config in table_facet_configs: + if isinstance(facet_config, str): type = "column" - metadata_config = {"simple": metadata_config} + facet_config = {"simple": facet_config} else: assert ( - len(metadata_config.values()) == 1 + len(facet_config.values()) == 1 ), "Metadata config dicts should be {type: config}" - type, metadata_config = list(metadata_config.items())[0] - if isinstance(metadata_config, str): - metadata_config = {"simple": metadata_config} + type, facet_config = list(facet_config.items())[0] + if isinstance(facet_config, str): + facet_config = {"simple": facet_config} facet_configs.setdefault(type, []).append( - {"source": "metadata", "config": metadata_config} + {"source": "metadata", "config": facet_config} ) qs_pairs = urllib.parse.parse_qs(request.query_string, keep_blank_values=True) for key, values in qs_pairs.items(): @@ -45,13 +45,12 @@ def load_facet_configs(request, table_metadata): elif key.startswith("_facet_"): type = key[len("_facet_") :] for value in values: - # The value is the config - either JSON or not - if value.startswith("{"): - config = json.loads(value) - else: - config = {"simple": value} + # The value is the facet_config - either JSON or not + facet_config = ( + json.loads(value) if value.startswith("{") else {"simple": value} + ) facet_configs.setdefault(type, []).append( - {"source": "request", "config": config} + {"source": "request", "config": facet_config} ) return facet_configs @@ -75,7 +74,7 @@ class Facet: sql=None, table=None, params=None, - metadata=None, + table_config=None, row_count=None, ): assert table or sql, "Must provide either table= or sql=" @@ -86,12 +85,12 @@ class Facet: self.table = table self.sql = sql or f"select * from [{table}]" self.params = params or [] - self.metadata = metadata + self.table_config = table_config # row_count can be None, in which case we calculate it ourselves: self.row_count = row_count def get_configs(self): - configs = load_facet_configs(self.request, self.metadata) + configs = load_facet_configs(self.request, self.table_config) return configs.get(self.type) or [] def get_querystring_pairs(self): diff --git a/datasette/views/table.py b/datasette/views/table.py index 3b812c01..22722847 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1275,7 +1275,7 @@ async def table_view_data( sql=sql_no_order_no_limit, params=params, table=table_name, - metadata=table_metadata, + table_config=table_metadata, row_count=extra_count, ) ) diff --git a/tests/test_facets.py b/tests/test_facets.py index 85c8f85b..76344108 100644 --- a/tests/test_facets.py +++ b/tests/test_facets.py @@ -82,7 +82,7 @@ async def test_column_facet_suggest_skip_if_enabled_by_metadata(ds_client): database="fixtures", sql="select * from facetable", table="facetable", - metadata={"facets": ["_city_id"]}, + table_config={"facets": ["_city_id"]}, ) suggestions = [s["name"] for s in await facet.suggest()] assert [ @@ -278,7 +278,7 @@ async def test_column_facet_from_metadata_cannot_be_hidden(ds_client): database="fixtures", sql="select * from facetable", table="facetable", - metadata={"facets": ["_city_id"]}, + table_config={"facets": ["_city_id"]}, ) buckets, timed_out = await facet.facet_results() assert [] == timed_out From 5d21057cf1e4171bc2741d3d8d9da83aee1165b5 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 6 Feb 2024 15:22:03 -0800 Subject: [PATCH 572/952] /-/config example, refs #2254 --- docs/introspection.rst | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/introspection.rst b/docs/introspection.rst index b62197ea..ff78ec78 100644 --- a/docs/introspection.rst +++ b/docs/introspection.rst @@ -110,7 +110,17 @@ Shows the :ref:`settings` for this instance of Datasette. `Settings example ` for this instance of Datasette. This is generally the contents of the :ref:`datasette.yaml or datasette.json ` file, which can include plugin configuration as well. +Shows the :ref:`configuration ` for this instance of Datasette. This is generally the contents of the :ref:`datasette.yaml or datasette.json ` file, which can include plugin configuration as well. `Config example `_: + +.. code-block:: json + + { + "settings": { + "template_debug": true, + "trace_debug": true, + "force_https_urls": true + } + } Any keys that include the one of the following substrings in their names will be returned as redacted ``***`` output, to help avoid accidentally leaking private configuration information: ``secret``, ``key``, ``password``, ``token``, ``hash``, ``dsn``. From 69c6e953231078ab18ba0807e5fe2a4e20e84093 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 6 Feb 2024 17:27:20 -0800 Subject: [PATCH 573/952] Fixed a bunch of unused imports spotted with ruff --- datasette/cli.py | 1 - datasette/forbidden.py | 1 - datasette/handle_exception.py | 4 +--- datasette/permissions.py | 2 +- datasette/url_builder.py | 2 +- datasette/views/base.py | 1 - datasette/views/index.py | 3 --- ruff.toml | 1 + tests/conftest.py | 2 +- tests/test_black.py | 2 -- tests/test_cli.py | 2 -- tests/test_cli_serve_get.py | 2 +- tests/test_config_dir.py | 2 -- tests/test_internals_datasette.py | 1 - tests/test_plugins.py | 7 ++----- tests/test_table_html.py | 2 +- 16 files changed, 9 insertions(+), 26 deletions(-) create mode 100644 ruff.toml diff --git a/datasette/cli.py b/datasette/cli.py index 1a5a8af3..0c8a8541 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -15,7 +15,6 @@ import sys import textwrap import webbrowser from .app import ( - OBSOLETE_SETTINGS, Datasette, DEFAULT_SETTINGS, SETTINGS, diff --git a/datasette/forbidden.py b/datasette/forbidden.py index 156a44d4..41c48396 100644 --- a/datasette/forbidden.py +++ b/datasette/forbidden.py @@ -1,4 +1,3 @@ -from os import stat from datasette import hookimpl, Response diff --git a/datasette/handle_exception.py b/datasette/handle_exception.py index bef6b4ee..1a0ac979 100644 --- a/datasette/handle_exception.py +++ b/datasette/handle_exception.py @@ -1,14 +1,12 @@ from datasette import hookimpl, Response -from .utils import await_me_maybe, add_cors_headers +from .utils import add_cors_headers from .utils.asgi import ( Base400, - Forbidden, ) from .views.base import DatasetteError from markupsafe import Markup import pdb import traceback -from .plugins import pm try: import rich diff --git a/datasette/permissions.py b/datasette/permissions.py index 152f1721..bd42158e 100644 --- a/datasette/permissions.py +++ b/datasette/permissions.py @@ -1,4 +1,4 @@ -from dataclasses import dataclass, fields +from dataclasses import dataclass from typing import Optional diff --git a/datasette/url_builder.py b/datasette/url_builder.py index 574bf3c1..9c6bbde0 100644 --- a/datasette/url_builder.py +++ b/datasette/url_builder.py @@ -1,4 +1,4 @@ -from .utils import tilde_encode, path_with_format, HASH_LENGTH, PrefixedUrlString +from .utils import tilde_encode, path_with_format, PrefixedUrlString import urllib diff --git a/datasette/views/base.py b/datasette/views/base.py index bdc1e9cf..9d7a854c 100644 --- a/datasette/views/base.py +++ b/datasette/views/base.py @@ -10,7 +10,6 @@ from markupsafe import escape import pint -from datasette import __version__ from datasette.database import QueryInterrupted from datasette.utils.asgi import Request from datasette.utils import ( diff --git a/datasette/views/index.py b/datasette/views/index.py index 595cf234..2cb18b1c 100644 --- a/datasette/views/index.py +++ b/datasette/views/index.py @@ -1,12 +1,9 @@ import json -from datasette.plugins import pm from datasette.utils import add_cors_headers, make_slot_function, CustomJSONEncoder from datasette.utils.asgi import Response from datasette.version import __version__ -from markupsafe import Markup - from .base import BaseView diff --git a/ruff.toml b/ruff.toml new file mode 100644 index 00000000..0deb884c --- /dev/null +++ b/ruff.toml @@ -0,0 +1 @@ +line-length = 160 \ No newline at end of file diff --git a/tests/conftest.py b/tests/conftest.py index 445de057..168194d2 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -7,7 +7,7 @@ import re import subprocess import tempfile import time -from dataclasses import dataclass, field +from dataclasses import dataclass from datasette import Event, hookimpl diff --git a/tests/test_black.py b/tests/test_black.py index d09b2514..ccf51171 100644 --- a/tests/test_black.py +++ b/tests/test_black.py @@ -1,8 +1,6 @@ import black from click.testing import CliRunner from pathlib import Path -import pytest -import sys code_root = Path(__file__).parent.parent diff --git a/tests/test_cli.py b/tests/test_cli.py index 9cc18c6e..bda17eed 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -4,7 +4,6 @@ from .fixtures import ( TestClient as _TestClient, EXPECTED_PLUGINS, ) -import asyncio from datasette.app import SETTINGS from datasette.plugins import DEFAULT_PLUGINS from datasette.cli import cli, serve @@ -19,7 +18,6 @@ import pytest import sys import textwrap from unittest import mock -import urllib def test_inspect_cli(app_client): diff --git a/tests/test_cli_serve_get.py b/tests/test_cli_serve_get.py index ff2429c6..1088d906 100644 --- a/tests/test_cli_serve_get.py +++ b/tests/test_cli_serve_get.py @@ -1,4 +1,4 @@ -from datasette.cli import cli, serve +from datasette.cli import cli from datasette.plugins import pm from click.testing import CliRunner import textwrap diff --git a/tests/test_config_dir.py b/tests/test_config_dir.py index 748412c3..66114a27 100644 --- a/tests/test_config_dir.py +++ b/tests/test_config_dir.py @@ -3,11 +3,9 @@ import pathlib import pytest from datasette.app import Datasette -from datasette.cli import cli from datasette.utils.sqlite import sqlite3 from datasette.utils import StartupError from .fixtures import TestClient as _TestClient -from click.testing import CliRunner PLUGIN = """ from datasette import hookimpl diff --git a/tests/test_internals_datasette.py b/tests/test_internals_datasette.py index c30bb748..2614e02e 100644 --- a/tests/test_internals_datasette.py +++ b/tests/test_internals_datasette.py @@ -7,7 +7,6 @@ from datasette import Forbidden, Context from datasette.app import Datasette, Database from itsdangerous import BadSignature import pytest -from typing import Optional @pytest.fixture diff --git a/tests/test_plugins.py b/tests/test_plugins.py index a53fc118..c02c94cc 100644 --- a/tests/test_plugins.py +++ b/tests/test_plugins.py @@ -1,6 +1,5 @@ from bs4 import BeautifulSoup as Soup from .fixtures import ( - app_client, app_client, make_app_client, TABLES, @@ -9,14 +8,12 @@ from .fixtures import ( TestClient as _TestClient, ) # noqa from click.testing import CliRunner -from dataclasses import dataclass from datasette.app import Datasette -from datasette import cli, hookimpl, Event, Permission +from datasette import cli, hookimpl, Permission from datasette.filters import FilterArguments from datasette.plugins import get_plugins, DEFAULT_PLUGINS, pm from datasette.utils.sqlite import sqlite3 -from datasette.utils import CustomRow, StartupError -from jinja2.environment import Template +from datasette.utils import StartupError from jinja2 import ChoiceLoader, FileSystemLoader import base64 import datetime diff --git a/tests/test_table_html.py b/tests/test_table_html.py index 0604d34c..2a658663 100644 --- a/tests/test_table_html.py +++ b/tests/test_table_html.py @@ -1,4 +1,4 @@ -from datasette.app import Datasette, Database +from datasette.app import Datasette from bs4 import BeautifulSoup as Soup from .fixtures import ( # noqa app_client, From f0491038523e000b97a18d2e9a23faee62208083 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 6 Feb 2024 17:33:18 -0800 Subject: [PATCH 574/952] datasette.table_metadata() is now await datasette.table_config(), refs #2247 --- datasette/app.py | 2 +- datasette/database.py | 2 +- datasette/filters.py | 2 +- datasette/views/row.py | 2 +- datasette/views/table.py | 6 +++--- 5 files changed, 7 insertions(+), 7 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 2e20d402..b0b8f041 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -1202,7 +1202,7 @@ class Datasette: def _actor(self, request): return {"actor": request.actor} - def table_metadata(self, database, table): + async def table_config(self, database, table): """Fetch table-specific metadata.""" return ( (self.metadata("databases") or {}) diff --git a/datasette/database.py b/datasette/database.py index f2c980d7..1c1b3e1b 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -418,7 +418,7 @@ class Database: return await self.execute_fn(lambda conn: detect_fts(conn, table)) async def label_column_for_table(self, table): - explicit_label_column = self.ds.table_metadata(self.name, table).get( + explicit_label_column = (await self.ds.table_config(self.name, table)).get( "label_column" ) if explicit_label_column: diff --git a/datasette/filters.py b/datasette/filters.py index 73eea857..4d9580d8 100644 --- a/datasette/filters.py +++ b/datasette/filters.py @@ -50,7 +50,7 @@ def search_filters(request, database, table, datasette): extra_context = {} # Figure out which fts_table to use - table_metadata = datasette.table_metadata(database, table) + table_metadata = await datasette.table_config(database, table) db = datasette.get_database(database) fts_table = request.args.get("_fts_table") fts_table = fts_table or table_metadata.get("fts_table") diff --git a/datasette/views/row.py b/datasette/views/row.py index 7b646641..7b43b893 100644 --- a/datasette/views/row.py +++ b/datasette/views/row.py @@ -89,7 +89,7 @@ class RowView(DataView): "columns": columns, "primary_keys": resolved.pks, "primary_key_values": pk_values, - "units": self.ds.table_metadata(database, table).get("units", {}), + "units": (await self.ds.table_config(database, table)).get("units", {}), } if "foreign_key_tables" in (request.args.get("_extras") or "").split(","): diff --git a/datasette/views/table.py b/datasette/views/table.py index 22722847..2b5b7c24 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -142,7 +142,7 @@ async def display_columns_and_rows( """Returns columns, rows for specified table - including fancy foreign key treatment""" sortable_columns = sortable_columns or set() db = datasette.databases[database_name] - table_metadata = datasette.table_metadata(database_name, table_name) + table_metadata = await datasette.table_config(database_name, table_name) column_descriptions = table_metadata.get("columns") or {} column_details = { col.name: col for col in await db.table_column_details(table_name) @@ -663,7 +663,7 @@ async def _columns_to_select(table_columns, pks, request): async def _sortable_columns_for_table(datasette, database_name, table_name, use_rowid): db = datasette.databases[database_name] - table_metadata = datasette.table_metadata(database_name, table_name) + table_metadata = await datasette.table_config(database_name, table_name) if "sortable_columns" in table_metadata: sortable_columns = set(table_metadata["sortable_columns"]) else: @@ -962,7 +962,7 @@ async def table_view_data( nocount = True nofacet = True - table_metadata = datasette.table_metadata(database_name, table_name) + table_metadata = await datasette.table_config(database_name, table_name) units = table_metadata.get("units", {}) # Arguments that start with _ and don't contain a __ are From 52a1dac5d2bac7e106c8d6ce8e0c6f1dc0141a7e Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 6 Feb 2024 21:00:55 -0800 Subject: [PATCH 575/952] Test proving $env works for datasette.yml, closes #2255 --- tests/test_plugins.py | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/tests/test_plugins.py b/tests/test_plugins.py index c02c94cc..40d01c71 100644 --- a/tests/test_plugins.py +++ b/tests/test_plugins.py @@ -231,10 +231,18 @@ async def test_plugin_config(ds_client): @pytest.mark.asyncio -async def test_plugin_config_env(ds_client): - os.environ["FOO_ENV"] = "FROM_ENVIRONMENT" - assert {"foo": "FROM_ENVIRONMENT"} == ds_client.ds.plugin_config("env-plugin") - del os.environ["FOO_ENV"] +async def test_plugin_config_env(ds_client, monkeypatch): + monkeypatch.setenv("FOO_ENV", "FROM_ENVIRONMENT") + assert ds_client.ds.plugin_config("env-plugin") == {"foo": "FROM_ENVIRONMENT"} + + +@pytest.mark.asyncio +async def test_plugin_config_env_from_config(monkeypatch): + monkeypatch.setenv("FOO_ENV", "FROM_ENVIRONMENT_2") + datasette = Datasette( + config={"plugins": {"env-plugin": {"setting": {"$env": "FOO_ENV"}}}} + ) + assert datasette.plugin_config("env-plugin") == {"setting": "FROM_ENVIRONMENT_2"} @pytest.mark.asyncio From 60c6692f6802dfae6a433f648f287be30ef52325 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 6 Feb 2024 21:57:09 -0800 Subject: [PATCH 576/952] table_config instead of table_metadata (#2257) Table configuration that was incorrectly placed in metadata is now treated as if it was in config. New await datasette.table_config() method. Closes #2247 --- datasette/app.py | 12 +++-- datasette/database.py | 10 ++-- datasette/utils/__init__.py | 69 +++++++++++++++++++----- datasette/views/table.py | 11 ++-- tests/test_api.py | 101 +++++++++++++++++++++++++++++++++++- tests/test_html.py | 2 +- 6 files changed, 175 insertions(+), 30 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index b0b8f041..373b3e95 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -75,6 +75,7 @@ from .utils import ( format_bytes, module_from_path, move_plugins, + move_table_config, parse_metadata, resolve_env_secrets, resolve_routes, @@ -346,7 +347,9 @@ class Datasette: # Move any "plugins" settings from metadata to config - updates them in place metadata = metadata or {} config = config or {} - move_plugins(metadata, config) + metadata, config = move_plugins(metadata, config) + # Now migrate any known table configuration settings over as well + metadata, config = move_table_config(metadata, config) self._metadata_local = metadata or {} self.sqlite_extensions = [] @@ -1202,10 +1205,11 @@ class Datasette: def _actor(self, request): return {"actor": request.actor} - async def table_config(self, database, table): - """Fetch table-specific metadata.""" + async def table_config(self, database: str, table: str) -> dict: + """Return dictionary of configuration for specified table""" return ( - (self.metadata("databases") or {}) + (self.config or {}) + .get("databases", {}) .get(database, {}) .get("tables", {}) .get(table, {}) diff --git a/datasette/database.py b/datasette/database.py index 1c1b3e1b..fba81496 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -487,13 +487,11 @@ class Database: ) ).rows ] - # Add any from metadata.json - db_metadata = self.ds.metadata(database=self.name) - if "tables" in db_metadata: + # Add any tables marked as hidden in config + db_config = self.ds.config.get("databases", {}).get(self.name, {}) + if "tables" in db_config: hidden_tables += [ - t - for t in db_metadata["tables"] - if db_metadata["tables"][t].get("hidden") + t for t in db_config["tables"] if db_config["tables"][t].get("hidden") ] # Also mark as hidden any tables which start with the name of a hidden table # e.g. "searchable_fts" implies "searchable_fts_content" should be hidden diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index 4c940645..fcaebe3f 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -2,6 +2,7 @@ import asyncio from contextlib import contextmanager import click from collections import OrderedDict, namedtuple, Counter +import copy import base64 import hashlib import inspect @@ -17,7 +18,7 @@ import time import types import secrets import shutil -from typing import Iterable +from typing import Iterable, Tuple import urllib import yaml from .shutil_backport import copytree @@ -1290,11 +1291,24 @@ def make_slot_function(name, datasette, request, **kwargs): return inner -def move_plugins(source, destination): +def prune_empty_dicts(d: dict): + """ + Recursively prune all empty dictionaries from a given dictionary. + """ + for key, value in list(d.items()): + if isinstance(value, dict): + prune_empty_dicts(value) + if value == {}: + d.pop(key, None) + + +def move_plugins(source: dict, destination: dict) -> Tuple[dict, dict]: """ Move 'plugins' keys from source to destination dictionary. Creates hierarchy in destination if needed. After moving, recursively remove any keys in the source that are left empty. """ + source = copy.deepcopy(source) + destination = copy.deepcopy(destination) def recursive_move(src, dest, path=None): if path is None: @@ -1316,18 +1330,49 @@ def move_plugins(source, destination): if not value: src.pop(key, None) - def prune_empty_dicts(d): - """ - Recursively prune all empty dictionaries from a given dictionary. - """ - for key, value in list(d.items()): - if isinstance(value, dict): - prune_empty_dicts(value) - if value == {}: - d.pop(key, None) - recursive_move(source, destination) prune_empty_dicts(source) + return source, destination + + +_table_config_keys = ( + "hidden", + "sort", + "sort_desc", + "size", + "sortable_columns", + "label_column", + "facets", + "fts_table", + "fts_pk", + "searchmode", + "units", +) + + +def move_table_config(metadata: dict, config: dict): + """ + Move all known table configuration keys from metadata to config. + """ + if "databases" not in metadata: + return metadata, config + metadata = copy.deepcopy(metadata) + config = copy.deepcopy(config) + for database_name, database in metadata["databases"].items(): + if "tables" not in database: + continue + for table_name, table in database["tables"].items(): + for key in _table_config_keys: + if key in table: + config.setdefault("databases", {}).setdefault( + database_name, {} + ).setdefault("tables", {}).setdefault(table_name, {})[ + key + ] = table.pop( + key + ) + prune_empty_dicts(metadata) + return metadata, config def redact_keys(original: dict, key_patterns: Iterable) -> dict: diff --git a/datasette/views/table.py b/datasette/views/table.py index 2b5b7c24..50d2b3c2 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -142,11 +142,11 @@ async def display_columns_and_rows( """Returns columns, rows for specified table - including fancy foreign key treatment""" sortable_columns = sortable_columns or set() db = datasette.databases[database_name] - table_metadata = await datasette.table_config(database_name, table_name) - column_descriptions = table_metadata.get("columns") or {} + column_descriptions = datasette.metadata("columns", database_name, table_name) or {} column_details = { col.name: col for col in await db.table_column_details(table_name) } + table_config = await datasette.table_config(database_name, table_name) pks = await db.primary_keys(table_name) pks_for_display = pks if not pks_for_display: @@ -193,7 +193,6 @@ async def display_columns_and_rows( "raw": pk_path, "value": markupsafe.Markup( '{flat_pks}'.format( - base_url=base_url, table_path=datasette.urls.table(database_name, table_name), flat_pks=str(markupsafe.escape(pk_path)), flat_pks_quoted=path_from_row_pks(row, pks, not pks), @@ -274,9 +273,9 @@ async def display_columns_and_rows( ), ) ) - elif column in table_metadata.get("units", {}) and value != "": + elif column in table_config.get("units", {}) and value != "": # Interpret units using pint - value = value * ureg(table_metadata["units"][column]) + value = value * ureg(table_config["units"][column]) # Pint uses floating point which sometimes introduces errors in the compact # representation, which we have to round off to avoid ugliness. In the vast # majority of cases this rounding will be inconsequential. I hope. @@ -591,7 +590,7 @@ class TableDropView(BaseView): try: data = json.loads(await request.post_body()) confirm = data.get("confirm") - except json.JSONDecodeError as e: + except json.JSONDecodeError: pass if not confirm: diff --git a/tests/test_api.py b/tests/test_api.py index 0a1f3725..8cb73dbb 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -771,7 +771,7 @@ def test_databases_json(app_client_two_attached_databases_one_immutable): @pytest.mark.asyncio async def test_metadata_json(ds_client): response = await ds_client.get("/-/metadata.json") - assert response.json() == METADATA + assert response.json() == ds_client.ds.metadata() @pytest.mark.asyncio @@ -1061,3 +1061,102 @@ async def test_config_json(config, expected): ds = Datasette(config=config) response = await ds.client.get("/-/config.json") assert response.json() == expected + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "metadata,expected_config,expected_metadata", + ( + ({}, {}, {}), + ( + # Metadata input + { + "title": "Datasette Fixtures", + "databases": { + "fixtures": { + "tables": { + "sortable": { + "sortable_columns": [ + "sortable", + "sortable_with_nulls", + "sortable_with_nulls_2", + "text", + ], + }, + "no_primary_key": {"sortable_columns": [], "hidden": True}, + "units": {"units": {"distance": "m", "frequency": "Hz"}}, + "primary_key_multiple_columns_explicit_label": { + "label_column": "content2" + }, + "simple_view": {"sortable_columns": ["content"]}, + "searchable_view_configured_by_metadata": { + "fts_table": "searchable_fts", + "fts_pk": "pk", + }, + "roadside_attractions": { + "columns": { + "name": "The name of the attraction", + "address": "The street address for the attraction", + } + }, + "attraction_characteristic": {"sort_desc": "pk"}, + "facet_cities": {"sort": "name"}, + "paginated_view": {"size": 25}, + }, + } + }, + }, + # Should produce a config with just the table configuration keys + { + "databases": { + "fixtures": { + "tables": { + "sortable": { + "sortable_columns": [ + "sortable", + "sortable_with_nulls", + "sortable_with_nulls_2", + "text", + ] + }, + "units": {"units": {"distance": "m", "frequency": "Hz"}}, + # These one get redacted: + "no_primary_key": "***", + "primary_key_multiple_columns_explicit_label": "***", + "simple_view": {"sortable_columns": ["content"]}, + "searchable_view_configured_by_metadata": { + "fts_table": "searchable_fts", + "fts_pk": "pk", + }, + "attraction_characteristic": {"sort_desc": "pk"}, + "facet_cities": {"sort": "name"}, + "paginated_view": {"size": 25}, + } + } + } + }, + # And metadata with everything else + { + "title": "Datasette Fixtures", + "databases": { + "fixtures": { + "tables": { + "roadside_attractions": { + "columns": { + "name": "The name of the attraction", + "address": "The street address for the attraction", + } + }, + } + } + }, + }, + ), + ), +) +async def test_upgrade_metadata(metadata, expected_config, expected_metadata): + ds = Datasette(metadata=metadata) + response = await ds.client.get("/-/config.json") + assert response.json() == expected_config + response2 = await ds.client.get("/-/metadata.json") + assert response2.json() == expected_metadata diff --git a/tests/test_html.py b/tests/test_html.py index 86895844..8229b166 100644 --- a/tests/test_html.py +++ b/tests/test_html.py @@ -753,7 +753,7 @@ async def test_metadata_json_html(ds_client): response = await ds_client.get("/-/metadata") assert response.status_code == 200 pre = Soup(response.content, "html.parser").find("pre") - assert METADATA == json.loads(pre.text) + assert ds_client.ds.metadata() == json.loads(pre.text) @pytest.mark.asyncio From 9ac9f0152f1d3396d01ceddfcaf07fd1cf3f7168 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 6 Feb 2024 22:18:38 -0800 Subject: [PATCH 577/952] Migrate allow from metadata to config if necessary, closes #2249 --- datasette/app.py | 6 +++--- datasette/utils/__init__.py | 9 +++++---- tests/test_permissions.py | 20 ++++++++++++-------- 3 files changed, 20 insertions(+), 15 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 373b3e95..af8cfeab 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -74,7 +74,7 @@ from .utils import ( find_spatialite, format_bytes, module_from_path, - move_plugins, + move_plugins_and_allow, move_table_config, parse_metadata, resolve_env_secrets, @@ -344,10 +344,10 @@ class Datasette: with config_files[0].open() as fp: config = parse_metadata(fp.read()) - # Move any "plugins" settings from metadata to config - updates them in place + # Move any "plugins" and "allow" settings from metadata to config - updates them in place metadata = metadata or {} config = config or {} - metadata, config = move_plugins(metadata, config) + metadata, config = move_plugins_and_allow(metadata, config) # Now migrate any known table configuration settings over as well metadata, config = move_table_config(metadata, config) diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index fcaebe3f..f2cd7eb0 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -1302,10 +1302,11 @@ def prune_empty_dicts(d: dict): d.pop(key, None) -def move_plugins(source: dict, destination: dict) -> Tuple[dict, dict]: +def move_plugins_and_allow(source: dict, destination: dict) -> Tuple[dict, dict]: """ - Move 'plugins' keys from source to destination dictionary. Creates hierarchy in destination if needed. - After moving, recursively remove any keys in the source that are left empty. + Move 'plugins' and 'allow' keys from source to destination dictionary. Creates + hierarchy in destination if needed. After moving, recursively remove any keys + in the source that are left empty. """ source = copy.deepcopy(source) destination = copy.deepcopy(destination) @@ -1315,7 +1316,7 @@ def move_plugins(source: dict, destination: dict) -> Tuple[dict, dict]: path = [] for key, value in list(src.items()): new_path = path + [key] - if key == "plugins": + if key in ("plugins", "allow"): # Navigate and create the hierarchy in destination if needed d = dest for step in path: diff --git a/tests/test_permissions.py b/tests/test_permissions.py index 9917b749..6713b850 100644 --- a/tests/test_permissions.py +++ b/tests/test_permissions.py @@ -89,10 +89,11 @@ def test_view_padlock(allow, expected_anon, expected_auth, path, padlock_client) ({"id": "root"}, 403, 200), ], ) -def test_view_database(allow, expected_anon, expected_auth): - with make_app_client( - config={"databases": {"fixtures": {"allow": allow}}} - ) as client: +@pytest.mark.parametrize("use_metadata", (True, False)) +def test_view_database(allow, expected_anon, expected_auth, use_metadata): + key = "metadata" if use_metadata else "config" + kwargs = {key: {"databases": {"fixtures": {"allow": allow}}}} + with make_app_client(**kwargs) as client: for path in ( "/fixtures", "/fixtures/compound_three_primary_keys", @@ -173,16 +174,19 @@ def test_database_list_respects_view_table(): ({"id": "root"}, 403, 200), ], ) -def test_view_table(allow, expected_anon, expected_auth): - with make_app_client( - config={ +@pytest.mark.parametrize("use_metadata", (True, False)) +def test_view_table(allow, expected_anon, expected_auth, use_metadata): + key = "metadata" if use_metadata else "config" + kwargs = { + key: { "databases": { "fixtures": { "tables": {"compound_three_primary_keys": {"allow": allow}} } } } - ) as client: + } + with make_app_client(**kwargs) as client: anon_response = client.get("/fixtures/compound_three_primary_keys") assert expected_anon == anon_response.status if allow and anon_response.status == 200: From ad01f9d3217f2447b0a321ee7731900dac7e3e6d Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 6 Feb 2024 22:24:24 -0800 Subject: [PATCH 578/952] 1.0a8 release notes Closes #2243 * Changelog for jinja2_environment_from_request and plugin_hook_slots * track_event() in changelog * Remove Using YAML for metadata section - no longer necessary now we show YAML and JSON examples everywhere. * Configuration via the command-line section - #2252 * JavaScript plugins in release notes, refs #2052 * /-/config in changelog, refs #2254 Refs #2052, #2156, #2243, #2247, #2249, #2252, #2254 --- docs/changelog.rst | 77 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 77 insertions(+) diff --git a/docs/changelog.rst b/docs/changelog.rst index f4b928e3..1f47b429 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -4,6 +4,83 @@ Changelog ========= +.. _v1_0_a8: + +1.0a8 (2024-02-01) +------------------ + +This alpha release continues the migration of Datasette's configuration from ``metadata.yaml`` to the new ``datasette.yaml`` configuration file, and adds several new plugin hooks. + +Configuration +~~~~~~~~~~~~~ + +- Plugin configuration now lives in the :ref:`datasette.yaml configuration file `, passed to Datasette using the ``-c/--config`` option. Thanks, Alex Garcia. (:issue:`2093`) + + .. code-block:: bash + + datasette -c datasette.yaml + + Where ``datasette.yaml`` contains configuration that looks like this: + + .. code-block:: yaml + + plugins: + datasette-cluster-map: + latitude_column: xlat + longitude_column: xlon + + Previously plugins were configured in ``metadata.yaml``, which was confusing as plugin settings were unrelated to database and table metadata. +- The ``-s/--setting`` option can now be used to set plugin configuration as well. See :ref:`configuration_cli` for details. (:issue:`2252`) + + The above YAML configuration example using ``-s/--setting`` looks like this: + + .. code-block:: bash + + datasette mydatabase.db \ + -s plugins.datasette-cluster-map.latitude_column xlat \ + -s plugins.datasette-cluster-map.longitude_column xlon + +- The new ``/-/config`` page shows the current instance configuration, after redacting keys that could contain sensitive data such as API keys or passwords. (:issue:`2254`) + +- Existing Datasette installations may already have configuration set in ``metadata.yaml`` that should be migrated to ``datasette.yaml``. To avoid breaking these installations, Datasette will silently treat table configuration, plugin configuration and allow blocks in metadata as if they had been specified in configuration instead. (:issue:`2247`) (:issue:`2248`) (:issue:`2249`) + +JavaScript plugins +~~~~~~~~~~~~~~~~~~ + +Datasette now includes a :ref:`JavaScript plugins mechanism `, allowing JavaScript to customize Datasette in a way that can collaborate with other plugins. + +This provides two initial hooks, with more to come in the future: + +- :ref:`makeAboveTablePanelConfigs() ` can add additional panels to the top of the table page. +- :ref:`makeColumnActions() ` can add additional actions to the column menu. + +Thanks `Cameron Yick `__ for contributing this feature. (`#2052 `__) + +Plugin hooks +~~~~~~~~~~~~ + +- New :ref:`plugin_hook_jinja2_environment_from_request` plugin hook, which can be used to customize the current Jinja environment based on the incoming request. This can be used to modify the template lookup path based on the incoming request hostname, among other things. (:issue:`2225`) +- New :ref:`family of template slot plugin hooks `: ``top_homepage``, ``top_database``, ``top_table``, ``top_row``, ``top_query``, ``top_canned_query``. Plugins can use these to provide additional HTML to be injected at the top of the corresponding pages. (:issue:`1191`) +- New :ref:`track_event() mechanism ` for plugins to emit and receive events when certain events occur within Datasette. (:issue:`2240`) + - Plugins can register additional event classes using :ref:`plugin_hook_register_events`. + - They can then trigger those events with the :ref:`datasette.track_event(event) ` internal method. + - Plugins can subscribe to notifications of events using the :ref:`plugin_hook_track_event` plugin hook. +- New internal function for plugin authors: :ref:`database_execute_isolated_fn`, for creating a new SQLite connection, executing code and then closing that connection, all while preventing other code from writing to that particular database. This connection will not have the :ref:`prepare_connection() ` plugin hook executed against it, allowing plugins to perform actions that might otherwise be blocked by existing connection configuration. (:issue:`2218`) + +Documentation +~~~~~~~~~~~~~ + +- Documentation describing :ref:`how to write tests that use signed actor cookies ` using ``datasette.client.actor_cookie()``. (:issue:`1830`) +- Documentation on how to :ref:`register a plugin for the duration of a test `. (:issue:`2234`) +- The :ref:`configuration documentation ` now shows examples of both YAML and JSON for each setting. + +Minor fixes +~~~~~~~~~~~ + +- Datasette no longer attempts to run SQL queries in parallel when rendering a table page, as this was leading to some rare crashing bugs. (:issue:`2189`) +- Fixed warning: ``DeprecationWarning: pkg_resources is deprecated as an API`` (:issue:`2057`) +- Fixed bug where ``?_extra=columns`` parameter returned an incorrectly shaped response. (:issue:`2230`) + .. _v0_64_6: 0.64.6 (2023-12-22) From c64453a4a137ea815f4d94a99cdc4c7709734839 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 6 Feb 2024 22:28:22 -0800 Subject: [PATCH 579/952] Fix the date on the 1.0a8 release (due to go tomorrow) Refs #2258 --- docs/changelog.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index 1f47b429..5e9e3ba2 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -6,7 +6,7 @@ Changelog .. _v1_0_a8: -1.0a8 (2024-02-01) +1.0a8 (2024-02-07) ------------------ This alpha release continues the migration of Datasette's configuration from ``metadata.yaml`` to the new ``datasette.yaml`` configuration file, and adds several new plugin hooks. From d0089ba7769d58b97c5dc1e07969246502d97544 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 6 Feb 2024 22:30:30 -0800 Subject: [PATCH 580/952] Note in changelog about datasette publish, refs #2195 --- docs/changelog.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/changelog.rst b/docs/changelog.rst index 5e9e3ba2..e17dc2f8 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -44,6 +44,8 @@ Configuration - Existing Datasette installations may already have configuration set in ``metadata.yaml`` that should be migrated to ``datasette.yaml``. To avoid breaking these installations, Datasette will silently treat table configuration, plugin configuration and allow blocks in metadata as if they had been specified in configuration instead. (:issue:`2247`) (:issue:`2248`) (:issue:`2249`) +Note that the ``datasette publish`` command has not yet been updated to accept a ``datasette.yaml`` configuration file. This will be addressed in :issue:`2195` but for the moment you can include those settings in ``metadata.yaml`` instead. + JavaScript plugins ~~~~~~~~~~~~~~~~~~ From df8d1c055a48a36693d9caec3915eb93c1acefc0 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 6 Feb 2024 22:59:58 -0800 Subject: [PATCH 581/952] Mention JS plugins in release intro --- docs/changelog.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index e17dc2f8..a73635a8 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -9,7 +9,7 @@ Changelog 1.0a8 (2024-02-07) ------------------ -This alpha release continues the migration of Datasette's configuration from ``metadata.yaml`` to the new ``datasette.yaml`` configuration file, and adds several new plugin hooks. +This alpha release continues the migration of Datasette's configuration from ``metadata.yaml`` to the new ``datasette.yaml`` configuration file, introduces a new system for JavaScript plugins and adds several new plugin hooks. Configuration ~~~~~~~~~~~~~ From 1e31821d9ff2bc81c62bcd442214e9098cf785a4 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 7 Feb 2024 08:25:47 -0800 Subject: [PATCH 582/952] Link to events docs from changelog --- docs/changelog.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/changelog.rst b/docs/changelog.rst index a73635a8..dfcf492f 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -67,6 +67,7 @@ Plugin hooks - Plugins can register additional event classes using :ref:`plugin_hook_register_events`. - They can then trigger those events with the :ref:`datasette.track_event(event) ` internal method. - Plugins can subscribe to notifications of events using the :ref:`plugin_hook_track_event` plugin hook. + - Datasette core now emits ``login``, ``logout``, ``create-token``, ``create-table``, ``drop-table``, ``insert-rows``, ``upsert-rows``, ``update-row``, ``delete-row`` events, :ref:`documented here `. - New internal function for plugin authors: :ref:`database_execute_isolated_fn`, for creating a new SQLite connection, executing code and then closing that connection, all while preventing other code from writing to that particular database. This connection will not have the :ref:`prepare_connection() ` plugin hook executed against it, allowing plugins to perform actions that might otherwise be blocked by existing connection configuration. (:issue:`2218`) Documentation From e0794ddd52697812848c0b59f68e49a2e9361693 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 7 Feb 2024 08:32:47 -0800 Subject: [PATCH 583/952] Link to annotated release notes blog post --- docs/changelog.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/changelog.rst b/docs/changelog.rst index dfcf492f..d164f71d 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -11,6 +11,8 @@ Changelog This alpha release continues the migration of Datasette's configuration from ``metadata.yaml`` to the new ``datasette.yaml`` configuration file, introduces a new system for JavaScript plugins and adds several new plugin hooks. +See `Datasette 1.0a8: JavaScript plugins, new plugin hooks and plugin configuration in datasette.yaml `__ for an annotated version of these release notes. + Configuration ~~~~~~~~~~~~~ From 9989f257094daaf26e0cb0cebe31f17f19d4cad2 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 7 Feb 2024 08:34:05 -0800 Subject: [PATCH 584/952] Release 1.0a8 Refs Refs #2052, #2156, #2243, #2247, #2249, #2252, #2254, #2258 --- datasette/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/datasette/version.py b/datasette/version.py index 75d44727..e43b9918 100644 --- a/datasette/version.py +++ b/datasette/version.py @@ -1,2 +1,2 @@ -__version__ = "1.0a8.dev1" +__version__ = "1.0a8" __version_info__ = tuple(__version__.split(".")) From 569aacd39bcc5529b2f463c89c616e3ada21c560 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 7 Feb 2024 22:53:14 -0800 Subject: [PATCH 585/952] Link to /en/latest/ changelog --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 57f17a5c..662f2a11 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ Datasette [![PyPI](https://img.shields.io/pypi/v/datasette.svg)](https://pypi.org/project/datasette/) -[![Changelog](https://img.shields.io/github/v/release/simonw/datasette?label=changelog)](https://docs.datasette.io/en/stable/changelog.html) +[![Changelog](https://img.shields.io/github/v/release/simonw/datasette?label=changelog)](https://docs.datasette.io/en/latest/changelog.html) [![Python 3.x](https://img.shields.io/pypi/pyversions/datasette.svg?logo=python&logoColor=white)](https://pypi.org/project/datasette/) [![Tests](https://github.com/simonw/datasette/workflows/Test/badge.svg)](https://github.com/simonw/datasette/actions?query=workflow%3ATest) [![Documentation Status](https://readthedocs.org/projects/datasette/badge/?version=latest)](https://docs.datasette.io/en/latest/?badge=latest) From 900d15bcb81c90d26cfebc3fe463c4b0465832c2 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 8 Feb 2024 12:21:13 -0800 Subject: [PATCH 586/952] alter table support for /db/-/create API, refs #2101 --- datasette/default_permissions.py | 10 ++++- datasette/events.py | 25 +++++++++++ datasette/views/database.py | 61 +++++++++++++++++++++++--- docs/authentication.rst | 12 ++++++ tests/test_api_write.py | 74 ++++++++++++++++++++++++++++++++ 5 files changed, 174 insertions(+), 8 deletions(-) diff --git a/datasette/default_permissions.py b/datasette/default_permissions.py index d29dbe84..c13f2ed2 100644 --- a/datasette/default_permissions.py +++ b/datasette/default_permissions.py @@ -8,7 +8,6 @@ from typing import Union, Tuple @hookimpl def register_permissions(): return ( - # name, abbr, description, takes_database, takes_resource, default Permission( name="view-instance", abbr="vi", @@ -109,6 +108,14 @@ def register_permissions(): takes_resource=False, default=False, ), + Permission( + name="alter-table", + abbr="at", + description="Alter tables", + takes_database=True, + takes_resource=True, + default=False, + ), Permission( name="drop-table", abbr="dt", @@ -129,6 +136,7 @@ def permission_allowed_default(datasette, actor, action, resource): "debug-menu", "insert-row", "create-table", + "alter-table", "drop-table", "delete-row", "update-row", diff --git a/datasette/events.py b/datasette/events.py index 96244779..ae90972d 100644 --- a/datasette/events.py +++ b/datasette/events.py @@ -108,6 +108,30 @@ class DropTableEvent(Event): table: str +@dataclass +class AlterTableEvent(Event): + """ + Event name: ``alter-table`` + + A table has been altered. + + :ivar database: The name of the database where the table was altered + :type database: str + :ivar table: The name of the table that was altered + :type table: str + :ivar before_schema: The table's SQL schema before the alteration + :type before_schema: str + :ivar after_schema: The table's SQL schema after the alteration + :type after_schema: str + """ + + name = "alter-table" + database: str + table: str + before_schema: str + after_schema: str + + @dataclass class InsertRowsEvent(Event): """ @@ -203,6 +227,7 @@ def register_events(): LogoutEvent, CreateTableEvent, CreateTokenEvent, + AlterTableEvent, DropTableEvent, InsertRowsEvent, UpsertRowsEvent, diff --git a/datasette/views/database.py b/datasette/views/database.py index 6d17b16c..bd55064f 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -10,7 +10,7 @@ import re import sqlite_utils import textwrap -from datasette.events import CreateTableEvent +from datasette.events import AlterTableEvent, CreateTableEvent from datasette.database import QueryInterrupted from datasette.utils import ( add_cors_headers, @@ -792,7 +792,17 @@ class MagicParameters(dict): class TableCreateView(BaseView): name = "table-create" - _valid_keys = {"table", "rows", "row", "columns", "pk", "pks", "ignore", "replace"} + _valid_keys = { + "table", + "rows", + "row", + "columns", + "pk", + "pks", + "ignore", + "replace", + "alter", + } _supported_column_types = { "text", "integer", @@ -876,6 +886,20 @@ class TableCreateView(BaseView): ): return _error(["Permission denied - need insert-row"], 403) + alter = False + if rows or row: + if not table_exists: + # if table is being created for the first time, alter=True + alter = True + else: + # alter=True only if they request it AND they have permission + if data.get("alter"): + if not await self.ds.permission_allowed( + request.actor, "alter-table", resource=database_name + ): + return _error(["Permission denied - need alter-table"], 403) + alter = True + if columns: if rows or row: return _error(["Cannot specify columns with rows or row"]) @@ -939,10 +963,18 @@ class TableCreateView(BaseView): return _error(["pk cannot be changed for existing table"]) pks = actual_pks + initial_schema = None + if table_exists: + initial_schema = await db.execute_fn( + lambda conn: sqlite_utils.Database(conn)[table_name].schema + ) + def create_table(conn): table = sqlite_utils.Database(conn)[table_name] if rows: - table.insert_all(rows, pk=pks or pk, ignore=ignore, replace=replace) + table.insert_all( + rows, pk=pks or pk, ignore=ignore, replace=replace, alter=alter + ) else: table.create( {c["name"]: c["type"] for c in columns}, @@ -954,6 +986,18 @@ class TableCreateView(BaseView): schema = await db.execute_write_fn(create_table) except Exception as e: return _error([str(e)]) + + if initial_schema is not None and initial_schema != schema: + await self.ds.track_event( + AlterTableEvent( + request.actor, + database=database_name, + table=table_name, + before_schema=initial_schema, + after_schema=schema, + ) + ) + table_url = self.ds.absolute_url( request, self.ds.urls.table(db.name, table_name) ) @@ -970,11 +1014,14 @@ class TableCreateView(BaseView): } if rows: details["row_count"] = len(rows) - await self.ds.track_event( - CreateTableEvent( - request.actor, database=db.name, table=table_name, schema=schema + + if not table_exists: + # Only log creation if we created a table + await self.ds.track_event( + CreateTableEvent( + request.actor, database=db.name, table=table_name, schema=schema + ) ) - ) return Response.json(details, status=201) diff --git a/docs/authentication.rst b/docs/authentication.rst index 8758765d..87ee6385 100644 --- a/docs/authentication.rst +++ b/docs/authentication.rst @@ -1217,6 +1217,18 @@ Actor is allowed to create a database table. Default *deny*. +.. _permissions_alter_table: + +alter-table +----------- + +Actor is allowed to alter a database table. + +``resource`` - tuple: (string, string) + The name of the database, then the name of the table + +Default *deny*. + .. _permissions_drop_table: drop-table diff --git a/tests/test_api_write.py b/tests/test_api_write.py index 9caf9fdf..30cbfbab 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -1349,3 +1349,77 @@ async def test_method_not_allowed(ds_write, path): "ok": False, "error": "Method not allowed", } + + +@pytest.mark.asyncio +async def test_create_uses_alter_by_default_for_new_table(ds_write): + token = write_token(ds_write) + response = await ds_write.client.post( + "/data/-/create", + json={ + "table": "new_table", + "rows": [ + { + "name": "Row 1", + } + ] + * 100 + + [ + {"name": "Row 2", "extra": "Extra"}, + ], + "pk": "id", + }, + headers=_headers(token), + ) + assert response.status_code == 201 + event = last_event(ds_write) + assert event.name == "create-table" + + +@pytest.mark.asyncio +@pytest.mark.parametrize("has_alter_permission", (True,)) # False)) +async def test_create_using_alter_against_existing_table( + ds_write, has_alter_permission +): + token = write_token( + ds_write, permissions=["ir", "ct"] + (["at"] if has_alter_permission else []) + ) + # First create the table + response = await ds_write.client.post( + "/data/-/create", + json={ + "table": "new_table", + "rows": [ + { + "name": "Row 1", + } + ], + "pk": "id", + }, + headers=_headers(token), + ) + assert response.status_code == 201 + # Now try to insert more rows using /-/create with alter=True + response2 = await ds_write.client.post( + "/data/-/create", + json={ + "table": "new_table", + "rows": [{"name": "Row 2", "extra": "extra"}], + "pk": "id", + "alter": True, + }, + headers=_headers(token), + ) + if not has_alter_permission: + assert response2.status_code == 403 + assert response2.json() == { + "ok": False, + "errors": ["Permission denied - need alter-table"], + } + else: + assert response2.status_code == 201 + # It should have altered the table + event = last_event(ds_write) + assert event.name == "alter-table" + assert "extra" not in event.before_schema + assert "extra" in event.after_schema From 574687834f4bd8e73281731b8ff01bfe093fecb5 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 8 Feb 2024 12:33:41 -0800 Subject: [PATCH 587/952] Docs for /db/-/create alter: true option, refs #2101 --- docs/json_api.rst | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/docs/json_api.rst b/docs/json_api.rst index 16b997eb..68a0c984 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -834,19 +834,22 @@ To create a table, make a ``POST`` to ``//-/create``. This requires th The JSON here describes the table that will be created: -* ``table`` is the name of the table to create. This field is required. -* ``columns`` is a list of columns to create. Each column is a dictionary with ``name`` and ``type`` keys. +* ``table`` is the name of the table to create. This field is required. +* ``columns`` is a list of columns to create. Each column is a dictionary with ``name`` and ``type`` keys. - - ``name`` is the name of the column. This is required. - - ``type`` is the type of the column. This is optional - if not provided, ``text`` will be assumed. The valid types are ``text``, ``integer``, ``float`` and ``blob``. + - ``name`` is the name of the column. This is required. + - ``type`` is the type of the column. This is optional - if not provided, ``text`` will be assumed. The valid types are ``text``, ``integer``, ``float`` and ``blob``. -* ``pk`` is the primary key for the table. This is optional - if not provided, Datasette will create a SQLite table with a hidden ``rowid`` column. +* ``pk`` is the primary key for the table. This is optional - if not provided, Datasette will create a SQLite table with a hidden ``rowid`` column. - If the primary key is an integer column, it will be configured to automatically increment for each new record. + If the primary key is an integer column, it will be configured to automatically increment for each new record. - If you set this to ``id`` without including an ``id`` column in the list of ``columns``, Datasette will create an integer ID column for you. + If you set this to ``id`` without including an ``id`` column in the list of ``columns``, Datasette will create an auto-incrementing integer ID column for you. -* ``pks`` can be used instead of ``pk`` to create a compound primary key. It should be a JSON list of column names to use in that primary key. +* ``pks`` can be used instead of ``pk`` to create a compound primary key. It should be a JSON list of column names to use in that primary key. +* ``ignore`` can be set to ``true`` to ignore existing rows by primary key if the table already exists. +* ``replace`` can be set to ``true`` to replace existing rows by primary key if the table already exists. +* ``alter`` can be set to ``true`` if you want to automatically add any missing columns to the table. This requires the :ref:`permissions_alter_table` permission. If the table is successfully created this will return a ``201`` status code and the following response: @@ -925,6 +928,8 @@ You can avoid this error by passing the same ``"ignore": true`` or ``"replace": To use the ``"replace": true`` option you will also need the :ref:`permissions_update_row` permission. +Pass ``"alter": true`` to automatically add any missing columns to the existing table that are present in the rows you are submitting. This requires the :ref:`permissions_alter_table` permission. + .. _TableDropView: Dropping tables From b5ccc4d60844a24fdf91c3f317d8cda4a285a58d Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 8 Feb 2024 12:35:12 -0800 Subject: [PATCH 588/952] Test for Permission denied - need alter-table --- tests/test_api_write.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_api_write.py b/tests/test_api_write.py index 30cbfbab..abf9a88a 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -1377,7 +1377,7 @@ async def test_create_uses_alter_by_default_for_new_table(ds_write): @pytest.mark.asyncio -@pytest.mark.parametrize("has_alter_permission", (True,)) # False)) +@pytest.mark.parametrize("has_alter_permission", (True, False)) async def test_create_using_alter_against_existing_table( ds_write, has_alter_permission ): From 528d89d1a3d6ff85047a7eef9a7623efdd2fb19f Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 8 Feb 2024 13:14:12 -0800 Subject: [PATCH 589/952] alter: true support for /-/insert and /-/upsert, refs #2101 --- datasette/views/table.py | 48 +++++++++++++++++++++++++++++++++++----- docs/json_api.rst | 6 ++++- tests/test_api_write.py | 48 ++++++++++++++++++++++++++++++++++++---- 3 files changed, 91 insertions(+), 11 deletions(-) diff --git a/datasette/views/table.py b/datasette/views/table.py index 50d2b3c2..fcbe253d 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -8,7 +8,12 @@ import markupsafe from datasette.plugins import pm from datasette.database import QueryInterrupted -from datasette.events import DropTableEvent, InsertRowsEvent, UpsertRowsEvent +from datasette.events import ( + AlterTableEvent, + DropTableEvent, + InsertRowsEvent, + UpsertRowsEvent, +) from datasette import tracer from datasette.utils import ( add_cors_headers, @@ -388,7 +393,7 @@ class TableInsertView(BaseView): extras = { key: value for key, value in data.items() if key not in ("row", "rows") } - valid_extras = {"return", "ignore", "replace"} + valid_extras = {"return", "ignore", "replace", "alter"} invalid_extras = extras.keys() - valid_extras if invalid_extras: return _errors( @@ -397,7 +402,6 @@ class TableInsertView(BaseView): if extras.get("ignore") and extras.get("replace"): return _errors(['Cannot use "ignore" and "replace" at the same time']) - # Validate columns of each row columns = set(await db.table_columns(table_name)) columns.update(pks_list) @@ -412,7 +416,7 @@ class TableInsertView(BaseView): ) ) invalid_columns = set(row.keys()) - columns - if invalid_columns: + if invalid_columns and not extras.get("alter"): errors.append( "Row {} has invalid columns: {}".format( i, ", ".join(sorted(invalid_columns)) @@ -476,10 +480,23 @@ class TableInsertView(BaseView): ignore = extras.get("ignore") replace = extras.get("replace") + alter = extras.get("alter") if upsert and (ignore or replace): return _error(["Upsert does not support ignore or replace"], 400) + initial_schema = None + if alter: + # Must have alter-table permission + if not await self.ds.permission_allowed( + request.actor, "alter-table", resource=(database_name, table_name) + ): + return _error(["Permission denied for alter-table"], 403) + # Track initial schema to check if it changed later + initial_schema = await db.execute_fn( + lambda conn: sqlite_utils.Database(conn)[table_name].schema + ) + should_return = bool(extras.get("return", False)) row_pk_values_for_later = [] if should_return and upsert: @@ -489,9 +506,13 @@ class TableInsertView(BaseView): table = sqlite_utils.Database(conn)[table_name] kwargs = {} if upsert: - kwargs["pk"] = pks[0] if len(pks) == 1 else pks + kwargs = { + "pk": pks[0] if len(pks) == 1 else pks, + "alter": alter, + } else: - kwargs = {"ignore": ignore, "replace": replace} + # Insert + kwargs = {"ignore": ignore, "replace": replace, "alter": alter} if should_return and not upsert: rowids = [] method = table.upsert if upsert else table.insert @@ -552,6 +573,21 @@ class TableInsertView(BaseView): ) ) + if initial_schema is not None: + after_schema = await db.execute_fn( + lambda conn: sqlite_utils.Database(conn)[table_name].schema + ) + if initial_schema != after_schema: + await self.ds.track_event( + AlterTableEvent( + request.actor, + database=database_name, + table=table_name, + before_schema=initial_schema, + after_schema=after_schema, + ) + ) + return Response.json(result, status=200 if upsert else 201) diff --git a/docs/json_api.rst b/docs/json_api.rst index 68a0c984..000f532d 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -618,6 +618,8 @@ Pass ``"ignore": true`` to ignore these errors and insert the other rows: Or you can pass ``"replace": true`` to replace any rows with conflicting primary keys with the new values. +Pass ``"alter: true`` to automatically add any missing columns to the table. This requires the :ref:`permissions_alter_table` permission. + .. _TableUpsertView: Upserting rows @@ -728,6 +730,8 @@ When using upsert you must provide the primary key column (or columns if the tab If your table does not have an explicit primary key you should pass the SQLite ``rowid`` key instead. +Pass ``"alter: true`` to automatically add any missing columns to the table. This requires the :ref:`permissions_alter_table` permission. + .. _RowUpdateView: Updating a row @@ -849,7 +853,7 @@ The JSON here describes the table that will be created: * ``pks`` can be used instead of ``pk`` to create a compound primary key. It should be a JSON list of column names to use in that primary key. * ``ignore`` can be set to ``true`` to ignore existing rows by primary key if the table already exists. * ``replace`` can be set to ``true`` to replace existing rows by primary key if the table already exists. -* ``alter`` can be set to ``true`` if you want to automatically add any missing columns to the table. This requires the :ref:`permissions_alter_table` permission. +* ``alter`` can be set to ``true`` if you want to automatically add any missing columns to the table. This requires the :ref:`permissions_alter_table` permission. If the table is successfully created this will return a ``201`` status code and the following response: diff --git a/tests/test_api_write.py b/tests/test_api_write.py index abf9a88a..9e1d73e0 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -60,6 +60,27 @@ async def test_insert_row(ds_write): assert not event.replace +@pytest.mark.asyncio +async def test_insert_row_alter(ds_write): + token = write_token(ds_write) + response = await ds_write.client.post( + "/data/docs/-/insert", + json={ + "row": {"title": "Test", "score": 1.2, "age": 5, "extra": "extra"}, + "alter": True, + }, + headers=_headers(token), + ) + assert response.status_code == 201 + assert response.json()["ok"] is True + assert response.json()["rows"][0]["extra"] == "extra" + # Analytics event + event = last_event(ds_write) + assert event.name == "alter-table" + assert "extra" not in event.before_schema + assert "extra" in event.after_schema + + @pytest.mark.asyncio @pytest.mark.parametrize("return_rows", (True, False)) async def test_insert_rows(ds_write, return_rows): @@ -278,16 +299,27 @@ async def test_insert_rows(ds_write, return_rows): 403, ["Permission denied: need both insert-row and update-row"], ), + # Alter table forbidden without alter permission + ( + "/data/docs/-/upsert", + {"rows": [{"id": 1, "title": "One", "extra": "extra"}], "alter": True}, + "update-and-insert-but-no-alter", + 403, + ["Permission denied for alter-table"], + ), ), ) async def test_insert_or_upsert_row_errors( ds_write, path, input, special_case, expected_status, expected_errors ): - token = write_token(ds_write) + token_permissions = [] if special_case == "insert-but-not-update": - token = write_token(ds_write, permissions=["ir", "vi"]) + token_permissions = ["ir", "vi"] if special_case == "update-but-not-insert": - token = write_token(ds_write, permissions=["ur", "vi"]) + token_permissions = ["ur", "vi"] + if special_case == "update-and-insert-but-no-alter": + token_permissions = ["ur", "ir"] + token = write_token(ds_write, permissions=token_permissions) if special_case == "duplicate_id": await ds_write.get_database("data").execute_write( "insert into docs (id) values (1)" @@ -309,7 +341,9 @@ async def test_insert_or_upsert_row_errors( actor_response = ( await ds_write.client.get("/-/actor.json", headers=kwargs["headers"]) ).json() - print(actor_response) + assert set((actor_response["actor"] or {}).get("_r", {}).get("a") or []) == set( + token_permissions + ) if special_case == "invalid_json": del kwargs["json"] @@ -434,6 +468,12 @@ async def test_insert_ignore_replace( {"id": 1, "title": "Two", "score": 1}, ], ), + ( + # Upsert with an alter + {"rows": [{"id": 1, "title": "One"}], "pk": "id"}, + {"rows": [{"id": 1, "title": "Two", "extra": "extra"}], "alter": True}, + [{"id": 1, "title": "Two", "extra": "extra"}], + ), ), ) @pytest.mark.parametrize("should_return", (False, True)) From 4e944c29e4a208f173f15ac2df6253ff90f6466f Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 8 Feb 2024 13:19:47 -0800 Subject: [PATCH 590/952] Corrected path used in test_update_row_check_permission --- tests/test_api_write.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_api_write.py b/tests/test_api_write.py index 9e1d73e0..b43ee5a6 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -633,7 +633,7 @@ async def test_update_row_check_permission(ds_write, scenario): pk = await _insert_row(ds_write) - path = "/data/{}/{}/-/delete".format( + path = "/data/{}/{}/-/update".format( "docs" if scenario != "bad_table" else "bad_table", pk ) From c954795f9af9007e7c04d9b472bfd2faef647a87 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 8 Feb 2024 13:30:48 -0800 Subject: [PATCH 591/952] alter: true for row/-/update, refs #2101 --- datasette/views/row.py | 12 +++++++++++- docs/json_api.rst | 2 ++ tests/test_api_write.py | 43 +++++++++++++++++++++++++++++++++++++++-- 3 files changed, 54 insertions(+), 3 deletions(-) diff --git a/datasette/views/row.py b/datasette/views/row.py index 7b43b893..4d20e41a 100644 --- a/datasette/views/row.py +++ b/datasette/views/row.py @@ -237,11 +237,21 @@ class RowUpdateView(BaseView): if not "update" in data or not isinstance(data["update"], dict): return _error(["JSON must contain an update dictionary"]) + invalid_keys = set(data.keys()) - {"update", "return", "alter"} + if invalid_keys: + return _error(["Invalid keys: {}".format(", ".join(invalid_keys))]) + update = data["update"] + alter = data.get("alter") + if alter and not await self.ds.permission_allowed( + request.actor, "alter-table", resource=(resolved.db.name, resolved.table) + ): + return _error(["Permission denied for alter-table"], 403) + def update_row(conn): sqlite_utils.Database(conn)[resolved.table].update( - resolved.pk_values, update + resolved.pk_values, update, alter=alter ) try: diff --git a/docs/json_api.rst b/docs/json_api.rst index 000f532d..c401d97e 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -787,6 +787,8 @@ The returned JSON will look like this: Any errors will return ``{"errors": ["... descriptive message ..."], "ok": false}``, and a ``400`` status code for a bad input or a ``403`` status code for an authentication or permission error. +Pass ``"alter: true`` to automatically add any missing columns to the table. This requires the :ref:`permissions_alter_table` permission. + .. _RowDeleteView: Deleting a row diff --git a/tests/test_api_write.py b/tests/test_api_write.py index b43ee5a6..7cc38674 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -622,12 +622,17 @@ async def test_delete_row(ds_write, table, row_for_create, pks, delete_path): @pytest.mark.asyncio -@pytest.mark.parametrize("scenario", ("no_token", "no_perm", "bad_table")) +@pytest.mark.parametrize( + "scenario", ("no_token", "no_perm", "bad_table", "cannot_alter") +) async def test_update_row_check_permission(ds_write, scenario): if scenario == "no_token": token = "bad_token" elif scenario == "no_perm": token = write_token(ds_write, actor_id="not-root") + elif scenario == "cannot_alter": + # update-row but no alter-table: + token = write_token(ds_write, permissions=["ur"]) else: token = write_token(ds_write) @@ -637,9 +642,13 @@ async def test_update_row_check_permission(ds_write, scenario): "docs" if scenario != "bad_table" else "bad_table", pk ) + json_body = {"update": {"title": "New title"}} + if scenario == "cannot_alter": + json_body["alter"] = True + response = await ds_write.client.post( path, - json={"update": {"title": "New title"}}, + json=json_body, headers=_headers(token), ) assert response.status_code == 403 if scenario in ("no_token", "bad_token") else 404 @@ -651,6 +660,36 @@ async def test_update_row_check_permission(ds_write, scenario): ) +@pytest.mark.asyncio +async def test_update_row_invalid_key(ds_write): + token = write_token(ds_write) + + pk = await _insert_row(ds_write) + + path = "/data/docs/{}/-/update".format(pk) + response = await ds_write.client.post( + path, + json={"update": {"title": "New title"}, "bad_key": 1}, + headers=_headers(token), + ) + assert response.status_code == 400 + assert response.json() == {"ok": False, "errors": ["Invalid keys: bad_key"]} + + +@pytest.mark.asyncio +async def test_update_row_alter(ds_write): + token = write_token(ds_write, permissions=["ur", "at"]) + pk = await _insert_row(ds_write) + path = "/data/docs/{}/-/update".format(pk) + response = await ds_write.client.post( + path, + json={"update": {"title": "New title", "extra": "extra"}, "alter": True}, + headers=_headers(token), + ) + assert response.status_code == 200 + assert response.json() == {"ok": True} + + @pytest.mark.asyncio @pytest.mark.parametrize( "input,expected_errors", From c62cfa6de836667834b5b9a7fef2b861307ac998 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 8 Feb 2024 13:32:36 -0800 Subject: [PATCH 592/952] Fix upsert test to detect new alter-table event --- tests/test_api_write.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/tests/test_api_write.py b/tests/test_api_write.py index 7cc38674..2d127e1a 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -499,10 +499,14 @@ async def test_upsert(ds_write, initial, input, expected_rows, should_return): # Analytics event event = last_event(ds_write) - assert event.name == "upsert-rows" - assert event.num_rows == 1 assert event.database == "data" assert event.table == "upsert_test" + if input.get("alter"): + assert event.name == "alter-table" + assert "extra" in event.after_schema + else: + assert event.name == "upsert-rows" + assert event.num_rows == 1 if should_return: # We only expect it to return rows corresponding to those we sent From dcd9ea3622520c99a1f921766dc36ca4c0e3b796 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 8 Feb 2024 14:14:58 -0800 Subject: [PATCH 593/952] datasette-events-db as an example of track_events() --- docs/plugin_hooks.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index 16f5cebb..960dc9b6 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -1817,6 +1817,7 @@ This example plugin logs details of all events to standard error: ) print(msg, file=sys.stderr, flush=True) +Example: `datasette-events-db `_ .. _plugin_hook_register_events: From bd9ed62e5d8821f9dc9e035b195452980c900b3c Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 8 Feb 2024 18:58:12 -0800 Subject: [PATCH 594/952] Make ds.pemrission_allawed(..., default=) a keyword-only argument, refs #2262 --- datasette/app.py | 2 +- datasette/views/table.py | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index af8cfeab..d943b97b 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -896,7 +896,7 @@ class Datasette: await await_me_maybe(hook) async def permission_allowed( - self, actor, action, resource=None, default=DEFAULT_NOT_SET + self, actor, action, resource=None, *, default=DEFAULT_NOT_SET ): """Check permissions using the permissions_allowed plugin hook""" result = None diff --git a/datasette/views/table.py b/datasette/views/table.py index fcbe253d..1c187692 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -444,10 +444,10 @@ class TableInsertView(BaseView): # Must have insert-row AND upsert-row permissions if not ( await self.ds.permission_allowed( - request.actor, "insert-row", database_name, table_name + request.actor, "insert-row", resource=(database_name, table_name) ) and await self.ds.permission_allowed( - request.actor, "update-row", database_name, table_name + request.actor, "update-row", resource=(database_name, table_name) ) ): return _error( From 398a92cf1e54f868ff80f01634d6a814d1c61998 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 8 Feb 2024 20:12:22 -0800 Subject: [PATCH 595/952] Include database in name of _execute_writes thread, closes #2265 --- datasette/database.py | 3 +++ tests/test_api.py | 6 +++++- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/datasette/database.py b/datasette/database.py index fba81496..becb552c 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -196,6 +196,9 @@ class Database: self._write_thread = threading.Thread( target=self._execute_writes, daemon=True ) + self._write_thread.name = "_execute_writes for database {}".format( + self.name + ) self._write_thread.start() task_id = uuid.uuid5(uuid.NAMESPACE_DNS, "datasette.io") reply_queue = janus.Queue() diff --git a/tests/test_api.py b/tests/test_api.py index 8cb73dbb..7a25b55e 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -780,7 +780,11 @@ async def test_threads_json(ds_client): expected_keys = {"threads", "num_threads"} if sys.version_info >= (3, 7, 0): expected_keys.update({"tasks", "num_tasks"}) - assert set(response.json().keys()) == expected_keys + data = response.json() + assert set(data.keys()) == expected_keys + # Should be at least one _execute_writes thread for __INTERNAL__ + thread_names = [thread["name"] for thread in data["threads"]] + assert "_execute_writes for database __INTERNAL__" in thread_names @pytest.mark.asyncio From 5d7997418664bcdfdba714c16bd5a67c241e8740 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sat, 10 Feb 2024 07:19:47 -0800 Subject: [PATCH 596/952] Call them "notable events" --- docs/plugin_hooks.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index 960dc9b6..5372ea5e 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -1765,7 +1765,7 @@ Returns HTML to be displayed at the top of the canned query page. Event tracking -------------- -Datasette includes an internal mechanism for tracking analytical events. This can be used for analytics, but can also be used by plugins that want to listen out for when key events occur (such as a table being created) and take action in response. +Datasette includes an internal mechanism for tracking notable events. This can be used for analytics, but can also be used by plugins that want to listen out for when key events occur (such as a table being created) and take action in response. Plugins can register to receive events using the ``track_event`` plugin hook. From b89cac3b6a63929325c067d0cf2d5748e4bf4d2e Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 13 Feb 2024 18:23:54 -0800 Subject: [PATCH 597/952] Use MD5 usedforsecurity=False on Python 3.9 and higher to pass FIPS Closes #2270 --- datasette/database.py | 4 ++-- datasette/utils/__init__.py | 10 +++++++++- 2 files changed, 11 insertions(+), 3 deletions(-) diff --git a/datasette/database.py b/datasette/database.py index becb552c..707d8f85 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -1,7 +1,6 @@ import asyncio from collections import namedtuple from pathlib import Path -import hashlib import janus import queue import sys @@ -15,6 +14,7 @@ from .utils import ( detect_spatialite, get_all_foreign_keys, get_outbound_foreign_keys, + md5_not_usedforsecurity, sqlite_timelimit, sqlite3, table_columns, @@ -74,7 +74,7 @@ class Database: def color(self): if self.hash: return self.hash[:6] - return hashlib.md5(self.name.encode("utf8")).hexdigest()[:6] + return md5_not_usedforsecurity(self.name)[:6] def suggest_name(self): if self.path: diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index f2cd7eb0..e3637f7a 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -713,7 +713,7 @@ def to_css_class(s): """ if css_class_re.match(s): return s - md5_suffix = hashlib.md5(s.encode("utf8")).hexdigest()[:6] + md5_suffix = md5_not_usedforsecurity(s)[:6] # Strip leading _, - s = s.lstrip("_").lstrip("-") # Replace any whitespace with hyphens @@ -1401,3 +1401,11 @@ def redact_keys(original: dict, key_patterns: Iterable) -> dict: return data return redact(original) + + +def md5_not_usedforsecurity(s): + try: + return hashlib.md5(s.encode("utf8"), usedforsecurity=False).hexdigest() + except TypeError: + # For Python 3.8 which does not support usedforsecurity=False + return hashlib.md5(s.encode("utf8")).hexdigest() From 97de4d6362ce5a6c1e3520ecdc73b305ab269910 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 15 Feb 2024 21:35:49 -0800 Subject: [PATCH 598/952] Use transaction in delete_everything(), closes #2273 --- datasette/utils/internal_db.py | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/datasette/utils/internal_db.py b/datasette/utils/internal_db.py index 2e5ac53b..dd0d3a9d 100644 --- a/datasette/utils/internal_db.py +++ b/datasette/utils/internal_db.py @@ -69,18 +69,20 @@ async def populate_schema_tables(internal_db, db): database_name = db.name def delete_everything(conn): - conn.execute( - "DELETE FROM catalog_tables WHERE database_name = ?", [database_name] - ) - conn.execute( - "DELETE FROM catalog_columns WHERE database_name = ?", [database_name] - ) - conn.execute( - "DELETE FROM catalog_foreign_keys WHERE database_name = ?", [database_name] - ) - conn.execute( - "DELETE FROM catalog_indexes WHERE database_name = ?", [database_name] - ) + with conn: + conn.execute( + "DELETE FROM catalog_tables WHERE database_name = ?", [database_name] + ) + conn.execute( + "DELETE FROM catalog_columns WHERE database_name = ?", [database_name] + ) + conn.execute( + "DELETE FROM catalog_foreign_keys WHERE database_name = ?", + [database_name], + ) + conn.execute( + "DELETE FROM catalog_indexes WHERE database_name = ?", [database_name] + ) await internal_db.execute_write_fn(delete_everything) From 47e29e948b26e8c003a03b4fc46cb635134a3958 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 16 Feb 2024 10:05:18 -0800 Subject: [PATCH 599/952] Better comments in permission_allowed_default() --- datasette/default_permissions.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/datasette/default_permissions.py b/datasette/default_permissions.py index c13f2ed2..757b3a46 100644 --- a/datasette/default_permissions.py +++ b/datasette/default_permissions.py @@ -144,7 +144,7 @@ def permission_allowed_default(datasette, actor, action, resource): if actor and actor.get("id") == "root": return True - # Resolve metadata view permissions + # Resolve view permissions in allow blocks in configuration if action in ( "view-instance", "view-database", @@ -158,7 +158,7 @@ def permission_allowed_default(datasette, actor, action, resource): if result is not None: return result - # Check custom permissions: blocks + # Resolve custom permissions: blocks in configuration result = await _resolve_config_permissions_blocks( datasette, actor, action, resource ) From 232a30459babebece653795d136fb6516444ecf0 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 16 Feb 2024 12:56:39 -0800 Subject: [PATCH 600/952] DATASETTE_TRACE_PLUGINS setting, closes #2274 --- datasette/plugins.py | 24 ++++++++++++++++++++++++ docs/writing_plugins.rst | 24 ++++++++++++++++++++++++ 2 files changed, 48 insertions(+) diff --git a/datasette/plugins.py b/datasette/plugins.py index f7a1905f..3769a209 100644 --- a/datasette/plugins.py +++ b/datasette/plugins.py @@ -1,6 +1,7 @@ import importlib import os import pluggy +from pprint import pprint import sys from . import hookspecs @@ -33,6 +34,29 @@ DEFAULT_PLUGINS = ( pm = pluggy.PluginManager("datasette") pm.add_hookspecs(hookspecs) +DATASETTE_TRACE_PLUGINS = os.environ.get("DATASETTE_TRACE_PLUGINS", None) + + +def before(hook_name, hook_impls, kwargs): + print(file=sys.stderr) + print(f"{hook_name}:", file=sys.stderr) + pprint(kwargs, width=40, indent=4, stream=sys.stderr) + print("Hook implementations:", file=sys.stderr) + pprint(hook_impls, width=40, indent=4, stream=sys.stderr) + + +def after(outcome, hook_name, hook_impls, kwargs): + results = outcome.get_result() + if not isinstance(results, list): + results = [results] + print(f"Results:", file=sys.stderr) + pprint(results, width=40, indent=4, stream=sys.stderr) + + +if DATASETTE_TRACE_PLUGINS: + pm.add_hookcall_monitoring(before, after) + + DATASETTE_LOAD_PLUGINS = os.environ.get("DATASETTE_LOAD_PLUGINS", None) if not hasattr(sys, "_called_from_test") and DATASETTE_LOAD_PLUGINS is None: diff --git a/docs/writing_plugins.rst b/docs/writing_plugins.rst index 5c8bc4c6..2bc6bd24 100644 --- a/docs/writing_plugins.rst +++ b/docs/writing_plugins.rst @@ -7,6 +7,30 @@ You can write one-off plugins that apply to just one Datasette instance, or you Want to start by looking at an example? The `Datasette plugins directory `__ lists more than 90 open source plugins with code you can explore. The :ref:`plugin hooks ` page includes links to example plugins for each of the documented hooks. +.. _writing_plugins_tracing: + +Tracing plugin hooks +-------------------- + +The ``DATASETTE_TRACE_PLUGINS`` environment variable turns on detailed tracing showing exactly which hooks are being run. This can be useful for understanding how Datasette is using your plugin. + +.. code-block:: bash + + DATASETTE_TRACE_PLUGINS=1 datasette mydb.db + +Example output:: + + actor_from_request: + { 'datasette': , + 'request': } + Hook implementations: + [ >, + >, + >] + Results: + [{'id': 'root'}] + + .. _writing_plugins_one_off: Writing one-off plugins From 8bfa3a51c222d653f45fb48ebcb6957a85f9ea6c Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 16 Feb 2024 13:29:39 -0800 Subject: [PATCH 601/952] Consider every plugins opinion in datasette.permission_allowed() Closes #2275, refs #2262 --- datasette/app.py | 14 +++++++++++++- docs/authentication.rst | 17 +++++++++++++++++ 2 files changed, 30 insertions(+), 1 deletion(-) diff --git a/datasette/app.py b/datasette/app.py index d943b97b..8591af6a 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -903,6 +903,8 @@ class Datasette: # Use default from registered permission, if available if default is DEFAULT_NOT_SET and action in self.permissions: default = self.permissions[action].default + opinions = [] + # Every plugin is consulted for their opinion for check in pm.hook.permission_allowed( datasette=self, actor=actor, @@ -911,9 +913,19 @@ class Datasette: ): check = await await_me_maybe(check) if check is not None: - result = check + opinions.append(check) + + result = None + # If any plugin said False it's false - the veto rule + if any(not r for r in opinions): + result = False + elif any(r for r in opinions): + # Otherwise, if any plugin said True it's true + result = True + used_default = False if result is None: + # No plugin expressed an opinion, so use the default result = default used_default = True self._permission_checks.append( diff --git a/docs/authentication.rst b/docs/authentication.rst index 87ee6385..a8dc5637 100644 --- a/docs/authentication.rst +++ b/docs/authentication.rst @@ -71,6 +71,23 @@ Datasette's built-in view permissions (``view-database``, ``view-table`` etc) de Permissions with potentially harmful effects should default to *deny*. Plugin authors should account for this when designing new plugins - for example, the `datasette-upload-csvs `__ plugin defaults to deny so that installations don't accidentally allow unauthenticated users to create new tables by uploading a CSV file. +.. _authentication_permissions_explained: + +How permissions are resolved +---------------------------- + +The :ref:`datasette.permission_allowed(actor, action, resource=None, default=...)` method is called to check if an actor is allowed to perform a specific action. + +This method asks every plugin that implements the :ref:`plugin_hook_permission_allowed` hook if the actor is allowed to perform the action. + +Each plugin can return ``True`` to indicate that the actor is allowed to perform the action, ``False`` if they are not allowed and ``None`` if the plugin has no opinion on the matter. + +``False`` acts as a veto - if any plugin returns ``False`` then the permission check is denied. Otherwise, if any plugin returns ``True`` then the permission check is allowed. + +The ``resource`` argument can be used to specify a specific resource that the action is being performed against. Some permissions, such as ``view-instance``, do not involve a resource. Others such as ``view-database`` have a resource that is a string naming the database. Permissions that take both a database name and the name of a table, view or canned query within that database use a resource that is a tuple of two strings, ``(database_name, resource_name)``. + +Plugins that implement the ``permission_allowed()`` hook can decide if they are going to consider the provided resource or not. + .. _authentication_permissions_allow: Defining permissions with "allow" blocks From 244f3ff83aac19e96fab85a95ddde349079a9827 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 16 Feb 2024 13:39:57 -0800 Subject: [PATCH 602/952] Test demonstrating fix for permisisons bug in #2262 --- tests/test_api_write.py | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/tests/test_api_write.py b/tests/test_api_write.py index 2d127e1a..2aea699b 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -365,6 +365,41 @@ async def test_insert_or_upsert_row_errors( assert before_count == after_count +@pytest.mark.asyncio +@pytest.mark.parametrize("allowed", (True, False)) +async def test_upsert_permissions_per_table(ds_write, allowed): + # https://github.com/simonw/datasette/issues/2262 + token = "dstok_{}".format( + ds_write.sign( + { + "a": "root", + "token": "dstok", + "t": int(time.time()), + "_r": { + "r": { + "data": { + "docs" if allowed else "other": ["ir", "ur"], + } + } + }, + }, + namespace="token", + ) + ) + response = await ds_write.client.post( + "/data/docs/-/upsert", + json={"rows": [{"id": 1, "title": "One"}]}, + headers={ + "Authorization": "Bearer {}".format(token), + }, + ) + if allowed: + assert response.status_code == 200 + assert response.json()["ok"] is True + else: + assert response.status_code == 403 + + @pytest.mark.asyncio @pytest.mark.parametrize( "ignore,replace,expected_rows", From 3a999a85fb431594ccee1adf38721de03de19500 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 16 Feb 2024 13:58:33 -0800 Subject: [PATCH 603/952] Fire insert-rows on /db/-/create if rows were inserted, refs #2260 --- datasette/views/database.py | 13 ++++++- tests/test_api_write.py | 71 +++++++++++++++++++++++++++++-------- 2 files changed, 69 insertions(+), 15 deletions(-) diff --git a/datasette/views/database.py b/datasette/views/database.py index bd55064f..2a8b40cc 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -10,7 +10,7 @@ import re import sqlite_utils import textwrap -from datasette.events import AlterTableEvent, CreateTableEvent +from datasette.events import AlterTableEvent, CreateTableEvent, InsertRowsEvent from datasette.database import QueryInterrupted from datasette.utils import ( add_cors_headers, @@ -1022,6 +1022,17 @@ class TableCreateView(BaseView): request.actor, database=db.name, table=table_name, schema=schema ) ) + if rows: + await self.ds.track_event( + InsertRowsEvent( + request.actor, + database=db.name, + table=table_name, + num_rows=len(rows), + ignore=ignore, + replace=replace, + ) + ) return Response.json(details, status=201) diff --git a/tests/test_api_write.py b/tests/test_api_write.py index 2aea699b..0eb915ba 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -857,13 +857,14 @@ async def test_drop_table(ds_write, scenario): @pytest.mark.asyncio @pytest.mark.parametrize( - "input,expected_status,expected_response", + "input,expected_status,expected_response,expected_events", ( # Permission error with a bad token ( {"table": "bad", "row": {"id": 1}}, 403, {"ok": False, "errors": ["Permission denied"]}, + [], ), # Successful creation with columns: ( @@ -910,6 +911,7 @@ async def test_drop_table(ds_write, scenario): ")" ), }, + ["create-table"], ), # Successful creation with rows: ( @@ -945,6 +947,7 @@ async def test_drop_table(ds_write, scenario): ), "row_count": 2, }, + ["create-table", "insert-rows"], ), # Successful creation with row: ( @@ -973,6 +976,7 @@ async def test_drop_table(ds_write, scenario): ), "row_count": 1, }, + ["create-table", "insert-rows"], ), # Create with row and no primary key ( @@ -992,6 +996,7 @@ async def test_drop_table(ds_write, scenario): "schema": ("CREATE TABLE [four] (\n" " [name] TEXT\n" ")"), "row_count": 1, }, + ["create-table", "insert-rows"], ), # Create table with compound primary key ( @@ -1013,6 +1018,7 @@ async def test_drop_table(ds_write, scenario): ), "row_count": 1, }, + ["create-table", "insert-rows"], ), # Error: Table is required ( @@ -1024,6 +1030,7 @@ async def test_drop_table(ds_write, scenario): "ok": False, "errors": ["Table is required"], }, + [], ), # Error: Invalid table name ( @@ -1036,6 +1043,7 @@ async def test_drop_table(ds_write, scenario): "ok": False, "errors": ["Invalid table name"], }, + [], ), # Error: JSON must be an object ( @@ -1045,6 +1053,7 @@ async def test_drop_table(ds_write, scenario): "ok": False, "errors": ["JSON must be an object"], }, + [], ), # Error: Cannot specify columns with rows or row ( @@ -1058,6 +1067,7 @@ async def test_drop_table(ds_write, scenario): "ok": False, "errors": ["Cannot specify columns with rows or row"], }, + [], ), # Error: columns, rows or row is required ( @@ -1069,6 +1079,7 @@ async def test_drop_table(ds_write, scenario): "ok": False, "errors": ["columns, rows or row is required"], }, + [], ), # Error: columns must be a list ( @@ -1081,6 +1092,7 @@ async def test_drop_table(ds_write, scenario): "ok": False, "errors": ["columns must be a list"], }, + [], ), # Error: columns must be a list of objects ( @@ -1093,6 +1105,7 @@ async def test_drop_table(ds_write, scenario): "ok": False, "errors": ["columns must be a list of objects"], }, + [], ), # Error: Column name is required ( @@ -1105,6 +1118,7 @@ async def test_drop_table(ds_write, scenario): "ok": False, "errors": ["Column name is required"], }, + [], ), # Error: Unsupported column type ( @@ -1117,6 +1131,7 @@ async def test_drop_table(ds_write, scenario): "ok": False, "errors": ["Unsupported column type: bad"], }, + [], ), # Error: Duplicate column name ( @@ -1132,6 +1147,7 @@ async def test_drop_table(ds_write, scenario): "ok": False, "errors": ["Duplicate column name: id"], }, + [], ), # Error: rows must be a list ( @@ -1144,6 +1160,7 @@ async def test_drop_table(ds_write, scenario): "ok": False, "errors": ["rows must be a list"], }, + [], ), # Error: rows must be a list of objects ( @@ -1156,6 +1173,7 @@ async def test_drop_table(ds_write, scenario): "ok": False, "errors": ["rows must be a list of objects"], }, + [], ), # Error: pk must be a string ( @@ -1169,6 +1187,7 @@ async def test_drop_table(ds_write, scenario): "ok": False, "errors": ["pk must be a string"], }, + [], ), # Error: Cannot specify both pk and pks ( @@ -1183,6 +1202,7 @@ async def test_drop_table(ds_write, scenario): "ok": False, "errors": ["Cannot specify both pk and pks"], }, + [], ), # Error: pks must be a list ( @@ -1196,12 +1216,14 @@ async def test_drop_table(ds_write, scenario): "ok": False, "errors": ["pks must be a list"], }, + [], ), # Error: pks must be a list of strings ( {"table": "bad", "row": {"id": 1, "name": "Row 1"}, "pks": [1, 2]}, 400, {"ok": False, "errors": ["pks must be a list of strings"]}, + [], ), # Error: ignore and replace are mutually exclusive ( @@ -1217,6 +1239,7 @@ async def test_drop_table(ds_write, scenario): "ok": False, "errors": ["ignore and replace are mutually exclusive"], }, + [], ), # ignore and replace require row or rows ( @@ -1230,6 +1253,7 @@ async def test_drop_table(ds_write, scenario): "ok": False, "errors": ["ignore and replace require row or rows"], }, + [], ), # ignore and replace require pk or pks ( @@ -1243,6 +1267,7 @@ async def test_drop_table(ds_write, scenario): "ok": False, "errors": ["ignore and replace require pk or pks"], }, + [], ), ( { @@ -1255,10 +1280,14 @@ async def test_drop_table(ds_write, scenario): "ok": False, "errors": ["ignore and replace require pk or pks"], }, + [], ), ), ) -async def test_create_table(ds_write, input, expected_status, expected_response): +async def test_create_table( + ds_write, input, expected_status, expected_response, expected_events +): + ds_write._tracked_events = [] # Special case for expected status of 403 if expected_status == 403: token = "bad_token" @@ -1272,12 +1301,9 @@ async def test_create_table(ds_write, input, expected_status, expected_response) assert response.status_code == expected_status data = response.json() assert data == expected_response - # create-table event - if expected_status == 201: - event = last_event(ds_write) - assert event.name == "create-table" - assert event.actor == {"id": "root", "token": "dstok"} - assert event.schema.startswith("CREATE TABLE ") + # Should have tracked the expected events + events = ds_write._tracked_events + assert [e.name for e in events] == expected_events @pytest.mark.asyncio @@ -1376,6 +1402,8 @@ async def test_create_table_ignore_replace(ds_write, input, expected_rows_after) ) assert first_response.status_code == 201 + ds_write._tracked_events = [] + # Try a second time second_response = await ds_write.client.post( "/data/-/create", @@ -1387,6 +1415,10 @@ async def test_create_table_ignore_replace(ds_write, input, expected_rows_after) rows = await ds_write.client.get("/data/test_insert_replace.json?_shape=array") assert rows.json() == expected_rows_after + # Check it fired the right events + event_names = [e.name for e in ds_write._tracked_events] + assert event_names == ["insert-rows"] + @pytest.mark.asyncio async def test_create_table_error_if_pk_changed(ds_write): @@ -1471,6 +1503,7 @@ async def test_method_not_allowed(ds_write, path): @pytest.mark.asyncio async def test_create_uses_alter_by_default_for_new_table(ds_write): + ds_write._tracked_events = [] token = write_token(ds_write) response = await ds_write.client.post( "/data/-/create", @@ -1490,8 +1523,8 @@ async def test_create_uses_alter_by_default_for_new_table(ds_write): headers=_headers(token), ) assert response.status_code == 201 - event = last_event(ds_write) - assert event.name == "create-table" + event_names = [e.name for e in ds_write._tracked_events] + assert event_names == ["create-table", "insert-rows"] @pytest.mark.asyncio @@ -1517,6 +1550,8 @@ async def test_create_using_alter_against_existing_table( headers=_headers(token), ) assert response.status_code == 201 + + ds_write._tracked_events = [] # Now try to insert more rows using /-/create with alter=True response2 = await ds_write.client.post( "/data/-/create", @@ -1536,8 +1571,16 @@ async def test_create_using_alter_against_existing_table( } else: assert response2.status_code == 201 + + event_names = [e.name for e in ds_write._tracked_events] + assert event_names == ["alter-table", "insert-rows"] + # It should have altered the table - event = last_event(ds_write) - assert event.name == "alter-table" - assert "extra" not in event.before_schema - assert "extra" in event.after_schema + alter_event = ds_write._tracked_events[0] + assert alter_event.name == "alter-table" + assert "extra" not in alter_event.before_schema + assert "extra" in alter_event.after_schema + + insert_rows_event = ds_write._tracked_events[1] + assert insert_rows_event.name == "insert-rows" + assert insert_rows_event.num_rows == 1 From 9906f937d92c79dcc457cb057d7222ed70aef0e0 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 16 Feb 2024 14:32:47 -0800 Subject: [PATCH 604/952] Release 1.0a9 Refs #2101, #2260, #2262, #2265, #2270, #2273, #2274, #2275 Closes #2276 --- datasette/version.py | 2 +- docs/changelog.rst | 45 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+), 1 deletion(-) diff --git a/datasette/version.py b/datasette/version.py index e43b9918..f5e07ac8 100644 --- a/datasette/version.py +++ b/datasette/version.py @@ -1,2 +1,2 @@ -__version__ = "1.0a8" +__version__ = "1.0a9" __version_info__ = tuple(__version__.split(".")) diff --git a/docs/changelog.rst b/docs/changelog.rst index d164f71d..e567f422 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -4,6 +4,51 @@ Changelog ========= +.. _v1_0_a9: + +1.0a9 (2024-02-16) +------------------ + +This alpha release adds basic alter table support to the Datasette Write API and fixes a permissions bug relating to the ``/upsert`` API endpoint. + +Alter table support for create, insert, upsert and update +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The :ref:`JSON write API ` can now be used to apply simple alter table schema changes, provided the acting actor has the new :ref:`permissions_alter_table` permission. (:issue:`2101`) + +The only alter operation supported so far is adding new columns to an existing table. + +* The :ref:`/db/-/create ` API now adds new columns during large operations to create a table based on incoming example ``"rows"``, in the case where one of the later rows includes columns that were not present in the earlier batches. This requires the ``create-table`` but not the ``alter-table`` permission. +* When ``/db/-/create`` is called with rows in a situation where the table may have been already created, an ``"alter": true`` key can be included to indicate that any missing columns from the new rows should be added to the table. This requires the ``alter-table`` permission. +* :ref:`/db/table/-/insert ` and :ref:`/db/table/-/upsert ` and :ref:`/db/table/row-pks/-/update ` all now also accept ``"alter": true``, depending on the ``alter-table`` permission. + +Operations that alter a table now fire the new :ref:`alter-table event `. + +Permissions fix for the upsert API +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The :ref:`/database/table/-/upsert API ` had a minor permissions bug, only affecting Datasette instances that had configured the ``insert-row`` and ``update-row`` permissions to apply to a specific table rather than the database or instance as a whole. Full details in issue :issue:`2262`. + +To avoid similar mistakes in the future the :ref:`datasette.permission_allowed() ` method now specifies ``default=`` as a keyword-only argument. + +Permission checks now consider opinions from every plugin +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The :ref:`datasette.permission_allowed() ` method previously consulted every plugin that implemented the :ref:`permission_allowed() ` plugin hook and obeyed the opinion of the last plugin to return a value. (:issue:`2275`) + +Datasette now consults every plugin and checks to see if any of them returned ``False`` (the veto rule), and if none of them did, it then checks to see if any of them returned ``True``. + +This is explained at length in the new documentation covering :ref:`authentication_permissions_explained`. + +Other changes +~~~~~~~~~~~~~ + +- The new :ref:`DATASETTE_TRACE_PLUGINS=1 environment variable ` turns on detailed trace output for every executed plugin hook, useful for debugging and understanding how the plugin system works at a low level. (:issue:`2274`) +- Datasette on Python 3.9 or above marks its non-cryptographic uses of the MD5 hash function as ``usedforsecurity=False``, for compatibility with FIPS systems. (:issue:`2270`) +- SQL relating to :ref:`internals_internal` now executes inside a transaction, avoiding a potential database locked error. (:issue:`2273`) +- The ``/-/threads`` debug page now identifies the database in the name associated with each dedicated write thread. (:issue:`2265`) +- The ``/db/-/create`` API now fires a ``insert-rows`` event if rows were inserted after the table was created. (:issue:`2260`) + .. _v1_0_a8: 1.0a8 (2024-02-07) From e1c80efff8f4b0a53619546bb03e6dfd6cb42a32 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 16 Feb 2024 14:43:36 -0800 Subject: [PATCH 605/952] Note about activating alpha documentation versions on ReadTheDocs --- docs/contributing.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/contributing.rst b/docs/contributing.rst index ef022a4d..b678e637 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -254,6 +254,7 @@ Datasette releases are performed using tags. When a new release is published on * Re-point the "latest" tag on Docker Hub to the new image * Build a wheel bundle of the underlying Python source code * Push that new wheel up to PyPI: https://pypi.org/project/datasette/ +* If the release is an alpha, navigate to https://readthedocs.org/projects/datasette/versions/ and search for the tag name in the "Activate a version" filter, then mark that version as "active" to ensure it will appear on the public ReadTheDocs documentation site. To deploy new releases you will need to have push access to the main Datasette GitHub repository. From 5e0e440f2c8a0771b761b02801456e55e95e2a04 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sat, 17 Feb 2024 20:28:15 -0800 Subject: [PATCH 606/952] database.execute_write_fn(transaction=True) parameter, closes #2277 --- datasette/database.py | 31 +++++++++++++++++++++++-------- docs/internals.rst | 15 ++++++++++----- tests/test_internals_database.py | 27 +++++++++++++++++++++++++++ 3 files changed, 60 insertions(+), 13 deletions(-) diff --git a/datasette/database.py b/datasette/database.py index 707d8f85..d34aac73 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -179,17 +179,25 @@ class Database: # Threaded mode - send to write thread return await self._send_to_write_thread(fn, isolated_connection=True) - async def execute_write_fn(self, fn, block=True): + async def execute_write_fn(self, fn, block=True, transaction=True): if self.ds.executor is None: # non-threaded mode if self._write_connection is None: self._write_connection = self.connect(write=True) self.ds._prepare_connection(self._write_connection, self.name) - return fn(self._write_connection) + if transaction: + with self._write_connection: + return fn(self._write_connection) + else: + return fn(self._write_connection) else: - return await self._send_to_write_thread(fn, block) + return await self._send_to_write_thread( + fn, block=block, transaction=transaction + ) - async def _send_to_write_thread(self, fn, block=True, isolated_connection=False): + async def _send_to_write_thread( + self, fn, block=True, isolated_connection=False, transaction=True + ): if self._write_queue is None: self._write_queue = queue.Queue() if self._write_thread is None: @@ -202,7 +210,9 @@ class Database: self._write_thread.start() task_id = uuid.uuid5(uuid.NAMESPACE_DNS, "datasette.io") reply_queue = janus.Queue() - self._write_queue.put(WriteTask(fn, task_id, reply_queue, isolated_connection)) + self._write_queue.put( + WriteTask(fn, task_id, reply_queue, isolated_connection, transaction) + ) if block: result = await reply_queue.async_q.get() if isinstance(result, Exception): @@ -244,7 +254,11 @@ class Database: pass else: try: - result = task.fn(conn) + if task.transaction: + with conn: + result = task.fn(conn) + else: + result = task.fn(conn) except Exception as e: sys.stderr.write("{}\n".format(e)) sys.stderr.flush() @@ -554,13 +568,14 @@ class Database: class WriteTask: - __slots__ = ("fn", "task_id", "reply_queue", "isolated_connection") + __slots__ = ("fn", "task_id", "reply_queue", "isolated_connection", "transaction") - def __init__(self, fn, task_id, reply_queue, isolated_connection): + def __init__(self, fn, task_id, reply_queue, isolated_connection, transaction): self.fn = fn self.task_id = task_id self.reply_queue = reply_queue self.isolated_connection = isolated_connection + self.transaction = transaction class QueryInterrupted(Exception): diff --git a/docs/internals.rst b/docs/internals.rst index bd7a70b5..6ca62423 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -1010,7 +1010,9 @@ You can pass additional SQL parameters as a tuple or dictionary. The method will block until the operation is completed, and the return value will be the return from calling ``conn.execute(...)`` using the underlying ``sqlite3`` Python library. -If you pass ``block=False`` this behaviour changes to "fire and forget" - queries will be added to the write queue and executed in a separate thread while your code can continue to do other things. The method will return a UUID representing the queued task. +If you pass ``block=False`` this behavior changes to "fire and forget" - queries will be added to the write queue and executed in a separate thread while your code can continue to do other things. The method will return a UUID representing the queued task. + +Each call to ``execute_write()`` will be executed inside a transaction. .. _database_execute_write_script: @@ -1019,6 +1021,8 @@ await db.execute_write_script(sql, block=True) Like ``execute_write()`` but can be used to send multiple SQL statements in a single string separated by semicolons, using the ``sqlite3`` `conn.executescript() `__ method. +Each call to ``execute_write_script()`` will be executed inside a transaction. + .. _database_execute_write_many: await db.execute_write_many(sql, params_seq, block=True) @@ -1033,10 +1037,12 @@ Like ``execute_write()`` but uses the ``sqlite3`` `conn.executemany() 5") - conn.commit() return conn.execute( "select count(*) from some_table" ).fetchone()[0] @@ -1069,7 +1074,7 @@ The value returned from ``await database.execute_write_fn(...)`` will be the ret If your function raises an exception that exception will be propagated up to the ``await`` line. -If you see ``OperationalError: database table is locked`` errors you should check that you remembered to explicitly call ``conn.commit()`` in your write function. +By default your function will be executed inside a transaction. You can pass ``transaction=False`` to disable this behavior, though if you do that you should be careful to manually apply transactions - ideally using the ``with conn:`` pattern, or you may see ``OperationalError: database table is locked`` errors. If you specify ``block=False`` the method becomes fire-and-forget, queueing your function to be executed and then allowing your code after the call to ``.execute_write_fn()`` to continue running while the underlying thread waits for an opportunity to run your function. A UUID representing the queued task will be returned. Any exceptions in your code will be silently swallowed. diff --git a/tests/test_internals_database.py b/tests/test_internals_database.py index dd68a6cb..57e75046 100644 --- a/tests/test_internals_database.py +++ b/tests/test_internals_database.py @@ -66,6 +66,33 @@ async def test_execute_fn(db): assert 2 == await db.execute_fn(get_1_plus_1) +@pytest.mark.asyncio +async def test_execute_fn_transaction_false(): + datasette = Datasette(memory=True) + db = datasette.add_memory_database("test_execute_fn_transaction_false") + + def run(conn): + try: + with conn: + conn.execute("create table foo (id integer primary key)") + conn.execute("insert into foo (id) values (44)") + # Table should exist + assert ( + conn.execute( + 'select count(*) from sqlite_master where name = "foo"' + ).fetchone()[0] + == 1 + ) + assert conn.execute("select id from foo").fetchall()[0][0] == 44 + raise ValueError("Cancel commit") + except ValueError: + pass + # Row should NOT exist + assert conn.execute("select count(*) from foo").fetchone()[0] == 0 + + await db.execute_write_fn(run, transaction=False) + + @pytest.mark.parametrize( "tables,exists", ( From 10f9ba1a0050724ba47a089861606bef58a4087f Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sat, 17 Feb 2024 20:51:19 -0800 Subject: [PATCH 607/952] Take advantage of execute_write_fn(transaction=True) A bunch of places no longer need to do manual transaction handling thanks to this change. Refs #2277 --- datasette/database.py | 9 +++------ datasette/utils/internal_db.py | 27 +++++++++++++-------------- tests/test_internals_database.py | 10 ++++------ 3 files changed, 20 insertions(+), 26 deletions(-) diff --git a/datasette/database.py b/datasette/database.py index d34aac73..4e590d3a 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -123,8 +123,7 @@ class Database: async def execute_write(self, sql, params=None, block=True): def _inner(conn): - with conn: - return conn.execute(sql, params or []) + return conn.execute(sql, params or []) with trace("sql", database=self.name, sql=sql.strip(), params=params): results = await self.execute_write_fn(_inner, block=block) @@ -132,8 +131,7 @@ class Database: async def execute_write_script(self, sql, block=True): def _inner(conn): - with conn: - return conn.executescript(sql) + return conn.executescript(sql) with trace("sql", database=self.name, sql=sql.strip(), executescript=True): results = await self.execute_write_fn(_inner, block=block) @@ -149,8 +147,7 @@ class Database: count += 1 yield param - with conn: - return conn.executemany(sql, count_params(params_seq)), count + return conn.executemany(sql, count_params(params_seq)), count with trace( "sql", database=self.name, sql=sql.strip(), executemany=True diff --git a/datasette/utils/internal_db.py b/datasette/utils/internal_db.py index dd0d3a9d..dbfcceb4 100644 --- a/datasette/utils/internal_db.py +++ b/datasette/utils/internal_db.py @@ -69,20 +69,19 @@ async def populate_schema_tables(internal_db, db): database_name = db.name def delete_everything(conn): - with conn: - conn.execute( - "DELETE FROM catalog_tables WHERE database_name = ?", [database_name] - ) - conn.execute( - "DELETE FROM catalog_columns WHERE database_name = ?", [database_name] - ) - conn.execute( - "DELETE FROM catalog_foreign_keys WHERE database_name = ?", - [database_name], - ) - conn.execute( - "DELETE FROM catalog_indexes WHERE database_name = ?", [database_name] - ) + conn.execute( + "DELETE FROM catalog_tables WHERE database_name = ?", [database_name] + ) + conn.execute( + "DELETE FROM catalog_columns WHERE database_name = ?", [database_name] + ) + conn.execute( + "DELETE FROM catalog_foreign_keys WHERE database_name = ?", + [database_name], + ) + conn.execute( + "DELETE FROM catalog_indexes WHERE database_name = ?", [database_name] + ) await internal_db.execute_write_fn(delete_everything) diff --git a/tests/test_internals_database.py b/tests/test_internals_database.py index 57e75046..1c155cf3 100644 --- a/tests/test_internals_database.py +++ b/tests/test_internals_database.py @@ -501,9 +501,8 @@ async def test_execute_write_has_correctly_prepared_connection(db): @pytest.mark.asyncio async def test_execute_write_fn_block_false(db): def write_fn(conn): - with conn: - conn.execute("delete from roadside_attractions where pk = 1;") - row = conn.execute("select count(*) from roadside_attractions").fetchone() + conn.execute("delete from roadside_attractions where pk = 1;") + row = conn.execute("select count(*) from roadside_attractions").fetchone() return row[0] task_id = await db.execute_write_fn(write_fn, block=False) @@ -513,9 +512,8 @@ async def test_execute_write_fn_block_false(db): @pytest.mark.asyncio async def test_execute_write_fn_block_true(db): def write_fn(conn): - with conn: - conn.execute("delete from roadside_attractions where pk = 1;") - row = conn.execute("select count(*) from roadside_attractions").fetchone() + conn.execute("delete from roadside_attractions where pk = 1;") + row = conn.execute("select count(*) from roadside_attractions").fetchone() return row[0] new_count = await db.execute_write_fn(write_fn) From a4fa1ef3bd6a6117118b5cdd64aca2308c21604b Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sat, 17 Feb 2024 20:56:15 -0800 Subject: [PATCH 608/952] Release 1.0a10 Refs #2277 --- datasette/version.py | 2 +- docs/changelog.rst | 11 +++++++++++ 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/datasette/version.py b/datasette/version.py index f5e07ac8..809c434f 100644 --- a/datasette/version.py +++ b/datasette/version.py @@ -1,2 +1,2 @@ -__version__ = "1.0a9" +__version__ = "1.0a10" __version_info__ = tuple(__version__.split(".")) diff --git a/docs/changelog.rst b/docs/changelog.rst index e567f422..92f198af 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -4,6 +4,17 @@ Changelog ========= +.. _v1_0_a10: + +1.0a10 (2024-02-17) +------------------- + +The only changes in this alpha correspond to the way Datasette handles database transactions. (:issue:`2277`) + +- The :ref:`database.execute_write_fn() ` method has a new ``transaction=True`` parameter. This defaults to ``True`` which means all functions executed using this method are now automatically wrapped in a transaction - previously the functions needed to roll transaction handling on their own, and many did not. +- Pass ``transaction=False`` to ``execute_write_fn()`` if you want to manually handle transactions in your function. +- Several internal Datasette features, including parts of the :ref:`JSON write API `, had been failing to wrap their operations in a transaction. This has been fixed by the new ``transaction=True`` default. + .. _v1_0_a9: 1.0a9 (2024-02-16) From 81629dbeffb5cee9086bc956ce3a9ab7d051f4d1 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sat, 17 Feb 2024 21:03:41 -0800 Subject: [PATCH 609/952] Upgrade GitHub Actions, including PyPI publishing --- .github/workflows/publish.yml | 60 ++++++++++++++--------------------- .github/workflows/test.yml | 13 +++----- 2 files changed, 27 insertions(+), 46 deletions(-) diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 64a03a77..55fc0eb9 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -12,20 +12,15 @@ jobs: runs-on: ubuntu-latest strategy: matrix: - python-version: ["3.8", "3.9", "3.10", "3.11"] + python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v4 + uses: actions/setup-python@v5 with: python-version: ${{ matrix.python-version }} - - uses: actions/cache@v3 - name: Configure pip caching - with: - path: ~/.cache/pip - key: ${{ runner.os }}-pip-${{ hashFiles('**/setup.py') }} - restore-keys: | - ${{ runner.os }}-pip- + cache: pip + cache-dependency-path: setup.py - name: Install dependencies run: | pip install -e '.[test]' @@ -36,47 +31,38 @@ jobs: deploy: runs-on: ubuntu-latest needs: [test] + environment: release + permissions: + id-token: write steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 - name: Set up Python - uses: actions/setup-python@v4 + uses: actions/setup-python@v5 with: - python-version: '3.11' - - uses: actions/cache@v3 - name: Configure pip caching - with: - path: ~/.cache/pip - key: ${{ runner.os }}-publish-pip-${{ hashFiles('**/setup.py') }} - restore-keys: | - ${{ runner.os }}-publish-pip- + python-version: '3.12' + cache: pip + cache-dependency-path: setup.py - name: Install dependencies run: | - pip install setuptools wheel twine - - name: Publish - env: - TWINE_USERNAME: __token__ - TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }} + pip install setuptools wheel build + - name: Build run: | - python setup.py sdist bdist_wheel - twine upload dist/* + python -m build + - name: Publish + uses: pypa/gh-action-pypi-publish@release/v1 deploy_static_docs: runs-on: ubuntu-latest needs: [deploy] if: "!github.event.release.prerelease" steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v4 - name: Set up Python - uses: actions/setup-python@v2 + uses: actions/setup-python@v5 with: python-version: '3.9' - - uses: actions/cache@v2 - name: Configure pip caching - with: - path: ~/.cache/pip - key: ${{ runner.os }}-publish-pip-${{ hashFiles('**/setup.py') }} - restore-keys: | - ${{ runner.os }}-publish-pip- + cache: pip + cache-dependency-path: setup.py - name: Install dependencies run: | python -m pip install -e .[docs] @@ -105,7 +91,7 @@ jobs: needs: [deploy] if: "!github.event.release.prerelease" steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v4 - name: Build and push to Docker Hub env: DOCKER_USER: ${{ secrets.DOCKER_USER }} diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 656b0b1c..3ac8756d 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -12,19 +12,14 @@ jobs: matrix: python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v4 + uses: actions/setup-python@v5 with: python-version: ${{ matrix.python-version }} allow-prereleases: true - - uses: actions/cache@v3 - name: Configure pip caching - with: - path: ~/.cache/pip - key: ${{ runner.os }}-pip-${{ hashFiles('**/setup.py') }} - restore-keys: | - ${{ runner.os }}-pip- + cache: pip + cache-dependency-path: setup.py - name: Build extension for --load-extension test run: |- (cd tests && gcc ext.c -fPIC -shared -o ext.so) From 3856a8cb244f1338d2c4bceb76a510022d88ade5 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 19 Feb 2024 12:51:14 -0800 Subject: [PATCH 610/952] Consistent Permission denied:, refs #2279 --- datasette/views/database.py | 6 +++--- tests/test_api_write.py | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/datasette/views/database.py b/datasette/views/database.py index 2a8b40cc..56fc6f8c 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -860,7 +860,7 @@ class TableCreateView(BaseView): if not await self.ds.permission_allowed( request.actor, "update-row", resource=database_name ): - return _error(["Permission denied - need update-row"], 403) + return _error(["Permission denied: need update-row"], 403) table_name = data.get("table") if not table_name: @@ -884,7 +884,7 @@ class TableCreateView(BaseView): if not await self.ds.permission_allowed( request.actor, "insert-row", resource=database_name ): - return _error(["Permission denied - need insert-row"], 403) + return _error(["Permission denied: need insert-row"], 403) alter = False if rows or row: @@ -897,7 +897,7 @@ class TableCreateView(BaseView): if not await self.ds.permission_allowed( request.actor, "alter-table", resource=database_name ): - return _error(["Permission denied - need alter-table"], 403) + return _error(["Permission denied: need alter-table"], 403) alter = True if columns: diff --git a/tests/test_api_write.py b/tests/test_api_write.py index 0eb915ba..634f5ee9 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -1316,7 +1316,7 @@ async def test_create_table( ["create-table"], {"table": "t", "rows": [{"name": "c"}]}, 403, - ["Permission denied - need insert-row"], + ["Permission denied: need insert-row"], ), # This should work: ( @@ -1330,7 +1330,7 @@ async def test_create_table( ["create-table", "insert-row"], {"table": "t", "rows": [{"id": 1}], "pk": "id", "replace": True}, 403, - ["Permission denied - need update-row"], + ["Permission denied: need update-row"], ), ), ) @@ -1567,7 +1567,7 @@ async def test_create_using_alter_against_existing_table( assert response2.status_code == 403 assert response2.json() == { "ok": False, - "errors": ["Permission denied - need alter-table"], + "errors": ["Permission denied: need alter-table"], } else: assert response2.status_code == 201 From b36a2d8f4b566a3a4902cdaa7a549241e0e8c881 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 19 Feb 2024 12:55:51 -0800 Subject: [PATCH 611/952] Require update-row to use insert replace, closes #2279 --- datasette/views/table.py | 5 +++++ docs/json_api.rst | 4 ++-- tests/test_api_write.py | 8 ++++++++ 3 files changed, 15 insertions(+), 2 deletions(-) diff --git a/datasette/views/table.py b/datasette/views/table.py index 1c187692..6d0d9885 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -485,6 +485,11 @@ class TableInsertView(BaseView): if upsert and (ignore or replace): return _error(["Upsert does not support ignore or replace"], 400) + if replace and not await self.ds.permission_allowed( + request.actor, "update-row", resource=(database_name, table_name) + ): + return _error(['Permission denied: need update-row to use "replace"'], 403) + initial_schema = None if alter: # Must have alter-table permission diff --git a/docs/json_api.rst b/docs/json_api.rst index c401d97e..366f74b2 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -616,7 +616,7 @@ Pass ``"ignore": true`` to ignore these errors and insert the other rows: "ignore": true } -Or you can pass ``"replace": true`` to replace any rows with conflicting primary keys with the new values. +Or you can pass ``"replace": true`` to replace any rows with conflicting primary keys with the new values. This requires the :ref:`permissions_update_row` permission. Pass ``"alter: true`` to automatically add any missing columns to the table. This requires the :ref:`permissions_alter_table` permission. @@ -854,7 +854,7 @@ The JSON here describes the table that will be created: * ``pks`` can be used instead of ``pk`` to create a compound primary key. It should be a JSON list of column names to use in that primary key. * ``ignore`` can be set to ``true`` to ignore existing rows by primary key if the table already exists. -* ``replace`` can be set to ``true`` to replace existing rows by primary key if the table already exists. +* ``replace`` can be set to ``true`` to replace existing rows by primary key if the table already exists. This requires the :ref:`permissions_update_row` permission. * ``alter`` can be set to ``true`` if you want to automatically add any missing columns to the table. This requires the :ref:`permissions_alter_table` permission. If the table is successfully created this will return a ``201`` status code and the following response: diff --git a/tests/test_api_write.py b/tests/test_api_write.py index 634f5ee9..6a7ddeb6 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -221,6 +221,14 @@ async def test_insert_rows(ds_write, return_rows): 400, ['Cannot use "ignore" and "replace" at the same time'], ), + ( + # Replace is not allowed if you don't have update-row + "/data/docs/-/insert", + {"rows": [{"title": "Test"}], "replace": True}, + "insert-but-not-update", + 403, + ['Permission denied: need update-row to use "replace"'], + ), ( "/data/docs/-/insert", {"rows": [{"title": "Test"}], "invalid_param": True}, From 392ca2e24cc93a3918d07718f40524857d626d14 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 19 Feb 2024 13:40:48 -0800 Subject: [PATCH 612/952] Improvements to table column cog menu display, closes #2263 - Repositions if menu would cause a horizontal scrollbar - Arrow tip on menu now attempts to align with cog icon on column --- datasette/static/table.js | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/datasette/static/table.js b/datasette/static/table.js index 778457c5..0c54a472 100644 --- a/datasette/static/table.js +++ b/datasette/static/table.js @@ -217,6 +217,17 @@ const initDatasetteTable = function (manager) { menuList.appendChild(menuItem); }); + // Measure width of menu and adjust position if too far right + const menuWidth = menu.offsetWidth; + const windowWidth = window.innerWidth; + if (menuLeft + menuWidth > windowWidth) { + menu.style.left = windowWidth - menuWidth - 20 + "px"; + } + // Align menu .hook arrow with the column cog icon + const hook = menu.querySelector('.hook'); + const icon = th.querySelector('.dropdown-menu-icon'); + const iconRect = icon.getBoundingClientRect(); + hook.style.left = (iconRect.left - menuLeft + 1) + 'px'; } var svg = document.createElement("div"); From 27409a78929b4baa017cce2cc0ca636603ed6d37 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 19 Feb 2024 14:01:55 -0800 Subject: [PATCH 613/952] Fix for hook position in wide column names, refs #2263 --- datasette/static/table.js | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/datasette/static/table.js b/datasette/static/table.js index 0c54a472..4f81b2e5 100644 --- a/datasette/static/table.js +++ b/datasette/static/table.js @@ -227,7 +227,15 @@ const initDatasetteTable = function (manager) { const hook = menu.querySelector('.hook'); const icon = th.querySelector('.dropdown-menu-icon'); const iconRect = icon.getBoundingClientRect(); - hook.style.left = (iconRect.left - menuLeft + 1) + 'px'; + const hookLeft = (iconRect.left - menuLeft + 1) + 'px'; + hook.style.left = hookLeft; + // Move the whole menu right if the hook is too far right + const menuRect = menu.getBoundingClientRect(); + if (iconRect.right > menuRect.right) { + menu.style.left = (iconRect.right - menuWidth) + 'px'; + // And move hook tip as well + hook.style.left = (menuWidth - 13) + 'px'; + } } var svg = document.createElement("div"); From 26300738e3c6e7ad515bd513063f57249a05000a Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Mon, 19 Feb 2024 14:17:37 -0800 Subject: [PATCH 614/952] Fixes for permissions debug page, closes #2278 --- datasette/templates/permissions_debug.html | 10 +++++----- datasette/views/special.py | 17 +++++++++-------- tests/test_permissions.py | 8 ++++++++ 3 files changed, 22 insertions(+), 13 deletions(-) diff --git a/datasette/templates/permissions_debug.html b/datasette/templates/permissions_debug.html index 36a12acc..5a5c1aa6 100644 --- a/datasette/templates/permissions_debug.html +++ b/datasette/templates/permissions_debug.html @@ -57,7 +57,7 @@ textarea {

@@ -71,19 +71,19 @@ textarea { +{% endif %} + {% endblock %} diff --git a/datasette/url_builder.py b/datasette/url_builder.py index 9c6bbde0..16b3d42b 100644 --- a/datasette/url_builder.py +++ b/datasette/url_builder.py @@ -31,6 +31,12 @@ class Urls: db = self.ds.get_database(database) return self.path(tilde_encode(db.route), format=format) + def database_query(self, database, sql, format=None): + path = f"{self.database(database)}/-/query?" + urllib.parse.urlencode( + {"sql": sql} + ) + return self.path(path, format=format) + def table(self, database, table, format=None): path = f"{self.database(database)}/{tilde_encode(table)}" if format is not None: diff --git a/datasette/views/table.py b/datasette/views/table.py index d71efeb0..ea044b36 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -929,6 +929,7 @@ async def table_view_traced(datasette, request): database=resolved.db.name, table=resolved.table, ), + count_limit=resolved.db.count_limit, ), request=request, view_name="table", @@ -1280,6 +1281,9 @@ async def table_view_data( if extra_extras: extras.update(extra_extras) + async def extra_count_sql(): + return count_sql + async def extra_count(): "Total count of rows matching these filters" # Calculate the total count for this query @@ -1299,8 +1303,11 @@ async def table_view_data( # Otherwise run a select count(*) ... if count_sql and count is None and not nocount: + count_sql_limited = ( + f"select count(*) from (select * {from_sql} limit 10001)" + ) try: - count_rows = list(await db.execute(count_sql, from_sql_params)) + count_rows = list(await db.execute(count_sql_limited, from_sql_params)) count = count_rows[0][0] except QueryInterrupted: pass @@ -1615,6 +1622,7 @@ async def table_view_data( "facet_results", "facets_timed_out", "count", + "count_sql", "human_description_en", "next_url", "metadata", @@ -1647,6 +1655,7 @@ async def table_view_data( registry = Registry( extra_count, + extra_count_sql, extra_facet_results, extra_facets_timed_out, extra_suggested_facets, From dc288056b81a3635bdb02a6d0121887db2720e5e Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 21 Aug 2024 19:56:02 -0700 Subject: [PATCH 713/952] Better handling of errors for count all button, refs #2408 --- datasette/templates/table.html | 29 +++++++++++++++++++---------- 1 file changed, 19 insertions(+), 10 deletions(-) diff --git a/datasette/templates/table.html b/datasette/templates/table.html index 187f0143..7246ff5d 100644 --- a/datasette/templates/table.html +++ b/datasette/templates/table.html @@ -42,7 +42,7 @@ {% if count or human_description_en %}

{% if count == count_limit + 1 %}>{{ "{:,}".format(count_limit) }} rows - {% if allow_execute_sql and query.sql %} count all rows{% endif %} + {% if allow_execute_sql and query.sql %} count all{% endif %} {% elif count or count == 0 %}{{ "{:,}".format(count) }} row{% if count == 1 %}{% else %}s{% endif %}{% endif %} {% if human_description_en %}{{ human_description_en }}{% endif %}

@@ -180,7 +180,7 @@ document.addEventListener('DOMContentLoaded', function() { const countLink = document.querySelector('a.count-sql'); if (countLink) { - countLink.addEventListener('click', function(ev) { + countLink.addEventListener('click', async function(ev) { ev.preventDefault(); // Replace countLink with span with same style attribute const span = document.createElement('span'); @@ -189,14 +189,23 @@ document.addEventListener('DOMContentLoaded', function() { countLink.replaceWith(span); countLink.setAttribute('disabled', 'disabled'); let url = countLink.href.replace(/(\?|$)/, '.json$1'); - fetch(url) - .then(response => response.json()) - .then(data => { - const count = data['rows'][0]['count(*)']; - const formattedCount = count.toLocaleString(); - span.closest('h3').textContent = formattedCount + ' rows'; - }) - .catch(error => countLink.textContent = 'error'); + try { + const response = await fetch(url); + console.log({response}); + const data = await response.json(); + console.log({data}); + if (!response.ok) { + console.log('throw error'); + throw new Error(data.title || data.error); + } + const count = data['rows'][0]['count(*)']; + const formattedCount = count.toLocaleString(); + span.closest('h3').textContent = formattedCount + ' rows'; + } catch (error) { + console.log('Update', span, 'with error message', error); + span.textContent = error.message; + span.style.color = 'red'; + } }); } }); From 92c4d41ca605e0837a2711ee52fde9cf1eea74d0 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 1 Sep 2024 17:20:41 -0700 Subject: [PATCH 714/952] results.dicts() method, closes #2414 --- datasette/database.py | 3 +++ datasette/views/row.py | 3 +-- datasette/views/table.py | 2 +- docs/internals.rst | 3 +++ tests/test_api_write.py | 23 +++++++++-------------- tests/test_internals_database.py | 11 +++++++++++ 6 files changed, 28 insertions(+), 17 deletions(-) diff --git a/datasette/database.py b/datasette/database.py index da0ab1de..a2e899bc 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -677,6 +677,9 @@ class Results: else: raise MultipleValues + def dicts(self): + return [dict(row) for row in self.rows] + def __iter__(self): return iter(self.rows) diff --git a/datasette/views/row.py b/datasette/views/row.py index d802994e..f374fd94 100644 --- a/datasette/views/row.py +++ b/datasette/views/row.py @@ -277,8 +277,7 @@ class RowUpdateView(BaseView): results = await resolved.db.execute( resolved.sql, resolved.params, truncate=True ) - rows = list(results.rows) - result["row"] = dict(rows[0]) + result["row"] = results.dicts()[0] await self.ds.track_event( UpdateRowEvent( diff --git a/datasette/views/table.py b/datasette/views/table.py index ea044b36..82dab613 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -558,7 +558,7 @@ class TableInsertView(BaseView): ), args, ) - result["rows"] = [dict(r) for r in fetched_rows.rows] + result["rows"] = fetched_rows.dicts() else: result["rows"] = rows # We track the number of rows requested, but do not attempt to show which were actually diff --git a/docs/internals.rst b/docs/internals.rst index 4289c815..facbc224 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -1093,6 +1093,9 @@ The ``Results`` object also has the following properties and methods: ``.rows`` - list of ``sqlite3.Row`` This property provides direct access to the list of rows returned by the database. You can access specific rows by index using ``results.rows[0]``. +``.dicts()`` - list of ``dict`` + This method returns a list of Python dictionaries, one for each row. + ``.first()`` - row or None Returns the first row in the results, or ``None`` if no rows were returned. diff --git a/tests/test_api_write.py b/tests/test_api_write.py index 9c2b9b45..04e61261 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -58,8 +58,8 @@ async def test_insert_row(ds_write, content_type): assert response.status_code == 201 assert response.json()["ok"] is True assert response.json()["rows"] == [expected_row] - rows = (await ds_write.get_database("data").execute("select * from docs")).rows - assert dict(rows[0]) == expected_row + rows = (await ds_write.get_database("data").execute("select * from docs")).dicts() + assert rows[0] == expected_row # Analytics event event = last_event(ds_write) assert event.name == "insert-rows" @@ -118,12 +118,9 @@ async def test_insert_rows(ds_write, return_rows): assert not event.ignore assert not event.replace - actual_rows = [ - dict(r) - for r in ( - await ds_write.get_database("data").execute("select * from docs") - ).rows - ] + actual_rows = ( + await ds_write.get_database("data").execute("select * from docs") + ).dicts() assert len(actual_rows) == 20 assert actual_rows == [ {"id": i + 1, "title": "Test {}".format(i), "score": 1.0, "age": 5} @@ -469,12 +466,10 @@ async def test_insert_ignore_replace( assert event.ignore == ignore assert event.replace == replace - actual_rows = [ - dict(r) - for r in ( - await ds_write.get_database("data").execute("select * from docs") - ).rows - ] + actual_rows = ( + await ds_write.get_database("data").execute("select * from docs") + ).dicts() + assert actual_rows == expected_rows assert response.json()["ok"] is True if should_return: diff --git a/tests/test_internals_database.py b/tests/test_internals_database.py index 0020668a..edfc6bc7 100644 --- a/tests/test_internals_database.py +++ b/tests/test_internals_database.py @@ -40,6 +40,17 @@ async def test_results_bool(db, expected): assert bool(results) is expected +@pytest.mark.asyncio +async def test_results_dicts(db): + results = await db.execute("select pk, name from roadside_attractions") + assert results.dicts() == [ + {"pk": 1, "name": "The Mystery Spot"}, + {"pk": 2, "name": "Winchester Mystery House"}, + {"pk": 3, "name": "Burlingame Museum of PEZ Memorabilia"}, + {"pk": 4, "name": "Bigfoot Discovery Museum"}, + ] + + @pytest.mark.parametrize( "query,expected", [ From 2170269258d1de38f4e518aa3e55e6b3ed202841 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Tue, 3 Sep 2024 08:37:26 -0700 Subject: [PATCH 715/952] New .core CSS class for inputs and buttons * Initial .core input/button classes, refs #2415 * Docs for the new .core CSS class, refs #2415 * Applied .core class everywhere that needs it, closes #2415 --- datasette/static/app.css | 33 +++++++++++++++------- datasette/templates/allow_debug.html | 2 +- datasette/templates/api_explorer.html | 4 +-- datasette/templates/create_token.html | 2 +- datasette/templates/database.html | 2 +- datasette/templates/logout.html | 2 +- datasette/templates/messages_debug.html | 2 +- datasette/templates/permissions_debug.html | 2 +- datasette/templates/query.html | 2 +- datasette/templates/table.html | 4 +-- docs/custom_templates.rst | 9 ++++++ docs/writing_plugins.rst | 3 +- tests/test_permissions.py | 2 +- 13 files changed, 46 insertions(+), 23 deletions(-) diff --git a/datasette/static/app.css b/datasette/static/app.css index 562d6adb..f975f0ad 100644 --- a/datasette/static/app.css +++ b/datasette/static/app.css @@ -528,8 +528,11 @@ label.sort_by_desc { pre#sql-query { margin-bottom: 1em; } -form input[type=text], -form input[type=search] { + +.core input[type=text], +input.core[type=text], +.core input[type=search], +input.core[type=search] { border: 1px solid #ccc; border-radius: 3px; width: 60%; @@ -540,17 +543,25 @@ form input[type=search] { } /* Stop Webkit from styling search boxes in an inconsistent way */ /* https://css-tricks.com/webkit-html5-search-inputs/ comments */ -input[type=search] { +.core input[type=search], +input.core[type=search] { -webkit-appearance: textfield; } -input[type="search"]::-webkit-search-decoration, -input[type="search"]::-webkit-search-cancel-button, -input[type="search"]::-webkit-search-results-button, -input[type="search"]::-webkit-search-results-decoration { +.core input[type="search"]::-webkit-search-decoration, +input.core[type="search"]::-webkit-search-decoration, +.core input[type="search"]::-webkit-search-cancel-button, +input.core[type="search"]::-webkit-search-cancel-button, +.core input[type="search"]::-webkit-search-results-button, +input.core[type="search"]::-webkit-search-results-button, +.core input[type="search"]::-webkit-search-results-decoration, +input.core[type="search"]::-webkit-search-results-decoration { display: none; } -form input[type=submit], form button[type=button] { +.core input[type=submit], +.core button[type=button], +input.core[type=submit], +button.core[type=button] { font-weight: 400; cursor: pointer; text-align: center; @@ -563,14 +574,16 @@ form input[type=submit], form button[type=button] { border-radius: .25rem; } -form input[type=submit] { +.core input[type=submit], +input.core[type=submit] { color: #fff; background: linear-gradient(180deg, #007bff 0%, #4E79C7 100%); border-color: #007bff; -webkit-appearance: button; } -form button[type=button] { +.core button[type=button], +button.core[type=button] { color: #007bff; background-color: #fff; border-color: #007bff; diff --git a/datasette/templates/allow_debug.html b/datasette/templates/allow_debug.html index 04181531..610417d2 100644 --- a/datasette/templates/allow_debug.html +++ b/datasette/templates/allow_debug.html @@ -35,7 +35,7 @@ p.message-warning {

Use this tool to try out different actor and allow combinations. See Defining permissions with "allow" blocks for documentation.

- +

diff --git a/datasette/templates/api_explorer.html b/datasette/templates/api_explorer.html index 109fb1e9..dc393c20 100644 --- a/datasette/templates/api_explorer.html +++ b/datasette/templates/api_explorer.html @@ -19,7 +19,7 @@

GET - +
@@ -29,7 +29,7 @@
POST - +
diff --git a/datasette/templates/create_token.html b/datasette/templates/create_token.html index 2be98d38..409fb8a9 100644 --- a/datasette/templates/create_token.html +++ b/datasette/templates/create_token.html @@ -39,7 +39,7 @@ {% endfor %} {% endif %} - +

diff --git a/datasette/templates/logout.html b/datasette/templates/logout.html index 4c4a7d11..c8fc642a 100644 --- a/datasette/templates/logout.html +++ b/datasette/templates/logout.html @@ -8,7 +8,7 @@

You are logged in as {{ display_actor(actor) }}

- +
diff --git a/datasette/templates/messages_debug.html b/datasette/templates/messages_debug.html index e0ab9a40..2940cd69 100644 --- a/datasette/templates/messages_debug.html +++ b/datasette/templates/messages_debug.html @@ -8,7 +8,7 @@

Set a message:

- +
diff --git a/datasette/templates/permissions_debug.html b/datasette/templates/permissions_debug.html index 5a5c1aa6..83891181 100644 --- a/datasette/templates/permissions_debug.html +++ b/datasette/templates/permissions_debug.html @@ -47,7 +47,7 @@ textarea {

This tool lets you simulate an actor and a permission check for that actor.

- +

diff --git a/datasette/templates/query.html b/datasette/templates/query.html index f7c8d0a3..a6e9a3aa 100644 --- a/datasette/templates/query.html +++ b/datasette/templates/query.html @@ -36,7 +36,7 @@ {% block description_source_license %}{% include "_description_source_license.html" %}{% endblock %} - +

Custom SQL query{% if display_rows %} returning {% if truncated %}more than {% endif %}{{ "{:,}".format(display_rows|length) }} row{% if display_rows|length == 1 %}{% else %}s{% endif %}{% endif %}{% if not query_error %} ({{ show_hide_text }}) {% endif %}

diff --git a/datasette/templates/table.html b/datasette/templates/table.html index 7246ff5d..c9e0e87b 100644 --- a/datasette/templates/table.html +++ b/datasette/templates/table.html @@ -48,7 +48,7 @@

{% endif %} - + {% if supports_search %}
{% endif %} @@ -152,7 +152,7 @@ object {% endif %}

- +

CSV options: diff --git a/docs/custom_templates.rst b/docs/custom_templates.rst index 534d8b33..8cc40f0f 100644 --- a/docs/custom_templates.rst +++ b/docs/custom_templates.rst @@ -83,6 +83,15 @@ database column they are representing, for example: +.. _customization_css: + +Writing custom CSS +~~~~~~~~~~~~~~~~~~ + +Custom templates need to take Datasette's default CSS into account. The pattern portfolio at ``/-/patterns`` (`example here `__) is a useful reference for understanding the available CSS classes. + +The ``core`` class is particularly useful - you can apply this directly to a ```` or ``