From 50c35b66a476c186549167140b6ebc0a6ec43fc1 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sat, 28 Aug 2021 04:14:38 -0700 Subject: [PATCH 0001/1135] Added missing space --- docs/plugin_hooks.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index be1624a3..d61dc727 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -626,7 +626,7 @@ Help text (from the docstring for the function plus any defined Click arguments datasette verify --help -Plugins can register multiple commands by making multiple calls to the ``@cli.command()`` decorator.Consult the `Click documentation `__ for full details on how to build a CLI command, including how to define arguments and options. +Plugins can register multiple commands by making multiple calls to the ``@cli.command()`` decorator. Consult the `Click documentation `__ for full details on how to build a CLI command, including how to define arguments and options. Note that ``register_commands()`` plugins cannot used with the :ref:`--plugins-dir mechanism ` - they need to be installed into the same virtual environment as Datasette using ``pip install``. Provided it has a ``setup.py`` file (see :ref:`writing_plugins_packaging`) you can run ``pip install`` directly against the directory in which you are developing your plugin like so:: From 67cbf0ae7243431bf13702e6e3ba466b619c4d6f Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sat, 28 Aug 2021 04:17:03 -0700 Subject: [PATCH 0002/1135] Example for register_commands, refs #1449 --- docs/plugin_hooks.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index d61dc727..23f19e38 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -80,7 +80,6 @@ You can now use this filter in your custom templates like so:: Table name: {{ table|uppercase }} - .. _plugin_hook_extra_template_vars: extra_template_vars(template, database, table, columns, view_name, request, datasette) @@ -632,6 +631,8 @@ Note that ``register_commands()`` plugins cannot used with the :ref:`--plugins-d pip install -e path/to/my/datasette-plugin +Example: `datasette-verify `_ + .. _plugin_register_facet_classes: register_facet_classes() From 772f9a07ce363869e0aaa7600617454dc00e6966 Mon Sep 17 00:00:00 2001 From: Robert Gieseke Date: Sat, 4 Sep 2021 18:31:38 +0200 Subject: [PATCH 0003/1135] Add scientists to target groups (#1455) --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 55160afe..95d430cc 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ Datasette is a tool for exploring and publishing data. It helps people take data of any shape or size and publish that as an interactive, explorable website and accompanying API. -Datasette is aimed at data journalists, museum curators, archivists, local governments and anyone else who has data that they wish to share with the world. +Datasette is aimed at data journalists, museum curators, archivists, local governments, scientists and anyone else who has data that they wish to share with the world. [Explore a demo](https://global-power-plants.datasettes.com/global-power-plants/global-power-plants), watch [a video about the project](https://simonwillison.net/2021/Feb/7/video/) or try it out by [uploading and publishing your own CSV data](https://docs.datasette.io/en/stable/getting_started.html#try-datasette-without-installing-anything-using-glitch). From d57ab156b35ec642549fb69d08279850065027d2 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sat, 4 Sep 2021 09:33:20 -0700 Subject: [PATCH 0004/1135] Added researchers too, refs #1455 --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 95d430cc..ee9d9a5a 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ Datasette is a tool for exploring and publishing data. It helps people take data of any shape or size and publish that as an interactive, explorable website and accompanying API. -Datasette is aimed at data journalists, museum curators, archivists, local governments, scientists and anyone else who has data that they wish to share with the world. +Datasette is aimed at data journalists, museum curators, archivists, local governments, scientists, researchers and anyone else who has data that they wish to share with the world. [Explore a demo](https://global-power-plants.datasettes.com/global-power-plants/global-power-plants), watch [a video about the project](https://simonwillison.net/2021/Feb/7/video/) or try it out by [uploading and publishing your own CSV data](https://docs.datasette.io/en/stable/getting_started.html#try-datasette-without-installing-anything-using-glitch). From b28b6cd2fe97f7e193a235877abeec2c8eb0a821 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sun, 12 Sep 2021 13:13:52 -0700 Subject: [PATCH 0005/1135] Warn that execute_write_fn(fn) should be a non-async function --- docs/internals.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/internals.rst b/docs/internals.rst index d5db7ffa..910f2c71 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -664,6 +664,10 @@ This method works like ``.execute_write()``, but instead of a SQL statement you The function can then perform multiple actions, safe in the knowledge that it has exclusive access to the single writable connection as long as it is executing. +.. warning:: + + ``fn`` needs to be a regular function, not an ``async def`` function. + For example: .. code-block:: python From 63886178a649586b403966a27a45881709d2b868 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 22 Sep 2021 15:44:28 -0700 Subject: [PATCH 0006/1135] Describe a common mistake using csrftoken() --- docs/internals.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/internals.rst b/docs/internals.rst index 910f2c71..411327eb 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -793,6 +793,10 @@ If your plugin implements a ``
`` anywhere you will need to i +If you are rendering templates using the :ref:`datasette_render_template` method the ``csrftoken()`` helper will only work if you provide the ``request=`` argument to that method. If you forget to do this you will see the following error:: + + form-urlencoded POST field did not match cookie + You can selectively disable CSRF protection using the :ref:`plugin_hook_skip_csrf` hook. .. _internals_internal: From 98dcabccbbf9c0800efa74df9b7d1fee81c3cd0c Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 8 Oct 2021 17:32:11 -0700 Subject: [PATCH 0007/1135] asyncio_run helper to deal with a 3.10 warning, refs #1482 --- datasette/cli.py | 13 +++++-------- datasette/utils/__init__.py | 9 +++++++++ 2 files changed, 14 insertions(+), 8 deletions(-) diff --git a/datasette/cli.py b/datasette/cli.py index 22e2338a..95ee9495 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -14,6 +14,7 @@ from runpy import run_module import webbrowser from .app import Datasette, DEFAULT_SETTINGS, SETTINGS, SQLITE_LIMIT_ATTACHED, pm from .utils import ( + asyncio_run, StartupError, check_connection, find_spatialite, @@ -136,9 +137,7 @@ def cli(): @click.option("--inspect-file", default="-") @sqlite_extensions def inspect(files, inspect_file, sqlite_extensions): - app = Datasette([], immutables=files, sqlite_extensions=sqlite_extensions) - loop = asyncio.get_event_loop() - inspect_data = loop.run_until_complete(inspect_(files, sqlite_extensions)) + inspect_data = asyncio_run(inspect_(files, sqlite_extensions)) if inspect_file == "-": sys.stdout.write(json.dumps(inspect_data, indent=2)) else: @@ -555,10 +554,10 @@ def serve( return ds # Run the "startup" plugin hooks - asyncio.get_event_loop().run_until_complete(ds.invoke_startup()) + asyncio_run(ds.invoke_startup()) # Run async soundness checks - but only if we're not under pytest - asyncio.get_event_loop().run_until_complete(check_databases(ds)) + asyncio_run(check_databases(ds)) if get: client = TestClient(ds) @@ -578,9 +577,7 @@ def serve( if open_browser: if url is None: # Figure out most convenient URL - to table, database or homepage - path = asyncio.get_event_loop().run_until_complete( - initial_path_for_datasette(ds) - ) + path = asyncio_run(initial_path_for_datasette(ds)) url = f"http://{host}:{port}{path}" webbrowser.open(url) uvicorn_kwargs = dict( diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index 70ac8976..1b7bfe05 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -1089,3 +1089,12 @@ async def derive_named_parameters(db, sql): return [row["p4"].lstrip(":") for row in results if row["opcode"] == "Variable"] except sqlite3.DatabaseError: return possible_params + + +def asyncio_run(coro): + if hasattr(asyncio, "run"): + # Added in Python 3.7 + return asyncio.run(coro) + else: + loop = asyncio.get_event_loop() + return loop.run_until_complete(coro) From 1163da89163f357003007b14f458a2c28f2e0a8e Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Fri, 8 Oct 2021 17:32:52 -0700 Subject: [PATCH 0008/1135] Update test to handle Python 3.10 error message differenc, refs #1482 --- tests/test_canned_queries.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_canned_queries.py b/tests/test_canned_queries.py index 4186a97c..b8c2baec 100644 --- a/tests/test_canned_queries.py +++ b/tests/test_canned_queries.py @@ -353,4 +353,4 @@ def test_magic_parameters_cannot_be_used_in_arbitrary_queries(magic_parameters_c "/data.json?sql=select+:_header_host&_shape=array" ) assert 400 == response.status - assert "You did not supply a value for binding 1." == response.json["error"] + assert response.json["error"].startswith("You did not supply a value for binding") From 875117c343e454221ef023be6ad977fdaea3ceda Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sat, 9 Oct 2021 18:14:56 -0700 Subject: [PATCH 0009/1135] Fix bug with ?_next=x&_sort=rowid, closes #1470 --- datasette/views/table.py | 6 ++++-- tests/test_html.py | 6 ++++++ 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/datasette/views/table.py b/datasette/views/table.py index 83f7c7cb..e8c66d3c 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -588,13 +588,15 @@ class TableView(RowTableShared): _next = _next or special_args.get("_next") offset = "" if _next: + sort_value = None if is_view: # _next is an offset offset = f" offset {int(_next)}" else: components = urlsafe_components(_next) - # If a sort order is applied, the first of these is the sort value - if sort or sort_desc: + # If a sort order is applied and there are multiple components, + # the first of these is the sort value + if (sort or sort_desc) and (len(components) > 1): sort_value = components[0] # Special case for if non-urlencoded first token was $null if _next.split(",")[0] == "$null": diff --git a/tests/test_html.py b/tests/test_html.py index e73ccd2f..c22d87da 100644 --- a/tests/test_html.py +++ b/tests/test_html.py @@ -1817,3 +1817,9 @@ def test_facet_total_shown_if_facet_max_size(use_facet_size_max): assert fragment in response.text else: assert fragment not in response.text + + +def test_sort_rowid_with_next(app_client): + # https://github.com/simonw/datasette/issues/1470 + response = app_client.get("/fixtures/binary_data?_size=1&_next=1&_sort=rowid") + assert response.status == 200 From 0d5cc20aeffa3537cfc9296d01ec24b9c6e23dcf Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Sat, 9 Oct 2021 18:25:33 -0700 Subject: [PATCH 0010/1135] Revert "asyncio_run helper to deal with a 3.10 warning, refs #1482" This reverts commit 98dcabccbbf9c0800efa74df9b7d1fee81c3cd0c. --- datasette/cli.py | 13 ++++++++----- datasette/utils/__init__.py | 9 --------- 2 files changed, 8 insertions(+), 14 deletions(-) diff --git a/datasette/cli.py b/datasette/cli.py index 95ee9495..22e2338a 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -14,7 +14,6 @@ from runpy import run_module import webbrowser from .app import Datasette, DEFAULT_SETTINGS, SETTINGS, SQLITE_LIMIT_ATTACHED, pm from .utils import ( - asyncio_run, StartupError, check_connection, find_spatialite, @@ -137,7 +136,9 @@ def cli(): @click.option("--inspect-file", default="-") @sqlite_extensions def inspect(files, inspect_file, sqlite_extensions): - inspect_data = asyncio_run(inspect_(files, sqlite_extensions)) + app = Datasette([], immutables=files, sqlite_extensions=sqlite_extensions) + loop = asyncio.get_event_loop() + inspect_data = loop.run_until_complete(inspect_(files, sqlite_extensions)) if inspect_file == "-": sys.stdout.write(json.dumps(inspect_data, indent=2)) else: @@ -554,10 +555,10 @@ def serve( return ds # Run the "startup" plugin hooks - asyncio_run(ds.invoke_startup()) + asyncio.get_event_loop().run_until_complete(ds.invoke_startup()) # Run async soundness checks - but only if we're not under pytest - asyncio_run(check_databases(ds)) + asyncio.get_event_loop().run_until_complete(check_databases(ds)) if get: client = TestClient(ds) @@ -577,7 +578,9 @@ def serve( if open_browser: if url is None: # Figure out most convenient URL - to table, database or homepage - path = asyncio_run(initial_path_for_datasette(ds)) + path = asyncio.get_event_loop().run_until_complete( + initial_path_for_datasette(ds) + ) url = f"http://{host}:{port}{path}" webbrowser.open(url) uvicorn_kwargs = dict( diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index 1b7bfe05..70ac8976 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -1089,12 +1089,3 @@ async def derive_named_parameters(db, sql): return [row["p4"].lstrip(":") for row in results if row["opcode"] == "Variable"] except sqlite3.DatabaseError: return possible_params - - -def asyncio_run(coro): - if hasattr(asyncio, "run"): - # Added in Python 3.7 - return asyncio.run(coro) - else: - loop = asyncio.get_event_loop() - return loop.run_until_complete(coro) From b50bf5d13fed10d0b930f62198d8f2658e15e1eb Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 13 Oct 2021 14:08:06 -0700 Subject: [PATCH 0011/1135] Don't persist _next in hidden field, closes #1483 --- datasette/views/table.py | 2 +- tests/test_html.py | 10 ++++++++++ 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/datasette/views/table.py b/datasette/views/table.py index e8c66d3c..efcef4d2 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -889,7 +889,7 @@ class TableView(RowTableShared): form_hidden_args = [] for key in request.args: - if key.startswith("_") and key not in ("_sort", "_search"): + if key.startswith("_") and key not in ("_sort", "_search", "_next"): for value in request.args.getlist(key): form_hidden_args.append((key, value)) diff --git a/tests/test_html.py b/tests/test_html.py index c22d87da..151ac5c3 100644 --- a/tests/test_html.py +++ b/tests/test_html.py @@ -563,6 +563,16 @@ def test_facets_persist_through_filter_form(app_client): ] +def test_next_does_not_persist_in_hidden_field(app_client): + response = app_client.get("/fixtures/searchable?_size=1&_next=1") + assert response.status == 200 + inputs = Soup(response.body, "html.parser").find("form").findAll("input") + hiddens = [i for i in inputs if i["type"] == "hidden"] + assert [(hidden["name"], hidden["value"]) for hidden in hiddens] == [ + ("_size", "1"), + ] + + @pytest.mark.parametrize( "path,expected_classes", [ From 68087440b3448633a62807c1623559619584f2ee Mon Sep 17 00:00:00 2001 From: Rhet Turnbull Date: Wed, 13 Oct 2021 14:09:10 -0700 Subject: [PATCH 0012/1135] Added instructions for installing plugins via pipx Closes #1486 --- docs/installation.rst | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/installation.rst b/docs/installation.rst index b6881bc0..723f1e3f 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -90,6 +90,25 @@ Once pipx is installed you can use it to install Datasette like this:: Then run ``datasette --version`` to confirm that it has been successfully installed. +Installing plugins using pipx +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can install additional datasette plugins with ``pipx inject`` like so:: + + $ pipx inject datasette datasette-json-html + injected package datasette-json-html into venv datasette + done! ✨ 🌟 ✨ + + $ datasette plugins + [ + { + "name": "datasette-json-html", + "static": false, + "templates": false, + "version": "0.6" + } + ] + Upgrading packages using pipx ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From 759fd97a54638c1a5e2cac65bac0ac7c07ce2305 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 13 Oct 2021 14:09:23 -0700 Subject: [PATCH 0013/1135] Update pytest-timeout requirement from <1.5,>=1.4.2 to >=1.4.2,<2.1 (#1485) Updates the requirements on [pytest-timeout](https://github.com/pytest-dev/pytest-timeout) to permit the latest version. - [Release notes](https://github.com/pytest-dev/pytest-timeout/releases) - [Commits](https://github.com/pytest-dev/pytest-timeout/commits) --- updated-dependencies: - dependency-name: pytest-timeout dependency-type: direct:development ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 84f32087..da62276e 100644 --- a/setup.py +++ b/setup.py @@ -72,7 +72,7 @@ setup( "pytest-asyncio>=0.10,<0.16", "beautifulsoup4>=4.8.1,<4.10.0", "black==21.7b0", - "pytest-timeout>=1.4.2,<1.5", + "pytest-timeout>=1.4.2,<2.1", "trustme>=0.7,<0.10", ], "rich": ["rich"], From 6aab0217f07bff4556cc92885a14279d5b295f84 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 13 Oct 2021 14:10:03 -0700 Subject: [PATCH 0014/1135] Update pytest-xdist requirement from <2.4,>=2.2.1 to >=2.2.1,<2.5 (#1476) Updates the requirements on [pytest-xdist](https://github.com/pytest-dev/pytest-xdist) to permit the latest version. - [Release notes](https://github.com/pytest-dev/pytest-xdist/releases) - [Changelog](https://github.com/pytest-dev/pytest-xdist/blob/master/CHANGELOG.rst) - [Commits](https://github.com/pytest-dev/pytest-xdist/compare/v2.2.1...v2.4.0) --- updated-dependencies: - dependency-name: pytest-xdist dependency-type: direct:development ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index da62276e..571bb35a 100644 --- a/setup.py +++ b/setup.py @@ -68,7 +68,7 @@ setup( "docs": ["sphinx_rtd_theme", "sphinx-autobuild", "codespell"], "test": [ "pytest>=5.2.2,<6.3.0", - "pytest-xdist>=2.2.1,<2.4", + "pytest-xdist>=2.2.1,<2.5", "pytest-asyncio>=0.10,<0.16", "beautifulsoup4>=4.8.1,<4.10.0", "black==21.7b0", From 31352914c427162f785d2610222a54a426d5215f Mon Sep 17 00:00:00 2001 From: Michael Tiemann <72577720+MichaelTiemannOSC@users.noreply.github.com> Date: Wed, 13 Oct 2021 17:10:23 -0400 Subject: [PATCH 0015/1135] Update full_text_search.rst (#1474) Change "above" to "below" to correct correspondence of reference to example. --- docs/full_text_search.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/full_text_search.rst b/docs/full_text_search.rst index 90b2e8c1..a285417f 100644 --- a/docs/full_text_search.rst +++ b/docs/full_text_search.rst @@ -47,7 +47,7 @@ If that option has been specified in the table metadata but you want to over-rid Configuring full-text search for a table or view ------------------------------------------------ -If a table has a corresponding FTS table set up using the ``content=`` argument to ``CREATE VIRTUAL TABLE`` shown above, Datasette will detect it automatically and add a search interface to the table page for that table. +If a table has a corresponding FTS table set up using the ``content=`` argument to ``CREATE VIRTUAL TABLE`` shown below, Datasette will detect it automatically and add a search interface to the table page for that table. You can also manually configure which table should be used for full-text search using query string parameters or :ref:`metadata`. You can set the associated FTS table for a specific table and you can also set one for a view - if you do that, the page for that SQL view will offer a search option. From a673a93b57e249f06b2d0265ce33f458258feeb0 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 13 Oct 2021 14:11:00 -0700 Subject: [PATCH 0016/1135] Update pluggy requirement from ~=0.13.0 to >=0.13,<1.1 (#1448) Updates the requirements on [pluggy](https://github.com/pytest-dev/pluggy) to permit the latest version. - [Release notes](https://github.com/pytest-dev/pluggy/releases) - [Changelog](https://github.com/pytest-dev/pluggy/blob/main/CHANGELOG.rst) - [Commits](https://github.com/pytest-dev/pluggy/compare/0.13.0...1.0.0) --- updated-dependencies: - dependency-name: pluggy dependency-type: direct:production ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 571bb35a..cb1cec66 100644 --- a/setup.py +++ b/setup.py @@ -49,7 +49,7 @@ setup( "hupper~=1.9", "httpx>=0.17", "pint~=0.9", - "pluggy~=0.13.0", + "pluggy>=0.13,<1.1", "uvicorn~=0.11", "aiofiles>=0.4,<0.8", "janus>=0.4,<0.7", From 763d0a0faabc6b53fa21ea2f0914e83ca12dfb34 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Wed, 13 Oct 2021 14:19:53 -0700 Subject: [PATCH 0017/1135] Fix for cog menu default facet bug, closes #1469 --- datasette/static/table.js | 9 ++++----- datasette/templates/table.html | 2 +- 2 files changed, 5 insertions(+), 6 deletions(-) diff --git a/datasette/static/table.js b/datasette/static/table.js index 85bf073f..3c88cc40 100644 --- a/datasette/static/table.js +++ b/datasette/static/table.js @@ -129,16 +129,15 @@ var DROPDOWN_ICON_SVG = ` el.dataset.column); var isFirstColumn = th.parentElement.querySelector("th:first-of-type") == th; var isSinglePk = th.getAttribute("data-is-pk") == "1" && document.querySelectorAll('th[data-is-pk="1"]').length == 1; - if ( - isFirstColumn || - params.getAll("_facet").includes(column) || - isSinglePk - ) { + if (isFirstColumn || displayedFacets.includes(column) || isSinglePk) { facetItem.parentNode.style.display = "none"; } else { facetItem.parentNode.style.display = "block"; diff --git a/datasette/templates/table.html b/datasette/templates/table.html index 6ba301b5..4b9df8e1 100644 --- a/datasette/templates/table.html +++ b/datasette/templates/table.html @@ -154,7 +154,7 @@ {% if facet_results %}
{% for facet_info in sorted_facet_results %} -
+

{{ facet_info.name }}{% if facet_info.type != "column" %} ({{ facet_info.type }}){% endif %} {% if show_facet_counts %} {% if facet_info.truncated %}>{% endif %}{{ facet_info.results|length }}{% endif %} From e1012e7098056734d9c90f081493991009253390 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 13 Oct 2021 14:47:42 -0700 Subject: [PATCH 0018/1135] Bump black from 21.7b0 to 21.9b0 (#1471) Bumps [black](https://github.com/psf/black) from 21.7b0 to 21.9b0. - [Release notes](https://github.com/psf/black/releases) - [Changelog](https://github.com/psf/black/blob/main/CHANGES.md) - [Commits](https://github.com/psf/black/commits) --- updated-dependencies: - dependency-name: black dependency-type: direct:development ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index cb1cec66..20576e72 100644 --- a/setup.py +++ b/setup.py @@ -71,7 +71,7 @@ setup( "pytest-xdist>=2.2.1,<2.5", "pytest-asyncio>=0.10,<0.16", "beautifulsoup4>=4.8.1,<4.10.0", - "black==21.7b0", + "black==21.9b0", "pytest-timeout>=1.4.2,<2.1", "trustme>=0.7,<0.10", ], From 2a8c6690399ee832ee62aafdede1794f5945d911 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 13 Oct 2021 15:35:36 -0700 Subject: [PATCH 0019/1135] Update beautifulsoup4 requirement (#1463) Updates the requirements on [beautifulsoup4](http://www.crummy.com/software/BeautifulSoup/bs4/) to permit the latest version. --- updated-dependencies: - dependency-name: beautifulsoup4 dependency-type: direct:development ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 20576e72..905fed16 100644 --- a/setup.py +++ b/setup.py @@ -70,7 +70,7 @@ setup( "pytest>=5.2.2,<6.3.0", "pytest-xdist>=2.2.1,<2.5", "pytest-asyncio>=0.10,<0.16", - "beautifulsoup4>=4.8.1,<4.10.0", + "beautifulsoup4>=4.8.1,<4.11.0", "black==21.9b0", "pytest-timeout>=1.4.2,<2.1", "trustme>=0.7,<0.10", From b267b5775436577a91a9f9655143908aecff05da Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 14 Oct 2021 11:03:44 -0700 Subject: [PATCH 0020/1135] Upgrade to httpx 0.20 * Upgrade to httpx 0.20, closes #1488 * TestClient.post() should not default to following redirects --- datasette/app.py | 5 +++- datasette/utils/asgi.py | 4 +-- datasette/utils/testing.py | 20 ++++++------- setup.py | 2 +- tests/test_api.py | 22 ++++++-------- tests/test_auth.py | 5 +--- tests/test_canned_queries.py | 12 +------- tests/test_custom_pages.py | 4 +-- tests/test_html.py | 38 ++++++++++++------------ tests/test_internals_datasette_client.py | 1 - tests/test_internals_request.py | 5 +++- tests/test_plugins.py | 13 ++++---- 12 files changed, 60 insertions(+), 71 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 06db740e..1f69c2b3 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -1139,6 +1139,7 @@ class DatasetteRouter: raw_path = scope.get("raw_path") if raw_path: path = raw_path.decode("ascii") + path = path.partition("?")[0] return await self.route_path(scope, receive, send, path) async def route_path(self, scope, receive, send, path): @@ -1192,7 +1193,9 @@ class DatasetteRouter: async def handle_404(self, request, send, exception=None): # If URL has a trailing slash, redirect to URL without it - path = request.scope.get("raw_path", request.scope["path"].encode("utf8")) + path = request.scope.get( + "raw_path", request.scope["path"].encode("utf8") + ).partition(b"?")[0] context = {} if path.endswith(b"/"): path = path.rstrip(b"/") diff --git a/datasette/utils/asgi.py b/datasette/utils/asgi.py index 5fa03b0a..696944df 100644 --- a/datasette/utils/asgi.py +++ b/datasette/utils/asgi.py @@ -75,7 +75,7 @@ class Request: @property def path(self): if self.scope.get("raw_path") is not None: - return self.scope["raw_path"].decode("latin-1") + return self.scope["raw_path"].decode("latin-1").partition("?")[0] else: path = self.scope["path"] if isinstance(path, str): @@ -122,7 +122,7 @@ class Request: "http_version": "1.1", "method": method, "path": path, - "raw_path": path.encode("latin-1"), + "raw_path": path_with_query_string.encode("latin-1"), "query_string": query_string.encode("latin-1"), "scheme": scheme, "type": "http", diff --git a/datasette/utils/testing.py b/datasette/utils/testing.py index a169a83d..94750b1f 100644 --- a/datasette/utils/testing.py +++ b/datasette/utils/testing.py @@ -55,10 +55,10 @@ class TestClient: @async_to_sync async def get( - self, path, allow_redirects=True, redirect_count=0, method="GET", cookies=None + self, path, follow_redirects=False, redirect_count=0, method="GET", cookies=None ): return await self._request( - path, allow_redirects, redirect_count, method, cookies + path, follow_redirects, redirect_count, method, cookies ) @async_to_sync @@ -67,7 +67,7 @@ class TestClient: path, post_data=None, body=None, - allow_redirects=True, + follow_redirects=False, redirect_count=0, content_type="application/x-www-form-urlencoded", cookies=None, @@ -90,7 +90,7 @@ class TestClient: body = urlencode(post_data, doseq=True) return await self._request( path=path, - allow_redirects=allow_redirects, + follow_redirects=follow_redirects, redirect_count=redirect_count, method="POST", cookies=cookies, @@ -103,7 +103,7 @@ class TestClient: async def request( self, path, - allow_redirects=True, + follow_redirects=True, redirect_count=0, method="GET", cookies=None, @@ -113,7 +113,7 @@ class TestClient: ): return await self._request( path, - allow_redirects=allow_redirects, + follow_redirects=follow_redirects, redirect_count=redirect_count, method=method, cookies=cookies, @@ -125,7 +125,7 @@ class TestClient: async def _request( self, path, - allow_redirects=True, + follow_redirects=True, redirect_count=0, method="GET", cookies=None, @@ -139,19 +139,19 @@ class TestClient: httpx_response = await self.ds.client.request( method, path, - allow_redirects=allow_redirects, + follow_redirects=follow_redirects, avoid_path_rewrites=True, cookies=cookies, headers=headers, content=post_body, ) response = TestResponse(httpx_response) - if allow_redirects and response.status in (301, 302): + if follow_redirects and response.status in (301, 302): assert ( redirect_count < self.max_redirects ), f"Redirected {redirect_count} times, max_redirects={self.max_redirects}" location = response.headers["Location"] return await self._request( - location, allow_redirects=True, redirect_count=redirect_count + 1 + location, follow_redirects=True, redirect_count=redirect_count + 1 ) return response diff --git a/setup.py b/setup.py index 905fed16..45eac040 100644 --- a/setup.py +++ b/setup.py @@ -47,7 +47,7 @@ setup( "click-default-group~=1.2.2", "Jinja2>=2.10.3,<3.1.0", "hupper~=1.9", - "httpx>=0.17", + "httpx>=0.20", "pint~=0.9", "pluggy>=0.13,<1.1", "uvicorn~=0.11", diff --git a/tests/test_api.py b/tests/test_api.py index 1e93c62e..38d1ba08 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -629,7 +629,7 @@ def test_no_files_uses_memory_database(app_client_no_files): ), ) def test_old_memory_urls_redirect(app_client_no_files, path, expected_redirect): - response = app_client_no_files.get(path, allow_redirects=False) + response = app_client_no_files.get(path) assert response.status == 301 assert response.headers["location"] == expected_redirect @@ -708,12 +708,8 @@ def test_table_not_exists_json(app_client): def test_jsono_redirects_to_shape_objects(app_client_with_hash): - response_1 = app_client_with_hash.get( - "/fixtures/simple_primary_key.jsono", allow_redirects=False - ) - response = app_client_with_hash.get( - response_1.headers["Location"], allow_redirects=False - ) + response_1 = app_client_with_hash.get("/fixtures/simple_primary_key.jsono") + response = app_client_with_hash.get(response_1.headers["Location"]) assert response.status == 302 assert response.headers["Location"].endswith("?_shape=objects") @@ -1488,7 +1484,7 @@ def test_settings_json(app_client): ), ) def test_config_redirects_to_settings(app_client, path, expected_redirect): - response = app_client.get(path, allow_redirects=False) + response = app_client.get(path) assert response.status == 301 assert response.headers["Location"] == expected_redirect @@ -1834,9 +1830,7 @@ def test_hash_parameter( current_hash = app_client_two_attached_databases_one_immutable.ds.databases[ "fixtures" ].hash[:7] - response = app_client_two_attached_databases_one_immutable.get( - path, allow_redirects=False - ) + response = app_client_two_attached_databases_one_immutable.get(path) assert response.status == 302 location = response.headers["Location"] assert expected_redirect.replace("HASH", current_hash) == location @@ -1844,7 +1838,7 @@ def test_hash_parameter( def test_hash_parameter_ignored_for_mutable_databases(app_client): path = "/fixtures/facetable.json?_hash=1" - response = app_client.get(path, allow_redirects=False) + response = app_client.get(path) assert response.status == 200 @@ -1976,7 +1970,9 @@ def test_cors(app_client_with_cors, path, status_code): ), ) def test_database_with_space_in_name(app_client_two_attached_databases, path): - response = app_client_two_attached_databases.get("/extra database" + path) + response = app_client_two_attached_databases.get( + "/extra database" + path, follow_redirects=True + ) assert response.status == 200 diff --git a/tests/test_auth.py b/tests/test_auth.py index 16397b7a..974f89ea 100644 --- a/tests/test_auth.py +++ b/tests/test_auth.py @@ -10,7 +10,6 @@ def test_auth_token(app_client): path = f"/-/auth-token?token={app_client.ds._root_token}" response = app_client.get( path, - allow_redirects=False, ) assert 302 == response.status assert "/" == response.headers["Location"] @@ -23,7 +22,6 @@ def test_auth_token(app_client): 403 == app_client.get( path, - allow_redirects=False, ).status ) @@ -78,14 +76,13 @@ def test_logout(app_client): in response2.text ) # If logged out you get a redirect to / - response3 = app_client.get("/-/logout", allow_redirects=False) + response3 = app_client.get("/-/logout") assert 302 == response3.status # A POST to that page should log the user out response4 = app_client.post( "/-/logout", csrftoken_from=True, cookies={"ds_actor": app_client.actor_cookie({"id": "test"})}, - allow_redirects=False, ) # The ds_actor cookie should have been unset assert response4.cookie_was_deleted("ds_actor") diff --git a/tests/test_canned_queries.py b/tests/test_canned_queries.py index b8c2baec..cea81ec7 100644 --- a/tests/test_canned_queries.py +++ b/tests/test_canned_queries.py @@ -59,7 +59,6 @@ def test_insert(canned_write_client): response = canned_write_client.post( "/data/add_name", {"name": "Hello"}, - allow_redirects=False, csrftoken_from=True, cookies={"foo": "bar"}, ) @@ -95,16 +94,13 @@ def test_insert_with_cookies_requires_csrf(canned_write_client): response = canned_write_client.post( "/data/add_name", {"name": "Hello"}, - allow_redirects=False, cookies={"foo": "bar"}, ) assert 403 == response.status def test_insert_no_cookies_no_csrf(canned_write_client): - response = canned_write_client.post( - "/data/add_name", {"name": "Hello"}, allow_redirects=False - ) + response = canned_write_client.post("/data/add_name", {"name": "Hello"}) assert 302 == response.status assert "/data/add_name?success" == response.headers["Location"] @@ -114,7 +110,6 @@ def test_custom_success_message(canned_write_client): "/data/delete_name", {"rowid": 1}, cookies={"ds_actor": canned_write_client.actor_cookie({"id": "root"})}, - allow_redirects=False, csrftoken_from=True, ) assert 302 == response.status @@ -129,7 +124,6 @@ def test_insert_error(canned_write_client): response = canned_write_client.post( "/data/add_name_specify_id", {"rowid": 1, "name": "Should fail"}, - allow_redirects=False, csrftoken_from=True, ) assert 302 == response.status @@ -145,7 +139,6 @@ def test_insert_error(canned_write_client): response = canned_write_client.post( "/data/add_name_specify_id", {"rowid": 1, "name": "Should fail"}, - allow_redirects=False, csrftoken_from=True, ) assert [["ERROR", 3]] == canned_write_client.ds.unsign( @@ -168,7 +161,6 @@ def test_json_post_body(canned_write_client): response = canned_write_client.post( "/data/add_name", body=json.dumps({"name": ["Hello", "there"]}), - allow_redirects=False, ) assert 302 == response.status assert "/data/add_name?success" == response.headers["Location"] @@ -189,7 +181,6 @@ def test_json_response(canned_write_client, headers, body, querystring): response = canned_write_client.post( "/data/add_name" + (querystring or ""), body=body, - allow_redirects=False, headers=headers, ) assert 200 == response.status @@ -331,7 +322,6 @@ def test_magic_parameters_csrf_json(magic_parameters_client, use_csrf, return_js f"/data/runme_post{qs}", {}, csrftoken_from=use_csrf or None, - allow_redirects=False, ) if return_json: assert response.status == 200 diff --git a/tests/test_custom_pages.py b/tests/test_custom_pages.py index 76c67397..66b7437a 100644 --- a/tests/test_custom_pages.py +++ b/tests/test_custom_pages.py @@ -67,13 +67,13 @@ def test_custom_content_type(custom_pages_client): def test_redirect(custom_pages_client): - response = custom_pages_client.get("/redirect", allow_redirects=False) + response = custom_pages_client.get("/redirect") assert 302 == response.status assert "/example" == response.headers["Location"] def test_redirect2(custom_pages_client): - response = custom_pages_client.get("/redirect2", allow_redirects=False) + response = custom_pages_client.get("/redirect2") assert 301 == response.status assert "/example" == response.headers["Location"] diff --git a/tests/test_html.py b/tests/test_html.py index 151ac5c3..5f2ba2f1 100644 --- a/tests/test_html.py +++ b/tests/test_html.py @@ -100,9 +100,9 @@ def test_not_allowed_methods(): def test_database_page_redirects_with_url_hash(app_client_with_hash): - response = app_client_with_hash.get("/fixtures", allow_redirects=False) - assert response.status == 302 response = app_client_with_hash.get("/fixtures") + assert response.status == 302 + response = app_client_with_hash.get("/fixtures", follow_redirects=True) assert "fixtures" in response.text @@ -161,22 +161,22 @@ def test_sql_time_limit(app_client_shorter_time_limit): def test_row_redirects_with_url_hash(app_client_with_hash): - response = app_client_with_hash.get( - "/fixtures/simple_primary_key/1", allow_redirects=False - ) + response = app_client_with_hash.get("/fixtures/simple_primary_key/1") assert response.status == 302 assert response.headers["Location"].endswith("/1") - response = app_client_with_hash.get("/fixtures/simple_primary_key/1") + response = app_client_with_hash.get( + "/fixtures/simple_primary_key/1", follow_redirects=True + ) assert response.status == 200 def test_row_strange_table_name_with_url_hash(app_client_with_hash): - response = app_client_with_hash.get( - "/fixtures/table%2Fwith%2Fslashes.csv/3", allow_redirects=False - ) + response = app_client_with_hash.get("/fixtures/table%2Fwith%2Fslashes.csv/3") assert response.status == 302 assert response.headers["Location"].endswith("/table%2Fwith%2Fslashes.csv/3") - response = app_client_with_hash.get("/fixtures/table%2Fwith%2Fslashes.csv/3") + response = app_client_with_hash.get( + "/fixtures/table%2Fwith%2Fslashes.csv/3", follow_redirects=True + ) assert response.status == 200 @@ -255,13 +255,13 @@ def test_add_filter_redirects(app_client): ) path_base = "/fixtures/simple_primary_key" path = path_base + "?" + filter_args - response = app_client.get(path, allow_redirects=False) + response = app_client.get(path) assert response.status == 302 assert response.headers["Location"].endswith("?content__startswith=x") # Adding a redirect to an existing query string: path = path_base + "?foo=bar&" + filter_args - response = app_client.get(path, allow_redirects=False) + response = app_client.get(path) assert response.status == 302 assert response.headers["Location"].endswith("?foo=bar&content__startswith=x") @@ -277,7 +277,7 @@ def test_add_filter_redirects(app_client): } ) ) - response = app_client.get(path, allow_redirects=False) + response = app_client.get(path) assert response.status == 302 assert response.headers["Location"].endswith("?content__isnull=5") @@ -299,7 +299,7 @@ def test_existing_filter_redirects(app_client): } path_base = "/fixtures/simple_primary_key" path = path_base + "?" + urllib.parse.urlencode(filter_args) - response = app_client.get(path, allow_redirects=False) + response = app_client.get(path) assert response.status == 302 assert_querystring_equal( "name__contains=hello&age__gte=22&age__lt=30&name__contains=world", @@ -309,7 +309,7 @@ def test_existing_filter_redirects(app_client): # Setting _filter_column_3 to empty string should remove *_3 entirely filter_args["_filter_column_3"] = "" path = path_base + "?" + urllib.parse.urlencode(filter_args) - response = app_client.get(path, allow_redirects=False) + response = app_client.get(path) assert response.status == 302 assert_querystring_equal( "name__contains=hello&age__gte=22&name__contains=world", @@ -317,7 +317,7 @@ def test_existing_filter_redirects(app_client): ) # ?_filter_op=exact should be removed if unaccompanied by _fiter_column - response = app_client.get(path_base + "?_filter_op=exact", allow_redirects=False) + response = app_client.get(path_base + "?_filter_op=exact") assert response.status == 302 assert "?" not in response.headers["Location"] @@ -336,7 +336,7 @@ def test_empty_search_parameter_gets_removed(app_client): } ) ) - response = app_client.get(path, allow_redirects=False) + response = app_client.get(path) assert response.status == 302 assert response.headers["Location"].endswith("?name__exact=chidi") @@ -360,7 +360,7 @@ def test_sort_by_desc_redirects(app_client): + "?" + urllib.parse.urlencode({"_sort": "sortable", "_sort_by_desc": "1"}) ) - response = app_client.get(path, allow_redirects=False) + response = app_client.get(path) assert response.status == 302 assert response.headers["Location"].endswith("?_sort_desc=sortable") @@ -1148,7 +1148,7 @@ def test_404(app_client, path): [("/fixtures/", "/fixtures"), ("/fixtures/simple_view/", "/fixtures/simple_view")], ) def test_404_trailing_slash_redirect(app_client, path, expected_redirect): - response = app_client.get(path, allow_redirects=False) + response = app_client.get(path) assert 302 == response.status assert expected_redirect == response.headers["Location"] diff --git a/tests/test_internals_datasette_client.py b/tests/test_internals_datasette_client.py index c538bef1..8c5b5bd3 100644 --- a/tests/test_internals_datasette_client.py +++ b/tests/test_internals_datasette_client.py @@ -42,7 +42,6 @@ async def test_client_post(datasette, prefix): data={ "message": "A message", }, - allow_redirects=False, ) assert isinstance(response, httpx.Response) assert response.status_code == 302 diff --git a/tests/test_internals_request.py b/tests/test_internals_request.py index fe273645..c42cfbd3 100644 --- a/tests/test_internals_request.py +++ b/tests/test_internals_request.py @@ -97,11 +97,14 @@ def test_request_url_vars(): [("/", "", "/"), ("/", "foo=bar", "/?foo=bar"), ("/foo", "bar", "/foo?bar")], ) def test_request_properties(path, query_string, expected_full_path): + path_with_query_string = path + if query_string: + path_with_query_string += "?" + query_string scope = { "http_version": "1.1", "method": "POST", "path": path, - "raw_path": path.encode("latin-1"), + "raw_path": path_with_query_string.encode("latin-1"), "query_string": query_string.encode("latin-1"), "scheme": "http", "type": "http", diff --git a/tests/test_plugins.py b/tests/test_plugins.py index a024c39b..7dac8002 100644 --- a/tests/test_plugins.py +++ b/tests/test_plugins.py @@ -71,7 +71,7 @@ def test_hook_plugin_prepare_connection_arguments(app_client): }, ), ( - "/fixtures/", + "/fixtures", { "template": "database.html", "database": "fixtures", @@ -106,6 +106,7 @@ def test_hook_plugin_prepare_connection_arguments(app_client): ) def test_hook_extra_css_urls(app_client, path, expected_decoded_object): response = app_client.get(path) + assert response.status == 200 links = Soup(response.body, "html.parser").findAll("link") special_href = [ l for l in links if l.attrs["href"].endswith("/extra-css-urls-demo.css") @@ -263,7 +264,7 @@ def test_plugin_config_file(app_client): }, ), ( - "/fixtures/", + "/fixtures", { "template": "database.html", "database": "fixtures", @@ -640,7 +641,7 @@ async def test_hook_permission_allowed(app_client, action, expected): def test_actor_json(app_client): assert {"actor": None} == app_client.get("/-/actor.json").json assert {"actor": {"id": "bot2", "1+1": 2}} == app_client.get( - "/-/actor.json/?_bot2=1" + "/-/actor.json?_bot2=1" ).json @@ -674,7 +675,7 @@ def test_hook_register_routes_with_datasette(configured_path): assert configured_path.upper() == response.text # Other one should 404 other_path = [p for p in ("path1", "path2") if configured_path != p][0] - assert client.get(f"/{other_path}/").status == 404 + assert client.get(f"/{other_path}/", follow_redirects=True).status == 404 def test_hook_register_routes_post(app_client): @@ -777,7 +778,7 @@ def test_hook_register_magic_parameters(restore_working_directory): }, ) as client: response = client.post("/data/runme", {}, csrftoken_from=True) - assert 200 == response.status + assert 302 == response.status actual = client.get("/data/logs.json?_sort_desc=rowid&_shape=array").json assert [{"rowid": 1, "line": "1.1"}] == actual # Now try the GET request against get_uuid @@ -794,7 +795,7 @@ def test_hook_forbidden(restore_working_directory): ) as client: response = client.get("/") assert 403 == response.status - response2 = client.get("/data2", allow_redirects=False) + response2 = client.get("/data2") assert 302 == response2.status assert "/login?message=view-database" == response2.headers["Location"] assert "view-database" == client.ds._last_forbidden_message From 827fa823d1b919c445f3141174ecb7a82717d99c Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 14 Oct 2021 11:10:42 -0700 Subject: [PATCH 0021/1135] Update pyyaml requirement from ~=5.3 to >=5.3,<7.0 (#1489) Updates the requirements on [pyyaml](https://github.com/yaml/pyyaml) to permit the latest version. - [Release notes](https://github.com/yaml/pyyaml/releases) - [Changelog](https://github.com/yaml/pyyaml/blob/master/CHANGES) - [Commits](https://github.com/yaml/pyyaml/compare/5.3...6.0) --- updated-dependencies: - dependency-name: pyyaml dependency-type: direct:production ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 45eac040..c1b87bd4 100644 --- a/setup.py +++ b/setup.py @@ -54,7 +54,7 @@ setup( "aiofiles>=0.4,<0.8", "janus>=0.4,<0.7", "asgi-csrf>=0.9", - "PyYAML~=5.3", + "PyYAML>=5.3,<7.0", "mergedeep>=1.1.1,<1.4.0", "itsdangerous>=1.1,<3.0", "python-baseconv==1.2.2", From 0fdbf004843850f200e077a3c87427fe16c18b85 Mon Sep 17 00:00:00 2001 From: "C. Titus Brown" Date: Thu, 14 Oct 2021 11:39:55 -0700 Subject: [PATCH 0022/1135] Rework the `--static` documentation Rework the `--static` documentation to better differentiate between the filesystem and serving locations. Closes #1457 Co-authored-by: Simon Willison --- docs/custom_templates.rst | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/custom_templates.rst b/docs/custom_templates.rst index efb5b842..3e4eb633 100644 --- a/docs/custom_templates.rst +++ b/docs/custom_templates.rst @@ -173,18 +173,18 @@ Datasette can serve static files for you, using the ``--static`` option. Consider the following directory structure:: metadata.json - static/styles.css - static/app.js + static-files/styles.css + static-files/app.js -You can start Datasette using ``--static static:static/`` to serve those -files from the ``/static/`` mount point:: +You can start Datasette using ``--static assets:static-files/`` to serve those +files from the ``/assets/`` mount point:: - $ datasette -m metadata.json --static static:static/ --memory + $ datasette -m metadata.json --static assets:static-files/ --memory The following URLs will now serve the content from those CSS and JS files:: - http://localhost:8001/static/styles.css - http://localhost:8001/static/app.js + http://localhost:8001/assets/styles.css + http://localhost:8001/assets/app.js You can reference those files from ``metadata.json`` like so: @@ -192,10 +192,10 @@ You can reference those files from ``metadata.json`` like so: { "extra_css_urls": [ - "/static/styles.css" + "/assets/styles.css" ], "extra_js_urls": [ - "/static/app.js" + "/assets/app.js" ] } @@ -205,10 +205,10 @@ Publishing static assets The :ref:`cli_publish` command can be used to publish your static assets, using the same syntax as above:: - $ datasette publish cloudrun mydb.db --static static:static/ + $ datasette publish cloudrun mydb.db --static assets:static-files/ -This will upload the contents of the ``static/`` directory as part of the -deployment, and configure Datasette to correctly serve the assets. +This will upload the contents of the ``static-files/`` directory as part of the +deployment, and configure Datasette to correctly serve the assets from ``/assets/``. .. _customization_custom_templates: From 85849935292e500ab7a99f8fe0f9546e903baad3 Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 14 Oct 2021 12:03:28 -0700 Subject: [PATCH 0023/1135] --cors Access-Control-Allow-Headers: Authorization Refs #1467, refs https://github.com/simonw/datasette-auth-tokens/issues/4 --- datasette/app.py | 3 ++- datasette/utils/__init__.py | 5 +++++ datasette/views/base.py | 9 +++++---- datasette/views/database.py | 3 ++- datasette/views/index.py | 4 ++-- datasette/views/special.py | 4 ++-- tests/test_api.py | 3 ++- 7 files changed, 20 insertions(+), 11 deletions(-) diff --git a/datasette/app.py b/datasette/app.py index 1f69c2b3..52c5e629 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -46,6 +46,7 @@ from .database import Database, QueryInterrupted from .utils import ( PrefixedUrlString, StartupError, + add_cors_headers, async_call_with_supported_arguments, await_me_maybe, call_with_supported_arguments, @@ -1321,7 +1322,7 @@ class DatasetteRouter: ) headers = {} if self.ds.cors: - headers["Access-Control-Allow-Origin"] = "*" + add_cors_headers(headers) if request.path.split("?")[0].endswith(".json"): await asgi_send_json(send, info, status=status, headers=headers) else: diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index 70ac8976..c339113c 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -1089,3 +1089,8 @@ async def derive_named_parameters(db, sql): return [row["p4"].lstrip(":") for row in results if row["opcode"] == "Variable"] except sqlite3.DatabaseError: return possible_params + + +def add_cors_headers(headers): + headers["Access-Control-Allow-Origin"] = "*" + headers["Access-Control-Allow-Headers"] = "Authorization" diff --git a/datasette/views/base.py b/datasette/views/base.py index 3333781c..01e90220 100644 --- a/datasette/views/base.py +++ b/datasette/views/base.py @@ -11,6 +11,7 @@ import pint from datasette import __version__ from datasette.database import QueryInterrupted from datasette.utils import ( + add_cors_headers, await_me_maybe, EscapeHtmlWriter, InvalidSql, @@ -163,7 +164,7 @@ class DataView(BaseView): async def options(self, request, *args, **kwargs): r = Response.text("ok") if self.ds.cors: - r.headers["Access-Control-Allow-Origin"] = "*" + add_cors_headers(r.headers) return r def redirect(self, request, path, forward_querystring=True, remove_args=None): @@ -174,7 +175,7 @@ class DataView(BaseView): r = Response.redirect(path) r.headers["Link"] = f"<{path}>; rel=preload" if self.ds.cors: - r.headers["Access-Control-Allow-Origin"] = "*" + add_cors_headers(r.headers) return r async def data(self, request, database, hash, **kwargs): @@ -417,7 +418,7 @@ class DataView(BaseView): headers = {} if self.ds.cors: - headers["Access-Control-Allow-Origin"] = "*" + add_cors_headers(headers) if request.args.get("_dl", None): if not trace: content_type = "text/csv; charset=utf-8" @@ -643,5 +644,5 @@ class DataView(BaseView): response.headers["Cache-Control"] = ttl_header response.headers["Referrer-Policy"] = "no-referrer" if self.ds.cors: - response.headers["Access-Control-Allow-Origin"] = "*" + add_cors_headers(response.headers) return response diff --git a/datasette/views/database.py b/datasette/views/database.py index e3070ce6..affded9b 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -8,6 +8,7 @@ from urllib.parse import parse_qsl, urlencode import markupsafe from datasette.utils import ( + add_cors_headers, await_me_maybe, check_visibility, derive_named_parameters, @@ -176,7 +177,7 @@ class DatabaseDownload(DataView): filepath = db.path headers = {} if self.ds.cors: - headers["Access-Control-Allow-Origin"] = "*" + add_cors_headers(headers) headers["Transfer-Encoding"] = "chunked" return AsgiFileDownload( filepath, diff --git a/datasette/views/index.py b/datasette/views/index.py index e37643f9..18454759 100644 --- a/datasette/views/index.py +++ b/datasette/views/index.py @@ -1,7 +1,7 @@ import hashlib import json -from datasette.utils import check_visibility, CustomJSONEncoder +from datasette.utils import add_cors_headers, check_visibility, CustomJSONEncoder from datasette.utils.asgi import Response from datasette.version import __version__ @@ -129,7 +129,7 @@ class IndexView(BaseView): if as_format: headers = {} if self.ds.cors: - headers["Access-Control-Allow-Origin"] = "*" + add_cors_headers(headers) return Response( json.dumps({db["name"]: db for db in databases}, cls=CustomJSONEncoder), content_type="application/json; charset=utf-8", diff --git a/datasette/views/special.py b/datasette/views/special.py index 9750dd06..3cb626a5 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -1,6 +1,6 @@ import json from datasette.utils.asgi import Response, Forbidden -from datasette.utils import actor_matches_allow +from datasette.utils import actor_matches_allow, add_cors_headers from .base import BaseView import secrets @@ -23,7 +23,7 @@ class JsonDataView(BaseView): if as_format: headers = {} if self.ds.cors: - headers["Access-Control-Allow-Origin"] = "*" + add_cors_headers(headers) return Response( json.dumps(data), content_type="application/json; charset=utf-8", diff --git a/tests/test_api.py b/tests/test_api.py index 38d1ba08..311ae464 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -1955,7 +1955,8 @@ def test_trace(trace_debug): def test_cors(app_client_with_cors, path, status_code): response = app_client_with_cors.get(path) assert response.status == status_code - assert "*" == response.headers["Access-Control-Allow-Origin"] + assert response.headers["Access-Control-Allow-Origin"] == "*" + assert response.headers["Access-Control-Allow-Headers"] == "Authorization" @pytest.mark.parametrize( From 81bf9a9f3cc5b30107a6b1adeee39d5e8312ecfc Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 14 Oct 2021 12:19:03 -0700 Subject: [PATCH 0024/1135] Updated --cors documentation, refs #1467 --- docs/json_api.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/json_api.rst b/docs/json_api.rst index 09cac1f9..7d3123b7 100644 --- a/docs/json_api.rst +++ b/docs/json_api.rst @@ -10,9 +10,10 @@ To access the API for a page, either click on the ``.json`` link on that page or edit the URL and add a ``.json`` extension to it. If you started Datasette with the ``--cors`` option, each JSON endpoint will be -served with the following additional HTTP header:: +served with the following additional HTTP headers:: Access-Control-Allow-Origin: * + Access-Control-Allow-Headers: Authorization This means JavaScript running on any domain will be able to make cross-origin requests to fetch the data. From 8934507cdc0029a598cf37cdb3818fd31af5e33c Mon Sep 17 00:00:00 2001 From: Simon Willison Date: Thu, 14 Oct 2021 12:22:19 -0700 Subject: [PATCH 0025/1135] Release 0.59 Refs #942, #1404, #1405, #1416, #1420, #1421, #1422, #1423, #1425, #1431, #1443, #1446, #1449, #1467, #1469, #1470, #1488 --- datasette/version.py | 2 +- docs/changelog.rst | 24 ++++++++---------------- 2 files changed, 9 insertions(+), 17 deletions(-) diff --git a/datasette/version.py b/datasette/version.py index 87b18fab..d78b2e90 100644 --- a/datasette/version.py +++ b/datasette/version.py @@ -1,2 +1,2 @@ -__version__ = "0.59a2" +__version__ = "0.59" __version_info__ = tuple(__version__.split(".")) diff --git a/docs/changelog.rst b/docs/changelog.rst index 737a151b..b2d95c49 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -4,36 +4,28 @@ Changelog ========= -.. _v0_59a2: +.. _v0_59: -0.59a2 (2021-08-27) -------------------- +0.59 (2021-10-14) +----------------- - Columns can now have associated metadata descriptions in ``metadata.json``, see :ref:`metadata_column_descriptions`. (:issue:`942`) - New :ref:`register_commands() ` plugin hook allows plugins to register additional Datasette CLI commands, e.g. ``datasette mycommand file.db``. (:issue:`1449`) - Adding ``?_facet_size=max`` to a table page now shows the number of unique values in each facet. (:issue:`1423`) +- Upgraded dependency `httpx 0.20 `__ - the undocumented ``allow_redirects=`` parameter to :ref:`internals_datasette_client` is now ``follow_redirects=``, and defalts to ``False`` where it previously defaulted to ``True``. (:issue:`1488`) +- The ``--cors`` option now causes Datasette to return the ``Access-Control-Allow-Headers: Authorization`` header, in addition to ``Access-Control-Allow-Origin: *``. (`#1467 `__) - Code that figures out which named parameters a SQL query takes in order to display form fields for them is no longer confused by strings that contain colon characters. (:issue:`1421`) - Renamed ``--help-config`` option to ``--help-settings``. (:issue:`1431`) - ``datasette.databases`` property is now a documented API. (:issue:`1443`) -- Datasette base template now wraps everything other than the ``