CREATE INDEX statements on table page, closes #618

It previously used csvs-to-sqlite but that had heavy dependencies.

See https://support.glitch.com/t/can-you-upgrade-python-to-latest-version/7980/33

.travis.yml

@@ -6,6 +6,7 @@ python:
   - "3.6"
   - "3.7"
   - "3.5"
+  - "3.8-dev"
 
 # Executed for 3.5 AND 3.5 as the first "test" stage:
 script:
@@ -23,12 +24,23 @@ jobs:
     - stage: deploy latest.datasette.io
       if: branch = master AND type = push
       script:
-        - pip install .
+        - pip install .[test]
         - npm install -g now
         - python tests/fixtures.py fixtures.db fixtures.json
         - export ALIAS=`echo $TRAVIS_COMMIT | cut -c 1-7`
         - datasette publish nowv1 fixtures.db -m fixtures.json --token=$NOW_TOKEN --branch=$TRAVIS_COMMIT --version-note=$TRAVIS_COMMIT --name=datasette-latest-$ALIAS --alias=latest.datasette.io --alias=$ALIAS.datasette.io
     - stage: release tagged version
+      if: tag IS present
+      python: 3.6
+      deploy:
+        - provider: pypi
+          user: simonw
+          distributions: bdist_wheel
+          password: ${PYPI_PASSWORD}
+          on:
+            branch: master
+            tags: true
+    - stage: publish docker image
       if: tag IS present
       python: 3.6
       script:
@@ -42,11 +54,3 @@ jobs:
         - docker build -f Dockerfile -t $REPO:$TRAVIS_TAG .
         - docker tag $REPO:$TRAVIS_TAG $REPO:latest
         - docker push $REPO
-      deploy:
-        - provider: pypi
-          user: simonw
-          distributions: bdist_wheel
-          password: ${PYPI_PASSWORD}
-          on:
-            branch: master
-            tags: true

README.md

@@ -1,6 +1,7 @@
 # Datasette
 
 [![PyPI](https://img.shields.io/pypi/v/datasette.svg)](https://pypi.org/project/datasette/)
+[![Python 3.x](https://img.shields.io/pypi/pyversions/datasette.svg?logo=python&logoColor=white)](https://pypi.org/project/datasette/)
 [![Travis CI](https://travis-ci.org/simonw/datasette.svg?branch=master)](https://travis-ci.org/simonw/datasette)
 [![Documentation Status](https://readthedocs.org/projects/datasette/badge/?version=latest)](http://datasette.readthedocs.io/en/latest/?badge=latest)
 [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette/blob/master/LICENSE)
@@ -20,6 +21,12 @@ Datasette is aimed at data journalists, museum curators, archivists, local gover
 
 ## News
 
+ * 18th October 2019: [Datasette 0.30](https://datasette.readthedocs.io/en/stable/changelog.html#v0-30)
+ * 13th July 2019: [Single sign-on against GitHub using ASGI middleware](https://simonwillison.net/2019/Jul/14/sso-asgi/) talks about the implementation of [datasette-auth-github](https://github.com/simonw/datasette-auth-github) in more detail.
+ * 7th July 2019: [Datasette 0.29](https://datasette.readthedocs.io/en/stable/changelog.html#v0-29) - ASGI, new plugin hooks, facet by date and much, much more...
+   * [datasette-auth-github](https://github.com/simonw/datasette-auth-github) - a new plugin for Datasette 0.29 that lets you require users to authenticate against GitHub before accessing your Datasette instance. You can whitelist specific users, or you can restrict access to members of specific GitHub organizations or teams.
+   * [datasette-cors](https://github.com/simonw/datasette-cors) - a plugin that lets you configure CORS access from a list of domains (or a set of domain wildcards) so you can make JavaScript calls to a Datasette instance from a specific set of other hosts.
+ * 23rd June 2019: [Porting Datasette to ASGI, and Turtles all the way down](https://simonwillison.net/2019/Jun/23/datasette-asgi/)
  * 21st May 2019: The anonymized raw data from [the Stack Overflow Developer Survey 2019](https://stackoverflow.blog/2019/05/21/public-data-release-of-stack-overflows-2019-developer-survey/) has been [published in partnership with Glitch](https://glitch.com/culture/discover-insights-explore-developer-survey-results-2019/), powered by Datasette.
  * 19th May 2019: [Datasette 0.28](https://datasette.readthedocs.io/en/stable/changelog.html#v0-28) - a salmagundi of new features!
    * No longer immutable! Datasette now supports [databases that change](https://datasette.readthedocs.io/en/stable/changelog.html#supporting-databases-that-change).
@@ -82,26 +89,31 @@ Now visiting http://localhost:8001/History/downloads will show you a web interfa
 
 ## datasette serve options
 
-    $ datasette serve --help
-
     Usage: datasette serve [OPTIONS] [FILES]...
 
       Serve up specified SQLite database files with a web UI
 
     Options:
       -i, --immutable PATH      Database files to open in immutable mode
-      -h, --host TEXT           host for server, defaults to 127.0.0.1
+      -h, --host TEXT           Host for server. Defaults to 127.0.0.1 which means
-      -p, --port INTEGER        port for server, defaults to 8001
+                                only connections from the local machine will be
+                                allowed. Use 0.0.0.0 to listen to all IPs and
+                                allow access from other machines.
+      -p, --port INTEGER        Port for server, defaults to 8001
       --debug                   Enable debug mode - useful for development
-      --reload                  Automatically reload if database or code change detected -
+      --reload                  Automatically reload if database or code change
-                                useful for development
+                                detected - useful for development
-      --cors                    Enable CORS by serving Access-Control-Allow-Origin: *
+      --cors                    Enable CORS by serving Access-Control-Allow-
+                                Origin: *
       --load-extension PATH     Path to a SQLite extension to load
-      --inspect-file TEXT       Path to JSON file created using "datasette inspect"
+      --inspect-file TEXT       Path to JSON file created using "datasette
-      -m, --metadata FILENAME   Path to JSON file containing license/source metadata
+                                inspect"
+      -m, --metadata FILENAME   Path to JSON file containing license/source
+                                metadata
       --template-dir DIRECTORY  Path to directory containing custom templates
       --plugins-dir DIRECTORY   Path to directory containing custom plugins
-      --static STATIC MOUNT     mountpoint:path-to-directory for serving static files
+      --static STATIC MOUNT     mountpoint:path-to-directory for serving static
+                                files
       --memory                  Make :memory: database available
       --config CONFIG           Set config option using configname:value
                                 datasette.readthedocs.io/en/latest/config.html

datasette/__main__.py

@@ -0,0 +1,4 @@
+from datasette.cli import cli
+
+if __name__ == "__main__":
+    cli()

datasette/_version.py

@@ -409,7 +409,7 @@ def render_pep440_old(pieces):
 
     The ".dev0" means dirty.
 
-    Eexceptions:
+    Exceptions:
     1: no tags. 0.postDISTANCE[.dev0]
     """
     if pieces["closest-tag"]:

datasette/app.py

@@ -1,11 +1,9 @@
 import asyncio
 import collections
 import hashlib
-import json
 import os
 import sys
 import threading
-import time
 import traceback
 import urllib.parse
 from concurrent import futures
@@ -14,10 +12,8 @@ from pathlib import Path
 import click
 from markupsafe import Markup
 from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader
-from sanic import Sanic, response
-from sanic.exceptions import InvalidUsage, NotFound
 
-from .views.base import DatasetteError, ureg
+from .views.base import DatasetteError, ureg, AsgiRouter
 from .views.database import DatabaseDownload, DatabaseView
 from .views.index import IndexView
 from .views.special import JsonDataView
@@ -36,7 +32,16 @@ from .utils import (
     sqlite_timelimit,
     to_css_class,
 )
-from .tracer import capture_traces, trace
+from .utils.asgi import (
+    AsgiLifespan,
+    NotFound,
+    asgi_static,
+    asgi_send,
+    asgi_send_html,
+    asgi_send_json,
+    asgi_send_redirect,
+)
+from .tracer import trace, AsgiTracer
 from .plugins import pm, DEFAULT_PLUGINS
 from .version import __version__
 
@@ -126,8 +131,8 @@ CONFIG_OPTIONS = (
 DEFAULT_CONFIG = {option.name: option.default for option in CONFIG_OPTIONS}
 
 
-async def favicon(request):
+async def favicon(scope, receive, send):
-    return response.text("")
+    await asgi_send(send, "", 200)
 
 
 class Datasette:
@@ -154,7 +159,7 @@ class Datasette:
             self.files = [MEMORY]
         elif memory:
             self.files = (MEMORY,) + self.files
-        self.databases = {}
+        self.databases = collections.OrderedDict()
         self.inspect_data = inspect_data
         for file in self.files:
             path = file
@@ -263,7 +268,21 @@ class Datasette:
         )
         if plugins is None:
             return None
-        return plugins.get(plugin_name)
+        plugin_config = plugins.get(plugin_name)
+        # Resolve any $file and $env keys
+        if isinstance(plugin_config, dict):
+            # Create a copy so we don't mutate the version visible at /-/metadata.json
+            plugin_config_copy = dict(plugin_config)
+            for key, value in plugin_config_copy.items():
+                if isinstance(value, dict):
+                    if list(value.keys()) == ["$env"]:
+                        plugin_config_copy[key] = os.environ.get(
+                            list(value.values())[0]
+                        )
+                    elif list(value.keys()) == ["$file"]:
+                        plugin_config_copy[key] = open(list(value.values())[0]).read()
+            return plugin_config_copy
+        return plugin_config
 
     def app_css_hash(self):
         if not hasattr(self, "_app_css_hash"):
@@ -413,6 +432,7 @@ class Datasette:
                 "full": sys.version,
             },
             "datasette": datasette_version,
+            "asgi": "3.0",
             "sqlite": {
                 "version": sqlite_version,
                 "fts_versions": fts_versions,
@@ -437,6 +457,15 @@ class Datasette:
             for p in ps
         ]
 
+    def threads(self):
+        threads = list(threading.enumerate())
+        return {
+            "num_threads": len(threads),
+            "threads": [
+                {"name": t.name, "ident": t.ident, "daemon": t.daemon} for t in threads
+            ],
+        }
+
     def table_metadata(self, database, table):
         "Fetch table-specific metadata."
         return (
@@ -450,20 +479,7 @@ class Datasette:
         def in_thread():
             conn = getattr(connections, db_name, None)
             if not conn:
-                db = self.databases[db_name]
+                conn = self.databases[db_name].connect()
-                if db.is_memory:
-                    conn = sqlite3.connect(":memory:")
-                else:
-                    # mode=ro or immutable=1?
-                    if db.is_mutable:
-                        qs = "mode=ro"
-                    else:
-                        qs = "immutable=1"
-                    conn = sqlite3.connect(
-                        "file:{}?{}".format(db.path, qs),
-                        uri=True,
-                        check_same_thread=False,
-                    )
                 self.prepare_connection(conn)
                 setattr(connections, db_name, conn)
             return fn(conn)
@@ -543,21 +559,7 @@ class Datasette:
             self.renderers[renderer["extension"]] = renderer["callback"]
 
     def app(self):
-        class TracingSanic(Sanic):
+        "Returns an ASGI app function that serves the whole of Datasette"
-            async def handle_request(self, request, write_callback, stream_callback):
-                if request.args.get("_trace"):
-                    request["traces"] = []
-                    request["trace_start"] = time.time()
-                    with capture_traces(request["traces"]):
-                        await super().handle_request(
-                            request, write_callback, stream_callback
-                        )
-                else:
-                    await super().handle_request(
-                        request, write_callback, stream_callback
-                    )
-
-        app = TracingSanic(__name__)
         default_templates = str(app_root / "datasette" / "templates")
         template_paths = []
         if self.template_dir:
@@ -588,134 +590,143 @@ class Datasette:
         pm.hook.prepare_jinja2_environment(env=self.jinja_env)
 
         self.register_renderers()
+
+        routes = []
+
+        def add_route(view, regex):
+            routes.append((regex, view))
+
         # Generate a regex snippet to match all registered renderer file extensions
         renderer_regex = "|".join(r"\." + key for key in self.renderers.keys())
 
-        app.add_route(IndexView.as_view(self), r"/<as_format:(\.jsono?)?$>")
+        add_route(IndexView.as_asgi(self), r"/(?P<as_format>(\.jsono?)?$)")
         # TODO: /favicon.ico and /-/static/ deserve far-future cache expires
-        app.add_route(favicon, "/favicon.ico")
+        add_route(favicon, "/favicon.ico")
-        app.static("/-/static/", str(app_root / "datasette" / "static"))
+
+        add_route(
+            asgi_static(app_root / "datasette" / "static"), r"/-/static/(?P<path>.*)$"
+        )
         for path, dirname in self.static_mounts:
-            app.static(path, dirname)
+            add_route(asgi_static(dirname), r"/" + path + "/(?P<path>.*)$")
+
         # Mount any plugin static/ directories
         for plugin in get_plugins(pm):
             if plugin["static_path"]:
-                modpath = "/-/static-plugins/{}/".format(plugin["name"])
+                add_route(
-                app.static(modpath, plugin["static_path"])
+                    asgi_static(plugin["static_path"]),
-        app.add_route(
+                    "/-/static-plugins/{}/(?P<path>.*)$".format(plugin["name"]),
-            JsonDataView.as_view(self, "metadata.json", lambda: self._metadata),
+                )
-            r"/-/metadata<as_format:(\.json)?$>",
+                # Support underscores in name in addition to hyphens, see https://github.com/simonw/datasette/issues/611
+                add_route(
+                    asgi_static(plugin["static_path"]),
+                    "/-/static-plugins/{}/(?P<path>.*)$".format(
+                        plugin["name"].replace("-", "_")
+                    ),
+                )
+        add_route(
+            JsonDataView.as_asgi(self, "metadata.json", lambda: self._metadata),
+            r"/-/metadata(?P<as_format>(\.json)?)$",
         )
-        app.add_route(
+        add_route(
-            JsonDataView.as_view(self, "versions.json", self.versions),
+            JsonDataView.as_asgi(self, "versions.json", self.versions),
-            r"/-/versions<as_format:(\.json)?$>",
+            r"/-/versions(?P<as_format>(\.json)?)$",
         )
-        app.add_route(
+        add_route(
-            JsonDataView.as_view(self, "plugins.json", self.plugins),
+            JsonDataView.as_asgi(self, "plugins.json", self.plugins),
-            r"/-/plugins<as_format:(\.json)?$>",
+            r"/-/plugins(?P<as_format>(\.json)?)$",
         )
-        app.add_route(
+        add_route(
-            JsonDataView.as_view(self, "config.json", lambda: self._config),
+            JsonDataView.as_asgi(self, "config.json", lambda: self._config),
-            r"/-/config<as_format:(\.json)?$>",
+            r"/-/config(?P<as_format>(\.json)?)$",
         )
-        app.add_route(
+        add_route(
-            JsonDataView.as_view(self, "databases.json", self.connected_databases),
+            JsonDataView.as_asgi(self, "threads.json", self.threads),
-            r"/-/databases<as_format:(\.json)?$>",
+            r"/-/threads(?P<as_format>(\.json)?)$",
         )
-        app.add_route(
+        add_route(
-            DatabaseDownload.as_view(self), r"/<db_name:[^/]+?><as_db:(\.db)$>"
+            JsonDataView.as_asgi(self, "databases.json", self.connected_databases),
+            r"/-/databases(?P<as_format>(\.json)?)$",
         )
-        app.add_route(
+        add_route(
-            DatabaseView.as_view(self),
+            DatabaseDownload.as_asgi(self), r"/(?P<db_name>[^/]+?)(?P<as_db>\.db)$"
-            r"/<db_name:[^/]+?><as_format:(" + renderer_regex + r"|.jsono|\.csv)?$>",
         )
-        app.add_route(
+        add_route(
-            TableView.as_view(self), r"/<db_name:[^/]+>/<table_and_format:[^/]+?$>"
+            DatabaseView.as_asgi(self),
-        )
+            r"/(?P<db_name>[^/]+?)(?P<as_format>"
-        app.add_route(
-            RowView.as_view(self),
-            r"/<db_name:[^/]+>/<table:[^/]+?>/<pk_path:[^/]+?><as_format:("
             + renderer_regex
-            + r")?$>",
+            + r"|.jsono|\.csv)?$",
+        )
+        add_route(
+            TableView.as_asgi(self),
+            r"/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)",
+        )
+        add_route(
+            RowView.as_asgi(self),
+            r"/(?P<db_name>[^/]+)/(?P<table>[^/]+?)/(?P<pk_path>[^/]+?)(?P<as_format>"
+            + renderer_regex
+            + r")?$",
         )
         self.register_custom_units()
 
-        # On 404 with a trailing slash redirect to path without that slash:
+        async def setup_db():
-        # pylint: disable=unused-variable
+            # First time server starts up, calculate table counts for immutable databases
-        @app.middleware("response")
-        def redirect_on_404_with_trailing_slash(request, original_response):
-            if original_response.status == 404 and request.path.endswith("/"):
-                path = request.path.rstrip("/")
-                if request.query_string:
-                    path = "{}?{}".format(path, request.query_string)
-                return response.redirect(path)
-
-        @app.middleware("response")
-        async def add_traces_to_response(request, response):
-            if request.get("traces") is None:
-                return
-            traces = request["traces"]
-            trace_info = {
-                "request_duration_ms": 1000 * (time.time() - request["trace_start"]),
-                "sum_trace_duration_ms": sum(t["duration_ms"] for t in traces),
-                "num_traces": len(traces),
-                "traces": traces,
-            }
-            if "text/html" in response.content_type and b"</body>" in response.body:
-                extra = json.dumps(trace_info, indent=2)
-                extra_html = "<pre>{}</pre></body>".format(extra).encode("utf8")
-                response.body = response.body.replace(b"</body>", extra_html)
-            elif "json" in response.content_type and response.body.startswith(b"{"):
-                data = json.loads(response.body.decode("utf8"))
-                if "_trace" not in data:
-                    data["_trace"] = trace_info
-                    response.body = json.dumps(data).encode("utf8")
-
-        @app.exception(Exception)
-        def on_exception(request, exception):
-            title = None
-            help = None
-            if isinstance(exception, NotFound):
-                status = 404
-                info = {}
-                message = exception.args[0]
-            elif isinstance(exception, InvalidUsage):
-                status = 405
-                info = {}
-                message = exception.args[0]
-            elif isinstance(exception, DatasetteError):
-                status = exception.status
-                info = exception.error_dict
-                message = exception.message
-                if exception.messagge_is_html:
-                    message = Markup(message)
-                title = exception.title
-            else:
-                status = 500
-                info = {}
-                message = str(exception)
-                traceback.print_exc()
-            templates = ["500.html"]
-            if status != 500:
-                templates = ["{}.html".format(status)] + templates
-            info.update(
-                {"ok": False, "error": message, "status": status, "title": title}
-            )
-            if request is not None and request.path.split("?")[0].endswith(".json"):
-                r = response.json(info, status=status)
-
-            else:
-                template = self.jinja_env.select_template(templates)
-                r = response.html(template.render(info), status=status)
-            if self.cors:
-                r.headers["Access-Control-Allow-Origin"] = "*"
-            return r
-
-        # First time server starts up, calculate table counts for immutable databases
-        @app.listener("before_server_start")
-        async def setup_db(app, loop):
             for dbname, database in self.databases.items():
                 if not database.is_mutable:
                     await database.table_counts(limit=60 * 60 * 1000)
 
-        return app
+        asgi = AsgiLifespan(
+            AsgiTracer(DatasetteRouter(self, routes)), on_startup=setup_db
+        )
+        for wrapper in pm.hook.asgi_wrapper(datasette=self):
+            asgi = wrapper(asgi)
+        return asgi
+
+
+class DatasetteRouter(AsgiRouter):
+    def __init__(self, datasette, routes):
+        self.ds = datasette
+        super().__init__(routes)
+
+    async def handle_404(self, scope, receive, send):
+        # If URL has a trailing slash, redirect to URL without it
+        path = scope.get("raw_path", scope["path"].encode("utf8"))
+        if path.endswith(b"/"):
+            path = path.rstrip(b"/")
+            if scope["query_string"]:
+                path += b"?" + scope["query_string"]
+            await asgi_send_redirect(send, path.decode("latin1"))
+        else:
+            await super().handle_404(scope, receive, send)
+
+    async def handle_500(self, scope, receive, send, exception):
+        title = None
+        if isinstance(exception, NotFound):
+            status = 404
+            info = {}
+            message = exception.args[0]
+        elif isinstance(exception, DatasetteError):
+            status = exception.status
+            info = exception.error_dict
+            message = exception.message
+            if exception.messagge_is_html:
+                message = Markup(message)
+            title = exception.title
+        else:
+            status = 500
+            info = {}
+            message = str(exception)
+            traceback.print_exc()
+        templates = ["500.html"]
+        if status != 500:
+            templates = ["{}.html".format(status)] + templates
+        info.update({"ok": False, "error": message, "status": status, "title": title})
+        headers = {}
+        if self.ds.cors:
+            headers["Access-Control-Allow-Origin"] = "*"
+        if scope["path"].split("?")[0].endswith(".json"):
+            await asgi_send_json(send, info, status=status, headers=headers)
+        else:
+            template = self.ds.jinja_env.select_template(templates)
+            await asgi_send_html(
+                send, template.render(info), status=status, headers=headers
+            )

datasette/cli.py

@@ -1,4 +1,5 @@
 import asyncio
+import uvicorn
 import click
 from click import formatting
 from click_default_group import DefaultGroup
@@ -229,9 +230,16 @@ def package(
     multiple=True,
 )
 @click.option(
-    "-h", "--host", default="127.0.0.1", help="host for server, defaults to 127.0.0.1"
+    "-h",
+    "--host",
+    default="127.0.0.1",
+    help=(
+        "Host for server. Defaults to 127.0.0.1 which means only connections "
+        "from the local machine will be allowed. Use 0.0.0.0 to listen to "
+        "all IPs and allow access from other machines."
+    ),
 )
-@click.option("-p", "--port", default=8001, help="port for server, defaults to 8001")
+@click.option("-p", "--port", default=8001, help="Port for server, defaults to 8001")
 @click.option(
     "--debug", is_flag=True, help="Enable debug mode - useful for development"
 )
@@ -354,4 +362,4 @@ def serve(
     asyncio.get_event_loop().run_until_complete(ds.run_sanity_checks())
 
     # Start the server
-    ds.app().run(host=host, port=port, debug=debug)
+    uvicorn.run(ds.app(), host=host, port=port, log_level="info")

datasette/database.py

@@ -33,6 +33,18 @@ class Database:
                     for key, value in self.ds.inspect_data[self.name]["tables"].items()
                 }
 
+    def connect(self):
+        if self.is_memory:
+            return sqlite3.connect(":memory:")
+        # mode=ro or immutable=1?
+        if self.is_mutable:
+            qs = "mode=ro"
+        else:
+            qs = "immutable=1"
+        return sqlite3.connect(
+            "file:{}?{}".format(self.path, qs), uri=True, check_same_thread=False
+        )
+
     @property
     def size(self):
         if self.is_memory:
@@ -220,7 +232,18 @@ class Database:
         )
         if not table_definition_rows:
             return None
-        return table_definition_rows[0][0]
+        bits = [table_definition_rows[0][0] + ";"]
+        # Add on any indexes
+        index_rows = list(
+            await self.ds.execute(
+                self.name,
+                "select sql from sqlite_master where tbl_name = :n and type='index' and sql is not null",
+                {"n": table},
+            )
+        )
+        for index_row in index_rows:
+            bits.append(index_row[0] + ";")
+        return "\n".join(bits)
 
     async def get_view_definition(self, view):
         return await self.get_table_definition(view, "view")

datasette/facets.py

@@ -60,7 +60,7 @@ def load_facet_configs(request, table_metadata):
 
 @hookimpl
 def register_facet_classes():
-    classes = [ColumnFacet, DateFacet, ManyToManyFacet]
+    classes = [ColumnFacet, DateFacet]
     if detect_json1():
         classes.append(ArrayFacet)
     return classes
@@ -257,6 +257,16 @@ class ColumnFacet(Facet):
 class ArrayFacet(Facet):
     type = "array"
 
+    def _is_json_array_of_strings(self, json_string):
+        try:
+            array = json.loads(json_string)
+        except ValueError:
+            return False
+        for item in array:
+            if not isinstance(item, str):
+                return False
+        return True
+
     async def suggest(self):
         columns = await self.get_columns(self.sql, self.params)
         suggested_facets = []
@@ -282,18 +292,37 @@ class ArrayFacet(Facet):
                 )
                 types = tuple(r[0] for r in results.rows)
                 if types in (("array",), ("array", None)):
-                    suggested_facets.append(
+                    # Now sanity check that first 100 arrays contain only strings
-                        {
+                    first_100 = [
-                            "name": column,
+                        v[0]
-                            "type": "array",
+                        for v in await self.ds.execute(
-                            "toggle_url": self.ds.absolute_url(
+                            self.database,
-                                self.request,
+                            "select {column} from ({sql}) where {column} is not null and json_array_length({column}) > 0 limit 100".format(
-                                path_with_added_args(
+                                column=escape_sqlite(column), sql=self.sql
-                                    self.request, {"_facet_array": column}
-                                ),
                             ),
-                        }
+                            self.params,
-                    )
+                            truncate=False,
+                            custom_time_limit=self.ds.config(
+                                "facet_suggest_time_limit_ms"
+                            ),
+                            log_sql_errors=False,
+                        )
+                    ]
+                    if first_100 and all(
+                        self._is_json_array_of_strings(r) for r in first_100
+                    ):
+                        suggested_facets.append(
+                            {
+                                "name": column,
+                                "type": "array",
+                                "toggle_url": self.ds.absolute_url(
+                                    self.request,
+                                    path_with_added_args(
+                                        self.request, {"_facet_array": column}
+                                    ),
+                                ),
+                            }
+                        )
             except (QueryInterrupted, sqlite3.OperationalError):
                 continue
         return suggested_facets
@@ -476,190 +505,3 @@ class DateFacet(Facet):
                 facets_timed_out.append(column)
 
         return facet_results, facets_timed_out
-
-
-class ManyToManyFacet(Facet):
-    type = "m2m"
-
-    async def suggest(self):
-        # This is calculated based on foreign key relationships to this table
-        # Are there any many-to-many tables pointing here?
-        suggested_facets = []
-        db = self.ds.databases[self.database]
-        all_foreign_keys = await db.get_all_foreign_keys()
-        if not all_foreign_keys.get(self.table):
-            # It's probably a view
-            return []
-        args = set(self.get_querystring_pairs())
-        incoming = all_foreign_keys[self.table]["incoming"]
-        # Do any of these incoming tables have exactly two outgoing keys?
-        for fk in incoming:
-            other_table = fk["other_table"]
-            other_table_outgoing_foreign_keys = all_foreign_keys[other_table][
-                "outgoing"
-            ]
-            if len(other_table_outgoing_foreign_keys) == 2:
-                destination_table = [
-                    t
-                    for t in other_table_outgoing_foreign_keys
-                    if t["other_table"] != self.table
-                ][0]["other_table"]
-                # Only suggest if it's not selected already
-                if ("_facet_m2m", destination_table) in args:
-                    continue
-                suggested_facets.append(
-                    {
-                        "name": destination_table,
-                        "type": "m2m",
-                        "toggle_url": self.ds.absolute_url(
-                            self.request,
-                            path_with_added_args(
-                                self.request, {"_facet_m2m": destination_table}
-                            ),
-                        ),
-                    }
-                )
-        return suggested_facets
-
-    async def facet_results(self):
-        facet_results = {}
-        facets_timed_out = []
-        args = set(self.get_querystring_pairs())
-        facet_size = self.ds.config("default_facet_size")
-        db = self.ds.databases[self.database]
-        all_foreign_keys = await db.get_all_foreign_keys()
-        if not all_foreign_keys.get(self.table):
-            return [], []
-        # We care about three tables: self.table, middle_table and destination_table
-        incoming = all_foreign_keys[self.table]["incoming"]
-        for source_and_config in self.get_configs():
-            config = source_and_config["config"]
-            source = source_and_config["source"]
-            # The destination_table is specified in the _facet_m2m=xxx parameter
-            destination_table = config.get("column") or config["simple"]
-            # Find middle table - it has fks to self.table AND destination_table
-            fks = None
-            middle_table = None
-            for fk in incoming:
-                other_table = fk["other_table"]
-                other_table_outgoing_foreign_keys = all_foreign_keys[other_table][
-                    "outgoing"
-                ]
-                if (
-                    any(
-                        o
-                        for o in other_table_outgoing_foreign_keys
-                        if o["other_table"] == destination_table
-                    )
-                    and len(other_table_outgoing_foreign_keys) == 2
-                ):
-                    fks = other_table_outgoing_foreign_keys
-                    middle_table = other_table
-                    break
-            if middle_table is None or fks is None:
-                return [], []
-            # Now that we have determined the middle_table, we need to figure out the three
-            # columns on that table which are relevant to us. These are:
-            #    column_to_table - the middle_table column with a foreign key to self.table
-            #    table_pk - the primary key column on self.table that is referenced
-            #    column_to_destination - the column with a foreign key to destination_table
-            #
-            # It turns out we don't actually need the fourth obvious column:
-            #    destination_pk = the primary key column on destination_table which is referenced
-            #
-            # These are both in the fks array - which now contains 2 foreign key relationships, e.g:
-            # [
-            #   {'other_table': 'characteristic', 'column': 'characteristic_id', 'other_column': 'pk'},
-            #   {'other_table': 'attractions', 'column': 'attraction_id', 'other_column': 'pk'}
-            # ]
-            column_to_table = None
-            table_pk = None
-            column_to_destination = None
-            for fk in fks:
-                if fk["other_table"] == self.table:
-                    table_pk = fk["other_column"]
-                    column_to_table = fk["column"]
-                elif fk["other_table"] == destination_table:
-                    column_to_destination = fk["column"]
-            assert all((column_to_table, table_pk, column_to_destination))
-            facet_sql = """
-                select
-                    {middle_table}.{column_to_destination} as value,
-                    count(distinct {middle_table}.{column_to_table}) as count
-                from {middle_table}
-                where {middle_table}.{column_to_table} in (
-                    select {table_pk} from ({sql})
-                )
-                group by {middle_table}.{column_to_destination}
-                order by count desc limit {limit}
-            """.format(
-                sql=self.sql,
-                limit=facet_size + 1,
-                middle_table=escape_sqlite(middle_table),
-                column_to_destination=escape_sqlite(column_to_destination),
-                column_to_table=escape_sqlite(column_to_table),
-                table_pk=escape_sqlite(table_pk),
-            )
-            try:
-                facet_rows_results = await self.ds.execute(
-                    self.database,
-                    facet_sql,
-                    self.params,
-                    truncate=False,
-                    custom_time_limit=self.ds.config("facet_time_limit_ms"),
-                )
-                facet_results_values = []
-                facet_results[destination_table] = {
-                    "name": destination_table,
-                    "type": self.type,
-                    "results": facet_results_values,
-                    "hideable": source != "metadata",
-                    "toggle_url": path_with_removed_args(
-                        self.request, {"_facet_m2m": destination_table}
-                    ),
-                    "truncated": len(facet_rows_results) > facet_size,
-                }
-                facet_rows = facet_rows_results.rows[:facet_size]
-
-                # Attempt to expand foreign keys into labels
-                values = [row["value"] for row in facet_rows]
-                expanded = await self.ds.expand_foreign_keys(
-                    self.database, middle_table, column_to_destination, values
-                )
-
-                for row in facet_rows:
-                    through = json.dumps(
-                        {
-                            "table": middle_table,
-                            "column": column_to_destination,
-                            "value": str(row["value"]),
-                        },
-                        separators=(",", ":"),
-                        sort_keys=True,
-                    )
-                    selected = ("_through", through) in args
-                    if selected:
-                        toggle_path = path_with_removed_args(
-                            self.request, {"_through": through}
-                        )
-                    else:
-                        toggle_path = path_with_added_args(
-                            self.request, {"_through": through}
-                        )
-                    facet_results_values.append(
-                        {
-                            "value": row["value"],
-                            "label": expanded.get(
-                                (column_to_destination, row["value"]), row["value"]
-                            ),
-                            "count": row["count"],
-                            "toggle_url": self.ds.absolute_url(
-                                self.request, toggle_path
-                            ),
-                            "selected": selected,
-                        }
-                    )
-            except QueryInterrupted:
-                facets_timed_out.append(destination_table)
-
-        return facet_results, facets_timed_out

datasette/filters.py

@@ -77,6 +77,20 @@ class InFilter(Filter):
         return "{} in {}".format(column, json.dumps(self.split_value(value)))
 
 
+class NotInFilter(InFilter):
+    key = "notin"
+    display = "not in"
+
+    def where_clause(self, table, column, value, param_counter):
+        values = self.split_value(value)
+        params = [":p{}".format(param_counter + i) for i in range(len(values))]
+        sql = "{} not in ({})".format(escape_sqlite(column), ", ".join(params))
+        return sql, values
+
+    def human_clause(self, column, value):
+        return "{} not in {}".format(column, json.dumps(self.split_value(value)))
+
+
 class Filters:
     _filters = (
         [
@@ -125,6 +139,7 @@ class Filters:
             TemplatedFilter("like", "like", '"{c}" like :{p}', '{c} like "{v}"'),
             TemplatedFilter("glob", "glob", '"{c}" glob :{p}', '{c} glob "{v}"'),
             InFilter(),
+            NotInFilter(),
         ]
         + (
             [

datasette/hookspecs.py

@@ -5,6 +5,11 @@ hookspec = HookspecMarker("datasette")
 hookimpl = HookimplMarker("datasette")
 
 
+@hookspec
+def asgi_wrapper(datasette):
+    "Returns an ASGI middleware callable to wrap our ASGI application with"
+
+
 @hookspec
 def prepare_connection(conn):
     "Modify SQLite connection in some way e.g. register custom SQL functions"
@@ -30,6 +35,11 @@ def extra_body_script(template, database, table, view_name, datasette):
     "Extra JavaScript code to be included in <script> at bottom of body"
 
 
+@hookspec
+def extra_template_vars(template, database, table, view_name, request, datasette):
+    "Extra template variables to be made available to the template - can return dict or callable or awaitable"
+
+
 @hookspec
 def publish_subcommand(publish):
     "Subcommands for 'datasette publish'"

datasette/publish/cloudrun.py

@@ -1,6 +1,7 @@
 from datasette import hookimpl
 import click
 import json
+import os
 from subprocess import check_call, check_output
 
 from .common import (
@@ -24,6 +25,11 @@ def publish_subcommand(publish):
         "--service", default="", help="Cloud Run service to deploy (or over-write)"
     )
     @click.option("--spatialite", is_flag=True, help="Enable SpatialLite extension")
+    @click.option(
+        "--show-files",
+        is_flag=True,
+        help="Output the generated Dockerfile and metadata.json",
+    )
     def cloudrun(
         files,
         metadata,
@@ -33,6 +39,7 @@ def publish_subcommand(publish):
         plugins_dir,
         static,
         install,
+        plugin_secret,
         version_note,
         title,
         license,
@@ -44,6 +51,7 @@ def publish_subcommand(publish):
         name,
         service,
         spatialite,
+        show_files,
     ):
         fail_if_publish_binary_not_installed(
             "gcloud", "Google Cloud", "https://cloud.google.com/sdk/"
@@ -52,6 +60,47 @@ def publish_subcommand(publish):
             "gcloud config get-value project", shell=True, universal_newlines=True
         ).strip()
 
+        if not service:
+            # Show the user their current services, then prompt for one
+            click.echo("Please provide a service name for this deployment\n")
+            click.echo("Using an existing service name will over-write it")
+            click.echo("")
+            existing_services = get_existing_services()
+            if existing_services:
+                click.echo("Your existing services:\n")
+                for existing_service in existing_services:
+                    click.echo(
+                        "  {name} - created {created} - {url}".format(
+                            **existing_service
+                        )
+                    )
+                click.echo("")
+            service = click.prompt("Service name", type=str)
+
+        extra_metadata = {
+            "title": title,
+            "license": license,
+            "license_url": license_url,
+            "source": source,
+            "source_url": source_url,
+            "about": about,
+            "about_url": about_url,
+        }
+
+        environment_variables = {}
+        if plugin_secret:
+            extra_metadata["plugins"] = {}
+            for plugin_name, plugin_setting, setting_value in plugin_secret:
+                environment_variable = (
+                    "{}_{}".format(plugin_name, plugin_setting)
+                    .upper()
+                    .replace("-", "_")
+                )
+                environment_variables[environment_variable] = setting_value
+                extra_metadata["plugins"].setdefault(plugin_name, {})[
+                    plugin_setting
+                ] = {"$env": environment_variable}
+
         with temporary_docker_directory(
             files,
             name,
@@ -64,21 +113,40 @@ def publish_subcommand(publish):
             install,
             spatialite,
             version_note,
-            {
+            extra_metadata,
-                "title": title,
+            environment_variables,
-                "license": license,
-                "license_url": license_url,
-                "source": source,
-                "source_url": source_url,
-                "about": about,
-                "about_url": about_url,
-            },
         ):
+            if show_files:
+                if os.path.exists("metadata.json"):
+                    print("=== metadata.json ===\n")
+                    print(open("metadata.json").read())
+                print("\n==== Dockerfile ====\n")
+                print(open("Dockerfile").read())
+                print("\n====================\n")
+
             image_id = "gcr.io/{project}/{name}".format(project=project, name=name)
             check_call("gcloud builds submit --tag {}".format(image_id), shell=True)
         check_call(
-            "gcloud beta run deploy --allow-unauthenticated --image {}{}".format(
+            "gcloud beta run deploy --allow-unauthenticated --platform=managed --image {} {}".format(
-                image_id, " {}".format(service) if service else ""
+                image_id, service,
             ),
             shell=True,
         )
+
+
+def get_existing_services():
+    services = json.loads(
+        check_output(
+            "gcloud beta run services list --platform=managed --format json",
+            shell=True,
+            universal_newlines=True,
+        )
+    )
+    return [
+        {
+            "name": service["metadata"]["name"],
+            "created": service["metadata"]["creationTimestamp"],
+            "url": service["status"]["address"]["url"],
+        }
+        for service in services
+    ]

datasette/publish/common.py

@@ -41,6 +41,14 @@ def add_common_publish_arguments_and_options(subcommand):
                 help="Additional packages (e.g. plugins) to install",
                 multiple=True,
             ),
+            click.option(
+                "--plugin-secret",
+                nargs=3,
+                type=(str, str, str),
+                callback=validate_plugin_secret,
+                multiple=True,
+                help="Secrets to pass to plugins, e.g. --plugin-secret datasette-auth-github client_id xxx",
+            ),
             click.option(
                 "--version-note", help="Additional note to show on /-/versions"
             ),
@@ -76,3 +84,10 @@ def fail_if_publish_binary_not_installed(binary, publish_target, install_link):
             err=True,
         )
         sys.exit(1)
+
+
+def validate_plugin_secret(ctx, param, value):
+    for plugin_name, plugin_setting, setting_value in value:
+        if "'" in setting_value:
+            raise click.BadParameter("--plugin-secret cannot contain single quotes")
+    return value

datasette/publish/heroku.py

@@ -33,6 +33,7 @@ def publish_subcommand(publish):
         plugins_dir,
         static,
         install,
+        plugin_secret,
         version_note,
         title,
         license,
@@ -61,6 +62,30 @@ def publish_subcommand(publish):
             )
             call(["heroku", "plugins:install", "heroku-builds"])
 
+        extra_metadata = {
+            "title": title,
+            "license": license,
+            "license_url": license_url,
+            "source": source,
+            "source_url": source_url,
+            "about": about,
+            "about_url": about_url,
+        }
+
+        environment_variables = {}
+        if plugin_secret:
+            extra_metadata["plugins"] = {}
+            for plugin_name, plugin_setting, setting_value in plugin_secret:
+                environment_variable = (
+                    "{}_{}".format(plugin_name, plugin_setting)
+                    .upper()
+                    .replace("-", "_")
+                )
+                environment_variables[environment_variable] = setting_value
+                extra_metadata["plugins"].setdefault(plugin_name, {})[
+                    plugin_setting
+                ] = {"$env": environment_variable}
+
         with temporary_heroku_directory(
             files,
             name,
@@ -72,15 +97,7 @@ def publish_subcommand(publish):
             static,
             install,
             version_note,
-            {
+            extra_metadata,
-                "title": title,
-                "license": license,
-                "license_url": license_url,
-                "source": source,
-                "source_url": source_url,
-                "about": about,
-                "about_url": about_url,
-            },
         ):
             app_name = None
             if name:
@@ -104,6 +121,11 @@ def publish_subcommand(publish):
                 create_output = check_output(cmd).decode("utf8")
                 app_name = json.loads(create_output)["name"]
 
+            for key, value in environment_variables.items():
+                call(
+                    ["heroku", "config:set", "-a", app_name, "{}={}".format(key, value)]
+                )
+
             call(["heroku", "builds:create", "-a", app_name, "--include-vcs-ignore"])
 
 

datasette/publish/now.py

@@ -1,6 +1,7 @@
 from datasette import hookimpl
 import click
 import json
+import os
 from subprocess import run, PIPE
 
 from .common import (
@@ -24,6 +25,11 @@ def publish_subcommand(publish):
     @click.option("--token", help="Auth token to use for deploy")
     @click.option("--alias", multiple=True, help="Desired alias e.g. yoursite.now.sh")
     @click.option("--spatialite", is_flag=True, help="Enable SpatialLite extension")
+    @click.option(
+        "--show-files",
+        is_flag=True,
+        help="Output the generated Dockerfile and metadata.json",
+    )
     def nowv1(
         files,
         metadata,
@@ -33,6 +39,7 @@ def publish_subcommand(publish):
         plugins_dir,
         static,
         install,
+        plugin_secret,
         version_note,
         title,
         license,
@@ -46,6 +53,7 @@ def publish_subcommand(publish):
         token,
         alias,
         spatialite,
+        show_files,
     ):
         fail_if_publish_binary_not_installed("now", "Zeit Now", "https://zeit.co/now")
         if extra_options:
@@ -54,6 +62,30 @@ def publish_subcommand(publish):
             extra_options = ""
         extra_options += "--config force_https_urls:on"
 
+        extra_metadata = {
+            "title": title,
+            "license": license,
+            "license_url": license_url,
+            "source": source,
+            "source_url": source_url,
+            "about": about,
+            "about_url": about_url,
+        }
+
+        environment_variables = {}
+        if plugin_secret:
+            extra_metadata["plugins"] = {}
+            for plugin_name, plugin_setting, setting_value in plugin_secret:
+                environment_variable = (
+                    "{}_{}".format(plugin_name, plugin_setting)
+                    .upper()
+                    .replace("-", "_")
+                )
+                environment_variables[environment_variable] = setting_value
+                extra_metadata["plugins"].setdefault(plugin_name, {})[
+                    plugin_setting
+                ] = {"$env": environment_variable}
+
         with temporary_docker_directory(
             files,
             name,
@@ -66,15 +98,8 @@ def publish_subcommand(publish):
             install,
             spatialite,
             version_note,
-            {
+            extra_metadata,
-                "title": title,
+            environment_variables,
-                "license": license,
-                "license_url": license_url,
-                "source": source,
-                "source_url": source_url,
-                "about": about,
-                "about_url": about_url,
-            },
         ):
             now_json = {"version": 1}
             open("now.json", "w").write(json.dumps(now_json, indent=4))
@@ -88,6 +113,13 @@ def publish_subcommand(publish):
             else:
                 done = run("now", stdout=PIPE)
             deployment_url = done.stdout
+            if show_files:
+                if os.path.exists("metadata.json"):
+                    print("=== metadata.json ===\n")
+                    print(open("metadata.json").read())
+                print("\n==== Dockerfile ====\n")
+                print(open("Dockerfile").read())
+                print("\n====================\n")
             if alias:
                 # I couldn't get --target=production working, so I call
                 # 'now alias' with arguments directly instead - but that

datasette/renderer.py

@@ -88,5 +88,5 @@ def json_renderer(args, data, view_name):
         content_type = "text/plain"
     else:
         body = json.dumps(data, cls=CustomJSONEncoder)
-        content_type = "application/json"
+        content_type = "application/json; charset=utf-8"
     return {"body": body, "status_code": status_code, "content_type": content_type}

datasette/static/app.css

@@ -1,5 +1,6 @@
 body {
-    margin: 0 1em;
+    margin: 0;
+    padding: 0;
     font-family: "Helvetica Neue", sans-serif;
     font-size: 1rem;
     font-weight: 400;
@@ -8,6 +9,9 @@ body {
     text-align: left;
     background-color: #fff;
 }
+.bd {
+    margin: 0 1em;
+}
 table {
     border-collapse: collapse;
 }
@@ -82,9 +86,22 @@ table a:visited {
 
 .hd {
     border-bottom: 2px solid #ccc;
+    padding: 0.2em 1em;
+    background-color: #eee;
+    overflow: hidden;
+    box-sizing: border-box;
+    min-height: 2rem;
+}
+.hd p {
+    margin: 0;
+    padding: 0;
+}
+.hd .crumbs {
+    float: left;
 }
 .ft {
     margin: 1em 0;
+    padding: 0.5em 1em 0 1em;
     border-top: 1px solid #ccc;
     font-size: 0.8em;
 }
@@ -149,22 +166,32 @@ form input[type=search] {
         width: 95%;
     }
 }
-form input[type=submit] {
+form input[type=submit], form button[type=button] {
-    color: #fff;
-    background-color: #007bff;
-    border-color: #007bff;
     font-weight: 400;
     cursor: pointer;
     text-align: center;
     vertical-align: middle;
-    border: 1px solid blue;
+    border-width: 1px;
+    border-style: solid;
     padding: .5em 0.8em;
     font-size: 0.9rem;
     line-height: 1;
     border-radius: .25rem;
+}
+
+form input[type=submit] {
+    color: #fff;
+    background-color: #007bff;
+    border-color: #007bff;
     -webkit-appearance: button;
 }
 
+form button[type=button] {
+    color: #007bff;
+    background-color: #fff;
+    border-color: #007bff;
+}
+
 .filter-row {
     margin-bottom: 0.6em;
 }
@@ -296,3 +323,7 @@ a.not-underlined {
     font-style: normal;
     font-size: 0.8em;
 }
+
+pre.wrapped-sql {
+    white-space: pre-wrap;
+}

datasette/templates/500.html

@@ -2,8 +2,14 @@
 
 {% block title %}{% if title %}{{ title }}{% else %}Error {{ status }}{% endif %}{% endblock %}
 
+{% block nav %}
+    <p class="crumbs">
+        <a href="/">home</a>
+    </p>
+    {{ super() }}
+{% endblock %}
+
 {% block content %}
-<div class="hd"><a href="/">home</a></div>
 
 <h1>{% if title %}{{ title }}{% else %}Error {{ status }}{% endif %}</h1>
 

datasette/templates/_codemirror.html

@@ -1,3 +1,4 @@
+<script src="/-/static/sql-formatter-2.3.3.min.js" defer></script>
 <script src="/-/static/codemirror-5.31.0.js"></script>
 <link rel="stylesheet" href="/-/static/codemirror-5.31.0-min.css" />
 <script src="/-/static/codemirror-5.31.0-sql.min.js"></script>

datasette/templates/_codemirror_foot.html

@@ -1,13 +1,37 @@
 <script>
-    var editor = CodeMirror.fromTextArea(document.getElementById("sql-editor"), {
+window.onload = () => {
-      lineNumbers: true,
+    const sqlFormat = document.querySelector("button#sql-format");
-      mode: "text/x-sql",
+    const readOnly = document.querySelector("pre#sql-query");
-      lineWrapping: true,
+    const sqlInput = document.querySelector("textarea#sql-editor");
-    });
+    if (sqlFormat && !readOnly) {
-    editor.setOption("extraKeys", {
+        sqlFormat.hidden = false;
-      "Shift-Enter": function() {
+    }
-        document.getElementsByClassName("sql")[0].submit();
+    if (sqlInput) {
-      },
+        var editor = CodeMirror.fromTextArea(sqlInput, {
-      Tab: false
+          lineNumbers: true,
-    });
+          mode: "text/x-sql",
+          lineWrapping: true,
+        });
+        editor.setOption("extraKeys", {
+          "Shift-Enter": function() {
+            document.getElementsByClassName("sql")[0].submit();
+          },
+          Tab: false
+        });
+        if (sqlFormat) {
+            sqlFormat.addEventListener("click", ev => {
+                editor.setValue(sqlFormatter.format(editor.getValue()));
+            })
+        }
+    }
+    if (sqlFormat && readOnly) {
+        const formatted = sqlFormatter.format(readOnly.innerHTML);
+        if (formatted != readOnly.innerHTML) {
+            sqlFormat.hidden = false;
+            sqlFormat.addEventListener("click", ev => {
+                readOnly.innerHTML = formatted;
+            })
+        }
+    }
+}
 </script>

datasette/templates/_footer.html

@@ -0,0 +1,21 @@
+Powered by <a href="https://github.com/simonw/datasette" title="Datasette v{{ datasette_version }}">Datasette</a>
+{% if query_ms %}&middot; Query took {{ query_ms|round(3) }}ms{% endif %}
+{% if metadata %}
+    {% if metadata.license or metadata.license_url %}&middot; Data license:
+        {% if metadata.license_url %}
+            <a href="{{ metadata.license_url }}">{{ metadata.license or metadata.license_url }}</a>
+        {% else %}
+            {{ metadata.license }}
+        {% endif %}
+    {% endif %}
+    {% if metadata.source or metadata.source_url %}&middot;
+        Data source: {% if metadata.source_url %}
+            <a href="{{ metadata.source_url }}">
+        {% endif %}{{ metadata.source or metadata.source_url }}{% if metadata.source_url %}</a>{% endif %}
+    {% endif %}
+    {% if metadata.about or metadata.about_url %}&middot;
+        About: {% if metadata.about_url %}
+            <a href="{{ metadata.about_url }}">
+        {% endif %}{{ metadata.about or metadata.about_url }}{% if metadata.about_url %}</a>{% endif %}
+    {% endif %}
+{% endif %}

datasette/templates/base.html

@@ -14,33 +14,15 @@
 </head>
 <body class="{% block body_class %}{% endblock %}">
 
+<nav class="hd">{% block nav %}{% endblock %}</nav>
+
+<div class="bd">
 {% block content %}
 {% endblock %}
-
-<div class="ft">
-    Powered by <a href="https://github.com/simonw/datasette" title="Datasette v{{ datasette_version }}">Datasette</a>
-    {% if query_ms %}&middot; Query took {{ query_ms|round(3) }}ms{% endif %}
-    {% if metadata %}
-        {% if metadata.license or metadata.license_url %}&middot; Data license:
-            {% if metadata.license_url %}
-                <a href="{{ metadata.license_url }}">{{ metadata.license or metadata.license_url }}</a>
-            {% else %}
-                {{ metadata.license }}
-            {% endif %}
-        {% endif %}
-        {% if metadata.source or metadata.source_url %}&middot;
-            Data source: {% if metadata.source_url %}
-                <a href="{{ metadata.source_url }}">
-            {% endif %}{{ metadata.source or metadata.source_url }}{% if metadata.source_url %}</a>{% endif %}
-        {% endif %}
-        {% if metadata.about or metadata.about_url %}&middot;
-            About: {% if metadata.about_url %}
-                <a href="{{ metadata.about_url }}">
-            {% endif %}{{ metadata.about or metadata.about_url }}{% if metadata.about_url %}</a>{% endif %}
-        {% endif %}
-    {% endif %}
 </div>
 
+<div class="ft">{% block footer %}{% include "_footer.html" %}{% endblock %}</div>
+
 {% for body_script in body_scripts %}
     <script>{{ body_script }}</script>
 {% endfor %}

datasette/templates/database.html

@@ -9,8 +9,14 @@
 
 {% block body_class %}db db-{{ database|to_css_class }}{% endblock %}
 
+{% block nav %}
+    <p class="crumbs">
+        <a href="/">home</a>
+    </p>
+    {{ super() }}
+{% endblock %}
+
 {% block content %}
-<div class="hd"><a href="/">home</a></div>
 
 <h1 style="padding-left: 10px; border-left: 10px solid #{{ database_color(database) }}">{{ metadata.title or database }}</h1>
 
@@ -19,8 +25,11 @@
 {% if config.allow_sql %}
     <form class="sql" action="{{ database_url(database) }}" method="get">
         <h3>Custom SQL query</h3>
-        <p><textarea name="sql">{% if tables %}select * from {{ tables[0].name|escape_sqlite }}{% else %}select sqlite_version(){% endif %}</textarea></p>
+        <p><textarea id="sql-editor" name="sql">{% if tables %}select * from {{ tables[0].name|escape_sqlite }}{% else %}select sqlite_version(){% endif %}</textarea></p>
-        <p><input type="submit" value="Run SQL"></p>
+        <p>
+            <button id="sql-format" type="button" hidden>Format SQL</button>
+            <input type="submit" value="Run SQL">
+        </p>
     </form>
 {% endif %}
 

datasette/templates/index.html

@@ -21,7 +21,8 @@
             {{ "{:,}".format(database.views_count) }} view{% if database.views_count != 1 %}s{% endif %}
         {% endif %}
     </p>
-    <p>{% for table in database.tables_and_views_truncated %}<a href="{{ database.path }}/{{ table.name|quote_plus }}"{% if table.count %} title="{{ table.count }} rows"{% endif %}>{{ table.name }}</a>{% if not loop.last %}, {% endif %}{% endfor %}{% if database.tables_and_views_more %}, <a href="{{ database.path }}">...</a>{% endif %}</p>
+    <p>{% for table in database.tables_and_views_truncated %}<a href="{{ database.path }}/{{ table.name|quote_plus 
+    }}"{% if table.count %} title="{{ table.count }} rows"{% endif %}>{{ table.name }}</a>{% if not loop.last %}, {% endif %}{% endfor %}{% if database.tables_and_views_more %}, <a href="{{ database.path }}">...</a>{% endif %}</p>
 {% endfor %}
 
 {% endblock %}

datasette/templates/query.html

@@ -18,9 +18,15 @@
 
 {% block body_class %}query db-{{ database|to_css_class }}{% endblock %}
 
-{% block content %}
+{% block nav %}
-<div class="hd"><a href="/">home</a> / <a href="{{ database_url(database) }}">{{ database }}</a></div>
+    <p class="crumbs">
+        <a href="/">home</a> /
+        <a href="{{ database_url(database) }}">{{ database }}</a>
+    </p>
+    {{ super() }}
+{% endblock %}
 
+{% block content %}
 <h1 style="padding-left: 10px; border-left: 10px solid #{{ database_color(database) }}">{{ metadata.title or database }}</h1>
 
 {% block description_source_license %}{% include "_description_source_license.html" %}{% endblock %}
@@ -31,7 +37,7 @@
         {% if editable and config.allow_sql %}
             <p><textarea id="sql-editor" name="sql">{% if query and query.sql %}{{ query.sql }}{% else %}select * from {{ tables[0].name|escape_sqlite }}{% endif %}</textarea></p>
         {% else %}
-            <pre>{% if query %}{{ query.sql }}{% endif %}</pre>
+            <pre id="sql-query">{% if query %}{{ query.sql }}{% endif %}</pre>
         {% endif %}
     {% else %}
         <input type="hidden" name="sql" value="{% if query and query.sql %}{{ query.sql }}{% else %}select * from {{ tables[0].name|escape_sqlite }}{% endif %}">
@@ -43,7 +49,10 @@
             <p><label for="qp{{ loop.index }}">{{ name }}</label> <input type="text" id="qp{{ loop.index }}" name="{{ name }}" value="{{ value }}"></p>
         {% endfor %}
     {% endif %}
-    <p><input type="submit" value="Run SQL"></p>
+    <p>
+        <button id="sql-format" type="button" hidden>Format SQL</button>
+        <input type="submit" value="Run SQL">
+    </p>
 </form>
 
 {% if display_rows %}

datasette/templates/row.html

@@ -15,16 +15,23 @@
 
 {% block body_class %}row db-{{ database|to_css_class }} table-{{ table|to_css_class }}{% endblock %}
 
-{% block content %}
+{% block nav %}
-<div class="hd"><a href="/">home</a> / <a href="{{ database_url(database) }}">{{ database }}</a> / <a href="{{ database_url(database) }}/{{ table|quote_plus }}">{{ table }}</a></div>
+    <p class="crumbs">
+        <a href="/">home</a> /
+        <a href="{{ database_url(database) }}">{{ database }}</a> /
+        <a href="{{ database_url(database) }}/{{ table|quote_plus }}">{{ table }}</a>
+    </p>
+    {{ super() }}
+{% endblock %}
 
+{% block content %}
 <h1 style="padding-left: 10px; border-left: 10px solid #{{ database_color(database) }}">{{ table }}: {{ ', '.join(primary_key_values) }}</a></h1>
 
 {% block description_source_license %}{% include "_description_source_license.html" %}{% endblock %}
 
 <p>This data as {% for name, url in renderers.items() %}<a href="{{ url }}">{{ name }}</a>{{ ", " if not loop.last }}{% endfor %}</p>
 
-{% include custom_rows_and_columns_templates %}
+{% include custom_table_templates %}
 
 {% if foreign_key_tables %}
     <h2>Links from other tables</h2>

datasette/templates/table.html

@@ -16,8 +16,15 @@
 
 {% block body_class %}table db-{{ database|to_css_class }} table-{{ table|to_css_class }}{% endblock %}
 
+{% block nav %}
+    <p class="crumbs">
+        <a href="/">home</a> /
+        <a href="{{ database_url(database) }}">{{ database }}</a>
+    </p>
+    {{ super() }}
+{% endblock %}
+
 {% block content %}
-<div class="hd"><a href="/">home</a> / <a href="{{ database_url(database) }}">{{ database }}</a></div>
 
 <h1 style="padding-left: 10px; border-left: 10px solid #{{ database_color(database) }}">{{ metadata.title or table }}{% if is_view %} (view){% endif %}</h1>
 
@@ -145,7 +152,7 @@
     </div>
 {% endif %}
 
-{% include custom_rows_and_columns_templates %}
+{% include custom_table_templates %}
 
 {% if next_url %}
      <p><a href="{{ next_url }}">Next page</a></p>
@@ -177,11 +184,11 @@
 {% endif %}
 
 {% if table_definition %}
-    <pre>{{ table_definition }}</pre>
+    <pre class="wrapped-sql">{{ table_definition }}</pre>
 {% endif %}
 
 {% if view_definition %}
-    <pre>{{ view_definition }}</pre>
+    <pre class="wrapped-sql">{{ view_definition }}</pre>
 {% endif %}
 
 {% endblock %}

datasette/tracer.py

@@ -1,6 +1,7 @@
 import asyncio
 from contextlib import contextmanager
 import time
+import json
 import traceback
 
 tracers = {}
@@ -32,15 +33,15 @@ def trace(type, **kwargs):
     start = time.time()
     yield
     end = time.time()
-    trace = {
+    trace_info = {
         "type": type,
         "start": start,
         "end": end,
         "duration_ms": (end - start) * 1000,
         "traceback": traceback.format_list(traceback.extract_stack(limit=6)[:-3]),
     }
-    trace.update(kwargs)
+    trace_info.update(kwargs)
-    tracer.append(trace)
+    tracer.append(trace_info)
 
 
 @contextmanager
@@ -53,3 +54,77 @@ def capture_traces(tracer):
     tracers[task_id] = tracer
     yield
     del tracers[task_id]
+
+
+class AsgiTracer:
+    # If the body is larger than this we don't attempt to append the trace
+    max_body_bytes = 1024 * 256  # 256 KB
+
+    def __init__(self, app):
+        self.app = app
+
+    async def __call__(self, scope, receive, send):
+        if b"_trace=1" not in scope.get("query_string", b"").split(b"&"):
+            await self.app(scope, receive, send)
+            return
+        trace_start = time.time()
+        traces = []
+
+        accumulated_body = b""
+        size_limit_exceeded = False
+        response_headers = []
+
+        async def wrapped_send(message):
+            nonlocal accumulated_body, size_limit_exceeded, response_headers
+            if message["type"] == "http.response.start":
+                response_headers = message["headers"]
+                await send(message)
+                return
+
+            if message["type"] != "http.response.body" or size_limit_exceeded:
+                await send(message)
+                return
+
+            # Accumulate body until the end or until size is exceeded
+            accumulated_body += message["body"]
+            if len(accumulated_body) > self.max_body_bytes:
+                await send(
+                    {
+                        "type": "http.response.body",
+                        "body": accumulated_body,
+                        "more_body": True,
+                    }
+                )
+                size_limit_exceeded = True
+                return
+
+            if not message.get("more_body"):
+                # We have all the body - modify it and send the result
+                # TODO: What to do about Content-Type or other cases?
+                trace_info = {
+                    "request_duration_ms": 1000 * (time.time() - trace_start),
+                    "sum_trace_duration_ms": sum(t["duration_ms"] for t in traces),
+                    "num_traces": len(traces),
+                    "traces": traces,
+                }
+                try:
+                    content_type = [
+                        v.decode("utf8")
+                        for k, v in response_headers
+                        if k.lower() == b"content-type"
+                    ][0]
+                except IndexError:
+                    content_type = ""
+                if "text/html" in content_type and b"</body>" in accumulated_body:
+                    extra = json.dumps(trace_info, indent=2)
+                    extra_html = "<pre>{}</pre></body>".format(extra).encode("utf8")
+                    accumulated_body = accumulated_body.replace(b"</body>", extra_html)
+                elif "json" in content_type and accumulated_body.startswith(b"{"):
+                    data = json.loads(accumulated_body.decode("utf8"))
+                    if "_trace" not in data:
+                        data["_trace"] = trace_info
+                    accumulated_body = json.dumps(data).encode("utf8")
+                await send({"type": "http.response.body", "body": accumulated_body})
+
+        with capture_traces(traces):
+            await self.app(scope, receive, wrapped_send)

datasette/utils/__init__.py

@@ -3,7 +3,6 @@ from collections import OrderedDict
 import base64
 import click
 import hashlib
-import imp
 import json
 import os
 import pkg_resources
@@ -11,6 +10,7 @@ import re
 import shlex
 import tempfile
 import time
+import types
 import shutil
 import urllib
 import numbers
@@ -167,6 +167,8 @@ allowed_sql_res = [
     re.compile(r"^explain select\b"),
     re.compile(r"^explain query plan select\b"),
     re.compile(r"^with\b"),
+    re.compile(r"^explain with\b"),
+    re.compile(r"^explain query plan with\b"),
 ]
 disallawed_sql_res = [(re.compile("pragma"), "Statement may not contain PRAGMA")]
 
@@ -261,27 +263,6 @@ def escape_sqlite(s):
         return "[{}]".format(s)
 
 
-_decode_path_component_re = re.compile(r"U\+([\da-h]{4})", re.IGNORECASE)
-_encode_path_component_re = re.compile(
-    "[{}]".format(
-        "".join(
-            re.escape(c)
-            for c in (";", "/", "?", ":", "@", "&", "=", "+", "$", ",", "~")
-        )
-    )
-)
-
-
-def decode_path_component(table_name):
-    return _decode_path_component_re.sub(lambda m: chr(int(m.group(1), 16)), table_name)
-
-
-def encode_path_component(table_name):
-    return _encode_path_component_re.sub(
-        lambda m: "U+{0:0{1}x}".format(ord(m.group(0)), 4).upper(), table_name
-    )
-
-
 def make_dockerfile(
     files,
     metadata_file,
@@ -293,6 +274,7 @@ def make_dockerfile(
     install,
     spatialite,
     version_note,
+    environment_variables=None,
 ):
     cmd = ["datasette", "serve", "--host", "0.0.0.0"]
     for filename in files:
@@ -328,11 +310,18 @@ FROM python:3.6
 COPY . /app
 WORKDIR /app
 {spatialite_extras}
+{environment_variables}
 RUN pip install -U {install_from}
 RUN datasette inspect {files} --inspect-file inspect-data.json
 ENV PORT 8001
 EXPOSE 8001
 CMD {cmd}""".format(
+        environment_variables="\n".join(
+            [
+                "ENV {} '{}'".format(key, value)
+                for key, value in (environment_variables or {}).items()
+            ]
+        ),
         files=" ".join(files),
         cmd=cmd,
         install_from=" ".join(install),
@@ -354,6 +343,7 @@ def temporary_docker_directory(
     spatialite,
     version_note,
     extra_metadata=None,
+    environment_variables=None,
 ):
     extra_metadata = extra_metadata or {}
     tmp = tempfile.TemporaryDirectory()
@@ -382,6 +372,7 @@ def temporary_docker_directory(
             install,
             spatialite,
             version_note,
+            environment_variables,
         )
         os.chdir(datasette_dir)
         if metadata_content:
@@ -480,6 +471,7 @@ def detect_fts_sql(table):
             where rootpage = 0
             and (
                 sql like '%VIRTUAL TABLE%USING FTS%content="{table}"%'
+                or sql like '%VIRTUAL TABLE%USING FTS%content=[{table}]%'
                 or (
                     tbl_name = "{table}"
                     and sql like '%VIRTUAL TABLE%USING FTS%'
@@ -609,7 +601,7 @@ def link_or_copy_directory(src, dst):
 
 def module_from_path(path, name):
     # Adapted from http://sayspy.blogspot.com/2011/07/how-to-import-module-from-just-file.html
-    mod = imp.new_module(name)
+    mod = types.ModuleType(name)
     mod.__file__ = path
     with open(path, "r") as file:
         code = compile(file.read(), path, "exec", dont_inherit=True)
@@ -641,6 +633,7 @@ def get_plugins(pm):
         distinfo = plugin_to_distinfo.get(plugin)
         if distinfo:
             plugin_info["version"] = distinfo.version
+            plugin_info["name"] = distinfo.project_name
         plugins.append(plugin_info)
     return plugins
 
@@ -718,13 +711,13 @@ class LimitedWriter:
         self.limit_bytes = limit_mb * 1024 * 1024
         self.bytes_count = 0
 
-    def write(self, bytes):
+    async def write(self, bytes):
         self.bytes_count += len(bytes)
         if self.limit_bytes and (self.bytes_count > self.limit_bytes):
             raise WriteLimitExceeded(
                 "CSV contains more than {} bytes".format(self.limit_bytes)
             )
-        self.writer.write(bytes)
+        await self.writer.write(bytes)
 
 
 _infinities = {float("inf"), float("-inf")}
@@ -746,7 +739,8 @@ class StaticMount(click.ParamType):
                 param,
                 ctx,
             )
-        path, dirpath = value.split(":")
+        path, dirpath = value.split(":", 1)
+        dirpath = os.path.abspath(dirpath)
         if not os.path.exists(dirpath) or not os.path.isdir(dirpath):
             self.fail("%s is not a valid directory path" % value, param, ctx)
         return path, dirpath
@@ -762,3 +756,16 @@ def format_bytes(bytes):
         return "{} {}".format(int(current), unit)
     else:
         return "{:.1f} {}".format(current, unit)
+
+
+class RequestParameters(dict):
+    def get(self, name, default=None):
+        "Return first value in the list, if available"
+        try:
+            return super().get(name)[0]
+        except (KeyError, TypeError):
+            return default
+
+    def getlist(self, name, default=None):
+        "Return full list"
+        return super().get(name, default)

datasette/utils/asgi.py

@@ -0,0 +1,384 @@
+import json
+from datasette.utils import RequestParameters
+from mimetypes import guess_type
+from urllib.parse import parse_qs, urlunparse
+from pathlib import Path
+from html import escape
+import re
+import aiofiles
+
+
+class NotFound(Exception):
+    pass
+
+
+class Request:
+    def __init__(self, scope):
+        self.scope = scope
+
+    @property
+    def method(self):
+        return self.scope["method"]
+
+    @property
+    def url(self):
+        return urlunparse(
+            (self.scheme, self.host, self.path, None, self.query_string, None)
+        )
+
+    @property
+    def scheme(self):
+        return self.scope.get("scheme") or "http"
+
+    @property
+    def headers(self):
+        return dict(
+            [
+                (k.decode("latin-1").lower(), v.decode("latin-1"))
+                for k, v in self.scope.get("headers") or []
+            ]
+        )
+
+    @property
+    def host(self):
+        return self.headers.get("host") or "localhost"
+
+    @property
+    def path(self):
+        if "raw_path" in self.scope:
+            return self.scope["raw_path"].decode("latin-1")
+        else:
+            return self.scope["path"].decode("utf-8")
+
+    @property
+    def query_string(self):
+        return (self.scope.get("query_string") or b"").decode("latin-1")
+
+    @property
+    def args(self):
+        return RequestParameters(parse_qs(qs=self.query_string))
+
+    @property
+    def raw_args(self):
+        return {key: value[0] for key, value in self.args.items()}
+
+    @classmethod
+    def fake(cls, path_with_query_string, method="GET", scheme="http"):
+        "Useful for constructing Request objects for tests"
+        path, _, query_string = path_with_query_string.partition("?")
+        scope = {
+            "http_version": "1.1",
+            "method": method,
+            "path": path,
+            "raw_path": path.encode("latin-1"),
+            "query_string": query_string.encode("latin-1"),
+            "scheme": scheme,
+            "type": "http",
+        }
+        return cls(scope)
+
+
+class AsgiRouter:
+    def __init__(self, routes=None):
+        routes = routes or []
+        self.routes = [
+            # Compile any strings to regular expressions
+            ((re.compile(pattern) if isinstance(pattern, str) else pattern), view)
+            for pattern, view in routes
+        ]
+
+    async def __call__(self, scope, receive, send):
+        # Because we care about "foo/bar" v.s. "foo%2Fbar" we decode raw_path ourselves
+        path = scope["path"]
+        raw_path = scope.get("raw_path")
+        if raw_path:
+            path = raw_path.decode("ascii")
+        for regex, view in self.routes:
+            match = regex.match(path)
+            if match is not None:
+                new_scope = dict(scope, url_route={"kwargs": match.groupdict()})
+                try:
+                    return await view(new_scope, receive, send)
+                except Exception as exception:
+                    return await self.handle_500(scope, receive, send, exception)
+        return await self.handle_404(scope, receive, send)
+
+    async def handle_404(self, scope, receive, send):
+        await send(
+            {
+                "type": "http.response.start",
+                "status": 404,
+                "headers": [[b"content-type", b"text/html"]],
+            }
+        )
+        await send({"type": "http.response.body", "body": b"<h1>404</h1>"})
+
+    async def handle_500(self, scope, receive, send, exception):
+        await send(
+            {
+                "type": "http.response.start",
+                "status": 404,
+                "headers": [[b"content-type", b"text/html"]],
+            }
+        )
+        html = "<h1>500</h1><pre{}></pre>".format(escape(repr(exception)))
+        await send({"type": "http.response.body", "body": html.encode("latin-1")})
+
+
+class AsgiLifespan:
+    def __init__(self, app, on_startup=None, on_shutdown=None):
+        self.app = app
+        on_startup = on_startup or []
+        on_shutdown = on_shutdown or []
+        if not isinstance(on_startup or [], list):
+            on_startup = [on_startup]
+        if not isinstance(on_shutdown or [], list):
+            on_shutdown = [on_shutdown]
+        self.on_startup = on_startup
+        self.on_shutdown = on_shutdown
+
+    async def __call__(self, scope, receive, send):
+        if scope["type"] == "lifespan":
+            while True:
+                message = await receive()
+                if message["type"] == "lifespan.startup":
+                    for fn in self.on_startup:
+                        await fn()
+                    await send({"type": "lifespan.startup.complete"})
+                elif message["type"] == "lifespan.shutdown":
+                    for fn in self.on_shutdown:
+                        await fn()
+                    await send({"type": "lifespan.shutdown.complete"})
+                    return
+        else:
+            await self.app(scope, receive, send)
+
+
+class AsgiView:
+    def dispatch_request(self, request, *args, **kwargs):
+        handler = getattr(self, request.method.lower(), None)
+        return handler(request, *args, **kwargs)
+
+    @classmethod
+    def as_asgi(cls, *class_args, **class_kwargs):
+        async def view(scope, receive, send):
+            # Uses scope to create a request object, then dispatches that to
+            # self.get(...) or self.options(...) along with keyword arguments
+            # that were already tucked into scope["url_route"]["kwargs"] by
+            # the router, similar to how Django Channels works:
+            # https://channels.readthedocs.io/en/latest/topics/routing.html#urlrouter
+            request = Request(scope)
+            self = view.view_class(*class_args, **class_kwargs)
+            response = await self.dispatch_request(
+                request, **scope["url_route"]["kwargs"]
+            )
+            await response.asgi_send(send)
+
+        view.view_class = cls
+        view.__doc__ = cls.__doc__
+        view.__module__ = cls.__module__
+        view.__name__ = cls.__name__
+        return view
+
+
+class AsgiStream:
+    def __init__(self, stream_fn, status=200, headers=None, content_type="text/plain"):
+        self.stream_fn = stream_fn
+        self.status = status
+        self.headers = headers or {}
+        self.content_type = content_type
+
+    async def asgi_send(self, send):
+        # Remove any existing content-type header
+        headers = dict(
+            [(k, v) for k, v in self.headers.items() if k.lower() != "content-type"]
+        )
+        headers["content-type"] = self.content_type
+        await send(
+            {
+                "type": "http.response.start",
+                "status": self.status,
+                "headers": [
+                    [key.encode("utf-8"), value.encode("utf-8")]
+                    for key, value in headers.items()
+                ],
+            }
+        )
+        w = AsgiWriter(send)
+        await self.stream_fn(w)
+        await send({"type": "http.response.body", "body": b""})
+
+
+class AsgiWriter:
+    def __init__(self, send):
+        self.send = send
+
+    async def write(self, chunk):
+        await self.send(
+            {
+                "type": "http.response.body",
+                "body": chunk.encode("utf-8"),
+                "more_body": True,
+            }
+        )
+
+
+async def asgi_send_json(send, info, status=200, headers=None):
+    headers = headers or {}
+    await asgi_send(
+        send,
+        json.dumps(info),
+        status=status,
+        headers=headers,
+        content_type="application/json; charset=utf-8",
+    )
+
+
+async def asgi_send_html(send, html, status=200, headers=None):
+    headers = headers or {}
+    await asgi_send(
+        send, html, status=status, headers=headers, content_type="text/html"
+    )
+
+
+async def asgi_send_redirect(send, location, status=302):
+    await asgi_send(
+        send,
+        "",
+        status=status,
+        headers={"Location": location},
+        content_type="text/html",
+    )
+
+
+async def asgi_send(send, content, status, headers=None, content_type="text/plain"):
+    await asgi_start(send, status, headers, content_type)
+    await send({"type": "http.response.body", "body": content.encode("latin-1")})
+
+
+async def asgi_start(send, status, headers=None, content_type="text/plain"):
+    headers = headers or {}
+    # Remove any existing content-type header
+    headers = dict([(k, v) for k, v in headers.items() if k.lower() != "content-type"])
+    headers["content-type"] = content_type
+    await send(
+        {
+            "type": "http.response.start",
+            "status": status,
+            "headers": [
+                [key.encode("latin1"), value.encode("latin1")]
+                for key, value in headers.items()
+            ],
+        }
+    )
+
+
+async def asgi_send_file(
+    send, filepath, filename=None, content_type=None, chunk_size=4096
+):
+    headers = {}
+    if filename:
+        headers["Content-Disposition"] = 'attachment; filename="{}"'.format(filename)
+    first = True
+    async with aiofiles.open(str(filepath), mode="rb") as fp:
+        if first:
+            await asgi_start(
+                send,
+                200,
+                headers,
+                content_type or guess_type(str(filepath))[0] or "text/plain",
+            )
+            first = False
+        more_body = True
+        while more_body:
+            chunk = await fp.read(chunk_size)
+            more_body = len(chunk) == chunk_size
+            await send(
+                {"type": "http.response.body", "body": chunk, "more_body": more_body}
+            )
+
+
+def asgi_static(root_path, chunk_size=4096, headers=None, content_type=None):
+    async def inner_static(scope, receive, send):
+        path = scope["url_route"]["kwargs"]["path"]
+        try:
+            full_path = (Path(root_path) / path).resolve().absolute()
+        except FileNotFoundError:
+            await asgi_send_html(send, "404", 404)
+            return
+        # Ensure full_path is within root_path to avoid weird "../" tricks
+        try:
+            full_path.relative_to(root_path)
+        except ValueError:
+            await asgi_send_html(send, "404", 404)
+            return
+        try:
+            await asgi_send_file(send, full_path, chunk_size=chunk_size)
+        except FileNotFoundError:
+            await asgi_send_html(send, "404", 404)
+            return
+
+    return inner_static
+
+
+class Response:
+    def __init__(self, body=None, status=200, headers=None, content_type="text/plain"):
+        self.body = body
+        self.status = status
+        self.headers = headers or {}
+        self.content_type = content_type
+
+    async def asgi_send(self, send):
+        headers = {}
+        headers.update(self.headers)
+        headers["content-type"] = self.content_type
+        await send(
+            {
+                "type": "http.response.start",
+                "status": self.status,
+                "headers": [
+                    [key.encode("utf-8"), value.encode("utf-8")]
+                    for key, value in headers.items()
+                ],
+            }
+        )
+        body = self.body
+        if not isinstance(body, bytes):
+            body = body.encode("utf-8")
+        await send({"type": "http.response.body", "body": body})
+
+    @classmethod
+    def html(cls, body, status=200, headers=None):
+        return cls(
+            body,
+            status=status,
+            headers=headers,
+            content_type="text/html; charset=utf-8",
+        )
+
+    @classmethod
+    def text(cls, body, status=200, headers=None):
+        return cls(
+            body,
+            status=status,
+            headers=headers,
+            content_type="text/plain; charset=utf-8",
+        )
+
+    @classmethod
+    def redirect(cls, path, status=302, headers=None):
+        headers = headers or {}
+        headers["Location"] = path
+        return cls("", status=status, headers=headers)
+
+
+class AsgiFileDownload:
+    def __init__(
+        self, filepath, filename=None, content_type="application/octet-stream"
+    ):
+        self.filepath = filepath
+        self.filename = filename
+        self.content_type = content_type
+
+    async def asgi_send(self, send):
+        return await asgi_send_file(send, self.filepath, content_type=self.content_type)

datasette/views/base.py

@@ -7,9 +7,8 @@ import urllib
 
 import jinja2
 import pint
-from sanic import response
+
-from sanic.exceptions import NotFound
+from html import escape
-from sanic.views import HTTPMethodView
 
 from datasette import __version__
 from datasette.plugins import pm
@@ -26,6 +25,14 @@ from datasette.utils import (
     sqlite3,
     to_css_class,
 )
+from datasette.utils.asgi import (
+    AsgiStream,
+    AsgiWriter,
+    AsgiRouter,
+    AsgiView,
+    NotFound,
+    Response,
+)
 
 ureg = pint.UnitRegistry()
 
@@ -49,7 +56,14 @@ class DatasetteError(Exception):
         self.messagge_is_html = messagge_is_html
 
 
-class RenderMixin(HTTPMethodView):
+class BaseView(AsgiView):
+    ds = None
+
+    async def head(self, *args, **kwargs):
+        response = await self.get(*args, **kwargs)
+        response.body = b""
+        return response
+
     def _asset_urls(self, key, template, context):
         # Flatten list-of-lists from plugins:
         seen_urls = set()
@@ -88,7 +102,7 @@ class RenderMixin(HTTPMethodView):
     def database_color(self, database):
         return "ff0000"
 
-    def render(self, templates, **context):
+    async def render(self, templates, request, context):
         template = self.ds.jinja_env.select_template(templates)
         select_templates = [
             "{}{}".format("*" if template_name == template.name else "", template_name)
@@ -104,7 +118,27 @@ class RenderMixin(HTTPMethodView):
             datasette=self.ds,
         ):
             body_scripts.append(jinja2.Markup(script))
-        return response.html(
+
+        extra_template_vars = {}
+        # pylint: disable=no-member
+        for extra_vars in pm.hook.extra_template_vars(
+            template=template.name,
+            database=context.get("database"),
+            table=context.get("table"),
+            view_name=self.name,
+            request=request,
+            datasette=self.ds,
+        ):
+            if callable(extra_vars):
+                extra_vars = extra_vars()
+            if asyncio.iscoroutine(extra_vars):
+                extra_vars = await extra_vars
+            assert isinstance(extra_vars, dict), "extra_vars is of type {}".format(
+                type(extra_vars)
+            )
+            extra_template_vars.update(extra_vars)
+
+        return Response.html(
             template.render(
                 {
                     **context,
@@ -123,12 +157,13 @@ class RenderMixin(HTTPMethodView):
                         "database_url": self.database_url,
                         "database_color": self.database_color,
                     },
+                    **extra_template_vars,
                 }
             )
         )
 
 
-class BaseView(RenderMixin):
+class DataView(BaseView):
     name = ""
     re_named_parameter = re.compile(":([a-zA-Z0-9_]+)")
 
@@ -136,7 +171,7 @@ class BaseView(RenderMixin):
         self.ds = datasette
 
     def options(self, request, *args, **kwargs):
-        r = response.text("ok")
+        r = Response.text("ok")
         if self.ds.cors:
             r.headers["Access-Control-Allow-Origin"] = "*"
         return r
@@ -146,7 +181,7 @@ class BaseView(RenderMixin):
             path = "{}?{}".format(path, request.query_string)
         if remove_args:
             path = path_with_removed_args(request, remove_args, path=path)
-        r = response.redirect(path)
+        r = Response.redirect(path)
         r.headers["Link"] = "<{}>; rel=preload".format(path)
         if self.ds.cors:
             r.headers["Access-Control-Allow-Origin"] = "*"
@@ -158,22 +193,23 @@ class BaseView(RenderMixin):
     async def resolve_db_name(self, request, db_name, **kwargs):
         hash = None
         name = None
-        if "-" in db_name:
+        if db_name not in self.ds.databases and "-" in db_name:
-            # Might be name-and-hash, or might just be
+            # No matching DB found, maybe it's a name-hash?
-            # a name with a hyphen in it
+            name_bit, hash_bit = db_name.rsplit("-", 1)
-            name, hash = db_name.rsplit("-", 1)
+            if name_bit not in self.ds.databases:
-            if name not in self.ds.databases:
+                raise NotFound("Database not found: {}".format(name))
-                # Try the whole name
+            else:
-                name = db_name
+                name = name_bit
-                hash = None
+                hash = hash_bit
         else:
             name = db_name
-        # Verify the hash
+        name = urllib.parse.unquote_plus(name)
         try:
             db = self.ds.databases[name]
         except KeyError:
             raise NotFound("Database not found: {}".format(name))
 
+        # Verify the hash
         expected = "000"
         if db.hash is not None:
             expected = db.hash[:HASH_LENGTH]
@@ -195,17 +231,17 @@ class BaseView(RenderMixin):
                 kwargs["table"] = table
                 if _format:
                     kwargs["as_format"] = ".{}".format(_format)
-            elif "table" in kwargs:
+            elif kwargs.get("table"):
                 kwargs["table"] = urllib.parse.unquote_plus(kwargs["table"])
 
             should_redirect = "/{}-{}".format(name, expected)
-            if "table" in kwargs:
+            if kwargs.get("table"):
                 should_redirect += "/" + urllib.parse.quote_plus(kwargs["table"])
-            if "pk_path" in kwargs:
+            if kwargs.get("pk_path"):
                 should_redirect += "/" + kwargs["pk_path"]
-            if "as_format" in kwargs:
+            if kwargs.get("as_format"):
                 should_redirect += kwargs["as_format"]
-            if "as_db" in kwargs:
+            if kwargs.get("as_db"):
                 should_redirect += kwargs["as_db"]
 
             if (
@@ -222,9 +258,12 @@ class BaseView(RenderMixin):
         assert NotImplemented
 
     async def get(self, request, db_name, **kwargs):
-        database, hash, correct_hash_provided, should_redirect = await self.resolve_db_name(
+        (
-            request, db_name, **kwargs
+            database,
-        )
+            hash,
+            correct_hash_provided,
+            should_redirect,
+        ) = await self.resolve_db_name(request, db_name, **kwargs)
         if should_redirect:
             return self.redirect(request, should_redirect, remove_args={"_hash"})
 
@@ -246,7 +285,7 @@ class BaseView(RenderMixin):
             response_or_template_contexts = await self.data(
                 request, database, hash, **kwargs
             )
-            if isinstance(response_or_template_contexts, response.HTTPResponse):
+            if isinstance(response_or_template_contexts, Response):
                 return response_or_template_contexts
             else:
                 data, _, _ = response_or_template_contexts
@@ -282,26 +321,30 @@ class BaseView(RenderMixin):
                     if not first:
                         data, _, _ = await self.data(request, database, hash, **kwargs)
                     if first:
-                        writer.writerow(headings)
+                        await writer.writerow(headings)
                         first = False
                     next = data.get("next")
                     for row in data["rows"]:
                         if not expanded_columns:
                             # Simple path
-                            writer.writerow(row)
+                            await writer.writerow(row)
                         else:
                             # Look for {"value": "label": } dicts and expand
                             new_row = []
-                            for cell in row:
+                            for heading, cell in zip(data["columns"], row):
-                                if isinstance(cell, dict):
+                                if heading in expanded_columns:
-                                    new_row.append(cell["value"])
+                                    if cell is None:
-                                    new_row.append(cell["label"])
+                                        new_row.extend(("", ""))
+                                    else:
+                                        assert isinstance(cell, dict)
+                                        new_row.append(cell["value"])
+                                        new_row.append(cell["label"])
                                 else:
                                     new_row.append(cell)
-                            writer.writerow(new_row)
+                            await writer.writerow(new_row)
                 except Exception as e:
                     print("caught this", e)
-                    r.write(str(e))
+                    await r.write(str(e))
                     return
 
         content_type = "text/plain; charset=utf-8"
@@ -315,7 +358,7 @@ class BaseView(RenderMixin):
             )
             headers["Content-Disposition"] = disposition
 
-        return response.stream(stream_fn, headers=headers, content_type=content_type)
+        return AsgiStream(stream_fn, headers=headers, content_type=content_type)
 
     async def get_format(self, request, database, args):
         """ Determine the format of the response from the request, from URL
@@ -327,6 +370,8 @@ class BaseView(RenderMixin):
         _format = request.args.get("_format", None)
         if not _format:
             _format = (args.pop("as_format", None) or "").lstrip(".")
+        else:
+            args.pop("as_format", None)
         if "table_and_format" in args:
             db = self.ds.databases[database]
 
@@ -352,7 +397,7 @@ class BaseView(RenderMixin):
             return await self.as_csv(request, database, hash, **kwargs)
 
         if _format is None:
-            # HTML views default to expanding all foriegn key labels
+            # HTML views default to expanding all foreign key labels
             kwargs["default_labels"] = True
 
         extra_template_data = {}
@@ -363,7 +408,7 @@ class BaseView(RenderMixin):
             response_or_template_contexts = await self.data(
                 request, database, hash, **kwargs
             )
-            if isinstance(response_or_template_contexts, response.HTTPResponse):
+            if isinstance(response_or_template_contexts, Response):
                 return response_or_template_contexts
 
             else:
@@ -414,17 +459,11 @@ class BaseView(RenderMixin):
             if result is None:
                 raise NotFound("No data")
 
-            response_args = {
+            r = Response(
-                "content_type": result.get("content_type", "text/plain"),
+                body=result.get("body"),
-                "status": result.get("status_code", 200),
+                status=result.get("status_code", 200),
-            }
+                content_type=result.get("content_type", "text/plain"),
-
+            )
-            if type(result.get("body")) == bytes:
-                response_args["body_bytes"] = result.get("body")
-            else:
-                response_args["body"] = result.get("body")
-
-            r = response.HTTPResponse(**response_args)
         else:
             extras = {}
             if callable(extra_template_data):
@@ -463,7 +502,7 @@ class BaseView(RenderMixin):
             }
             if "metadata" not in context:
                 context["metadata"] = self.ds.metadata
-            r = self.render(templates, **context)
+            r = await self.render(templates, request=request, context=context)
             r.status = status_code
 
         ttl = request.args.get("_ttl", None)

datasette/views/database.py

@@ -1,22 +1,26 @@
 import os
 
-from sanic import response
-
 from datasette.utils import to_css_class, validate_sql_select
+from datasette.utils.asgi import AsgiFileDownload
 
-from .base import BaseView, DatasetteError
+from .base import DatasetteError, DataView
 
 
-class DatabaseView(BaseView):
+class DatabaseView(DataView):
     name = "database"
 
     async def data(self, request, database, hash, default_labels=False, _size=None):
+        metadata = (self.ds.metadata("databases") or {}).get(database, {})
+        self.ds.update_with_inherited_metadata(metadata)
+
         if request.args.get("sql"):
             if not self.ds.config("allow_sql"):
                 raise DatasetteError("sql= is not allowed", status=400)
             sql = request.raw_args.pop("sql")
             validate_sql_select(sql)
-            return await self.custom_sql(request, database, hash, sql, _size=_size)
+            return await self.custom_sql(
+                request, database, hash, sql, _size=_size, metadata=metadata
+            )
 
         db = self.ds.databases[database]
 
@@ -25,9 +29,6 @@ class DatabaseView(BaseView):
         hidden_table_names = set(await db.hidden_table_names())
         all_foreign_keys = await db.get_all_foreign_keys()
 
-        metadata = (self.ds.metadata("databases") or {}).get(database, {})
-        self.ds.update_with_inherited_metadata(metadata)
-
         tables = []
         for table in table_counts:
             table_columns = await db.table_columns(table)
@@ -65,7 +66,7 @@ class DatabaseView(BaseView):
         )
 
 
-class DatabaseDownload(BaseView):
+class DatabaseDownload(DataView):
     name = "database_download"
 
     async def view_get(self, request, database, hash, correct_hash_present, **kwargs):
@@ -79,8 +80,8 @@ class DatabaseDownload(BaseView):
         if not db.path:
             raise DatasetteError("Cannot download database", status=404)
         filepath = db.path
-        return await response.file_stream(
+        return AsgiFileDownload(
             filepath,
             filename=os.path.basename(filepath),
-            mime_type="application/octet-stream",
+            content_type="application/octet-stream",
         )

datasette/views/index.py

@@ -1,12 +1,11 @@
 import hashlib
 import json
 
-from sanic import response
-
 from datasette.utils import CustomJSONEncoder
+from datasette.utils.asgi import Response
 from datasette.version import __version__
 
-from .base import RenderMixin
+from .base import BaseView
 
 
 # Truncate table list on homepage at:
@@ -16,7 +15,7 @@ TRUNCATE_AT = 5
 COUNT_TABLE_LIMIT = 30
 
 
-class IndexView(RenderMixin):
+class IndexView(BaseView):
     name = "index"
 
     def __init__(self, datasette):
@@ -98,21 +97,22 @@ class IndexView(RenderMixin):
                 }
             )
 
-        databases.sort(key=lambda database: database["name"])
-
         if as_format:
             headers = {}
             if self.ds.cors:
                 headers["Access-Control-Allow-Origin"] = "*"
-            return response.HTTPResponse(
+            return Response(
                 json.dumps({db["name"]: db for db in databases}, cls=CustomJSONEncoder),
-                content_type="application/json",
+                content_type="application/json; charset=utf-8",
                 headers=headers,
             )
         else:
-            return self.render(
+            return await self.render(
                 ["index.html"],
-                databases=databases,
+                request=request,
-                metadata=self.ds.metadata(),
+                context={
-                datasette_version=__version__,
+                    "databases": databases,
+                    "metadata": self.ds.metadata(),
+                    "datasette_version": __version__,
+                },
             )

datasette/views/special.py

@@ -1,9 +1,9 @@
 import json
-from sanic import response
+from datasette.utils.asgi import Response
-from .base import RenderMixin
+from .base import BaseView
 
 
-class JsonDataView(RenderMixin):
+class JsonDataView(BaseView):
     name = "json_data"
 
     def __init__(self, datasette, filename, data_callback):
@@ -17,9 +17,15 @@ class JsonDataView(RenderMixin):
             headers = {}
             if self.ds.cors:
                 headers["Access-Control-Allow-Origin"] = "*"
-            return response.HTTPResponse(
+            return Response(
-                json.dumps(data), content_type="application/json", headers=headers
+                json.dumps(data),
+                content_type="application/json; charset=utf-8",
+                headers=headers,
             )
 
         else:
-            return self.render(["show_json.html"], filename=self.filename, data=data)
+            return await self.render(
+                ["show_json.html"],
+                request=request,
+                context={"filename": self.filename, "data": data},
+            )

datasette/views/table.py

@@ -3,13 +3,12 @@ import itertools
 import json
 
 import jinja2
-from sanic.exceptions import NotFound
-from sanic.request import RequestParameters
 
 from datasette.plugins import pm
 from datasette.utils import (
     CustomRow,
     QueryInterrupted,
+    RequestParameters,
     append_querystring,
     compound_keys_after_sql,
     escape_sqlite,
@@ -24,8 +23,9 @@ from datasette.utils import (
     urlsafe_components,
     value_as_boolean,
 )
+from datasette.utils.asgi import NotFound
 from datasette.filters import Filters
-from .base import BaseView, DatasetteError, ureg
+from .base import DataView, DatasetteError, ureg
 
 LINK_WITH_LABEL = (
     '<a href="/{database}/{table}/{link_id}">{label}</a>&nbsp;<em>{id}</em>'
@@ -33,7 +33,36 @@ LINK_WITH_LABEL = (
 LINK_WITH_VALUE = '<a href="/{database}/{table}/{link_id}">{id}</a>'
 
 
-class RowTableShared(BaseView):
+class Row:
+    def __init__(self, cells):
+        self.cells = cells
+
+    def __iter__(self):
+        return iter(self.cells)
+
+    def __getitem__(self, key):
+        for cell in self.cells:
+            if cell["column"] == key:
+                return cell["raw"]
+        raise KeyError
+
+    def display(self, key):
+        for cell in self.cells:
+            if cell["column"] == key:
+                return cell["value"]
+        return None
+
+    def __str__(self):
+        d = {
+            key: self[key]
+            for key in [
+                c["column"] for c in self.cells if not c.get("is_special_link_column")
+            ]
+        }
+        return json.dumps(d, default=repr, indent=2)
+
+
+class RowTableShared(DataView):
     async def sortable_columns_for_table(self, database, table, use_rowid):
         db = self.ds.databases[database]
         table_metadata = self.ds.table_metadata(database, table)
@@ -76,18 +105,18 @@ class RowTableShared(BaseView):
             # Unless we are a view, the first column is a link - either to the rowid
             # or to the simple or compound primary key
             if link_column:
+                is_special_link_column = len(pks) != 1
+                pk_path = path_from_row_pks(row, pks, not pks, False)
                 cells.append(
                     {
                         "column": pks[0] if len(pks) == 1 else "Link",
+                        "is_special_link_column": is_special_link_column,
+                        "raw": pk_path,
                         "value": jinja2.Markup(
                             '<a href="/{database}/{table}/{flat_pks_quoted}">{flat_pks}</a>'.format(
                                 database=database,
                                 table=urllib.parse.quote_plus(table),
-                                flat_pks=str(
+                                flat_pks=str(jinja2.escape(pk_path)),
-                                    jinja2.escape(
-                                        path_from_row_pks(row, pks, not pks, False)
-                                    )
-                                ),
                                 flat_pks_quoted=path_from_row_pks(row, pks, not pks),
                             )
                         ),
@@ -159,8 +188,8 @@ class RowTableShared(BaseView):
                     if truncate_cells and len(display_value) > truncate_cells:
                         display_value = display_value[:truncate_cells] + u"\u2026"
 
-                cells.append({"column": column, "value": display_value})
+                cells.append({"column": column, "value": display_value, "raw": value})
-            cell_rows.append(cells)
+            cell_rows.append(Row(cells))
 
         if link_column:
             # Add the link column header.
@@ -206,21 +235,24 @@ class TableView(RowTableShared):
             raise NotFound("Table not found: {}".format(table))
 
         pks = await db.primary_keys(table)
+        table_columns = await db.table_columns(table)
+
+        select_columns = ", ".join(escape_sqlite(t) for t in table_columns)
+
         use_rowid = not pks and not is_view
         if use_rowid:
-            select = "rowid, *"
+            select = "rowid, {}".format(select_columns)
             order_by = "rowid"
             order_by_pks = "rowid"
         else:
-            select = "*"
+            select = select_columns
             order_by_pks = ", ".join([escape_sqlite(pk) for pk in pks])
             order_by = order_by_pks
 
         if is_view:
             order_by = ""
 
-        # We roll our own query_string decoder because by default Sanic
+        # Ensure we don't drop anything with an empty value e.g. ?name__exact=
-        # drops anything with an empty value e.g. ?name__exact=
         args = RequestParameters(
             urllib.parse.parse_qs(request.query_string, keep_blank_values=True)
         )
@@ -229,12 +261,10 @@ class TableView(RowTableShared):
         # That's so if there is a column that starts with _
         # it can still be queried using ?_col__exact=blah
         special_args = {}
-        special_args_lists = {}
         other_args = []
         for key, value in args.items():
             if key.startswith("_") and "__" not in key:
                 special_args[key] = value[0]
-                special_args_lists[key] = value
             else:
                 for v in value:
                     other_args.append((key, v))
@@ -467,18 +497,6 @@ class TableView(RowTableShared):
         if order_by:
             order_by = "order by {} ".format(order_by)
 
-        # _group_count=col1&_group_count=col2
-        group_count = special_args_lists.get("_group_count") or []
-        if group_count:
-            sql = 'select {group_cols}, count(*) as "count" from {table_name} {where} group by {group_cols} order by "count" desc limit 100'.format(
-                group_cols=", ".join(
-                    '"{}"'.format(group_count_col) for group_count_col in group_count
-                ),
-                table_name=escape_sqlite(table),
-                where=where_clause,
-            )
-            return await self.custom_sql(request, database, hash, sql, editable=True)
-
         extra_args = {}
         # Handle ?_size=500
         page_size = _size or request.raw_args.get("_size")
@@ -558,9 +576,10 @@ class TableView(RowTableShared):
             )
 
         for facet in facet_instances:
-            instance_facet_results, instance_facets_timed_out = (
+            (
-                await facet.facet_results()
+                instance_facet_results,
-            )
+                instance_facets_timed_out,
+            ) = await facet.facet_results()
             facet_results.update(instance_facet_results)
             facets_timed_out.extend(instance_facets_timed_out)
 
@@ -608,7 +627,7 @@ class TableView(RowTableShared):
                     new_row = CustomRow(columns)
                     for column in row.keys():
                         value = row[column]
-                        if (column, value) in expanded_labels:
+                        if (column, value) in expanded_labels and value is not None:
                             new_row[column] = {
                                 "value": value,
                                 "label": expanded_labels[(column, value)],
@@ -692,6 +711,9 @@ class TableView(RowTableShared):
             for arg in ("_fts_table", "_fts_pk"):
                 if arg in special_args:
                     form_hidden_args.append((arg, special_args[arg]))
+            if request.args.get("_where"):
+                for where_text in request.args["_where"]:
+                    form_hidden_args.append(("_where", where_text))
             return {
                 "supports_search": bool(fts_table),
                 "search": search or "",
@@ -716,14 +738,14 @@ class TableView(RowTableShared):
                 "sort": sort,
                 "sort_desc": sort_desc,
                 "disable_sort": is_view,
-                "custom_rows_and_columns_templates": [
+                "custom_table_templates": [
-                    "_rows_and_columns-{}-{}.html".format(
+                    "_table-{}-{}.html".format(
                         to_css_class(database), to_css_class(table)
                     ),
-                    "_rows_and_columns-table-{}-{}.html".format(
+                    "_table-table-{}-{}.html".format(
                         to_css_class(database), to_css_class(table)
                     ),
-                    "_rows_and_columns.html",
+                    "_table.html",
                 ],
                 "metadata": metadata,
                 "view_definition": await db.get_view_definition(table),
@@ -800,14 +822,14 @@ class RowView(RowTableShared):
                 ),
                 "display_columns": display_columns,
                 "display_rows": display_rows,
-                "custom_rows_and_columns_templates": [
+                "custom_table_templates": [
-                    "_rows_and_columns-{}-{}.html".format(
+                    "_table-{}-{}.html".format(
                         to_css_class(database), to_css_class(table)
                     ),
-                    "_rows_and_columns-row-{}-{}.html".format(
+                    "_table-row-{}-{}.html".format(
                         to_css_class(database), to_css_class(table)
                     ),
-                    "_rows_and_columns.html",
+                    "_table.html",
                 ],
                 "metadata": (self.ds.metadata("databases") or {})
                 .get(database, {})

docs/changelog.rst

@@ -4,6 +4,159 @@
 Changelog
 =========
 
+.. _v0_30_2:
+
+0.30.2 (2019-11-02)
+-------------------
+
+- ``/-/plugins`` page now uses distribution name e.g. ``datasette-cluster-map`` instead of the name of the underlying Python package (``datasette_cluster_map``) (`#606 <https://github.com/simonw/datasette/issues/606>`__)
+- Array faceting is now only suggested for columns that contain arrays of strings (`#562 <https://github.com/simonw/datasette/issues/562>`__)
+- Better documentation for the ``--host`` argument (`#574 <https://github.com/simonw/datasette/issues/574>`__)
+- Don't show ``None`` with a broken link for the label on a nullable foreign key (`#406 <https://github.com/simonw/datasette/issues/406>`__)
+
+.. _v0_30_1:
+
+0.30.1 (2019-10-30)
+-------------------
+
+- Fixed bug where ``?_where=`` parameter was not persisted in hidden form fields (`#604 <https://github.com/simonw/datasette/issues/604>`__)
+- Fixed bug with .JSON representation of row pages - thanks, Chris Shaw (`#603 <https://github.com/simonw/datasette/issues/603>`__)
+
+.. _v0_30:
+
+
+0.30 (2019-10-18)
+-----------------
+
+- Added ``/-/threads`` debugging page
+- Allow ``EXPLAIN WITH...`` (`#583 <https://github.com/simonw/datasette/issues/583>`__)
+- Button to format SQL - thanks, Tobias Kunze (`#136 <https://github.com/simonw/datasette/issues/136>`__)
+- Sort databases on homepage by argument order - thanks, Tobias Kunze (`#585 <https://github.com/simonw/datasette/issues/585>`__)
+- Display metadata footer on custom SQL queries - thanks, Tobias Kunze (`#589 <https://github.com/simonw/datasette/pull/589>`__)
+- Use ``--platform=managed`` for ``publish cloudrun`` (`#587 <https://github.com/simonw/datasette/issues/587>`__)
+- Fixed bug returning non-ASCII characters in CSV (`#584 <https://github.com/simonw/datasette/issues/584>`__)
+- Fix for ``/foo`` v.s. ``/foo-bar`` bug (`#601 <https://github.com/simonw/datasette/issues/601>`__)
+
+.. _v0_29_3:
+
+0.29.3 (2019-09-02)
+-------------------
+
+- Fixed implementation of CodeMirror on database page (`#560 <https://github.com/simonw/datasette/issues/560>`__)
+- Documentation typo fixes - thanks, Min ho Kim (`#561 <https://github.com/simonw/datasette/pull/561>`__)
+- Mechanism for detecting if a table has FTS enabled now works if the table name used alternative escaping mechanisms (`#570 <https://github.com/simonw/datasette/issues/570>`__) - for compatibility with `a recent change to sqlite-utils <https://github.com/simonw/sqlite-utils/pull/57>`__.
+
+.. _v0_29_2:
+
+0.29.2 (2019-07-13)
+-------------------
+
+- Bumped `Uvicorn <https://www.uvicorn.org/>`__ to 0.8.4, fixing a bug where the querystring was not included in the server logs. (`#559 <https://github.com/simonw/datasette/issues/559>`__)
+- Fixed bug where the navigation breadcrumbs were not displayed correctly on the page for a custom query. (`#558 <https://github.com/simonw/datasette/issues/558>`__)
+- Fixed bug where custom query names containing unicode characters caused errors.
+
+.. _v0_29_1:
+
+0.29.1 (2019-07-11)
+-------------------
+
+- Fixed bug with static mounts using relative paths which could lead to traversal exploits (`#555 <https://github.com/simonw/datasette/issues/555>`__) - thanks Abdussamet Kocak!
+- Datasette can now be run as a module: ``python -m datasette`` (`#556 <https://github.com/simonw/datasette/issues/556>`__) - thanks, Abdussamet Kocak!
+
+.. _v0_29:
+
+0.29 (2019-07-07)
+-----------------
+
+ASGI, new plugin hooks, facet by date and much, much more...
+
+ASGI
+~~~~
+
+`ASGI <https://asgi.readthedocs.io/>`__ is the Asynchronous Server Gateway Interface standard. I've been wanting to convert Datasette into an ASGI application for over a year - `Port Datasette to ASGI #272 <https://github.com/simonw/datasette/issues/272>`__ tracks thirteen months of intermittent development - but with Datasette 0.29 the change is finally released. This also means Datasette now runs on top of `Uvicorn <https://www.uvicorn.org/>`__ and no longer depends on `Sanic <https://github.com/huge-success/sanic>`__.
+
+I wrote about the significance of this change in `Porting Datasette to ASGI, and Turtles all the way down <https://simonwillison.net/2019/Jun/23/datasette-asgi/>`__.
+
+The most exciting consequence of this change is that Datasette plugins can now take advantage of the ASGI standard.
+
+New plugin hook: asgi_wrapper
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The :ref:`plugin_asgi_wrapper` plugin hook allows plugins to entirely wrap the Datasette ASGI application in their own ASGI middleware. (`#520 <https://github.com/simonw/datasette/issues/520>`__)
+
+Two new plugins take advantage of this hook:
+
+* `datasette-auth-github <https://github.com/simonw/datasette-auth-github>`__ adds a authentication layer: users will have to sign in using their GitHub account before they can view data or interact with Datasette. You can also use it to restrict access to specific GitHub users, or to members of specified GitHub `organizations <https://help.github.com/en/articles/about-organizations>`__ or `teams <https://help.github.com/en/articles/organizing-members-into-teams>`__.
+
+* `datasette-cors <https://github.com/simonw/datasette-cors>`__ allows you to configure `CORS headers <https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS>`__ for your Datasette instance. You can use this to enable JavaScript running on a whitelisted set of domains to make ``fetch()`` calls to the JSON API provided by your Datasette instance.
+
+New plugin hook: extra_template_vars
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The :ref:`plugin_hook_extra_template_vars` plugin hook allows plugins to inject their own additional variables into the Datasette template context. This can be used in conjunction with custom templates to customize the Datasette interface. `datasette-auth-github <https://github.com/simonw/datasette-auth-github>`__ uses this hook to add custom HTML to the new top navigation bar (which is designed to be modified by plugins, see `#540 <https://github.com/simonw/datasette/issues/540>`__).
+
+Secret plugin configuration options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Plugins like `datasette-auth-github <https://github.com/simonw/datasette-auth-github>`__ need a safe way to set secret configuration options. Since the default mechanism for configuring plugins exposes those settings in ``/-/metadata`` a new mechanism was needed. :ref:`plugins_configuration_secret` describes how plugins can now specify that their settings should be read from a file or an environment variable::
+
+    {
+        "plugins": {
+            "datasette-auth-github": {
+                "client_secret": {
+                    "$env": "GITHUB_CLIENT_SECRET"
+                }
+            }
+        }
+    }
+
+These plugin secrets can be set directly using ``datasette publish``. See :ref:`publish_custom_metadata_and_plugins` for details. (`#538 <https://github.com/simonw/datasette/issues/538>`__ and `#543 <https://github.com/simonw/datasette/issues/543>`__)
+
+Facet by date
+~~~~~~~~~~~~~
+
+If a column contains datetime values, Datasette can now facet that column by date. (`#481 <https://github.com/simonw/datasette/issues/481>`__)
+
+.. _v0_29_medium_changes:
+
+Easier custom templates for table rows
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you want to customize the display of individual table rows, you can do so using a ``_table.html`` template include that looks something like this::
+
+    {% for row in display_rows %}
+        <div>
+            <h2>{{ row["title"] }}</h2>
+            <p>{{ row["description"] }}<lp>
+            <p>Category: {{ row.display("category_id") }}</p>
+        </div>
+    {% endfor %}
+
+This is a **backwards incompatible change**. If you previously had a custom template called ``_rows_and_columns.html`` you need to rename it to ``_table.html``.
+
+See :ref:`customization_custom_templates` for full details.
+
+?_through= for joins through many-to-many tables
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The new ``?_through={json}`` argument to the Table view allows records to be filtered based on a many-to-many relationship. See :ref:`json_api_table_arguments` for full documentation - here's `an example <https://latest.datasette.io/fixtures/roadside_attractions?_through={%22table%22:%22roadside_attraction_characteristics%22,%22column%22:%22characteristic_id%22,%22value%22:%221%22}>`__. (`#355 <https://github.com/simonw/datasette/issues/355>`__)
+
+This feature was added to help support `facet by many-to-many <https://github.com/simonw/datasette/issues/551>`__, which isn't quite ready yet but will be coming in the next Datasette release.
+
+Small changes
+~~~~~~~~~~~~~
+
+* Databases published using ``datasette publish`` now open in :ref:`performance_immutable_mode`. (`#469 <https://github.com/simonw/datasette/issues/469>`__)
+* ``?col__date=`` now works for columns containing spaces
+* Automatic label detection (for deciding which column to show when linking to a foreign key) has been improved. (`#485 <https://github.com/simonw/datasette/issues/485>`__)
+* Fixed bug where pagination broke when combined with an expanded foreign key. (`#489 <https://github.com/simonw/datasette/issues/489>`__)
+* Contributors can now run ``pip install -e .[docs]`` to get all of the dependencies needed to build the documentation, including ``cd docs && make livehtml`` support.
+* Datasette's dependencies are now all specified using the ``~=`` match operator. (`#532 <https://github.com/simonw/datasette/issues/532>`__)
+* ``white-space: pre-wrap`` now used for table creation SQL. (`#505 <https://github.com/simonw/datasette/issues/505>`__)
+
+
+`Full list of commits <https://github.com/simonw/datasette/compare/0.28...0.29>`__ between 0.28 and 0.29.
+
 .. _v0_28:
 
 0.28 (2019-05-19)
@@ -31,7 +184,7 @@ Datasette can still run against immutable files and gains numerous performance b
 Faceting improvements, and faceting plugins
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Datasette :ref:`facets` provide an intuitive way to quickly summarize and interact with data. Previously the only supported faceting technique was column faceting, but 0.28 introduces two powerful new capibilities: facet-by-JSON-array and the ability to define further facet types using plugins.
+Datasette :ref:`facets` provide an intuitive way to quickly summarize and interact with data. Previously the only supported faceting technique was column faceting, but 0.28 introduces two powerful new capabilities: facet-by-JSON-array and the ability to define further facet types using plugins.
 
 Facet by array (`#359 <https://github.com/simonw/datasette/issues/359>`__) is only available if your SQLite installation provides the ``json1`` extension. Datasette will automatically detect columns that contain JSON arrays of values and offer a faceting interface against those columns - useful for modelling things like tags without needing to break them out into a new table. See :ref:`facet_by_json_array` for more.
 
@@ -42,7 +195,7 @@ The new :ref:`plugin_register_facet_classes` plugin hook (`#445 <https://github.
 datasette publish cloudrun
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-`Google Cloud Run <https://cloud.google.com/run/>`__ is a brand new serverless hosting platform from Google, which allows you to build a Docker container which will run only when HTTP traffic is recieved and will shut down (and hence cost you nothing) the rest of the time. It's similar to Zeit's Now v1 Docker hosting platform which sadly is `no longer accepting signups <https://hyperion.alpha.spectrum.chat/zeit/now/cannot-create-now-v1-deployments~d206a0d4-5835-4af5-bb5c-a17f0171fb25?m=MTU0Njk2NzgwODM3OA==>`__ from new users.
+`Google Cloud Run <https://cloud.google.com/run/>`__ is a brand new serverless hosting platform from Google, which allows you to build a Docker container which will run only when HTTP traffic is received and will shut down (and hence cost you nothing) the rest of the time. It's similar to Zeit's Now v1 Docker hosting platform which sadly is `no longer accepting signups <https://hyperion.alpha.spectrum.chat/zeit/now/cannot-create-now-v1-deployments~d206a0d4-5835-4af5-bb5c-a17f0171fb25?m=MTU0Njk2NzgwODM3OA==>`__ from new users.
 
 The new ``datasette publish cloudrun`` command was contributed by Romain Primet (`#434 <https://github.com/simonw/datasette/pull/434>`__) and publishes selected databases to a new Datasette instance running on Google Cloud Run.
 
@@ -481,7 +634,7 @@ Mostly new work on the :ref:`plugins` mechanism: plugins can now bundle static a
 - Longer time limit for test_paginate_compound_keys
 
   It was failing intermittently in Travis - see `#209 <https://github.com/simonw/datasette/issues/209>`_
-- Use application/octet-stream for downloadable databses
+- Use application/octet-stream for downloadable databases
 - Updated PyPI classifiers
 - Updated PyPI link to pypi.org
 

docs/contributing.rst

@@ -38,7 +38,7 @@ The next step is to create a virtual environment for your project and use it to
     # Now activate the virtual environment, so pip can install into it
     source venv/bin/activate
     # Install Datasette and its testing dependencies
-    python3 -m pip -e .[test]
+    python3 -m pip install -e .[test]
 
 That last line does most of the work: ``pip install -e`` means "install this package in a way that allows me to edit the source code in place". The ``.[test]`` option means "use the setup.py in this directory and install the optional testing dependencies as well".
 
@@ -91,7 +91,7 @@ You can build it locally by installing ``sphinx`` and ``sphinx_rtd_theme`` in yo
     source venv/bin/activate
 
     # Install the dependencies needed to build the docs
-    pip install sphinx sphinx_rtd_theme
+    pip install -e .[docs]
 
     # Now build the docs
     cd docs/
@@ -103,16 +103,14 @@ This will create the HTML version of the documentation in ``docs/_build/html``.
 
 Any time you make changes to a ``.rst`` file you can re-run ``make html`` to update the built documents, then refresh them in your browser.
 
-For added productivity, you can run Sphinx in auto-build mode. This will run a local webserver serving the docs that automatically rebuilds them and refreshes the page any time you hit save in your editor.
+For added productivity, you can use use `sphinx-autobuild <https://pypi.org/project/sphinx-autobuild/>`__ to run Sphinx in auto-build mode. This will run a local webserver serving the docs that automatically rebuilds them and refreshes the page any time you hit save in your editor.
 
-To enable auto-build mode, first install `sphinx-autobuild <https://pypi.org/project/sphinx-autobuild/>`__::
+``sphinx-autobuild`` will have been installed when you ran ``pip install -e .[docs]``. In your ``docs/`` directory you can start the server by running the following::
-
-    pip install sphinx-autobuild
-
-Now start the server by running::
 
     make livehtml
 
+Now browse to ``http://localhost:8000/`` to view the documentation. Any edits you make should be instantly reflected in your browser.
+
 .. _contributing_release:
 
 Release process

docs/custom_templates.rst

@@ -102,6 +102,8 @@ database column they are representing, for example::
         </tbody>
     </table>
 
+.. _customization_custom_templates:
+
 Custom templates
 ----------------
 
@@ -144,15 +146,15 @@ The lookup rules Datasette uses are as follows::
         row-mydatabase-mytable.html
         row.html
 
-    Rows and columns include on table page:
+    Table of rows and columns include on table page:
-        _rows_and_columns-table-mydatabase-mytable.html
+        _table-table-mydatabase-mytable.html
-        _rows_and_columns-mydatabase-mytable.html
+        _table-mydatabase-mytable.html
-        _rows_and_columns.html
+        _table.html
 
-    Rows and columns include on row page:
+    Table of rows and columns include on row page:
-        _rows_and_columns-row-mydatabase-mytable.html
+        _table-row-mydatabase-mytable.html
-        _rows_and_columns-mydatabase-mytable.html
+        _table-mydatabase-mytable.html
-        _rows_and_columns.html
+        _table.html
 
 If a table name has spaces or other unexpected characters in it, the template
 filename will follow the same rules as our custom ``<body>`` CSS classes - for
@@ -189,38 +191,28 @@ content you can do so by creating a ``row.html`` template like this::
 Note the ``default:row.html`` template name, which ensures Jinja will inherit
 from the default template.
 
-The ``_rows_and_columns.html`` template is included on both the row and the table
+The ``_table.html`` template is included by both the row and the table pages,
-page, and displays the content of the row. The default ``_rows_and_columns.html`` template
+and a list of rows. The default ``_table.html`` template renders them as an
-`can be seen here <https://github.com/simonw/datasette/blob/master/datasette/templates/_rows_and_columns.html>`_.
+HTML template and `can be seen here <https://github.com/simonw/datasette/blob/master/datasette/templates/_table.html>`_.
 
 You can provide a custom template that applies to all of your databases and
 tables, or you can provide custom templates for specific tables using the
 template naming scheme described above.
 
-Say for example you want to output a certain column as unescaped HTML. You could
+If you want to present your data in a format other than an HTML table, you
-provide a custom ``_rows_and_columns.html`` template like this::
+can do so by looping through ``display_rows`` in your own ``_table.html``
+template. You can use ``{{ row["column_name"] }}`` to output the raw value
+of a specific column.
 
-    <table>
+If you want to output the rendered HTML version of a column, including any
-        <thead>
+links to foreign keys, you can use ``{{ row.display("column_name") }}``.
-            <tr>
+
-                {% for column in display_columns %}
+Here is an example of a custom ``_table.html`` template::
-                    <th scope="col">{{ column }}</th>
+
-                {% endfor %}
+    {% for row in display_rows %}
-            </tr>
+        <div>
-        </thead>
+            <h2>{{ row["title"] }}</h2>
-        <tbody>
+            <p>{{ row["description"] }}<lp>
-        {% for row in display_rows %}
+            <p>Category: {{ row.display("category_id") }}</p>
-            <tr>
+        </div>
-                {% for cell in row %}
+    {% endfor %}
-                    <td>
-                        {% if cell.column == 'description' %}
-                            {{ cell.value|safe }}
-                        {% else %}
-                            {{ cell.value }}
-                        {% endif %}
-                    </td>
-                {% endfor %}
-            </tr>
-        {% endfor %}
-        </tbody>
-    </table>

docs/datasette-publish-cloudrun-help.txt

@@ -3,22 +3,26 @@ $ datasette publish cloudrun --help
 Usage: datasette publish cloudrun [OPTIONS] [FILES]...
 
 Options:
-  -m, --metadata FILENAME   Path to JSON file containing metadata to publish
+  -m, --metadata FILENAME         Path to JSON file containing metadata to publish
-  --extra-options TEXT      Extra options to pass to datasette serve
+  --extra-options TEXT            Extra options to pass to datasette serve
-  --branch TEXT             Install datasette from a GitHub branch e.g. master
+  --branch TEXT                   Install datasette from a GitHub branch e.g. master
-  --template-dir DIRECTORY  Path to directory containing custom templates
+  --template-dir DIRECTORY        Path to directory containing custom templates
-  --plugins-dir DIRECTORY   Path to directory containing custom plugins
+  --plugins-dir DIRECTORY         Path to directory containing custom plugins
-  --static STATIC MOUNT     mountpoint:path-to-directory for serving static files
+  --static STATIC MOUNT           mountpoint:path-to-directory for serving static files
-  --install TEXT            Additional packages (e.g. plugins) to install
+  --install TEXT                  Additional packages (e.g. plugins) to install
-  --version-note TEXT       Additional note to show on /-/versions
+  --plugin-secret <TEXT TEXT TEXT>...
-  --title TEXT              Title for metadata
+                                  Secrets to pass to plugins, e.g. --plugin-secret
-  --license TEXT            License label for metadata
+                                  datasette-auth-github client_id xxx
-  --license_url TEXT        License URL for metadata
+  --version-note TEXT             Additional note to show on /-/versions
-  --source TEXT             Source label for metadata
+  --title TEXT                    Title for metadata
-  --source_url TEXT         Source URL for metadata
+  --license TEXT                  License label for metadata
-  --about TEXT              About label for metadata
+  --license_url TEXT              License URL for metadata
-  --about_url TEXT          About URL for metadata
+  --source TEXT                   Source label for metadata
-  -n, --name TEXT           Application name to use when building
+  --source_url TEXT               Source URL for metadata
-  --service TEXT            Cloud Run service to deploy (or over-write)
+  --about TEXT                    About label for metadata
-  --spatialite              Enable SpatialLite extension
+  --about_url TEXT                About URL for metadata
-  --help                    Show this message and exit.
+  -n, --name TEXT                 Application name to use when building
+  --service TEXT                  Cloud Run service to deploy (or over-write)
+  --spatialite                    Enable SpatialLite extension
+  --show-files                    Output the generated Dockerfile and metadata.json
+  --help                          Show this message and exit.

docs/datasette-publish-heroku-help.txt

@@ -3,20 +3,23 @@ $ datasette publish heroku --help
 Usage: datasette publish heroku [OPTIONS] [FILES]...
 
 Options:
-  -m, --metadata FILENAME   Path to JSON file containing metadata to publish
+  -m, --metadata FILENAME         Path to JSON file containing metadata to publish
-  --extra-options TEXT      Extra options to pass to datasette serve
+  --extra-options TEXT            Extra options to pass to datasette serve
-  --branch TEXT             Install datasette from a GitHub branch e.g. master
+  --branch TEXT                   Install datasette from a GitHub branch e.g. master
-  --template-dir DIRECTORY  Path to directory containing custom templates
+  --template-dir DIRECTORY        Path to directory containing custom templates
-  --plugins-dir DIRECTORY   Path to directory containing custom plugins
+  --plugins-dir DIRECTORY         Path to directory containing custom plugins
-  --static STATIC MOUNT     mountpoint:path-to-directory for serving static files
+  --static STATIC MOUNT           mountpoint:path-to-directory for serving static files
-  --install TEXT            Additional packages (e.g. plugins) to install
+  --install TEXT                  Additional packages (e.g. plugins) to install
-  --version-note TEXT       Additional note to show on /-/versions
+  --plugin-secret <TEXT TEXT TEXT>...
-  --title TEXT              Title for metadata
+                                  Secrets to pass to plugins, e.g. --plugin-secret
-  --license TEXT            License label for metadata
+                                  datasette-auth-github client_id xxx
-  --license_url TEXT        License URL for metadata
+  --version-note TEXT             Additional note to show on /-/versions
-  --source TEXT             Source label for metadata
+  --title TEXT                    Title for metadata
-  --source_url TEXT         Source URL for metadata
+  --license TEXT                  License label for metadata
-  --about TEXT              About label for metadata
+  --license_url TEXT              License URL for metadata
-  --about_url TEXT          About URL for metadata
+  --source TEXT                   Source label for metadata
-  -n, --name TEXT           Application name to use when deploying
+  --source_url TEXT               Source URL for metadata
-  --help                    Show this message and exit.
+  --about TEXT                    About label for metadata
+  --about_url TEXT                About URL for metadata
+  -n, --name TEXT                 Application name to use when deploying
+  --help                          Show this message and exit.

docs/datasette-publish-nowv1-help.txt

@@ -3,24 +3,28 @@ $ datasette publish nowv1 --help
 Usage: datasette publish nowv1 [OPTIONS] [FILES]...
 
 Options:
-  -m, --metadata FILENAME   Path to JSON file containing metadata to publish
+  -m, --metadata FILENAME         Path to JSON file containing metadata to publish
-  --extra-options TEXT      Extra options to pass to datasette serve
+  --extra-options TEXT            Extra options to pass to datasette serve
-  --branch TEXT             Install datasette from a GitHub branch e.g. master
+  --branch TEXT                   Install datasette from a GitHub branch e.g. master
-  --template-dir DIRECTORY  Path to directory containing custom templates
+  --template-dir DIRECTORY        Path to directory containing custom templates
-  --plugins-dir DIRECTORY   Path to directory containing custom plugins
+  --plugins-dir DIRECTORY         Path to directory containing custom plugins
-  --static STATIC MOUNT     mountpoint:path-to-directory for serving static files
+  --static STATIC MOUNT           mountpoint:path-to-directory for serving static files
-  --install TEXT            Additional packages (e.g. plugins) to install
+  --install TEXT                  Additional packages (e.g. plugins) to install
-  --version-note TEXT       Additional note to show on /-/versions
+  --plugin-secret <TEXT TEXT TEXT>...
-  --title TEXT              Title for metadata
+                                  Secrets to pass to plugins, e.g. --plugin-secret
-  --license TEXT            License label for metadata
+                                  datasette-auth-github client_id xxx
-  --license_url TEXT        License URL for metadata
+  --version-note TEXT             Additional note to show on /-/versions
-  --source TEXT             Source label for metadata
+  --title TEXT                    Title for metadata
-  --source_url TEXT         Source URL for metadata
+  --license TEXT                  License label for metadata
-  --about TEXT              About label for metadata
+  --license_url TEXT              License URL for metadata
-  --about_url TEXT          About URL for metadata
+  --source TEXT                   Source label for metadata
-  -n, --name TEXT           Application name to use when deploying
+  --source_url TEXT               Source URL for metadata
-  --force                   Pass --force option to now
+  --about TEXT                    About label for metadata
-  --token TEXT              Auth token to use for deploy
+  --about_url TEXT                About URL for metadata
-  --alias TEXT              Desired alias e.g. yoursite.now.sh
+  -n, --name TEXT                 Application name to use when deploying
-  --spatialite              Enable SpatialLite extension
+  --force                         Pass --force option to now
-  --help                    Show this message and exit.
+  --token TEXT                    Auth token to use for deploy
+  --alias TEXT                    Desired alias e.g. yoursite.now.sh
+  --spatialite                    Enable SpatialLite extension
+  --show-files                    Output the generated Dockerfile and metadata.json
+  --help                          Show this message and exit.

docs/datasette-serve-help.txt

@@ -6,8 +6,11 @@ Usage: datasette serve [OPTIONS] [FILES]...
 
 Options:
   -i, --immutable PATH      Database files to open in immutable mode
-  -h, --host TEXT           host for server, defaults to 127.0.0.1
+  -h, --host TEXT           Host for server. Defaults to 127.0.0.1 which means only
-  -p, --port INTEGER        port for server, defaults to 8001
+                            connections from the local machine will be allowed. Use
+                            0.0.0.0 to listen to all IPs and allow access from other
+                            machines.
+  -p, --port INTEGER        Port for server, defaults to 8001
   --debug                   Enable debug mode - useful for development
   --reload                  Automatically reload if database or code change detected -
                             useful for development

docs/ecosystem.rst

@@ -70,6 +70,11 @@ datasette-vega
 
 `datasette-vega <https://github.com/simonw/datasette-vega>`__ exposes the powerful  `Vega <https://vega.github.io/vega/>`__ charting library, allowing you to construct line, bar and scatter charts against your data and share links to your visualizations.
 
+datasette-auth-github
+---------------------
+
+`datasette-auth-github <https://github.com/simonw/datasette-auth-github>`__ adds an authentication layer to Datasette. Users will have to sign in using their GitHub account before they can view data or interact with Datasette. You can also use it to restrict access to specific GitHub users, or to members of specified GitHub `organizations <https://help.github.com/en/articles/about-organizations>`__ or `teams <https://help.github.com/en/articles/organizing-members-into-teams>`__.
+
 datasette-json-html
 -------------------
 
@@ -80,6 +85,11 @@ datasette-jellyfish
 
 `datasette-jellyfish <https://github.com/simonw/datasette-jellyfish>`__ exposes custom SQL functions for a range of common fuzzy string matching functions, including soundex, porter stemming and levenshtein distance. It builds on top of the `Jellyfish Python library <https://jellyfish.readthedocs.io/>`__.
 
+datasette-doublemetaphone
+-------------------------
+
+`datasette-doublemetaphone <https://github.com/dracos/datasette-doublemetaphone>`__ by Matthew Somerville adds custom SQL functions for applying the Double Metaphone fuzzy "sounds like" algorithm.
+
 datasette-jq
 ------------
 
@@ -90,6 +100,11 @@ datasette-render-images
 
 `datasette-render-images <https://github.com/simonw/datasette-render-images>`__ works with SQLite tables that contain binary image data in BLOB columns. It converts any images it finds into ``data-uri`` image elements, allowing you to view them directly in the Datasette interface.
 
+datasette-render-binary
+-----------------------
+
+`datasette-render-binary <https://github.com/simonw/datasette-render-binary>`__ renders binary data in a slightly more readable fashion: it shows ASCII characters as they are, and shows all other data as monospace octets. Useful as a tool for exploring new unfamiliar databases as it makes it easier to spot if a binary column may contain a decipherable binary format.
+
 datasette-pretty-json
 ---------------------
 
@@ -99,3 +114,13 @@ datasette-sqlite-fts4
 ---------------------
 
 `datasette-sqlite-fts4 <https://github.com/simonw/datasette-sqlite-fts4>`__ provides search relevance ranking algorithms that can be used with SQLite's FTS4 search module. You can read more about it in `Exploring search relevance algorithms with SQLite <https://simonwillison.net/2019/Jan/7/exploring-search-relevance-algorithms-sqlite/>`__.
+
+datasette-bplist
+----------------
+
+`datasette-bplist <https://github.com/simonw/datasette-bplist>`__ provides tools for working with Apple's binary plist format embedded in SQLite database tables. If you use OS X you already have dozens of SQLite databases hidden away in your ``~/Library`` folder that include data in this format - this plugin allows you to view the decoded data and run SQL queries against embedded values using a ``bplist_to_json(value)`` custom SQL function.
+
+datasette-cors
+--------------
+
+`datasette-cors <https://github.com/simonw/datasette-cors>`__ allows you to configure `CORS headers <https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS>`__ for your Datasette instance. You can use this to enable JavaScript running on a whitelisted set of domains to make ``fetch()`` calls to the JSON API provided by your Datasette instance.

docs/facets.rst

@@ -129,17 +129,6 @@ The performance of facets can be greatly improved by adding indexes on the colum
     Enter ".help" for usage hints.
     sqlite> CREATE INDEX Food_Trucks_state ON Food_Trucks("state");
 
-.. _facet_by_m2m:
-
-Facet by many-to-many
----------------------
-
-Datasette can detect many-to-many SQL tables - defined as SQL tables which have foreign key relationships to two other tables.
-
-If a many-to-many table exists pointing at the table you are currently viewing, Datasette will suggest you facet the table based on that relationship.
-
-Example here: `latest.datasette.io/fixtures/roadside_attractions?_facet_m2m=attraction_characteristic <https://latest.datasette.io/fixtures/roadside_attractions?_facet_m2m=attraction_characteristic>`__
-
 .. _facet_by_json_array:
 
 Facet by JSON array

docs/full_text_search.rst

@@ -28,7 +28,28 @@ To set up full-text search for a table, you need to do two things:
 * Create a new FTS virtual table associated with your table
 * Populate that FTS table with the data that you would like to be able to run searches against
 
-To enable full-text search for a table called ``items`` that works against the ``name`` and ``description`` columns, you would run the following SQL to create a new ``items_fts`` FTS virtual table:
+Configuring FTS using sqlite-utils
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+`sqlite-utils <https://sqlite-utils.readthedocs.io/>`__ is a CLI utility and Python library for manipulating SQLite databases. You can use `it from Python code <https://sqlite-utils.readthedocs.io/en/latest/python-api.html#enabling-full-text-search>`__ to configure FTS search, or you can achieve the same goal `using the accompanying command-line tool <https://sqlite-utils.readthedocs.io/en/latest/cli.html#configuring-full-text-search>`__.
+
+Here's how to use ``sqlite-utils`` to enable full-text search for an ``items`` table across the ``name`` and ``description`` columns::
+
+    $ sqlite-utils enable-fts mydatabase.db items name description
+
+Configuring FTS using csvs-to-sqlite
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If your data starts out in CSV files, you can use Datasette's companion tool `csvs-to-sqlite <https://github.com/simonw/csvs-to-sqlite>`__ to convert that file into a SQLite database and enable full-text search on specific columns. For a file called ``items.csv`` where you want full-text search to operate against the ``name`` and ``description`` columns you would run the following::
+
+    $ csvs-to-sqlite items.csv items.db -f name -f description
+
+Configuring FTS by hand
+~~~~~~~~~~~~~~~~~~~~~~~
+
+We recommend using `sqlite-utils <https://sqlite-utils.readthedocs.io/>`__, but if you want to hand-roll a SQLite full-text search table you can do so using the following SQL.
+
+To enable full-text search for a table called ``items`` that works against the ``name`` and ``description`` columns, you would run this SQL to create a new ``items_fts`` FTS virtual table:
 
 .. code-block:: sql
 
@@ -71,8 +92,6 @@ And then populate it like this:
 
 You can use this technique to populate the full-text search index from any combination of tables and joins that makes sense for your project.
 
-The `sqlite-utils tool <https://sqlite-utils.readthedocs.io/en/latest/cli.html#configuring-full-text-search>`__ provides a command-line mechanism that can be used to implement the above steps.
-
 .. _full_text_search_table_or_view:
 
 Configuring full-text search for a table or view
@@ -103,13 +122,6 @@ Here is an example which enables full-text search for a ``display_ads`` view whi
       }
     }
 
-Setting up full-text search using csvs-to-sqlite
-------------------------------------------------
-
-If your data starts out in CSV files, you can use Datasette's companion tool `csvs-to-sqlite <https://github.com/simonw/csvs-to-sqlite>`_ to convert that file into a SQLite database and enable full-text search on specific columns. For a file called ``items.csv`` where you want full-text search to operate against the ``name`` and ``description`` columns you would run the following::
-
-    csvs-to-sqlite items.csv items.db -f name -f description
-
 The table view API
 ------------------
 

docs/getting_started.rst

@@ -25,7 +25,7 @@ Glitch allows you to "remix" any project to create your own copy and start editi
 .. image:: https://cdn.glitch.com/2703baf2-b643-4da7-ab91-7ee2a2d00b5b%2Fremix-button.svg
    :target: https://glitch.com/edit/#!/remix/datasette-csvs
 
-Find a CSV file and drag it onto the Glitch file explorer panel - ``datasette-csvs`` will automatically convert it to a SQLite database (using `csvs-to-sqlite <https://github.com/simonw/csvs-to-sqlite>`__) and allow you to start exploring it using Datasette.
+Find a CSV file and drag it onto the Glitch file explorer panel - ``datasette-csvs`` will automatically convert it to a SQLite database (using `sqlite-utils <https://github.com/simonw/sqlite-utils>`__) and allow you to start exploring it using Datasette.
 
 If your CSV file has a ``latitude`` and ``longitude`` column you can visualize it on a map by uncommenting the ``datasette-cluster-map`` line in the ``requirements.txt`` file using the Glitch file editor.
 

docs/introspection.rst

@@ -90,6 +90,8 @@ Shows the :ref:`config` options for this instance of Datasette. `Config example
         "sql_time_limit_ms": 1000
     }
 
+.. _JsonDataView_databases:
+
 /-/databases
 ------------
 
@@ -105,3 +107,26 @@ Shows currently attached databases. `Databases example <https://latest.datasette
             "size": 225280
         }
     ]
+
+.. _JsonDataView_threads:
+
+/-/threads
+----------
+
+Shows details of threads. `Threads example <https://latest.datasette.io/-/threads>`_::
+
+    {
+        "num_threads": 2,
+        "threads": [
+            {
+                "daemon": false,
+                "ident": 4759197120,
+                "name": "MainThread"
+            },
+            {
+                "daemon": true,
+                "ident": 123145319682048,
+                "name": "Thread-1"
+            },
+        ]
+    }

docs/json_api.rst

@@ -228,6 +228,9 @@ You can filter the data returned by the table based on column values using a que
 
     ``?column__in=["value","value,with,commas"]``
 
+``?column__notin=value1,value2,value3``
+    Rows where column does not match any of the provided values. The inverse of ``__in=``. Also supports JSON arrays.
+
 ``?column__arraycontains=value``
     Works against columns that contain JSON arrays - matches if any of the values in that array match.
 
@@ -318,15 +321,6 @@ Special table arguments
 
     Here's `an example <https://latest.datasette.io/fixtures/roadside_attractions?_through={%22table%22:%22roadside_attraction_characteristics%22,%22column%22:%22characteristic_id%22,%22value%22:%221%22}>`__.
 
-
-``?_group_count=COLUMN``
-    Executes a SQL query that returns a count of the number of rows matching
-    each unique value in that column, with the most common ordered first.
-
-``?_group_count=COLUMN1&_group_count=column2``
-    You can pass multiple ``_group_count`` columns to return counts against
-    unique combinations of those columns.
-
 ``?_next=TOKEN``
     Pagination by continuation token - pass the token that was returned in the
     ``"next"`` property by the previous page.

docs/metadata.rst

@@ -62,7 +62,7 @@ Each of the top-level metadata fields can be used at the database and table leve
 Source, license and about
 -------------------------
 
-The three visible metadata fields you can apply to everything, specific databases or specific tables are source, license and about. All three are optionaly.
+The three visible metadata fields you can apply to everything, specific databases or specific tables are source, license and about. All three are optional.
 
 **source** and **source_url** should be used to indicate where the underlying data came from.
 

docs/performance.rst

@@ -7,6 +7,8 @@ Datasette runs on top of SQLite, and SQLite has excellent performance.  For smal
 
 That said, there are a number of tricks you can use to improve Datasette's performance.
 
+.. _performance_immutable_mode:
+
 Immutable mode
 --------------
 
@@ -37,7 +39,7 @@ Then later you can start Datasette against the ``counts.json`` file and use it t
 
     datasette -i data.db --inspect-file=counts.json
 
-You need to use the ``-i`` immutable mode agaist the databse file here or the counts from the JSON file will be ignored.
+You need to use the ``-i`` immutable mode against the databse file here or the counts from the JSON file will be ignored.
 
 You will rarely need to use this optimization in every-day use, but several of the ``datasette publish`` commands described in :ref:`publishing` use this optimization for better performance when deploying a database file to a hosting provider.
 

docs/plugins.rst

@@ -219,6 +219,48 @@ Here is an example of some plugin configuration for a specific table::
 
 This tells the ``datasette-cluster-map`` column which latitude and longitude columns should be used for a table called ``Street_Tree_List`` inside a database file called ``sf-trees.db``.
 
+.. _plugins_configuration_secret:
+
+Secret configuration values
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Any values embedded in ``metadata.json`` will be visible to anyone who views the ``/-/metadata`` page of your Datasette instance. Some plugins may need configuration that should stay secret - API keys for example. There are two ways in which you can store secret configuration values.
+
+**As environment variables**. If your secret lives in an environment variable that is available to the Datasette process, you can indicate that the configuration value should be read from that environment variable like so::
+
+    {
+        "plugins": {
+            "datasette-auth-github": {
+                "client_secret": {
+                    "$env": "GITHUB_CLIENT_SECRET"
+                }
+            }
+        }
+    }
+
+**As values in separate files**. Your secrets can also live in files on disk. To specify a secret should be read from a file, provide the full file path like this::
+
+    {
+        "plugins": {
+            "datasette-auth-github": {
+                "client_secret": {
+                    "$file": "/secrets/client-secret"
+                }
+            }
+        }
+    }
+
+If you are publishing your data using the :ref:`datasette publish <cli_publish>` family of commands, you can use the ``--plugin-secret`` option to set these secrets at publish time. For example, using Heroku you might run the following command::
+
+    $ datasette publish heroku my_database.db \
+        --name my-heroku-app-demo \
+        --install=datasette-auth-github \
+        --plugin-secret datasette-auth-github client_id your_client_id \
+        --plugin-secret datasette-auth-github client_secret your_client_secret
+
+Writing plugins that accept configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
 When you are writing plugins, you can access plugin configuration like this using the ``datasette.plugin_config()`` method. If you know you need plugin configuration for a specific table, you can access it like this::
 
     plugin_config = datasette.plugin_config(
@@ -400,7 +442,7 @@ you have one:
     @hookimpl
     def extra_js_urls():
         return [
-            '/-/static-plugins/your_plugin/app.js'
+            '/-/static-plugins/your-plugin/app.js'
         ]
 
 .. _plugin_hook_publish_subcommand:
@@ -529,6 +571,8 @@ If the value matches that pattern, the plugin returns an HTML link element:
 extra_body_script(template, database, table, view_name, datasette)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
+Extra JavaScript to be added to a ``<script>`` block at the end of the ``<body>`` element on the page.
+
 ``template`` - string
     The template that is being rendered, e.g. ``database.html``
 
@@ -544,14 +588,74 @@ extra_body_script(template, database, table, view_name, datasette)
 ``datasette`` - Datasette instance
     You can use this to access plugin configuration options via ``datasette.plugin_config(your_plugin_name)``
 
-Extra JavaScript to be added to a ``<script>`` block at the end of the ``<body>`` element on the page.
-
 The ``template``, ``database`` and ``table`` options can be used to return different code depending on which template is being rendered and which database or table are being processed.
 
 The ``datasette`` instance is provided primarily so that you can consult any plugin configuration options that may have been set, using the ``datasette.plugin_config(plugin_name)`` method documented above.
 
 The string that you return from this function will be treated as "safe" for inclusion in a ``<script>`` block directly in the page, so it is up to you to apply any necessary escaping.
 
+
+.. _plugin_hook_extra_template_vars:
+
+extra_template_vars(template, database, table, view_name, request, datasette)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Extra template variables that should be made available in the rendered template context.
+
+``template`` - string
+    The template that is being rendered, e.g. ``database.html``
+
+``database`` - string or None
+    The name of the database, or ``None`` if the page does not correspond to a database (e.g. the root page)
+
+``table`` - string or None
+    The name of the table, or ``None`` if the page does not correct to a table
+
+``view_name`` - string
+    The name of the view being displayed. (`database`, `table`, and `row` are the most important ones.)
+
+``request`` - object
+    The current HTTP request object. ``request.scope`` provides access to the ASGI scope.
+
+``datasette`` - Datasette instance
+    You can use this to access plugin configuration options via ``datasette.plugin_config(your_plugin_name)``
+
+This hook can return one of three different types:
+
+Dictionary
+    If you return a dictionary its keys and values will be merged into the template context.
+
+Function that returns a dictionary
+    If you return a function it will be executed. If it returns a dictionary those values will will be merged into the template context.
+
+Function that returns an awaitable function that returns a dictionary
+    You can also return a function which returns an awaitable function which returns a dictionary. This means you can execute additional SQL queries using ``datasette.execute()``.
+
+Here's an example plugin that returns an authentication object from the ASGI scope:
+
+.. code-block:: python
+
+    @hookimpl
+    def extra_template_vars(request):
+        return {
+            "auth": request.scope.get("auth")
+        }
+
+And here's an example which returns the current version of SQLite:
+
+.. code-block:: python
+
+    @hookimpl
+    def extra_template_vars(datasette):
+        async def inner():
+            first_db = list(datasette.databases.keys())[0]
+            return {
+                "sqlite_version": (
+                    await datasette.execute(first_db, "select sqlite_version()")
+                ).rows[0][0]
+            }
+        return inner
+
 .. _plugin_register_output_renderer:
 
 register_output_renderer(datasette)
@@ -564,12 +668,12 @@ Allows the plugin to register a new output renderer, to output data in a custom
 
 .. code-block:: python
 
-        @hookimpl
+    @hookimpl
-        def register_output_renderer(datasette):
+    def register_output_renderer(datasette):
-            return {
+        return {
-                'extension': 'test',
+            'extension': 'test',
-                'callback': render_test
+            'callback': render_test
-            }
+        }
 
 This will register `render_test` to be called when paths with the extension `.test` (for example `/database.test`, `/database/table.test`, or `/database/table/row.test`) are requested. When a request is received, the callback function is called with three positional arguments:
 
@@ -597,10 +701,10 @@ A simple example of an output renderer callback function:
 
 .. code-block:: python
 
-        def render_test(args, data, view_name):
+    def render_test(args, data, view_name):
-            return {
+        return {
-                'body': 'Hello World'
+            'body': 'Hello World'
-            }
+        }
 
 .. _plugin_register_facet_classes:
 
@@ -617,9 +721,9 @@ Each Facet subclass implements a new type of facet operation. The class should l
         # This key must be unique across all facet classes:
         type = "special"
 
-        async def suggest(self, sql, params, filtered_table_rows_count):
+        async def suggest(self):
+            # Use self.sql and self.params to suggest some facets
             suggested_facets = []
-            # Perform calculations to suggest facets
             suggested_facets.append({
                 "name": column, # Or other unique name
                 # Construct the URL that will enable this facet:
@@ -631,8 +735,9 @@ Each Facet subclass implements a new type of facet operation. The class should l
             })
             return suggested_facets
 
-        async def facet_results(self, sql, params):
+        async def facet_results(self):
-            # This should execute the facet operation and return results
+            # This should execute the facet operation and return results, again
+            # using self.sql and self.params as the starting point
             facet_results = {}
             facets_timed_out = []
             # Do some calculations here...
@@ -657,7 +762,7 @@ Each Facet subclass implements a new type of facet operation. The class should l
 
             return facet_results, facets_timed_out
 
-See ``datasette/facets.py`` for examples of how these classes can work.
+See `datasette/facets.py <https://github.com/simonw/datasette/blob/master/datasette/facets.py>`__ for examples of how these classes can work.
 
 The plugin hook can then be used to register the new facet class like this:
 
@@ -666,3 +771,44 @@ The plugin hook can then be used to register the new facet class like this:
     @hookimpl
     def register_facet_classes():
         return [SpecialFacet]
+
+
+.. _plugin_asgi_wrapper:
+
+asgi_wrapper(datasette)
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Return an `ASGI <https://asgi.readthedocs.io/>`__ middleware wrapper function that will be applied to the Datasette ASGI application.
+
+This is a very powerful hook. You can use it to manipulate the entire Datasette response, or even to configure new URL routes that will be handled by your own custom code.
+
+You can write your ASGI code directly against the low-level specification, or you can use the middleware utilites provided by an ASGI framework such as `Starlette <https://www.starlette.io/middleware/>`__.
+
+This example plugin adds a ``x-databases`` HTTP header listing the currently attached databases:
+
+.. code-block:: python
+
+    from datasette import hookimpl
+    from functools import wraps
+
+
+    @hookimpl
+    def asgi_wrapper(datasette):
+        def wrap_with_databases_header(app):
+            @wraps(app)
+            async def add_x_databases_header(scope, recieve, send):
+                async def wrapped_send(event):
+                    if event["type"] == "http.response.start":
+                        original_headers = event.get("headers") or []
+                        event = {
+                            "type": event["type"],
+                            "status": event["status"],
+                            "headers": original_headers + [
+                                [b"x-databases",
+                                ", ".join(datasette.databases.keys()).encode("utf-8")]
+                            ],
+                        }
+                    await send(event)
+                await app(scope, recieve, wrapped_send)
+            return add_x_databases_header
+        return wrap_with_databases_header

docs/publish.rst

@@ -6,6 +6,8 @@
 
 Datasette includes tools for publishing and deploying your data to the internet. The ``datasette publish`` command will deploy a new Datasette instance containing your databases directly to a Heroku, Google Cloud or Zeit Now hosting account. You can also use ``datasette package`` to create a Docker image that bundles your databases together with the datasette application that is used to serve them.
 
+.. _cli_publish:
+
 datasette publish
 =================
 
@@ -41,14 +43,16 @@ You will first need to install and configure the Google Cloud CLI tools by follo
 
 You can then publish a database to Google Cloud Run using the following command::
 
-    datasette publish cloudrun mydatabase.db
+    datasette publish cloudrun mydatabase.db --service=my-database
+
+A Cloud Run **service** is a single hosted application. The service name you specify will be used as part of the Cloud Run URL. If you deploy to a service name that you have used in the past your new deployment will replace the previous one.
+
+If you omit the ``--service`` option you will be asked to pick a service name interactively during the deploy.
 
 You may need to interact with prompts from the tool. Once it has finished it will output a URL like this one::
 
-    Service [datasette] revision [datasette-00001] has been deployed
+    Service [my-service] revision [my-service-00001] has been deployed
-    and is serving traffic at https://datasette-j7hipcg4aq-uc.a.run.app
+    and is serving traffic at https://my-service-j7hipcg4aq-uc.a.run.app
-
-During the deployment the tool will prompt you for the name of your service. You can reuse an existing name to replace your previous deployment with your new version, or pick a new name to deploy to a new URL.
 
 .. literalinclude:: datasette-publish-cloudrun-help.txt
 
@@ -79,6 +83,8 @@ You can also use custom domains, if you `first register them with Zeit Now <http
 
 .. literalinclude:: datasette-publish-nowv1-help.txt
 
+.. _publish_custom_metadata_and_plugins:
+
 Custom metadata and plugins
 ---------------------------
 
@@ -86,19 +92,26 @@ Custom metadata and plugins
 
 You can define your own :ref:`metadata` and deploy that with your instance like so::
 
-    datasette publish nowv1 mydatabase.db -m metadata.json
+    datasette publish cloudrun --service=my-service mydatabase.db -m metadata.json
 
 If you just want to set the title, license or source information you can do that directly using extra options to ``datasette publish``::
 
-    datasette publish nowv1 mydatabase.db \
+    datasette publish cloudrun mydatabase.db --service=my-service \
         --title="Title of my database" \
         --source="Where the data originated" \
         --source_url="http://www.example.com/"
 
 You can also specify plugins you would like to install. For example, if you want to include the `datasette-vega <https://github.com/simonw/datasette-vega>`_ visualization plugin you can use the following::
 
-    datasette publish nowv1 mydatabase.db --install=datasette-vega
+    datasette publish cloudrun mydatabase.db --service=my-service --install=datasette-vega
 
+If a plugin has any :ref:`plugins_configuration_secret` you can use the ``--plugin-secret`` option to set those secrets at publish time. For example, using Heroku with `datasette-auth-github <https://github.com/simonw/datasette-auth-github>`__ you might run the following command::
+
+    $ datasette publish heroku my_database.db \
+        --name my-heroku-app-demo \
+        --install=datasette-auth-github \
+        --plugin-secret datasette-auth-github client_id your_client_id \
+        --plugin-secret datasette-auth-github client_secret your_client_secret
 
 datasette package
 =================

pytest.ini

@@ -4,7 +4,5 @@ filterwarnings=
     ignore:Using or importing the ABCs::jinja2
     # https://bugs.launchpad.net/beautifulsoup/+bug/1778909
     ignore:Using or importing the ABCs::bs4.element
-    # Sanic verify_ssl=True
-    ignore:verify_ssl is deprecated::sanic
     # Python 3.7 PendingDeprecationWarning: Task.current_task()
     ignore:.*current_task.*:PendingDeprecationWarning

setup.py

@@ -25,7 +25,7 @@ def get_version():
 # Only install black on Python 3.6 or higher
 maybe_black = []
 if sys.version_info > (3, 6):
-    maybe_black = ["black"]
+    maybe_black = ["black~=19.10b0"]
 
 setup(
     name="datasette",
@@ -37,17 +37,18 @@ setup(
     author="Simon Willison",
     license="Apache License, Version 2.0",
     url="https://github.com/simonw/datasette",
-    packages=find_packages(exclude='tests'),
+    packages=find_packages(exclude="tests"),
     package_data={"datasette": ["templates/*.html"]},
     include_package_data=True,
     install_requires=[
-        "click>=6.7",
+        "click~=7.0",
-        "click-default-group==1.2",
+        "click-default-group~=1.2.1",
-        "Sanic==0.7.0",
+        "Jinja2~=2.10.1",
-        "Jinja2==2.10.1",
+        "hupper~=1.0",
-        "hupper==1.0",
+        "pint~=0.8.1",
-        "pint==0.8.1",
+        "pluggy~=0.12.0",
-        "pluggy>=0.12.0",
+        "uvicorn~=0.8.4",
+        "aiofiles~=0.4.0",
     ],
     entry_points="""
         [console_scripts]
@@ -55,13 +56,15 @@ setup(
     """,
     setup_requires=["pytest-runner"],
     extras_require={
+        "docs": ["sphinx_rtd_theme", "sphinx-autobuild"],
         "test": [
-            "pytest==4.6.1",
+            "pytest~=5.0.0",
-            "pytest-asyncio==0.10.0",
+            "pytest-asyncio~=0.10.0",
-            "aiohttp==3.5.3",
+            "aiohttp~=3.5.3",
-            "beautifulsoup4==4.6.1",
+            "beautifulsoup4~=4.6.1",
+            "asgiref~=3.1.2",
         ]
-        + maybe_black
+        + maybe_black,
     },
     tests_require=["datasette[test]"],
     classifiers=[

tests/conftest.py

@@ -1,3 +1,7 @@
+import os
+import pytest
+
+
 def pytest_configure(config):
     import sys
 
@@ -22,3 +26,14 @@ def move_to_front(items, test_name):
     test = [fn for fn in items if fn.name == test_name]
     if test:
         items.insert(0, items.pop(items.index(test[0])))
+
+
+@pytest.fixture
+def restore_working_directory(tmpdir, request):
+    previous_cwd = os.getcwd()
+    tmpdir.chdir()
+
+    def return_to_previous():
+        os.chdir(previous_cwd)
+
+    request.addfinalizer(return_to_previous)

tests/fixtures.py

@@ -1,5 +1,7 @@
 from datasette.app import Datasette
 from datasette.utils import sqlite3
+from asgiref.testing import ApplicationCommunicator
+from asgiref.sync import async_to_sync
 import itertools
 import json
 import os
@@ -10,16 +12,88 @@ import sys
 import string
 import tempfile
 import time
+from urllib.parse import unquote, quote
+
+
+# This temp file is used by one of the plugin config tests
+TEMP_PLUGIN_SECRET_FILE = os.path.join(tempfile.gettempdir(), "plugin-secret")
+
+
+class TestResponse:
+    def __init__(self, status, headers, body):
+        self.status = status
+        self.headers = headers
+        self.body = body
+
+    @property
+    def json(self):
+        return json.loads(self.text)
+
+    @property
+    def text(self):
+        return self.body.decode("utf8")
 
 
 class TestClient:
-    def __init__(self, sanic_test_client):
+    max_redirects = 5
-        self.sanic_test_client = sanic_test_client
 
-    def get(self, path, allow_redirects=True):
+    def __init__(self, asgi_app):
-        return self.sanic_test_client.get(
+        self.asgi_app = asgi_app
-            path, allow_redirects=allow_redirects, gather_request=False
+
+    @async_to_sync
+    async def get(self, path, allow_redirects=True, redirect_count=0, method="GET"):
+        return await self._get(path, allow_redirects, redirect_count, method)
+
+    async def _get(self, path, allow_redirects=True, redirect_count=0, method="GET"):
+        query_string = b""
+        if "?" in path:
+            path, _, query_string = path.partition("?")
+            query_string = query_string.encode("utf8")
+        if "%" in path:
+            raw_path = path.encode("latin-1")
+        else:
+            raw_path = quote(path, safe="/:,").encode("latin-1")
+        scope = {
+            "type": "http",
+            "http_version": "1.0",
+            "method": method,
+            "path": unquote(path),
+            "raw_path": raw_path,
+            "query_string": query_string,
+            "headers": [[b"host", b"localhost"]],
+        }
+        instance = ApplicationCommunicator(self.asgi_app, scope)
+        await instance.send_input({"type": "http.request"})
+        # First message back should be response.start with headers and status
+        messages = []
+        start = await instance.receive_output(2)
+        messages.append(start)
+        assert start["type"] == "http.response.start"
+        headers = dict(
+            [(k.decode("utf8"), v.decode("utf8")) for k, v in start["headers"]]
         )
+        status = start["status"]
+        # Now loop until we run out of response.body
+        body = b""
+        while True:
+            message = await instance.receive_output(2)
+            messages.append(message)
+            assert message["type"] == "http.response.body"
+            body += message["body"]
+            if not message.get("more_body"):
+                break
+        response = TestResponse(status, headers, body)
+        if allow_redirects and response.status in (301, 302):
+            assert (
+                redirect_count < self.max_redirects
+            ), "Redirected {} times, max_redirects={}".format(
+                redirect_count, self.max_redirects
+            )
+            location = response.headers["Location"]
+            return await self._get(
+                location, allow_redirects=True, redirect_count=redirect_count + 1
+            )
+        return response
 
 
 def make_app_client(
@@ -32,6 +106,8 @@ def make_app_client(
     is_immutable=False,
     extra_databases=None,
     inspect_data=None,
+    static_mounts=None,
+    template_dir=None,
 ):
     with tempfile.TemporaryDirectory() as tmpdir:
         filepath = os.path.join(tmpdir, filename)
@@ -73,9 +149,11 @@ def make_app_client(
             plugins_dir=plugins_dir,
             config=config,
             inspect_data=inspect_data,
+            static_mounts=static_mounts,
+            template_dir=template_dir,
         )
         ds.sqlite_functions.append(("sleep", 1, lambda n: time.sleep(float(n))))
-        client = TestClient(ds.app().test_client)
+        client = TestClient(ds.app())
         client.ds = ds
         yield client
 
@@ -88,7 +166,7 @@ def app_client():
 @pytest.fixture(scope="session")
 def app_client_no_files():
     ds = Datasette([])
-    client = TestClient(ds.app().test_client)
+    client = TestClient(ds.app())
     client.ds = ds
     yield client
 
@@ -96,14 +174,21 @@ def app_client_no_files():
 @pytest.fixture(scope="session")
 def app_client_two_attached_databases():
     yield from make_app_client(
-        extra_databases={"extra_database.db": EXTRA_DATABASE_SQL}
+        extra_databases={"extra database.db": EXTRA_DATABASE_SQL}
+    )
+
+
+@pytest.fixture(scope="session")
+def app_client_conflicting_database_names():
+    yield from make_app_client(
+        extra_databases={"foo.db": EXTRA_DATABASE_SQL, "foo-bar.db": EXTRA_DATABASE_SQL}
     )
 
 
 @pytest.fixture(scope="session")
 def app_client_two_attached_databases_one_immutable():
     yield from make_app_client(
-        is_immutable=True, extra_databases={"extra_database.db": EXTRA_DATABASE_SQL}
+        is_immutable=True, extra_databases={"extra database.db": EXTRA_DATABASE_SQL}
     )
 
 
@@ -174,7 +259,11 @@ METADATA = {
     "source_url": "https://github.com/simonw/datasette/blob/master/tests/fixtures.py",
     "about": "About Datasette",
     "about_url": "https://github.com/simonw/datasette",
-    "plugins": {"name-of-plugin": {"depth": "root"}},
+    "plugins": {
+        "name-of-plugin": {"depth": "root"},
+        "env-plugin": {"foo": {"$env": "FOO_ENV"}},
+        "file-plugin": {"foo": {"$file": TEMP_PLUGIN_SECRET_FILE}},
+    },
     "databases": {
         "fixtures": {
             "description": "Test tables description",
@@ -211,6 +300,7 @@ METADATA = {
                 },
             },
             "queries": {
+                "𝐜𝐢𝐭𝐢𝐞𝐬": "select id, name from facet_cities order by id limit 1;",
                 "pragma_cache_size": "PRAGMA cache_size;",
                 "neighborhood_search": {
                     "sql": """
@@ -296,10 +386,21 @@ def render_cell(value, column, table, database, datasette):
             table=table,
         )
     })
+
+
+@hookimpl
+def extra_template_vars(template, database, table, view_name, request, datasette):
+    return {
+        "extra_template_vars": json.dumps({
+            "template": template,
+            "scope_path": request.scope["path"]
+        }, default=lambda b: b.decode("utf8"))
+    }
 """
 
 PLUGIN2 = """
 from datasette import hookimpl
+from functools import wraps
 import jinja2
 import json
 
@@ -341,6 +442,41 @@ def render_cell(value, database):
             label=jinja2.escape(data["label"] or "") or "&nbsp;"
         )
     )
+
+
+@hookimpl
+def extra_template_vars(template, database, table, view_name, request, datasette):
+    async def inner():
+        return {
+            "extra_template_vars_from_awaitable": json.dumps({
+                "template": template,
+                "scope_path": request.scope["path"],
+                "awaitable": True,
+            }, default=lambda b: b.decode("utf8"))
+        }
+    return inner
+
+
+@hookimpl
+def asgi_wrapper(datasette):
+    def wrap_with_databases_header(app):
+        @wraps(app)
+        async def add_x_databases_header(scope, recieve, send):
+            async def wrapped_send(event):
+                if event["type"] == "http.response.start":
+                    original_headers = event.get("headers") or []
+                    event = {
+                        "type": event["type"],
+                        "status": event["status"],
+                        "headers": original_headers + [
+                            [b"x-databases",
+                            ", ".join(datasette.databases.keys()).encode("utf-8")]
+                        ],
+                    }
+                await send(event)
+            await app(scope, recieve, wrapped_send)
+        return add_x_databases_header
+    return wrap_with_databases_header
 """
 
 TABLES = (
@@ -378,6 +514,7 @@ CREATE TABLE compound_three_primary_keys (
   content text,
   PRIMARY KEY (pk1, pk2, pk3)
 );
+CREATE INDEX idx_compound_three_primary_keys_content ON compound_three_primary_keys(content);
 
 CREATE TABLE foreign_key_references (
   pk varchar(30) primary key,
@@ -525,26 +662,27 @@ CREATE TABLE facetable (
     city_id integer,
     neighborhood text,
     tags text,
+    complex_array text,
     FOREIGN KEY ("city_id") REFERENCES [facet_cities](id)
 );
 INSERT INTO facetable
-    (created, planet_int, on_earth, state, city_id, neighborhood, tags)
+    (created, planet_int, on_earth, state, city_id, neighborhood, tags, complex_array)
 VALUES
-    ("2019-01-14 08:00:00", 1, 1, 'CA', 1, 'Mission', '["tag1", "tag2"]'),
+    ("2019-01-14 08:00:00", 1, 1, 'CA', 1, 'Mission', '["tag1", "tag2"]', '[{"foo": "bar"}]'),
-    ("2019-01-14 08:00:00", 1, 1, 'CA', 1, 'Dogpatch', '["tag1", "tag3"]'),
+    ("2019-01-14 08:00:00", 1, 1, 'CA', 1, 'Dogpatch', '["tag1", "tag3"]', '[]'),
-    ("2019-01-14 08:00:00", 1, 1, 'CA', 1, 'SOMA', '[]'),
+    ("2019-01-14 08:00:00", 1, 1, 'CA', 1, 'SOMA', '[]', '[]'),
-    ("2019-01-14 08:00:00", 1, 1, 'CA', 1, 'Tenderloin', '[]'),
+    ("2019-01-14 08:00:00", 1, 1, 'CA', 1, 'Tenderloin', '[]', '[]'),
-    ("2019-01-15 08:00:00", 1, 1, 'CA', 1, 'Bernal Heights', '[]'),
+    ("2019-01-15 08:00:00", 1, 1, 'CA', 1, 'Bernal Heights', '[]', '[]'),
-    ("2019-01-15 08:00:00", 1, 1, 'CA', 1, 'Hayes Valley', '[]'),
+    ("2019-01-15 08:00:00", 1, 1, 'CA', 1, 'Hayes Valley', '[]', '[]'),
-    ("2019-01-15 08:00:00", 1, 1, 'CA', 2, 'Hollywood', '[]'),
+    ("2019-01-15 08:00:00", 1, 1, 'CA', 2, 'Hollywood', '[]', '[]'),
-    ("2019-01-15 08:00:00", 1, 1, 'CA', 2, 'Downtown', '[]'),
+    ("2019-01-15 08:00:00", 1, 1, 'CA', 2, 'Downtown', '[]', '[]'),
-    ("2019-01-16 08:00:00", 1, 1, 'CA', 2, 'Los Feliz', '[]'),
+    ("2019-01-16 08:00:00", 1, 1, 'CA', 2, 'Los Feliz', '[]', '[]'),
-    ("2019-01-16 08:00:00", 1, 1, 'CA', 2, 'Koreatown', '[]'),
+    ("2019-01-16 08:00:00", 1, 1, 'CA', 2, 'Koreatown', '[]', '[]'),
-    ("2019-01-16 08:00:00", 1, 1, 'MI', 3, 'Downtown', '[]'),
+    ("2019-01-16 08:00:00", 1, 1, 'MI', 3, 'Downtown', '[]', '[]'),
-    ("2019-01-17 08:00:00", 1, 1, 'MI', 3, 'Greektown', '[]'),
+    ("2019-01-17 08:00:00", 1, 1, 'MI', 3, 'Greektown', '[]', '[]'),
-    ("2019-01-17 08:00:00", 1, 1, 'MI', 3, 'Corktown', '[]'),
+    ("2019-01-17 08:00:00", 1, 1, 'MI', 3, 'Corktown', '[]', '[]'),
-    ("2019-01-17 08:00:00", 1, 1, 'MI', 3, 'Mexicantown', '[]'),
+    ("2019-01-17 08:00:00", 1, 1, 'MI', 3, 'Mexicantown', '[]', '[]'),
-    ("2019-01-17 08:00:00", 2, 0, 'MC', 4, 'Arcadia Planitia', '[]')
+    ("2019-01-17 08:00:00", 2, 0, 'MC', 4, 'Arcadia Planitia', '[]', '[]')
 ;
 
 CREATE TABLE binary_data (
@@ -617,6 +755,7 @@ INSERT INTO primary_key_multiple_columns VALUES (1, 'hey', 'world');
 INSERT INTO primary_key_multiple_columns_explicit_label VALUES (1, 'hey', 'world2');
 
 INSERT INTO foreign_key_references VALUES (1, 1, 1);
+INSERT INTO foreign_key_references VALUES (2, null, null);
 
 INSERT INTO complex_foreign_keys VALUES (1, 1, 2, 1);
 INSERT INTO custom_foreign_key_label VALUES (1, 1);

tests/test_api.py

@@ -6,7 +6,9 @@ from .fixtures import (  # noqa
     app_client_shorter_time_limit,
     app_client_larger_cache_size,
     app_client_returned_rows_matches_page_size,
+    app_client_two_attached_databases,
     app_client_two_attached_databases_one_immutable,
+    app_client_conflicting_database_names,
     app_client_with_cors,
     app_client_with_dot,
     generate_compound_rows,
@@ -22,6 +24,7 @@ import urllib
 def test_homepage(app_client):
     response = app_client.get("/.json")
     assert response.status == 200
+    assert "application/json; charset=utf-8" == response.headers["content-type"]
     assert response.json.keys() == {"fixtures": 0}.keys()
     d = response.json["fixtures"]
     assert d["name"] == "fixtures"
@@ -193,6 +196,7 @@ def test_database_page(app_client):
                 "city_id",
                 "neighborhood",
                 "tags",
+                "complex_array",
             ],
             "primary_keys": ["pk"],
             "count": 15,
@@ -213,7 +217,7 @@ def test_database_page(app_client):
             "name": "foreign_key_references",
             "columns": ["pk", "foreign_key_with_label", "foreign_key_with_no_label"],
             "primary_keys": ["pk"],
-            "count": 1,
+            "count": 2,
             "hidden": False,
             "fts_table": None,
             "foreign_keys": {
@@ -607,7 +611,8 @@ def test_table_json(app_client):
     assert response.status == 200
     data = response.json
     assert (
-        data["query"]["sql"] == "select * from simple_primary_key order by id limit 51"
+        data["query"]["sql"]
+        == "select id, content from simple_primary_key order by id limit 51"
     )
     assert data["query"]["params"] == {}
     assert data["rows"] == [
@@ -771,8 +776,8 @@ def test_paginate_tables_and_views(app_client, path, expected_rows, expected_pag
         fetched.extend(response.json["rows"])
         path = response.json["next_url"]
         if path:
-            assert response.json["next"]
             assert urllib.parse.urlencode({"_next": response.json["next"]}) in path
+            path = path.replace("http://localhost", "")
         assert count < 30, "Possible infinite loop detected"
 
     assert expected_rows == len(fetched)
@@ -812,6 +817,8 @@ def test_paginate_compound_keys(app_client):
         response = app_client.get(path)
         fetched.extend(response.json["rows"])
         path = response.json["next_url"]
+        if path:
+            path = path.replace("http://localhost", "")
         assert page < 100
     assert 1001 == len(fetched)
     assert 21 == page
@@ -833,6 +840,8 @@ def test_paginate_compound_keys_with_extra_filters(app_client):
         response = app_client.get(path)
         fetched.extend(response.json["rows"])
         path = response.json["next_url"]
+        if path:
+            path = path.replace("http://localhost", "")
     assert 2 == page
     expected = [r[3] for r in generate_compound_rows(1001) if "d" in r[3]]
     assert expected == [f["content"] for f in fetched]
@@ -881,6 +890,8 @@ def test_sortable(app_client, query_string, sort_key, human_description_en):
         assert human_description_en == response.json["human_description_en"]
         fetched.extend(response.json["rows"])
         path = response.json["next_url"]
+        if path:
+            path = path.replace("http://localhost", "")
     assert 5 == page
     expected = list(generate_sortable_rows(201))
     expected.sort(key=sort_key)
@@ -1021,15 +1032,25 @@ def test_table_filter_queries_multiple_of_same_type(app_client):
 def test_table_filter_json_arraycontains(app_client):
     response = app_client.get("/fixtures/facetable.json?tags__arraycontains=tag1")
     assert [
-        [1, "2019-01-14 08:00:00", 1, 1, "CA", 1, "Mission", '["tag1", "tag2"]'],
+        [
-        [2, "2019-01-14 08:00:00", 1, 1, "CA", 1, "Dogpatch", '["tag1", "tag3"]'],
+            1,
+            "2019-01-14 08:00:00",
+            1,
+            1,
+            "CA",
+            1,
+            "Mission",
+            '["tag1", "tag2"]',
+            '[{"foo": "bar"}]',
+        ],
+        [2, "2019-01-14 08:00:00", 1, 1, "CA", 1, "Dogpatch", '["tag1", "tag3"]', "[]"],
     ] == response.json["rows"]
 
 
 def test_table_filter_extra_where(app_client):
     response = app_client.get("/fixtures/facetable.json?_where=neighborhood='Dogpatch'")
     assert [
-        [2, "2019-01-14 08:00:00", 1, 1, "CA", 1, "Dogpatch", '["tag1", "tag3"]']
+        [2, "2019-01-14 08:00:00", 1, 1, "CA", 1, "Dogpatch", '["tag1", "tag3"]', "[]"]
     ] == response.json["rows"]
 
 
@@ -1099,6 +1120,15 @@ def test_row(app_client):
     assert [{"id": "1", "content": "hello"}] == response.json["rows"]
 
 
+def test_row_format_in_querystring(app_client):
+    # regression test for https://github.com/simonw/datasette/issues/563
+    response = app_client.get(
+        "/fixtures/simple_primary_key/1?_format=json&_shape=objects"
+    )
+    assert response.status == 200
+    assert [{"id": "1", "content": "hello"}] == response.json["rows"]
+
+
 def test_row_strange_table_name(app_client):
     response = app_client.get(
         "/fixtures/table%2Fwith%2Fslashes.csv/3.json?_shape=objects"
@@ -1159,7 +1189,7 @@ def test_databases_json(app_client_two_attached_databases_one_immutable):
     databases = response.json
     assert 2 == len(databases)
     extra_database, fixtures_database = databases
-    assert "extra_database" == extra_database["name"]
+    assert "extra database" == extra_database["name"]
     assert None == extra_database["hash"]
     assert True == extra_database["is_mutable"]
     assert False == extra_database["is_memory"]
@@ -1191,6 +1221,7 @@ def test_plugins_json(app_client):
 def test_versions_json(app_client):
     response = app_client.get("/-/versions.json")
     assert "python" in response.json
+    assert "3.0" == response.json.get("asgi")
     assert "version" in response.json["python"]
     assert "full" in response.json["python"]
     assert "datasette" in response.json
@@ -1227,7 +1258,7 @@ def test_config_json(app_client):
 
 
 def test_page_size_matching_max_returned_rows(
-    app_client_returned_rows_matches_page_size
+    app_client_returned_rows_matches_page_size,
 ):
     fetched = []
     path = "/fixtures/no_primary_key.json"
@@ -1236,6 +1267,8 @@ def test_page_size_matching_max_returned_rows(
         fetched.extend(response.json["rows"])
         assert len(response.json["rows"]) in (1, 50)
         path = response.json["next_url"]
+        if path:
+            path = path.replace("http://localhost", "")
     assert 201 == len(fetched)
 
 
@@ -1433,6 +1466,7 @@ def test_suggested_facets(app_client):
         {"name": "city_id", "querystring": "_facet=city_id"},
         {"name": "neighborhood", "querystring": "_facet=neighborhood"},
         {"name": "tags", "querystring": "_facet=tags"},
+        {"name": "complex_array", "querystring": "_facet=complex_array"},
         {"name": "created", "querystring": "_facet_date=created"},
     ]
     if detect_json1():
@@ -1468,6 +1502,7 @@ def test_expand_labels(app_client):
             "city_id": {"value": 1, "label": "San Francisco"},
             "neighborhood": "Dogpatch",
             "tags": '["tag1", "tag3"]',
+            "complex_array": "[]",
         },
         "13": {
             "pk": 13,
@@ -1478,6 +1513,7 @@ def test_expand_labels(app_client):
             "city_id": {"value": 3, "label": "Detroit"},
             "neighborhood": "Corktown",
             "tags": "[]",
+            "complex_array": "[]",
         },
     } == response.json
 
@@ -1485,7 +1521,7 @@ def test_expand_labels(app_client):
 def test_expand_label(app_client):
     response = app_client.get(
         "/fixtures/foreign_key_references.json?_shape=object"
-        "&_label=foreign_key_with_label"
+        "&_label=foreign_key_with_label&_size=1"
     )
     assert {
         "1": {
@@ -1603,6 +1639,11 @@ def test_infinity_returned_as_invalid_json_if_requested(app_client):
     ] == response.json
 
 
+def test_custom_query_with_unicode_characters(app_client):
+    response = app_client.get("/fixtures/𝐜𝐢𝐭𝐢𝐞𝐬.json?_shape=array")
+    assert [{"id": 1, "name": "San Francisco"}] == response.json
+
+
 def test_trace(app_client):
     response = app_client.get("/fixtures/simple_primary_key.json?_trace=1")
     data = response.json
@@ -1637,3 +1678,50 @@ def test_cors(app_client_with_cors, path, status_code):
     response = app_client_with_cors.get(path)
     assert response.status == status_code
     assert "*" == response.headers["Access-Control-Allow-Origin"]
+
+
+@pytest.mark.parametrize(
+    "path",
+    (
+        "/",
+        ".json",
+        "/searchable",
+        "/searchable.json",
+        "/searchable_view",
+        "/searchable_view.json",
+    ),
+)
+def test_database_with_space_in_name(app_client_two_attached_databases, path):
+    response = app_client_two_attached_databases.get("/extra database" + path)
+    assert response.status == 200
+
+
+def test_common_prefix_database_names(app_client_conflicting_database_names):
+    # https://github.com/simonw/datasette/issues/597
+    assert ["fixtures", "foo", "foo-bar"] == [
+        d["name"]
+        for d in json.loads(
+            app_client_conflicting_database_names.get("/-/databases.json").body.decode(
+                "utf8"
+            )
+        )
+    ]
+    for db_name, path in (("foo", "/foo.json"), ("foo-bar", "/foo-bar.json")):
+        data = json.loads(
+            app_client_conflicting_database_names.get(path).body.decode("utf8")
+        )
+        assert db_name == data["database"]
+
+
+def test_null_foreign_keys_are_not_expanded(app_client):
+    response = app_client.get(
+        "/fixtures/foreign_key_references.json?_shape=array&_labels=on"
+    )
+    assert [
+        {
+            "pk": "1",
+            "foreign_key_with_label": {"value": "1", "label": "hello"},
+            "foreign_key_with_no_label": {"value": "1", "label": "1"},
+        },
+        {"pk": "2", "foreign_key_with_label": None, "foreign_key_with_no_label": None,},
+    ] == response.json

tests/test_csv.py

@@ -21,22 +21,30 @@ world
 )
 
 EXPECTED_TABLE_WITH_LABELS_CSV = """
-pk,created,planet_int,on_earth,state,city_id,city_id_label,neighborhood,tags
+pk,created,planet_int,on_earth,state,city_id,city_id_label,neighborhood,tags,complex_array
-1,2019-01-14 08:00:00,1,1,CA,1,San Francisco,Mission,"[""tag1"", ""tag2""]"
+1,2019-01-14 08:00:00,1,1,CA,1,San Francisco,Mission,"[""tag1"", ""tag2""]","[{""foo"": ""bar""}]"
-2,2019-01-14 08:00:00,1,1,CA,1,San Francisco,Dogpatch,"[""tag1"", ""tag3""]"
+2,2019-01-14 08:00:00,1,1,CA,1,San Francisco,Dogpatch,"[""tag1"", ""tag3""]",[]
-3,2019-01-14 08:00:00,1,1,CA,1,San Francisco,SOMA,[]
+3,2019-01-14 08:00:00,1,1,CA,1,San Francisco,SOMA,[],[]
-4,2019-01-14 08:00:00,1,1,CA,1,San Francisco,Tenderloin,[]
+4,2019-01-14 08:00:00,1,1,CA,1,San Francisco,Tenderloin,[],[]
-5,2019-01-15 08:00:00,1,1,CA,1,San Francisco,Bernal Heights,[]
+5,2019-01-15 08:00:00,1,1,CA,1,San Francisco,Bernal Heights,[],[]
-6,2019-01-15 08:00:00,1,1,CA,1,San Francisco,Hayes Valley,[]
+6,2019-01-15 08:00:00,1,1,CA,1,San Francisco,Hayes Valley,[],[]
-7,2019-01-15 08:00:00,1,1,CA,2,Los Angeles,Hollywood,[]
+7,2019-01-15 08:00:00,1,1,CA,2,Los Angeles,Hollywood,[],[]
-8,2019-01-15 08:00:00,1,1,CA,2,Los Angeles,Downtown,[]
+8,2019-01-15 08:00:00,1,1,CA,2,Los Angeles,Downtown,[],[]
-9,2019-01-16 08:00:00,1,1,CA,2,Los Angeles,Los Feliz,[]
+9,2019-01-16 08:00:00,1,1,CA,2,Los Angeles,Los Feliz,[],[]
-10,2019-01-16 08:00:00,1,1,CA,2,Los Angeles,Koreatown,[]
+10,2019-01-16 08:00:00,1,1,CA,2,Los Angeles,Koreatown,[],[]
-11,2019-01-16 08:00:00,1,1,MI,3,Detroit,Downtown,[]
+11,2019-01-16 08:00:00,1,1,MI,3,Detroit,Downtown,[],[]
-12,2019-01-17 08:00:00,1,1,MI,3,Detroit,Greektown,[]
+12,2019-01-17 08:00:00,1,1,MI,3,Detroit,Greektown,[],[]
-13,2019-01-17 08:00:00,1,1,MI,3,Detroit,Corktown,[]
+13,2019-01-17 08:00:00,1,1,MI,3,Detroit,Corktown,[],[]
-14,2019-01-17 08:00:00,1,1,MI,3,Detroit,Mexicantown,[]
+14,2019-01-17 08:00:00,1,1,MI,3,Detroit,Mexicantown,[],[]
-15,2019-01-17 08:00:00,2,0,MC,4,Memnonia,Arcadia Planitia,[]
+15,2019-01-17 08:00:00,2,0,MC,4,Memnonia,Arcadia Planitia,[],[]
+""".lstrip().replace(
+    "\n", "\r\n"
+)
+
+EXPECTED_TABLE_WITH_NULLABLE_LABELS_CSV = """
+pk,foreign_key_with_label,foreign_key_with_label_label,foreign_key_with_no_label,foreign_key_with_no_label_label
+1,1,hello,1,1
+2,,,,
 """.lstrip().replace(
     "\n", "\r\n"
 )
@@ -46,7 +54,7 @@ def test_table_csv(app_client):
     response = app_client.get("/fixtures/simple_primary_key.csv")
     assert response.status == 200
     assert not response.headers.get("Access-Control-Allow-Origin")
-    assert "text/plain; charset=utf-8" == response.headers["Content-Type"]
+    assert "text/plain; charset=utf-8" == response.headers["content-type"]
     assert EXPECTED_TABLE_CSV == response.text
 
 
@@ -59,27 +67,43 @@ def test_table_csv_cors_headers(app_client_with_cors):
 def test_table_csv_with_labels(app_client):
     response = app_client.get("/fixtures/facetable.csv?_labels=1")
     assert response.status == 200
-    assert "text/plain; charset=utf-8" == response.headers["Content-Type"]
+    assert "text/plain; charset=utf-8" == response.headers["content-type"]
     assert EXPECTED_TABLE_WITH_LABELS_CSV == response.text
 
 
+def test_table_csv_with_nullable_labels(app_client):
+    response = app_client.get("/fixtures/foreign_key_references.csv?_labels=1")
+    assert response.status == 200
+    assert "text/plain; charset=utf-8" == response.headers["content-type"]
+    assert EXPECTED_TABLE_WITH_NULLABLE_LABELS_CSV == response.text
+
+
 def test_custom_sql_csv(app_client):
     response = app_client.get(
         "/fixtures.csv?sql=select+content+from+simple_primary_key+limit+2"
     )
     assert response.status == 200
-    assert "text/plain; charset=utf-8" == response.headers["Content-Type"]
+    assert "text/plain; charset=utf-8" == response.headers["content-type"]
     assert EXPECTED_CUSTOM_CSV == response.text
 
 
 def test_table_csv_download(app_client):
     response = app_client.get("/fixtures/simple_primary_key.csv?_dl=1")
     assert response.status == 200
-    assert "text/csv; charset=utf-8" == response.headers["Content-Type"]
+    assert "text/csv; charset=utf-8" == response.headers["content-type"]
     expected_disposition = 'attachment; filename="simple_primary_key.csv"'
     assert expected_disposition == response.headers["Content-Disposition"]
 
 
+def test_csv_with_non_ascii_characters(app_client):
+    response = app_client.get(
+        "/fixtures.csv?sql=select%0D%0A++%27%F0%9D%90%9C%F0%9D%90%A2%F0%9D%90%AD%F0%9D%90%A2%F0%9D%90%9E%F0%9D%90%AC%27+as+text%2C%0D%0A++1+as+number%0D%0Aunion%0D%0Aselect%0D%0A++%27bob%27+as+text%2C%0D%0A++2+as+number%0D%0Aorder+by%0D%0A++number"
+    )
+    assert response.status == 200
+    assert "text/plain; charset=utf-8" == response.headers["content-type"]
+    assert "text,number\r\n𝐜𝐢𝐭𝐢𝐞𝐬,1\r\nbob,2\r\n" == response.body.decode("utf8")
+
+
 def test_max_csv_mb(app_client_csv_max_mb_one):
     response = app_client_csv_max_mb_one.get(
         "/fixtures.csv?sql=select+randomblob(10000)+"

tests/test_facets.py

@@ -1,8 +1,7 @@
-from datasette.facets import ColumnFacet, ArrayFacet, DateFacet, ManyToManyFacet
+from datasette.facets import ColumnFacet, ArrayFacet, DateFacet
 from datasette.utils import detect_json1
 from .fixtures import app_client  # noqa
 from .utils import MockRequest
-from collections import namedtuple
 import pytest
 
 
@@ -24,6 +23,10 @@ async def test_column_facet_suggest(app_client):
         {"name": "city_id", "toggle_url": "http://localhost/?_facet=city_id"},
         {"name": "neighborhood", "toggle_url": "http://localhost/?_facet=neighborhood"},
         {"name": "tags", "toggle_url": "http://localhost/?_facet=tags"},
+        {
+            "name": "complex_array",
+            "toggle_url": "http://localhost/?_facet=complex_array",
+        },
     ] == suggestions
 
 
@@ -58,6 +61,10 @@ async def test_column_facet_suggest_skip_if_already_selected(app_client):
             "name": "tags",
             "toggle_url": "http://localhost/?_facet=planet_int&_facet=on_earth&_facet=tags",
         },
+        {
+            "name": "complex_array",
+            "toggle_url": "http://localhost/?_facet=planet_int&_facet=on_earth&_facet=complex_array",
+        },
     ] == suggestions
 
 
@@ -79,6 +86,7 @@ async def test_column_facet_suggest_skip_if_enabled_by_metadata(app_client):
         "state",
         "neighborhood",
         "tags",
+        "complex_array",
     ] == suggestions
 
 
@@ -207,6 +215,20 @@ async def test_array_facet_suggest(app_client):
     ] == suggestions
 
 
+@pytest.mark.asyncio
+@pytest.mark.skipif(not detect_json1(), reason="Requires the SQLite json1 module")
+async def test_array_facet_suggest_not_if_all_empty_arrays(app_client):
+    facet = ArrayFacet(
+        app_client.ds,
+        MockRequest("http://localhost/"),
+        database="fixtures",
+        sql="select * from facetable where tags = '[]'",
+        table="facetable",
+    )
+    suggestions = await facet.suggest()
+    assert [] == suggestions
+
+
 @pytest.mark.asyncio
 @pytest.mark.skipif(not detect_json1(), reason="Requires the SQLite json1 module")
 async def test_array_facet_results(app_client):
@@ -303,60 +325,3 @@ async def test_date_facet_results(app_client):
             "truncated": False,
         }
     } == buckets
-
-
-@pytest.mark.asyncio
-async def test_m2m_facet_suggest(app_client):
-    facet = ManyToManyFacet(
-        app_client.ds,
-        MockRequest("http://localhost/"),
-        database="fixtures",
-        sql="select * from roadside_attractions",
-        table="roadside_attractions",
-    )
-    suggestions = await facet.suggest()
-    assert [
-        {
-            "name": "attraction_characteristic",
-            "type": "m2m",
-            "toggle_url": "http://localhost/?_facet_m2m=attraction_characteristic",
-        }
-    ] == suggestions
-
-
-@pytest.mark.asyncio
-async def test_m2m_facet_results(app_client):
-    facet = ManyToManyFacet(
-        app_client.ds,
-        MockRequest("http://localhost/?_facet_m2m=attraction_characteristic"),
-        database="fixtures",
-        sql="select * from roadside_attractions",
-        table="roadside_attractions",
-    )
-    buckets, timed_out = await facet.facet_results()
-    assert [] == timed_out
-    assert {
-        "attraction_characteristic": {
-            "name": "attraction_characteristic",
-            "type": "m2m",
-            "results": [
-                {
-                    "value": 2,
-                    "label": "Paranormal",
-                    "count": 3,
-                    "toggle_url": "http://localhost/?_facet_m2m=attraction_characteristic&_through=%7B%22column%22%3A%22characteristic_id%22%2C%22table%22%3A%22roadside_attraction_characteristics%22%2C%22value%22%3A%222%22%7D",
-                    "selected": False,
-                },
-                {
-                    "value": 1,
-                    "label": "Museum",
-                    "count": 2,
-                    "toggle_url": "http://localhost/?_facet_m2m=attraction_characteristic&_through=%7B%22column%22%3A%22characteristic_id%22%2C%22table%22%3A%22roadside_attraction_characteristics%22%2C%22value%22%3A%221%22%7D",
-                    "selected": False,
-                },
-            ],
-            "hideable": True,
-            "toggle_url": "/",
-            "truncated": False,
-        }
-    } == buckets

tests/test_filters.py

@@ -47,6 +47,9 @@ import pytest
             ["foo in (:p0, :p1)"],
             ["dog,cat", "cat[dog]"],
         ),
+        # Not in, and JSON array not in
+        ((("foo__notin", "1,2,3"),), ["foo not in (:p0, :p1, :p2)"], ["1", "2", "3"]),
+        ((("foo__notin", "[1,2,3]"),), ["foo not in (:p0, :p1, :p2)"], [1, 2, 3]),
     ],
 )
 def test_build_where(args, expected_where, expected_params):

tests/test_html.py

@@ -8,6 +8,7 @@ from .fixtures import (  # noqa
     METADATA,
 )
 import json
+import pathlib
 import pytest
 import re
 import urllib.parse
@@ -16,6 +17,7 @@ import urllib.parse
 def test_homepage(app_client_two_attached_databases):
     response = app_client_two_attached_databases.get("/")
     assert response.status == 200
+    assert "text/html; charset=utf-8" == response.headers["content-type"]
     soup = Soup(response.body, "html.parser")
     assert "Datasette Fixtures" == soup.find("h1").text
     assert (
@@ -24,12 +26,12 @@ def test_homepage(app_client_two_attached_databases):
     )
     # Should be two attached databases
     assert [
-        {"href": "/extra_database", "text": "extra_database"},
         {"href": "/fixtures", "text": "fixtures"},
+        {"href": "/extra database", "text": "extra database"},
     ] == [{"href": a["href"], "text": a.text.strip()} for a in soup.select("h2 a")]
     # The first attached database should show count text and attached tables
-    h2 = soup.select("h2")[0]
+    h2 = soup.select("h2")[1]
-    assert "extra_database" == h2.text.strip()
+    assert "extra database" == h2.text.strip()
     counts_p, links_p = h2.find_all_next("p")[:2]
     assert (
         "2 rows in 1 table, 5 rows in 4 hidden tables, 1 view" == counts_p.text.strip()
@@ -39,11 +41,36 @@ def test_homepage(app_client_two_attached_databases):
         {"href": a["href"], "text": a.text.strip()} for a in links_p.findAll("a")
     ]
     assert [
-        {"href": "/extra_database/searchable", "text": "searchable"},
+        {"href": "/extra database/searchable", "text": "searchable"},
-        {"href": "/extra_database/searchable_view", "text": "searchable_view"},
+        {"href": "/extra database/searchable_view", "text": "searchable_view"},
     ] == table_links
 
 
+def test_http_head(app_client):
+    response = app_client.get("/", method="HEAD")
+    assert response.status == 200
+
+
+def test_static(app_client):
+    response = app_client.get("/-/static/app2.css")
+    assert response.status == 404
+    response = app_client.get("/-/static/app.css")
+    assert response.status == 200
+    assert "text/css" == response.headers["content-type"]
+
+
+def test_static_mounts():
+    for client in make_app_client(
+        static_mounts=[("custom-static", str(pathlib.Path(__file__).parent))]
+    ):
+        response = client.get("/custom-static/test_html.py")
+        assert response.status == 200
+        response = client.get("/custom-static/not_exists.py")
+        assert response.status == 404
+        response = client.get("/custom-static/../LICENSE")
+        assert response.status == 404
+
+
 def test_memory_database_page():
     for client in make_app_client(memory=True):
         response = client.get("/:memory:")
@@ -92,6 +119,39 @@ def test_row_strange_table_name_with_url_hash(app_client_with_hash):
     assert response.status == 200
 
 
+@pytest.mark.parametrize(
+    "path,expected_definition_sql",
+    [
+        (
+            "/fixtures/facet_cities",
+            """
+CREATE TABLE facet_cities (
+    id integer primary key,
+    name text
+);
+        """.strip(),
+        ),
+        (
+            "/fixtures/compound_three_primary_keys",
+            """
+CREATE TABLE compound_three_primary_keys (
+  pk1 varchar(30),
+  pk2 varchar(30),
+  pk3 varchar(30),
+  content text,
+  PRIMARY KEY (pk1, pk2, pk3)
+);
+CREATE INDEX idx_compound_three_primary_keys_content ON compound_three_primary_keys(content);
+            """.strip(),
+        ),
+    ],
+)
+def test_definition_sql(path, expected_definition_sql, app_client):
+    response = app_client.get(path)
+    pre = Soup(response.body, "html.parser").select_one("pre.wrapped-sql")
+    assert expected_definition_sql == pre.string
+
+
 def test_table_cell_truncation():
     for client in make_app_client(config={"truncate_cells_html": 5}):
         response = client.get("/fixtures/facetable")
@@ -576,7 +636,12 @@ def test_table_html_foreign_key_links(app_client):
             '<td class="col-pk"><a href="/fixtures/foreign_key_references/1">1</a></td>',
             '<td class="col-foreign_key_with_label"><a href="/fixtures/simple_primary_key/1">hello</a>\xa0<em>1</em></td>',
             '<td class="col-foreign_key_with_no_label"><a href="/fixtures/primary_key_multiple_columns/1">1</a></td>',
-        ]
+        ],
+        [
+            '<td class="col-pk"><a href="/fixtures/foreign_key_references/2">2</a></td>',
+            '<td class="col-foreign_key_with_label">\xa0</td>',
+            '<td class="col-foreign_key_with_no_label">\xa0</td>',
+        ],
     ]
     assert expected == [
         [str(td) for td in tr.select("td")] for tr in table.select("tbody tr")
@@ -584,7 +649,7 @@ def test_table_html_foreign_key_links(app_client):
 
 
 def test_table_html_disable_foreign_key_links_with_labels(app_client):
-    response = app_client.get("/fixtures/foreign_key_references?_labels=off")
+    response = app_client.get("/fixtures/foreign_key_references?_labels=off&_size=1")
     assert response.status == 200
     table = Soup(response.body, "html.parser").find("table")
     expected = [
@@ -710,6 +775,18 @@ def test_database_metadata(app_client):
     assert_footer_links(soup)
 
 
+def test_database_metadata_with_custom_sql(app_client):
+    response = app_client.get("/fixtures?sql=select+*+from+simple_primary_key")
+    assert response.status == 200
+    soup = Soup(response.body, "html.parser")
+    # Page title should be the default
+    assert "fixtures" == soup.find("h1").text
+    # Description should be custom
+    assert "Custom SQL query returning" in soup.find("h3").text
+    # The source/license should be inherited
+    assert_footer_links(soup)
+
+
 def test_table_metadata(app_client):
     response = app_client.get("/fixtures/simple_primary_key")
     assert response.status == 200
@@ -916,6 +993,12 @@ def test_extra_where_clauses(app_client):
         "/fixtures/facetable?_where=city_id%3D1",
         "/fixtures/facetable?_where=neighborhood%3D%27Dogpatch%27",
     ] == hrefs
+    # These should also be persisted as hidden fields
+    inputs = soup.find("form").findAll("input")
+    hiddens = [i for i in inputs if i["type"] == "hidden"]
+    assert [("_where", "neighborhood='Dogpatch'"), ("_where", "city_id=1")] == [
+        (hidden["name"], hidden["value"]) for hidden in hiddens
+    ]
 
 
 def test_binary_data_display(app_client):
@@ -939,3 +1022,16 @@ def test_metadata_json_html(app_client):
     assert response.status == 200
     pre = Soup(response.body, "html.parser").find("pre")
     assert METADATA == json.loads(pre.text)
+
+
+def test_custom_table_include():
+    for client in make_app_client(
+        template_dir=str(pathlib.Path(__file__).parent / "test_templates")
+    ):
+        response = client.get("/fixtures/complex_foreign_keys")
+        assert response.status == 200
+        assert (
+            '<div class="custom-table-row">'
+            '1 - 2 - <a href="/fixtures/simple_primary_key/1">hello</a> <em>1</em>'
+            "</div>"
+        ) == str(Soup(response.text, "html.parser").select_one("div.custom-table-row"))

tests/test_plugins.py

@@ -1,7 +1,9 @@
 from bs4 import BeautifulSoup as Soup
-from .fixtures import app_client  # noqa
+from .fixtures import app_client, make_app_client, TEMP_PLUGIN_SECRET_FILE  # noqa
 import base64
 import json
+import os
+import pathlib
 import re
 import pytest
 import urllib
@@ -125,6 +127,26 @@ def test_plugin_config(app_client):
     assert None is app_client.ds.plugin_config("unknown-plugin")
 
 
+def test_plugin_config_env(app_client):
+    os.environ["FOO_ENV"] = "FROM_ENVIRONMENT"
+    assert {"foo": "FROM_ENVIRONMENT"} == app_client.ds.plugin_config("env-plugin")
+    # Ensure secrets aren't visible in /-/metadata.json
+    metadata = app_client.get("/-/metadata.json")
+    assert {"foo": {"$env": "FOO_ENV"}} == metadata.json["plugins"]["env-plugin"]
+    del os.environ["FOO_ENV"]
+
+
+def test_plugin_config_file(app_client):
+    open(TEMP_PLUGIN_SECRET_FILE, "w").write("FROM_FILE")
+    assert {"foo": "FROM_FILE"} == app_client.ds.plugin_config("file-plugin")
+    # Ensure secrets aren't visible in /-/metadata.json
+    metadata = app_client.get("/-/metadata.json")
+    assert {"foo": {"$file": TEMP_PLUGIN_SECRET_FILE}} == metadata.json["plugins"][
+        "file-plugin"
+    ]
+    os.remove(TEMP_PLUGIN_SECRET_FILE)
+
+
 @pytest.mark.parametrize(
     "path,expected_extra_body_script",
     [
@@ -162,3 +184,33 @@ def test_plugins_extra_body_script(app_client, path, expected_extra_body_script)
     json_data = r.search(app_client.get(path).body.decode("utf8")).group(1)
     actual_data = json.loads(json_data)
     assert expected_extra_body_script == actual_data
+
+
+def test_plugins_asgi_wrapper(app_client):
+    response = app_client.get("/fixtures")
+    assert "fixtures" == response.headers["x-databases"]
+
+
+def test_plugins_extra_template_vars(restore_working_directory):
+    for client in make_app_client(
+        template_dir=str(pathlib.Path(__file__).parent / "test_templates")
+    ):
+        response = client.get("/-/metadata")
+        assert response.status == 200
+        extra_template_vars = json.loads(
+            Soup(response.body, "html.parser").select("pre.extra_template_vars")[0].text
+        )
+        assert {
+            "template": "show_json.html",
+            "scope_path": "/-/metadata",
+        } == extra_template_vars
+        extra_template_vars_from_awaitable = json.loads(
+            Soup(response.body, "html.parser")
+            .select("pre.extra_template_vars_from_awaitable")[0]
+            .text
+        )
+        assert {
+            "template": "show_json.html",
+            "awaitable": True,
+            "scope_path": "/-/metadata",
+        } == extra_template_vars_from_awaitable

tests/test_publish_cloudrun.py

@@ -1,6 +1,7 @@
 from click.testing import CliRunner
 from datasette import cli
 from unittest import mock
+import json
 
 
 @mock.patch("shutil.which")
@@ -26,23 +27,127 @@ def test_publish_cloudrun_invalid_database(mock_which):
 @mock.patch("shutil.which")
 @mock.patch("datasette.publish.cloudrun.check_output")
 @mock.patch("datasette.publish.cloudrun.check_call")
-def test_publish_cloudrun(mock_call, mock_output, mock_which):
+@mock.patch("datasette.publish.cloudrun.get_existing_services")
+def test_publish_cloudrun_prompts_for_service(
+    mock_get_existing_services, mock_call, mock_output, mock_which
+):
+    mock_get_existing_services.return_value = [
+        {"name": "existing", "created": "2019-01-01", "url": "http://www.example.com/"}
+    ]
     mock_output.return_value = "myproject"
     mock_which.return_value = True
     runner = CliRunner()
     with runner.isolated_filesystem():
         open("test.db", "w").write("data")
-        result = runner.invoke(cli.cli, ["publish", "cloudrun", "test.db"])
+        result = runner.invoke(
+            cli.cli, ["publish", "cloudrun", "test.db"], input="input-service"
+        )
+        assert (
+            """
+Please provide a service name for this deployment
+
+Using an existing service name will over-write it
+
+Your existing services:
+
+  existing - created 2019-01-01 - http://www.example.com/
+
+Service name: input-service
+""".strip()
+            == result.output.strip()
+        )
         assert 0 == result.exit_code
-        tag = "gcr.io/{}/datasette".format(mock_output.return_value)
+        tag = "gcr.io/myproject/datasette"
         mock_call.assert_has_calls(
             [
                 mock.call("gcloud builds submit --tag {}".format(tag), shell=True),
                 mock.call(
-                    "gcloud beta run deploy --allow-unauthenticated --image {}".format(
+                    "gcloud beta run deploy --allow-unauthenticated --platform=managed --image {} input-service".format(
                         tag
                     ),
                     shell=True,
                 ),
             ]
         )
+
+
+@mock.patch("shutil.which")
+@mock.patch("datasette.publish.cloudrun.check_output")
+@mock.patch("datasette.publish.cloudrun.check_call")
+def test_publish_cloudrun(mock_call, mock_output, mock_which):
+    mock_output.return_value = "myproject"
+    mock_which.return_value = True
+    runner = CliRunner()
+    with runner.isolated_filesystem():
+        open("test.db", "w").write("data")
+        result = runner.invoke(
+            cli.cli, ["publish", "cloudrun", "test.db", "--service", "test"]
+        )
+        assert 0 == result.exit_code
+        tag = "gcr.io/{}/datasette".format(mock_output.return_value)
+        mock_call.assert_has_calls(
+            [
+                mock.call("gcloud builds submit --tag {}".format(tag), shell=True),
+                mock.call(
+                    "gcloud beta run deploy --allow-unauthenticated --platform=managed --image {} test".format(
+                        tag
+                    ),
+                    shell=True,
+                ),
+            ]
+        )
+
+
+@mock.patch("shutil.which")
+@mock.patch("datasette.publish.cloudrun.check_output")
+@mock.patch("datasette.publish.cloudrun.check_call")
+def test_publish_cloudrun_plugin_secrets(mock_call, mock_output, mock_which):
+    mock_which.return_value = True
+    mock_output.return_value = "myproject"
+
+    runner = CliRunner()
+    with runner.isolated_filesystem():
+        open("test.db", "w").write("data")
+        result = runner.invoke(
+            cli.cli,
+            [
+                "publish",
+                "cloudrun",
+                "test.db",
+                "--service",
+                "datasette",
+                "--plugin-secret",
+                "datasette-auth-github",
+                "client_id",
+                "x-client-id",
+                "--show-files",
+            ],
+        )
+        dockerfile = (
+            result.output.split("==== Dockerfile ====\n")[1]
+            .split("\n====================\n")[0]
+            .strip()
+        )
+        expected = """FROM python:3.6
+COPY . /app
+WORKDIR /app
+
+ENV DATASETTE_AUTH_GITHUB_CLIENT_ID 'x-client-id'
+RUN pip install -U datasette
+RUN datasette inspect test.db --inspect-file inspect-data.json
+ENV PORT 8001
+EXPOSE 8001
+CMD datasette serve --host 0.0.0.0 -i test.db --cors --inspect-file inspect-data.json --metadata metadata.json --port $PORT""".strip()
+        assert expected == dockerfile
+        metadata = (
+            result.output.split("=== metadata.json ===\n")[1]
+            .split("\n==== Dockerfile ====\n")[0]
+            .strip()
+        )
+        assert {
+            "plugins": {
+                "datasette-auth-github": {
+                    "client_id": {"$env": "DATASETTE_AUTH_GITHUB_CLIENT_ID"}
+                }
+            }
+        } == json.loads(metadata)

tests/test_publish_heroku.py

@@ -46,7 +46,7 @@ def test_publish_heroku_invalid_database(mock_which):
 @mock.patch("datasette.publish.heroku.check_output")
 @mock.patch("datasette.publish.heroku.call")
 def test_publish_heroku(mock_call, mock_check_output, mock_which):
-    mock_which.return_varue = True
+    mock_which.return_value = True
     mock_check_output.side_effect = lambda s: {
         "['heroku', 'plugins']": b"heroku-builds",
         "['heroku', 'apps:list', '--json']": b"[]",
@@ -60,3 +60,47 @@ def test_publish_heroku(mock_call, mock_check_output, mock_which):
         mock_call.assert_called_once_with(
             ["heroku", "builds:create", "-a", "f", "--include-vcs-ignore"]
         )
+
+
+@mock.patch("shutil.which")
+@mock.patch("datasette.publish.heroku.check_output")
+@mock.patch("datasette.publish.heroku.call")
+def test_publish_heroku_plugin_secrets(mock_call, mock_check_output, mock_which):
+    mock_which.return_value = True
+    mock_check_output.side_effect = lambda s: {
+        "['heroku', 'plugins']": b"heroku-builds",
+        "['heroku', 'apps:list', '--json']": b"[]",
+        "['heroku', 'apps:create', 'datasette', '--json']": b'{"name": "f"}',
+    }[repr(s)]
+    runner = CliRunner()
+    with runner.isolated_filesystem():
+        open("test.db", "w").write("data")
+        result = runner.invoke(
+            cli.cli,
+            [
+                "publish",
+                "heroku",
+                "test.db",
+                "--plugin-secret",
+                "datasette-auth-github",
+                "client_id",
+                "x-client-id",
+            ],
+        )
+        assert 0 == result.exit_code, result.output
+        mock_call.assert_has_calls(
+            [
+                mock.call(
+                    [
+                        "heroku",
+                        "config:set",
+                        "-a",
+                        "f",
+                        "DATASETTE_AUTH_GITHUB_CLIENT_ID=x-client-id",
+                    ]
+                ),
+                mock.call(
+                    ["heroku", "builds:create", "-a", "f", "--include-vcs-ignore"]
+                ),
+            ]
+        )

tests/test_publish_now.py

@@ -1,6 +1,7 @@
 from click.testing import CliRunner
 from datasette import cli
 from unittest import mock
+import json
 import subprocess
 
 
@@ -105,3 +106,58 @@ def test_publish_now_multiple_aliases(mock_run, mock_which):
                 ),
             ]
         )
+
+
+@mock.patch("shutil.which")
+@mock.patch("datasette.publish.now.run")
+def test_publish_now_plugin_secrets(mock_run, mock_which):
+    mock_which.return_value = True
+    mock_run.return_value = mock.Mock(0)
+    mock_run.return_value.stdout = b"https://demo.example.com/"
+
+    runner = CliRunner()
+    with runner.isolated_filesystem():
+        open("test.db", "w").write("data")
+        result = runner.invoke(
+            cli.cli,
+            [
+                "publish",
+                "now",
+                "test.db",
+                "--token",
+                "XXX",
+                "--plugin-secret",
+                "datasette-auth-github",
+                "client_id",
+                "x-client-id",
+                "--show-files",
+            ],
+        )
+        dockerfile = (
+            result.output.split("==== Dockerfile ====\n")[1]
+            .split("\n====================\n")[0]
+            .strip()
+        )
+        expected = """FROM python:3.6
+COPY . /app
+WORKDIR /app
+
+ENV DATASETTE_AUTH_GITHUB_CLIENT_ID 'x-client-id'
+RUN pip install -U datasette
+RUN datasette inspect test.db --inspect-file inspect-data.json
+ENV PORT 8001
+EXPOSE 8001
+CMD datasette serve --host 0.0.0.0 -i test.db --cors --inspect-file inspect-data.json --metadata metadata.json --config force_https_urls:on --port $PORT""".strip()
+        assert expected == dockerfile
+        metadata = (
+            result.output.split("=== metadata.json ===\n")[1]
+            .split("\n==== Dockerfile ====\n")[0]
+            .strip()
+        )
+        assert {
+            "plugins": {
+                "datasette-auth-github": {
+                    "client_id": {"$env": "DATASETTE_AUTH_GITHUB_CLIENT_ID"}
+                }
+            }
+        } == json.loads(metadata)

tests/test_templates/_table.html

@@ -0,0 +1,3 @@
+{% for row in display_rows %}
+    <div class="custom-table-row">{{ row["f1"] }} - {{ row["f2"] }} - {{ row.display("f3") }}</div>
+{% endfor %}

tests/test_templates/show_json.html

@@ -0,0 +1,8 @@
+{% extends "base.html" %}
+
+{% block content %}
+{{ super() }}
+Test data for extra_template_vars:
+<pre class="extra_template_vars">{{ extra_template_vars|safe }}</pre>
+<pre class="extra_template_vars_from_awaitable">{{ extra_template_vars_from_awaitable|safe }}</pre>
+{% endblock %}

tests/test_utils.py

@@ -3,11 +3,11 @@ Tests for various datasette helper functions.
 """
 
 from datasette import utils
+from datasette.utils.asgi import Request
 from datasette.filters import Filters
 import json
 import os
 import pytest
-from sanic.request import Request
 import sqlite3
 import tempfile
 from unittest.mock import patch
@@ -53,7 +53,7 @@ def test_urlsafe_components(path, expected):
     ],
 )
 def test_path_with_added_args(path, added_args, expected):
-    request = Request(path.encode("utf8"), {}, "1.1", "GET", None)
+    request = Request.fake(path)
     actual = utils.path_with_added_args(request, added_args)
     assert expected == actual
 
@@ -67,11 +67,11 @@ def test_path_with_added_args(path, added_args, expected):
     ],
 )
 def test_path_with_removed_args(path, args, expected):
-    request = Request(path.encode("utf8"), {}, "1.1", "GET", None)
+    request = Request.fake(path)
     actual = utils.path_with_removed_args(request, args)
     assert expected == actual
     # Run the test again but this time use the path= argument
-    request = Request("/".encode("utf8"), {}, "1.1", "GET", None)
+    request = Request.fake("/")
     actual = utils.path_with_removed_args(request, args, path=path)
     assert expected == actual
 
@@ -84,7 +84,7 @@ def test_path_with_removed_args(path, args, expected):
     ],
 )
 def test_path_with_replaced_args(path, args, expected):
-    request = Request(path.encode("utf8"), {}, "1.1", "GET", None)
+    request = Request.fake(path)
     actual = utils.path_with_replaced_args(request, args)
     assert expected == actual
 
@@ -151,15 +151,20 @@ def test_validate_sql_select_bad(bad_sql):
         "select count(*) from airports",
         "select foo from bar",
         "select 1 + 1",
+        "explain select 1 + 1",
+        "explain query plan select 1 + 1",
         "SELECT\nblah FROM foo",
         "WITH RECURSIVE cnt(x) AS (SELECT 1 UNION ALL SELECT x+1 FROM cnt LIMIT 10) SELECT x FROM cnt;",
+        "explain WITH RECURSIVE cnt(x) AS (SELECT 1 UNION ALL SELECT x+1 FROM cnt LIMIT 10) SELECT x FROM cnt;",
+        "explain query plan WITH RECURSIVE cnt(x) AS (SELECT 1 UNION ALL SELECT x+1 FROM cnt LIMIT 10) SELECT x FROM cnt;",
     ],
 )
 def test_validate_sql_select_good(good_sql):
     utils.validate_sql_select(good_sql)
 
 
-def test_detect_fts():
+@pytest.mark.parametrize("open_quote,close_quote", [('"', '"'), ("[", "]")])
+def test_detect_fts(open_quote, close_quote):
     sql = """
     CREATE TABLE "Dumb_Table" (
       "TreeID" INTEGER,
@@ -175,9 +180,11 @@ def test_detect_fts():
       "qCaretaker" TEXT
     );
     CREATE VIEW Test_View AS SELECT * FROM Dumb_Table;
-    CREATE VIRTUAL TABLE "Street_Tree_List_fts" USING FTS4 ("qAddress", "qCaretaker", "qSpecies", content="Street_Tree_List");
+    CREATE VIRTUAL TABLE {open}Street_Tree_List_fts{close} USING FTS4 ("qAddress", "qCaretaker", "qSpecies", content={open}Street_Tree_List{close});
     CREATE VIRTUAL TABLE r USING rtree(a, b, c);
-    """
+    """.format(
+        open=open_quote, close=close_quote
+    )
     conn = utils.sqlite3.connect(":memory:")
     conn.executescript(sql)
     assert None is utils.detect_fts(conn, "Dumb_Table")
@@ -363,7 +370,7 @@ def test_table_columns():
     ],
 )
 def test_path_with_format(path, format, extra_qs, expected):
-    request = Request(path.encode("utf8"), {}, "1.1", "GET", None)
+    request = Request.fake(path)
     actual = utils.path_with_format(request, format, extra_qs)
     assert expected == actual
 
@@ -381,19 +388,3 @@ def test_path_with_format(path, format, extra_qs, expected):
 )
 def test_format_bytes(bytes, expected):
     assert expected == utils.format_bytes(bytes)
-
-
-@pytest.mark.parametrize(
-    "name,expected",
-    [
-        ("table", "table"),
-        ("table/and/slashes", "tableU+002FandU+002Fslashes"),
-        ("~table", "U+007Etable"),
-        ("+bobcats!", "U+002Bbobcats!"),
-        ("U+007Etable", "UU+002B007Etable"),
-    ],
-)
-def test_encode_decode_path_component(name, expected):
-    encoded = utils.encode_path_component(name)
-    assert encoded == expected
-    assert name == utils.decode_path_component(encoded)

versioneer.py

@@ -180,7 +180,7 @@ two common reasons why `setup.py` might not be in the root:
   `setup.cfg`, and `tox.ini`. Projects like these produce multiple PyPI
   distributions (and upload multiple independently-installable tarballs).
 * Source trees whose main purpose is to contain a C library, but which also
-  provide bindings to Python (and perhaps other langauges) in subdirectories.
+  provide bindings to Python (and perhaps other languages) in subdirectories.
 
 Versioneer will look for `.git` in parent directories, and most operations
 should get the right version string. However `pip` and `setuptools` have bugs
@@ -805,7 +805,7 @@ def render_pep440_old(pieces):
 
     The ".dev0" means dirty.
 
-    Eexceptions:
+    Exceptions:
     1: no tags. 0.postDISTANCE[.dev0]
     """
     if pieces["closest-tag"]:
@@ -1306,7 +1306,7 @@ def render_pep440_old(pieces):
 
     The ".dev0" means dirty.
 
-    Eexceptions:
+    Exceptions:
     1: no tags. 0.postDISTANCE[.dev0]
     """
     if pieces["closest-tag"]: