Use pytest-xdist to speed up tests (#1290)

* Run tests in CI using pytest-xdist
* Documentation for pytest-xdist

Closes #1289

.github/workflows/test.yml

@@ -26,4 +26,5 @@ jobs:
         pip install -e '.[test]'
     - name: Run tests
       run: |
-        pytest
+        pytest -n auto -m "not serial"
+        pytest -m "serial"

docs/contributing.rst

@@ -43,10 +43,28 @@ The next step is to create a virtual environment for your project and use it to
 
 That last line does most of the work: ``pip install -e`` means "install this package in a way that allows me to edit the source code in place". The ``.[test]`` option means "use the setup.py in this directory and install the optional testing dependencies as well".
 
+.. _contributing_running_tests:
+
+Running the tests
+-----------------
+
 Once you have done this, you can run the Datasette unit tests from inside your ``datasette/`` directory using `pytest <https://docs.pytest.org/>`__ like so::
 
     pytest
 
+You can run the tests faster using multiple CPU cores with `pytest-xdist <https://pypi.org/project/pytest-xdist/>`__ like this::
+
+    pytest -n auto -m "not serial"
+
+``-n auto`` detects the number of available cores automatically. The ``-m "not serial"`` skips tests that don't work well in a parallel test environment. You can run those tests separately like so::
+
+    pytest -m "serial"
+
+.. _contributing_using_fixtures:
+
+Using fixtures
+--------------
+
 To run Datasette itself, type ``datasette``.
 
 You're going to need at least one SQLite database. A quick way to get started is to use the fixtures database that Datasette uses for its own tests.

pytest.ini

@@ -6,3 +6,5 @@ filterwarnings=
     ignore:Using or importing the ABCs::bs4.element
     # Python 3.7 PendingDeprecationWarning: Task.current_task()
     ignore:.*current_task.*:PendingDeprecationWarning
+markers =
+    serial: tests to avoid using with pytest-xdist

setup.py

@@ -70,6 +70,7 @@ setup(
         "docs": ["sphinx_rtd_theme", "sphinx-autobuild"],
         "test": [
             "pytest>=5.2.2,<6.3.0",
+            "pytest-xdist>=2.2.1,<2.3",
             "pytest-asyncio>=0.10,<0.15",
             "beautifulsoup4>=4.8.1,<4.10.0",
             "black==20.8b1",

tests/test_cli_serve_server.py

@@ -1,6 +1,8 @@
 import httpx
+import pytest
 
 
+@pytest.mark.serial
 def test_serve_localhost_http(ds_localhost_http_server):
     response = httpx.get("http://localhost:8041/_memory.json")
     assert {
@@ -10,6 +12,7 @@ def test_serve_localhost_http(ds_localhost_http_server):
     }.items() <= response.json().items()
 
 
+@pytest.mark.serial
 def test_serve_localhost_https(ds_localhost_https_server):
     _, client_cert = ds_localhost_https_server
     response = httpx.get("https://localhost:8042/_memory.json", verify=client_cert)