.. _contributing: Contributing ============ Datasette is an open source project. We welcome contributions! This document describes how to contribute to Datasette core. You can also contribute to the wider Datasette ecosystem by creating new :ref:`plugins`. General guidelines ------------------ * **main should always be releasable**. Incomplete features should live in branches. This ensures that any small bug fixes can be quickly released. * **The ideal commit** should bundle together the implementation, unit tests and associated documentation updates. The commit message should link to an associated issue. * **New plugin hooks** should only be shipped if accompanied by a separate release of a non-demo plugin that uses them. * **New user-facing views and documentation** should be added or updated alongside their implementation. The `/docs` folder includes pages for plugin hooks and built-in views—please ensure any new hooks or views are reflected there so the documentation tests continue to pass. .. _devenvironment: Setting up a development environment ------------------------------------ If you have Python 3.10 or higher installed on your computer (on OS X the quickest way to do this `is using homebrew `__) you can install an editable copy of Datasette using the following steps. If you want to use GitHub to publish your changes, first `create a fork of datasette `__ under your own GitHub account. Now clone that repository somewhere on your computer:: git clone git@github.com:YOURNAME/datasette If you want to get started without creating your own fork, you can do this instead:: git clone git@github.com:simonw/datasette The quickest way to set up a development environment is to use `uv `__. From the repository root you can run the tests directly:: cd datasette uv run pytest This will create a local ``.venv/`` and install Datasette plus its development dependencies. If you prefer to manage your own virtual environment with pip, create and activate one and then install the development dependency group:: python3 -m venv ./venv source venv/bin/activate python3 -m pip install -e . --group dev .. _contributing_running_tests: Running the tests ----------------- Once you have done this, you can run the Datasette unit tests from inside your ``datasette/`` directory using `pytest `__ like so:: uv run pytest You can run the tests faster using multiple CPU cores with `pytest-xdist `__ like this:: uv run pytest -n auto -m "not serial" ``-n auto`` detects the number of available cores automatically. The ``-m "not serial"`` skips tests that don't work well in a parallel test environment. You can run those tests separately like so:: uv run pytest -m "serial" .. _contributing_playwright: Running Playwright tests ~~~~~~~~~~~~~~~~~~~~~~~~ Datasette includes a small number of browser automation tests using Playwright_. These tests are skipped by default, so you can run the main test suite with ``uv run pytest`` without installing Playwright or any browser binaries. .. _Playwright: https://playwright.dev/python/ The Playwright tests use a separate dependency group. The easiest way to run them is using ``just``. First install the browser engine you want to test against. Chromium is used by default: .. code-block:: bash just playwright-install Then run the Playwright test module: .. code-block:: bash just playwright You can also run the same tests against Firefox or WebKit by installing that browser engine and passing it to ``just playwright``: .. code-block:: bash just playwright-install firefox just playwright firefox just playwright-install webkit just playwright webkit To install every supported browser engine and run the tests against all of them, use: .. code-block:: bash just playwright-install-all just playwright-all You can pass extra ``pytest`` options after the browser name: .. code-block:: bash just playwright chromium -k permissions just playwright-all -x You can add the ``--headed`` option to have Playwright open a browser window that you can see while it runs the tests. This only works if you specify a browser, for example: .. code-block:: bash just playwright firefox --headed Combine this with ``-k`` to watch a specific test: .. code-block:: bash just playwright chromium --headed -k test_insert_row If you are not using ``just``, the equivalent ``uv run`` commands are: .. code-block:: bash uv run --group playwright playwright install chromium uv run --group playwright pytest tests/test_playwright.py --playwright --browser chromium .. _contributing_modals: Modal dialogs ------------- Core dialogs use the same ```` component available to plugins. See :ref:`javascript_plugins_modals` for examples, lifecycle methods, dismissal guards and shared styles. The implementation lives in ``datasette/static/modal.js`` and ``datasette/static/modal.css``. The wrapper keeps each native ```` and its content in the caller's DOM tree, preserving form associations, accessible labels and plugin controls. Components such as ```` use the same wrapper and stylesheet inside their shadow roots. Keep focus restoration, backdrop hit testing, busy-state dismissal guards and the Safari Escape/confirmation workaround in the shared component. Each consumer owns its content, submission logic, discard-confirmation policy and cleanup. In particular, preserve the intentional differences between Cancel and Escape in the editing dialogs. Add lifecycle coverage to ``tests/test_playwright.py`` when changing the shared component. Exercise both light DOM and shadow roots, focus restoration, busy state, nested controls consuming Escape, backdrop clicks and disconnect cleanup. Run these checks in Chromium, Firefox and WebKit; keyboard changes should include real confirmation prompts in WebKit. .. _contributing_using_fixtures: Using fixtures -------------- To run Datasette itself, type ``datasette``. You're going to need at least one SQLite database. A quick way to get started is to use the fixtures database that Datasette uses for its own tests. You can create a copy of that database by running this command:: uv run python tests/fixtures.py fixtures.db Now you can run Datasette against the new fixtures database like so:: uv run datasette fixtures.db This will start a server at ``http://127.0.0.1:8001/``. Any changes you make in the ``datasette/templates`` or ``datasette/static`` folder will be picked up immediately (though you may need to do a force-refresh in your browser to see changes to CSS or JavaScript). If you want to change Datasette's Python code you can use the ``--reload`` option to cause Datasette to automatically reload any time the underlying code changes:: uv run datasette --reload fixtures.db This also enables development mode for static asset cache busting, described in :ref:`customization_static_files`. You can also use the ``fixtures.py`` script to recreate the testing version of ``metadata.json`` used by the unit tests. To do that:: uv run python tests/fixtures.py fixtures.db fixtures-metadata.json Or to output the plugins used by the tests, run this:: uv run python tests/fixtures.py fixtures.db fixtures-metadata.json fixtures-plugins Test tables written to fixtures.db - metadata written to fixtures-metadata.json Wrote plugin: fixtures-plugins/register_output_renderer.py Wrote plugin: fixtures-plugins/view_name.py Wrote plugin: fixtures-plugins/my_plugin.py Wrote plugin: fixtures-plugins/messages_output_renderer.py Wrote plugin: fixtures-plugins/my_plugin_2.py Then run Datasette like this:: uv run datasette fixtures.db -m fixtures-metadata.json --plugins-dir=fixtures-plugins/ .. _contributing_debugging: Debugging --------- Any errors that occur while Datasette is running while display a stack trace on the console. You can tell Datasette to open an interactive ``pdb`` (or ``ipdb``, if present) debugger session if an error occurs using the ``--pdb`` option:: uv run datasette --pdb fixtures.db For `ipdb `__, first run this:: uv run datasette install ipdb .. _contributing_formatting: Code formatting --------------- Datasette uses opinionated code formatters: `Black `__ for Python and `Prettier `__ for JavaScript. These formatters are enforced by Datasette's continuous integration: if a commit includes Python or JavaScript code that does not match the style enforced by those tools, the tests will fail. When developing locally, you can verify and correct the formatting of your code using these tools. If you are using `Just `__ the quickest way to run these is like so:: just black just prettier Or run both at the same time:: just format .. _contributing_formatting_black: Running Black ~~~~~~~~~~~~~ Black is installed as part of the development dependency group. To test that your code complies with Black, run the following in your root ``datasette`` repository checkout:: uv run black . --check :: All done! ✨ 🍰 ✨ 95 files would be left unchanged. If any of your code does not conform to Black you can run this to automatically fix those problems:: uv run black . :: reformatted ../datasette/app.py All done! ✨ 🍰 ✨ 1 file reformatted, 94 files left unchanged. .. _contributing_formatting_blacken_docs: blacken-docs ~~~~~~~~~~~~ The `blacken-docs `__ command applies Black formatting rules to code examples in the documentation. Run it like this:: uv run blacken-docs -l 60 docs/*.rst .. _contributing_formatting_prettier: Prettier ~~~~~~~~ To install Prettier, `install Node.js `__ and then run the following in the root of your ``datasette`` repository checkout:: npm install This will install Prettier in a ``node_modules`` directory. You can then check that your code matches the coding style like so:: npm run prettier -- --check :: > prettier > prettier 'datasette/static/*[!.min].js' "--check" Checking formatting... [warn] datasette/static/plugins.js [warn] Code style issues found in the above file(s). Forgot to run Prettier? You can fix any problems by running:: npm run fix .. _contributing_documentation: Editing and building the documentation -------------------------------------- Datasette's documentation lives in the ``docs/`` directory and is deployed automatically using `Read The Docs `__. The documentation is written using reStructuredText. You may find this article on `The subset of reStructuredText worth committing to memory `__ useful. You can build it locally once you have installed the development dependency group (which includes Sphinx and related tools) and then running ``make html`` directly in the ``docs/`` directory:: cd docs/ uv run make html This will create the HTML version of the documentation in ``docs/_build/html``. You can open it in your browser like so:: open _build/html/index.html Any time you make changes to a ``.rst`` file you can re-run ``make html`` to update the built documents, then refresh them in your browser. For added productivity, you can use use `sphinx-autobuild `__ to run Sphinx in auto-build mode. This will run a local webserver serving the docs that automatically rebuilds them and refreshes the page any time you hit save in your editor. ``sphinx-autobuild`` is included in the development dependency group. In your ``docs/`` directory you can start the server by running the following:: uv run make livehtml Now browse to ``http://localhost:8000/`` to view the documentation. Any edits you make should be instantly reflected in your browser. .. _contributing_documentation_cog: Running Cog ~~~~~~~~~~~ Some pages of documentation (in particular the :ref:`cli_reference`) are automatically updated using `Cog `__. To update these pages, run the following command:: uv run cog -r docs/*.rst .. _contributing_template_contexts: Documented template contexts ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Datasette's documented template contexts are part of the public API for custom templates. They are defined as dataclasses next to the view code that renders them, for example ``DatabaseContext`` and ``QueryContext`` in ``datasette/views/database.py``. Every documented context class inherits from ``datasette.views.Context``. Fields that are added directly by view code should be declared as dataclass fields with ``help`` metadata, which is used to generate :ref:`template_context`. Fields resolved through the page extras system should use ``from_extra()`` so their documentation comes from the matching ``Extra`` class. Use ``documented_template`` on each context class to record the canonical template named in the generated documentation. This should be a string such as ``"database.html"``. Runtime template selection still happens in the view code, since most pages consider more specific template names before falling back to the canonical one. When a context field contains repeated structured data, prefer a small nested dataclass over an anonymous dictionary. For example, a field containing table summaries should be annotated as ``list[DatabaseTable]`` where ``DatabaseTable`` is a dataclass describing the keys and value types. This keeps the Python contract and generated documentation clear. JSON responses and ``?_context=1`` debug output will convert nested dataclasses back to JSON objects at the response boundary. .. _contributing_continuous_deployment: Continuously deployed demo instances ------------------------------------ The demo instance at `latest.datasette.io `__ is re-deployed automatically to Google Cloud Run for every push to ``main`` that passes the test suite. This is implemented by the GitHub Actions workflow at `.github/workflows/deploy-latest.yml `__. Specific branches can also be set to automatically deploy by adding them to the ``on: push: branches`` block at the top of the workflow YAML file. Branches configured in this way will be deployed to a new Cloud Run service whether or not their tests pass. The Cloud Run URL for a branch demo can be found in the GitHub Actions logs. .. _contributing_release: Release process --------------- Datasette releases are performed using tags. When a new release is published on GitHub, a `GitHub Action workflow `__ will perform the following: * Run the unit tests against all supported Python versions. If the tests pass... * Build a Docker image of the release and push a tag to https://hub.docker.com/r/datasetteproject/datasette * Re-point the "latest" tag on Docker Hub to the new image * Build a wheel bundle of the underlying Python source code * Push that new wheel up to PyPI: https://pypi.org/project/datasette/ * If the release is an alpha, navigate to https://readthedocs.org/projects/datasette/versions/ and search for the tag name in the "Activate a version" filter, then mark that version as "active" to ensure it will appear on the public ReadTheDocs documentation site. To deploy new releases you will need to have push access to the main Datasette GitHub repository. Datasette follows `Semantic Versioning `__:: major.minor.patch We increment ``major`` for backwards-incompatible releases. Datasette is currently pre-1.0 so the major version is always ``0``. We increment ``minor`` for new features. We increment ``patch`` for bugfix releass. :ref:`contributing_alpha_beta` may have an additional ``a0`` or ``b0`` prefix - the integer component will be incremented with each subsequent alpha or beta. To release a new version, first create a commit that updates the version number in ``datasette/version.py`` and the :ref:`the changelog ` with highlights of the new version. An example `commit can be seen here `__:: # Update changelog git commit -m " Release 0.51a1 Refs #1056, #1039, #998, #1045, #1033, #1036, #1034, #976, #1057, #1058, #1053, #1064, #1066" -a git push Referencing the issues that are part of the release in the commit message ensures the name of the release shows up on those issue pages, e.g. `here `__. You can generate the list of issue references for a specific release by copying and pasting text from the release notes or GitHub changes-since-last-release view into this `Extract issue numbers from pasted text `__ tool. To create the tag for the release, create `a new release `__ on GitHub matching the new version number. You can convert the release notes to Markdown by copying and pasting the rendered HTML into this `Paste to Markdown tool `__. Don't forget to create the release from the correct branch - usually ``main``, but sometimes ``0.64.x`` or similar for a bugfix release. While the release is running you can confirm that the correct commits made it into the release using the https://github.com/simonw/datasette/compare/0.64.6...0.64.7 URL. Finally, post a news item about the release on `datasette.io `__ by editing the `news.yaml `__ file in that site's repository. Use `this preview tool `__ to preview the edits to the YAML. .. _contributing_alpha_beta: Alpha and beta releases ----------------------- Alpha and beta releases are published to preview upcoming features that may not yet be stable - in particular to preview new plugin hooks. You are welcome to try these out, but please be aware that details may change before the final release. Please join `discussions on the issue tracker `__ to share your thoughts and experiences with on alpha and beta features that you try out. .. _contributing_bug_fix_branch: Releasing bug fixes from a branch --------------------------------- If it's necessary to publish a bug fix release without shipping new features that have landed on ``main`` a release branch can be used. Create it from the relevant last tagged release like so:: git branch 0.52.x 0.52.4 git checkout 0.52.x Next cherry-pick the commits containing the bug fixes:: git cherry-pick COMMIT Write the release notes in the branch, and update the version number in ``version.py``. Then push the branch:: git push -u origin 0.52.x Once the tests have completed, publish the release from that branch target using the GitHub `Draft a new release `__ form. Finally, cherry-pick the commit with the release notes and version number bump across to ``main``:: git checkout main git cherry-pick COMMIT git push .. _contributing_upgrading_codemirror: Upgrading CodeMirror -------------------- Datasette bundles `CodeMirror `__ for the SQL editing interface, e.g. on `this page `__. Here are the steps for upgrading to a new version of CodeMirror: * Install the packages with:: npm i codemirror @codemirror/lang-sql * Build the bundle using the version number from package.json with:: node_modules/.bin/rollup datasette/static/cm-editor-6.0.1.js \ -f iife \ -n cm \ -o datasette/static/cm-editor-6.0.1.bundle.js \ -p @rollup/plugin-node-resolve \ -p @rollup/plugin-terser * Update the version reference in the ``codemirror.html`` template.