remote/datasette
1
0
Fork 0
mirror of https://github.com/simonw/datasette.git synced 2026-09-24 19:04:08 +02:00
Code Issues Projects Releases Packages Wiki Activity

Remove the hand-rolled tracer now that OpenTelemetry covers the same ground

Browse source 
Datasette had two tracing systems since the OpenTelemetry spans landed. The
hand-rolled one measures the wrong thing - issue 1730, open since 2022, is
about exactly that - and it cannot be rebuilt on top of the new spans without
core owning a TracerProvider, which is the one thing the OTel design refuses
to do. Rather than carry duplicate instrumentation on the db.execute() hot
path indefinitely, the old system goes.

Deleted: datasette/tracer.py, the trace_debug setting, the AsgiTracer
response-rewriting middleware and the ?_trace=1 query-string argument.

- datasette/database.py: the four `with trace(...)` wrappers PR 1 deliberately
  nested the OTel spans inside are removed and the bodies dedented. That also
  retires the `# noqa: SIM117` comments those wrappers required - a leftover
  unnecessary noqa trips ruff's RUF100 - and `kwargs["count"] = count` in
  execute_write_many, which fed the old tracer only. `git diff -w` on this file
  shows nothing but the deleted lines.
- datasette/views/base.py: stream_csv() still read ?_trace=1 to wrap CSV output
  in an HTML <textarea> debug page. That whole branch, including the
  EscapeHtmlWriter selection and the conditional content-type, is gone. The
  EscapeHtmlWriter class itself stays in datasette.utils - it is an importable
  public name and removing it would widen the API break.
- .github/workflows/deploy-latest.yml no longer passes --setting trace_debug 1.
  Worth stating precisely, because the ticket claimed otherwise: this would not
  have broken the deploy. Setting.convert() in cli.py only rewrites a bare name
  to settings.<name> for *known* settings, so `--setting trace_debug 1` would
  have been silently accepted as a meaningless top-level config key. The flag is
  removed because it is dead, not because it errors.

Tests. tests/test_tracer.py is deleted outright (6 items). Four other tests used
?_trace=1 as an assertion instrument rather than testing tracing:

- test_csv_trace tested the trace mechanism itself - deleted.
- test_table_csv_stream_does_not_calculate_facets,
  test_table_csv_stream_does_not_calculate_counts and
  test_nocount_nofacet_if_shape_is_object test real behaviour, and are rebuilt
  against captured spans. All three had silently stopped being able to fail: the
  facets test looked for "select content, count(*) as n", which facet suggestion
  has not emitted since it moved to a `with limited as (...)` CTE, and none of
  the three requested the count or facet work whose suppression they claim to
  check. The rebuilt versions ask for it explicitly, match strings the current
  SQL contains, and carry a guard assertion so an empty span list cannot
  masquerade as a pass. Each was confirmed to fail with the covered code broken.
- test_trace_correctly_escaped is kept, renamed test_query_page_escapes_sql,
  with ?_trace=1 dropped. It ran against ds_client, which has no trace_debug, so
  it never exercised the tracer - what it actually covered is the query page
  echoing user SQL into HTML, the surface of the two reflected-XSS advisories in
  issue 1360, and nothing else in the suite covers it. Deleting it would have
  quietly dropped that.

tests/test_utils.py's pairs_to_nested_config case used settings.trace_debug to
check that a later key overrides an earlier one; it now uses template_debug
rather than losing the case.

Docs: the datasette.tracer section of internals.rst, the trace_debug section of
settings.rst, the ?_trace=1 entries in json_api.rst and introspection.rst, and
the regenerated cli-reference.rst. changelog.rst gets a breaking-change entry
and keeps all its historical ?_trace=1 entries - two of them had to lose a
:ref: role pointing at a label this commit deletes, or Sphinx warns on every
build.

2368 passed, 39 skipped, 6 xfailed, 15 xpassed, 140 subtests, against 2375 /
141 before. Net -7 tests, fully accounted for: -6 test_tracer.py, -1
test_csv_trace, -1 test_trace_correctly_escaped, +1 test_query_page_escapes_sql.
The lost subtest is the per-setting case trace_debug generated in
test_settings_are_documented.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Alex Garcia 2026-07-30 20:14:08 -07:00
commit 745b7872b8
21 changed files with 170 additions and 557 deletions
Download patch file Download diff file Expand all files Collapse all files

7
docs/changelog.rst
View file

@ -11,8 +11,7 @@ Unreleased
- Datasette's database layer now emits `OpenTelemetry <https://opentelemetry.io/>`__ spans: one per query, covering the full round trip including time spent waiting for a SQL worker thread, plus separate child spans for the execution itself and for time spent in the write queue. Datasette core depends on ``opentelemetry-api`` only and never installs an SDK provider, an exporter or a sampler, so there is no effect and no measurable overhead unless tracing is switched on externally - normally with the standard ``opentelemetry-instrument`` agent. See :ref:`internals_telemetry`. (:issue:`1730`)
- :ref:`db.execute(sql, ..., table=None) <database_execute>` has a new optional ``table=`` parameter, naming the table a query is about so it can be recorded on that query's OpenTelemetry span. It has no effect on query execution, and Datasette never derives it from the SQL. (:issue:`1730`)
Nothing is removed by this change: the ``?_trace=1`` query string parameter, the ``trace_debug`` setting and the :ref:`internals_tracer` module all continue to work as before.
- **Breaking change:** Datasette's hand-rolled tracer has been removed, now that OpenTelemetry covers the same ground. The ``?_trace=1`` query string parameter, the ``trace_debug`` setting and the ``datasette.tracer`` module are all gone. ``datasette.tracer.trace()`` and ``datasette.tracer.trace_child_tasks()`` were documented plugin APIs, so any plugin importing them will now raise ``ModuleNotFoundError`` and needs a new release. `datasette-pretty-traces <https://datasette.io/plugins/datasette-pretty-traces>`__ does not import that module, but it renders ``?_trace=1`` output, so it no longer has anything to display. (:issue:`1730`)
.. _v1_0_a38:
@ -1161,7 +1160,7 @@ Datasette also now requires Python 3.7 or higher.
- ``sqlite_stat`` tables are now hidden by default. (:issue:`1587`)
- SpatiaLite tables ``data_licenses``, ``KNN`` and ``KNN2`` are now hidden by default. (:issue:`1601`)
- SQL query tracing mechanism now works for queries executed in ``asyncio`` sub-tasks, such as those created by ``asyncio.gather()``. (:issue:`1576`)
- :ref:`internals_tracer` mechanism is now documented.
- ``datasette.tracer`` mechanism is now documented.
- Common Datasette symbols can now be imported directly from the top-level ``datasette`` package, see :ref:`internals_shortcuts`. Those symbols are ``Response``, ``Forbidden``, ``NotFound``, ``hookimpl``, ``actor_matches_allow``. (:issue:`957`)
- ``/-/versions`` page now returns additional details for libraries used by SpatiaLite. (:issue:`1607`)
- Documentation now links to the `Datasette Tutorials <https://datasette.io/tutorials>`__.
@ -1327,7 +1326,7 @@ New features
- ``?_facet_size=max`` sets that to the maximum, which defaults to 1,000 and is controlled by the the :ref:`setting_max_returned_rows` setting. If facet results are truncated the … at the bottom of the facet list now links to this parameter. (:issue:`1337`)
- ``?_nofacet=1`` option to disable all facet calculations on a page, used as a performance optimization for CSV exports and ``?_shape=array/object``. (:issue:`1349`, :issue:`263`)
- ``?_nocount=1`` option to disable full query result counts. (:issue:`1353`)
- ``?_trace=1`` debugging option is now controlled by the new :ref:`setting_trace_debug` setting, which is turned off by default. (:issue:`1359`)
- ``?_trace=1`` debugging option is now controlled by the new ``trace_debug`` setting, which is turned off by default. (:issue:`1359`)
Bug fixes and other improvements
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

2
docs/cli-reference.rst
View file

@ -283,8 +283,6 @@ These can be passed to ``datasette serve`` using ``datasette serve --setting nam
protocol (default=False)
template_debug Allow display of template debug information with
?_context=1 (default=False)
trace_debug Allow display of SQL trace debug information with
?_trace=1 (default=False)
base_url Datasette URLs should use this base path
(default=/)

80
docs/internals.rst
View file

@ -2327,8 +2327,6 @@ Turning tracing on is entirely an operational decision made outside of Datasette
Everything Datasette emits carries the instrumentation scope ``datasette``, versioned with the running Datasette version and declaring the `semantic conventions schema <https://opentelemetry.io/docs/specs/otel/schemas/>`__ its attribute names follow.
This is separate from, and does not replace, the built-in :ref:`internals_tracer` mechanism behind ``?_trace=1`` and the :ref:`setting_trace_debug` setting. Both continue to work exactly as before.
.. _internals_telemetry_turning_on:
Turning tracing on
@ -2773,8 +2771,6 @@ Async version of :ref:`call_with_supported_arguments <internals_utils_call_with_
.. autofunction:: datasette.utils.async_call_with_supported_arguments
.. _internals_tracer:
JSON encoding
-------------
@ -2782,82 +2778,6 @@ JSON encoding
.. autoclass:: datasette.utils.CustomJSONEncoder
datasette.tracer
================
Running Datasette with ``--setting trace_debug 1`` enables trace debug output, which can then be viewed by adding ``?_trace=1`` to the query string for any page.
You can see an example of this at the bottom of `latest.datasette.io/fixtures/facetable?_trace=1 <https://latest.datasette.io/fixtures/facetable?_trace=1>`__. The JSON output shows full details of every SQL query that was executed to generate the page.
The `datasette-pretty-traces <https://datasette.io/plugins/datasette-pretty-traces>`__ plugin can be installed to provide a more readable display of this information. You can see `a demo of that here <https://latest-with-plugins.datasette.io/github/commits?_trace=1>`__.
You can add your own custom traces to the JSON output using the ``trace()`` context manager. This takes a string that identifies the type of trace being recorded, and records any keyword arguments as additional JSON keys on the resulting trace object.
The start and end time, duration and a traceback of where the trace was executed will be automatically attached to the JSON object.
This example uses trace to record the start, end and duration of any HTTP GET requests made using the function:
.. code-block:: python
from datasette.tracer import trace
import httpx
async def fetch_url(url):
with trace("fetch-url", url=url):
async with httpx.AsyncClient() as client:
return await client.get(url)
.. _internals_tracer_trace_child_tasks:
Tracing child tasks
-------------------
If your code uses a mechanism such as ``asyncio.gather()`` to execute code in additional tasks you may find that some of the traces are missing from the display.
You can use the ``trace_child_tasks()`` context manager to ensure these child tasks are correctly handled.
.. code-block:: python
from datasette import tracer
with tracer.trace_child_tasks():
results = await asyncio.gather(
# ... async tasks here
)
This example uses the :ref:`register_routes() <plugin_register_routes>` plugin hook to add a page at ``/parallel-queries`` which executes two SQL queries in parallel using ``asyncio.gather()`` and returns their results.
.. code-block:: python
from datasette import hookimpl
from datasette import tracer
@hookimpl
def register_routes():
async def parallel_queries(datasette):
db = datasette.get_database()
with tracer.trace_child_tasks():
one, two = await asyncio.gather(
db.execute("select 1"),
db.execute("select 2"),
)
return Response.json(
{
"one": one.single_value(),
"two": two.single_value(),
}
)
return [
(r"/parallel-queries$", parallel_queries),
]
Note that running parallel SQL queries in this way has `been known to cause problems in the past <https://github.com/simonw/datasette/issues/2189>`__, so treat this example with caution.
Adding ``?_trace=1`` will show that the trace covers both of those child tasks.
.. _internals_shortcuts:
Import shortcuts

1
docs/introspection.rst
View file

@ -124,7 +124,6 @@ Shows the :ref:`configuration <configuration>` for this instance of Datasette. T
"ok": true,
"settings": {
"template_debug": true,
"trace_debug": true,
"force_https_urls": true
}
}

10
docs/json_api.rst
View file

@ -313,16 +313,6 @@ query string arguments:
For how many seconds should this response be cached by HTTP proxies? Use
``?_ttl=0`` to disable HTTP caching entirely for this request.
``?_trace=1``
Turns on tracing for this page: SQL queries executed during the request will
be gathered and included in the response, either in a new ``"_traces"`` key
for JSON responses or at the bottom of the page if the response is in HTML.
The structure of the data returned here should be considered highly unstable
and very likely to change.
Only available if the :ref:`setting_trace_debug` setting is enabled.
.. _json_api_extra:
Expanding JSON responses

18
docs/settings.rst
View file

@ -335,24 +335,6 @@ Some examples:
* https://latest.datasette.io/fixtures?_context=1
* https://latest.datasette.io/fixtures/roadside_attractions?_context=1
.. _setting_trace_debug:
trace_debug
~~~~~~~~~~~
This setting enables appending ``?_trace=1`` to any page in order to see the SQL queries and other trace information that was used to generate that page.
Enable it like this::
datasette mydatabase.db --setting trace_debug 1
Some examples:
* https://latest.datasette.io/?_trace=1
* https://latest.datasette.io/fixtures/roadside_attractions?_trace=1
See :ref:`internals_tracer` for details on how to hook into this mechanism as a plugin author.
.. _setting_base_url:
base_url