Extend the kit with metric-side conformance: kind, unit and enum checks

The five surveyed plugin plans all kept a hand-rolled metrics-vs-registry
diff because the kit's conformance helpers covered spans only. This adds
the metric side:

- metric_for() in the registry (the span_for analogue - no prefix/dynamic
  machinery, metric names are static), and the attribute helpers are
  documented as accepting MetricName entries.
- MetricsCollector.collect() now retains the instrumentation scope per
  collected metric, so a plugin is judged against its own meter only.
- assert_metrics_conform(): every collected metric in scope is registered,
  was created as the instrument kind and unit its registry entry declares
  (drift between the registry entry and the meter.create_*() call was
  previously caught by nothing, in core or any plugin), sets only
  registered attributes, and respects values= enums - the check that makes
  a metric dimension provably bounded.
- assert_metrics_covered(): every registered metric collected at least
  once with every non-optional attribute seen. Both *_covered helpers now
  exempt optional=True attributes, so a workload is not forced to
  manufacture every error path; pin those with targeted tests instead.
- datasette.operation declares values={"read", "write"} - core dogfoods
  the enum enforcement on the dimension where it matters most.
- Core's generic metric conformance tests are now calls to the kit
  helpers with scope_name="datasette"; the stricter literal-pinning and
  optional-attribute-coverage tests stay hand-written on purpose.
- The metric reference docs render attributes through the same helper as
  spans, so *(optional)* markers and enum values now appear there too.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012U7coQfVu8nK2R4q2mCULA
This commit is contained in:
Alex Garcia 2026-09-02 12:55:47 -07:00
commit 2a05263ab8
8 changed files with 304 additions and 59 deletions

View file

@ -163,27 +163,35 @@ Wire your registry to reality with the conformance helpers - the two directions
.. code-block:: python
from datasette.telemetry_testing import (
assert_metrics_conform,
assert_metrics_covered,
assert_package_never_imports_sdk,
assert_registry_covered,
assert_spans_conform,
)
from my_plugin.telemetry import SPANS
from my_plugin.telemetry import METRICS, SPANS
def test_conformance(otel_spans):
def test_conformance(otel_spans, otel_metrics):
run_a_workload_that_exercises_everything()
finished = otel_spans.get_finished_spans()
# Everything emitted is registered (and enum values are legal):
assert_spans_conform(SPANS, finished, scope_name="my-plugin")
# Everything registered was emitted:
assert_registry_covered(SPANS, finished, scope_name="my-plugin")
# Same two directions for metrics - one collect() after the workload:
otel_metrics.collect()
assert_metrics_conform(METRICS, otel_metrics, scope_name="my-plugin")
assert_metrics_covered(METRICS, otel_metrics, scope_name="my-plugin")
def test_api_only_dependency():
assert_package_never_imports_sdk("my_plugin")
Always pass ``scope_name`` - the exporter also holds core's spans, and your registry should only be judged against your own.
Always pass ``scope_name`` - the exporter and reader also hold core's signals, and your registry should only be judged against your own.
The metric helpers check more than names: ``assert_metrics_conform`` asserts each instrument was created as the **kind** and **unit** its registry entry declares (the registry entry and the ``meter.create_*()`` call are separate statements, and a dashboard built on the registry's word breaks silently if they drift), and that every value on a ``values=`` enum attribute is a member - which is what makes a metric dimension *provably* bounded rather than bounded by intent. Both ``*_covered`` helpers exempt attributes marked ``optional=True`` (an ``error.type`` only present on failures should not force your workload to manufacture errors - pin those with targeted tests instead), and the metrics reader uses delta temporality, so run one broad workload followed by a single ``collect()``.
.. _plugin_telemetry_caveats: