mirror of
https://github.com/simonw/datasette.git
synced 2026-09-22 18:04:07 +02:00
Extend the kit with metric-side conformance: kind, unit and enum checks
The five surveyed plugin plans all kept a hand-rolled metrics-vs-registry
diff because the kit's conformance helpers covered spans only. This adds
the metric side:
- metric_for() in the registry (the span_for analogue - no prefix/dynamic
machinery, metric names are static), and the attribute helpers are
documented as accepting MetricName entries.
- MetricsCollector.collect() now retains the instrumentation scope per
collected metric, so a plugin is judged against its own meter only.
- assert_metrics_conform(): every collected metric in scope is registered,
was created as the instrument kind and unit its registry entry declares
(drift between the registry entry and the meter.create_*() call was
previously caught by nothing, in core or any plugin), sets only
registered attributes, and respects values= enums - the check that makes
a metric dimension provably bounded.
- assert_metrics_covered(): every registered metric collected at least
once with every non-optional attribute seen. Both *_covered helpers now
exempt optional=True attributes, so a workload is not forced to
manufacture every error path; pin those with targeted tests instead.
- datasette.operation declares values={"read", "write"} - core dogfoods
the enum enforcement on the dimension where it matters most.
- Core's generic metric conformance tests are now calls to the kit
helpers with scope_name="datasette"; the stricter literal-pinning and
optional-attribute-coverage tests stay hand-written on purpose.
- The metric reference docs render attributes through the same helper as
spans, so *(optional)* markers and enum values now appear there too.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012U7coQfVu8nK2R4q2mCULA
This commit is contained in:
parent
f28db54eda
commit
2a05263ab8
8 changed files with 304 additions and 59 deletions
|
|
@ -163,27 +163,35 @@ Wire your registry to reality with the conformance helpers - the two directions
|
|||
.. code-block:: python
|
||||
|
||||
from datasette.telemetry_testing import (
|
||||
assert_metrics_conform,
|
||||
assert_metrics_covered,
|
||||
assert_package_never_imports_sdk,
|
||||
assert_registry_covered,
|
||||
assert_spans_conform,
|
||||
)
|
||||
|
||||
from my_plugin.telemetry import SPANS
|
||||
from my_plugin.telemetry import METRICS, SPANS
|
||||
|
||||
|
||||
def test_conformance(otel_spans):
|
||||
def test_conformance(otel_spans, otel_metrics):
|
||||
run_a_workload_that_exercises_everything()
|
||||
finished = otel_spans.get_finished_spans()
|
||||
# Everything emitted is registered (and enum values are legal):
|
||||
assert_spans_conform(SPANS, finished, scope_name="my-plugin")
|
||||
# Everything registered was emitted:
|
||||
assert_registry_covered(SPANS, finished, scope_name="my-plugin")
|
||||
# Same two directions for metrics - one collect() after the workload:
|
||||
otel_metrics.collect()
|
||||
assert_metrics_conform(METRICS, otel_metrics, scope_name="my-plugin")
|
||||
assert_metrics_covered(METRICS, otel_metrics, scope_name="my-plugin")
|
||||
|
||||
|
||||
def test_api_only_dependency():
|
||||
assert_package_never_imports_sdk("my_plugin")
|
||||
|
||||
Always pass ``scope_name`` - the exporter also holds core's spans, and your registry should only be judged against your own.
|
||||
Always pass ``scope_name`` - the exporter and reader also hold core's signals, and your registry should only be judged against your own.
|
||||
|
||||
The metric helpers check more than names: ``assert_metrics_conform`` asserts each instrument was created as the **kind** and **unit** its registry entry declares (the registry entry and the ``meter.create_*()`` call are separate statements, and a dashboard built on the registry's word breaks silently if they drift), and that every value on a ``values=`` enum attribute is a member - which is what makes a metric dimension *provably* bounded rather than bounded by intent. Both ``*_covered`` helpers exempt attributes marked ``optional=True`` (an ``error.type`` only present on failures should not force your workload to manufacture errors - pin those with targeted tests instead), and the metrics reader uses delta temporality, so run one broad workload followed by a single ``collect()``.
|
||||
|
||||
.. _plugin_telemetry_caveats:
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue