OpenTelemetry · Sentinel Docs

What Sentinel supports

OTLP over HTTP protobuf at /v1/traces
local trace ingestion during Sentinel ingest commands
metadata-only spans for HTTP, gRPC, and DB activity
event-detail correlation using sentinel.session_id

Sentinel does not store request bodies, response bodies, cookies, headers, or query strings.

Best path: `sentinel run` and `sentinel dev`

sentinel run -- <your command>

run is the easiest OTEL path for a single process because Sentinel can:

start a local OTLP receiver
inject exporter variables into the child process
append sentinel.session_id and sentinel.project into OTEL_RESOURCE_ATTRIBUTES

sentinel dev does the same injection for every command service in .sentinel.yml. So if your local stack is configured for sentinel dev and uses an OTel SDK that respects OTEL_* env vars, you do not need to set anything else: traces show up in sentinel traces and on the issue detail page automatically.

sentinel dev

The variables Sentinel sets for each command service:

OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_ENDPOINT=<local OTLP/HTTP receiver>
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=<endpoint>/v1/traces
OTEL_RESOURCE_ATTRIBUTES=...,sentinel.session_id=<id>,sentinel.project=<project>

If a variable is already set in the parent environment, Sentinel keeps your value (except for OTEL_RESOURCE_ATTRIBUTES, which is merged so sentinel.session_id and sentinel.project always make it through).

compose_service services are not instrumented for you. If you want their traces too, set the same variables in their Compose file.

Manual setup for `stdin` and `watch`

For sentinel stdin and sentinel watch, configure OTEL yourself and include the current Sentinel session id in resource attributes.

Minimal example:

export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318
export OTEL_RESOURCE_ATTRIBUTES="service.name=my-service,sentinel.session_id=<session>,sentinel.project=default"

Without sentinel.session_id, Sentinel may ingest spans but cannot link them exactly to occurrences.

Language examples

Python

sentinel run -- opentelemetry-instrument python app.py

Node.js

sentinel run -- node --require @opentelemetry/auto-instrumentations-node/register server.js

Java

sentinel run -- java \
  -javaagent:/path/to/opentelemetry-javaagent.jar \
  -Dotel.exporter.otlp.protocol=http/protobuf \
  -Dotel.exporter.otlp.endpoint=http://127.0.0.1:4318 \
  -jar app.jar

Runtime flags

Important OTEL-related global flags:

Flag	Purpose
`--otlp-enabled`	Enable or disable the local OTLP receiver
`--otlp-http-addr`	Change the OTLP listen address
`--otlp-correlation-lookback`	How far before an event to search for spans
`--otlp-correlation-lookahead`	How far after an event to search for spans
`--otlp-max-spans-per-event`	Cap spans returned on one event detail page
`--otlp-trace-slow-threshold-ms`	Always keep traces at or above this duration

Retention and drop defaults

Sentinel applies a few defaults to keep the local trace database manageable when an app emits a lot of low-value spans (health checks, metrics scrapes, internal HTTP send/receive plumbing):

Default	Value	What it does
`OTLPNoisyRoutePatterns`	`/health`, `/ready`, `/live`, `/metrics`, plus a few common polling patterns	Marks these routes as “noisy”.
`OTLPNoisyTraceSampleRatio`	`0.20`	Keep 20% of traces whose root path matches a noisy pattern (and which are not slow or errors).
`OTLPTraceSlowThresholdMS`	`100`	Always keep any trace that contains a span at or above this duration, even if it matches a noisy pattern.
`OTLPDropNoisyInternalHTTPSendReceive`	`true`	Drop very short internal `http send`/`http receive` spans on noisy routes.
`OTLPDropLowValueInternalHTTPSendReceive`	`true`	Drop the same kind of span across all routes when it adds no value.
`OTLPMaxSpansPerEvent`	`100`	Limit how many correlated spans the issue detail page returns for one event.

Each is a flag with the same name (kebab-cased) and can be tuned per command. To turn the noisy-route filter off entirely:

sentinel dev \
  --otlp-noisy-trace-sample-ratio 1.0 \
  --otlp-drop-noisy-internal-http-send-receive=false \
  --otlp-drop-low-value-internal-http-send-receive=false

For the full flag list with defaults, see the CLI reference.

Troubleshooting

No spans in the UI: Check http/protobuf, the OTLP endpoint, and sentinel.session_id.
Spans appear late: Exporters often batch. Refresh the event detail after a few seconds.
Port 4318 already in use: Run Sentinel with --otlp-http-addr 127.0.0.1:14318 and point your exporter there.
Health-check or metrics traces missing: That is the noisy-route filter. Lower the threshold or raise the sample ratio (see above).