§reference

Observability

What the framework reports through OpenTelemetry - spans, their attributes, and metrics - and what it logs through tracing. It depends on the OpenTelemetry API only, not the SDK: with no SDK installed, every span and metric call does nothing. Install one to collect them.

Installing an SDK

The ekbatan crate depends on opentelemetry (the API) and nothing else for telemetry. To collect spans and metrics, add opentelemetry_sdk and an exporter to your app, and install the providers globally, once, at start-up:

use opentelemetry::global;
use opentelemetry_sdk::metrics::{PeriodicReader, SdkMeterProvider};
use opentelemetry_sdk::trace::SdkTracerProvider;

let tracer_provider = SdkTracerProvider::builder()
    .with_simple_exporter(span_exporter)      // the exporter of your choice
    .build();
global::set_tracer_provider(tracer_provider.clone());

let meter_provider = SdkMeterProvider::builder()
    .with_reader(PeriodicReader::builder(metric_exporter).build())
    .build();
global::set_meter_provider(meter_provider.clone());

The framework asks the global providers each time it starts a span or records a metric. So a provider installed after the executor and the jobs were built is still the one used.

Spans are made with OpenTelemetry’s API directly, not as tracing spans, so their names and attributes don’t depend on a bridge.

Span tree

Each executor.execute(...) gives:

ekbatan.action.execute               the whole call
+-- ekbatan.action.perform           your perform(), once per attempt
+-- ekbatan.action.persist           the writes, once per attempt
    +-- ekbatan.transaction          one per shard
        +-- ekbatan.repository       one per staged type and kind (insert, update)
        +-- ekbatan.event.persist    the event rows

Outside an action:

Attributes

attributetypespanmeaning
ekbatan.action.namestringaction.execute, event.persistthe action’s type name: WalletDepositMoneyAction
ekbatan.action.principalstringaction.executethe principal; empty when None
ekbatan.action.outcomestringaction.executesuccess or error
ekbatan.action.retry.countintaction.executethe number of retries; 0 if none
ekbatan.shard.cross_shardboolaction.persistpresent, and true, when the changes span shards
ekbatan.shard.groupinttransactionthe shard’s group
ekbatan.shard.memberinttransactionthe shard’s member
db.operation.namestringrepositoryBATCH_INSERT or BATCH_UPDATE
ekbatan.entity.typestringrepositorythe model or entity’s type name: Wallet
ekbatan.batch.sizeintrepositoryrows in the statement
ekbatan.event.countintevent.persistevents written on this shard

db.operation.name is OpenTelemetry’s standard attribute name, with the framework’s own values. Every other attribute starts with ekbatan..

ekbatan.action.retry.count is set once, when the call ends, with the final count.

On a partial cross-shard commit

When a cross-shard action commits some shards and then fails on another, ekbatan.action.persist also gets:

attributemeaning
ekbatan.shard.partial_commit_failuretrue: this happened
ekbatan.shard.committed_shardsthe shards already committed, which won’t roll back, as group/member: [0/0]
ekbatan.shard.failed_shardthe shard that failed

The same facts are logged at ERROR, on a line starting CRITICAL:. They are set only when a shard’s write or commit returns an error; see ActionExecutor for what that misses.

The instrumentation scope

Spans are reported under the scope io.ekbatan.core, metrics under io.ekbatan.events.localeventhandler, both with the ekbatan crate’s version.

A service built on Ekbatan for Java reports the same span names, attribute names, metric names and scopes. One filter on ekbatan.action.name, one dashboard or one alert covers both services.

Errors

A failing span gets status ERROR, with the error’s text, and the error recorded as an exception event. An action that fails marks ekbatan.action.execute and the span where it failed (ekbatan.action.perform, or ekbatan.action.persist and the spans under it).

Context

Each span is a child of OpenTelemetry’s current context (opentelemetry::Context::current()) when it starts, and the framework makes its own spans current for the work they cover, across .awaits. To put an action’s spans under a span of yours, make yours current around the call: .with_context(cx) on the future (from opentelemetry::context::FutureExt), or cx.attach() in blocking code. With nothing current, ekbatan.action.execute starts a new trace.

Metrics

Metrics come from the local event handler only. Actions are traced, not counted.

instrumenttypeunitattributesfrom
ekbatan.events.fanned_outcounter{event}-fan-out job
ekbatan.events.notifications_createdcounter{notification}-fan-out job
ekbatan.events.handledcounter{notification}outcome, handlerhandling job
ekbatan.events.handler.durationhistogramshandler, outcomehandling job
ekbatan.events.delivery.laghistogramshandlerhandling job

outcome values

valuemeaning
succeededthe handler returned Ok
failed_retryit returned Err, or no handler has that name; a retry is scheduled, with backoff
expired_preflightthe row was already past the retention window when claimed; the handler never ran
expired_postfailureit failed, and the next retry would land past the retention window

handler.duration has a narrower outcome: succeeded or failed. When a handler returns, the only thing known is whether it failed. Whether that becomes a retry or an expiry is decided after.

What to alert on

Logs

The framework logs through the tracing crate. To see the lines, install a subscriber in your app, such as tracing-subscriber. Each line’s target is the module it comes from, so a filter on ekbatan covers them all (RUST_LOG=ekbatan=info with tracing-subscriber’s EnvFilter).

targetlogs
ekbatan::action::aio::executoreach action: executing, completed (INFO), retrying, cross-shard (WARN), failed, rejected cross-shard, CRITICAL: partial commit (ERROR); shards and row counts (DEBUG). Both executors log here
ekbatan::migrationeach database migrated, and a failed migration
ekbatan::distributed_jobs::registrythe job registry starting and stopping; each run (DEBUG); a failed run, a lost heartbeat, a revived dead run, a job abandoned at shutdown
ekbatan::local_event_handler::jobsa failed fan-out or handling round; a handler that failed or is missing (WARN); DISCARDED notifications (ERROR)
ekbatan::keyed_lock::aioa lock force-released because it was held past its max hold

The executor’s lines read like Java’s: Executing WalletDepositMoneyAction [principal=alice], WalletDepositMoneyAction completed in 12ms [principal=alice].

sqlx, underneath, logs under its own sqlx targets.

Not instrumented

See also