Observability

This page explains how Behemoth reports what it does: logs, audit events, metrics, and traces. It covers what each one is for, how to turn it on, and what to watch in production. Read How Behemoth fits together first.

Behemoth is quiet until you give it a destination. It writes to a logger, a metrics sink, and a tracer that you provide, and it stays silent when you leave them out. The audit log is the exception: it is on by default and writes to your database.

The four signals #

Each signal answers a different question, so use them together:

SignalIt answersGoes toDefault
LogsWhat happened in the application, and why did something fail?A logger you provideOff
Audit eventsWho did what to which account, and when?The audit_log tableOn
MetricsHow often does it happen, and how long does it take?A metrics sink you provideOff
TracesWhere did the time of one request go?A tracer you provideOff

A log line can be dropped, so use it to diagnose. An audit event is stored in your database and can be queried, so use it as a record of account activity.

Set up telemetry #

Build a Telemetry with telemetry.New and pass it to Boot. The three arguments are the logger, the audit recorder, and the metrics sink. A nil logger or metrics sink means none, and a nil audit recorder means the default table.

go
tel := telemetry.New(
    telemetry.NewJSONLogger(os.Stdout, slog.LevelInfo),
    nil, // audit: the audit_log table
    nil, // metrics: none
)

ac, err := engine.Boot(
    ctx,
    app,
    db,
    engine.BootConfig{
        Crypto:    cryptoCfg,
        Telemetry: tel,
    },
)
if err != nil {
    log.Fatal(err)
}

Inside your plugins and hook handlers, ac.Telemetry and all of its fields are never nil, so you need no checks.

Logs #

Behemoth writes through a logger interface with four levels. Use a ready-made logger for text or JSON on standard output, wrap your own *slog.Logger with telemetry.NewSlogLogger, or implement telemetry.Logger. To use zap or zerolog, pass their slog handler to NewSlogLogger.

Levels #

LevelWhen
ErrorBehemoth failed at something it should have done: a request answered with a 5xx, a hook handler that returned an error or panicked.
WarnBehemoth continued with something degraded: a failed cache write, an unreachable rate-limit store, or a configuration that is probably a mistake.
InfoLifecycle events, such as the boot summary.
DebugPer-request detail, such as a rejected request and SQL statements.

A rejected request is not an error. A wrong password or an expired token is logged at Debug only, so the Error level stays for problems on your side. Every line has a component field that names its source, such as router, session, ratelimit, or the name of a plugin.

Safe by default #

Whichever logger you pass, Behemoth wraps it first, so your backend receives lines that are already safe:

  • Redacted. The value of a field whose key is, or ends with, password, token, secret, hash, cookie, authorization, or email is replaced by [REDACTED]. By default, logs identify a user by ID only.
  • Correlated. A line written while handling a request has a request_id field. The same ID is in the X-Request-ID header of the response, so a user who reports a problem can quote it.

Redaction works on field keys. It does not inspect text inside a log message, so do not put a secret in a message. Pass telemetry.WithEmailsInLogs() to let email addresses through, and telemetry.WithRedactedKeys(...) to redact more keys.

Log from a plugin #

A plugin takes the logger from the AuthContext and names itself, so its lines can be told apart. telemetry.ErrorFields adds the error, its code, and its category.

go
func (p *Plugin) Init(ac *core.AuthContext) error {
    p.log = telemetry.Named(ac.Telemetry.Logger, "invites")
    return nil
}

func (p *Plugin) accept(rctx *httpapi.RequestContext) error {
    err := p.redeem(rctx.Ctx, code)
    if err != nil {
        p.log.Warn(
            rctx.Ctx,
            "invite could not be redeemed",
            telemetry.ErrorFields(err),
        )
        return err
    }
    return nil
}

The router logs an error that your route handler returns, so log only the errors you handle yourself.

Audit events #

The audit log records actions on accounts. It is on from the first start and stores each event in the audit_log table. Events include:

  • User creation, update, and deletion
  • Sign-up, sign-in, and sign-out, with failures
  • Session creation and revocation
  • Token issue, use, and refusal
  • The first rate-limit refusal in each window
  • Applied migrations and secret rotations

Each event records who acted (a user, a service, an anonymous caller, or named system code), who acted for them if anyone did, what the action was done to, the session and request involved, and the client's address and user agent. A failed sign-in names the account that was tried and an anonymous actor, because Behemoth does not know who typed the password.

Read events #

Behemoth has no HTTP route for the audit log. Query it from your code, and put your own access check in front of it:

go
page, err := ac.Store.QueryAuditEvents(
    ctx,
    telemetry.AuditFilter{
        SubjectType: "users",
        SubjectID:   userID,
        Limit:       50,
    },
)
if err != nil {
    return err
}
for _, e := range page.Events {
    fmt.Println(e.Timestamp, e.Type, e.Outcome)
}

Events come newest first. To read on, pass the page's NextCursor back as Cursor. An empty NextCursor means the last page.

Record your own events #

go
ac.Telemetry.RecordAudit(ctx, telemetry.AuditEvent{
    Type:        "billing.plan.changed",
    ActorID:     adminID,
    SubjectType: "users",
    SubjectID:   userID,
    Metadata:    behemoth.M{"from": "free", "to": "pro"},
})

What to expect #

  • User changes are exact. An event for a user created, updated, or deleted is written in the same database transaction as the change. The event exists exactly when the change does.
  • Other events are best effort. They are recorded after the action. If the write fails, the action still happened, and Behemoth logs the failure at Error.
  • Email addresses are kept. Audit metadata keeps the email tried in a sign-in, where logs redact it. Passwords, tokens, and other secrets are redacted in both.
  • Addresses and agents are recorded. Set BootConfig.Audit to core.AuditConfig{OmitIPAndAgent: true} to keep neither.
  • Behemoth never deletes events. Remove old ones from a scheduled job with ac.Store.PurgeAuditEvents. There is no way to change or delete a single event through Behemoth.
  • The table must exist first. User changes write to audit_log in their transaction, so apply your migrations before you serve requests.

To send events to another destination too, such as a security information system, pass telemetry.MultiRecorder(store.NewAuditRecorder(st), yours) as the audit recorder.

Metrics #

Behemoth counts and times its own work, such as requests, sign-ins, sessions, tokens, rate-limit checks, hook handlers, database operations, and mail and SMS sends. It hands the numbers to a sink with two methods, Counter and Histogram. The OpenTelemetry adapter below is a ready-made sink. For another library, such as the Prometheus client, implement the two methods yourself.

Durations are in seconds. Attribute values come from small fixed sets, such as a route pattern, a table name, or a failure code. They never hold a user ID, an IP address, or an email, so every attribute is safe to use as a label.

What to watch #

Start with these. The Telemetry reference lists every metric:

To seeWatch
A password-guessing runbehemoth.auth.sign_in with outcome="failure", split by reason. A rise in invalid_credentials for a steady success rate is the pattern.
Database troublebehemoth.store.errors with error_category="database". A not_found is a normal answer.
Rate limitingbehemoth.ratelimit.checks with result="limited", by rule. A result="error" means the counter store is down.
Silent mail or SMS failuresbehemoth.mail.sent and behemoth.sms.sent with result="error" or "dropped"
Password hashing capacitybehemoth.password.wait rising, or any behemoth.password.busy. Both are per process, so they signal that you need more replicas.
A slow hook handlerbehemoth.hook.duration by plugin. Data hooks run inside a database transaction, so a slow one also holds locks.
Lost audit eventsbehemoth.audit.record_failures

A plugin uses the same sink for its own numbers. Keep attribute values to a small fixed set, as Behemoth does.

go
ac.Telemetry.Count(
    ctx,
    "invites.redeemed",
    behemoth.M{"plan": plan},
)

Traces #

With a tracer, Behemoth records a span for each piece of its work: the request, each hook chain and handler, each database operation and transaction, session and token operations, password hashing, rate-limit checks, and outbound calls. A trace of one request then shows where its time went. Set the tracer with an option:

go
tel := telemetry.New(
    logger,
    nil,
    metrics,
    telemetry.WithTracer(tracer),
)

Keep the following in mind:

  • Behemoth joins your trace. It does not read traceparent or any other header. Your framework's tracing middleware starts the server span, and Behemoth's request span becomes its child. Without such middleware, each request to a Behemoth route is a trace of its own.
  • Only real failures mark a span as failed. A database error or a panic does. A wrong password, an expired session, or a hook handler that refuses an operation does not. Its span carries the error_code and stays unfailed.
  • Your code appears in the right place. A hook handler already runs inside a span, so calls you make with hctx.Ctx are traced under it. The same holds for rctx.Ctx in a route handler.

OpenTelemetry #

For metrics and traces, the OpenTelemetry adapter connects Behemoth to a collector, Jaeger, or any other OTLP destination. It is a separate module, so you download it only if you use it:

bash
go get github.com/go-behemoth/behemoth/telemetry/adapters/otel
FunctionGives Behemoth
behemothotel.NewTracer(tracerProvider)A tracer
behemothotel.NewMetrics(meterProvider)A metrics sink
behemothotel.WithTraceIDs(logger)A logger that adds trace_id and span_id to its lines

You build the OpenTelemetry SDK, its providers, and its exporters. The adapter needs only the providers. Then connect them:

go
tel := telemetry.New(
    behemothotel.WithTraceIDs(
        telemetry.NewJSONLogger(os.Stdout, slog.LevelInfo),
    ),
    nil,
    behemothotel.NewMetrics(meterProvider),
    telemetry.WithTracer(behemothotel.NewTracer(tracerProvider)),
)
Shut the providers down before exiting

The providers export in batches. A process that stops without calling Shutdown on them loses its last few seconds of traces and metrics. For a server, handle SIGINT and SIGTERM. For a short command, call Shutdown before it returns.

The examples/init directory in the repository wires all of this into one application, with a text logger, the default audit table, and OTLP export when an endpoint is set. Its audit command reads the audit log.

Test what is emitted #

The telemetry/telemetrytest package records logs, audit events, metrics, and spans, so a test can check them:

go
tel, rec := telemetrytest.New()
// Boot with tel and exercise the code.

events := rec.Audit.OfType("auth.signIn.failed")

Passing tel replaces the default audit recorder, so in such a test events go to rec.Audit and not to the audit_log table.

Next steps #

To go deeper, read the following pages:

  • Telemetry: Every field name, event type, metric, span, and filter.
  • Rate limiting: The audit event and metric for refusals.
  • Request IDs: How a request is identified across logs and responses.
  • Data and migrations: Create the audit_log table before you serve requests.