Observability

@fougere/observability is optional in the strong sense: core holds no tracing code at all, only a trace field on the invocation that it carries and never reads. An app that does not install it pays nothing.

pnpm add @fougere/observability

import { trace, onSpan, otlp, metrics } from '@fougere/observability';

app.use(trace());                       // every operation
app.use('post', trace());               // one entity's

It is an ordinary app middleware because an operation already has a lifecycle and it is that one — a second hook system would be a second answer to a settled question.

One span per operation

A span carries an op's duration and its verdict:

onSpan((span) => console.log(span.frond, span.entity, span.operation, span.ms, span.error));

Two clocks, and they are not the same measurement: the wall clock says when, so two processes land on one timeline; the monotonic one says how long, without being moved by an NTP correction mid-call.

Nothing is opened while no sink is set — observing is a decision, and the cost of one nobody asked for is zero rather than small. onSpan returns the way to withdraw.

The trace survives the wire

The parent rides the invocation, not a header:

app                     post.list        traceId a3f… spanId 01
  └─ blog (split)       post.list        traceId a3f… spanId 02, parent 01

A header is HTTP's alone, and the same call over a socket would have arrived untraced. The invocation is what every transport carries, so the trace crosses whatever the topology became — and the format is W3C Trace Context, so a collector reads it beside anything else.

The same middleware runs on both halves of a split: at the door a call arrives at, and at the stand-in it leaves from. That is why the numbers line up across processes — and why the difference between the two spans is what the wire cost.

One case the wire cannot describe: a handler reaching a second Frond builds a fresh invocation, so the parent is not on that call. It is in the context the first one is still running inside, which currentSpan() reads.

The four signals, from the span that already exists

const measured = metrics(app);
onSpan(measured.sink);

measured.snapshot();   // what to publish now

Rate, errors and duration are one metric — a histogram of durations, dimensioned by the op and by whether it refused. Its count is the rate, its dimension is the error rate, its buckets are the latency. Splitting it into three counters would publish the same numbers three times and let them disagree.

The fourth, saturation, cannot come from a finished span, because it is about the ones that have not finished. The middleware counts them and activeCalls() reads it.

Passing app is optional and only feeds the topology, which is discovered rather than declared, from the two things a process can honestly say: a Frond it SCANNED runs here, and a Frond it CALLED without having scanned runs elsewhere.

processus catalog   catalog → local        it holds the code
processus shop      shop    → local
processus shop      catalog → remote       it called it, it never scanned it

Deliberately not read from remotes: — a config key states an intent, while a Frond that answered a call is a fact. The two disagree exactly when something is misconfigured, which is when a dashboard has to be right.

The third signal: logs, and the only thing that makes them one

A log shipped without a trace id is a log stored somewhere else. What makes it the third pillar is landing on a trace and reading the lines that call produced.

The logger has a door — onLog in core — that hands out a structured record before any formatting, because what reaches the console is already cooked: ANSI codes, a badge, a timestamp. Core emits the record and knows nothing more; this package attaches the span and speaks OTLP.

import { onLog, loggerMiddleware, Logger } from '@fougere/core';
import { logs } from '@fougere/observability';

app.use(trace());                                  // first — the span must exist
app.use(loggerMiddleware(new Logger('blog')));     // one line in, one line out, timed

onLog(logs({ service: 'blog' }).sink);

The span is read at the moment the line is written, never at flush time — by then the call it belongs to is over and the context is somebody else's. A line written outside any call (a boot line, a shutdown line) leaves with no trace id rather than a zeroed one: absent means "not inside a call", and a forged zero would gather every process's startup into one phantom trace.

Forwarding is an addition, never a replacement: the console keeps its line, and a sink that throws costs nobody theirs. setLogLevel filters first, so a level that never printed is never shipped either — one place where the level lives.

Sending it somewhere

const exporter = otlp({ service: 'blog', metrics: measured });
onSpan(exporter.sink);

OTLP over HTTP, batched every second, in the JSON encoding — no protobuf, no dependency. Endpoints default to the convention (http://localhost:4318/v1/traces, and the metrics one with its last segment swapped) — name metricsUrl when traces and metrics go to two different engines rather than a collector in front. The three signals are three paths on the same door: /v1/traces, /v1/metrics, /v1/logs.

A process about to exit calls flush(); stop() ends the timer and sends what is left. onError is told when a batch could not go, and defaults to silence: a trace must never break a call, and a sink that throws is a broken exporter, not a broken call.

Metrics go out on every beat even when no span finished — a gauge that stops being published reads as "gone", not as "idle".

Wiring it, once

Everything on this page is behind one member of the app's ascent:

const app = await createApp({
  root,
  createContainer,
  extensions: [observability({ service: 'blog', otlp: 'http://localhost:4318' })],
});

up installs the middleware, the accumulator and the sinks in the order that matters — trace() opens the span every log line inside the call will carry, so installed the other way round the lines leave uncorrelated. down withdraws them and flushes what is buffered, inside dispose().

That release is not decoration. onSpan and onLog have always RETURNED their withdrawal and nothing called it: a discarded app kept feeding the sinks of the app that replaced it, so every metric counted twice. Invisible until the ring turns, and then permanent.

otlp absent, nothing leaves the process — and the topology below still answers.

Under a host that writes the boot file itself, an extension is named rather than passed: it is a function and the module writes a file, so the options travel beside the name as data.

// nuxt.config.ts
fougere: {
  observability: { service: 'blog', otlp: 'http://localhost:4318' },
  calls: { panel: true },        // `false` turns a key off; absent is the same
}

The keys are declared, not open: one per extension package, the key being both the package suffix and the export name — calls is import { calls } from '@fougere/calls'. An open record would accept callz: {} in silence.

The dev panel — what this process dispatched

@fougere/calls is the second optional extension, and it watches rather than participates:

pnpm add -D @fougere/calls

extensions: [calls({ max: 500, panel: true })],

It subscribes to app.observe — passive, and its own failure is swallowed — so it sees what a middleware cannot: a call refused before any handler (an unknown route, an entity hosted elsewhere, a call arriving while the door drains), and the route kind of every call, so a local execution and a hop to another process read the same way. Beside the calls it keeps a bounded ring of log lines, of errors, and — when @fougere/adapter-sql is there — of the statements each call issued.

Nothing is stored and no port is opened unless panel says so. The ring is served as rpc.calls, and the reader is fougere devtools over /_fougere/call, like any other consumer. An app that never installed it answers Unknown rpc operation 'calls'. It serves discover.

The shape of the system, on the wire

An app can be asked what shape it is in, on the same wire as everything else:

await call({ entity: 'rpc', op: 'topology' });
// { fronds: [{ frond: 'shop', placement: 'local', entities: 2, doors: 2 },
//             { frond: 'catalog', placement: 'remote', entities: 0, doors: 0 }],
//   edges:  [{ from: 'shop', to: 'catalog', count: 12, errors: 1 }],
//   active: 0, since: 1755861234567 }

Nothing in it is declared. A Frond is local because this process scanned it, remote because it answered a call nobody here hosts — deliberately not read from remotes:, which states an intent. The two disagree exactly when something is misconfigured, which is when a dashboard has to be right.

rpc.topology is declared by this package, not by core — so an app that never installed it refuses the op by name, and that refusal is the whole degradation a reader needs. It sits beside rpc.discover: the card says what a process HOSTS, this says what shape it is in.

A remote's entities: 0, doors: 0 is not an empty Frond — its shape is published by the process that owns it, under its own service name.

@fougere/admin reads it on its Topology page, which is the same answer drawn: what runs here, what answered from elsewhere, and every call path observed between them.

What is signal, and what is not

Spans are named for the subject rather than for today's reading of it: a duration and a verdict are the matter of a metric as much as of a trace, and an exporter for either hangs off the same onSpan. That is why sinks is a list — the two exporters read the same value instead of the middleware producing it twice.

The caller of a split call is established, not asserted: once a receiver verifies an envelope, invocation.caller names the Frond that signed, and absent means nothing was established — never "unknown peer".

Nothing here reads it yet. A service graph drawn from a verified caller would be a fact rather than an inference from an address, and the dimension is bounded like the others, so the two fit together — but the span carries no caller today, and this page would rather say so than let you look for it.

On a dashboard

Anything that speaks OTLP reads this as-is. Two collectors were run against it while it was being written — Jaeger with Prometheus, then SigNoz — and moving between them changed two URL strings: no instrumentation, no span field, no metric. That is what following the standard buys, and it is the same bargain remotes: makes for topology.

pnpm -C demos/observability dev      # three Fronds, three processes
pnpm -C demos/observability load     # k6, in stages
pnpm -C demos/observability signoz   # a collector, OTLP on 4318, UI on 8080

demos/observability is this page, running: a cart.checkout that reaches a catalog and a shipping Frond in two other processes, under staged load. Its domain code says nothing about being observed.

What to build there, in the order a reader needs it:

Two things are worth more than a fifth chart. A heatmap of sum by (le) (rate(…_bucket[1m])) shows where calls actually live — a p95 hides a bimodal distribution, a heatmap cannot. And a table of operations sorted by p95: one slow op at 20 % of the traffic drags a global quantile far from anything real, and only the sort says which one.