Observability

@fougere/observability is optional in the strong sense: core holds no tracing code at all, only a trace field on the invocation that it carries and never reads. An app that does not install it pays nothing.

npm install @fougere/observability

pnpm add @fougere/observability

bun add @fougere/observability

yarn add @fougere/observability

import { tracing, otlp, metrics, type SpanSink } from '@fougere/observability';

const spans: SpanSink[] = [];             // who takes them — this app's, not the process's
const tracer = tracing(spans);
app.use(tracer.middleware);               // every operation
app.use('post', tracer.middleware);       // one entity's

It is an ordinary app middleware because an operation already has a lifecycle and it is that one — a second hook system would be a second answer to a settled question.

One span per operation

A span carries an op's duration and its verdict:

spans.push((span) => console.log(span.frond, span.address, span.operation, span.ms, span.error));

Two clocks, and they are not the same measurement: the wall clock says when, so two processes land on one timeline; the monotonic one says how long, without being moved by an NTP correction mid-call.

Nothing is opened while the list is empty — observing is a decision, and the cost of one nobody asked for is zero rather than small. The list belongs to the app, so a released one takes its takers with it.

The trace survives the wire

The parent rides the invocation, not a header:

app                     post.list        traceId a3f… spanId 01
  └─ blog (split)       post.list        traceId a3f… spanId 02, parent 01

A header is HTTP's alone, and the same call over a socket would have arrived untraced. The invocation is what every transport carries, so the trace crosses whatever the topology became — and the format is W3C Trace Context, so a collector reads it beside anything else.

The same middleware runs on both halves of a split: at the facade a call arrives at, and at the stand-in it leaves from. That is why the numbers line up across processes — and why the difference between the two spans is what the wire cost.

One case the wire cannot describe: a handler reaching a second Frond builds a fresh invocation, so the parent is not on that call. It is in the context the first one is still running inside, which currentSpan() reads.

The four signals, from the span that already exists

const measured = metrics(app);
spans.push(measured.sink);

measured.snapshot();   // what to publish now

Rate, errors and duration are one metric — a histogram of durations, dimensioned by the op and by whether it refused. Its count is the rate, its dimension is the error rate, its buckets are the latency. Splitting it into three counters would publish the same numbers three times and let them disagree.

The fourth, saturation, cannot come from a finished span, because it is about the ones that have not finished. The middleware counts them and activeCalls() reads it.

What a duration cannot say

A span also carries selfMs — its duration minus what its observed children account for. An op that reaches two other Fronds reports milliseconds and almost no self: it is not slow, it is waiting, and a column of durations reads the two the same way.

That figure is only honest if what runs under an op is observed too, so a SQL statement is charged to the op that issued it: its time comes out of selfMs, and statements counts it. Nothing is wired for that — a Kysely sink runs inside the async context the tracer already opened, so it finds the operation in flight while the app holds no source. The engine is optional: an app on another storage observes no statement, and every operation still reports its own time.

{ kind: 'operation', address: 'cart', operation: 'checkout', ms: 31.0, selfMs: 0.2, statements: 40 }

One query per row is what an N+1 looks like, and that line says it without a trace being opened. Against the call count, statements reads as statements per call.

The count detects, the detail explains

Which queries ran is a second question, and it is off by default:

observability({ service: 'shop', otlp: '…', spanPerStatement: true })

item.listOneByOne   [0.54ms, self 0.47ms, 4 statements]     ← the default, one span

item.listOneByOne   [0.54ms, self 0.47ms, 4 statements]     ← and with it, five
  items.select      [0.01ms]
  items.select      [0.01ms]
  items.select      [0.01ms]
  items.select      [0.01ms]

The split is the one profiling has with monitoring. The count runs always and costs one span; the detail is turned on over the operation the count named, and turned off after. What decides is not the CPU but the volume: a backend charges per span, and a page of forty rows exports forty of them.

kind is what a reader owes the detail: a statement is a step, not a call this process answered. Counted as one it would publish product.select as an operation nothing serves, and appear in the topology and the saturation figure. metrics.ts says so in a line, and it is the only reader that has to — an exporter sends both, and a viewer draws them as it draws any tree.

Passing app is optional and only feeds the topology, whose counted half is discovered rather than declared, from the two things a process can honestly say: a Frond it SCANNED and hosts runs here, and a Frond it CALLED runs elsewhere.

process catalog   catalog → local        it holds the code
process shop      shop    → local
process shop      catalog → remote       it called it, it never scanned it

fronds: is not read for any of that — a config key states an intent, while a Frond that answered a call is a fact. It is read for the half named declared, which sits beside them and counts nothing, because the fact is missing exactly when it matters: a Frond that never answered is absent from the counted half, so a system with a node down reports as a smaller healthy one.

What an operation reaches, and what that costs

placement says where an operation ANSWERS. reach says where its work GOES:

fougere explain CommandeHandler.servable --root .
# Placement: commande / local
# Reach:     stock — 1 hop(s)

It answers here, and it leaves. Comment the fronds: line out and the same operation reports 0 hop(s) — same code, same signature, different number.

It is read from the HANDLER, not the operation: a dependency is declared on the constructor, so every operation of one handler reaches the same Fronds. Saying otherwise would dress a structural fact as a precise effect.

The third signal: logs, and the only thing that makes them one

A log shipped without a trace id is a log stored somewhere else. What makes it the third pillar is landing on a trace and reading the lines that call produced.

A line is an announced fact, LogLine, and a destination is a handler that accepts it. The signature is the subscription: nothing is registered, and a second destination changes no code anywhere else. observability() brings such a handler with it, so naming otlp: is the whole wiring.

import { observability } from '@fougere/observability';

createApp({
  extensions: [observability({ service: 'blog', otlp: 'http://localhost:4318' })],
});

Your own destination is a handler in one of your fronds, and it is declared the same way anything else in Fougere is — by what it takes:

import type { Fact, LogLine } from '@fougere/core';

export default class KeepHandler {
  async record(line: Fact<LogLine>): Promise<void> { … }
}

The span is read at the moment the line is written, never at flush time — by then the call it belongs to is over and the context is somebody else's. A line written outside any call (a boot line, a shutdown line) leaves with no trace id rather than a zeroed one: absent means "not inside a call", and a forged zero would gather every process's startup into one phantom trace.

Forwarding is an addition, never a replacement: the console keeps its line, and a sink that throws costs nobody theirs. setLogLevel filters first, so a level that never printed is never shipped either — one place where the level lives.

Sending it somewhere

const exporter = otlp({ service: 'blog', metrics: measured });
spans.push(exporter.sink);

OTLP over HTTP, batched every second, in the JSON encoding — no protobuf, no dependency. Endpoints default to the convention (http://localhost:4318/v1/traces, and the metrics one with its last segment swapped) — name metricsUrl when traces and metrics go to two different engines rather than a collector in front. The three signals are three paths on the same facade: /v1/traces, /v1/metrics, /v1/logs.

A process about to exit calls flush(); stop() ends the timer and sends what is left. onError is told when a batch could not go, and defaults to silence: a trace must never break a call, and a sink that throws is a broken exporter, not a broken call.

Metrics go out on every beat even when no span finished — a gauge that stops being published reads as "gone", not as "idle".

Wiring it, once

Everything on this page is behind one member of the app's ascent:

const app = await createApp({
  root,
  createContainer,
  extensions: [observability({ service: 'blog', otlp: 'http://localhost:4318' })],
});

A frond that runs in its own process carries its half of the trace with it. Put the extension in the frond's extensions/ directory, and it mounts on whichever process serves that frond — fougere serve blog included, which names no extension of its own:

// fronds/blog/extensions/observe.ts
import { observability } from '@fougere/observability';

export default observability({ service: 'blog', otlp: 'http://localhost:4318' });

up installs the middleware, the accumulator and the sinks in the order that matters — tracing() opens the span every log line inside the call will carry, so installed the other way round the lines leave uncorrelated. down withdraws them and flushes what is buffered, inside dispose().

That release is not decoration. The takers used to live in a module-level array whose withdrawal nothing called: a discarded app kept feeding the takers of the app that replaced it, so every metric counted twice. The list is the app's now, so it cannot. Invisible until the ring turns, and then permanent.

otlp absent, nothing leaves the process — and the topology below still answers.

Under a host that writes the boot file itself, an extension is named rather than passed: it is a function and the module writes a file, so the options travel beside the name as data.

// nuxt.config.ts
fougere: {
  observability: { service: 'blog', otlp: 'http://localhost:4318' },
  calls: { panel: true },        // `false` turns a key off; absent is the same
}

The keys are declared, not open: one per extension package, the key being both the package suffix and the export name — calls is import { calls } from '@fougere/calls'. An open record would accept callz: {} in silence.

The dev panel — what this process dispatched

@fougere/calls is the second optional extension, and it watches rather than participates:

npm install -D @fougere/calls

pnpm add -D @fougere/calls

bun add -d @fougere/calls

yarn add -D @fougere/calls

extensions: [calls({ max: 500, panel: true })],

It subscribes to app.observe — passive, and its own failure is swallowed — so it sees what a middleware cannot: a call refused before any handler (an unknown route, an entity hosted elsewhere, a call arriving while the facade drains), and the route kind of every call, so a local execution and a hop to another process read the same way. Beside the calls it keeps a bounded ring of log lines, of errors, and — when @fougere/adapter-sql is there — of the statements each call issued.

Nothing is stored and no port is opened unless panel says so. The ring is served as rpc.calls, and the reader is fougere devtools over /_fougere/call, like any other consumer. An app that never installed it answers Unknown rpc operation 'calls'. It serves discover.

The shape of the system, on the wire

An app can be asked what shape it is in, on the same wire as everything else:

await call({ address: 'rpc', op: 'topology' });
// { fronds:   [{ frond: 'shop', placement: 'local', entities: 2, facades: 2 },
//              { frond: 'catalog', placement: 'remote', entities: 0, facades: 0 }],
//   edges:    [{ from: 'shop', to: 'catalog', count: 12, errors: 1 }],
//   declared: { fronds: [{ frond: 'shop', placement: 'local' },
//                        { frond: 'catalog',  placement: 'remote', at: 'http://127.0.0.1:4100' },
//                        { frond: 'shipping', placement: 'remote', at: 'http://127.0.0.1:4300' }],
//               edges:  [{ from: 'shop', to: 'catalog' }, { from: 'shop', to: 'shipping' }] },
//   active: 0, since: 1755861234567 }

fronds and edges declare nothing. A Frond is local there because this process scanned it AND hosts it, remote because it ANSWERED a call. Neither reads fronds:.

declared is the other half, and it counts nothing: what the config says, beside what the runtime saw. The two disagree exactly when something is misconfigured, which is when a dashboard has to be right — and without the second one, a Frond that never answered is simply absent from the report. shipping above is declared and has returned nothing: a dead edge, visible, instead of a healthy two-node system. An address is reduced to its host and port, because a declared one may carry credentials and this answer leaves the process.

rpc.topology is declared by this package, not by core — so an app that never installed it refuses the op by name, and that refusal is the whole degradation a reader needs. It sits beside rpc.discover: the card says what a process HOSTS, this says what shape it is in.

A remote's entities: 0, facades: 0 is not an empty Frond — its shape is published by the process that owns it, under its own service name.

@fougere/admin reads it on its Topology page, which is the same answer drawn: what runs here, what answered from elsewhere, and every call path observed between them.

What is signal, and what is not

Spans are named for the subject rather than for today's reading of it: a duration and a verdict are the matter of a metric as much as of a trace, and an exporter for either reads the same span. That is why the takers are a LIST — the two exporters read one value instead of the middleware producing it twice.

The caller of a split call is established, not asserted: once a receiver verifies an envelope, invocation.caller names the Frond that signed, and absent means nothing was established — never "unknown peer".

Nothing here reads it yet. A service graph drawn from a verified caller would be a fact rather than an inference from an address, and the dimension is bounded like the others, so the two fit together — but the span carries no caller today, and this page would rather say so than let you look for it.

On a dashboard

Anything that speaks OTLP reads this as-is. Two collectors were run against it while it was being written — Jaeger with Prometheus, then SigNoz — and moving between them changed two URL strings: no instrumentation, no span field, no metric. That is what following the standard buys, and it is the same bargain fronds: makes for topology.

pnpm -C demos/observability dev      # three Fronds, three processes
pnpm -C demos/observability load     # k6, in stages
pnpm -C demos/observability signoz   # a collector, OTLP on 4318, UI on 8080

demos/observability is this page, running: a cart.checkout that reaches a catalog and a shipping Frond in two other processes, under staged load. Its domain code says nothing about being observed.

import { entity, primary, text, number } from '@fougere/schema';

export default class Product extends entity({
  id: primary(),
  title: text({ min: 1 }),
  stock: number({ min: 0 }),
}) {}

What to build there, in the order a reader needs it:

The two middle rows are the ones that name a culprit rather than a symptom: the first says which op is actually spending the time, the second turns an N+1 into a number that climbs with the page instead of a shape someone has to notice in a trace.

Two things are worth more than a fifth chart. A heatmap of sum by (le) (rate(…_bucket[1m])) shows where calls actually live — a p95 hides a bimodal distribution, a heatmap cannot. And a table of operations sorted by p95: one slow op at 20 % of the traffic drags a global quantile far from anything real, and only the sort says which one.