Fibrescope

Methodology

What is measured, from where, and what each word on this site means. The source of truth is docs/verdicts.md and docs/research/R4-probe-etiquette.md in the repository; this page restates them.

What the chain cannot tell you

Everything about Fibre that a node can answer over RPC is about the moment a blob was published. Nothing on chain describes a validator’s Fibre service after that.

On chain, and queryable from any node: the host each validator registered in x/valaddr and when it first appeared; every MsgPayForFibre with its promise hash, commitment, blob size, signer and the signature set carried at settlement; and the x/fibre parameters in force. From the signature set you can derive who was proven to hold a shard at upload time, and nothing more.

Not on chain, at all: whether an endpoint answers now; whether a validator still serves a shard it signed for; whether its TLS certificate is still endorsed by its consensus key; how fast it responds; whether it pruned before its deadline. There is no serving proof, no challenge protocol and no slashing for not serving. Retention is an obligation the chain records and never checks.

That is by design, and the Celestia core team has said so directly. Asked in the CIP-51 research thread how a server that signs without storing, or prunes early, would be detected, the answer was that the protocol relies on an honest majority, that only a third of the validator set needs to be honest for the rows to be retrievable, and that beyond that the final layer is social consensus. Asked whether the core team would ship external probing of registered endpoints, the answer was that the community is expected to build the dashboards that periodically check whether the hosts in x/valaddr are reachable and whether validators are serving the data. (forum thread)

Two things follow, and they are why this site is built the way it is.

First, where there is no enforcement, a public and checkable record is the enforcement. That raises the bar on the record rather than lowering it: every figure here has to be reproducible from stored rows anyone can download, every accusation has to be stated in the direction it accuses, and anything this observer cannot distinguish has to be published as undistinguished rather than resolved in either direction. A reputational mechanism built on a number nobody can check is worse than none.

Concretely: every record file is published as a daily export with digests (/v1/exports); every process records the build and flags it ran under (/v1/runs); the API answers any window as of any past moment (?as_of=); and sentinel-recompute re-derives every row’s verdict and every obligation figure from an export, in a second implementation of the rules kept apart from the one the site runs, and prints what differs.

Second, “a third of the set is honest” is not a slogan here, it is the Blob availability figure on the overview. The assignment hands out roughly three times the rows needed to rebuild a blob, so any honest third of the voting power holds enough. Rebuildable — could this blob still be rebuilt from the rows that actually came back at the latest complete probe point — is that assumption measured rather than assumed; fully served is the stricter reading, where every validator proven to hold a shard served it. Both are observations of rows, never an actual rebuild.

A validator can already watch its own Fibre server: the binary exports OTLP metrics and ships a Grafana dashboard. That answers a different question. A server’s own metrics are its own account of itself, from inside its own network: they cannot say whether it is reachable from outside, whether the certificate it presents is one a client will accept, or whether it is doing better or worse than the rest of the set. An indexer of the chain, however good, inherits the limits above: it can show how many blobs were published and who signed, not whether anyone is still serving them.

The promise being checked

When a client publishes a blob to Fibre, the validators sign that they received their assigned rows and the chain records the payment (MsgPayForFibre). Each signing validator then owes serving of its rows to anyone who asks until must_serve_until = creation_timestamp + max(payment_promise_timeout, shard_retention), using the x/fibre parameters in force when the blob settled. The chain does not observe whether serving continues. This site does, from outside, as an ordinary client.

One probe

A probe is one attempt to fetch one validator's shard for one blob: resolve the registered host (x/valaddr), open TCP, complete a TLS 1.3 handshake, check that the certificate's extension is endorsed by the validator's consensus key (fibre-tlsverify), call DownloadShard, verify every returned row against the blob commitment, and check that the returned row indices are exactly the ones assigned (fibre-assign, a bit-identical reimplementation of celestia-app's assignment, verified across about 890 differential scenarios). Each layer is timed and recorded separately.

Schedule

For every publication the prober plans four in-window points at 12%, 45%, 72% and 92% of the window, packed toward the deadline where early pruning shows, one grace point 30 s after must_serve_until, and one post point 3.5 min after it where “not found” is the expected answer. An honest server prunes on a one-minute loop with minute-resolution keys, so a shard outlives the deadline by anything up to two minutes (the prune key is the deadline truncated to the minute, and the loop runs once a minute); the grace tolerance (2m30s) exists so that lag is never called a fault.

The last of those four is also held to an absolute margin: whatever the fraction works out to, it is moved forward to sit no more than 2m30s before must_serve_until. A fraction alone does not survive a change of scale. On a ten-minute window 92% is 48 seconds out; on a four-hour retention it is nineteen minutes out, and since a missing shard at the grace point is tolerated by construction, a validator that pruned inside those nineteen minutes would have served every probe it was given. The margin cannot usefully be tighter than the chain’s own prune granularity, which is what sets it: below a minute or two, an early reading would be accusing an operator of the clock.

Verdicts

Each probe gets exactly one class from the probe outcome, the phase of the window at the actual start time, whether the validator was assigned the shard, and whether the settled promise proves it stored the shard.

What a FAULT claims, exactly. That this site reached the validator and it did not hand over a shard the chain records it as obliged to hold, at a moment inside that obligation. Not more. The chain calls the parameter behind that obligation a minimum local retention — upstream’s own words are “the minimum local duration validators keep uploaded shards” and “the on-chain local retention floor for uploaded shards” — and it neither checks it nor penalises missing it. So a fault here is an observation with a timestamp and a promise hash, not a finding of intent and not evidence for a penalty that does not exist.

One thing a fault cannot rule out: a power cut. The Fibre server writes the shard file first and then commits the metadata that makes it discoverable — the promise record, the /shard/ marker, the prune index — in one pebble batch with NoSync, which hands the write to the operating system without waiting for it to reach the disk. A power loss in between leaves the shard on disk with nothing pointing at it, and the server answers NotFound for data it still physically holds; upstream’s own code names that case. From outside it is the same wire answer, the same row and the same fault as an early prune. The shape that tells them apart is in the record rather than in any single row: a machine event puts a validator’s faults at one moment across many promises, a retention policy does not. If that is what happened to you, the rows carry the times and the promise hashes, and the dispute route puts the correction on the record beside the original.

Who is actually obliged

Assignment is the publisher’s arithmetic. A validator becomes obliged only once it holds the shard, and the only on-chain evidence of that is a signature from the validator on the settled MsgPayForFibre: a Fibre server writes the shard to its store before it signs. This site verifies those signatures itself against each validator’s consensus key rather than trusting the count in the transaction, because the chain’s own check runs in the ante handler and is skipped when the block is finalised or when the node has already seen the message, so a settled transaction carries no guarantee that its signature entries are valid.

A missing signature is not evidence that a validator failed to store the shard. The publisher stops collecting signatures the moment it has enough voting power to be safe and keeps delivering to the rest in the background, so a validator can hold a shard whose signature never reached the chain. Absence means unproven, never absent.

So a probe of an assigned but unattested validator is recorded as UNATTESTED whatever happened on the wire, and sits outside the serve rate in both directions. A failure the validator was never proven to owe cannot count against it, and a success it was never proven to owe cannot count for it. The wire outcome is still published, and every page that shows a serve rate also shows how much of the set the rate speaks for.

Two thirds, and why most validators are unproven

This is the single fact most likely to be misread on this site, so it is stated plainly: on any given blob, roughly a third of the validator set will have no signature on chain, and that is the protocol working as designed. It is not a missed duty, it is not downtime, and this site never counts it as either.

The publisher uploads shards to every bonded validator at once and collects their signatures as they arrive. It stops the moment the signatures it holds represent two thirds of total voting power — the same threshold the chain’s own check applies when the transaction settles — and submits. Deliveries already in flight to the remaining validators are not cancelled; they simply finish after the transaction is on chain, where nothing records them. Those validators very often do hold the shard. The chain is silent about it, and this site will not fill that silence in either direction.

Which validators land inside the quorum is a race, not a list. The publisher fans every upload out at once, one goroutine per validator, and detaches the moment the collected power crosses the threshold, so membership is upload-completion order. The same validator is inside on one blob and outside on the next.

How large that quorum is can be bounded. Over the mocha validator set (79 validators, 322,778,683 total voting power) simulated across 20,000 uniformly random arrival orders it holds a median of 53 of 79 validators; the smallest it can produce is 30, the case where the largest validators all answer first, and that is a bound rather than an expectation.

Who is in it cannot be bounded that way, and this site does not claim to know. Completion order is not uniformly random: it depends on the size of the shard, which scales with stake, so a large validator has more bytes to receive; on the network path between that particular publisher and that validator; and on the validator’s own write-and-sign latency. This observer measures none of those. So a validator persistently outside the quorum may well be telling you something about itself — that it is slow, or far from where blobs are published — and the serve rate’s population is exactly the validators that answered fast enough. Whether that makes the rate read better or worse than the network’s true retention, this site does not know: it would take a measurement of upload speed against retention, and this site measures only the second. The likelier of the two is that it flatters. It is not corrected in either direction, and the size of what is left out is published beside it. An earlier version of this page said inclusion was flat across stake: that was a property of the simulation’s assumption rather than a measurement, and it is withdrawn.

This is the main reason each validator’s page leads with reachability and certificate validity rather than with the serve rate. Those come from a TLS handshake with every registered endpoint every five minutes whether or not anything was assigned, so their coverage does not depend on a quorum anyone happened to be in, and they carry no selection effect of this kind.

Counts of unproven obligations are published per blob, never per probe. Each obligation is visited at four schedule points, so a probe count of the same fact reads about four times larger than the fact — and a four-figure number beside an operator’s name reads as an accusation where the chain is merely quiet.

What the serve rate leaves out

The rate is HEALTHY / (HEALTHY + FAULT) over probes of an assigned shard, in window, where the settled promise proves the validator stored it. Everything else is published under its own name beside the rate, never folded into it:

Verdict coverage, published with every rate in the API, is how much of that population produced a verdict at all.

One thing this site cannot see and an operator should know about: a Fibre server derives its storage budget from its validator’s stake; a server that starts outside the active set refuses to start, and one that leaves it keeps the last budget it derived. When the budget is full the server rejects the upload (ResourceExhausted) rather than dropping shards it already holds, so a budget problem shows up here as UNATTESTED — no signature, no obligation, nothing counted against the validator — and never as a fault. The server does not evict a shard before its prune time.

healthyAssigned rows served correctly while the promise held.
faultAn identity-verified endpoint, for a shard it signed for, said it has no such shard, or returned bytes that verify against neither the blob commitment nor any settled promise's assignment. The only class that counts against a validator. Rows that do verify but are not this promise's set are never a fault: the store serves by promise-hash order, so another promise's shard can answer in this one's place.
unreachableThis site could not complete a conversation with the endpoint while the validator was under obligation. From one location that is not distinguishable from a route, firewall or peering problem on this site's own path, so it is recorded and shown but kept out of the serve rate.
identity expiredThe certificate is endorsed by the right consensus key, but its signed validity window has lapsed or has not started. A renewal running late, not someone else answering on this endpoint.
bad certificateThe certificate is not endorsed by this validator's consensus key, so no client can download from the endpoint. A statement about the endpoint, shown as its status; not about any shard, so outside the serve rate.
server errorThe endpoint was reached and answered with an application error instead of the shard. It did not say it lacks the shard; from one probe that is not distinguishable from a transient fault, so it is shown beside the rate, not inside it.
rate limitedThe endpoint was reached and refused the download with a rate limit. That says nothing about the shard, so it is shown beside the rate, not inside it, and the prober backs off from a validator that says so.
unattestedThe settled promise carries no verified signature from this validator, so nothing on chain proves it ever stored the shard. Whatever the probe found is recorded but kept out of the serve rate, in both directions.
not registeredThe validator had no Fibre host in x/valaddr when the probe ran, so nobody could fetch its rows. Jailing and unbonding remove a provider from the bonded list while the chain keeps the registration, so this is a registry state, not a refusal to serve.
shadowed shardThe rows returned are genuine rows of this blob and are exactly the set another settled promise over the same blob assigns to this validator. DownloadShard is addressed by the commitment alone and the store serves the first shard by promise-hash order, so that promise answers in this one's place; the validator has no way to tell them apart.
unmatched genuine rowsThe rows returned are genuine rows of this blob but match no settled promise's assignment for this validator. The store serves the first shard by promise-hash order, and a shard uploaded for a promise that never settled is on disk until its prune and never on chain, so a validator can answer with it honestly. Not a fault the evidence supports; held out of the rate and counted beside it, with the row indices on the row.
toleratedNot found or unreachable just after must_serve_until, within the measured prune lag. Not counted against the validator.
expected goneNot found after the window plus tolerance. Correct behaviour.
served after windowStill serving after the obligation ended. Not a fault.
unreachable after windowUnreachable after the obligation ended. Not a retention fault.
unassignedValidator was not assigned this shard.
serving unassignedValidator returned a shard it was not assigned. Flagged for review.
deadline unverifiedFibre's parameters changed without an event somewhere in a range of heights covering this blob's upload, and this site has not read the parameters at every height in that range. How long the shard had to be kept is computed from those parameters, so this site cannot say when the obligation ended. It publishes neither the failure nor the credit; the verdict returns once the range has been read.
probe errorThe observer's own probe failed. A gap, not a verdict.
not probedThe slot elapsed unprobed, or the policy sampled it out. A gap, not a verdict.

FAULT is the only class that counts against a validator. It means one of two things, for a shard the validator signed for, from an endpoint whose certificate its consensus key endorsed: it said it has no shard of this blob at all while the promise held, or it returned bytes that do not verify against the blob commitment. Genuine rows of the blob that are not this promise’s assignment are never a fault: the store serves whichever promise’s shard sorts first by hash, including one whose promise never settled, so the validator may be answering honestly; such rows are shown beside the rate under their own name. Anyone who repeats the probe gets the same answer. Being unreachable, answering with a server error, refusing with a rate limit, or presenting a wrong certificate are each their own class and never a fault. One margin is deliberate: a not-found answer that arrives within thirty seconds of the deadline is tolerated rather than faulted, because the server prunes on its own clock and the observer’s need not agree with it to the second.

Rates

Gaps

NOT_PROBED and PROBE_ERROR rows are the observer's own gaps: the slot elapsed while the observer was down, the probe policy sampled the blob out, or the probe itself failed. They are counted and shown, and never enter a rate. Observer run spans are recorded (/v1/runs), so downtime is rendered as a gap, not as zero.

That holds per obligation, not only per probe, and it took a fix to make true. An obligation this site watched early and then lost sight of — the exact shape an outage here leaves in the record — used to keep its last good verdict and publish as kept, so the serve rate rose while nobody was watching. It is now end unobserved: outside the rate, counted beside it. The two directions are not symmetric, and the remaining bias belongs on this page rather than in a commit message. A fault is conclusive from one reading, because the shard was gone at that minute; a kept promise is a claim about a whole window, so it needs a reading near the end of one. While this site is blind, the obligations it can still judge are therefore enriched for faults. What its blindness cannot do is manufacture one: downtime here can only withhold credit from an operator, never accuse them, and the number of times it did is printed next to the rate.

The observer also reports on itself. Each of its four processes keeps a status file, and /v1/health turns them into one verdict: which process is alive, when it last succeeded, how far the scanner is behind the chain, how much disk is left. The chip in the header is that verdict. Blocks the observer’s own node could not serve are recorded as scan gaps and listed on the overview: a publication settled in one of them is unknown here, not counted as served or unserved.

Load policy and sampling

A probe transfers the validator's whole shard; there is no partial-row download in the protocol. To stay a small fraction of a validator's capacity, the prober enforces per-validator caps (requests per minute, bytes per hour and per day, scaled by assigned rows) and global caps. When the projected load for the next hour would exceed a cap, publications are sampled at probability p = cap / projected. The sample is deterministic and unpredictable: a blob is probed iff H(promise_hash || day_secret) < p · 2^64, where day_secret = HMAC(master, date). Every row this site stores, probed or sampled out, carries the probability it was decided at and a commitment to that day’s secret (SHA-256 of the secret). Those commitments are published at /v1/sampling. Seven days after a day ends, its secret is published beside the commitment (the delay clears any retention window this site schedules, so the draw stays unpredictable while it matters), and from then on anyone can recompute the draw for every MsgPayForFibre settled that day and check which publications this site should have probed against which ones it did, from /v1/probes; sentinel-recompute -sampling does it from the daily export. A sampled-in blob gets its whole schedule; a sampled-out blob is recorded NOT_PROBED with the reason and p. Backoff only ever removes the download step, after three consecutive transport failures or rate limits from one validator and for twenty minutes; it never adds requests, and the slots it skips are recorded as not probed, which is a gap, never a verdict.

Publishers and fees

The Publishers page is the one part of this site that measures nothing. Every figure on it is a count of something the chain recorded: a MsgPayForFibre settling a promise, a MsgPaymentPromiseTimeout charging an abandoned one, a deposit to or withdrawal from escrow, and the escrow balance the chain holds for each account. The scanner reads these from the same blocks it reads publications from; nothing comes from a third-party index.

What kind of claim each number is, and through which block

Every headline figure carries a small tag saying which of three kinds of evidence it rests on, because the three are different claims and a reader should never have to guess. chain: a count of something the chain recorded — publications, signatures, fees, registered endpoints — nothing here was measured by this site. verified: bytes this site fetched and checked against the on-chain commitment, or a certificate checked against the validator’s consensus key — the serve rate, faults, obligations, endorsement. observed: what this site’s own network saw from one location — reachability, throughput — which says nothing about any shard. On chain does not make a figure true of Fibre service, and a verified download is not a statement about anyone else’s path to the endpoint. The same map is published under evidence in /v1/meta.

Every snapshot also says through which block it was computed: record_through on /v1/network, /v1/validators, /v1/market and each validator’s page carries the scanner’s checkpoint height and its block time as they stood when the figures were taken, beside the chain tip the collector had last seen. computed_at says when; this says over which part of the record. The record is indexed by height and the clock is not, so this is the number to quote when checking a figure against the chain.

Vantage

All measurements on this site come from one vantage, named in the API (/v1/meta) and stamped on every probe row. A failed probe means this vantage could not fetch the rows at that time; it is not proof the validator is down. A successful probe is not proof of availability from elsewhere. Adding vantages multiplies load on validators, so a second one comes after the policy above has been published.

What this site does not do