Monitoring & Observability

03

Monitoring is a product, not a side quest

Many teams buy a monitoring tool, wire a few dashboards to green, and call observability done. They discover their mistake the night something breaks and the dashboard is a wallpaper of unreadable panels that answer a question nobody asked. We treat monitoring as a product: it has users (engineers, support, the on-call human), it has jobs to do (detect, diagnose, route, prove), and it has a maintenance cost. It is designed, not attached.

• It is engineered like the product it watches — dashboards and alerts are specified, reviewed and versioned the same way code is; a monitoring change is a code change.
• It is built around questions, not panels — every screen answers 'is there money moving?', 'are callbacks confirming?', 'are wallets settling?' — not 'does this graph exist?'.
• It has a maintenance owner — alerts that page people are owned by a human who keeps them true; nobody accepts a permanent false alarm in the rotation.
• It is documented and discoverable — a new engineer can open the observability front door and understand the system in an hour, not a quarter.
• It is measured by its outcomes — we track time-to-detection and time-to-understanding, not the number of green panels; a monitoring system that finds nothing late is a failed product.
• It degrades honestly — when telemetry itself fails, that failure is visible and paged before it silently blinds the team.

A payment platform cannot afford 'we did not notice'. So we build monitoring the way we build the money layer: deliberately, owned, and tested against the failure it is meant to catch.

04

What each signal sees — and what it cannot

Observability rests on three classic pillars plus the context that makes them useful. Each one sees a different slice, and each one is blind to some things. The craft is knowing which signal answers which question — and gluing them together with a shared correlation key so the slices become a whole picture.

• Metrics — numbers that trend: request rates, latencies, error rates, queue depths, balances. They tell you 'something is wrong' fast, but not which customer, which line, or why.
• Logs — the story of one request: structured records with timestamps, actors, references and outcomes. They tell you what happened, but only if something was logged at the right moment.
• Traces — the path one operation took through every service, database call and gateway hop. They tell you where the time went, but only on the paths that are instrumented.
• Events — discrete happenings: a deploy, a schema migration, a config change, a cron run. They tell you what changed, which is often the cause of the drift metrics just showed.
• The shared correlation key — a request ID or transaction reference carried from the phone through the API into the ledger, so a log, a trace and an event that belong to the same customer can be joined in one search.
• The gaps are the design — we deliberately decide what is not worth logging, tracing or alerting, because an observability layer that records everything NoSQL-style is expensive and usually unreadable.

On money paths the correlation key is the transaction reference — the same one the gateway echoes back in its callback. That single decision is why 'which payment is this?' has a one-line answer in our logs.

05

Metrics: numbers that trend (the four golden signals)

The phrase 'the system is fine' is only a fact if it is backed by numbers. We instrument the classic golden signals on every service that matters, and we wire them to the business reality underneath — because a latency rise on a payment endpoint is not a curve, it is a roomful of people staring at a pin pad.

• Latency — how long a request takes, measured in distributions (percentiles), because the average hides the customer whose STK push took 40 seconds while the mean stayed beautiful.
• Traffic — request rate and concurrent load, watched for the spikes that precede saturation and for the sudden drop that means requests are failing before they reach you.
• Errors — the rate of 4xx, 5xx and application-level failures, broken out by error class so 'payment surface is erroring' becomes 'callbacks are failing verification' within one click.
• Saturation — how close the database, the queues, the connections and the gateway budget are to their physical limits; the warning that arrives an hour before the outage.
• Business metrics beside the technical ones — pending transactions aging, wallets settling, reconciliations matching; the technical signals tell you something may be wrong, the business metrics tell you what it is.
• Percentiles over means — money software is about the worst user experience, so p50, p95 and p99 are charted, and the SLO is written against the tail, not the average.

When a metric crosses a threshold, the human needs to know why within minutes. That is what the log and trace layers exist for — the metric says 'look here', and the story layer explains what happened when you did.

06

Logs: the story of one request, made searchable

A log that cannot be searched is a receipt, not a tool. We centralize structured logs so that any actor, any transaction reference, any gateway callback can be found across the whole stack in seconds — and we log the fields that incident response actually needs, not the ones that are convenient.

• Structured, always — JSON with named fields (timestamp, user, reference, outcome, latency, error code), never free-text prose that cannot be queried or joined.
• Centralized and searchable — logs from every service, every edge function and every scheduled job land in one place, so a request that crossed five hops is one search, not five console sessions.
• Correlated by reference — the transaction reference and the request ID ride every log line, so 'show me everything that touched this payment' is a single query returning a single timeline.
• Verbosity tuned per path — money endpoints log request, verification result, outcome and timing; static pages log far less; we spend log budget where the answers live.
• The payment-gateway log is sacred — raw callbacks and their verification results are stored even longer than ordinary logs, because 'where is my money' is answered from that record months after the fact.
• Errors carry context, not noise — an error log includes the state, the reference and the payload-shaped clue an engineer needs, and it is deduplicated so one customer erroring a hundred times is one signal.

Support desks tell us the truth that dashboards cannot: a customer calling about 'my money did not arrive' is a search away from the answer when every payment is logged with a reference from the moment it was born.

07

Traces: following one request through the whole stack

A payment request does not live in one service. It starts on a phone, enters an API, writes to a wallet, calls the gateway, gets a callback, verifies against the gateway's own status API, and settles the ledger. When it is slow, or hangs, or fails in a way no single service can explain, distributed tracing is what shows the actual path and the actual time each hop took.

• Trace IDs across every hop — the transaction reference is propagated through every call, so the whole journey from pin pad to ledger is one trace with one root.
• Spans per meaningful step — the push fires on whatever rail carries it (STK, card, bank or wallet), the callback lands, the verification query runs, the credit commits — each is a span with a duration, so 'where did the 12 seconds go' has an answer.
• Latency attributed to services — a trace shows which hop ballooned: the gateway, the database lock, the verification query, the wallet write — not a shrug across the whole stack.
• Errors attached to spans — a failed verification or a timed-out query is visible at exactly the span where it happened, with the payload nearby.
• Sampling done honestly — trace every money path in full, sample the rest; the traces that answer the expensive questions are the ones we refuse to lose.
• Joined with metrics and logs — the trace is reachable from the dashboard panel that flagged the latency, and every span links to the logs of that hop.

The transition is real: before tracing, an incident gets argued about; after tracing, it gets resolved. Our own payment incident reviews are timelines pulled from traces, not reconstructions from memory.

08

SLOs and error budgets: 'good enough' made measurable

'Good enough' is a fight until it is a number. Service level objectives turn reliability into a budget: we agree on the target (say, 99.9% of payments confirm successfully this quarter), the error budget is what remains, and every change that risks consuming the budget is a decision made openly instead of a surprise discovered on the day of reckoning.

• SLOs written against what users feel — availability of the payment path, callback confirmation on time, wallet settlement under a latency line — not uptime of a server nobody touches.
• Error budgets make trade-offs explicit — the team can spend the budget on a bold change, but when the budget is gone, reliability becomes the top priority for the rest of the period.
• Burn-rate alerts, not threshold whining — alerts fire when the budget is being consumed faster than the plan allows, not on arbitrary percentage wobbles at 3am.
• Money paths get tighter SLOs — the paths that move money are held to stricter targets than the paths that move pixels, because the cost of a missed payment is a person's trust.
• SLOs are reviewed, not fossilized — every quarter the targets are questioned against what the business actually needs; a perfect-but-expensive SLO is loosened deliberately.
• They are honest about the past — the error-budget ledger keeps score, so a quarter's saved-up budget is a real license the team earned and can spend.

The discipline changed how our own team argues: nobody says 'but it's basically fine' against a number — the SLO settles it, and the budget makes the risk visible before it is spent.

09

Alerting that respects the human on the other end

Alert fatigue is not a culture problem; it is an engineering failure. Every alert that fires and is ignored trains the on-call human to ignore the next one — including the one that matters. We build alerting where every page is actionable, routed to the right person, and worth waking up for.

• Page on symptoms, not every cause — one alert for 'payments are failing' instead of fifteen for each likely cause; the runbook contains the diagnosis, the pager contains the fact.
• Severity is a promise — sev1 pages the responsible engineer now, sev2 pages within minutes, sev3 goes to chat; the on-call human is never forced to guess how urgent a page is.
• Route by ownership, not by list — money-path alerts reach the payments engineer, reconciliation alerts reach the ops rotation; the right person, the right channel, the right expectation.
• Escalation exists and is rehearsed — if the primary does not acknowledge, the alert climbs; escalation to the platform owner is a designed path, not an improvisation at 4am.
• Quiet hours are engineered — detecting the difference between a 3am page that changes nothing and a 3am page that saves a customer's rent payment is a design decision, not a policy hope.
• Every alert has a test date — alert rules are reviewed and re-validated on a cadence; an alert that cannot fire a real incident in rehearsal is a risk wearing a costume.

When we say 'alerting that respects humans', we mean it literally: we measure how many pages are false, how many are loners, how many lead to nothing — and we fix the alerts before we blame the person ignoring them.

10

Dashboards that answer real questions

A dashboard is a lie detector for a specific question, not an aquarium. We build dashboards around the questions teams actually ask — 'is money moving?', 'are callbacks confirming?', 'is reconciliation matching?', 'is the bill still sane?' — so that the first thirty seconds of an incident produce the answer, not a scavenger hunt.

• One question per dashboard — a payments dashboard, a callbacks dashboard, a reconciliation dashboard, a cost dashboard — never one wall of everything for everyone.
• Top-down drill paths — a red health line leads to the error breakdown, which leads to the trace, which leads to the log line; a dashboard is the front door of a corridor, not the whole corridor.
• Business numbers beside technical ones — money moved today, pending aging, verification failures and reconciliation variance sit with the latencies and error rates, because 'is it broke' is a business sentence.
• Built for the wall and the pocket — large panels for the operations screen, compact versions on phones for the person being paged away from their desk.
• Uptime and health at the door — a glance answers 'is the platform up, moving money and settling' before anyone digs anywhere.
• Named after questions, maintained like code — dashboard changes go through review; nobody tiles a new panel onto a busy wall and calls it done.

The difference between our dashboards and a thousand wallpapers is one test: a new team member asked 'what's happening right now?' should get a truthful answer in under a minute, or the dashboard loses a public debate.

11

Monitoring the money path specifically

Ordinary observability watches servers; money observability watches the lifecycle of a payment. Because we operate a payment platform, we hold the money path to a stricter standard — the path that moves a tenant's rent to a landlord's wallet is watched end-to-end, state by state, and its failures are business events, not just log noise.

• Pending transactions aging is a business signal — a push, a card charge or a transfer that sits pending past its window is flagged, swept and reported; money does not get to linger in a half state quietly.
• Callbacks are watched as first-class citizens — the arrival rate, the verification results and the failures of gateway callbacks are charted, because a callback that never confirms is money stuck in limbo.
• Reconciliation variance has its own dashboard — the daily match against the gateway statement is visible the moment it runs, so a penny of drift surfaces as a workflow, not a month-end discovery.
• Fail-to-confirm ratio is a SLO, not a diagnosis — the fraction of payments that do not confirm is tracked over time; a rising ratio is a business problem being born, long before customers call.
• Wallet and float position are observed — the float that backs customer balances, the locked-versus-available split, the fee revenue booked — numbers that tell you the platform is solvent, visibly and continuously.
• Scheduled sweeps are observable — the cron jobs that expire stale transactions and release holds publish their run results, so 'did the cleanup actually run' has an answer, every time.

On money paths we do not wait for a customer to discover the failure. The system is watched so that 'is this payment stuck?' is answered by a dashboard before the support queue hears the question.

12

Payment gateway logs: the 'where is my money' answer path

Gateway callbacks arrive over the ordinary internet, as claims about what the gateway did — M-Pesa and Daraja at home, and every other rail a platform runs. Every claim is verified against that gateway's own status API before money moves, and every raw interaction is recorded. That record is the foundation of the single most important support promise a payment platform can make: 'where is my money' gets a truthful, inspectable answer in minutes.

• Every callback is archived raw — the untouched HTTP body, the headers, the timestamp and the endpoint that received it, stored beyond ordinary log retention.
• Every verification is logged beside it — what the gateway's query API said in response, so 'the claim said paid, the verification said X' is a record, not a memory.
• Support can search by reference — a customer's transaction reference returns the whole story: initiated, pushed, callback received, verified, credited — or the exact step where it stopped.
• Gateway errors are classified — transport failure versus definitive refusal versus amount mismatch are distinct log classes with distinct runbook paths; the response to the gateway differs for each.
• Mismatches never vanish — what the gateway said and what we credited are correlated, and any disagreement is surfaced to reconciliation, not parked.
• The log is queryable forever — the money-path record is not rotated away on a routine; it is the platform's memory and support's answer key combined.

When a customer says 'my money did not arrive', the correct honest reply is a search away, not a mystery. That is what gateway logging is for, and why we treat it as a financial record rather than a technical log.

13

The on-call reality (not the brochure version)

Monitoring produces pages, and pages land on people. The difference between a great on-call rotation and a burning one is mostly engineering: clear ownership, honest severity, runbooks that work, and a culture where being paged is a designed reality, not a punishment drawing.

• Small, owned rotations — the on-call list is the people who can actually fix the thing; a page reaching a generic 'all hands' mailbox is a design failure labelled as urgency.
• Handovers are sacred — the outgoing on-call writes the state of the world: what happened, what is limping, what the incoming should watch; a calm handover is worth two runbooks.
• Runbooks are beside the alert — the same page that wakes a human links to the exact procedure, so a 3am brain does not have to reason from first principles.
• Page triage is trained, not hoped — the on-call is briefed on what each severity means and what to do when unsure; 'acknowledge, then escalate' is a real policy.
• The discharged incident is reviewed — every real page gets a short postmortem, even a quiet one; two quiet nights in a row is data, not congratulations.
• Overwork is monitored — if a rotation is being woken every night, the alert rules are wrong and the system, not the human, is changed.

We have been the 2am on-call for our own payments platform, so we design rotations the way we would want to be woken: only for what matters, with everything needed to act, and a team that debriefs instead of blaming.

14

Incident runbooks: the right person knows exactly what to do

An incident runbook is the difference between 'someone should do something' and 'the third step of the procedure says to check X and call Y'. We write runbooks for every failure the system can plausibly offer — the ones that happened before and the ones the design review could foresee — and we write them to be followed by the human at 3am with a phone.

• Written from real scars — the highest-value runbook is the one describing the incident that already hurt; every postmortem that produced an action may produce a runbook page.
• Step-shaped with owners — each phase names who does it: the on-call acknowledges, the primary diagnoses, the escalation point is written down, not hoped for.
• Decision trees for the ambiguous — 'callback not arriving': check the gateway log; check the verification query; check the sweep; each branch has an action and an exit.
• Money incidents have precedent — because we operate payments, the money runbooks are rehearsed: a stuck batch, a reconciliation variance, a failed B2B payout all have scripts.
• Drilled in calm so they work in chaos — runbooks are rehearsed in game days so that a live incident follows a muscle memory, not a first read.
• Versioned like code — a runbook that drifted from reality is a trap; they are reviewed whenever the system changes and whenever they are used.

The runbook is the memory of the incident written before the incident. We own that memory for the money system, and we bring the same runbook discipline to every system we watch.

15

Post-incident reviews that produce adopted learnings

A postmortem that produces a slide deck is theatre. The only thing that matters is the action list that is actually adopted: the alert that gets added, the check that gets written, the runbook page that gets created. We run reviews that make the system measurably safer each time, and we keep the learnings enforced rather than applauded.

• Blameless by rule — the subject is the system, the timeline and the actions; the word 'who' does not appear as an accusation.
• Timeline first — the review starts from the traces, logs and events, and the story is built from evidence before anyone offers explanation.
• Every finding maps to an action — for each 'what should have caught this', there is a named next step with an owner and a review date.
• Class-level fixes over one-offs — we fix the class (the missing alert, the weak verification, the blind spot) so the same incident cannot return in a different costume.
• The learnings reach the tools — if the incident taught us a signal, a runbook or a test, that artifact is created in the same week, not the same quarter.
• A cadence that demands it — every real incident gets a review, even the quiet ones; the discipline does not depend on whoever happened to be angry that week.

The honest measure of our observability is that the system measurably gets better after every failure. Each incident closed with adopted learnings is the product improving — and that is the whole point of watching.

16

Scheduled checks and synthetic probes: catching weekly drift

Some rot only shows weekly: a certificate about to expire, a job that stopped quietly, an integration that drifted from the real gateway. Synthetic checks and scheduled probes exist to catch the drift that never triggers a live incident — the slow leaks that would otherwise surface as a mystery on a quiet Tuesday.

• Synthetic payment probes — a test transaction through the real flow (sandbox keys where possible, harmless real moves where safe) that verifies the path still works end-to-end.
• Certificate and expiry checks — TLS certificates, gateway keys, and OAuth tokens are watched so 'expired credential' never becomes the headline of an outage.
• Job heartbeat checks — every cron that matters publishes a heartbeat; a missed run pages before the effect of the missed run is discovered by a customer.
• Integration drift checks — the gateway's documented contracts, the signing requirements, the SDK versions are re-validated on a cadence, because sandboxes drift.
• Playbook-based diagnostics — a scheduled check runs the same diagnostic a runbook would, so 'is X healthy' is answered every day, not only during the incident.
• The quiet failures are loud in aggregate — drift that would be invisible per-run is visible as a trend, because the checks themselves are monitored like telemetry.

The alert that never fires because the check caught the rot first is the best alert we have. Synthetic probing is how we rehearse 'fine' until it becomes a fact checked daily.

17

Cost observability: the bill is observable too

Infrastructure cost surprises are the one incident no alert ever fires for — unless you watch it. We wire cost and usage into the same observability so the bill is a trend with a drill-down instead of a month-end shock, and so features, environments and customers are visible in the number.

• Cost as a first-class metric — spend is charted and trended beside latency and error rate, because an infra decision is a money decision before it is a performance one.
• Usage attribution — cost is broken down by feature, environment, service and (where it matters) by customer load, so 'what is this actually costing' is a query, not a guess.
• Unit economics on the money platform — the cost of processing a transaction, of running a top-up, of holding a wallet is computed, because a platform's margins live in its unit costs.
• Anomaly detection on spend — a bill that jumps overnight is flagged like a latency spike, because runaway spend is a failure mode with its own runbook.
• Capacity and budget forecasts — current spend and growth trends project next month's bill, so the budget conversation starts before January starts costing.
• Cost reviews on a cadence — the bill is reviewed like the dashboards: what is this for, is it earning its keep, what can legitimately shrink; observability makes shrinkage a decision, not a surprise.

Monitoring that stops at uptime is blind to the roof. The bill is part of the system's health — and on a payment platform, unit economics are how you know a feature is really working or quietly eating the margin.

18

Observability as a product for support and the business

The consumers of observability are not only the on-call engineer. Support needs 'where is my money' answered fast; operations needs pending queues visible; leadership needs 'is the platform healthy and solvent' in one sentence. We build observability surfaces for each audience, from the front-line desk to the board.

• Support surfaces are read-oriented — a support person looks up a reference and sees the full payment story without permission to change anything; inquiry is a view, not a write.
• The health snapshot for leadership — one screen that says money is flowing, settling and reconciling, with a trend and a budget line, so the business uses the same truth the engineers do.
• Operations dashboards for the daily run — pending approvals, stuck jobs, exception queues, gateway incidents — the work of keeping the machine moving is itself visible work.
• The support case becomes the trace — a ticket about a payment links to the transaction timeline, so support stops being a switchboard and starts being the first line of diagnosis.
• Reports are generated from the same telemetry — daily, weekly and monthly operational reports reuse the dashboards' data rather than inventing a parallel spreadsheet truth.
• Feedback loops back into engineering — support questions ('why is this state possible?', 'how do I tell this apart from that?') become monitoring improvements when repeated.

A monitoring system only the engineers can read is half a product. We build observability for everyone who has to answer a question about the system — and we hold support's 'where is my money' answer time as one of our own SLOs.

19

Honesty about monitoring and observability

Observability has limits, and a vendor who tells you monitoring makes you safe is selling you comfort. We are direct about what it can and cannot do, because the point is a system that stands up — not a dashboard that impresses a tour.

• Monitoring reduces time-to-know, not the universe of failure — it cannot prevent every outage; it decides how fast and how calmly you respond to the ones that happen.
• It is a cost like any other — logs, traces and alert platforms have price tags; we size the observability to the risk and the revenue, not to the vendor's largest plan.
• It must stay true or it dies — false alerts and stale dashboards are how monitoring earns being ignored; we treat truthfulness as a maintenance requirement with a calendar.
• The perfect system is not the goal — an alert that would have caught an incident you can now prevent is worth more than an alert riding herd on every possible event.
• Some failures are invisible until they hurt — a decaying code path, a dependency going cold, a human process lapsing; monitoring finds what produces a signal, and the signal is designed deliberately.
• Telemetry is not judgement — a chart tells you a number moved; deciding what the number means and what to do about it belongs to people, which is why the people layer (runbooks, SLOs, reviews) is the other half of observability.

We would rather tell you the truth about what monitoring buys than sell you a promise a graph cannot keep. And when it matters — the money paths — we build the monitoring tight enough that the truth it tells is the truth you need at 3am.

20

How we build and run observability

The build never really ends, because the system it watches keeps moving. But the shape of the work is stable and disciplined: instrument, define, alert, dashboard, rehearse, review, tune — and do it again next month, because observability is a standing service, not a one-off project.

• 01 · Instrument first — metrics, structured logs, traces and events wired into the code as it is written, not bolted on after the incident.
• 02 · Define what matters — the SLOs, the severity ladder, the ownership map and the questions each dashboard must answer are agreed before a panel is placed.
• 03 · Build the money dashboards — payments, callbacks, reconciliation, pending aging and float position, each answering its named question.
• 04 · Write the alert rules — page-on-symptom, routed to owners, with burn-rate honesty and rehearsed escalation.
• 05 · Write and rehearse runbooks — every alert links to a procedure, and the top money runbooks are practiced in game days.
• 06 · Wire support surfaces — the 'where is my money' search path, the gateway log access, the operations views, all built for the people who answer humans.
• 07 · Watch cost and capacity — the bill, the unit economics and the forecast joined to the same dashboards as uptime.
• 08 · Rehearse the incident — a simulated failure that the on-call follows through runbook and postmortem, so the first time it matters is not the first time it happens.
• 09 · Review, adopt and tune humanely — every real incident produces adopted learnings, and alert rules are pruned and validated on a cadence while the on-call's page load is treated as a product metric too.

You keep ownership of the telemetry, the dashboards, the runbooks and every access key. No monitoring hostageware — the system that watches your money is yours, tuned the way only a team that has been woken by their own platform tunes it.

The toolchain

The monitoring & observability toolchain

This is the stack we run for a live payment platform — telemetry, query, alerting, dashboards, money-path probes, cost and process. Every layer exists because a live system demanded it, usually at 2am.

Lifecycle

The observability lifecycle — from first signal to tuned product

Observability is a standing service: it is built, rehearsed, reviewed and tuned as long as the system it watches exists. This is the arc every engagement passes through — the same arc that runs our own platform around the clock.

Closing

More than development

Monitoring is how you know, and observability is how you know why. We build both as a product — with the discipline of a team that has been woken by its own payment platform more times than it wants to admit, and tuned it from every one of those nights. That includes:

Monitoring is a product too: it has users, jobs, cost and a maintenance burden — and it earns its keep only when the right person hears the right thing at the right time.

We do not sell dashboards. We sell the ability to know — fast, truthfully, and with a clear path to acting. On a payment platform, that is the difference between a bad afternoon and a bad month.