Backend Engineering

03

What the backend actually is — and why it decides everything

The backend is not 'the server'. It is every decision the product makes that no screen shows: who is allowed in, what they may do, what is true about the data, what happens when two people act at the same moment, what runs when nobody is watching, and how every record is protected and explained. When a product fails, it almost always fails here — silently, in production, at the worst moment.

• Business logic lives deliberately in one place — one trusted layer owns the rules, instead of rules scattered across screens, edge functions and client code where each copy eventually disagrees.
• The machine that never sleeps — scheduled jobs, queues, retries and reconciliations run unattended at 2am, and are engineered to run for years without a human holding the torch.
• Logs explain everything — what happened, in what order, by whom, with what error — so a production mystery becomes a findable line instead of a tale told around a server.
• The source of truth is a real database — with constraints, transactions and integrity, so a client that lies is refused, not forgiven.
• Every surface is a guest — the web app, the mobile app, the desktop tool and the partner's API are all clients of one backend, so all of them can never tell different stories.

There is a reason the question every serious builder asks first is 'where is the truth kept, and who is allowed to change it?'. The backend is the answer — and getting it wrong is the most expensive mistake a product can make.

04

The architecture: layers that hold their shape

A good backend is not a pile of endpoints; it is a set of layers with a clear job each and a direction of trust. The shape of the system is what keeps it safe and fast years later, when nobody remembers the early decisions.

• Presentation at the edge — the API accepts requests, authenticates them, validates them and hands them inward; the transport cares about format, never about truth.
• Application logic in the middle — the orchestration, the rules, the business decisions — one layer, in one language family, so the product's brain is reviewable.
• Data at the bottom, protected — the database owns integrity, constraints and transactions; the application asks, the database refuses the impossible.
• The direction of trust — everything funnels to the core; no surface can peek around the core or reach the database directly to avoid a rule.
• Modular, not monolithic-cursed — services or modules split at real seams (auth, billing, notifications, reports) without paying the integration tax of a distributed system too early.
• Contracts between the layers — schemas and types at every boundary, so the web team and the mobile team and the job runner all speak the same shape.

The test of an architecture is not how nice it looks in a diagram. It is whether a new feature, a new client or a new attacker can arrive and the shape still holds.

05

The language and stack decision

Backend languages are a matter of engineering, not fashion. We choose per project — matching the stack to the work, the team and the funding reality of a Kenyan business — and we are honest about the trade-offs of each:

• TypeScript / Node — the invisible majority of modern products: one language across web, mobile and backend, an enormous ecosystem, and an installed base that hires from easily. Right for most products, especially when a web team must also build the backend.
• Go — the language of high-concurrency infrastructure: brutal simplicity, compiled speed, tiny deployed binaries and concurrency as a first citizen. Right for high-throughput services, gateways and tools that must never stutter.
• Rust — memory-safe near-C performance: the modern choice for the deepest trust boundaries — payment engines, crypto-adjacent work, protocol code — where a buffer overflow is simply not acceptable. We use Rust where it earns its reputation, not everywhere.
• Python — the fastest path for AI, data and machine learning services, with mature frameworks and a developer economy that moves like lightning for business logic.
• Managed PostgreSQL as the resident database — the trust backbone of financial systems: real ACID, row-level security, JSON, full-text search, and the same engine our live platform runs millions of financial rows through.
• Queues and workers — Redis, NATS, or managed job queues: work that should not block a request (emails, webhooks, heavy jobs) goes to a queue with retries and dead-letter honesty.

The honest rule: pick the language the team can be great in, and let the database carry the trust. Every stack on this list has shipped money systems at production scale; the difference is never the logo, it is the discipline.

06

The database: where the truth is protected

Every serious product is a database with a front door. The database is where money lives, where promises are kept and where attacks ultimately aim. We treat it as the surface most worth protecting and most worth designing.

• Schema that argues back — constraints, NOT NULLs, foreign keys, unique keys, enums and CHECK constraints refuse lies at the database, not in the app where lies are easier to write.
• Transactions that cannot split — all-or-nothing for every operation that must hold together, so an interrupted save can never leave half a record.
• Row-level security for isolation — the database itself refuses cross-tenant and cross-owner reads, so a bug in the app cannot leak another customer's world.
• Indexes chosen, not sprinkled — indexes follow the actual queries and the growth of the data, reviewed with EXPLAIN before a slow wall appears.
• Migrations that go both ways — schema changes run forward safely and reversibly, in order, reviewed, with the data preserved — the way a money system must evolve.
• Backups restored on a schedule — restores rehearsed, not assumed, because a backup that has never come back is a rumour, not a safety net.

We have caught products' flaws that only live in the database — CHECK constraints bypassed at the app layer, generated columns that must never be written, enum values the code forgot. The database is the final honest referee, and we program it to argue.

07

Correct under concurrency: the hardest discipline

Two users paying at once, a retry arriving late, a webhook firing twice, a cron overlapping itself at midnight — these are not edge cases, they are normal life. A backend that is not correct under concurrency is a backend that will lose money or lose data with a straight face.

• Row-level locks (FOR UPDATE) — the row is locked before the decision is made, so two payments or two withdrawals on the same wallet physically cannot both win.
• Atomic transactions — the balance check, the lock and the decrement happen as one unit; the system cannot observe a state that was never true.
• Idempotency keys make retries harmless — a retry carrying the same operation key is recognised and answered, never executed twice — the reason a network hiccup cannot double-charge.
• Advisory locks for cron safety — scheduled jobs claim a lock so two overlapping runs cannot double-send, double-deduct or double-notify.
• Optimistic checks where locks are heavy — versioned rows and conditional updates refuse stale writes with a clean retry, when a lock would be overkill.
• Dead-letter honesty — work that fails repeatedly goes to a visible dead-letter queue with its payload intact, so nothing is ever silently lost.

Concurrency bugs are the most expensive bugs a product can have, because they are reproducible only by production. We build the proof out of production — locks, idempotency and transactions are the default, not the special case.

The money-specific versions of these rules — this is where products actually break:

• Three layers of balance protection — a pre-flight check inside the row lock, an atomic decrement, and a database CHECK constraint as the final wall; no single bug — not even a bypassed check — can carry a balance negative.
• Escrow semantics — initiation moves available→locked; success consumes the lock, failure releases it; a balance is never simultaneously spendable and pending.
• Callback trust is earned, not given — a payment callback is re-verified by querying the gateway's own status API before credit; transport failure keeps money pending and asks for retry, a definitive 'not paid' is acknowledged but never credited.
• Advisory locks plus SKIP LOCKED on money crons — scheduled batches claim a lock and skip rows another run holds, so overlapping runs physically cannot double-deduct, double-notify or double-send.
• Audit born with the event — the audit row is written inside the same transaction as the balance change, so the trail cannot be detached from the money.
• A pending minute is a known state — every money state is explicit and re-drivable: pending, initiated, confirmed, reversed — and stale states are swept by a job, never left to rust.

08

Security: the backend is the attack surface

A frontend can be ignored by attackers; a backend is what they actually want. Every day, the internet scans for the backend, tests its seams and sends its invented payloads. We build backends that treat being attacked as the default condition of being online.

• Authentication that is real — password hashing done properly (bcrypt/argon2), sessions and JWTs with short lives, MFA where the surface matters, and hashed_token/email flows handled with the same care as passwords.
• Authorization at the data, not the button — RLS and owner checks inside the logic, so 'hide the button' is never the actual protection.
• Input is never trusted — validation at the border, parameterised queries everywhere (no string-built SQL), and every payload checked against a shape before it touches a rule.
• SQL injection refused structurally — parameterised statements and an ORM/policy layer that make raw, interpolated SQL impossible by design.
• The OWASP reality — IDOR, privilege escalation, enumeration, rate-limit abuse, mass assignment, CSRF, SSRF, JWT confusion and the payloads the internet actually sends — each one a concrete control, not a slide.
• Secrets that are never in the code — API keys, credentials and salts in managed secrets, fetched at runtime, rotated, and never committed to anything with an audience.
• Rate limits enforced, not requested — per-user and per-IP windows at the gateway and in the logic, so brute force and rapid clicks are structurally slowed.
• Logging that catches the attack — security events recorded with the timestamp, actor and outcome, so an incident has a trial, not a mystery.

We have hardened live platforms after real attacks — enumeration that shipped as a feature, escalation paths hidden behind a button, callbacks that accepted anything signed by nothing. Security is not a report we attach; it is the default posture of every line.

09

The backend security checklist in practice

Below is the concrete checklist we run against every backend we build or audit. Each item is a real control with a real test — the same ones that catch problems on live systems:

• Authentication — bcrypt/argon2 hashing; short-lived sessions; MFA on admin and money surfaces; login rate-limited; password reset tokens one-shot and hashed.
• Authorization — every endpoint checks the caller's own scope (auth.uid() == owner) or an admin role, server-side; RLS at the database backs it up.
• Data isolation — cross-tenant and cross-owner queries refused by the database itself; tests attempt the cross-read and fail the build if it passes.
• Injection — parameterised queries everywhere; no string-built SQL; shape validation on every input; a static scan that looks for interpolated query strings.
• Enumeration resistance — identical responses for 'wrong email' and 'no such user'; registration and password-reset rate-limited; phone/email existence not queryable by the public.
• Secrets — no keys in code or bundles; managed secrets with rotation; service accounts carry least privilege and are not shared.
• Rate limits — per-user and per-IP windows on every public action that costs work or money; enforced at the gateway and inside business logic.
• Callbacks and webhooks — verified, idempotent, and replay-threat-aware: a payment callback carries genuine proof and can never double-credit.
• Logging and audit — every money and admin action logged with actor, timestamp and result; logs append-only and protected from tampering.
• Dependency hygiene — dependencies scanned and current, because the backend is only as honest as the code it imported.

This checklist is not decoration. Every item corresponds to a class of bug we have found in production — including in code that believed it was already secure — and each one has a test that runs before a deploy.

What the checklist means when the thing behind it is a live payments platform:

• Balance and ledger writes run only through privileged procedures — the client can request; the server, under its own rights, decides; a direct insert or update to money tables is not possible from the outside.
• Row-level security at the database is the actual wall — ownership checks (`auth.uid() == owner`) plus RLS mean cross-customer reads and writes are refused by the database, not by politeness in code.
• Clients can never mint balances — wallet and account creation is validated server-side and constrained by CHECK constraints, so a crafted request cannot invent funds.
• Enumeration resistance is tested, not assumed — identical responses whether an email, phone or ID exists, verified by tests that attempt the probe and fail the build if they succeed.
• Credentials live in managed secrets, never bundles — gateway keys are injected at runtime under least privilege, and rotated.
• Every disbursement is gated and audited — identity checks, balances, entitlements and rate limits run before money leaves, and the outcome lands in an append-only log with actor, timestamp and result.
• Distinct accounts for money and entitlements — user float, company revenue and subscription entitlements are tracked apart, so no single write can steal from another pot by accident.

10

Scheduled jobs, queues and the machine that never sleeps

Half of a backend's work happens when nobody is looking: billing cycles advance, reminders fire, stale things get cleaned, exports run, reconciliations run against third parties. This is where products are outrun by their own promises when the jobs crash silently.

• Jobs engineered to run unattended — every scheduled task is idempotent, batched and advisory-locked, so an overlap, a retry or a re-run cannot cause double work.
• Cron with a heartbeat — every job logs its runs and failures to a visible channel, so 'the job crashed at 2am three months ago' is a findable fact, not a haunting.
• Queues absorb the spikes — emails, webhooks and heavy work leave the request path through a queue with retry and backoff, so a burst can never take the checkout down.
• Retries with backoff and a dead letter — a webhook failure retries gently, then rests in a visible dead-letter queue with its payload, not in the void.
• Sweeps that heal the system — stale pending records, expired sessions and stuck statuses are reclaimed on a schedule, so the system recovers itself.
• Exactly-once intent, at-least-once reality — the job is designed to be safe when a retry delivers the same work again, because real infrastructure retries.

We run these exact mechanisms on our own platform — advisory-locked cron, idempotent STK expiry sweeps, bank-funding timeouts, KYC refund schedules. The midnight machine on our own system is the same discipline we ship in your backend.

11

APIs, webhooks and the integrations that connect the business

Backends do not stand alone; they talk to payment providers, banks, SMS gateways, cloud storage and partners' systems. Every integration is a second backend with its own opinions, and the seams are where products lose money if they are built naively.

• APIs as products — versioned, documented, protected and measured, with consistent error contracts so clients can handle failure instead of guessing at it.
• Webhooks verified, never trusted — signatures checked, payloads schema-validated, and callbacks idempotent, so a spoofed or replayed webhook cannot create money.
• OAuth and scoped keys — integrations carry the narrowest token that work allows, revocable per consumer, so one partner's key is not a skeleton key.
• Timeout and retry discipline — every outbound call has a timeout, a retry with backoff and a circuit breaker, so a slow partner is a queue, not a crash.
• The payment rails done properly — M-Pesa via Daraja (STK, C2B, B2B, B2C), PayBill, Till, bank transfer and cards — each with the callback verification, idempotency and reconciliation that money requires.
• Reconciliation over hope — internal ledgers match the provider's statements on a schedule, so a drifted payment is found by the machine, not discovered by a customer.

The seam with a payment provider is where a product's trust is made or broken. We treat every webhook as hostile until proven, every retry as certain, and every penny as reconciled.

12

Performance: fast at 100 users and at a million

A backend that is fast at 100 users and dying at 100,000 was designed for the smaller number. The discipline below is what keeps a product's promises as the data, traffic and customers grow — for real, not on a slide.

• Indexes and queries designed for the real load — query plans reviewed with EXPLAIN before they become the slow wall, and re-reviewed as indexes age with the data.
• Caching at the right layers — hot reads served from a cache whose invalidation is correct, so nothing stale is ever served near money or entitlements.
• Connection pooling and the database underneath — a pool that survives spikes, connection limits understood, and long transactions kept short so the pool never stalls.
• Batching instead of braces — N+1 queries replaced by joined or in-batch reads, so a list page that query-bombs is never shipped.
• Timeouts everywhere — every dependency has a timeout, so a slow webhook or a sleepy partner becomes 'we retried', not 'everything froze'.
• Capacity with numbers — requests per second, tenants per box, rows per tenant — every scaling decision has a measured number behind it, not a hope.

We have profiled backends where one missing index turned a 'simple' page into a thirty-second crawl, and one N+1 query quietly tripled database load. Performance is a design property, found and fixed before the users feel it — and measured again after.

13

Observability: seeing the system while it runs

A backend that cannot be seen cannot be run. Observability is not 'a dashboard'; it is the ability to answer, in minutes, what happened to a specific transaction, a specific user, a specific job — including the ones that failed.

• Structured logs with a story — request IDs that travel through the stack, so one user's journey is traceable across calls, jobs and callbacks.
• Metrics that answer questions — error rates, latency percentiles, queue depth, job success — the numbers a team argues with, watched before they become incidents.
• Alerts that call for help — real thresholds, real escalation, sent to the person who can act — not a wall of noise that everyone has learned to ignore.
• Health checks that mean it — readiness and liveness probes that catch a dying dependency before customers do.
• Traceability of money — every payment, every callback and every reconciliation distinguishable in the logs by reference, so 'where is my money' has a precise, provable answer.
• The post-mortem becomes a change — every incident produces a runbook fix that prevents the class, not just the instance.

On our own live platform we audit payment gateway logs, cron runs and transaction ledgers the same way we tell every client to — because an incident that cannot be seen is an incident that repeats unseen. Observability is how a backend earns the right to run unattended.

14

Testing and deploying the backend

Backend bugs have a way of being discovered by customers — which is exactly what the discipline below exists to prevent. The same standard that governs money governs every release.

• Unit tests for the rules — business logic, fees, entitlements and validations verified by the machine, so a human's change cannot silently break the rules.
• Integration tests for the seams — the API against the database, jobs against the queue, callbacks against the ledger — the places where bugs actually live.
• Concurrency tests — two at once, a retry arriving late, a webhook firing twice — proven harmless by design, in the lab, not in production.
• Migration tests — schema changes run against a copy of real-shaped data, forward and back, so a deploy never loses a column's meaning.
• Security tests as a discipline — the OWASP checklist as executable tests, run before every deploy, not as a quarterly audit.
• Deployments that can roll back — CI/CD that builds, tests and ships in steps, with the rollback path rehearsed — because reverting on Monday is a plan, not a miracle.

We deploy to production regularly — the same pipeline that hardened a payment platform against its failures. A release that passes our backend suite is a release we can ship before lunch and stand behind after dinner.

15

How we build and run backends

The same engineering discipline shown everywhere in this list — visible, iterative and ending with the customer owning everything — is how a backend gets built:

• 01 · The truth first — the data model, the constraints and the ownership rules decided before any screen, because the backend is the product's ground truth.
• 02 · The seams — the layers, the language, the queuing and the integrations chosen honestly, with numbers attached to every choice.
• 03 · The security posture — auth, authorization, isolation and the OWASP controls wired as the foundation, not the finish.
• 04 · The engine — business rules, jobs, hooks and validation built inside the trusted layer, with concurrency safety as the default.
• 05 · The integrations — payment rails, webhooks and partners connected with idempotency, timeouts and reconciliation from the first wire.
• 06 · The observability — logs, metrics, alerts and money-traceability from day one, because a backend that cannot be seen cannot be run.
• 07 · The tests and the pipeline — unit, integration, concurrency and security tests wired to CI/CD with a rehearsed rollback — a deploy that can sleep through the night.
• 08 · The pilot and the run — real traffic, real failures, real fixes — and then the long running, where the machine, the logs and the alerts do the watching.
• 09 · The handover — source, migrations, pipeline, monitoring and documentation delivered — the backend the customer owns and can take anywhere.

You own the code, the database, the pipeline and every key. No lock-in, no hostageware, no 'it only works in our basement' — the backend that runs your business runs wherever you say.

16

Honesty about backends

Because the backend is where trust is won or lost, we are direct about the realities you will meet on every build:

• The database and concurrency are the hardest seventy percent — most real defects live in locked rows, retried calls and crossed transactions, not in the features that demo well.
• 'Offline-friendly' and 'eventually consistent' are promises with real costs — sync, conflict rules and reconciliation are engineering, and we price them honestly.
• Security is a posture, not a dependency — a library update is not a security program; the data isolation, verification and audit discipline decide whether the backend survives.
• Legacy backends can be saved — modernisation is real work done in slices, and we will show you the plan, not hand you a moonlight pitch.
• Bad performance is a design fact, not a mystery — indexes, caching and query discipline make the difference long before any 'infrastructure upgrade' is needed.
• You own the machine — the code and the pipeline are yours, always, so the backend can never become a hostage to its builder.

The toolchain

The backend toolchain

Every layer below is chosen to carry real production trust: a database that argues back, logic that is correct under concurrency, security that treats attack as the default condition, jobs that run unattended and observability that makes every incident findable. These are the exact patterns behind our live platform — money-grade, hardened and scaled.

Lifecycle

The backend lifecycle — from truth to a machine that runs for years

A backend is not a delivery; it is the product's engine, evolving with every feature and surviving every attack. This is the lifecycle we carry every backend through.

Closing

More than development

Backend Engineering is where a product earns the right to run: the truth protected, the concurrency correct, the security real, the jobs faithful and the failures findable. That includes:

The backend is the part of the product nobody applauds and everybody discovers: the moment a rule is bent, a payment is duplicated or a record disappears, the frontend is instantly forgiven and the backend is judged. It is the layer where products are either proven or quietly found out — and it is the layer we have spent a career making boring and reliable.

We build backends the way we build the money platform we run ourselves — locked, atomic, verified, audited and asleep only when it should be. Every query, every job, every webhook and every midnight reconciliation is held to the standard of real trust.