DevOps & CI/CD

03

Shipping is a discipline, not an event

The difference between a fearful organization and a confident one is rarely talent — it is how often the organization practices the most frightening thing it does. If releasing to production happens quarterly, every release is a once-a-quarter near-death experience. If it happens weekly, the first hundred releases have already taught you every way the second hundred can break.

• Every commit is buildable — the instant a change lands on a shared branch, the pipeline compiles it, so 'works on my machine' dies in CI, not in production.
• Every build is testable — automated tests run against the exact artifact that will ship, not a separate 'test build' that quietly drifts from reality.
• Every test gates the release — a failing suite stops the pipeline at the gate; there is no human override at 11pm that sneaks a red build into production.
• Every deploy is reversible — every release knows how to go back to the previous known-good state, and the rollback has been rehearsed, not improvised.
• Every change is traceable — a running version points to a commit, a pull request, a reviewer and a decision record, so 'who changed this?' is a query, not an interrogation.
• Small changes ship more often — a small diff is easy to review, easy to reason about and easy to revert; the big-bang release is an organizational failure, not a badge of honour.

Shipping becomes a muscle. The goal is that deploying production software stops being an exceptional event and becomes the ordinary completion of every working day.

04

The pipeline: from commit to deployable artifact

A continuous integration pipeline is the assembly line under everything else. It runs on every push, fails loudly, and produces a single immutable artifact — a versioned build that can be promoted through environments without ever being rebuilt. For payments, the artifact is not just code; it is the migration scripts, the edge functions, the configuration and the verification tests that have to travel together.

• Fast feedback first — lint, type-check, unit tests and dependency checks run in the early stages so a developer knows within minutes, not hours, whether their change is safe.
• The artifact is born once — CI produces a single build with a content hash, and every later environment consumes that same artifact; we never compile twice and risk two truths.
• Migrations ride with the code — database migrations are versioned files validated in the pipeline against a real test database before anything else happens.
• Verification tools ship in the pipeline — migration dry-runs, schema diff checks and regression suites run inside CI, which is how a money platform catches a breaking change before it reaches the ledger.
• Artifacts are immutable and addressable — every build is stored by its hash and can be re-promoted, re-deployed or rolled back to, years later, from the same one artifact.
• The pipeline itself is code — pipeline definitions live in the repository, are reviewable, and change through the same review process as the software they ship.

When the pipeline is a boring, reliable assembly line, the team's attention moves to what the software does instead of how it gets out the door. That is where the trust begins.

05

Testing that actually gates the release

A pipeline that only compiles is a conveyor belt with no inspector. The tests are the inspector, and they must be meaningful, fast enough to run every time, and trusted by the team. We shape test suites so they protect money systems specifically, not just page renders.

• Unit tests lock the logic — fee arithmetic, state transitions and validators are covered at the pure-logic layer, where the cheapest bugs live.
• Integration tests prove the seams — database calls, edge functions and gateway requests run against real test endpoints, so 'it worked in my head' is not a deployment strategy.
• Money tests are non-negotiable — idempotency re-delivery, concurrent double-spend attempts, escrow lock/release and negative-balance attacks are rehearsed as tests on the payments platform, and we bring the same discipline to client money systems.
• End-to-end smoke tests protect the journey — a handful of critical user journeys (top up, pay, withdraw, receive a callback) run after deployment so the real flow is verified, not assumed.
• Performance smoke tests catch regressions — a query that tripled in cost or a callback that doubled in latency fails a budgeted check before it ever bothers a customer at scale.
• Test flakiness is treated as a bug — a suite that randomly fails is a suite the team starts ignoring; we fix the flake, not the symptoms, so the gate stays trusted.

A test suite is a promise the software makes about its own behaviour. We make those promises specific to money, and we keep them on every single commit.

06

Environments: staging that mirrors production

Most teams debug in an environment that shares almost nothing with production — different data, different size, different neighbours, different config. Invariably, the thing that breaks is the thing that was different. We build staging that is production-shaped: same topology, same configuration shape, same secrets-handling pattern, same gateway wiring — but pointed at test rails and test wallets so mistakes are cheap.

• Structure parity — the same services, databases, queues and cron jobs exist in staging, so a change that passes staging has actually passed through the same machine it will traverse live.
• Configuration parity — staging shares the exact env-var names and wiring as production, differing only in the values, so 'works in staging' actually predicts production behaviour.
• Data-shape parity — schema and realistic data volumes in staging surface slow queries and migration timing that a three-row demo database can never reveal.
• Gateway test rails — on the payments platform, staging talks to the Daraja test environment and the sandboxes of the card, PayPal, Stripe and bank-transfer rails, each with the real callback and verification loop, so callback handling on every rail is rehearsed against a genuine gateway, not mocked into silence.
• Environment drift is monitored — we track when staging and production no longer mean the same thing, and fix it before someone deploys 'like staging' into a different reality.
• One promotion path — the same pipeline promotes the same artifact through the same steps for every environment; if staging and production are promoted differently, they are different products already.

Staging exists so that production is boring. When staging mirrors reality closely enough, the surprise budget is spent in a sandbox instead of on customers.

07

Deployment strategies: blue-green, canary, rolling

Deploying is not one action — it is a family of strategies with different risk, speed and blast-radius profiles. For a payments platform, the default is cautious: new code arrives fully built but only gradually trusted, traffic shifts in measured steps, and the switch back to the old version is a flip, not a rebuild.

• Blue-green deployment — two fully provisioned environments; the new version is deployed and verified on the unused one while the live one still serves, then traffic flips atomically, and rolling back is flipping back — the safest pattern for a ledger that cannot tolerate partial states.
• Canary releases — a small percentage of traffic, or a small cohort of tenants and wallets, is shifted to the new build, monitored for errors and money-path health, then widened in steps until full.
• Rolling updates — instances are replaced gradually with zero-downtime health checks; useful for stateless services where a partial population during the window is acceptable.
• Feature flags decouple deploy from release — the new code ships dark behind a flag, and the flag turns on for a cohort, so a bad feature is reverted with a config change, not an emergency redeploy.
• The strategy is chosen per service — the ledger, the edge functions and the app shell each get the strategy their blast radius deserves; a landing page does not need a canary, the wallet engine never gets a hard cutover.
• Every deploy still has a rollback — whatever the strategy, the moment before the deploy is a recorded, restorable state, and the rollback path is part of the deploy plan, not an afterthought.

Deployment strategy is risk management made mechanical. We pick the pattern by asking who gets hurt if this is wrong — and money always gets the gentlest pattern there is.

08

Rollback: rehearsed, fast and boring

Every team claims it can roll back. Very few have ever actually rolled back a money system under pressure. Rollback is not a button; it is a rehearsed capability with four hard constraints — speed, completeness, data safety and the absence of shame. A rollback that takes an hour while wallets are frozen is not a rollback; it is a rescue mission.

• Data stays attached — the previous version of the code must be able to read the current database, so schema migrations are designed to be backward-compatible on the day they ship, not a week after everyone forgot.
• Rollback for databases is pre-planned — every migration has a reverse path, and destructive changes (column drops, table rewrites) are sequenced so the door back stays open for at least one full release cycle.
• Blue-green makes rollback a flip — with the previous environment still warm, rolling back is changing the pointer back, measured in seconds, not a battlefield rebuild.
• Rollback drills are scheduled — the team actually practices reverting on staging on a calendar cadence, because the rollback that has never been run is the rollback that fails when it matters.
• Automatic triggers watch for the bad deploy — error-rate and money-path health loops detect the regression and either alert the on-call engineer or execute the rehearsed rollback automatically.
• Post-rollback is rehearsed too — the team knows the order of verification after a revert: check the money path first, then the ledger, then the app, then the logs — so 'we reverted' actually means 'we are safe'.

Rollback is the seatbelt of deployment: you hope never to need it, and you insist it works every time. We build it, rehearse it and monitor it until rolling back is boring.

09

Observability into deploys: the deploy is not finished until it proves itself

A deploy is a moment; a rollout is a claimed state. The honest question after any deployment is not 'did it happen?' but 'is it healthy?' Observability into deploys answers that question — comparing the moments before and after the release so a regression is caught in minutes, not found at month-end reconciliation.

• Deploy markers land in the telemetry — every build stamps its logs, metrics and traces, so 'which version handled this callback?' is answered by the data, not by memory.
• Before-and-after comparison is automatic — error rate, latency, throughput and money-path success are compared across the deploy boundary; a regression that did not fail a unit test still fails the health check.
• On the payment platform, the money path is the health signal — deposit confirmations, settlement callbacks and ledger balances have their own health dashboards, so a deploy that breaks money verification is seen before a customer reports it.
• Alerts carry the deploy context — the notification includes which build, which environment and which change candidate caused the anomaly, so the on-call engineer starts from context instead of an empty terminal.
• Slow-onboarding metrics are watched — login success, onboarding completion and transaction attempt rates dipping after a release are product-level regressions that plain 'server up' checks will never see.
• Release dashboards are reviewed, not just automatic — the smoke tests run, the health checks pass, and a human glances at the release dashboard before signing off, because machines alone have never been a substitute for one careful read.

The deployment pipeline should answer 'is it better now?' with data, not hope. We treat observability as part of the release, not an optional add-on after it.

10

Database migrations: the riskiest deploy of all

Code can be reverted; a bad migration can rewrite or lock the data beneath everything. On a payments platform the database is the ledger, and the ledger is the product — so migrations get a discipline that code deploys never need. Every migration we ship is additive-first, backward-compatible and rehearsed.

• Additive changes ship before dependent code — a new column or table is created in an earlier release than the code that reads it, so neither the old code nor the new code ever queries something missing.
• Destructive changes ship after a full cycle — columns, constraints and indexes are dropped only once no running version references them, which on the payment platform means 'never drop before confirming safe.'
• Locks and size are respected — a migration that holds a table lock is a migration that has blocked a wallet read on an active evening; heavy migrations run in low-traffic windows and in batches.
• Backfills are batched and idempotent — data migration runs in bounded chunks with the ability to stop and resume, because a single 4-million-row UPDATE on a live table is an outage wearing a migration costume.
• Staging runs the real migration on production-shaped data — the exact scripts, in order, against realistic volumes, so timing surprises are exposed in the sandbox, not on the ledger.
• Every migration has a documented reverse — the file records what undo would look like, and automatic schema-diff guards compare the migration's result to the intended schema before it is allowed toward production.

We have seen the ledger-locked, callback-queue-stalled, frozen-wallet migration play out in the industry more times than we can count. We engineer so it does not happen to the platforms we build and operate.

11

Secrets and configuration: never in code, always injected

The pipeline needs credentials to deploy, the app needs secrets to run, and the payment gateway needs keys to move money. Compromised secrets are how most serious breaches begin — and on our platform a leaked Daraja credential could move real money. That is why secrets are managed, injected and rotated, never committed.

• Zero secrets in the repository — API keys, database passwords, gateway credentials and signing keys never appear in source, config files, build logs or Docker layers.
• Secrets come from a managed store — edge-function secrets, database passwords and gateway keys live in a secrets manager, injected at runtime with permissions scoped to the workload that needs them.
• Per-environment secrets are separated — staging keys cannot touch production wallets; the credential that moves real money and the credential that moves Monopoly money are different keys.
• Rotation is a routine, not a panic — keys are rotated on a schedule and the pipeline re-verifies after rotation, so a leaked key is survivable because the old one is already dead.
• Honest checkout discipline — repository scanning and pre-commit hooks look for accidental keys, and leaks are treated as incidents with immediate rotation and audit, not quiet panics.
• Least privilege is the default — a CI token can only build, a deploy role can only deploy, and a payments function can only call the payment endpoints it owns; no service holds keys to everything.

On a money platform, secrets management is not a hygiene checkbox — it is the property boundary of the business. We treat it that way, for ourselves and for every client.

12

Release cadence and the humans around the machine

The pipeline is machinery, but delivery is a team sport. The best pipeline on earth is defeated by a team that does not trust it, or an organisation whose release process depends on tribal knowledge. We build the human layer with the same care as the machine layer.

• Release windows and runbooks — a defined, communicated schedule for releases that touch user-facing or money-moving surfaces, with a written runbook of what happens at each step, known to everyone.
• Back-to-back deploys are the norm — teams ship small, often, at chosen cadences, so the muscle of releasing is exercised continuously rather than remembered nervously once a month.
• Feature flags keep deploy and release separate — the deploy is a technical event; the release is a business decision; flags let the two move independently, which gives back the ability to ship safely any hour.
• Code review pairs with pipeline review — a human reads the diff and the pipeline verifies the build, and both are treated as mandatory, not optional; a merge that bypasses either is an incident.
• Post-deploy checklists are short and real — after a release the team confirms the money path, the health dashboards and the supporting crons, then gets back to work — the check takes minutes because the pipeline did the heavy lifting.
• Blame goes to the process, credit to the team — when a release goes wrong, the question is 'what failed in the machinery?' not 'who pressed the button?', so the system gets fixed instead of scapegoats manufactured.

Great release culture is invisible — it is simply what a team expects of itself. We build the machine that makes that culture the path of least resistance.

13

The pipeline we run for KodiiPay: money-moving deployment

This is not a catalogue of good ideas; it is how a real payments platform deploys. KodiiPay moves real money hourly, so its release pipeline is the discipline this whole page describes, applied with a money-specific edge.

• Edge functions deploy together — the gateway-facing functions (STK push, C2B callback, B2B payout, withdrawal confirmation) deploy through the same pipeline as the app and the schema, so a mismatch between function versions and ledger procedures never silently arises.
• Declarative schema is the single source of truth — the database, RLS policies, SECURITY-DEFINER procedures and cron jobs are defined in versioned SQL, deployed from the repository, so the ledger has a reviewable ancestry, not a click-built reality.
• Money-path health gates the release — after a deploy, the pipeline verifies that callbacks still verify, wallets still balance, and a test transaction still completes the real verification loop before it calls the release done.
• Rate limits and escrow travel with the release — the rules that cap abuse and escrow funds are part of the shipped schema, so a deploy cannot accidentally strip the financial controls from production.
• Realtime infrastructure is reviewed like code — publications, subscriptions and monitoring configuration are versioned, because a UI change in the dashboard has the same blast radius as a UI change in the app.
• Rollbacks are genuinely inherited — because the schema and functions are declarative and versioned, rolling back the application also rolls back the procedures it depends on, to the exact known-good combination.

Every platform we ship DevOps for inherits this standard. If it moves money, it deploys the way money deserves.

14

The honest limits: when CI is overkill

We will tell you when you do not need the full machine. A landing page, a brochure site or an internal tool with two users does not need blue-green deployments, canaries and a deployment dashboard; the ceremony would cost more than the product. The honest discipline is matching machinery to stakes.

• The pipeline scales with the risk — a static site ships with a simple build-and-publish flow; a money platform ships with the full gauntlet; everything in between is sized to its blast radius.
• A simple deploy is not shameful — for a product with nobody depending on uptime, a reliable script that rebuilds and republishes is better engineering than a CI cathedral nobody maintains.
• The pipeline must out-earn its cost — every stage adds time and complexity; we ask whether it prevents a failure worth more than it costs, and we cut the stages that fail that test.
• Complexity that is unmaintained is worse than none — a broken, mysterious pipeline teaches a team to deploy around it manually, which is the most dangerous habit of all; we keep pipelines simple enough to be genuinely owned.
• The staging investment is matched to relevance — a cash-register demo with production-shaped data stages only what its actual risk profile needs, not the full money-platform parade.
• The rules are the same in miniature — even the smallest project keeps no secrets in code and every commit buildable, because those two rules are free and always worth keeping.

We will name the moment honestly, whatever it is: 'you need the full pipeline, period' or 'a script is correct here'. The machinery should fit the mission — and we will tell you which mission you actually have.

15

Integration with monitoring, alerting and observability

Delivery and operations are the same lifecycle wearing different hats. The pipeline that ships a release is the sibling of the monitoring that watches it; we wire the two together so deployment events and runtime signals share one vocabulary.

• Deploy events are operational events — a release records itself into the incident timeline, so a page after a deploy already knows which candidate is suspect.
• Dashboards compare release-to-release — key metrics carry the version label and the deploy marker, so 'was this better before the release?' is a query, not a debate.
• Alert routing follows the on-call rota — severity-aware routing sends a broke-money-path alert to the engineer who actually holds the pager, with the runbook attached.
• Logs, metrics and traces are one system — a slow callback is traced across the edge function, the database and the gateway so the bottleneck is visible, not guessed.
• Smoke tests after every deploy — the pipeline verifies synthetic money-path transactions in the live environment, so the whole loop — STK push, callback, ledger credit — is proven on every release.
• Observability pays down over time — we instrument what the platform actually needs, tune alert noise, and archive the data that has outlived its usefulness, keeping the signal honest and the cost sane.

The pipeline ships the code, and observability watches the code; together they are the operating system of delivery. We build both, and they share a brain.

16

How we run DevOps engagements

We do not hand over a pipeline PDF and disappear. We build the machinery, run it beside the client, and then hand it over with the humans trained to own it. The engagement shape is honest and staged.

• Audit first — we map the current delivery state: what ships manually, what breaks, what the release history hides, and what the money paths need most urgently.
• Establish the foundation — versioned code, a first CI pipeline, secrets hygiene and the smoke tests that protect the critical journeys, landed in the first sprint.
• Mirror the environments and add staging — production-shaped staging becomes the place where the client learns to rehearse the hard releases before the real calendar demands them.
• Add the deployment strategies that earn their place — blue-green where the ledger needs it, canary where cohorts exist, flags where release timing is a business decision.
• Wire monitoring and runbooks — the alerts, dashboards and on-call documentation arrive with the pipeline, so operating the release is possible on day one.
• Train and hand over — the client's team runs the pipeline, we coach the drills, and the ownership transfers cleanly with documentation the next hire can read.
• Stay for the honest work — we remain available for the incident week, the migration night and the quarterly pipeline review, because handover is a process, not a ceremony.

We have run money-platform delivery long enough to know the moment a handover is actually complete: it is the first deploy the client ships without calling us, and it goes fine. That is the day we aim for from the start.

17

Honesty about continuous delivery

Continuous delivery is one of the most oversold ideas in software, and one of the most genuine. Here is the honest version — the parts that survive contact with real teams and real money.

• A pipeline cannot fix a bad product — deploying faster just ships badness faster; CD multiplies what exists, it does not judge it.
• Rollback is not always clean — distributed systems, mobile apps in the wild and irreversible ledger writes do not all 'flip back'; we design the reversible paths and are honest that some changes are one-way.
• CI excludes the hardest customer — your mobile app in production cannot roll back like a server; app-store deployment is inherently slower, so the pipeline keeps native change small and the backend the load-bearing half.
• Team culture is the load-bearing wall — all the machinery is decoration if the organisation still punishes mistakes or rewards hidden work; we build the process and the confidence together.
• Green builds can still ship broken software — tests are an approximation of the future; observability and honest post-release review are the second wall, always; we do not claim tests alone make a release safe.
• Vendor pipelines are not the same as owning delivery — buying CI minutes is not the same as owning the release discipline; we sell the discipline and the machinery together.

Continuous delivery is not magic — it is a thousand honest, boring decisions repeated until they are automatic. We make the honest decisions for you, and we stay long enough that the team makes them without us.

The toolchain

The DevOps & CI/CD toolchain

The same classes of machinery that keep our money platform deployable to weekly, with specific tools for each layer. Every choice is a tool because the discipline demanded it.

Lifecycle

The DevOps lifecycle — from audit to owned pipeline

Whether we are building delivery from scratch or rescuing a release process that stops the product, the work moves through the same twelve stages — the ones that keep our own payments platform shipping safely.

Closing

More than development

DevOps and CI/CD are how a product keeps shipping after the founders stop pressing every button themselves. We bring the discipline that keeps our own money platform live — and we bring it to yours. The work includes:

Deployment is where software either keeps its promises or betrays them — and delivery discipline is what determines which.

We deploy a live money platform weekly, rehearse its rollbacks, and operate its alerts. When your product deserves delivery that strong, that is the machine you get.