Data Migration

03

Data migration is an operation, not a procedure

A migration is commonly treated as a technical task: write a script, run it, verify a count, call it done. We treat it as an operation — a full-time project with named phases, owned risks and a definition of done anchored to the business trust in the data, not to a script exit code.

• Named phases — inventory, profile, map, extract, transform, load, validate, reconcile, dry run, cutover and verify; each has a gate before the next opens.
• Owned risks — every risk, a table with no owner, a column with mixed meanings, an undocumented field, is a named item with an owner and a decision, not a hope.
• Definition of done is parity — the migration is finished when old and new agree to the last record, the last reference and the last penny, and someone signs it.
• Meaning is a first-class citizen — a migration that preserves rows but loses their meaning has only moved data around, and that is not the operation we run.
• Idempotent machinery — every script can be re-run safely, because a migration that cannot be retried without side effects is a migration that cannot be trusted.
• An audit trail of its own — the migration itself is logged: what ran, when, how many rows, what changed; a migration without a log is un-auditable by definition.

A migration is paid for in trust, and trust is earned by evidence. We run migrations as operations with a paper trail, because the data is only as good as the proof it arrived correctly.

04

The inventory — what there actually is

You cannot move what you cannot name. Before a single row is extracted we build an inventory of everything the source holds — tables, views, files, documents, archives, the spreadsheet behind the door, the export that finance started three years ago. The undocumented sources are exactly the ones that hide the truth.

• Every source, honoured — live databases, old backups, CSV exports, spreadsheets, scanned files and paper-going-digital, each discovered and listed, not assumed.
• Row counts and volumes — how much there actually is, so the move is dimensioned like the real thing it is, not like the sample on a demo server.
• Data quality snapshot — duplicates, nulls, typos and impossibilities profiled up front, so the migration plan can decide what to carry, clean and flag.
• Owner and meaning per data set — who actually understands each part of the data; if no one owns it, that is flagged as the risk it genuinely is.
• The hidden dependencies — which data feeds reports, integrations and regulators day to day; what looks orphaned in the database is often load-bearing in the business.
• Sensitive data sorted — what contains personal, financial or regulated information, so handling, masking and retention rules ride with the records from the start.

The inventory is the migration contract with reality. It names what the move touches — including the undocumented corners where a careless migration quietly loses history.

05

Extraction from any source, documented or not

The hardest sources to extract are not the exotic ones; they are the ordinary ones nobody documented properly — the database with no schema notes, the vendor system with a read-only export, the Excel file where one sheet is the real truth and the others are decoration. Extraction reads all of them, and reads them reliably, which is itself a discipline.

• Direct and read-only approaches — database queries, vendor exports, CSV and XML pulls, each matched to what the source actually permits, never guessed from its name.
• Reading, not rewriting — extraction changes nothing in the source; the legacy keeps running untouched while its data is being read, and that is a requirement, not a preference.
• Full snapshots for truth — the extracted raw copy is kept as a sealed snapshot, so the migration has a fixed, reviewable source instead of a moving target.
• Deliberate reads to the last row — batches, pagination and checkpoint resume, so large or fragile sources are drained completely even when the network drops.
• Encoding and character honesty — encodings, line endings and date formats are handled explicitly, because the classic silent loss is content that arrived but changed meaning on the way.
• Extraction provenance — each extracted set is logged with source, time, volume and checksum, so where this came from and is it complete is answered from the log, not the memory.

Extraction is where the migration is won or lost in microcosm: if the read is unreliable, everything downstream — transform, validate, reconcile — is polishing an untrustworthy foundation.

06

Schema mapping — the decision layer

The schema map is where every assumption becomes visible. Each source field is matched to a target field, and every transformation is written down: the status codes that change value, the IDs that get re-pointed, the dates that shift format. Nothing is left to the moment when someone spots a column in the new system and thinks it means something it does not.

• Field-by-field maps — a row per source field with target, rule and rationale; the map is a document the business can read, approve and argue with.
• Transformation rules attached — every conversion is explicit and reviewed: a coded status, a merged column, a currency normalization.
• Ambiguity surfaced, not guessed away — a field with shared meanings is flagged for an owner decision before load, never assigned a neutral default that hides the truth.
• ID and reference strategy — every foreign key, every cross-reference and every external ID has a mapping that keeps relationships alive in the new world.
• Auditability of every change — the map shows the before and after of each meaning, so the business can answer what happened to this column without a code tour.
• Assumptions are approvals — each risky mapping carries an owner name; a migration decisions are signed, not inferred, which is what makes them defensible afterward.

The schema map is the paper contract between two worlds. When every field is decided, reviewed and signed, the move becomes execution of an agreement rather than a leap of inference.

07

Transformation that preserves meaning

Transformation is where shape-changes happen, and shape is exactly the wrong thing to preserve: what must survive is meaning. A status that was a single letter must become its spelled word, a region code becomes a name, and a proportion that was stored as a percentage must remain the same proportion after the decimal moved. We transform so a person reading the new data understands what the old data meant.

• Value translation tables — code-to-meaning conversions are explicit, reversible and documented on both sides; a code equals a meaning is a record, not a convention.
• Units and formats normalized — currencies, dates, weights and identifiers converted with their context intact, so a figure is the same figure in the new system.
• Derived fields recomputed correctly — totals, balances and proportions are recalculated from their parts under the same rules, never copied blind from a column that might already be stale.
• Null versus zero versus absent — the three are different facts and the transform respects the difference; a never-recorded value never becomes a false zero that a report then sums.
• History keeps its timestamp — event times and effective dates survive the move, so the story can be reconstructed with the same chronology it had before.
• Regenerated references — new system IDs are mapped cleanly and each legacy reference is preserved alongside, so an old record can always be found by its old name.

The test of a great transformation is simple: a domain expert who knew the old data can read the new data and recognize every record, every status and every exception.

08

Reference integrity — the relationships survive

Two tables that referenced each other must still reference each other after the move, or the new system does not know who owes whom, which unit belongs to which property, which invoice belongs to which customer. Reference integrity is the difference between migrated data and data that behaves like the business expects it to.

• Foreign keys verified end to end — every relationship the source enforced is asserted again in the target, in both directions, before the move is accepted.
• Orphan hunting — records whose parent is missing are found and decided: repaired, excluded deliberately, or carried with a documented dangling reference.
• Cross-references rebuilt — join tables, link records and composite keys are migrated as the plumbing they are, so the new system queries return the same joined truth.
• External ID continuity — M-Pesa transaction IDs, PayBill references, card-authorization codes, bank references and receipt numbers that other systems and gateways cite keep travelling in the new world under their own names.
• Parent-child load order — the load is sequenced so a child never lands before its parent, and where a temporary reference is needed it is tracked and resolved, never left as a placeholder.
• Referential verification at the end — a final pass runs the same join the business actually uses against old and new, and the answers are the same.

Data that has lost its relationships is not data; it is a pile of independent facts with no knowledge. We verify that the new system knows the same connections the old one did — because those connections are the business.

09

Lookups and statuses translated to the new language

Every migrated system carries its own vocabulary — statuses, reasons, types, codes. When the new system speaks a different language, every record must be translated, and the translation must be complete and consistent, or the new system will show records in states the business does not recognize.

• A complete code dictionary — every distinct value in every lookup column is enumerated — all of them, including the stray values that should not exist but do.
• New-status mapping agreed — how each legacy status becomes a new one is a business decision signed by an owner, because a wrong mapping means wrong behaviour in the new system.
• Unknown values are caught, not invented — a code the dictionary does not know is surfaced and decided; it is never allowed to become a generic other that hides the truth.
• The dictionary travels — the old-to-new translation is stored as data with its own audit, so future lookups and reports can always decode the history.
• Enums and constraints in the target — the new database enforces the translated vocabulary, so a value the business never defined cannot quietly appear later.
• Translated meaning, not translated labels — pairs of statuses that meant the same thing in different contexts are mapped by their intent, so behaviour stays correct even where the labels differ.

A filtered report in the new system is a promise about the data it returns. If the statuses were translated wrong, that promise is quietly broken in every report the business runs.

10

Validation gates — the data refuses to enter wrong

Validation in a migration is not a report you glance at; it is a gate the data physically cannot pass without meeting the rules. The database, the load process and the pre-load checks each enforce their layer, so invalid data stops at the boundary instead of entering the new system dressed up as truth.

• Row-level rules — required fields, value ranges, format checks and referential existence are enforced per record before it is accepted.
• Set-level rules — counts, totals, uniqueness and aggregate invariants are checked per load batch, so the batch is internally consistent is proven, not assumed.
• Business-rule checks — the invariants the company actually depends on — a balance can never go negative, a paid invoice is never re-opened, a tenant has exactly one primary unit — asserted in the new system own terms.
• Sampling on top of totals — automated rules are complemented by human spot checks on real records, because rules catch what was specified and people catch what was missed.
• Failures are visible and bucketed — every rejected record carries its reason and lands in a review queue; nothing is swallowed, nothing is silently logged.
• The gate is replay-safe — re-running validation over the same load produces the same verdicts, so the sign-off rests on determinism, not a fluke.

The cleanest way to end up with a clean database is to refuse to fill it with dirty data in the first place. Our gates exist so a migration quality is decided at the boundary, where it is cheap — not in production, where it is expensive.

11

Reconciliation — the parity check is the product

The single most honest sentence in data migration is this: the parity check is the product. Every other piece of the operation exists to make old and new agree, and the migration is not finished until they agree — to the last record, the last reference, the last penny. We build the parity engine the week the migration plan is born, not as an after-action report.

• Count parity — row counts per table and per filter agree between source and target; a difference to the last row is found in minutes, not discovered by a customer.
• Total parity — sums of every meaningful numeric column agree; totals, balances and amounts are the arithmetic truth of the move.
• Checksum parity — hashes of full record sets agree, catching the invisible corruptions that counts and sums both miss.
• Reference parity — every ID, foreign key and external reference resolves to the same thing on both sides, so relationships survived the move.
• Sampled record parity — spot checks pull the same record from both worlds and compare it field by field, in the hands of a person who knows the data.
• The mismatch is a workflow — when parity fails, the specific differing rows are surfaced with their reasons, so answering which records and why is a query, not a scandal.
• For money, to the penny — balances, ledgers, floats and gateway references reconcile to the cent; a rounding drift in migrated money is an incident, not a debate.

We have reconciled migrated money on our own platform, where the price of a missed row is real cash and real trust. That is why the parity check is built, budgeted and non-negotiable in every migration we run — the move is only done when old and new say the same thing.

12

Migrating money data — the highest-stakes kind

There is a category of migration that tolerates no rounding: money. Wallets, transaction histories, ledgers, floats, pending settlements — a single mis-mapped row is not an inconvenience but a real loss. Migrating money data is payment engineering, and we run it with the exact machinery our live payments platform uses every day.

• Balances move atomically — a wallet row, its ledger history and its audit references migrate together; a customer balance is never changed by the platform changing.
• Ledger continuity — every transaction retains its reference and its place in the ledger, so the story behind each balance is as reconstructable after the move as before.
• Pending money is settled or traced — in-flight STK pushes, pending card captures, unsettled bank transfers and escrowed holds are matched to their gateway and resolved at cutover; nothing is orphaned while pending.
• Idempotent by construction — every money-load step can be re-run without double-crediting; a retried batch is a safe retry, not a duplication risk.
• Three-way truth — customer balances, the platform ledger and the gateway statement are checked together; the money matches the move from all three angles.
• Escrow and holds survive as holds — a KYC hold, a subscription gate or a freeze is a property of the money and migrates as such, never as a silent release.
• An audit trail a regulator accepts — the migration of money is logged like a transaction itself: what moved, when, by which script, under whose sign-off.

Money data is the same discipline as our own platform: atomic, idempotent, reconciled to the penny and provable in minutes. Everything else in this document is preparation; for money, this is the whole game.

13

Idempotent re-runs — the retry is a friend

Migrations go wrong, and the response is to fix and re-run — not to improvise. Every load, transform and validation step is built idempotent, so re-running it over the same source produces the same result without duplication. The migration can be repaired by re-running, which is the only sane way to repair something you trust with history.

• Deterministic steps — a step given the same input produces the same output; chaos is eliminated from the machinery itself, not managed around it.
• Natural keys and upserts — records are keyed by what they actually are, so a re-run updates rather than duplicates; the re-run works with the data, not against it.
• Checkpoint and resume — a large load that fails in the middle resumes from the last committed point, so progress is preserved and nothing redoes what it already did.
• Cleanup safe to re-run — the reset-slice path is itself idempotent, so the repair sequence can be repeated without compounding the damage.
• Dry-run of reparability — we prove the repair path in rehearsal, not for the first time at 2am on cutover night.
• Versioned migration scripts — each fix is a named, reviewed re-run with its own log; the migration history is itself a record the business can audit.

The unspoken requirement of any migration that carries trust is being able to try again safely. Idempotency is what makes trying again a working recovery instead of a gamble.

14

Dry runs — the rehearsal is the real thing

Nobody's first landing should be the real landing, and no migration's first full execution should be the go-live. We run dry runs that exercise the entire operation — extraction, transformation, load, validation, reconciliation, the full freeze window — against the real data shapes and real volumes, so what the business sees on cutover night is a repeat of something we have already proven.

• Full-scope rehearsals — the dry run is the real run twin: same sources, same scripts, same volumes, same checks; only the destination and the calendar differ.
• The freeze window is rehearsed too — the cutover timing, coordination and holds are practised, because the choreography fails as often as the scripts do.
• Parity is asserted in the rehearsal — the dry run finishes with the parity check against a source snapshot, so will it reconcile is answered before it matters.
• Failure is injected — on purpose, we make a step fail in rehearsal to prove the recovery path works for real, in seconds, without a panic.
• Rehearsal results are written — timings, row counts and exceptions from the dry run become the baseline the real cutover is compared to.
• The production cutover is the second run — the business gets the confidence of having watched the move succeed once; go-live is a repeat, not a debut.

A rehearsal that ends with parity and a clean recovery is the closest thing a migration has to certainty. We spend the nervousness in the rehearsal so the business does not spend it on the night.

15

The frozen cutover — controlled, coordinated, short

The frozen cutover is the one disciplined window during which data stops changing at the source, everything moves, and the switch is thrown. Its entire purpose is to buy determinism: a single, bounded moment when old and new can be compared with nothing moving underneath either one.

• A named freeze window — the source is held stable for a defined period, agreed with the business and scheduled on the calendar that matters to them.
• Coordination is written down — who holds the source, who runs the load, who runs the checks, who holds the go or no-go and who reverts; every seat named in advance.
• The go or no-go is explicit — a defined check passes or the window is aborted and the backout runs; there is no improvisation at 2am.
• The final load is small and verified — whatever changed in the last hours before the freeze is migrated in a final bounded pass, the last delta before the switch.
• The switch is a checkpoint, not a ceremony — one named step, idempotent, observable and reversible; the world flips, and the parity check runs while it settles.
• Everything is timed and reported — the window duration, volumes and results are written down, so the next freeze, and the next migration, is planned from evidence.

The freeze is where a migration earns its discipline or spends it — a short, named, rehearsed window beats a long, improvised weekend every single time.

16

Backout that actually works

Every cutover has a stated exit unless it has a rehearsed one — and an un-rehearsed backout is a story you tell after the fact, not a plan you execute during the crisis. We design the rollback the same week we design the move, and we prove it in the dry run, so the exit is a routine, not a rescue.

• A defined trigger — the conditions that call for rollback are written in advance, such as parity failed, a showstopper, a gate refused, so going back is decided by criteria, not by adrenaline.
• Restore by construction — the backout returns the world to the state it was in before the cutover; source and integrations resume exactly as they were.
• Verification of the rollback itself — after the backout, we re-prove the pre-cutover state with the same parity eyes, so being back is verified, not assumed.
• Rollback is rehearsed like the move — the dry run exercises the reversal path once, so cutover night go-back is a repeated move, not a first attempt.
• Reversible in stages — where possible the rollback is per-slice: only the failing slice returns if only the failing slice is broken, so a minor fault does not cost a full abort.
• Post-mortem without shame — an aborted cutover is a data point; the parity numbers and the failure reason are captured as the raw material of the next, better attempt.

A rollback that has never run is a rollback that has never been tested — which is to say, a rollback you do not have. We rehearse the exit so that going back is boring, and boring is what you want when it is 2am and the parity check says no.

17

Post-migration verification and sign-off

The cutover ends, and the migration is still not finished: the move is only done when the new system has been verified in its own working life, and an owner has signed that the old world truth has genuinely transferred. Sign-off is the formal bridge between the data arrived and the business runs on it.

• Real-work verification — the reports, searches and workflows the business actually runs are exercised against the new data; it is verified doing real jobs, not modelled on paper.
• Parity re-checked after the switch — old and new still agree after a period of real use; the final closeness of the move is proven with the system working, not just after loading.
• Exceptions are tracked to zero — every flagged record, rejected row and owner decision is followed to resolution or an accepted, documented exception; none are left in the drawer.
• The old system is parked, not assumed away — access is cut gradually with read-for-audit preserved, until the sign-off says the new system is the truth.
• A named owner signs — an accountable person certifies each scope on evidence; sign-off is the permanent record that this data transfer was deliberate.
• Lessons are written down — the migration surprises, timings and fixes become the playbook for the next one, and there is always a next one.

Verification without sign-off is a report no one owns; sign-off without verification is a signature on hope. We deliver both, attached to each other.

18

Honest limits of data migration

Because we have migrated money and history for real, we are direct about what a migration cannot fix. It cannot invent data that was never captured, cannot restore meaning that was lost before we arrived, and cannot make bad source data clean by wishing. The honest conversation about those limits happens before the contract, not after the cutover.

• Garbage in, garbage out, elegantly moved — if the source is corrupt, duplicated or missing, the perfect move produces a perfect copy of an imperfect truth; we profile it and say so up front.
• We cannot reconstruct lost context — a column whose meaning no one alive can explain can be flagged, mapped by best judgement and documented as such — never silently guessed.
• We cannot un-drop history — if the source retention already deleted old records, the new system inherits that hole; we restore what backups permit and say plainly what is gone.
• Cleanup decisions cost time and money — deduplicating and repairing dirty data is a real phase with real effort; we price it honestly rather than folding it silently into the migration.
• A perfect move does not fix a bad process — if the business that feeds the data will feed garbage in later, we say that the migration job is the move, and the process is a separate change.
• The move is ours; the meaning is both of ours — a value we map on best evidence still needs an owner confirmation where the source was ambiguous; the business signs the decisions only it can make.

A good migration leaves you with the truth you had, intact — and a clean, reconcilable place to build the truth you want. We deliver exactly the first, and we are honest about exactly everything else.

The toolchain

The data migration toolchain

A migration is a pipeline with a parity engine — every layer exists to carry records, preserve meaning and prove the move. This is the toolkit we run to take data from any source into a new home with its story intact.

Lifecycle

The data migration lifecycle

Every migration we run follows the same arc — inventory, map, transform, prove, cut over and verify — with the parity check as the centre of gravity the whole operation rotates around.

Closing

More than development

Data migration is where history either survives or quietly dies — and the difference is machinery, not luck. We move data the way we move money, and the move is only done when old and new say the same thing. That includes:

A migration is not a script that runs once — it is an operation with a paper trail, a rehearsed exit and a definition of done measured in parity, not in rows moved.

We have migrated live money and history for real. When your data is the story your business needs tomorrow, it moves through machinery that has survived the highest-stakes kind of move — and it arrives intact.