← Selected work

Client implementation / Python · Salesforce

A reusable migration pipeline
you can trace and trust.

Designed for reuse across client projects, the framework uses CSV as its input contract and keeps source relationships, field mappings, and destination connectors at clear boundaries. This implementation uses SAP Commerce exports and Salesforce; other destination APIs require their own connector.

Source
CSV exports · SAP Commerce here
Target
Salesforce · current connector
Design focus
Run-level traceability
Core stack
Python · pandas · DuckDB

Client and environment identifiers are omitted. The pipeline starts from exported files; it does not connect to a live SAP instance.

01 / Architecture

Two ways to prepare.
One controlled path.

A direct run or a prepared batch enters the same validated, run-scoped pipeline.

File-based source boundary SAP Commerce CSV exports Customer · Address · Order · related files
A / Root-batched run

Partition related source records

A configured relationship graph lets DuckDB plan Customer-rooted batches. Related files stay grouped, and each batch receives its own raw inputs.

prepare_root_batches.py
B / Direct single run

Start with one source set

A run launcher creates one run-scoped raw directory for the supplied CSV files.

start_migration_run.ps1

Both preparation paths converge at registry validation

  1. 01 Validate Object registry and dependencies
  2. 02 Resolve Source relationships and run scope
  3. 03 Transform Configured target object CSVs
  4. 04 Load Current connector: Salesforce Bulk API 2.0
  5. 05 Reconcile Run evidence and sign-off checks
Salesforce · current target Run-specific records available for verification
Optional profiling and source-dictionary notebooks support preparation; they are not required stages in the shared pipeline.
One identity follows the run

A unique migration_run_id scopes inputs, intermediate data, loader evidence, reports, and the Salesforce MigrationRunId__c field.

Designed for reuse

A reusable core. Client and platform specifics at the edges.

Source schemas, relationship rules, field mappings, and run settings are configured per migration wherever possible, rather than embedded in the shared processing stages.

Input / source boundary

Start with straightforward CSV files

SAP Commerce is the source in this implementation, not a required live dependency. Another source can use the same pipeline when its CSV files are described by the relevant schemas, relationships, and mappings.

Output / connector boundary

Adapt transformed CSVs to a destination API

The current connector loads Salesforce-shaped CSVs through Bulk API 2.0. A connector for another platform could consume target-shaped CSVs and translate records into that platform’s API requests.

Scope: Salesforce is the implemented destination here. Each additional platform would need its own connector, authentication, mappings, error handling, and verification.

02 / Run lifecycle

Make every stage
an explicit contract.

Inputs, outputs, and control points are visible from profiling through sign-off.

  1. 01

    Profile the source boundary

    Inspect schemas, row counts, nulls, duplicates, delimiters, and file volume to establish a data-quality baseline.

  2. 02

    Establish run identity

    Create a canonical run root and carry the same identifier through each stage and into target records.

  3. 03

    Partition when needed

    Validate the relationship graph and split Customer-rooted records into restartable batches; standalone file graphs get independent roots.

  4. 04

    Resolve source relationships

    Reconstruct links across flat files, scope child data, and record missing, orphaned, or ambiguous relationships.

  5. 05

    Assemble deterministic inputs

    Combine resolved and required raw files under transform_input/ so transformation receives a stable, explicit input contract.

  6. 06

    Apply mappings and business rules

    Normalize fields, convert types, and apply configured filters and rules to produce target-shaped object CSVs. In this use case, the target schema is Salesforce.

  7. 07

    Load through the target connector

    The current Salesforce connector validates the object plan before creating asynchronous Bulk API 2.0 jobs, then retains job state and row-level results.

  8. 08

    Reconcile for sign-off

    Compare local evidence and optional read-only Salesforce checks. Complete, partial, and inconclusive results remain distinct.

03 / Operational design

Keep controls separate.
Keep evidence with the run.

Configuration

Define behavior outside orchestration

Registries, relationship graphs, source maps, field mappings, loader metadata, and reconciliation rules are configured independently.

Orchestration

Make the stage sequence explicit

PowerShell wrappers coordinate run-root-scoped Python entry points and expose operational parameters.

Processing

Separate each migration concern

Python modules handle profiling, batching, relationship resolution, transformation, loading, results, reporting, and rollback.

Persistence

Retain reviewable run evidence

Canonical folders and JSON/CSV artifacts keep stage outputs, recovery state, result files, metrics, and reports available for analysis.

Run-root contract

One root for inputs, outputs, and proof.

data/runs/<env>/<load-mode>/<migration-run-id>/ 
raw/
working/
transform_input/
transform_output/
bulk2/
reports/
raw/Original source files for a direct run or prepared batch.
working/Resolved data and relationship evidence.
transform_input/Deterministic inputs assembled for mappings.
transform_output/Target-shaped object CSVs; Salesforce for this implementation.
bulk2/Job state, row results, and recovery manifests.
reports/Flow and reconciliation reports for the same run.

04 / Recovery and verification

Failure is a path
the design accounts for.

Recovery, rollback, and reconciliation have different purposes and consequences.

Before a load

Fail early on invalid inputs

Missing or unexpected files, invalid mappings, unknown object keys, bad registry configuration, and missing delta External IDs fail before authentication or job creation.

Interrupted load

Resume from durable evidence

Recovery manifests and chunks support safe resumption while preserving accepted, failed, and unprocessed results for review.

Incorrect target records

Keep rollback explicit

Rollback is a separate Bulk API delete workflow. It is not an automatic side effect of recovery or reconciliation.

Evidence gap

Do not mask inconclusive checks

An inconclusive required reconciliation check is not eligible for sign-off; the report surfaces the evidence gap.

05 / Scope

Clear boundaries. Verifiable outcomes.

File-based source input

The pipeline starts from SAP Commerce CSV exports. It does not extract data from or connect to a live SAP system.

Read-only reconciliation

Local checks and optional Salesforce SOQL queries inspect outcomes; they do not modify Salesforce records.

Credentials stay local

Private keys and local authentication profiles remain outside source control.

Run evidence stays scoped

Prepared roots, loader results, recovery state, and reports remain associated with their migration run.

The result is not just a way to load records. It is a controlled migration that can be explained, resumed, challenged, and signed off using durable evidence.

Featured engineering

Complex migrations.
Clear evidence at every stage.

Discuss a migration challenge