Workload scope
ETL and pipeline migration covers scheduled transformation logic, incremental loads, orchestration dependencies, restart behavior, and late-arriving data handling across source-to-target migrations.
Shared migration method
Model a pipeline as an operational contract, not a sequence of scripts. Inventory schedules, dependencies, ownership, source windows, state stores, retries, and downstream consumers. Then make the target implementation explicit about staging, typing, dedupe rules, apply boundaries, orchestration, alerts, and recovery.
The durable design separates extraction, validation, and apply stages. Each stage should have a bounded input window, observable outputs, and a restart-safe state transition so a partial run cannot silently double-apply or skip data.
Delivery controls
- Define full-refresh, incremental, late-arrival, and backfill policies per dataset.
- Persist watermarks and applied-window markers rather than deriving state from job runtime.
- Make dedupe ordering, tie-breakers, and schema-drift handling executable rules.
- Rehearse retries, partial failures, and dependency delay before cutover.
When to use a pair-specific guide
Use a retained pair-specific page when the source introduces a material operational difference such as Delta schema evolution, Oracle CDC, Hadoop/Oozie dependencies, or target-specific warehouse orchestration. The shared guide owns the repeatable pipeline method; pair pages document the exceptions.
Review points
- Incremental and full-refresh semantics
- Late-arriving data and rerun behavior
- Scheduler, dependency, and ownership changes
- Idempotency, dedupe, and restart controls
Acceptance criteria
A etl / pipeline migration page is complete when the migration team can identify source assets, understand conversion assumptions, review unsupported or ambiguous constructs, run validation checks, and decide whether the workload is ready for cutover.
For each related path, confirm that the child page explains what changes, how conversion works, which patterns are supported, what can drift, and which evidence is required before production traffic moves to the target platform.