Cloud Migration Process: Breaking Down the Cloud Data Migration Lifecycle

If your organization is about to migrate its data to the cloud or even if you are simply considering it as a decision-maker, it is important to understand the different stages of a migration project, as each phase requires different levels of investment in terms of time and money. 

The lifecycle typically consists of three distinct phases: planning, execution, and post-migration. Let’s break it all down in detail.

Cloud Migration Process Stage 1: Planning

First comes the planning stage, which generally involves the essential steps of discovery and planning.

Discovery 

The initiation step in any migration is the discovery phase, where we explore and document the data assets and business requirements and evaluate the associated risks. Data assets encapsulate all datasets, pipelines and downstream applications. There is a need to identify and document these data assets, along with the interdependencies between them. The inventory built in this step serves as a ground truth we can validate our migrated solution against. 

Business requirement discovery is particularly challenging because it requires domain knowledge and a consulting framework to discover the implicit, hidden requirements. These are critical because many migration projects fail due to these unspoken expectations or requirements. Risk comes from different places. Our main risk profile can contain technical risk, business risk, organizational risk, etc. It’s important to be explicit about these and document them in one place for an overview. This will help in later stages to focus on the areas where the project might run out of budget or trust. 

It cannot be stated enough that a thorough, detailed, and well-defined discovery is critical to a successful migration. It’s impossible to migrate a data platform without efficient discovery. 

HAS YOUR COMPANY OUTGROWN ITS INFRASTRUCTURE?

TALK TO DATAPAO TODAY ABOUT MIGRATING TO THE CLOUD TO STREAMLINE YOUR OPERATIONS AND REACH NEW HEIGHTS.

Planning

During planning, it’s time to define the migration strategy, the timeline and the staffing. A migration framework that helps us structure our thoughts around the migration is instrumental. The more established and tested the framework to be used, the lower the probability of unknown risks arising during execution. 

At the end of the planning stage, we can deliver a preliminary migration plan which is not verified. The preliminary plan contains the proposed architecture and technology stack, migration strategy, migration inventory and associated risk profiles. When the migration plan is ready, a pilot run will help verify it.

Cloud Migration Process Stage 2: Execution

Execution of the cloud migration project can be considered as consisting of the piloting step, the execution step, and the validation step – with execution being the longest and costliest stage of cloud migration.

MVP/Piloting

In software development, there are two types of unknowns. There are things we know we don’t know. These are predictable issues that cannot be accurately estimated. For instance, one might not know how they will accommodate a feature disparity between Oozie and Databricks workflows, but do know that time needs to be allocated to solving this problem.

A much bigger problem is unknown unknowns – things we don’t even know we don’t know about. These are the issues that can arise that one cannot anticipate. We don’t even have a way to account for them within the plan. Therefore, every plan is inaccurate before execution. 

Conducting tests of the plans will allow for the discovery of these unknown unknowns. Piloting or MVP (minimum viable product) migration aims to select a few datasets, pipelines, and applications, migrate them end-to-end, and document every unanticipated issue. In essence, this is the rehearsal of the migration plan, which is being field-tested in order for us to understand what we need to adjust. Therefore, it’s important to select MVP migration targets that cover a wider range of sources/pipelines/applications where issues are suspected to arise. 

Execution

The execution stage is the biggest part of a migration in terms of both time and cost. During the execution, it’s time to migrate the data platform, with every aspect of the software development covered: QA, project management, DevOps, etc.

Validation

A key difference between migration and greenfield development comes from the obvious fact that there is a source system to replace. Therefore, validation, which answers the question, “Is our migration complete?” is vital. There is a need to make sure that we cover every element and every aspect of the migration where divergence could arise.

We must incorporate deliberate changes without missing any differences we didn’t anticipate. Do note that no migration can replicate the source system 100%, but a well-planned and executed migration will ensure that all the functionality is preserved or even enhanced.

Cloud Migration Process Stage 3: Post-Migration

It is important to consider the support of the new infrastructure post-migration. Let’s look more closely at this and the go-live stage.

Go-Live

Go-Live (GL) is the most exciting and most stressful part of the migration. This is where we go live with the new system and decommission the old system. This is usually not a single go-live but an iterative process with multiple go-lives, some of which have been selected as milestones. 

GL has its own process and framework, which is out of the scope of this paper, but note that it’s important to have a GL plan, mitigation plan, and validation criteria in place. Dry runs are also valuable tools to test the team’s readiness for the GL event.

Support

After going live, it is time to switch to support mode, safeguard the SLA, and accommodate new business requests. It’s important to distinguish between the support mode (or team) and the migration (process or team). It is not a good idea to constantly incorporate new requests into the migration, as that would be shooting at a moving target. 

When a project transitions to the support phase, it usually changes from a deadline-based and scoped project into an agile approach with sprint and epic-based planning. In many cases, this move supports business continuity best. However, it’s critical to articulate this to business stakeholders and agree with every party whether the approach is agile or should follow a plan with minor changes. 

Sometimes, finding the right balance between being agile and not losing predictability and deadlines is difficult. The problem lies not in where the approach falls on the agile-fixed scope spectrum but in miscommunication and misalignment between stakeholders and delivery teams.

WANT TO LEARN MORE ABOUT CLOUD MIGRATION?

READ DATAPAO’S IN-DEPTH GUIDE HERE.

Are You Ready for Your Data Migration Project?

The right level of engagement on the part of stakeholders, meticulous planning and careful execution are all key to a successful cloud data migration.

We hope the above has been useful in breaking down the specific steps to be followed. If you need a trusted partner who will ensure you migrate not just efficiently but smartly, book a free 30-minute consultation with DATAPAO today.

We’ll go through your circumstances, needs, and goals and outline the approach that can work the best for you, including solid steps to achieving it. And we’ll work with you every step of the way to make your cloud data migration a real boost to your operations.