Migrating a Kubernetes PVC from One Cluster to Another

Overview

This post will go through the steps required to migrate a stateful app with a PersistentVolumeClaim (PVC) backed by an EBS Volume in AWS from one Kubernetes cluster to another. These clusters can even be in different VPCs and this strategy will still work.

This will incur downtime. Please plan for this accordingly.

Steps

1. Scale down the application that is attached to the PVC. This will detach the application from the PVC and leave the data in a more consistent state when we take a snapshot for DR purposes.

    kubectl scale deployment <name> -n <namespace> --replicas=0
   

2. Find the PersistentVolume (PV) you are planning to migrate

    kubectl get pv
   

3. Find the EBS Volume associated with it in AWS

    kubectl get pv <name> -o jsonpath='{.spec.awsElasticBlockStore.volumeID}'
   

4. Take a snapshot of the EBS Volume. This is just in case something goes wrong. (Optional, but recommended)

   In the EC2 Console in AWS, go to Volumes. Find your volume in the table and select it. Click Actions → Create Snapshot.

5. Mark the PersistentVolume’s reclaim policy as Retain. This will keep the EBS volume in AWS even if the PV or PVC is accidentally deleted from the cluster.

    kubectl patch pv <name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
   

6. Save the PV and PVC yaml definitions to files on your local machine.

    kubectl get pv <name> -o yaml > pv.yaml
    kubectl get pvc <name> -n <namespace> -o yaml > pvc.yaml
   

7. You can now deploy out the application to the new cluster (using helm, kubectl, etc.). It is okay for it to spin up a new PVC/PV for the application. We are going to be removing them later on.

   Once the application is installed, I recommend scaling down the new application to 0 replicas so nothing is attached to any PVC in the new environment

    kubectl scale deployment <name> -n <namespace> --replicas=0
   

8. Delete the new PVC that was created. This should in turn delete the PV and EBS Volume in AWS.

    kubectl delete pvc <name> -n <namespace>
   

9. Apply the saved PVC to the new cluster.

    kubectl apply -f pvc.yaml
   

10. Grab the uid of the PVC. This is needed when applying the PV to the cluster.

    kubectl get pvc <name> -n <namespace> -o jsonpath='{.metadata.uid}'
    

11. Now we can apply the PV. We will additionally set the claimRef to the uid found in the previous step.

    kubectl apply -f pv.yaml
    kubectl patch pv <name> -p '{"spec":{"claimRef":{"uid":"<uid>"}}}"
    

12. At this point it is safe to delete the PVC and PV from the old cluster. Because the reclaim policy was previously set to Retain, the EBS Volume will stay active in AWS.

13. Scale the application on the new cluster to its original replica count. It may take a few minutes for the volume to attach to the new node.

    kubectl scale deployment <name> -n <namespace> --replicas=<original count>