Portworx
PORTWORX.COM

Migration with Stork on Kubernetes

This document will walk you through how to migrate your Portworx volumes between clusters with Stork on Kubernetes.

Prerequisites

Before we begin, please make sure the following prerequisites are met:

  • Version: The source AND destination clusters need Portworx Enterprise v2.0 or later release. As future releases are made, the two clusters can have different Portworx Enterprise versions (e.g. v2.1 and v2.3).
  • Stork v2.0+ is required on the source cluster.
  • Stork helper : storkctl is a command-line tool for interacting with a set of scheduler extensions.

    Perform the following steps to download storkctl from the Stork pod:

    • Linux:

      STORK_POD=$(kubectl get pods -n kube-system -l name=stork -o jsonpath='{.items[0].metadata.name}') &&
kubectl cp -n kube-system $STORK_POD:/storkctl/linux/storkctl ./storkctl
sudo mv storkctl /usr/local/bin &&
sudo chmod +x /usr/local/bin/storkctl

    • OS X:

      STORK_POD=$(kubectl get pods -n kube-system -l name=stork -o jsonpath='{.items[0].metadata.name}') &&
kubectl cp -n kube-system $STORK_POD:/storkctl/darwin/storkctl ./storkctl
sudo mv storkctl /usr/local/bin &&
sudo chmod +x /usr/local/bin/storkctl

    • Windows:

      1. Copy storkctl.exe from the stork pod:

        STORK_POD=$(kubectl get pods -n kube-system -l name=stork -o jsonpath='{.items[0].metadata.name}') &&
kubectl cp -n kube-system $STORK_POD:/storkctl/windows/storkctl.exe ./storkctl.exe

      2. Move storkctl.exe to a directory in your PATH

  • Secret Store : Make sure you have configured a secret store on both clusters. This will be used to store the credentials for the objectstore.
  • Network Connectivity: Ports 9001 and 9010 on the destination cluster should be reachable by the source cluster.

Generate and Apply a ClusterPair Spec

In Kubernetes, you must define a trust object called ClusterPair. Portworx requires this object to communicate with the destination cluster. The ClusterPair object pairs the Portworx storage driver with the Kubernetes scheduler, allowing the volumes and resources to be migrated between clusters.

The ClusterPair is generated and used in the following way:

  • The ClusterPair spec is generated on the destination cluster.
  • The generated spec is then applied on the source cluster

Perform the following steps to create a cluster pair:

NOTE: You must run the pxctl commands in this document either on your Portworx nodes directly, or from inside the Portworx containers on your Kubernetes control plane node.

Create object store credentials for cloud clusters

You must create object store credentials on both the destination and source clusters before you can create a cluster pair. The options you use to create your object store credentials differ based on which object store you use:

Create Amazon s3 credentials

  1. Find the UUID of your destination cluster

  2. Enter the pxctl credentials create command, specifying the following:

    • The --provider flag with the name of the cloud provider (s3).
    • The --s3-access-key flag with your secret access key
    • The --s3-secret-key flag with your access key ID
    • The --s3-region flag with the name of the S3 region (us-east-1)
    • The --s3-endpoint flag with the name of the endpoint (s3.amazonaws.com)
    • The optional --s3-storage-class flag with either the STANDARD or STANDARD-IA value, depending on which storage class you prefer

    • clusterPair_ with the UUID of your destination cluster. Enter the following command into your cluster to find its UUID:

      PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}')
kubectl exec $PX_POD -n kube-system --  /opt/pwx/bin/pxctl status | grep UUID | awk '{print $3}'
      /opt/pwx/bin/pxctl credentials create \
--provider s3 \
--s3-access-key <aws_access_key> \
--s3-secret-key <aws_secret_key> \
--s3-region us-east-1  \
--s3-endpoint s3.amazonaws.com \
--s3-storage-class STANDARD \
clusterPair_<UUID_of_destination_cluster>

Create Microsoft Azure credentials

  1. Find the UUID of your destination cluster

  2. Enter the pxctl credentials create command, specifying the following:

    • --provider as azure
    • --azure-account-name with the name of your Azure account
    • --azure-account-key with your Azure account key

    • clusterPair_ with the UUID of your destination cluster appended. Enter the following command into your cluster to find its UUID:

      PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}')
kubectl exec $PX_POD -n kube-system --  /opt/pwx/bin/pxctl status | grep UUID | awk '{print $3}'
      /opt/pwx/bin/pxctl credentials create \
--provider azure \
--azure-account-name <your_azure_account_name> \
--azure-account-key <your_azure_account_key> \
clusterPair_<UUID_of_destination_cluster>

Create Google Cloud Platform credentials

  1. Find the UUID of your destination cluster

  2. Enter the pxctl credentials create command, specifying the following:

    • --provider as google
    • --google-project-id with the string of your Google project ID
    • --google-json-key-file with the filename of your GCP JSON key file

    • clusterPair_ with the UUID of your destination cluster appended. Enter the following command into your cluster to find its UUID:

      PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}')
kubectl exec $PX_POD -n kube-system --  /opt/pwx/bin/pxctl status | grep UUID | awk '{print $3}'
      /opt/pwx/bin/pxctl credentials create \
--provider google \
--google-project-id <your_google_project_ID> \
--google-json-key-file <your_GCP_JSON_key_file> \
clusterPair_<UUID_of_destination_cluster>

Generate a ClusterPair on the destination cluster

To generate the ClusterPair spec, run the following command on the destination cluster:

storkctl generate clusterpair -n <migrationnamespace> <remotecluster>

Here, remotecluster is the Kubernetes object that will be created on the source cluster representing the pair relationship, and migrationnamespace is the Kubernetes namespace of the source cluster that you want to migrate to the destination cluster.

During the actual migration, you will reference this name to identify the destination of your migration.

apiVersion: stork.libopenstorage.org/v1alpha1
kind: ClusterPair
metadata:
    creationTimestamp: null
    name: remotecluster
    namespace: migrationnamespace
spec:
   config:
      clusters:
         kubernetes:
            LocationOfOrigin: /etc/kubernetes/admin.conf
            certificate-authority-data: <CA_DATA>
            server: https://192.168.56.74:6443
      contexts:
         kubernetes-admin@kubernetes:
            LocationOfOrigin: /etc/kubernetes/admin.conf
            cluster: kubernetes
            user: kubernetes-admin
      current-context: kubernetes-admin@kubernetes
      preferences: {}
      users:
         kubernetes-admin:
            LocationOfOrigin: /etc/kubernetes/admin.conf
            client-certificate-data: <CLIENT_CERT_DATA>
            client-key-data: <CLIENT_KEY_DATA>
    options:
       <insert_storage_options_here>: ""
       mode: DisasterRecovery
status:
  remoteStorageId: ""
  schedulerStatus: ""
  storageStatus: ""

Save the resulting spec to a file named clusterpair.yaml.

Get the destination cluster token

On the destination cluster, run the following command from one of the Portworx nodes to get the cluster token. You’ll need this token in later steps:

pxctl cluster token show

Insert Storage Options

NOTE: When using ClusterPair in the context of Async DR, you must have a DR enabled Portworx license at both the source and destination cluster to enable disaster recovery mode. If you do not have these licenses, enabling disaster recovery mode will fail.

To pair storage specifying the following fields in the options section of your ClusterPair:

  • ip, with the IP address of the remote Portworx node
  • port, with the port of the remote Portworx node
  • token, the token of the destination cluster obtained fom the previous step.
  • mode: by default, every seventh migration is a full migration. If you specify mode: DisasterRecovery, then every migration is incremental. When doing a one time Migration (and not DR) this option can be skipped.
In the updated spec, ensure values for all fields under options are quoted.

Using Rancher Projects with ClusterPair

NOTE: If you are not using Rancher, skip to the next section.

Rancher has a concept of Projects that allow grouping of resources and Kubernetes namespaces. Depending on the resource and how it is created, Rancher adds the following label or annotation:

field.cattle.io/projectID: <project-short-UUID>

The projectID uniquely identifies the project, and the annotation or label on the Kubernetes object provides a way to tie a Kubernetes object back to a Rancher project.

From version 2.11.2 or newer, Stork has the capability to map projects from the source cluster to the destination cluster when it migrates Kubernetes resources. It will ensure that the following are transformed when migrating Kubernetes resources to a destination cluster: * Labels and annotations for projectID field.cattle.io/projectID on any Kubernetes resource on the source cluster are transformed to their respective projectIDs on the destination cluster. * Namespace Selectors on a NetworkPolicy object which refer to the field.cattle.io/projectID label will be transformed to their respective projectIDs on the destination cluster. * Namespace Selectors on a Pod object (Kubernetes version 1.24 or newer) which refer to the field.cattle.io/projectID label will be transformed to their respective projectIDs on the destination cluster.

NOTE:

  • Rancher project mappings are supported only with Stork version 2.11.2 or newer.
  • All the Rancher projects need to be created on both the source and the destination cluster.

While creating the ClusterPair, use the argument --project-mappings to indicate which projectID on the source cluster maps to a projectID on the destination cluster. For example:

storkctl generate clusterpair -n <migrationnamespace> <remotecluster> --project-mappings  <projectID-A1>=<projectID-A2>,<projectID-B1>: <projectID-B2>

The project mappings are provided as a comma-separate key=value pairs. In this example, projectID-A1 on source cluster maps to projectID-A2 on the destination cluster, while projectID-B1 on the source cluster maps to projectID-B2 on the destination cluster.

Apply the ClusterPair spec on the source cluster

A typical ClusterPair spec should like the following once you have followed the previous steps

apiVersion: stork.libopenstorage.org/v1alpha1
kind: ClusterPair
metadata:
  creationTimestamp: null
  name: remotecluster
spec:
  config:
    clusters:
      kubernetes:
        LocationOfOrigin: /etc/kubernetes/admin.conf
        certificate-authority-data: <CA_DATA>
        server: https://192.168.56.74:6443
    contexts:
      kubernetes-admin@kubernetes:
        LocationOfOrigin: /etc/kubernetes/admin.conf
        cluster: kubernetes
        user: kubernetes-admin
    current-context: kubernetes-admin@kubernetes
    preferences: {}
    users:
      kubernetes-admin:
        LocationOfOrigin: /etc/kubernetes/admin.conf
        client-certificate-data: <CLIENT_CERT_DATA>
        client-key-data: <CLIENT_KEY_DATA>
  options:
    ip:     <ip_of_remote_px_node>
    port:   <port_of_remote_px_node_default_9001>
    token:  <token_from_step_3>
    mode: DisasterRecovery
  platformOptions:
    # This section will be set only when using Rancher
    rancher:
      projectMappings:
        <projectID-A1>: <projectID-A2>
        <projectID-B1>: <projectID-B2>

status:
  remoteStorageId: ""
  schedulerStatus: ""
  storageStatus: ""

To create the ClusterPair, apply the ClusterPair YAML spec on the source cluster. Run the following command from a location where you have kubectl access to the source cluster:

kubectl apply -f clusterpair.yaml -n <namespace>

Verify the ClusterPair

To verify that you have generated the ClusterPair and that it is ready, run the following command:

storkctl -n <namespace> get clusterpair
NAME            STORAGE-STATUS   SCHEDULER-STATUS   CREATED
remotecluster   Ready            Ready              07 Mar 22 19:01 PST

On a successful pairing, you should see the STORAGE-STATUS and SCHEDULER-STATUS as Ready.

If you see an error instead, you can get more information by running the following command:

kubectl describe clusterpair remotecluster -n <namespace>
NOTE: You might need to perform additional steps if you are using GKE or EKS.

Migrating Volumes and Resources

Once the pairing is configured, applications can be migrated repeatedly to the destination cluster.

NOTE: If your cluster has a DR license applied to it, you can only perform migrations in DR mode; this includes operations involving the pxctl cluster migrate command.

Migrating Large Volumes

When the clusterpair gets created, Portworx automatically creates a 100G volume named ObjectstoreVolume. If you attempt to migrate a volume significantly larger than 100G, you will find out that the ObjectStore volume doesn’t provide sufficient disk space and the migration will fail.

NOTE: Pure Storage does not recommend using the Portworx internal objectstore for facilitating Async DR or Migrations in production. Instead, use an external objectstore which is setup outside the Kubernetes clusters, such as: * Minio s3 * AWS s3 * GCE Object Storage * Azure Blob Storage

As an example, say you want to migrate a 1 TB volume. If so, you would need to update the size of the ObjectstoreVolume by running:

pxctl volume update --size 1005 ObjectstoreVolume
Portworx uses compression when transferring data. So, if the data is easily compressible, a 100G Objectstore could allow you to transfer more than 100G of data. However, there is no way to tell beforehand how much the data will be compressed. In our example, we migrated a few docker images that were not easily compressible. Thus, reallocating the Objectstore to > 1TB was a safe thing to do.

Here’s how you can check if a migration failed due to insufficient space:

storkctl -n nexus get migration nexusmigration -o yaml
apiVersion: v1
items:
- metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"stork.libopenstorage.org/v1alpha1","kind":"Migration","metadata":{"annotations":{},"name":"nexusmigration","namespace":"nexus"},"spec":{"clusterPair":"remoteclusteroldgreatdanetonewgreatdane","includeResources":true,"namespaces":["nexus"],"startApplications":true}}
    creationTimestamp: "2019-07-02T23:49:13Z"
    generation: 1
    name: nexusmigration
    namespace: nexus
    resourceVersion: "77951964"
    selfLink: /apis/stork.libopenstorage.org/v1alpha1/namespaces/nexus/migrations/nexusmigration
    uid: fec861ca-9d23-11e9-beb8-0cc47ab5f9a2
  spec:
    clusterPair: remoteclusteroldgreatdanetonewgreatdane
    includeResources: true
    namespaces:
    - nexus
    selectors: null
    startApplications: true
  status:
    resources: null
    stage: Final
    status: Failed
    volumes:
    - namespace: nexus
      persistentVolumeClaim: nexus-pvc
      reason: "Migration Backup failed for volume: Backup failed: XMinioStorageFull:
        Storage backend has reached its minimum free disk threshold. Please delete
        a few objects to proceed.\n\tstatus code: 500, request id: 15ADBC3B90C3A97F,
        host id: "
      status: Failed
      volume: pvc-9b776615-3f5e-11e8-83b6-0cc47ab5f9a2
kind: List
metadata: {}

If you see an error similar to the one above, you should increase the size of the ObjectstoreVolume and restart the migration.

Alternatively, you can use your own cloud (S3, Azure, or Google) instead of ObjectStore on the destination cluster. Note that the credentials must be named clusterPair_<clusterUUID_of_destination> and you are required to create them on both the source and the destination cluster.

Start a migration

  1. Create a file called migration.yaml file, specifying the following fields and values:

    • apiVersion: as stork.libopenstorage.org/v1alpha1
    • kind: as Migration
    • metadata.name: with the name of the object that performs the migration
    • metadata.namespace: with the name of the namespace in which you want to create the object
    • spec.clusterPair: with the name of the ClusterPair object created in the Create a ClusterPair section
    • spec.includeResources: with a boolean value specifying if the migration should include PVCs and other applications specs. If you set this field to false, Portworx will only migrate your volumes
    • spec.startApplications: with a boolean value specifying if Portworx should automatically start applications on the destination cluster. If you set this field to false, then the Deployment and StatefulSet objects on the destination cluster will be set to zero. Note that, on the destination cluster, Portworx uses the stork.openstorage.org/migrationReplicas annotation to store the number of replicas from the source cluster
    • spec.namespaces: with the list of namespaces you want to migrate
    • spec.adminClusterPair: with the name of the ClusterPair object created in the admin namespace that Portworx should use to migrate your cluster-scoped resources. Use this if your regular users lack permission to create or delete cluster-scoped resources on the destination cluster. If you don’t specify this field, then STORK will use the object specified in the spec.clusterPair field. This feature was introduced in STORK v2.3.0.

    • spec.purgeDeletedResources: with a boolean value specifying if STORK should automatically purge a resource from the destination cluster when you delete it from the source cluster. The default value is false. This feature was introduced in STORK v2.3.2.

      apiVersion: stork.libopenstorage.org/v1alpha1
kind: Migration
metadata:
name: <YOUR_MIGRATION_OBJECT>
namespace: <YOUR_MIGRATION_NAMESPACE>
spec:
clusterPair: <YOUR_CLUSTER_PAIR>
includeResources: true
startApplications: true
namespaces:
- <NAMESPACE_TO_MIGRATE>
adminClusterPair: <YOUR_ADMIN_CLUSTER_PAIR>
purgeDeletedResources: false

  2. Apply the spec by entering the following command:

    kubectl apply -f migration.yaml

STORK users: You can run the following example command to create a migration:

storkctl create migration <YOUR_MIGRATION_OBJECT> --clusterPair <YOUR_CLUSTER_PAIR> --namespaces <NAMESPACE_TO_MIGRATE> --includeResources --startApplications -n <YOUR_NAMESPACE>

Migration scope

Currently, you can only migrate namespaces in which the object is created. You can also designate one namespace as an admin namespace. This will allow an admin who has access to that namespace to migrate any namespace from the source cluster. Instructions for setting this admin namespace to stork can be found here

Monitoring a migration

Once the migration has been started using the above commands, you can check the status using storkctl:

storkctl get migration -n migrationnamespace

First, you should see something like this:

NAME            CLUSTERPAIR     STAGE     STATUS       VOLUMES   RESOURCES   CREATED
mysqlmigration  remotecluster   Volumes   InProgress   0/1       0/0         26 Oct 18 20:04 UTC

If the migration is successful, the Stage will go from Volumes→ Application→Final.

Here is how the output of a successful migration would look like:

NAME            CLUSTERPAIR     STAGE     STATUS       VOLUMES   RESOURCES   CREATED
mysqlmigration  remotecluster   Final     Successful   1/1       3/3         26 Oct 18 20:04 UTC

Troubleshooting

If there is a failure or you want more information about what resources were migrated you can describe the migration object using kubectl:

kubectl -n <migrationnamespace> describe migration mysqlmigration
Name:         mysqlmigration
Namespace:    migrationnamespace
Labels:       <none>
Annotations:  <none>
API Version:  stork.libopenstorage.org/v1alpha1
Kind:         Migration
Metadata:
  Creation Timestamp:  2018-10-26T20:04:19Z
  Generation:          1
  Resource Version:    2148620
  Self Link:           /apis/stork.libopenstorage.org/v1alpha1/migrations/ctlmigration3
  UID:                 be63bf72-d95a-11e8-ba98-0214683e8447
Spec:
  Cluster Pair:       remotecluster
  Include Resources:  true
  Namespaces:
      migrationnamespace
  Selectors:           <nil>
  Start Applications:  true
Status:
  Resources:
    Group:      core
    Kind:       PersistentVolume
    Name:       pvc-34bacd62-d7ee-11e8-ba98-0214683e8447
    Namespace:
    Reason:     Resource migrated successfully
    Status:     Successful
    Version:    v1
    Group:      core
    Kind:       PersistentVolumeClaim
    Name:       mysql-data
    Namespace:  mysql
    Reason:     Resource migrated successfully
    Status:     Successful
    Version:    v1
    Group:      apps
    Kind:       Deployment
    Name:       mysql
    Namespace:  mysql
    Reason:     Resource migrated successfully
    Status:     Successful
    Version:    v1
  Stage:        Final
  Status:       Successful
  Volumes:
    Namespace:                mysql
    Persistent Volume Claim:  mysql-data
    Reason:                   Migration successful for volume
    Status:                   Successful
    Volume:                   pvc-34bacd62-d7ee-11e8-ba98-0214683e8447
Events:
  Type    Reason      Age    From   Message
  ----    ------      ----   ----   -------
  Normal  Successful  2m42s  stork  Volume pvc-34bacd62-d7ee-11e8-ba98-0214683e8447 migrated successfully
  Normal  Successful  2m39s  stork  /v1, Kind=PersistentVolume /pvc-34bacd62-d7ee-11e8-ba98-0214683e8447: Resource migrated successfully
  Normal  Successful  2m39s  stork  /v1, Kind=PersistentVolumeClaim mysql/mysql-data: Resource migrated successfully
  Normal  Successful  2m39s  stork  apps/v1, Kind=Deployment mysql/mysql: Resource migrated successfully

Pre and Post Exec rules

Similar to snapshots, a PreExec and PostExec rule can be specified when creating a Migration object. This will result in the PreExec rule being run before the migration is triggered and the PostExec rule to be run after the Migration has been triggered. If the rules do not exist, the Migration will log an event and will stop.

If the PreExec rule fails for any reason, it will log an event against the object and retry. The Migration will not be marked as failed.

If the PostExec rule fails for any reason, it will log an event and mark the Migration as failed. It will also try to cancel the migration that was started from the underlying storage driver.

As an example, to add pre and post rules to our migration, we could edit our migration.yaml file like this:

apiVersion: stork.libopenstorage.org/v1alpha1
kind: Migration
metadata:
  name: mysqlmigration
  namespace: mysql
spec:
  clusterPair: remotecluster
  includeResources: true
  startApplications: true
  preExecRule: mysql-pre-rule
  postExecRule: mysql-post-rule
  namespaces:
  - mysql

Advanced Operations

link Set up a Cluster Admin namespace for Migration
link Configure migrations to use service accounts
Last edited: Friday, Jan 13, 2023
Questions? Visit the Portworx forum.