Skip to content

Restoring from large replicated volumes

Note

This guide is meant as a helpful example of how to perform these actions; however, it is not a supported action.

Prerequisites

For an overview and detailed explanation of the preliminary steps refer to the snapshot tutorial.

  1. Primary HyperCloud site dashboard is able to connect to the remote instance using SSH keys.
  2. Create and configure the /var/run/cluster-control/facts/snapshot/remote.json file
  3. Add a SNAPSHOT_SCHEDULE to the Attributes header of the target VM (e.g. hourly:local=3,remote=6)

Please see Restoring replicated VMs for the basic steps.

Warning

These instructions are for use where large, multi-terabyte images have been replicated. Please read these instructions carefully - if an error is made, it may be necessary to manually clean up any errors.

Procedure

Restoring from snapshot(s)

Download the script from GitHub and copy the file to the HyperCloud dashboard under /var/cores/restore.sh, making sure that the script is executable.

To retrieve a basic list of available snapshots on the master cluster (i.e. the source cluster), run the following command:

snapctl list snapshots --remote

The resultant output will appear similar to the content below:

root@hypercloud-dashboard:~# snapctl list snapshots --remote
TYPE  VM PERIOD WHEN TAG                           TIME                          

[local snaps omitted]

TYPE   VM                   PERIOD WHEN TAG                                                TIME                          

[hypercloud-dashboard snaps omitted]

TYPE   VM PERIOD WHEN TAG                           TIME                          IMAGE             SIZE        
remote 1  hourly 0    auto-v2-hourly-202310050800-1 2023-10-05 08:00:00 +0000 UTC Master_Site-1-1-0 4294967296  
remote 1  hourly 0    auto-v2-hourly-202310050900-1 2023-10-05 09:00:00 +0000 UTC Master_Site-1-1-0 4294967296  
remote 1  hourly 0    auto-v2-hourly-202310050900-1 2023-10-05 09:00:00 +0000 UTC Master_Site-2-1-2 10737418240 
remote 1  hourly 0    auto-v2-hourly-202310051000-1 2023-10-05 10:00:00 +0000 UTC Master_Site-1-1-0 4294967296  
remote 1  hourly 0    auto-v2-hourly-202310051000-1 2023-10-05 10:00:00 +0000 UTC Master_Site-2-1-2 10737418240 
remote 1  hourly 0    auto-v2-hourly-202310051000-1 2023-10-05 10:00:00 +0000 UTC Master_Site-2-1-3 5368709120  
remote 1  hourly 0    auto-v2-hourly-202310051100-1 2023-10-05 11:00:00 +0000 UTC Master_Site-1-1-0 4294967296  
remote 1  hourly 0    auto-v2-hourly-202310051100-1 2023-10-05 11:00:00 +0000 UTC Master_Site-2-1-2 10737418240 
remote 1  hourly 0    auto-v2-hourly-202310051100-1 2023-10-05 11:00:00 +0000 UTC Master_Site-2-1-3 5368709120  
remote 1  hourly 0    auto-v2-hourly-202310051200-1 2023-10-05 12:00:00 +0000 UTC Master_Site-1-1-0 4294967296  
remote 1  hourly 0    auto-v2-hourly-202310051200-1 2023-10-05 12:00:00 +0000 UTC Master_Site-2-1-2 10737418240 
remote 1  hourly 0    auto-v2-hourly-202310051200-1 2023-10-05 12:00:00 +0000 UTC Master_Site-2-1-3 5368709120  
remote 1  hourly 0    auto-v2-hourly-202310051300-1 2023-10-05 13:00:00 +0000 UTC Master_Site-1-1-0 4294967296  
remote 1  hourly 0    auto-v2-hourly-202310051300-1 2023-10-05 13:00:00 +0000 UTC Master_Site-2-1-2 10737418240 
remote 1  hourly 0    auto-v2-hourly-202310051300-1 2023-10-05 13:00:00 +0000 UTC Master_Site-2-1-3 5368709120

All snaps present on the destination cluster will be visible, listed by VM ID. Additional information about these snapshot images can be gleaned by running:

rbd -p [pool_name] ls -l | grep [image_prefix]

root@hypercloud-dashboard:~# rbd -p rbd ls -l | grep Master
Master_Site-1-1-0                                                          4 GiB                     2            
Master_Site-1-1-0@auto-v2-hourly-202310050800-1                            4 GiB                     2            
Master_Site-1-1-0@auto-v2-hourly-202310050900-1                            4 GiB                     2            
Master_Site-1-1-0@auto-v2-hourly-202310051000-1                            4 GiB                     2            
Master_Site-1-1-0@auto-v2-hourly-202310051100-1                            4 GiB                     2            
Master_Site-1-1-0@auto-v2-hourly-202310051200-1                            4 GiB                     2            
Master_Site-1-1-0@auto-v2-hourly-202310051300-1                            4 GiB                     2            
Master_Site-2-1-2                                                         10 GiB                     2            
Master_Site-2-1-2@auto-v2-hourly-202310050900-1                           10 GiB                     2            
Master_Site-2-1-2@auto-v2-hourly-202310051000-1                           10 GiB                     2            
Master_Site-2-1-2@auto-v2-hourly-202310051100-1                           10 GiB                     2            
Master_Site-2-1-2@auto-v2-hourly-202310051200-1                           10 GiB                     2            
Master_Site-2-1-2@auto-v2-hourly-202310051300-1                           10 GiB                     2            
Master_Site-2-1-3                                                          5 GiB                     2            
Master_Site-2-1-3@auto-v2-hourly-202310051000-1                            5 GiB                     2            
Master_Site-2-1-3@auto-v2-hourly-202310051100-1                            5 GiB                     2            
Master_Site-2-1-3@auto-v2-hourly-202310051200-1                            5 GiB                     2            
Master_Site-2-1-3@auto-v2-hourly-202310051300-1                            5 GiB                     2

For an explanation of the naming convention, refer to the Restoring replicated VMs page.

In order to restore the VM, create a command line based on the image name from the list above.

The format will resemble:

restore.sh [pool_name] [datastore_id] [snap_name(#without the @ symbol!)] [rbd_name]

Any number of RBD images can be added, as needed, with the minimum requirement of one.

List pools with:

ceph osd pools ls

List datastores with:

hypercloud datastore list

For multiple volumes the [snap_name] must exist for all images.

For example, to restore the VM above:

/var/cores/restore.sh rbd 100 auto-v2-hourly-202310051100-1 Master_Site-1-1-0 Master_Site-2-1-2 Master_Site-2-1-3

Ensure that the disks are listed in the correct order; most importantly, the root disk is entered first. The script will generate the necessary images and create a template in HyperCloud. Lastly, it will provide the command that can be used to unprotect the snapshot(s) once the restore process is completed.

Instantiating the VM

Within the HyperCloud GUI, navigate to Templates → VMs and click on the template created by the script, for instance:

restore_template-auto-v2-hourly-202310051000-1

The VM can be instantiated as any other template, ensuring to give it a name and verifying the settings are correct, especially, the disk(s) are connected, showing a green box for available to use, and in the correct order.

When multiple disks are attached to the template, it would appear as below:

Instantiate Restored VM

Take notice that the disks are listed as having only 1 gigabyte in size; this is intentional and does not affect the actual disk size(s) once they've been attached to the VM instance.

When the VM has finished booting up, disks can be attached to it to recover files, etc.

Clean up

After the completion of the recovery process, it is recommended that the instance be terminated, and all artifacts deleted. This does not affect the replicated snapshots; however, it will recover storage space.

The steps are:

  • Terminate the VM.
  • Remove the template, selecting the Delete all images button.
  • In the HyperCloud dashboard CLI, unprotect the snapshot(s) with the command(s) provided from the script output.
    • If this step is not carried out, the snapshot(s) will not be removed by the automated replication process and will continue to consume space within the cluster.