Replace Storage Drives with Rsync in Arch Linux

Upgrade Storage with Rsync in Arch Linux

Rsync is a widely used file level synchronisation utility. When upgrading or replacing a failing disk drive it is important to make sure no data or even metadata loss occurs. Rsync can provide near mirror copies of files and directories. Only block-level copying can create a perfect mirror but it is not needed or wanted in all but some cases such as when a hardware defect is present in which case block level duplication can reduce data-loss.

 

After a disk drive has been physically connected it’s time to partition & format the disk drive then mount the partition on a temporary mount point. The files can now be transferred from the old or failing storage device to the temporarily mounted partition on the new replacement storage device. If a disk is being upgraded rsync is the best option as it can preserve all data and nearly all metadata. In case of a damaged storage device it may be advisable to backup at block-level with DD.

 

Synchronise data with Rsync

Run a test synchronisation.

sudo rsync -avihXP --info=progress2 --stats --dry-run /mnt/disk?/ /mnt/temp/

A test command is enabled with -n or --dry-run which will execute the command but will not perform any writes to the storage device.

 

Perform the initial synchronisation of files between the 2 mount points.

sudo rsync -avihXP --info=progress2 --stats --timeout=5 /mnt/disk?/ /mnt/temp/

Archive mode is enabled with -a or --archive options which is equivalent to -rlptgoD options combined.

Recurse into directories with -r option. Copy symbolic links with -l or --links option. Permissions can be preserved with -p or --perms option, modification times with -t or --times option and owner with -o or --owner. The -D option is a combination of --devices and --specials options. Device and special files are preserved with --devices and --specials options respectively.

Verbosity can be increased with -v or --verbose option. List a summary of all changes with -i or --itemize-changes option. The storage units can be displayed in human readable format with -h or --human-readable option. Extended attributes are preserved with -X or --xattrs options. The -P option is the same as --partial and --progress options. Files that have not finished transferring can be kept with --partial option otherwise these files are deleted. The progress of the current file being transferred can be shown with --progress option while total progress can be show with --info=progress2 option. Additional file transfer statistics can be displayed with --stats option. By default Rsync has no timeout meaning it will never exit and partial files will never get renamed which means they cannot be resumed. Setting a timeout value will make Rsync exit when no data is received after a specified number of seconds. It should prevent temporary files from being saved and never removed. This The last 2 options are the source and destination paths respectively. The trailing slashes of the source and destination paths will sync contents of the directory but not directory itself.

 

Perform an additional synchronisation and remove synced files from the source path.

sudo rsync -avihXP --info=progress2 --stats --timeout=5 --remove-source-files /mnt/disk?/ /mnt/temp/

To remove files from the source path that have been synced to the destination path with --remove-source-files option.

 

Verify Transferred Data

Once the synchronisation has completed its worth manually inspecting the files on the new storage device to verify the general directory and file structure. A quick way to verify that all files have transferred is to compare the total space used on the old and the new partitions. If the size matches it’s reasonable to assume all files have been transferred though it’s still possible that files could be corrupt.

 

Show total directory sizes of all mount points.

du -sh /mnt/* | sort -h

 

Once the files have been transferred and verified the old storage device can be unmounted and disconnected. If it is to be sold or disposed it should be securely wiped first to prevent data theft.

dom