k8s:misc:copy_dirs_using_tar_pipes_and_a_speeed_monitor

**This is an old revision of the document!**

Copy dirs using tar, pipes and a speeed monitor

rsync is good at copying or syncing dirs but shows progress and transfer rate only per file. For longer transfers an averaged transfer rate would be helpful.

export PATH_TO_SOURCE=/path/to/source
export PATH_TO_SOURCE=/path/to/dest
rsync -aux --out-format='%n %i' --dry-run $PATH_TO_SOURCE $PATH_TO_DEST | grep '>f+\+' | sed 's/ >f[^\s]*//' > $PATH_TO_DEST/tarlist.txt
(cd $PATH_TO_SOURCE; tar -cf - -T $PATH_TO_DEST/tarlist.txt) | pv -agp | (cd $PATH_TO_DEST; tar -xf -)

Note: Older versions of pv do not have the -g = gauge switch. Use pv -a for just the average transfer speed.

Run both commands again later to reduce the tar list and skip files copied already.

k8s/misc/copy_dirs_using_tar_pipes_and_a_speeed_monitor.1734898911.txt.gz · Last modified: by admin