k8s:misc:copy_dirs_using_tar_pipes_and_a_speeed_monitor
**This is an old revision of the document!**
Copy dirs using tar, pipes and a speeed monitor
rsync is good at copying or syncing dirs but shows progress and transfer rate only per file. For longer transfers an averaged transfer rate would be helpful.
tar can provide a pipeable datastream and a utility called Pipe Viewer (pv) can output an averaged transfer speed. Packages for el9 are available on the homepage, other distros include it already (but maybe an older version).
export PATH_TO_SOURCE=/path/to/source export PATH_TO_SOURCE=/path/to/dest rsync -aux --out-format='%n %i' --dry-run $PATH_TO_SOURCE $PATH_TO_DEST | grep '>f+\+' | sed 's/ >f[^\s]*//' > $PATH_TO_DEST/tarlist.txt (cd $PATH_TO_SOURCE; tar -cf - -T $PATH_TO_DEST/tarlist.txt) | pv -agpB 1000000 | (cd $PATH_TO_DEST; tar -xf -)
Note: Older versions of pv do not have the -g = gauge switch. Use pv -aB 1000000 for just the average transfer speed.
Run both commands again later to reduce the tar list and skip files copied already.
k8s/misc/copy_dirs_using_tar_pipes_and_a_speeed_monitor.1734899164.txt.gz · Last modified: by admin