[Quick Tips: Large File Transfer]: How To Quickly Transfer Large Files Over Network In Linux And Unix

How To Quickly Transfer Large Files Over Network In Linux And Unix

Today, I had to reinstall my Ubuntu server where I regularly test different applications and softwares. My Ubuntu server has over 200 GB data, and I don’t want to lose it. I can transfer those data via scp, or setup NFS, FTP to copy files, but I am pretty sure it would take hours to transfer such big files. While searching for an alternative method, I came across the following method. We can quickly transfer large files between between two systems over Network using netcat, tar, and pv commands in Linux and Unix operating systems. Unlike other methods, I find it very fast and quick. For those who don’t know, Netcat is a simple Unix utility which reads and writes data across network connections, using TCP or UDP protocol. tar is a commandline archiving tool, and pv, pipe viewer, is used to monitor the progress of data.

Quickly Transfer Large Files Over Network Between Two Systems In Linux

Make sure you have installed netcat and pv on your systems. If they are not installed already, you can install them as shown below. The tar command is available by default on most Linux systems, so you don’t have to install it separately.

On Arch Linux and its derivatives:

sudo pacman -S netcat pv

On RHEL, CentOS, Fedora:

sudo yum install epel-release

sudo yum install nc pv

Or,

sudo dnf install nc pv

On Debian, Ubuntu, Linux Mint:

sudo apt-get install netcat pv

Now, to copy a large file, we do the following.

On the receiving node (destination system), run the following command as root user:

netcat -l -p 7000 | pv | tar x

On the sending node (source system), run this command as root user:

tar cf - * | pv | netcat 192.168.1.105 7000

Here, 192.168.1.105 is my destination system. tar cf – * will copy everything in the current working directory to the destination system, and the files will be extracted at the other end.

Note: On RHEL, CentOS systems, use “nc” instead of “netcat” as shown below. And, you need to add the port “7000” to the iptables / firewall-cmd on the target system.

After adding the port on target system, you can transfer the larger files as shown below.

On destination system:

nc -l -p 7000 | pv | tar x

On source system:

tar cf - * | pv | nc 192.168.1.105 7000

Also, You can specify a particular file like below.

tar cf - /home/sk/test.file | pv | netcat 192.168.1.105 7000

Please be mindful that both system should have netcat installed. Now, grab a cup of coffee. You’ll see that the files will be copied very quickly than the traditional methods like scp.

Also, you will not see any sign of the file transfer completion on both sides. These commands will keep running until you manually stop them. You need manually check the file sizes on both systems using “du -h <filename>” command. If the file size in destination system is same as in source system, then file transfer process is completed. Afterwords, you can quit the command by pressing CTRL+C.

Quickly Transfer Large Files Between Two Systems In Unix

On Unix operating systems, netcat is called as nc. So, to copy large files between systems over Ethernet, the command would be:

On destination system:

nc -l 7000 | pv | tar -xpf -

On source system:

tar -cf - * | pv | nc 192.168.1.105 7000

Again, these commands should be run as root user. And, both source and destination systems should have netcat and pv installed. Transferring large files over LAN using netcat and tar can indeed save a lot of time.


Disclaimer: There is no security in this method. Because, as you see in the above example, there is no authentication either side. All you need to know the destination system’s IP address. Transferring files using netcat is recommended only in protected networks. If you concern about security, I strongly suggest you to use scp command.

That’s it.