For large file transfers within the NAS enclave, use the commands shiftc, mcp, or cxfscp. The slower cp command also can be used.

The following file transfer commands can be used when both the source and destination locations are accessible on the same host where the command is issued.

Shift Command

Shift (shiftc) is a NAS-developed tool for performing automated local and remote file transfers. Shift utilizes a variety of underlying file transports to achieve maximum performance for files of any size on any file system.

Where is it installed at NAS?

shiftc is installed on all the front ends of the NAS compute and storage systems (PFEs, AFEs, CFEs, CGHFEs, and LFEs).

When to use it?

The command shiftc can be used as a drop-in replacement for cp at any time on any system on which it is installed.

Examples

lfe5% shiftc /nobackup/username/filename $HOME
pfe25% shiftc $HOME/filename /nobackup/username

Performance

shiftc utilizes mcp, when available, so it can be up to 10 times faster than cp for large files (2+ GB) and can achieve up to 1.8 GB/sec on a single host. Using the --hosts option, it has been measured to achieve up to 5.0 GB/sec on 8 hosts.

For more information, see Shift File Transfer Overview.

cxfscp

cxfscp is a program for quickly copying large files to and from a CXFS filesystem, such as the Lou front-end (LFE) home file system. It can be significantly faster than cp on CXFS filesystems since it uses multiple threads and large direct I/Os to fully utilize the bandwidth to the storage hardware.

For files less than 64 kilobytes in size, which will not benefit from large direct I/Os, cxfscp will use a separate thread for copying these files using buffered I/O similar to cp.

Where is it installed at NAS?

cxfscp is installed on the LFEs.

When to use it?

The NAS Lustre filesystems (/nobackuppX) are mounted on lfe[5-8]. The command cxfscp can be issued on either of these hosts to copy large files between the LFE's CXFS home file system and NAS Lustre /nobackup. This is an easy way to transfer files between Lou and NAS compute resources without the need for scp or scp.

Examples

lfe5% cxfscp -rp --bi /nobackupp17/username/testdir_a /u/username/tests
lfe5% cxfscp -p --bo /u/username/data* /nobackupp17/username/data_dir

Performance

Some benchmarks done by NAS staff show that cxfscp is typically 4-7 times faster than cp for large files (2+ GB) and can achieve up to 650 MB/sec.

For more information, read man cxfscp.

cp

cp is a Unix command for copying files between two locations (for example, two different directories of the same filesystem or two different filesystems such as NFS, CXFS or Lustre).

Where is it installed at NAS?

cp is available on all NAS systems except the secure front ends (SFEs).

Use the -p option to preserve the timestamp on the file.

Examples

pfe25% cp -p $HOME/filename $HOME/newdir/filename2
pfe25% cp -p $HOME/filename /nobackup/username