For transfers between two host systems within the NAS enclave—for example, from a PFE to an LFE—the syntax for the shiftc command is nearly the same as that of local transfers. 

Note: If source and destination paths are not specified on the command line, any number of source/destination combinations will be read from stdin (one combination per line).

shiftc [OPTION]... source dest
shiftc [OPTION]... source... directory
shiftc [OPTION]...

For information about Shift options, including those used in the examples in this article, see Shift Command Options.

Transferring Files Within the Enclave

The examples in this section show you how to transfer files between two hosts within the enclave.

Note: The first example includes output; subsequent examples show only the command line.

Perform basic host-to-host transfers (for example, /u/username/file1 on pfe21 to your data/dir2 directory on Lou):

pfe21% shiftc /u/username/file1 lfe:data/dir2
Shift id is 2
Detaching process (use --status option to monitor progress)

• Copy local file1 in the current directory to your Lou home directory while waiting for completion:

  pfe21% shiftc --wait file1 lfe:

• Synchronize the local directory /u/username/dir1 with the Lou directory /u/username/dir2/dir1 while waiting for completion:

  pfe21% shiftc -r --sync --wait /u/username/dir1 lfe:/u/username/dir2

Creating and Extracting Tar Files Within the Enclave

You can use Shift to transfer a directory and write it into a tar file it in one step, resulting in a portable tar file that can be read by either shiftc or tar. In the same step, you can also create a table-of-contents (.toc) file that lists the files contained in the archive along with their sizes and attributes (recommended).

Because of their sequential nature, tar files cannot be efficiently updated in place. As a workaround, incremental tar files can be used, which are separate tar files that consist of files updated after the time the original or subsequent incremental updates were created.

Creating Tar Files

This example creates a tar file (dir1.tar) of the dir1 directory in /nobackup/user1 directly into your Lou home directory, along with a corresponding table of contents (dir1.tar.toc):

pfe21% cd /nobackup/user1
pfe21% shiftc --create-tar --index-tar dir1 lfe:dir1.tar

Note: If the dir1 directory is over 1 TB, it will be split into multiple tar files prefixed with dir1.

Creating Incremental Tar Files

The examples in this section show you how to create incremental tar files from one host to another within the enclave.

• Create an incremental tar file (dir1-2020.tar) of all files modified on or after January 1st, 2020 in the directory /nobackup/username/dir1 directly to your Lou home directory, along with a corresponding table of contents (dir1-2020.tar.toc):

  pfe21% cd /nobackup/username
  pfe21% shiftc --create-tar --index-tar --newer="Jan 1 2020" dir1 lfe:dir1-2020.tar
  

• Create an incremental tar file (dir1-update.tar) of all files modified in the dir1 directory of your nobackup directory after the original tar (dir1.tar) was successfully completed, to the data_dir directory in your Lou home directory:

  pfe21% cd /nobackupp/username/data_dir
  pfe21% shiftc --create-tar --index-tar --newer=`ssh lfe stat -c %Y dir1.tar` dir1 lfe:dir1-update.tar
  

  Note: The version of stat currently deployed on the systems does not allow file creation time to be retrieved, so the above example might miss files that were modified or created while the original tar was being written.

Extracting Tar Files

These examples show you how to extract tar files from one host to another within the enclave.

• Extract the dir1.tar file from your Lou home directory to the current directory:

  pfe21% shiftc --extract-tar lfe:dir1.tar .

• If dir1 was over 1 TB and therefore split into multiple tar files, extract them from your Lou home directory to the current directory as follows:

  pfe21% shiftc --extract-tar lfe:'dir1.*tar' .

  Note: As shown in the example, quotation marks are required when using wildcards (e.g., *) with sources on a remote host.

• Extract the files 1g.20 through 1g.29 from dir1.tar to the current directory:

  pfe21% shiftc --extract-tar --include='1g\.2[0-9]' lfe:dir1.tar .

  Note: As shown in this example, shiftc uses Perl-style regular expressions for some options.