Transfers between a remote system and a host system within the NAS enclave—for example, between your local system and a PFE—must be carried out using the Secure Unattended Proxy (SUP).

To use the SUP for Shift transfers to NAS systems, you must first download the SUP client, authorize one or more NAS hosts for SUP operations, and then authorize one or more NAS directories for writes. A brief summary of these steps is shown below. For a full overview, see Using the Secure Unattended Proxy.

Downloading SUP to Enable Remote Transfers

Complete the following steps to set up your system for remote transfers:

1. Download and install SUP client (one time):

   your_local_system% wget -O sup https://www.nas.nasa.gov/hecc/support/kb/file/9
   your_local_system% chmod 700 sup
   your_local_system% mv sup ~/bin

2. Authorize host for SUP operations (one time per host):

   your_local_system% ssh pfe21
   pfe21% touch ~/.meshrc

3. Authorize directories for writes (one or more times per host):

   your_local_system% ssh pfe21
   pfe21% echo /tmp >>~/.meshrc
   pfe21% echo /nobackup/$USER >>~/.meshrc
   pfe21% echo /u/$USER >>~/.meshrc

Performing Remote Transfers

For remote transfers, the shiftc syntax is similar to local transfers and transfers between two hosts within the enclave, except that you must prepend sup (the SUP client) to each shiftc command. Also, remote Shift transfers must always be initiated from the system that is external to the NAS enclave, but files may be transferred in either direction.

sup shiftc [OPTION]... source dest
sup shiftc [OPTION]... source... directory
sup shiftc [OPTION]...

For example, the following enclave-to-enclave transfer copies file1 into the directory ~/data/run2 on lfe5:

pfe21% shiftc /home/username/file1 lfe5:~/data/run2

If the above example were changed into a remote transfer, it would become:

your_local_system% sup shiftc /home/username/file1 lfe5:~/data/run2

In general, shiftc [args] becomes sup shiftc [args].

Note: For information about Shift options, including those used in the examples in this article, see Shift Command Options.

File Transfer Examples

The examples in this section show you how to transfer files between your local system and a host within the enclave. The first example includes output; subsequent examples show only the command line.

• Perform a remote transfer via the Secure Unattended Proxy (for example, /u/username/file1 on your local system to your home directory on pfe21):

  your_local_system% sup shiftc /u/username/file1 pfe21: 
  Shift id is 3
  Detaching process (use --status option to monitor progress)

• Recursively copy local directory /u/username/dir1 on your local system to directory /u/username/dir2 on lfe5:

  your_local_system% sup shiftc -r /u/username/dir1 lfe5:/u/username/dir2

• Recursively copy remote directory /u/username/dir2 on lfe5 to the current directory on your local system using an encrypted transport:

  your_local_system% sup shiftc -r --secure lfe5:/u/username/dir2 .

Creating and Extracting Tar Files Remotely

You can use Shift to transfer a directory and write it into a tar file it in one step, resulting in a portable tar file that can be read by either shiftc or tar. In the same step, we recommend that you also create a table-of-contents (.toc) file that lists the files contained in the archive along with their sizes and attributes.

Because of their sequential nature, tar files cannot be efficiently updated in place. As a workaround, incremental tar files can be used, which are separate tar files that consist of files updated after the time the original or subsequent incremental updates were created.

Creating Tar Files

This example creates a tar file (dir1.tar) of the directory dir1 on your local system directly into your Lou home directory, along with a corresponding table of contents (dir1.tar.toc):

your_local_system% sup shiftc --create-tar --index-tar dir1 lfe:dir1.tar

Note: If the dir1 directory is over 1 TB, it will be split into multiple tar files prefixed with dir1.

Creating Incremental Tar Files

The example in this section shows you how to create incremental tar files from your local system to your Lou home directory.

Create an incremental tar file (dir1-2020.tar) of all files modified on or after January 1st, 2020 in the directory dir1 on your local system directly to your Lou home directory, along with a corresponding table of contents (dir1-2020.tar.toc):

your_local_system% sup shiftc --create-tar --index-tar --newer="Jan 1 2020" dir1 lfe:dir1-2020.tar

Extracting Tar Files

These examples show you how to extract tar files from your local system to your Lou home directory.

• Extract the dir1.tar file from your local system to your Lou home directory:

  your_local_system% sup shiftc --extract-tar lfe:dir1.tar .

• If dir1 was over 1 TB and therefore split into multiple tar files:

  your_local_system% sup shiftc --extract-tar lfe:'dir1.*tar' .

  Note: As shown in this example, quotation marks are required when using wildcards (e.g., *) with sources on a remote host.

• Extract the files 1g.20 through 1g.29 from dir1.tar from your local system to your Lou home directory:

  your_local_system% sup shiftc --extract-tar --include='1g\.2[0-9]' lfe:dir1.tar .

  Note: As shown in this example, shiftc uses Perl-style regular expressions for some options.