For transfers on the same host within the NAS enclave—for example, between two directories on pfe21—the syntax for shiftc is similar to the cp command.

Note: If source and destination paths are not specified on the command line, any number of source/destination combinations will be read from the standard input, stdin (one combination per line).

The syntax for local Shift transfers is as follows:

shiftc [option]... source dest
shiftc [option]... source... directory
shiftc [option]...

For information about Shift options, including those used in the examples in this article, see Shift Command Options.

Transferring Files Locally

The examples in this section show you how use the shiftc command to transfer files on the same NAS host.

Note: The first example includes output; subsequent examples show only the command line.

• Copy local file1 into existing directory /u/username:

  pfe21% shiftc file1 /u/username 
  Shift id is 1
  Detaching process (use --status option to monitor progress)

• Copy file1 from /u/username  to /nobackup/username/dir1 on pfe21:

  pfe21% shiftc /u/username/file1 /nobackup/username/dir1

• Recursively copy the directory inputs inside another directory, /nobackup/username/dir2:

  % shiftc -r inputs /nobackup/username/dir2

  Complete the same operation with data verification turned off:

  % shiftc -r --no-verify inputs /nobackup/username/dir2

  TIP: Although using the --no-verify option can improve the speed of your transfer, it is not recommended because without data verification, the integrity of your data cannot be ensured.
• Copy local file1 in the current directory to existing local directory /u/username/dir1:

  pfe21% shiftc file1 /u/username/dir1

• Recursively copy local directory /nobackup/username/dir1 to local directory /nobackup/username/dir2 using 2 client hosts to perform the transfer:

  pfe21% shiftc -r --hosts=2 /nobackup/username/dir1 /nobackup/username/dir2

• Recursively copy local directory nobackup/username/dir1 to local directory dir2, but exclude files ending in .log:

  pfe21% shiftc -r --exclude='\.log$' nobackup/username/dir1 /dir2

Creating and Extracting Tar Files Locally

You can use Shift to transfer a directory and write it into a tar file it in one step, resulting in a portable tar file that can be read by either shiftc or tar. In the same step, you can also create a table-of-contents (.toc) file that lists the files contained in the archive along with their sizes and attributes (recommended).

Because of their sequential nature, tar files cannot be efficiently updated in place. As a workaround, incremental tar files can be used, which are separate tar files that consist of files updated after the time the original or subsequent incremental updates were created.

Creating Tar Files

The examples in this section show you how to create tar files on the same NAS host. 

• Create a tar file (dir1.tar) of the directory /nobackup/username/dir1 and put it in the current directory, along with a corresponding table of contents (dir1.tar.toc):

  pfe21% cd /nobackup/username
  pfe21% shiftc --create-tar --index-tar dir1 dir1.tar

  Note:? If the dir1 directory is over 1 TB, it will be split into multiple tar files prefixed with dir1.

• The NAS nobackup filesystems are mounted on Lou. This makes it possible to create local tar files from your nobackup filesystem directly to your Lou home directory when you are logged into Lou (lfe5-8).

  For example, to transfer the dir1 directory as a tar file from your nobackup filesystem to the data_dir directory in your Lou home directory:

  lfe5% cd /nobackupp17/username/data_dir
  lfe5% shiftc --create-tar --index-tar dir1 /u/username/data_dir/dir1.tar

  Note: In the above example, the first step is to change into the directory /nobackupp17/username/data_dir, which is the parent directory of dir1 (the directory you are transferring). This prevents the tar operation from creating extraneous prepended directories when the tar file is extracted.

Creating Incremental Tar Files

The examples in this section show you how to create incremental tar files on the same NAS host.

• Create an incremental tar file (dir1-2020.tar) of all files modified on or after January 1st, 2020 in the /nobackup/username/dir1 directory, and put it in the current directory, along with a corresponding table of contents (dir1-2020.tar.toc):

  pfe21% cd /nobackup/username
  pfe21% shiftc --create-tar --index-tar --newer="Jan 1 2020" dir1 dir1-2020.tar
  

• Create an incremental tar file (dir1-update.tar) of all files modified in the dir1 directory of your nobackup directory after the original tar (dir1.tar) was successfully completed, to the data_dir directory in your Lou home directory:

  lfe5% cd /nobackupp/username/data_dir
  lfe5% shiftc --create-tar --index-tar --newer=`stat -c %Y /u/username/data_dir/dir1.tar` \ 
  dir1 /u/username/data_dir/dir1-update.tar
  

  Notes:

  • The shiftc command line shown above is too long to be formatted as one line, so it is broken with a backslash (\).
  • The version of stat currently deployed on the systems does not allow file creation time to be retrieved, so the above example might miss files that were modified or created while the original tar was being written.

Extracting Tar Files

The examples in this section show you how to extract files from a tar file on the same host.

• Extract the tar file dir1.tar on Lou directly into your /nobackup directory:

  lfe5% cd /nobackup/username/data_dir
  lfe5% shiftc --extract-tar /u/username/data_dir/dir1.tar .

• If dir1 was over 1 TB and therefore split into multiple tar files:

  pfe21% shiftc --extract-tar /u/username/data_dir/dir1.*tar .
  
  

• Extract the files 1g.20 through 1g.29 from dir1.tar to the current directory:

  pfe21% shiftc --extract-tar --include='1g\.2[0-9]' dir1.tar .

  Note: As shown in this example, shiftc uses Perl-style regular expressions for some options.