Upgrade to Shift Automated Transfer Components

22 Oct, 2018

During the dedicated time for all HECC systems (scheduled for October 26-29), the Shift transfer tool will be upgraded with several enhancements, which are listed below. The upgraded Shift tool will be available when the systems come back online after the dedicated time.

Please note that any transfer active at the time of the upgrade will be stopped and cannot be restarted. Also, status information for transfers completed before the upgrade will no longer be available.
    
Important: If you use Shift for remote transfers through the Secure Unattended Proxy (SUP), you should answer "y" when prompted to upgrade during key generation. Remote transfers without the upgrade may operate erratically and will not have the enhancements listed below.

The following options will be added to Shift:

--monitor
Shows real-time status of currently running transfers.
--pipeline
Allows parallel processing of verified files to begin while the transfer is still ongoing.
--last-sum
Queries the last computed checksum of any file or tar entry previously transferred with Shift (for transfers after the upgrade).
--force
Allows overwrites of existing read-only files.
--restart=ignore
Allows transfers with unrecoverable errors to run to completion by ignoring all files and/or directories to which the errors pertain.

In addition, Shift will have the following new features:

  • Silent corruption detection of files previously transferred with Shift (for transfers after the upgrade) during transfers thereafter.
  • Advanced --older/--newer syntax to more precisely specify atime/ctime/mtime combinations of interest.
  • Ability to exclude a subset of attributes from {acl,mode,owner,stripe,time,xattr} during --no-preserve.
  • Advanced striping syntax to allow different files in the transfer to be striped dynamically according to different formulas.
  • Optimization of file attribute collection to increase speed of initial source directory traversal.
  • SSL encryption of data streams in fish-tcp transport when --secure specified (requires perl IO::Socket::SSL module installed on the client system).
  • Ability to exclude status emails in a subset of states from {alert,done,error,run,stop,throttle,warn}.

For more information about using Shift, see the following section in the Knowledge Base:  Using Shift for Local and Remote Transfers 

If you have any questions or need assistance, please contact the Control Room at (800) 331-8737, (650) 604-4444, or support@nas.nasa.gov.

Contact Us

General User Assistance

Security

  • Report security issues 24x7x365
  • Toll-free: 1-877-NASA-SEC (1-877-627-2732)
  • E-mail: soc@nasa.gov

User Documentation

High-End Computing Capability (HECC) Portfolio Office

NASA High-End Computing Program

Tell Us About It

Our goal is furnish all the information you need to efficiently and effectively use the HECC resources needed for your NASA computational projects.

We welcome your input on features and topics that you would like to see included on this website.

Please send us email with your wish list and other feedback.