Verifying Files Transferred to the Lou Mass Storage System

Article ID: 246
Last updated: 15 Mar, 2019

It is a good practice to confirm whether files are copied correctly to the Lou mass storage system after you transfer them. 

TIP: The easiest way to verify the integrity of your file contents is to use the NAS-developed Shift tool to transfer the files. By default, Shift automatically performs a checksum operation on the data at both the source and the destination, as part of the transfer. If corruption is detected, partial file transfers and checksum operations will be performed until the problem is fixed.

If you transfer files using a command other than Shift, such as scp, the simplest and most lightweight method to verify transferred files is to compare the size (disk usage) and number of files of the original with that of the copy. For example, if you use scp to transfer dir1 from pfe21 to lfe5:

pfe21% du -sk dir1
353760  dir1
pfe21% find dir1 | wc -l
51

pfe21% scp -rp dir1 lfe5:

lfe5% du -sk --apparent-size dir1
353684  dir1
lfe5% find dir1 | wc -l
51

In the example, the directory sizes nearly match (353760; 353684), and the number of files matches exactly (51).

Note: In most cases the sizes will not match exactly. Using the --apparent-size option of the du command is necessary on the Lou systems because the data may reside on either tape or disk. 

Contact Us

General User Assistance

Security

  • Report security issues 24x7x365
  • Toll-free: 1-877-NASA-SEC (1-877-627-2732)
  • E-mail: soc@nasa.gov

User Documentation

High-End Computing Capability (HECC) Portfolio Office

NASA High-End Computing Program

Tell Us About It

Our goal is furnish all the information you need to efficiently and effectively use the HECC resources needed for your NASA computational projects.

We welcome your input on features and topics that you would like to see included on this website.

Please send us email with your wish list and other feedback.