Checking File Integrity

Article ID: 243
Last updated: 26 May, 2023

It is a good practice to confirm whether your files are complete and accurate before you transfer the files to or from NAS, and again after the transfer is complete.

The easiest way to verify the integrity of file transfers is to use the NAS-developed Shift tool for the transfer, with the --verify option enabled. As part of the transfer, Shift will automatically checksum the data at both the source and destination to detect corruption. If corruption is detected, partial file transfers/checksums will be performed until the corruption is rectified.

For example:

pfe21% shiftc --verify $HOME/filename /nobackuppX/username
lou% shiftc --verify /nobackuppX/username/filename $HOME
your_localhost% sup shiftc --verify filename pfe:

In addition to Shift, there are several algorithms and programs you can use to compute a checksum. If the results of the pre-transfer checksum match the results obtained after the transfer, you can be reasonably certain that the data in the transferred files is not corrupted. If data is corrupted during a transfer, a good checksum algorithm will yield different results before and after the transfer, with high probability.

The following checksum programs are available on HECC systems:

sum
Computes a checksum using the BSD sum or System V sum algorithm; also counts the number of blocks (1 KB-block or 512 B-block) in a file
cksum
Computes a cyclic redundancy check (CRC) checksum; also counts the number of bytes in a file
md5sum
Computes a 128-bit MD5 checksum, which is represented by a 32-character hexadecimal number
msum
High performance multi-threaded checksum utility, developed at NAS. By default, computes 128-bit MD5 checksums, but you can compute other types using --hash-type. Note that for full compatibility with md5sum output, you must use the --split-size=0 option, which will also decrease performance on large files. For more information, see man msum on any HECC front-end system.

For example:

% ls -l filename
-rw------- 1 username group_id  67358 Nov 15 11:49 filename

% sum filename
50063    66

% cksum filename
269056887 67358 filename

% md5sum filename
cfe0fc62607e9dc6ea0c231982316b75  filename

% msum filename
cfe0fc62607e9dc6ea0c231982316b75  filename

To check the integrity of an existing tar file against a directory:


% tar -tf dir_name.tar | sort > sums.dir_name.tar 
% find dir_name | xargs md5sum | sort>sums.dir_name.dir 
% diff sums.dir_name.tar sums.dir_name.dir

The md5sum utility is more reliable than the sum or cksum commands for detecting accidental file corruption, as the chances of accidentally having two files with identical MD5 checksums are extremely small. The utility is installed by default in most Unix, Linux, and Unix-like operating systems. We recommend that you compute the md5sum of a file before and after the transfer.

The following example shows that the file filename is complete and accurate after the transfer, based on its md5sum.


pfe21% md5sum filename
cfe0fc62607e9dc6ea0c231982316b75  filename

pfe21% scp filename local_username@your_localhost:

your_localhost%md5sum filename
cfe0fc62607e9dc6ea0c231982316b75  filename

See the sum, cksum, md5sum, and msum man pages for more information on these commands.

Contact Us

General User Assistance

Security

  • Report security issues 24x7x365
  • Toll-free: 1-877-NASA-SEC (1-877-627-2732)
  • E-mail: soc@nasa.gov

User Documentation

High-End Computing Capability (HECC) Portfolio Office

NASA High-End Computing Program

Tell Us About It

Our goal is furnish all the information you need to efficiently and effectively use the HECC resources needed for your NASA computational projects.

We welcome your input on features and topics that you would like to see included on this website.

Please send us email with your wish list and other feedback.