Data corruption

Computer, transmission, and storage systems use a number of measures to provide end-to-end data integrity, or lack of errors.

Background radiation, head crashes, and aging or wear of the storage device fall into the former category, while software failure typically occurs due to bugs in the code.

For instance, cables might be slightly loose, the power supply might be unreliable,[3] external vibrations such as a loud sound,[4] the network might introduce undetected corruption,[5] cosmic radiation and many other causes of soft memory errors, etc.

[6] All in all, the error rates as observed by a CERN study on silent corruption are far higher than one in every 1016 bits.

[9][10] One problem is that hard disk drive capacities have increased substantially, but their error rates remain unchanged.

In modern disks the probability is much larger because they store much more data, whilst not being safer.

That way, silent data corruption has not been a serious concern while storage devices remained relatively small and slow.

[11] As an example, ZFS creator Jeff Bonwick stated that the fast database at Greenplum, which is a database software company specializing in large-scale data warehousing and analytics, faces silent corruption every 15 minutes.

[14] Silent data corruption may result in cascading failures, in which the system may run for a period of time with undetected initial error causing increasingly more problems until it is ultimately detected.

If an uncorrectable data corruption is detected, procedures such as automatic retransmission or restoration from backups can be applied.

and tools available for most operating systems to automatically check the disk drive for impending failures by watching for deteriorating SMART parameters.

This is particularly important in commercial applications (e.g. banking), where an undetected error could either corrupt a database index or change data to drastically affect an account balance, and in the use of encrypted or compressed data, where a small error can make an extensive dataset unusable.