File Hashes Explained: MD5, SHA-1, SHA-256 and How They Identify Malware

A file hash is a fixed-length fingerprint computed from every byte of a file, so two files with the same hash are, in practice, the same file. Scanners use hashes as the fastest kind of malware signature and as the key for reputation lookups, while you can use SHA-256 to confirm a download is exactly what the publisher released. MD5 and SHA-1 should no longer be trusted for that job.

What a hash actually is

A cryptographic hash function takes input of any length and produces output of a fixed length: 128 bits for MD5, 160 bits for SHA-1, 256 bits for SHA-256. The output is usually written as hexadecimal, so a SHA-256 hash is 64 hex characters. Three properties make hashes useful for security:

• Determinism. The same bytes always produce the same hash, on any machine, with any implementation.
• Avalanche. Changing a single bit of the input changes roughly half the bits of the output, so similar files have unrelated hashes.
• One-wayness and collision resistance. You cannot work backwards from a hash to the file, and you should not be able to find two different files with the same hash.

The first two properties hold for MD5 and SHA-1 as well as SHA-256. It is the third, collision resistance, that MD5 and SHA-1 have lost, which is the whole reason the industry moved on. The avalanche effect is easy to see. The SHA-256 of an empty file is a well-known constant; add one byte and nothing survives:

$ printf '' | shasum -a 256
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  -
$ printf 'a' | shasum -a 256
ca978112ca1bbdcafac231b39a23dc4da786eff8147c4e72b9807785afee48bb  -

MD5, SHA-1 and SHA-256 compared

A collision is two different inputs with the same hash. For MD5, researchers can produce colliding files at will, and since the chosen-prefix work used by the Flame malware to forge a Microsoft certificate in 2012, an attacker can make two files that each start with content of their choosing and still share a hash. For SHA-1, the 2017 SHAttered project produced two different PDFs with the same hash, and later work reduced chosen-prefix collisions to a cost within reach of a well-funded group. NIST's policy is that SHA-1 should not be used for digital signatures and is being retired entirely from federal use. The practical consequence is simple: a publisher who gives you an MD5 or SHA-1 for a download is asking you to check against a fingerprint that a capable attacker could reproduce on a different file. Prefer SHA-256, and treat an MD5-only checksum as protection against a corrupted download rather than a malicious one.

How scanners use hashes

As the fastest signature

The cheapest possible malware signature is "this exact file": hash it, look the hash up. ClamAV's .hdb and .hsb databases are lists of MD5 and SHA-256 hashes with the file size and a name, and every commercial engine has the equivalent. A hash lookup is one comparison against a sorted index, so millions of signatures cost almost nothing. The weakness is exactly the avalanche property that makes hashes useful: change any byte and the hash no longer matches. Malware builders exploit this by appending random data or re-encrypting the payload for every campaign, which is why hash signatures are only the first layer of a scanner's pipeline and why pattern signatures and heuristics exist.

As a reputation key

The more powerful use of a hash is as a question to the rest of the world. Threat-intelligence services index every sample they have seen by SHA-256, so the hash of your file can return how many machines have it, when it was first seen, which engines flag it and whether it is signed. The same idea runs in reverse with known-good sets such as NIST's National Software Reference Library, which lets an analyst discard thousands of operating-system files by hash before looking at anything. Scan.now reports this as the file hash reputation check.

Similarity hashes

Because a cryptographic hash changes completely on any edit, analysts also use hashes designed to change gradually. ssdeep and TLSH produce values that can be compared for similarity, so two builds of the same family score as near matches. For Windows executables, the import hash (imphash) fingerprints the list of imported functions in order, which survives most repacking of the surrounding file. These are not for verifying downloads; they are for clustering samples.

How to hash a file yourself

Every operating system ships a tool. Use SHA-256 unless the publisher only offers something else:

# macOS and most Linux
shasum -a 256 installer.dmg
# Linux (coreutils)
sha256sum installer.tar.gz
# Windows, Command Prompt
certutil -hashfile installer.msi SHA256
# Windows, PowerShell
Get-FileHash .\installer.msi -Algorithm SHA256

The output is the hash followed by the filename (or, for certutil, the hash on its own line with no spaces). Compare it against the value the publisher lists. A single differing character means the file is not the one they published, whether through corruption, a mirror serving a stale build or tampering. Do not trust your eyes for 64 characters; paste both into a text editor and use find, or let the shell do it:

echo "expectedhashhere  installer.dmg" | shasum -a 256 -c
installer.dmg: OK

What a matching hash proves, and what it does not

A matching SHA-256 proves that the bytes on your disk are the bytes the publisher hashed. That is worth having, because it defeats a compromised mirror, a tampered download in transit and an accidental truncation. It proves nothing about whether the publisher's file was safe to begin with, and it is worthless if you fetched the expected hash from the same place you fetched the file. If an attacker can replace the download, they can replace the checksum text next to it. Three rules follow:

1. Get the expected hash from a different channel than the file where possible: the project's own site over HTTPS when the file came from a mirror or CDN, or the release notes rather than the download link.
2. Prefer a cryptographic signature when the publisher offers one. A GPG signature, a Windows Authenticode signature or macOS notarisation ties the file to a key the publisher controls, which a hash alone cannot do.
3. Treat a hash match as necessary, not sufficient. Still scan the file, and still ask whether the source is one you trust. The step-by-step in is this file safe puts hashing in its place among the other checks.

Where else you meet hashes

Once you can read a hash, you will notice them everywhere. Package managers such as npm, pip and apt pin every dependency to a SHA-256 or SHA-512 in a lock file so that a build fails if a registry serves different bytes. Browsers do the same for scripts through Subresource Integrity, where a sha384- value in the script tag refuses a tampered CDN copy. Docker images are addressed by the SHA-256 digest of their manifest. Certificate fingerprints, torrent info-hashes and Git commit identifiers are all hashes of content. The common thread is that a hash lets one party describe a file so precisely that any other party can check it, without either side trusting the network in between.

Hashes and malware in practice

When a security team shares "indicators of compromise", they are usually sharing hashes, because a hash is the most unambiguous description of a file there is. If your scan report shows a SHA-256, you can search it directly in public feeds such as abuse.ch's MalwareBazaar or a multi-engine service and see whether anyone has already analysed it. A hash with no history is not clean, it is unknown; brand-new malware builds look exactly the same. A hash with detections from several engines and a family name is as close to certainty as a file scan gets. A hash that appears in a vendor's own release manifest is a strong reason to relax. Scan.now's file scanner reports the file's hashes alongside its structural findings so that you can carry them into these lookups; the file itself is analysed in memory and not retained.