How antivirus detection actually works

Five mechanisms run in series behind every alert. Following one file through them explains most of the vocabulary, including why clean files are sometimes flagged.

The sequence behind a single alert

An antivirus product is not one mechanism. It is four or five running in series, each cheaper and blinder than the one after it, arranged so that most files are settled by the cheap checks and only a few reach the expensive ones. Following one file through that sequence explains most of what the vocabulary in the glossary refers to.

Suppose a file arrives in your downloads folder. Before it runs, the product's on-access component intercepts the write and asks a series of questions.

1. Have I seen exactly this file before? The product calculates a hash and compares it against its local list. A match against a known-bad entry ends the sequence immediately. A match against a known-good entry, such as a signed operating system component, also ends it.
2. Does it contain a pattern I recognise? Byte sequences characteristic of known malicious families are matched against the file's contents, which catches variants that are not byte-for-byte identical to anything on the list.
3. Does its structure look wrong? Static heuristics examine how the file is built: whether it is packed or obfuscated, which system functions it imports, whether its declared type matches its contents.
4. What does the vendor's network say? If the file is still unresolved, a cloud lookup asks the vendor's servers whether this fingerprint has been seen elsewhere, how recently it appeared, and how many machines have encountered it.
5. What does it do when it runs? Once execution begins, behavioural monitoring watches the actions themselves and can stop the process partway.

Signature matching, and what it cannot see

A signature is a description precise enough to identify one known thing. In the simplest form it is a hash: change a single byte and the match fails completely. In practice vendors also use patterns that tolerate variation, so that a family of related samples is covered by one rule.

Signature matching is exact, fast and cheap, and it produces almost no false positives, because a match means the file is the thing described. Its limitation follows from the same property: a signature can only describe something already collected and analysed. Between a new sample appearing and a definition update reaching your machine, signature matching has nothing to say about it.

That gap is why every other method exists, and why the frequency of definition updates is a reasonable question to ask of any product.

Heuristics: judging by construction

Static heuristics look at what a file is made of rather than what it matches. A document that contains an embedded script, an executable whose code is compressed so that its contents are not visible until it runs, an installer that requests permissions unrelated to anything it claims to do — each of these is a signal, none is proof.

Products combine signals into a score and act above a threshold. The threshold is the whole design problem. Set it low and unusual but legitimate software gets flagged: a small developer's utility, a compiled script, an old installer. Set it high and variants pass through. Every product chooses a position on that scale, and some let you move it in the settings.

Behavioural monitoring: judging by action

Behavioural monitoring gives up on deciding in advance and watches instead. A process that enumerates a user's documents and rewrites each one in place is doing what ransomware does, whatever the file looked like on disk. A process that injects code into another process, or that adds itself to the startup sequence and then contacts an unfamiliar server, produces patterns that can be recognised without knowing anything about the specific software.

This is the only method that works against file-less activity, where there is nothing substantial on disk to scan. Its cost is inherent: the program has to begin running before there is anything to observe. Products mitigate that by acting on the first few actions and, in the case of some ransomware protection features, by keeping copies of files that a monitored process modifies.

Cloud lookups and reputation

A cloud lookup converts the vendor's whole installed base into a sensor network. When your product asks about an unknown file, the answer can draw on how many other machines have seen it, when it first appeared, whether it is digitally signed, and what it did on machines that ran it. A file seen for the first time anywhere, an hour ago, unsigned, is treated differently from one present on millions of machines for three years.

Two consequences follow. The first is practical: some detection capability depends on a working connection, and products behave differently offline. The second is a privacy question. Something about the files you run is being sent to the vendor — at minimum a fingerprint, in some configurations the file itself. What is transmitted, how long it is kept and where it is processed belong in the vendor's privacy documentation, and are worth reading there rather than assuming.

Sandboxing, emulation and machine learning

When a file is unresolved and consequential, some products run it in a sandbox first — an isolated environment where its actions can be observed without reaching the real system. Emulation is the lighter version: the file's instructions are interpreted rather than executed, far enough to see what the first stage of a packed program unpacks into.

A machine learning classifier sits across several of these stages. Trained on large collections of known-good and known-bad files, it scores new files by resemblance along features no human selected. It generalises better than hand-written heuristics, and it fails in ways that are harder to explain, which is why it is used as one input among several rather than as the decision.

The two ways detection is wrong

Every threshold in the sequence trades one error against the other. A false positive is a clean file called malicious; a false negative is malicious software not detected. Tuning a product to produce fewer of one produces more of the other, and no setting eliminates both.

False positives are visible and annoying; false negatives are invisible and worse. This is the reason independent testing laboratories measure both together, and the reason a detection figure quoted without its corresponding false-positive figure says very little. AgileBase publishes no such figures, because it runs no tests — see the editorial method.

What to do when something is flagged

An alert is a claim, and the reasonable response is to check it rather than to act immediately.

• Read the detection name. A specific family name means an identification; a generic one means a resemblance.
• Leave the file in quarantine rather than deleting it. Quarantined files cannot run, and a deletion cannot be undone.
• If a program stopped working immediately after a scan, the quarantine list is the first place to look.
• If you believe the detection is wrong, most products offer a route to submit the file to the vendor for review, which is more useful than adding a permanent exclusion.
• Use exclusions sparingly and specifically. An excluded folder stays excluded long after you have forgotten why.

What detection is not for

Detection is one layer, and several of the measures that matter most are not detection at all. The Australian Cyber Security Centre's guidance on securing devices is a reasonable free reference, and the emphasis there is consistent: keep software updated, use multi-factor authentication on accounts that support it, and keep backups you could actually restore from.

Those three do work no scanner does. Updates close the vulnerabilities that exploits rely on. Multi-factor authentication limits what a stolen password is worth. A backup held offline or in a separate account is the only measure that helps after encryption has already happened. Antivirus software sits alongside them rather than replacing any of them, and a product page that implies otherwise is overselling.

The next practical step is the comparison itself: choosing security software sets out what to compare and what to read in the billing terms.