A security control that can't do its job rarely says so. It returns the same green result it returns when everything is fine. That makes silent failure the most expensive failure mode in security tooling, because nobody investigates a problem the dashboard says isn't there.
We found it in our own code. aur-scanner is the open-source scanner Kief Studio maintains for the Arch User Repository (AUR). It reads a package's build files before anything is built and flags malicious patterns. Before promoting version 2.2.0 to stable, we audited the code against its own documentation, one claim at a time.
The serious findings weren't missed attack patterns. They were places where the scanner reported a clean result without having fully looked:
- A package file it couldn't read was skipped with a log line, and the scan came back clean.
- A dependency it couldn't identify was assumed to come from the trusted official repositories.
- An install wrapper that couldn't parse an argument passed the request through instead of stopping it.
- A user-level rules file could replace a built-in detection with a weaker one.
None of these shows up in normal use. Each one looks exactly like a clean result.
Three checks to run on your own security tools
1. Feed it something it can't read. Give your scanner, gateway, or DLP tool an input that is malformed, oversized, or unreadable and that also contains something it should catch. If the result is clean, you've found a silent failure. The correct result is "could not be analyzed," handled as a failure.
2. Find out what "unknown" defaults to. Every tool has a branch for inputs it can't classify: an unrecognized dependency, an argument it can't parse, a reputation service that doesn't answer. Check whether that branch allows or blocks. In a security control, unknown should block, or at minimum report clearly that the check didn't happen.
3. Test in a clean room, not on the machine that built it. Our full test suite, more than 600 tests, passed on the development machine. A fresh Arch Linux install found three more defects, including one that would have broken the upgrade for every existing user. A developer's machine is already configured correctly, which makes it the one environment where a packaging defect can't appear.
What changed in aur-scanner 2.2.0
- Files that can't be fully read are reported as a Critical finding (
SCAN-001), and every install gate treats that package as unreviewed. - Dependencies are classified by asking pacman, Arch's package manager. Anything that can't be resolved fails the check.
- The AUR-helper wrapper, the shell integrations, and the pacman hook refuse what they can't scan instead of passing it through.
- Community rules can add detections but can't replace or lower a built-in one.
- 142 detection codes, valid JSON, SARIF, and CycloneDX output, and a 201-check acceptance run on a fresh system as the release gate.
Install it with paru -S aur-scanner or yay -S aur-scanner. The package builds from a GPG-signed release tag and verifies the signature during the build. The full list of changes is in the 2.2.0 release notes.
The useful question about any control in your stack is what it reports when it can't tell.