GCP – Peer reviewed ‘allow’ and ‘deny’ software installation decisions to enable scalable protection
Among the controls any IT organization needs to have firmly in place, nothing ranks higher than blocking malware. As opposed to a “default allow” policy, which blocks only known bad software, the safer way to do that is through a “default deny” policy that blocks everything except allowed software. The obvious difficulty is that the more freedom you want to allow over the software your workforce can install outside your pre-vetted software, the more unmanageable the policy becomes. Multiply that freedom by the size of an enterprise like Google and you have a near-impossible scenario if you hope to dedicate a central staff to administering the approval.
How then do you approach the challenge of scaling this task given the infeasibility of scaling people to manage it? Our answer is Upvote, a system that distributes the workload of allowing software to the users themselves—through a process we call “social voting.” Think of it as a peer review for policy approvals.
How it works
Built on Google App Engine, Upvote consists of both a web-based frontend for user voting and a policy server that works with the Santa system for Mac OS and the Carbon Black Protection (formerly Bit9) system for Windows.
When a user (a Mac user, in this example) tries to run an unknown binary Santa—running in “lockdown” mode, allowing only allowed software to run—blocks the binary and Upvote allows the user to vote to allow it, surfacing a VirusTotal analysis so that they can make an informed decision. If others also vote to allow it and the total number of votes reaches a certain threshold, the voters—and only these voters—can then run the software.
This threshold is the first of two thresholds—a “local” one and a “global” one—that Upvote enforces. Voting continues even after the local threshold has been reached and anyone else who wants to run the software will still need to vote to allow it before they can run it. The voting stops only when the higher global threshold is reached, and only then is the software allowed for all users. You set the levels for these thresholds.
If in the course of voting, a user flags the binary as malware, the downvote temporarily disables the voting until an admin reviews the binary and either unflags it or downvotes it further to deny it as malware.
As the central policy server, Upvote regularly syncs binary metadata and policy updates with Santa so that users know within minutes whether they can run a given binary or not.
To avoid unnecessary blockage of software from known trusted sources, Upvote allows certificate approval at the admin level so that users can automatically run any software with an approved certificate in its signing chain. For Mac users, Upvote also features bundle voting: When a user tries to run an app that gets blocked, having to vote on its numerous binaries individually would be particularly onerous. So Upvote allows users to vote on an app’s entire set of binaries as a bundle.
Can users be trusted?
Allowlisting via social voting of course raises the question: Is it actually safe? Can users be trusted not to allow malware, however unintentional their doing so might be?
No doubt this approach comes with a certain degree of risk. Upvote minimizes this risk by enabling users to make informed decisions based on VirusTotal analyses. But even more important is Upvote’s threshold enforcement: Any potential infection is restricted by default to the subset of computers whose users have voted. The fleet as a whole is protected until the global threshold—which you’d naturally want to set as a very high bar—is reached.
Another significant benefit is that user vigilance is established by default. Since Upvote automatically blocks silent exploitation, like drive-by installs, it forces users to review the execution of any software that they didn’t deliberately intend. It encourages all users to become better stewards of their technology and data.
These benefits, we believe, more than offset the risks of user voting. Beyond the scaling that it allows for large or growing organizations, the convenience and, even more importantly, the productivity that users stand to gain from avoiding administrative blockage and having greater control over the software they need are a huge boon to any organization.
Development continues
We’re still actively developing and improving Upvote internally, evolving as we iterate. Based on the open source code we’ve shared, we have decided to also implement:
- Push notifications: Santa clients no longer have to wait for the next sync, which can take up to 10 minutes, to receive policy updates from Upvote. Users are notified and allowed to run binaries within seconds of voting on it. It also means that malware bans will propagate throughout the entire fleet within minutes.
- Lockdown exemption: Users who build and run new binaries may find lockdown mode onerous, so they are able to request to opt out of the default lockdown mode. If their policy checks pass, their machine is switched to “monitor” mode, which blocks only denied software. The granting of this request should be limited to exceptional cases, depending on your organization’s risk tolerance.
- Transitive allow/deny: Developers who frequently compile code can be especially burdened by a lockdown system, as every compile is subject to being locked down. This feature allows admins to “bless” a compiler so that every binary compiled by that compiler can run on the machine used to compile.
See the code to learn more
While we continue to evolve this solution inside Google, we’ve released an example repo to help you see how we handled early implementation. We won’t be publishing further updates to the codebase; we hope this can be helpful to other companies looking to better handle their allow/deny decisions at scale.
To learn more about Upvote and see a demo, check out this presentation at MacDevOpsYVR.
This post would not have been possible without the insights and expertise of Ben Grooters and Matt Doyle of the Upvote engineering teams who patiently answered our endless series of questions about the topic.
Read More for the details.
