Our HackerOne campaign & how it affects security at Vanilla
Linc
Admin
Hi all, I wanted to give you an update on the state of security practices at Vanilla, especially as it relates to HackerOne and our open source patch releases. I'll start with some background info, update you on the latest news, and then do a brief question & answer.
HackerOne background
HackerOne is a security bug bounty platform, which means it's a place for hackers to report discovered security issues in exchange for money, rather than using them maliciously. We launched our campaign there over a year ago and overall it's been very successful. Our rationale for doing so was that our open source users overlap very little with security researchers and therefore we receive little security scrutiny naturally. We decided it was in the best interest of both the company and the community to invest in more experienced folks investigating our software for problems.
In short, it's a very tangible way to prioritize security and invest in it, so we did so.
We've had an uptick in security releases since the program started, and moreso in the last few months. We expect to do more security releases before the end of the year. Basically: the program is working and our code base is being scrutinized more than it's ever been, so we're shipping patches as fast as we're able.
Some current metrics:
- We've awarded over $34,000 in bounties.
- In total, we've now evaluated 600 reports.
- Among them, we identified 139 legitimate reports from 73 different hackers.
- We average 1 day to ticket response, 6 days to triage, and 1 month to resolution.
"Resolution" for us means shipping an approved patch to both cloud and open source (all supported versions).
In the month of September we saw an abnormal surge in activity (more than 3x our normal level of legitimate reports) which caused some minor delays in delivering all patches.
Latest news
Yesterday, our company was blackmailed on Twitter regarding the open source release of a security patch. (Cloud customers were not affected by this threat; they were previously patched.) This was done by one of the HackerOne security researchers ("hackers") participating in our program (thus breaking the terms of service on HackerOne) who also made several accusations about our security practices. I will indirectly address some of those via question and answer below. We had awarded the hacker $3,000 in bounties, including $900 for the security patch in question prior to the blackmail.
The issue was submitted on Sept 18. 20 days ago, the hacker asked for an additional status update on the open ticket. We had previously explained the cause of delay (trying to bundle multiple security patches into 1 release to reduce the need for OSS updates) so we did not reply this time. 5 days ago, they asked again for an update and minutes later publicly messaged me a passive aggressive inquiry to my personal Twitter account (not the company's). We have escalation points listed in our campaign, we have a public company Twitter account, and my personal account is not tied to HackerOne in any way. I viewed this as an implicit threat and immediately blocked the user and ceased engaging over HackerOne. In retrospect, I should have passed on the HackerOne ticket to a colleague.
Yesterday, they included our company account on threats to 0-day our open source community (release details on an exploit for which no patch has been released) on Monday morning if the situation wasn't rectified to their liking. I am still unclear whether that happened. I later unblocked and engaged with them on Twitter to address some of the accusations made.
The direct consequence was a rushed late Sunday night release of Vanilla 2.6.4. In our haste, we accidentally included a config.php in the initial release that caused problems for some users. This was fixed by 8:20am ET this morning (thank you to Softaculous, @pioc34, and Twitter user console for quickly pointing out my mistake). We apologize for the oversight. We have fixed our tools to prevent a repeat.
You may recall the 2.6.1 release was similarly rushed. This was a process error on our part, not malicious activity. We accidentally agreed to disclosure of a security vulnerability that had not been patched in the 2.6 release. We've altered our process to prevent a repeat of that incident.
There's a lot of communication overhead to dealing with a HackerOne campaign in a growing company. We're learning quickly and adapting, and we apologize for the missteps we make along the way.
Question & Answer
How does Vanilla organize security releases?
We have a 5-day SLA on patching Critical exploits (usually we patch in 1 day). We backport all 'Critical' and 'High' severity patches to the latest stable releases of Vanilla. We patch cloud before open source. For Critical issues, this is usually a delay of minutes. For High issues, it may be delayed to open source by days or weeks to roll together multiple patches if they will soon be available. (We optionally backport 'Medium' severity patches at our discretion and do not backport 'Low'.)
Why does Vanilla sometimes hold security patches before releasing to open source?
The terms of our HackerOne campaign forbid disclosure until the report is resolved. We may hold a patch to group with additional patches in a single release if we have other similar severity reports open. We do this to avoid "update fatigue", where users start skipping updates if they are issued too frequently. We do not hold 'Critical' severity reports; they are always released immediately.
Why doesn't Vanilla issue CVE identifiers for reports?
Neither our customers nor open source community have requested this. It would require more time and steps to our workflow to accommodate this. In general, the only folks who have requested this are hackers seeking recognition. We previously provided public recognition in our release notes on this forum. We now provide it via HackerOne report disclosure. We do not contest the disclosure of resolved issues on HackerOne.
Why doesn't Vanilla make more explicit commit messages when patching security issues?
Sometimes we make commit messages that are either vague or technically accurate but not entirely forthcoming. We do this in the interest of not raising attention to an issue before we can finish the open source release process. Because our core software is entirely open source, a careful observer could gain awareness of a vulnerability and figure out how to abuse it before anyone has the option to update. Basically, we're buying you as much time as we can by not detailing every exploit and drawing attention to them. We understand some hackers take issue with this. We are more interested in protecting our users.
Why doesn't Vanilla have an automatic updater?
This is a huge feature fraught with its own security implications. We would support an open source effort to add this functionality to Vanilla, but we're not able to dedicate the resources necessary to it at this time so it seems very unlikely at this time. It remains the responsibility of each forum administrator to monitor our accounts for updates and do them in a timely manner. We use all available channels to communicate new releases when they are available (social media, email, and in-Dashboard feed). Before anyone says "but WordPress did it," kindly recall they are 1000x our size.
Thanks for reading and let us know if you have questions or suggestions in the comments.
Comments
One final postscript on a more personal issue.
It would be easy to rhetorically ask "Why would Lincoln think it's harassment to ask for an update over Twitter?" but there's a lot more to it than that. First, we had no prior relationship outside fairly anonymous interactions via HackerOne ticket. Second, my account was not linked or referenced, so they searched around to find me. Third, they chose to come after an individual employee rather than address our official account or any of the officially listed escalation points. Fourth, it was not an '@' reply but a full-view tweet that called me out, which was immediately liked by several of his hacker followers. Fifth, there is a severe power imbalance when an anonymous hacker engages with a person living very publicly on the Internet. He has all the info; me, none.
The impression I got was that this person was out to abuse their power. It turns out, I was correct.
I amended the number of legitimate issues and hackers in the OP. I'd posted the lower 3-month running total rather than the all-time total.
Thank you @Linc for taking the time to explain this to us.
Joining hacker one was a good idea, but no good deed goes unpunished.
grep is your friend.
Thank you
I love the transparency displayed here. Thanks for letting us know.