I don't know how many data breaches I'm sitting on that I'm yet to process. 100? 200? It's hard to tell because often I'm sent collections of multiple incidents in a single archive, often there's junk in there and often there's redundancy across those collections. All I really know is that there's hundreds of gigabytes spread across thousands of files. Sometimes - like in the case of the recent South Africa situation - I could be sitting on data for months that's actually very serious in nature and needs to be brought public awareness.
The biggest barrier by far to processing these is the effort involved in disclosure. I want to ensure that any incidents I load into Have I Been Pwned (HIBP) are first brought to the awareness of the organisations involved and whilst that may seem straight forward, it's often quite the opposite. There are notable exceptions (such as the recent Disqus disclosure), but more often than not, it's a laborious process of varying success. Because this is something I do over and over again, I want to streamline the process and more than that, I want to seek community input.
Tell me if I'm doing this right. This post documents how I intend to handle serious incidents with real consequences and frankly, I don't want to stuff it up.
What I'm going to do below is document the process I follow then apply it to 3 separate breach disclosures of different types. They'll each culminate with data being loaded into HIBP, subscribers receiving breach notifications and the incidents possibly landing up in the media. The consequences for the organisations involved can be serious; we've just seen VTech fined $650k for an incident I was involved in disclosing a few years back. I don't take this lightly I want to ensure there's broad consensus on the way to handle these incidents. Let me explain my approach.
Defining Levels of Escalation
One of the biggest challenges with disclosure is getting a response from the organisation involved in the first place. Following my recent Congressional testimony, I wrote a series on fixing data breaches and in part 3 I covered the ease of disclosure, specifically how organisations could make the process easier for folks such as myself to report security vulnerabilities or indeed previous breaches. In that post, I spoke about people giving up when it gets too hard:
Many well-intentioned people simply give up and don't report serious security incidents when the effort is too high or the risk is too great. That has to change.
That's often the case when it comes to reporting security flaws, but when you're already sitting on the data that someone has taken from an organisation's system, it can play out quite differently. In this situation, the question is not whether disclosure happens or not, but rather how much effort I should go to before the data ends up in HIBP.
Most of this post is going to talk about levels of escalation, that is the phases I go through with each one increasing in effort. I'm going to start with the most obvious channels for reporting a breach and move through increasingly indirect ways get the organisation's attention. These channels and the timeframes in which I escalate through stages are essential - please comment on them!
Let me detail those levels then jump immediately into the 3 use cases I'll be applying the process to. Do note that I'm writing this before loading those 3 breaches; that'll happen immediately after I publish this blog post and they'll each reference it.
Level 1: Published Security Contact Info
Ideally, organisations acknowledge that security incidents may occur and they provide dedicated channels through which to communicate with them. They may adhere to a convention or be published elsewhere on their website:
- Look for a security.txt file at /.well-known/security.txt (read more about this)
- Look for a security vulnerability reporting policy (Tesla has a great example)
- Look for a published security email address
The reality is that it will be a rarity to find this information, but it's always the first preference and I wanted to include it here as encouragement to organisations. If no published security contact info can be found, immediately fall through to level 2:
Level 2: Published General Contact Channels
General contact channels are less ideal as organisations receive all sorts of communication via them. Reports of serious security incidents are likely to land next to Viagra spam; there's a poor signal-to-noise ratio and it may mean an expeditious response will not be forthcoming.
- Look for a contact us form
- Look for publicised email addresses
- Look for social media accounts that accept private communications
- Email "standard" addresses: security@, admin@, webmaster@, support@, postmaster@, hostmaster@
All the above channels may be used simultaneously - the same message can be sent to each. A period of 3 business days should be allowed for a response before proceeding to the next level.
Edit: Thanks to the comment from Stuart, point 4 was added after his feedback.
Level 3: Indirect Contact Information (Optional)
Once level 3 is reached, "reasonable attempts" have been made to contact the company. By this time, either no readily accessible contact information has been found or the company has not been responsive to the report. Level 3 involves either increasingly hard work or a decreasing likelihood of success so is flagged as optional.
- WHOIS records
- Company employees via LinkedIn
- Other sources of OSINT contact info
- Email addresses on the domain already in HIBP
If level 3 is used, provide a period of 3 business days for a response before proceeding to the next level.
Edit: Thanks to the comment from Rob, point 4 was added after his feedback.
Level 4: Public Requests for Information
A public request for a security contact can be construed as implying the company has a security issue. Whilst this may be true, alerting the general public to this (even without providing details of the issue) before the company themselves should be a last resort.
- Request security contact via social media
By this time, 3 to 6 business days have already passed (we're potentially more than a week in) and reasonable efforts have been made to contact the organisation. Public requests via the likes of Twitter get many eyes very quickly so if there are no solid leads within 24 hours, progress to the final level.
Level 5: Full Public Disclosure
By this time, there have usually been many hours invested (including verifying the incident), security contacts have been sought, all publicised contacts tried and sufficient time for a response has passed. The organisation simply won't respond and it's now time to publicly disclose the incident via HIBP.
Disclosure in Practice 1: The Fly on the Wall
It all started with this tweet:
Just hijacked some big MySQL database server containing 53K credit card details with complete CVV2 happy new years to the 4 million users pic.twitter.com/pXda5DbNCz
— Taylor (@0x55Taylor) December 31, 2017
The data was sent to me and after inspecting it, I found identified 84k email addresses in the breach. The data included passwords stored in plain text and a quick password reset check on a Mailinator account delivers the precise password in the breach to the public mailbox. It's almost certainly legit, let's move onto the disclosure process.
Level 1: No security.txt file at theflyonthewall.com/.well-known/security.txt nor a security vulnerability reporting policy or a security email address anywhere.
Level 2: Generic contact form located and completed, submitted at 23:44 UTC on Jan 6 with the following message:
Hi, my name is Troy Hunt, I'm a security researcher you can read about here: https://www.troyhunt.com/about/ I was recently sent a data breach alleged to have come from theflyonthewall.com and upon verifying it, I believe it's legitimate. The data indicates that your website has had a large amount of data extracted from it, including credit card numbers. Could someone in a security capacity please get in touch with me via email so I can provide further information.
Confirmation:
7 minutes later, I also sent the same message via their Facebook page:
And via DM on their Twitter account:
Level 4: I've given it more than 5 days (well beyond the 3-day target) and there's been zero response. Remember, I've tried the contact form, Facebook and Twitter by now so it's not from lack of trying. At 08:58 UTC on 12 Jan, I send out a tweet:
Anyone got a security contact at @theflynews? I've been trying to get in touch with them about a serious security incident for 5 days now and can't get a response by any published channel.
— Troy Hunt (@troyhunt) January 12, 2018
Level 5: No replies of substance or any other contact made, the data is loaded into HIBP at 06:47 UTC on 15 Jan and 386 individual and 815 domain subscribers are notified of the incident.
New breach: Stock market news website The Fly on the Wall had 84k unique email addresses breached along with purchase histories and credit cards. 64% were already in @haveibeenpwned. Read more: https://t.co/tv36UH1SX5
— Have I been pwned? (@haveibeenpwned) January 15, 2018
Disclosure in Practice 2: Open CS:GO (Counter-Strike: Global Offensive)
I'm handed a 10GB MySQL backup file with 512k unique email addresses titled csgo_20171128.sql which allegedly came from opencsgo.com. That URL promptly redirects to dropgun.com but the site doesn't exhibit any of the usual verification vectors I'd use because access is only granted via external services (Facebook, Twitter, Steam login). I fire off an email to the 30 most recent HIBP subscribers I find in the data (there are 816 subscribers in total in the data):
Hi, I’m emailing you as someone who has recently subscribed to the service I run, “Have I been pwned?” Your email address has appeared in a new data breach I’ve been handed and I’m after your support to help verify whether is legitimate or not. I’d like to be confident it’s not a fake before I load the data and people such as yourself receive notifications. If you’re willing to assist, I’ll send you further information on the incident and include a small snippet of your (allegedly) breached record, enough for you to verify if it’s accurate. Is this something you’re willing to help with? For verification, I’m on the about page of the site: https://haveibeenpwned.com/About
Within 14 minutes, the first person has responded, I've sent them information on their account and they've confirmed the accuracy. Others chime in over the next 24 hours, each confirming the legitimacy of their data and that they had indeed used the service.
So let's apply the process again:
Level 1: No security.txt file at dropgun.com/.well-known/security.txt nor a security vulnerability reporting policy or a security email address anywhere.
Level 2: Onto level 2, I prepare a message for them:
Hi, my name is Troy Hunt, I'm a security researcher you can read about here: https://www.troyhunt.com/about/ I was recently sent a data breach alleged to have come from opencsgo.com (which now redirects to dropgun.com) and upon verifying it, I believe it's legitimate. The data indicates that your website has had a large amount of data extracted from it, including personal information. Could someone in a security capacity please get in touch with me via email so I can provide further information.
This goes out via both email to their published address and Twitter DM at 21:04 UTC on Jan 7:
Their Facebook page doesn't allow messages.
Level 4: Same deal as The Fly on the Wall and at the same time as tweeting out for public support re them (08:58 UTC on 12 Jan), I send out a public tweet about Open CS:GO:
Anyone got a security contact at @dropguncom? I've been trying to get in touch with them about a serious security incident for 5 days now and can't get a response by any published channel.
— Troy Hunt (@troyhunt) January 12, 2018
And again, nada.
Level 5: Just like The Fly on the Wall, there's no replies of substance or any other contact made so the data is loaded into HIBP at 06:17 UTC on 15 Jan and 826 individual and 243 domain subscribers are notified of the incident.
New breach: Counter-Strike skins website Open CS:GO had 512k records breached including usernames, email addresses and purchase histories. 32% were already in @haveibeenpwned. Read more: https://t.co/tv36UH1SX5
— Have I been pwned? (@haveibeenpwned) January 15, 2018
Disclosure in Practice 3: Lyrics Mania
I receive a 4MB CSV file with 109k lines of email addresses, usernames and plain text passwords. The Lyrics Mania website is a basic site listing song lyrics and has no obvious means of registration or authentication so again, that makes verification hard. A bit of Googling locates a login page on another subdomain (oddly, without HTTPS when the primary site has it), but a password reset with a Mailinator address returns the same page (seems to just reload it and not properly submit). I literally diff the responses and there's nothing there to confirm or deny the request has even been received and no email lands in the Mailinator inbox.
It's borderline in the "too hard basket" for the return given the (relatively speaking) small number of accounts, but the plain text passwords make it a more serious incident so here I go again. I find 233 accounts of HIBP subscribers and email the most recent 30 as I'd just done with the CSGO breach. I immediately get back 2 "out of office" replies, both in German. Over the next day, multiple other parties reply and confirm their data is legitimate and that they've used the service. It's real, so let's apply the process again.
Level 1: No security.txt file at lyricsmania.com/.well-known/security.txt nor a security vulnerability reporting policy or a security email address anywhere.
Level 2: Similar message to before:
Hi, my name is Troy Hunt, I'm a security researcher you can read about here: https://www.troyhunt.com/about/ I was recently sent a data breach alleged to have come from Lyrics Mania and upon verifying it, I believe it's legitimate. The data indicates that your website has had a large amount of data extracted from it, including personal information. Could someone in a security capacity please get in touch with me via email so I can provide further information.
I find a contact page, fill out the form and submit it. It doesn't go so well:
There's a Facebook page but that doesn't allow messages to be sent. Without any means of contact directly published via their website, contacting these guys is becoming increasingly hard. Let's fall through to level 3:
Level 3:
Their WHOIS record has privacy enabled so there's no contact info of any use there. I search through LinkedIn and find one person with "Lyrics Mania" on their profile:
It looks like it's a hobby project for the guy:
Obviously, I'm going to be sympathetic to someone who's stood this up in their spare time for fun, but by the same token we're looking at over 100k passwords that people are using on their email accounts, banking and who knows what else. It's a serious incident and it has to be seen through.
I send him a message at 21:37 UTC on 7 Jan:
I also find that the Lyrics Mania Google+ page has an email address:
So I follow up 4 minutes later with an email to that address:
I'm not overly confident of an expeditious response, but I'll give it 3 business days and see what happens.
About a day and a half later, Midhun connects:
It's late for me so first thing the next morning my time (20:16 UTC on 9 Jan), I message him the following:
Hi Midhun, thanks for connecting. I run the data breach notification service "Have I Been Pwned" (HIBP): https://haveibeenpwned.com/ People regularly send me data exposed in data breaches and someone recently sent me 109,219 records of email addresses and plain text passwords allegedly from Lyrics Mania. A number of HIBP subscribers were in there and I've confirmed with them that the data is indeed legitimate. For your verification, here are 3 of the Mailinator addresses in the file: [redacted]@mailinator.com [redacted]@mailinator.net [redacted]@mailinator.com> Is there somewhere I can send you the data so that you can review it? If it is indeed legitimate, you'll need to get in touch with those in the breach and notify them, especially given there's plain text passwords which will be usable on other services too.
Less than 48 hours to get acknowledgement isn't too bad, it's the fastest turnaround of all 3 and it's the only one that's a hobby project too. Neither of the other commercial operations have responded yet.
Level 5: Unfortunately, more than 5 days later and even after accepting my LinkedIn invitation (which clearly flagged the security incident), I've had nothing back from Midhun. I load the data into HIBP at 06:34 UTC on 15 Jan and 233 individual and 284 domain subscribers are notified of the incident.
New breach: Song Lyrics website Lyrics Mania had 109k records breached including usernames, email addresses and plain text passwords. 78% were already in @haveibeenpwned. Read more: https://t.co/tv36UH1SX5
— Have I been pwned? (@haveibeenpwned) January 15, 2018
Incidentally, I know some people may be critical of my naming Midhun here. But if I'm to be transparent about communications then it's going to result in identifying those I've attempted to contact and who are responsible for the data. And it is a big responsibility too because as I said earlier, these are plain text passwords for all sorts of important accounts people will have reused them across and accountability must be taken when an incident like this occurs.
Request for Feedback
Reflecting on the levels I originally outlined and the process with these 3 breaches, was this the right approach? Did I attempt to contact the right channels, leave enough time and ultimately, do the right thing by loading the data?
It feels right to me but then I'm obviously very close to the whole thing and have my own unique take on breach disclosure. One of the most important things for me is to ensure that I can confidently say I took all reasonable steps to notify the organisation before loading the data. Plus, of course, I want to know that I've done the right thing by my HIBP subscribers too!
Thoughts?