I imagine this is what it’s like when one of your kids gets old enough to finally beat you at something you’ve poured your heart into teaching them. Yes, I’m proud and it’s awesome that it has turned out so well, but I was still a little disappointed to get this the other day:
This came totally out of the blue for me which, of course, is exactly how it’s meant to work. If all this is unfamiliar to you, this is the paste monitoring feature of “Have I been pwned?” (HIBP) which I launched last month. As it happens, one of the domains I monitor for work had a hit in a paste titled VikingElectronics.com Email/PW Dump. (7.5k-ish Users.) – it was one of 7,381 unique emails in that paste actually.
Seeing this feature used in anger is a great opportunity to see how everything is ticking along. I’d done as much testing as I feasibly could, but there’s nothing like seeing real results from the tool organically doing its thing. Here’s what happened in the lifecycle from the paste being made right through to the last time anything related to it happened:
- 2014-09-27 04:34:42.00 (+0s): Paste was posted
- 2014-09-27 04:35:22.00 (+40s): Dump Monitor Tweet made
- 2014-09-27 04:35:26.84 (+45s): Tweet stored in HIBP DB and inserted into Azure message queue
- 2014-09-27 04:35:30.75 (+49s): Message retrieved from queue by worker role
- 2014-09-27 04:35:32.77 (+51s): Paste retrieved from Pastebin
- 2014-09-27 04:39:39.81 (+298s): Last email saved into HIBP
- 2014-09-27 04:40:10.20 (+328s): SendGrid SMTP server gets the message
- 2014-09-27 04:40:11.08 (+329): Email lands in my inbox
(Actually, there’s a point 9 that comes a couple of hours after all this – the paste is deleted.)
I can’t do anything about the initial 40 second delay and I’m pretty happy with it only being 11 seconds after it’s made that I’ve retrieved the paste from Pastebin. It’s a little longer than the median figures I observed in the aforementioned post I made when I launched the service, but by any reasonable measure it’s still very, very quick.
Obviously the big delay here is actually saving the email addresses from the paste into HIBP. I’ve gone backwards and forwards on the implementation of this a bit, but in short the process of locating and updating a row in Azure Table Storage isn’t super-fast, at least not without async’ing it across rows. Having said that, we’re looking at 30 records a second across a fairly large paste which isn’t too shabby. The email is sent right at the end of saving all the accounts to HIBP because I want to make sure that if someone gets a mail saying “Your account was found in a paste”, they can go and search for it immediately and actually find it. I also consolidate the notification when domain emails are sent (i.e. “There were 3 email addresses from your domain found”) so I need to fully process them all and save them to Table Storage before sending the email notification.
It’s only been a few weeks since I launched this new service and we’re already looking at 1,000 new pastes since then encompassing 800,000 accounts. That’s over a quarter of a million additional accounts being added every single week. In fact it’s significant enough that between me beginning to write this blog post in response the alert above and clicking on the publish button, I got two more of them. One of the alerts actually included accounts from two separate domains I monitor. That’s what I really love about this service – it just sits there in the background ticking away, watching what’s going on and sending out emails as required.
Notifications are awesome. They’re also free and there are going on 100,000 people getting them when verified data breaches or pastes are loaded into the system. Get on over and set up notifications because chances are you’re like me – we’ve either been pwned or almost certainly will be pwned and when that happens, I want to know about it ASAP.