Sponsored by:

Too big for Google – when Analytics fails you

This is one of those “I’m writing this because it will be useful for someone else when they really need it” posts. The other day, Google Analytics stopped logging traffic to “Have I been pwned?”:

Graph showing no data for the last two days

The thing had launched with a bang and everything was awesome then… nothing. Heaps of data for Dec 5 (the first full launch day) through 7 then absolutely nothing for the 8th and the day I took the image on the 9th. Odd thing was, I’d just seen the stats during the day on the 8th and when looking at things on an hourly basis, everything was just fine so where had the data gone?! Whilst this image is from the AnlyticsPro iOS app, it was the same deal when opening the Analytics website.

This is a disconcerting feeling because you begin thinking you’ve lost data. Did I inadvertently delete or purge the account? Did I screw up the tracking tags? Is my account locked out?

However, when I jumped over to the browser and look at the Google Analytics Real Time report, everything looked very healthy indeed:

Analytics Real Time showing 543 active visitors on the site

So the tracking tags are good and activity is certainly getting sent to Google, what gives?! As it turns out, there are limits on these things:

Data processing latency: 24-48 hours. Standard accounts that send more than 200,000 visits per day to Google Analytics will result in the reports being refreshed only once a day. Daily processing begins at 12:00 UTC and continues for approximately 10 hours. This can delay updates to reports and metrics for up to two days. To restore intra-day processing, reduce the number of visits you send to < 200,000 per day. For Premium accounts, this limit is extended to 2 billion hits per month.

Whilst I didn’t think I was quite at 200,000 visits (the 7th was 93k with 129k page views), sure enough the next day things came good again:


Graph now showing data collected for an additional day

Great that I haven’t lost any data, not so great that I can’t easily analyse things on an hourly basis any more. More worrying though is this statement:

Data collection limit: If you exceed 10 million hits per month, there is no assurance that the excess hits will be processed.

That’s 322k “hits” per day but in combination with the event tracking I’ve just dropped in (I had no easy way of monitoring the numbers of searches and successes versus failures), a busy period could start pushing that limit. Ok, so I could always just get the premium service they offer but, um, that’s $150k. Annually. For a website that cost me about $50 to build (I bought a few domains early as I wasn’t sure which one I’d use). Think I’ll be scaling down the sample rate before that happens!

So that’s it – no loss of data, just some lag on the reporting. One of the many things I’m now discovering by running a website of this scale!

Tweet Post Share Update Email RSS

Hi, I'm Troy Hunt, I write this blog, create courses for Pluralsight and am a Microsoft Regional Director and MVP who travels the world speaking at events and training technology professionals