Dynadot

Reliable stats?

NameSilo
Watch

D Haynes

Top Member
Impact
5,866
I have a very basic site I'm working on and I noticed some wild differences in the traffic reports from the host provider and Google analytics. Hostgator is telling me I'm getting between 400 and 500 unique visitors a day and Google is telling me I'm getting about 20 a day. I know that Hostgators stats will include bots etc but they do detail how many of these are part of the final number. I also read somewhere that Google doesn't report all visits, including people not using a java script. So I was just hoping the more experienced of you could tell me who or what I should be trusting in these reports or if there are better tools available etc. Thanks in advance.
 
0
•••
The views expressed on this page by users and staff are their own, not those of NamePros.
I don't know but I've used google analytics and I found it very confusing.

Have you ever tried StatCounter?

They have a section when you create a new project where you can click a setting that "removes bot traffic".

-Omar
 
0
•••
I don't know but I've used google analytics and I found it very confusing.

Have you ever tried StatCounter?

They have a section when you create a new project where you can click a setting that "removes bot traffic".

-Omar

That sounds very useful! Thanks Omar I'll be sure to check it out!
 
1
•••
That sounds very useful! Thanks Omar I'll be sure to check it out!

Anytime! Also, I forgot to mention, its free. They do ask for you to upgrade if you have a lot of traffic but to start and get the hang of it, the free version is very good.

-Omar
 
1
•••
There is no standard definition of unique visitor. What is yours ? What do you expect a unique visitor to be ?
 
0
•••
Hi there, I use Google Analytics and I 100% agree with you that there is an issue with bot visits being counted and giving you a larger number than what is really happening.

This is how I manage to count 'real' visitors using Google Analytics. It's a little clunky but it works:

1. When you're on your site data, on the vertical column of options on the left of the screen, click on "Aquisition", then on the "Overview" sub-menu.
2. It'll pull up some data on what method your various visitors got to your site. I total up my estimate of "Real Visitors" by adding up the number of visits from "Organic Search" + "Social". I pretty much ignore "Referral". Most of those are bots, at least for me. The exception is if you know you've legitimately been featured/linked to from someone else's site. If that's the case, click on your "referral" visits to break them down by site, and see how many you received from that particular site you'd been featured on. Even for my site that has been featured strongly on someone else's site, I personally found that their referrals count for only a small percent of the total referrals, the rest of which I always assume are bot visits. I would not recommend clicking on any of those bot sites that come up on your Referrals list, even if they look legitimate. Many are malware/spam sites.

Like I said, this method is clunky, but it'll give you a much closer estimate than the raw data. I think this issue is a big loophole in Google Analytics, and I can't believe they don't have better bot filters. I do have a couple of other options for you to get a feel for your visitors, but the other 2 options are in a sense even more limited than Goolge Analytics.

Extra option 1: If you're running AdSense on your site, I actually use AdSense as a fairly accurate measure of traffic. If you look at your AdSense stats under "Performance Reports", it'll tell you how many page views you had. NOTE: Don't get confused between page views and impressions (=ad views) - if you have 2 ads per page then your ad views will generally = 2x your page views. Page views is what you need. AdSense does a pretty good job of filtering out bot traffic and your own visits. I find that usually, the number of views from my Google Analytics "real visitors" that I told you about above, is pretty close to the number of AdSense page views. It's not the same exact number, but it's in the ball-park. Obviously this only works for sites that are running AdSense. It won't tell you which visitors are unique visitors. FYI - If you have multiple sites you can get Adsense results per site by setting up URL channels.

Extra option 2: You can get a feel for your real visitors by checking your Alexa.com stats. It's not very effective because your site won't even register on Alexa until it gets a certain amount of traffic. Also, your Alexa rank won't necessarily tell you much about actual numbers of visitors to the site. I did find they do a pretty good job of estimating 'real' traffic only, not bot traffic though. It's the least accurate way to measure anything, but it does give you a good relative feel of how your site is doing over time, relative to other sites out there.

@Kate - I'm not sure if I have possibly misunderstood your question? Unique visitors is generally defined as visits originating from a specific IP address. In other words, if you have a site that I visit 6 times that week from my home computer, it would be 6 total visits, but my visits would only count as 1 unique visit over that time period. (However, if I visited you once from my home computer, once from work, once from my friend's place etc then those would count as unique visits since they're from different IP's).

So in conclusion @UniflexDomains , you have identified a very real problem and there is no one-size-fits-all answer to it. My preferred method is the Google Analytics calculation I outlined at the beginning of my reply, but it's far from perfect. I'm interested to learn what alternative methods others are using, and I'd love to hear from others on this.
 
1
•••
There is no standard definition of unique visitor. What is yours ? What do you expect a unique visitor to be ?

Well I don't know if this is the correct definition on the internet but to me a unique visitor is someone who visits your website which counts as 1. If they visit your website again on that same day, they WILL not count as a unique visitor. They will count as a website visitor (or a hit) and add to your overall stats but are not a unique visitor since this is not the first time of them coming to your website.

So that's what a unique visitor is to me. Basically every unique IP address coming to your website on a daily basis.

-Omar
 
1
•••
@Kate - I'm not sure if I have possibly misunderstood your question? Unique visitors is generally defined as visits originating from a specific IP address. In other words, if you have a site that I visit 6 times that week from my home computer, it would be 6 total visits, but my visits would only count as 1 unique visit over that time period. (However, if I visited you once from my home computer, once from work, once from my friend's place etc then those would count as unique visits since they're from different IP's).
Something like that. A unique IP address.
However, you can have two or more different people connecting from the same IP address (especially from proxified/corporate networks). The cookies and browser fingerprints could be different, so you could tell them apart, but it could still be the same person running two browsers. Cookies can help tracking, but they may not be persistent (some users wipe them).

Another problem is how many times do you count each unique IP address. Every 24 hours ? 48 ? Something else ? IP addresses can be dynamic and recycled. There is a small chance your IP address will be allocated to a completely different person later.

Then you have the bots. Some have a specific user agent and are easy to isolate, but others choose to masquerade as ordinary browsers. It can be hard to spot them without looking thoroughly at the logs.

The only way to get a good approximation is to use a combination of different techniques. Some of which may be controversial from a privacy POV because they potentially aim at deanonymizing users.

This is the problem here, there is no definition of what a unique visitor is, and many stat counters do not explain their criteria. Therefore the results will vary a lot, and it's comparing apples to oranges.

Maybe the best is to decide for yourself. If you host the sites, you have access to the logs and you can parse them according to your own criteria.
 
2
•••
Wow, I see from what @Omar Negron and @Kate wrote that defining a unique visitor is not necessarily as straightforward as it first seems! I can see the issues here.

Another problem is how many times do you count each unique IP address. Every 24 hours ? 48 ? Something else ?
To answer this, as far as I understand, Google Analytics counts unique IP's over whatever selected time period you ask it to look at. See https://support.google.com/analytics/answer/2992042?hl=en If it's a week, then it'll be unique IP's over that week. If it's a month, unique IP's over that month.

However, I 100% take your point @Kate of the criteria varying between different stat counting sites/methods:

many stat counters do not explain their criteria. Therefore the results will vary a lot, and it's comparing apples to oranges.
So true.

I still think that Google Analytics gives a reasonable estimate if you ignore "direct" traffic and "referral" traffic (unless you happen to know you have a lot of legitimate direct or referral traffic, but for most sites that's not the case).

@UniflexDomains - I also noticed I totally forgot to address your discrepancy with HostGator. Again I think this is probably an "apples vs oranges" situation as Kate mentioned, but I also wanted to add that hosting companies can often be a lot less sophisticated than Google Analytics when it comes to counting visitors. And Google Analytics is hardly a stellar tool in itself, as has been mentioned in the discussions, so the HostGator stats may be very 'raw' stats. The reason hosting companies can be less sophisticated is that they may just simply track total number of visits - these may include Google's own crawlers, or other legitimate web-crawling. Thus you may get a higher number from your hosting company. I've never used HostGator so again this is something where you need to look into their tracking/visitor criteria if you really want to get into the discrepancy, even if it's a large discrepancy such as what you mentioned. As @Kate said, this may not be something they are willing to tell you.

FYI - Hosting companies have a slightly vested interest in using the most 'optimistic' numbers (so that you think your site is doing really well on there), so they are more likely to be using the most permissive parameters when measuring traffic.

But to get to your question in the simplest possible manner without having to delve deeply into who is measuring what and how, I feel the best rough approximation that I've found over the years is still Google Analytics but without including referral or direct visits. I've developed a bunch of sites over the years. At any time I usually have about 7 to 9 running at any one time. I've found that the Analytics calculation I gave seems pretty reasonable. It seems to tally well with AdSense (which is quite picky about trying to distinguish 'real' visits).

Still, the issue of who is measuring what and how is still a real issue, and I'd definitely be eager to hear more on this.
 
1
•••
Still, the issue of who is measuring what and how is still a real issue, and I'd definitely be eager to hear more on this.

I wish I could help you out more but I'm honestly not that smart with those analytical things. I log into my stat counter and look at the stats they give me. I may sure I click "remove bot traffic" and hopefully the stats I am looking at our correct. These companies spend a lot of money to build out their systems so boggling my mind trying to figure out everything within it hurts my head.

Also, if you have a question with their specific program possibly send them an email and see if you can get a response straight from the company. Like I mentioned earlier, I use Stat Counter so I believe they would answer you personally. Writing google for G analytics would probably take a long time to get a personal response from them.

I wish I could help out more though!

-Omar
 
1
•••
Google analytics is excellent and reliable. Stat counter is meh but pretty lightweight - as I recall very limited in the detailed information they provide, especially the free version.
1. When you're on your site data, on the vertical column of options on the left of the screen, click on "Aquisition", then on the "Overview" sub-menu.
2. It'll pull up some data on what method your various visitors got to your site. I total up my estimate of "Real Visitors" by adding up the number of visits from "Organic Search" + "Social". I pretty much ignore "Referral". Most of those are bots, at least for me. The exception is if you know you've legitimately been featured/linked to from someone else's site. If that's the case, click on your "referral" visits to break them down by site, and see how many you received from that particular site you'd been featured on. Even for my site that has been featured strongly on someone else's site, I personally found that their referrals count for only a small percent of the total referrals, the rest of which I always assume are bot visits. I would not recommend clicking on any of those bot sites that come up on your Referrals list, even if they look legitimate. Many are malware/spam sites.

Like I said, this method is clunky, but it'll give you a much closer estimate than the raw data. I think this issue is a big loophole in Google Analytics, and I can't believe they don't have better bot filters. I do have a couple of other options for you to get a feel for your visitors, but the other 2 options are in a sense even more limited than Goolge Analytics.
.

Ignoring your referral and direct traffic isn't the way to go...most sites get a lot of legit traffic from both channels.

What you're talking about is referrer spam, and it's a total pain in the butt. Much of it now is ghost referrer spam - which spoofs a visit. They send their data directly into analytics and never actually visit your site.

You can set up filters in analytics to remove it from your view (and the stats shown in that view). You have to maintain the filter though, because new ones are constantly popping up. If you google it, people have shared lists of urls to filter.

Questions about analytics - ask in the Google product forums - https://productforums.google.com/forum/#!forum/analytics
(And they have a section on referrer spam and how to get rid of it.)
 
4
•••
Thanks for stepping in @enlytend. I guess I had been roughly equating "number of traffic visits" to "traffic I care about" but upon reflection they are not the exact same thing. I tend to only count traffic I care about (i.e. that I perceive to be engaged traffic). But true, it's not the same thing as total traffic, which I guess is what the OP was asking.

I do think it's important to pose the question of "Should I count traffic that has a session duration of 0.00?" I tend not to, and most of my referrals are in that category, which is why I ignore it. But from a purely definition-based point of view, I can see how an argument could be made to count it as actual traffic.

Anyway I am just trying to point out why I use the approach I use for my sites and why *I* ignore the referrals (except for sites I personally know or that have session durations greater than 0.0 - that in itself is another whole question/discussion in itself!). I agree definitely setting up Analytics filters to remove ghost spam is the best way to go - I've not done it because I don't want the hassle of adding a new filter every time a new referral spam site pops up.

So yeah, you should look into your Google Analytics referrals or direct traffic to see what's going on with them - maybe like @enlytend said, they might be a good source of traffic for you. They've not been for me, but that doesn't mean it might not be for you, so I definitely take that point.

Thanks @enlytend for stepping in, it helped me realize that "traffic I care about" isn't necessarily any relation to "number of traffic visits" - they are 2 different things.

No matter what method you use, scarily, almost any number or stat you care to look at has its own limitations in measurement and/or significance....!
 
0
•••
Well I don't know if this is the correct definition on the internet but to me a unique visitor is someone who visits your website which counts as 1. If they visit your website again on that same day, they WILL not count as a unique visitor. They will count as a website visitor (or a hit) and add to your overall stats but are not a unique visitor since this is not the first time of them coming to your website.

So that's what a unique visitor is to me. Basically every unique IP address coming to your website on a daily basis.

-Omar
This is also what I thought was unique visitor too @Kate

@Vivi thank you very much for you detailed answers. I very much appreciate the time you've spent and the information you have given.

It appears that getting a 100% accurate read of human visitors is near on impossible and different providers give you a different ball park figure in different ways which just makes it a choice of who's system you trust more I guess although some selling platforms like flippa use Google stats as a proof of traffic when from what some have said and from what I've seen myself, it isn't very reliable.
 
2
•••
2
•••
0
•••
This is how you go about compiling your own stats. The basic concept is simple to understand.
Skills required: scripting (PHP + mysql or other).

Locate and parse your Apache log file. It is a regular text file. The format is not hard to understand: http://www.the-art-of-web.com/system/logs/
As an exercise import just these 3 fields into a mySQL DB:
  • IP address
  • timestamp (date/time of visit)
  • User agent (browser string)
Next step is to compile a list of user agents you want to exclude ie GoogleBot Yahoobot etc. You can get a list from your own stats.
=> SELECT DISTINCT in SQL
Flag those user agents for exclusion.

Then you can start counting the number of unique IP addresses on a day to day basis
=> GROUP BY in SQL
For this you need to 'truncate' the timestamp to a day value.

Here you go !

Now there is a lot of possible fine-tuning and improvements. For example you can exclude unique hits: browsers that have hit your home page, but didn't make any further requests. You also look at how much time each IP address has spent on your site. For example, if the time difference between the first and the second request is 30 seconds, you can assume the visitor has stared at (and hopefully read) the home page for 30 seconds.
If the time difference between the first and the last request recorded is 5 minutes, you can assume the visitor has stayed for a total of 5 minutes but probably has been doing other stuff at the same time (opening multiple browser tabs).
 
3
•••
So yeah, you should look into your Google Analytics referrals or direct traffic to see what's going on with them - maybe like @enlytend said, they might be a good source of traffic for you. They've not been for me, but that doesn't mean it might not be for you, so I definitely take that point.

Businesses generally get a lot of both that are genuine.

If you have a domain that gets type-ins or people know the name of your business /site, you'll get direct traffic worth paying attention to.

Referrals - from backlinks, but also some 2nd tier search engine traffic has been known to end up in here. Let's say you have a business that arranges housing for seniors and have links from local government and eldercare sites - you probably want to see your referrals!

If you only want to see and measure sessions of a certain duration, check out the advanced options above the stats on the right :).

Piwik is a great analytics clone, and worth using if you are paranoid about sharing your data with Google ;). On the plus side, you wont get ghost referrer spam. On the minus side, it's more labor intensive to install and maintain, and I don't think it can integrate with your Google Search Console data like analytics does. And a lot of SEO tools can integrate with the analytics api but not with piwik.
 
2
•••
Thanks everyone for your replies. It seems this is an area where someone can get a lot of information that just entering a google tracking code if they're willing to put some time in. Certainly a lot more in depth than I though. Thank you all again.
 
2
•••
Now i just read over everyone's input and one thought that came to mind is, wouldn't the best way judge how your site is doing by simply looking at the interaction?

  • What I mean is how many Facebook shares are your post getting?
  • How many shares by Twitter or any other social venue available?
  • How many comments?

I believe these are the true measurements of people coming to the blog. As far as the number of unique hits and visitors even though the number you see might not be 100 percent reliable it should give you an idea.

Also looking at different tools like Stat Counter an then cross referencing to something else like Google Analytics or even the traffic analytic tools already within your hosting should help. I know Hostgator has some type of analytics and traffic stats you can see within the Cpanel.

- Will
 
2
•••
A good point you make Will. Sales/comments/replies/inquiries etc don't lie.
 
0
•••
Now i just read over everyone's input and one thought that came to mind is, wouldn't the best way judge how your site is doing by simply looking at the interaction?

  • What I mean is how many Facebook shares are your post getting?
  • How many shares by Twitter or any other social venue available?
  • How many comments?

I believe these are the true measurements of people coming to the blog. As far as the number of unique hits and visitors even though the number you see might not be 100 percent reliable it should give you an idea.

Conversions are very valid to measure, but we were talking about measuring traffic - conversions don't reliably indicate how much traffic you're getting or the nature of that traffic. A site can get a ton of traffic and convert poorly, or get a modest trickle of visitors and have a great conversion rate. And all sites don't perform equally well on social media

Having the whole picture is ideal. If you know how much traffic you're getting, what that traffic is, and how different segments of it perform for relevant metrics (could be sales, leads, newsletter sign ups, social shares, phone calls...) then you know whether the answer to getting more conversions lies in getting more traffic or in optimizing for better conversions. For example, if you see that your mobile traffic has significantly poorer performance than desktop you may want to revisit how your site looks and functions on mobile before just throwing more traffic at it.

Btw, in Google analytics (and piwik) you can tie traffic data directly to conversions by configuring "goals". Can't do that with the simple log analysis programs or statcounter - at least not the free version.

Hostgator and other Cpanel hosting have awstats, analog and webalizer, which are very basic tools which analyze your activity logs.
 
2
•••
you can filter out all this ghost spam with custom segments in google analytics.
don't be confused between filter and custom segments in analytics.
if you set a filter, the data will be filtered beginning with the day you set the filter (historic data won't be filtered).
If you create a custom segment and activate it, you can add it to your data view and it also filters historical data. you can add and remove it whenever you want.

this may helps:

click "add segment" - on the left side "Conditions" - set "Medium exactly matches referral"

AND
Source contains buttons-for-your-website

OR
Source contains semalt

OR
Source contains snip.to

OR
Source contains .....

OR
Source contains .....
....

also check out your organic traffic for ghost spam by choosing "hostname" as secondary dimension. you may find ghost spam here as well, add those to your custom segment as well by adding more filters in your custom segment:

Hostname contains iloveitaly.com
Hostname contains с.новым.годом.рф
Hostname contains.....

you're likely to find new spam again and again in your stats so you should check your referrals and hosts and update your custom segment before diving deeper into the data.
 
1
•••
Back