Category Archives: Spam Filter

Direct ‘Spam’ in Google Analytics

A number of my clients have seen unexplained direct traffic in their Google Analytics accounts. There is no apparent purpose or reason behind this traffic — it just exists; sometimes for a short period of time, and sometimes for months. A number of theories have been thrown around, including ad click fraud, affiliation fraud, purpose-built crawlers that hide their identity, and more. Whatever the reason, people need to be able to see the non-bot traffic to their websites.

What can you do if you are affected?

Check your Referral Exclusion List

The Referral Exclusion List prevents traffic from the listed domains from starting a new referral session. If your site links to a partner site (like a payment gateway), this would join an initial session on your site with a returning visit from the partner site.

If you list other domains (like spam referrals), it will strip the referrals off the session, but the sessions remain — as direct visits. Use exclude filters in your view for spam referrals.

 

Check City, Service Provider and Network Domain

Sometimes there is a single crawler, bot or service causing the whole thing. Maybe it is an ‘uptime’ monitor that you initiated. If it all comes from one place, you can easily build a filter to eliminate it.

 

Check Browser and Operating System Versions

If the traffic comes from various locations, it may be generated from one of several bots that have been propagated around the internet. Look for a spike in direct traffic from old browser versions like Firefox 11.0, Internet Explorer 7.0, or Chrome 18.0.1015.168. Programmers sometimes use code they found on the ‘net and it uses embedded versions of these browsers. Real people tend to upgrade their devices; bots tend to stay the same. If you aren’t sure about a particular version, check the traffic using that version over the past 6 months and see if it is all direct, or if it includes any organic search visits.

 

Check Browser Size

When programmer’s make a bot, the code can spread across browser versions, and maybe even operating systems, but if they open a hidden browser window to visit your page, it is probably the same size, every time. In particular, look for sizes like 620×480 or 610×480 that have a lot of direct visits but no other sources.

 

Helpful Filters I Have Used

After performing a significant analysis on several client sites, I came up with this series of filters, which must be implemented IN ORDER. It works on the theory that most of the direct spam was from old browsers or specific browser sizes, and it landed only on the home page of the web site. Note that there were always a few people running really old computers that happen to match the profile, and they visit other pages of the site as well. I did not want to filter out this other traffic, especially if it included ecommerce transactions!

Filtering out visits from a  specific service provider is fairly simple, but how do you eliminate visits based on multiple criteria, like visits to the home page from a specific browser and version? Google Analytics Advanced Filters offers a solution.

A note of caution: this is not a comprehensive guide to advanced filters — that really is an advanced topic. I offer it as an example of what I have used for my own clients so that others can benefit. Also, it worked for me in the past…it may not work for you in the future.

Capture the Browser and Browser Version

The first filter gets the browser (Firefox, Internet Explorer, Chrome, etc) and the browser version (11.0, 43.0.2357.130, etc) into a single blob that I can work with — I used a field called “User Defined”. After the filter runs, the User Defined field will contain things like “Firefox 11.0”.

Tag Unwanted Traffic

Next comes a series of filters that change the User Defined value to “Direct Spam” if it happens to use a specific browser version AND is a hit to the home page (in this case, “/Home”; yours might be “/” in which case use the expression “^/$“).

The list of browser versions I have used includes (verify on your own site before you adopt these):

Firefox 11.0
Internet Explorer 7.0
Chrome 18.0.1025.168
Chrome 39.0.2171.95
Safari 5.1
Chrome 43.0.2357.130
Internet Explorer 9.0
Internet Explorer 8.0
Firefox 3.6.28
Firefox 12.0

Tag Other Unwanted Traffic

You may have other criteria, like browser size, that you want to use – include similar filters for them as well.

Exclude Tagged Traffic

Finally, look at the value in User Defined, and if it has been tagged as “Direct Spam”, then exclude it.

 

Put The Filters In Order!

This is critical! The first filter (capturing the browser and version) MUST be first, and the last filter (excluding the spam) MUST be last.  Google will execute the filters in order, and that is the only way they will accomplish the task.

 

Enjoy It While It Lasts

You should now see a drop in direct spam visits as the filters take effect (use a segment for historical reporting). Just remember that things change and you will need to revisit the effectiveness of the filters as time passes, especially if you see a jump in direct visits again.

This article describes a technique I used that was effective for me. Every website is different– you need to do your own analysis before implementing a solution.

Spam in Google Analytics?

Yes, the popularity of Google Analytics has gotten to the point that it has become a popular target for spammer. They have discovered that they can spoof the system into thinking that real traffic went to your website from fake referrals, searches and events.

Their objective: make you go to their website to see what is going on. Don’t do it! 

What is a ghost referral?

A ghost referral is a fake visit created by spammers spoofing the Google Analytics tracking servers into thinking that your website had referral traffic from their site. In reality, no one ever visited your site! They don’t even know what it is – they randomly select tracking id numbers. A Valid Hostname filter is the best defense.

What is a ghost visit?

A ghost referral is a fake visit created by spammers spoofing the Google Analytics tracking servers into thinking that your website had real traffic, and it could appear as a referral, as an organic search with fake keywords, as direct visits with fake landing pages and even as fake events. In reality, no one ever visited your site! They don’t even know what it is – they randomly select tracking id numbers. A Valid Hostname filter is the best defense.

What is a valid hostname filter?

When traffic hits your web server, it records the server name in the Google Analytics ‘hostname‘ field. Since spoofed traffic doesn’t actually know what your website is, it contains fake hostname values. Your can easily create a filter to let in only traffic that contains your valid hostname value. See the Definitive Guide for detailed instructions.

Definitive Guide to Removing All Google Analytics Spam

This is a PROVEN WORKING SOLUTION with complete filter expressions [2017-07-17].

How to Prevent and Remove Spam:

  1. BEFORE YOU START: Make an Unfiltered View!
  2. Implement a Valid Hostname Filter to eliminate ghost visits (like sell-fb-group-here.com, piulatte.cz and track-rankings.online). Also eliminates fake keywords like cdn site:nodepm.com.
  3. Implement Spam Crawler Filters to eliminate the targeted spam visits (like share-buttons-for-free.com and foxweber.com)
  4. Create a Custom Segment with these filters to use for reporting

All the information you need (and more!) is provided below in this step-by-step guide. Filter expressions are updated within a day or two as needed.


Tired of fighting spam? Let a professional do it for you!

Daily monitoring and filter updates: only $75/year. Start Today!


Current Spam Filter Expressions

Read the rest of the article below to learn more about how spam has changed over time and how to implement these filters properly. See the bottom of the article for a running list of spam referrals these filters block (updated almost daily).

Valid Hostname Filter: customize to suit your web server domain (see discussion below)

mydomain.com

Spam Crawlers Filter 1: [ 2015-06-01]  Custom > Exclude > Campaign Source

semalt|anticrawler|best-seo-offer|best-seo-solution|buttons-for-website|buttons-for-your-website|7makemoneyonline|-musicas*-gratis|kambasoft|savetubevideo|ranksonic|medispainstitute|offers.bycontext|100dollars-seo|sitevaluation|dailyrank

Spam Crawlers Filter 2: [2017-03-26]  Custom > Exclude > Campaign Source

videos-for-your-business|success-seo|rankscanner|doktoronline.no|adviceforum.info|video--production|sharemyfile.ru|seo-platform|justprofit.xyz|127.0.0.1|nexus.search-helper.ru|dbutton.net|o00.in|wordpress-crew.net|amazon-seo-service|enbersoft.com

Spam Crawlers Filter 3: [2017-03-20]  Custom > Exclude > Campaign Source

fast-wordpress-start.com|top1-seo-service.com|uptimechecker.com|uptimebot.net|rankings-analytics.com|^uptime.com|.responsive-test.net|dogsrun.net|free-video-tool.com|keywords-monitoring(-your)?-success.com|a.pr-cy.ru|share-buttons-for-free

Spam Crawlers Filter 4: [2017-03-01]  Custom > Exclude > Campaign Source

fix-website-errors.com|seo-2-0.com|platezhka.net|timer4web.com|1-99seo.com|1-free-share-buttons.com|uptime-alpha.net|3-letter-domains.net|datract.com|lifehacĸer.com|top10-way.com|google-liar.ru|motherboard.vice.com|petitions.whitehouse.gov|^vc.ru

Spam Crawlers Filter 5: [2016-12-13]  Custom > Exclude > Language Settings

.{13,}|\.

Spam Crawlers Filter 6: [2017-06-17]  Custom > Exclude > Campaign Source

slifty.github.io|foxweber.com|stalawfirm|free-fbook-traffic|goodwriterssales|spinnerco.ca|justkillingti.me|suralink.com|worldtraveler.world|yellowstonevisitortours|hollywoodweeklymagazine|tourcroatia|losangeles-ads|oldfaithfultaxi|christopherlane

Spam Crawlers Filter 7: [2017-07-03]  Custom > Exclude > Campaign Source

anniemation.com|timdreby.com|pcimforum.com|yellowstonesafaritours|autoseo-traffic.com|blogarama.com|money-for-placing-articles

Editorial note: please do not take the presence of a domain in this article to mean the related businesses are all ‘spammers’. In several cases, unsuspecting businesses have had their domains referenced by spammers or have unintentionally left referrals and since corrected their practices.

spam-language-filter

spam-language-filter

spam-referral-filter

spam-referral-filter

 

Continue reading

Why You Should Not Use the Referral Exclusion List for Spam

While all of us fight to get Google Analytics referral spam under control, one piece of well-intentioned, but bad advice comes up over and over again: use the Referral Exclusion List. Some people (myself included) have stated that is a bad idea, but no one has taken the time to really explain why. Here is the Why

[if you want to know what you should do, read the Definitive Guide to Removing Referral Spam]

Continue reading

*Payment and Access to Your Google Account

 

If you want the personalized spam filter service for your web site, you need to:

  1. Pay here:   fastspring.com
  2. Give me access to your Google Analytics Account (as follows)

Granting Access to Your Google Analytics Account

Do NOT send me your account and password! Instead, follow these simple steps:

  • log into Google Analytics
  • select the Admin section
  • select the Google Analytics ACCOUNT to be filtered
  • click User Management in the left (ACCOUNT) column

user-management

  • Add permissions for: [email protected](note this is different from the image)
  • Select Edit in the list of access permissions
  • Check the box to Notify this user by email
  • Click Add

Providing access

If you have multiple websites in your Account, please send me an email identifying which ONE property/website you want me to process: [email protected]

 

What Will Happen Next

After payment and proper access is granted. I will:

  • analyze your site
  • add 2 new Views (unfiltered and test) if needed
  • install spam filters into the test view
  • send you an email explaining what I found, what filters I installed
  • send you a new Custom Segment that you can use immediately with your reports

After a week, I will confirm that the new filters are working properly in the test view, and if there are no issues, I will apply them to your main website view.

At any time, you are free to ask any questions you want about what has been done.

 

Removing Access

When your service period is ended, you should remove my access to your account.  (I will remove myself eventually if you don’t)

In the Google Analytics Admin panel, select the User Management section for your account, identify the entry for [email protected] and click the delete link.

removing-access

All Users (property-name) Segment

Filters will prevent new spam referrals from appearing in your account, but they do not work retroactively. As part of the service, I create a new advanced segment called All Users (website-name). It can be used for historical reporting, eliminating the spam traffic and giving you clean metrics.

Getting Your Spam Filter Segment

Segments are associated with login accounts. To get the new Segment, you will be provided with a link by email. Log into Google Analytics, then click the link provided. Select the option Any viewNote: your segment will appear with a different name than shown.

shared-segment

The click the Save button to save the segment in your account.

 

Using the New Spam Filter Segment

To use it, open your Google Analytics report and click on the All Users box at the top. In the list of Segment Names, you should see that All Users is selected (checked) and a new entry just below it All Users (<your-website-name>).

Uncheck All Users and check All Users (<your-website-name>). Your reports now reflect only real traffic to your website. Note: your segment will appear with a different name than shown.

spam-segment-selection

 

AdWords Users:  note that AdWords data is imported into Google Analytics without a hostname value, so the Valid Hostname filter will cause the segment to drop all AdWords cost and ad information if you apply it. View the AdWords reports with the All Sessions segment.

*Personalized Spam Filter Service

Proactive filtering, daily monitoring, personalized service
Only $75 for 12 months of protection!  Start Now!
“Thank you, Mike.  This was clearly worth the small investment.”

spam-summary-2016

Spam sources appearing in Google Analytics in 2016

If you have Google Analytics, you have referral spam (and language spam, landing page spam, and even keyword spam). I will make it go away! I install and maintain the filters you need to keep spam out of your reports. I will even send you a custom segment you can use to clean up your existing Google Analytics reports. Includes protection from language spam, event spam, landing page spam, and plain old referral spam.

spam-filtered-report

FAST: Results usually in the same day. I will analyze the traffic to your website and identify valid host names and spammy crawlers, then build the filters and advanced segment appropriate for your website.

SAFE: I will not put your website data at risk. I will create an Unfiltered view before I apply any filters. I will install the filters into a new Test view, so you can see exactly what the impact of the filters will be. After a week of testing, I will apply the filters to your main view (usually called All Web Site Data).

PERSONALIZED: This isn’t a cookie-cutter solution — I review your website configuration and make sure everything will work for you. Multiple domains, ecommerce, and/or call tracking events all have subtle requirements that need to be taken into account when filtering traffic.

EXPERIENCED: I coined the term “ghost visit” and wrote the Definitive Guide to Removing All Google Analytics Spam and have updated it regularly since January 2015. I have helped thousands of people eliminate the spam from their Google Analytics accounts, and I can help you!

TRUSTED: Michael-Sullivan-150x150My identity and contact information is no secret. Check me out on Google: “Mike Sullivan Analytics Edge”. Read my LinkedIn profile. Send me an email; ask me questions. This is my business. This is what I do.

You are in good hands, so jump right in.  Start Now!

Perfect! Thank you so much!

…thank you very much, this was the fastest service I have ever received online.

“…your segment works…which means I can report on accurate data for the first time…

Wow.  Clean data.  This feels good.

Real traffic from real websites — orange: with spam, blue: without spam.

spam-combo