This is a quick experiment to add to my list of SERP snippet experiments. I got the idea for this after reading a blog post by Everfluxx, titled: Why I deleted the AddToAny and TweetMeme plugins. In that post, Everfluxx links to a file on my site that is blocked by my robots.txt file.
The file contains nothing but a bunch of social media buttons, which I lazy-load into my blog posts. I do this because it cuts down page load times and prevents the theoretical evaporation of PageRank caused by nofollow attributes.
Here’s a glimpse at the post that links to my iframe file (the one with the anchor text that says “custom solution”):
Today as I was looking at the results of a site: command (keep the bookmarklet; it’s a free gift!), I noticed this amongst the results:
If it surprises you to realize that blocking URLs in your robots.txt file does not prevent them from appearing in Google’s search results, then you need to study the difference between crawling and indexing. The only thing I found interesting about this particular SERP listing is the title. As far as I can remember, Google has always used the URL as the title for blocked pages, but in this case, it’s quite obvious that Google is using Everfluxx’s anchor text + my home page title. Maybe Google has always had the ability to do this, but I’ve just never seen it?
In any case, as soon as I realized where this title was coming from, it made me realize: I can spam other websites’ search results! The process is theoretically pretty simple:
- Pick a website.
- Review their robots.txt file (another free gift, bitches!).
- Link to a URL they’ve blocked, using a spammy phrase as anchor text!
Alright now…it’s time to find some poor bastard to try this out on. Hmm…
Jackpot!!! I just checked out Matt Cutts’ robots.txt file, which contains the following:
User-agent: * Disallow: /files/
Naturally, I immediately tried accessing that URL, but all I got was this:
Sorry.
It actually doesn’t matter what the page request returns, because well-behaved bots like googlebot will never request a URL that’s blocked by robots.txt. So for the purposes of this experiment, I can use any URL that appears to be in the /files directory, even if it returns a 404 status. Stop and think about this for a second. This means I can theoretically convince Google to add non-existent URLs to Matt’s SERPs…AND…I can choose the title of the snippet?!
Before I kick off this experiment, there’s one foot deeper I’ll dig my own grave more thing I’ll mention about mattcutts.com: it appears as though his DNS record is configured to point all subdomains to his canonical name/IP address, so http://[anything].mattcutts.com will return the same thing as http://www.mattcutts.com. So as an added bonus, I’m going to create my own subdomain on Matt’s site. Thanks, Matt!
It’s now time for me to go update my global navigation links. One last thing I’ll mention…did you know that Matt Cutts personally endorses my SEO services? It’s true–just the other day I heard him say SEOmofo is the World’s Greatest SEO.
You’re probably wondering if this experiment has succeeded yet. Last I checked…no, it hasn’t. But feel free to check for yourself: non-www subdomains on mattcutts.com
2/16/2011 UPDATE: It’s been over 10 weeks since this post was first published, and none of the major search engines have indexed my fake URL. So I’m making some changes to the experiment. The first thing I’m going to try changing is the fake URL: from http://seomofo.loves.mattcutts.com/files/spam-me-tender.html to http://seomofo.mattcutts.com/files/spam-me-tender.html. This is to rule out the possibility that it’s the double subdomain that’s preventing the URL from being indexed. I don’t think this is the case, since I’ve seen www.ww.mattcutts.com/blog/disclaimer indexed in both Yahoo and Google, but I’m knocking off a subdomain…just to be sure.
3/6/2011 UPDATE: I gave the single subdomain URL about 2.5 weeks, and still nothing, so I’m changing this again. This time, no subdomains. I’m changing the URL: from http://seomofo.mattcutts.com/files/spam-me-tender.html to
http://mattcutts.com/files/seomofo-is-worlds-greatest-seo.html. I’m also going to put some paragraph text around the global link.
10/13/2011 UPDATE: It has been almost a year since I published this post, and honestly I gave up on these experiments a long time ago. I removed all the links to seomofo.mattcutts.com and didn’t give it a second thought…until now. A guy named Will left a comment below that didn’t seem to acknowledge how miserably this post failed, so I re-checked the search results and was happy to see that http://seomofo.mattcutts.com/robots.txt has been indexed!
So thank you, Will, for bringing this to my attention. If I ever need to buy laboratory supplies to supply my lab with science lab equipment, I’ll be sure to check out Alkali Scientific’s offerings. ;)
Oh yeah…and thanks for the subdomain, Matt! :D
12/19/2011 UPDATE: HOLY SHIT! WTF?!!!
I just checked the results again and found that every variation of this experiment I ever tried has been indexed! I don’t know why they’re all showing up (only one variation was live at any given time) or why it took a YEAR for them to show up, but in any case…have a look at these epic results.
{ comment Leave a comment }
Missed you maaaan. Good ish as usual, very funny.
Your free gifts always come at a price….to my competitors
LOL, sorry bro, didn’t mean to pollute your site: query results. :D
Actually I had thought of the possible consequences of that link; however, a respectable professional once told me that linking out to other SEOs with nofollow is “lame” and equals to “peeing on their shoes” [sic], so I chose to give you uncondomed link love instead. Of course, if you would like to have it nofollowed instead, just email me.
Great post, however, and great idea! I’m sure MC will appreciate your creativity. (BTW, I was a bit shocked by the wildcard subdomain setting of his DNS: I wonder if it’s intentional?)
And yes, it’s true: AFAIK, Google can stitch together anchor text and home page title tag contents to construct a more usable, meaningful SERP title when the anchor text alone is too short –which is not too different from what it does for pages with no title tag.
No worries, brother. If you hadn’t linked to it…I wouldn’t have been inspired to write this amazing blog post. ;)
Full marks for writing a post on SEO techniques which caused me to laugh out loud.
I don’t think that’s happened before.
Sir I am reading you blog from last few months and specially your last post and all google threads on topic – Javascript to hide links .
This post is also amazing with lots of info but I think we must not do bad for some poor bastard without any reason :P .
Thanks
Lol. I bet people are going to start up businesses around this type of thing :)
This is ridiculously awesome – did it get indexed that way? Just did a quick search but I didn’t see it.
Immense post. One thing though. Maybe you should add a link to a Matt cutts “page” on the www version since google knows that subdomain does actually exist. It might be pushing it to link to a sub that doesn’t exist too, and it’d make your test closer to the control version on the Everflux site.
I know the sub does actually exist because of the wildcard but still. Don’t know if I’m talking sense there.
Makes perfect sense. This experiment is already 2+ weeks old and Google still hasn’t indexed my test URL, so I’m beginning to wonder.
However, if you click the site: link at the end of the post, you’ll notice that there’s a URL showing up that’s on the “nonexistent” subdomain “ww.” So I’m thinking it might be the double subdomain (“seomofo.loves.mattcutts.com”) that Google doesn’t like.
Good point. Maybe do all three versions, see what wins. Just trying to think of a site I wanna make public that I could do this with. Hmmmmm.
Sorry to spam your comments further, but out of interest how long after the Exverflux post (Oct 2nd) did you see this in the SERPs?
When doing anchor-text-passing experiments before I’ve noticed single blog pages take a while to pass the anchor (months), while homepage or blog root pages pass it a lot quicker.
If I remember correctly, I noticed the SERP right before I wrote this post, so…December 1st?
Regarding single-page-hosted anchor experiments vs. home-page-hosted experiments: I placed the link in my sidebar, so it’s on every page. Perhaps a more-prominent placement (e.g. in post content) would improve my chances of getting the URL indexed. Unfortunately, I might be too lazy to implement that. :)
Holy crap, I have to try this. NICE!!! Hey, and I’m with Bill, I think I LOLed at SEO talk. Yup, I’m officially a nerd.
Sounds very tricky :)
Love it, you are playing with fire. Once Matt sees what you did on his domain you are gone :)
Why those Google crawl the pages which are mentioned as Disallow in robots.txt ? Can we use meta robots tag.
Genius. After reading your post you inspired me to take a better look at my robot.txt file. Is it bad that I have a bunch of files blocked?
Oh…Great ending by the way…haha The link was perfect
Hey actually it’s working !!!
Did you check recently, here is what I got today :
http://www.flickr.com/photos/49522832@N08/6486375343/
and this one :
http://mattcutts.com/files/seomofo-is-worlds-greatest-seo.html
has been indexed !
Stupid bot ;-)
Cheers,
u are weird in a good way.
how do you come up with these wacky ideas ?