Step 2 - Robots & Ranking Issues

You’ve looked at key indicators and variables within your SEO and Paid Search campaigns in the last step.   Now, we'll just focus on issues and potential problems relating to your organic search rankings. 

It's possible, though unlikely, that Google or another major search engine has banned your site for violating their guidelines.  A more common problem is a noticeable drop in rankings for some of your target keywords.

      Quick Check to Confirm Site Indexing

1.    Type your entire URL into Google's query box:   www.yourURLhere.com

If Google doesn't return any pages from your site, it's likely that your entire site has been dropped.

2.    Grab a unique section of text from your site (like a sentence containing one of your brand's products or services), put it in quotes, and search for it

If Google displays the page from your site that contains this text, then your site is still included in Google's index.

Insight into Why Your Rankings Have Dropped

• Web hosting issue? It's possible that search robots crawled your site when your server was down.   In this case, there was nothing there to spider momentarily...hang tight and wait until the bots visit again and your positioning should be fine.  You could also request server stats from your hosting provider and see what's up. 

• Something may have been radically altered your site.  What if you wanted to update the HTML page title on only one of your landing pages but you accidentally replaced all of your page titles with this same phrase?  

• Your competition has become more sophisticated and is working diligently on their SEO campaigns.  This could drop your rankings but definitely not considerably.  If your positioning drops by more than 50 spots it's probably not due to competitors taking it up a notch all at the same time.  

A huge decrease in rankings could be attributed to some form of spamming, whether it be purposive or inadvertent.

• Your site may be found culpable of spamming, even if you had nothing to do with it or if it happened inadvertently.  Typically, only ridiculously obvious black hat tactics lead to precipitous drops in rankings or bans.  

• Your external links may be pointing to link farms or spammy sites.   You may have linked to a site last year that seemed fine but it's been commandeered by a spammer since then.   Check who you're linking to on MSN and make sure you're uplinking to reputable sites.  

   Here's a quick check to see who Starbucks is linking to.

• Google did it!   It's possible that Google changed their acclaimed algorithm, a minor tweak or major update.  Check out Matt Cutts' blog (a well-known Google Engineer) to keep apprised of the latest changes to Google's algorithm.  

• More...

  Talking to Search Bots

If none of the aforementioned possibilities apply, you might simply be placing too many restrictions on the spiders that scan your site.  You already know that search bots like to be fed clean, structurally sound text links and we’ve already discussed how they find your site.  

Now is an opportune time to show you how to communicate with bots like Mr. Spider-man...sound crazy yet?  Let's see how. 

While humans have manifold ways of communicating with each other, we only have one way to directly “talk to” a robot and this method is limited at that - it's the robots.txt file.

The simple robots.txt file adheres to something called the Robots Exclusion Protocol.   Here's how this file functions and what it can and cannot do.  More...

When the robots.txt file comes in handy

  CREATING a robots.txt file for your site

Let’s talk a bit more about how it works and we’ll give you an example.  Ok, so a robot wants to visit your site URL, which is http://www.yoursite.com/. 

As alluded to previously, the robot first checks for the http://www.yoursite.com/robots.txt before proceeding.

robots.txt must be all lower-case

Now, if you want to tell all search engine robots to exclude indexing files from two of your directories containing some kind of image content, then your file will look like this below:

• The asterisk as shown here, "User-agent: *" indicates that this section applies to ALL robots.

• What if you only want to tell Google not to index this very same image content within your site?  Instead of the asterisk (*) then you type in googlebot, like so...

• The "Disallow: /" tells the robot that it should not visit nor index the pages within the directory you specify. 

• If you remove the forward slash (leave the area blank after Disallow: ), then you’re permitting robots full access.   Note that this is the same as not using a robots.txt at all.

  Where do I put my robots.txt file?

Place it in the top-level directory (root directory) of your site.  Typically, this is where you put your site's main welcome page.  Where exactly that is, and how to put the file there, depends on the software on your server.   More...
 

  Another option:  the robots meta tag

The robots meta tag is used for similar reasons as with the robots.txt file.   By using it, you can control whether or not a snippet is displayed in search engine results and if a page is cached.   It’s a component of the page that tells search engine robots certain things.

The robots meta tag is written within the <head> of an HTML file and appears as such:

You can send the robots different commands by inserting them inside of the quotation marks where we wrote “command” – check out these examples:  More...

Make certain that you go ahead and triple check your code when using the robots meta tag.

Improperly formatted tags or the wrong usage could keep search engines from indexing your key landing pages or entire website altogether.

If you use robots.txt file to block search engine spiders, you do not need to use the robots meta tag.   Also, you should never put more than one robot meta tag on a page or more than one directive in the content area of the tag.

If you want the search engines to index your site, you really don't need to use the tag either because the spiders will index it by default.   More...

 

You've been booted from Google's index...now what?

We'll defer to Matt Cutts for Google's insider tips on how to get reincluded.   If you haven't been banned, use this collection of the right diagnostic questions to ask if your rankings take a hit.  More...

Remember that neither the robots.txt file nor the robot meta tag guarantees indexing.  Just stick to the simple paradigm we shared with you at the beginning of SEO Central and you'll be well on your way to a profitable SEO campaign.

 A snapshot of how your

SEO campaign is doing.