Googlebot Interpreter Testing

Print Friendly, PDF & Email

This page is a hub for my simple experiments with G Bot. To determine what it can or can’t do. The nature of these experiments is such that they are running all the time. Maybe it’s a good idea to set up some simple alarms if G suddenly changes behavior, but it does it rarely enough to do the checks manually like once a year or so. Having these experiments monitoring Google’s behavior continuously is essential. It’s quite useful especially if you’re involved in a lot of technical SEO and work with all kinds of web-technologies and developers, who would often demand evidence to SEO claims. Well, these are my evidence 🙂 he-he

This is not the kind of testing SEO people got used to. It’s not something that becomes obsolete in a year. This is a lot more technical and might be more helpful for web-devs than for regular SEOs. But I’ve always thought a good SEO has to be a good full-stack. Anyways, I’m conducting them mostly for myself, but feel free to get involved.

And yeah, I’m deeply sorry for how the code looks in the source code. It’s just WP doesn’t like new lines #13#10 🙂 And I’m not as good at coding as I should be. #Node.js community in FreeNode seduces me with Functional Programming. Successfully so.

(by the way, Google === G)

Conducted Experiments

  1. It’s a legend page. Which also tests how G treats pages with JS-injected titles and h1s. (G indexes them)
  2. How Google treats JS-injected meta noindex (G sees and respects it)
  3. Will Google index display:none? (it will)
  4. Will Google index text within nonsensical position:absolutes? (it will)
  5. Does G execute/index Base64? Any kind of it? Huge and tedious base64? base64 in JS? (Google works with Base64)
  6. Google works with web-storage&cookies? (Google uses both)
  7. Does G interpret text on images if it’s really easy to recognize? (G doesn’t recognize the text)
  8. How does Google treat text and words with foreign symbols in it? Symbols that don’t belong to the text’s language alphabet?
  9. Google and Noscript: Does google index what’s in <noscript>?

Http headers experiments

This section includes simple experiments that show how Google treats certain http codes. I just put a simple piece of php in the footer.php to set the headers. Here’s the code:

if (is_numeric(substr($_SERVER['REQUEST_URI'],-4,3)) &amp;amp;&amp;amp; strpos($_SERVER['REQUEST_URI'], '/gtestg-') !== false)
{
var_dump(http_response_code(substr($_SERVER['REQUEST_URI'],-4,3)));
}

I had to change the logic of this implementation since http_response_code doesn’t work as I expected for the codes, definitions of which are not known. Hence I reimplemented it via

@header( $status_header, true, $code );

Also my setup gets destroyed every time I update the engine. Will have to implement it via child themes. Updated again and it seems to work. Neat. Sometimes WP doesn’t update the header.php.

  1. 201
  2. 202
  3. 204
  4. 283: How will Google treat a custom, non-existing http status code with a random description?
  5. To Be Continued…

In-progress Experiments

What we know a-priori

(As if we needed to state the obvious)

  • G executes and indexes pure JS
  • G executes and indexes JQuery
  • G executes and indexes Ajax, hence the whole rest stack, hence all kinds of POSTs
  • G executes and indexes Angular
  • G executes and indexes Polymer
  • G executes and indexes Any popular not-obscure JS framework 🙂 (react, meteor, webgl, whatever)
  • G executes and indexes Web Sockets implemented in most common ways
  • G works with cookies (eats AND shares) like anyone (but we’ll test it to make sure)

Because we know these are true, we’re not gonna check them.

Pending questions (for future research)

  • What kinds of JS events does G trigger? Not onclick for sure, but how about hovers, scrolls, etc?
  • Does G execute/index CSS? Any kind of it? display:none? things hidden in a class? base64 in css?
  • What is the limit for Google in terms of JS obscurity/obfuscation?

2
Leave a Reply

avatar
2 Comment threads
0 Thread replies
0 Followers
 
Most reacted comment
Hottest comment thread
2 Comment authors
JamesJames Recent comment authors
  Subscribe  
newest oldest most voted
Notify of
James
Guest
James

Aw, this was a very nice post. In thought I want to put in writing like this moreover taking time and actual effort to make an excellent article but what can I say I procrastinate alot and not at all seem to get something done.

James
Guest
James

Awesome article post.Thanks Again. Much obliged.