Comparison Between URLs Requested by Anything and by Busiest Googlebot

« Back to: Analysis of Searchbots Behaviour During 27/Apr/11 to 19/May/11

Next Steps:

If you rollover the two horizontal lines next to "learn-us", you can see that Googlebot requested nine "/learn-us/" URLs during this time period, whereas there were in fact one hundred and seventy eight "/learn-us/" URLs requested in total.

In essence, Googlebot only requested 5% of all "/learn-us/" URLs.

To try to identify whether this is indeed a real problem, to see if Googlebot is spending its time inefficiently and thereby not indexing all content, a possible next step is to compare "/learn-us/" URLs that people request via natural search with "/learn-us/" URLs requested by Googlebot.

For an enterprise-scale ecommerce site this kind of analysis can be revealing: there might be URLs that people simply never request or that receive very little traffic but that lead Googlebot down rabbit holes; it might be that there are parameters in URLs that confuse Google.

The end goal though is simple: to identify ways in which to increase Googlebot's efficiency, and thereby increase the volume of content that Googlebot indexes. The more content indexed, the greater the visibility, the greater the chance of traffic.