Skip to main content
Search Engine Optimisation (SEO)

Google spiders, ‘crawl budget’ & why it matters for your SEO

By October 31, 2019March 4th, 2021No Comments

SEO Hacks for SMEs logo

Welcome to our video series on useful SEO tips that any small business can implement.

To receive these small business SEO tips in your inbox every two weeks, subscribe below.

SEO Hacks for SMEs logoWelcome to our video series on useful SEO tips that any small business can implement.

To receive these small business SEO tips in your inbox every two weeks, subscribe below.

Video Transcription

Today I want to talk about spiders.  No, not that kind – but Google spiders. If you want good SEO, your website needs to encourage these ‘bots’ to crawl like crazy. Problem is, most websites effectively treat the Google spiders as a pest!

I’m Ben Carew, the co-host ‘SEO Hacks for SME’s’. In this video I’ll reveal how you can optimise a website to welcome the Google spiders, and achieve better SEO results for your business.

130 trillion.

That’s how many web pages there are on Google. And every single one was indexed at some point by the ‘Googlebot’ –  or ‘spiders’ as they’re known. These tiny bots are sent by the search engine to crawl websites in order to collect the data needed to index, and rank web pages. 

Which brings us to the first point:

Key Insight #1: There is a limited ‘crawl budget’ for each site.

Contrary to popular belief, Google doesn’t send it’s spiders to your site crawl your entire website content from start to finish, and then update Google’s index accordingly. There are simply too many pages on the internet. 

And like the Universe (and climate change), the web is growing exponentially every day.

So the spiders can only spend so much time on any given website. Many web pages are never indexed at all. 

Key insight #2: A rational website structure is super important.

To give the spiders the best opportunity to understand your website in the limited time available, you need a page taxonomy and URL structure that is logical, and ‘frictionless’.

What does this mean for you?

Here are 3 simple steps you can take to optimise your website for Google spiders.

  1. Review your website structure.
    Look at it with fresh eyes, and think like a customer. How can you improve the URL structure navigation to make it more rational for the user?
  2. Organise pages around topics.
    Many Australian SEO companies  (including Webbuzz) recommend organising your site into ‘topics’ under which other pages in the same topic sit. In Google’s eyes these sub pages are effectively your ‘supporting documents’ – they tell Google that you have expertise in that particular topic.
  3. Use internal linking to help the spiders crawl.
    Think of internal website links as a motorway for the spiders. If your internal linking structure complements your ‘topic silos’, Google’s spiders can move faster – and further – into your website content.This is best for your SEO. It will help your site get fully indexed, and if Google has a detailed understanding of the value you offer to Google users, it will be more likely to serve your pages high in the SERPs for relevant search queries.

OK so hopefully that’s helped you understand Google spiders, crawl budget and why page structure is so important for your SEO. 

If you have any questions, leave a comment below and I’ll answer it!

And if you want to get SEO hacks delivered to your inbox every two weeks, don’t forget to subscribe.

Ben Carew

Ben Carew

Ben is the Director of SEO Services & Analytics at Webbuzz. He started in graphic design and web development in 1999 and has ridden many of technology's waves since then. He co-founded social media site Housenet in 2013 and began moving into digital marketing & SEO from his experiences growing this tech start up. Ben co-founded agency Webbuzz with former band mate Darren in 2014.

Leave a Reply

Close Menu
Antoinette Tyrrell

“Their work is consistently excellent, and has already moved the dial for us in a big way.”

Antoinette Tyrrell
CCO, Retirement Essentials

Luke Andersen

“WebBuzz have been wonderful … they’ve transformed our online presence over the last 12 months and they’ve been great people to work with.”

Luke Andersen
General Manager, Property Mastermind

Professor Stan Sidhu

“I’m happy to recommend them. They are reliable and deliver on their promises – my website is much more visible on Google.”

Prof. Stan Sidhu
Endocrine Surgeon