Video Transcription
Today I want to talk about spiders. No, not that kind – but Google spiders. If you want good SEO, your website needs to encourage these ‘bots’ to crawl like crazy. Problem is, most websites effectively treat the Google spiders as a pest!
I’m Ben Carew, the co-host ‘SEO Hacks for SME’s’. In this video I’ll reveal how you can optimise a website to welcome the Google spiders, and achieve better SEO results for your business.
130 trillion.
That’s how many web pages there are on Google. And every single one was indexed at some point by the ‘Googlebot’ – or ‘spiders’ as they’re known. These tiny bots are sent by the search engine to crawl websites in order to collect the data needed to index, and rank web pages.
Which brings us to the first point:
Key Insight #1: There is a limited ‘crawl budget’ for each site.
Contrary to popular belief, Google doesn’t send it’s spiders to your site crawl your entire website content from start to finish, and then update Google’s index accordingly. There are simply too many pages on the internet.
And like the Universe (and climate change), the web is growing exponentially every day.
So the spiders can only spend so much time on any given website. Many web pages are never indexed at all.
Key insight #2: A rational website structure is super important.
To give the spiders the best opportunity to understand your website in the limited time available, you need a page taxonomy and URL structure that is logical, and ‘frictionless’.
What does this mean for you?
Here are 3 simple steps you can take to optimise your website for Google spiders.
- Review your website structure.
Look at it with fresh eyes, and think like a customer. How can you improve the URL structure navigation to make it more rational for the user? - Organise pages around topics.
Many Australian SEO companies (including Webbuzz) recommend organising your site into ‘topics’ under which other pages in the same topic sit. In Google’s eyes these sub pages are effectively your ‘supporting documents’ – they tell Google that you have expertise in that particular topic. - Use internal linking to help the spiders crawl.
Think of internal website links as a motorway for the spiders. If your internal linking structure complements your ‘topic silos’, Google’s spiders can move faster – and further – into your website content.This is best for your SEO. It will help your site get fully indexed, and if Google has a detailed understanding of the value you offer to Google users, it will be more likely to serve your pages high in the SERPs for relevant search queries.
OK so hopefully that’s helped you understand Google spiders, crawl budget and why page structure is so important for your SEO.
If you have any questions, leave a comment below and I’ll answer it!
And if you want to get SEO hacks delivered to your inbox every two weeks, don’t forget to subscribe.
