Search engines like Google and Bing operate transparently and it is quite fascinating to know the basic way in which they crawl a page, index it and then rank it.
Search engines are the first tools we turn to, when we have questions. We depend so much on search engines from answering kids’ questions to doing comparison shopping before buying anything for the home or family.
What’s To Know About Web Crawlers?
Just as everyone driving a car does not need to know the underlying mechanism on which a car operates or the parts below the hood, it is not a must to know how search engines work. It is pretty cool to have an idea, as Google and Bing spend megabucks to keep their information up to date by crawling the web and its trillions of pages periodically.
This in turn helps them answer billions of questions of every possible type posed by several others in the world just like you. There is a lot going on to keep searchers like you happy and to get you coming back to the search engines for more.
What Is A Spider?
A spider (also going by the name of crawler or robot) is a software program that the search engines use to find out the changes on the web, primarily the content. New sites come up everyday, loss of interest causes domains not to be renewed leading to sites vanishing and old sites with outdated information just disappear into the vast cyberworld expanse.
The Crawling Spider
At the basic level, spiders crawl the web by visiting a website, go through different URLs by following the hyperlinks and find new content which can be just pages of content, pdf, word document or audio/video files.
With sites linking to other sites, this following of links eventually leads the multitude of spider bots, released by Google and Bing to crawl several servers, to find all possible new sites along with their content.
Fetching A Page
When a spider crawls a web page, it first fetches the page, then reads the content in words into a massive database called the search engine’s index. It then follows the links on the page or schedules these links to be visited later on another visit.
If for some reason the page that a spider visits is offline, then the page gets deleted from the index. This is the reason for having a reliable company hosting your website.
The First Thing A Spider Bot Looks For
The spider looks for a file called robots.txt when it visits a site. This file is a set of directives for the search engine bots, as to what parts of the site they are allowed to follow. In short, you can control what the spiders see on your site.
For a WordPress site, it can be instructions like:
User-agent: *
Disallow: /wp-admin
This tells all the search bots of all search engines (user-agent: *) not to crawl the wp-admin area. For the best part, all search engine spiders follow the rules laid out by the robots.txt file. There are lot of rogue bots that don’t as well.
Why Is Crawling So Important?
It is imperative that your site gets crawled and indexed first before there is any possibility of the page getting ranked on the search engines.
Ensure that you roll the red carpet to the crawler bots so they can crawl your site without any hindrance. Site navigation, site structure and internal linking are all vital to make this happen. Maintaining a high indexation of your website pages is essential.
How To Check Page Indexation?
To make sure your site is getting crawled, there are two quick easy ways.
1. If your site has been around for a while, then go to google.com and type in the search box the command site:yourdomain.com
For example, the number of pages indexed on the site disney.com is 60,800. This is an approximation and it lists the URLs on the Disney site that are in the Google index.
2. Go to www.google.com/webmasters and click Search Console. On the next page, click Start Now. If you are a verified owner of your site that has been around, you can login to see a wealth of information Google provides inside. If it is a new site, it will take time to populate.
All you need is a free Gmail account. Register for a free account and verify your website in one of the several ways Google provides. You are all set to view all the technical information like indexation rate, URLs in the index, Sitemaps and so many other things.
The same goes for Bing. Visit www.bing.com/webmasters to set up a Bing Webmasters Tools account where you can see similar metrics for Bing as what you saw in the Google search console.
What Is Indexing Really All About?
Indexing a page is the process of storing information about the content on that page. It is about the words and the position at which they occurred on that page.
Images, media files and code can be indexed and stored by different kinds of crawlers. But the search engine bots give priority to words, as they are used by searchers to find what they are looking for.
The search engines also store information about the hyperlinks visible on every page. The words in a hyperlink also known as anchor text serve an important purpose in giving the bots a cue about the contents of the page that a link is pointing to. Following external links on the site leads the search engines to new sites and pages of content.
There are other pieces of special content like meta description (which appear as the title of a site along with the meta description on the Search Engine Results Pages or SERPS).
The image tags have a property called ALT or Alternative text that enables visually impaired people using screen readers or users with Javascript turned off to understand what the image is about.
Search engines cannot understand images (though they are getting better at it with every passing day) and use the ALT tag to understand what the image is about.
A human user needs to hover her mouse over the image and the alt tag value (in Firefox, it is called title attribute inside the image tag) pops up more like a tool tip.
Controlling Indexing and Following Of Links
Another cool way to tell the search engines not to index your page (whatever may be your reason for doing so) without using robots.txt file is to use the Robots meta tag. These are placed between the sections on a web page.
Meta directives are used for 2 main purposes. One is to block the search engine bots from indexing a page using the meta robots = “noindex”. This will prevent the page from getting indexed.
Second use of this tag is to allow the link juice from the page, through its various links on that page, to either flow or not flow to other pages. Using the meta robots = “nofollow” will accomplish this task. If you want the links to pass on their link juice, then use meta robots = “dofollow”.
Finding Your Site On Google
If you have a new site, you can use the above 2 methods listed under Indexation to find out how many pages are being crawled and indexed by the search engines. It will take a while before the bots get into a routine of crawling and indexing your site pages.
NOTE: All the pages indexed will not be shown on Google. You have to login to Google Console to get the true picture. The same goes for Bing as well.
If you have a brand new site on a new domain, you can expect Google to cold shoulder you from 6 to 8 months before it starts ranking your site pages on the SERPs, provided your site’s content is worthy of getting ranked at the top.
If you have bought an old existing mature domain, it can be problematic depending on the previous chain of ownership of that domain and if any black hat techniques had been used on that site in the past. You simply inherit those problems.
How The Crawlers Find Your Site
The most common way is for the spiders to follow the hyperlinks on other sites and land on your site.
Back in the day, all the major search engines had a provision to Submit Your URL. It was a form that had a field for your site URL and then a user hit the Submit button. The search engine crawlers would add this URL to their list of links to crawl the next time around.
Login To Search Console
Today, you can login to the Search Console and submit a sitemap generated by say Yoast or any other good SEO WordPress plugin. There are XML generator tools available for non WordPress sites. The free version has a hard limit on number of URLs that can be submitted.
Post On Social Media
The best way of getting your site found by search bots is to post your article, image or video on Facebook, Youtube, Instagram or Twitter. The crawlers frequent these social media giants all the time as they find a continuous source of new content.
The Ranking Of Web Pages
Ranking is a complex process by which indexed pages stored in the Google index are subjected to at least 400 ranking factors and ranked in order of highest to lowest relevance for a user’s search query on the Google SERPs. All this is done based on the ranking algorithm.
Google has updated its ranking algorithm huge number of times over the years. The basis for Google’s famous ranking algorithm was (and still is) Page Rank. This is based on the concept that a maximum number of votes to a site by way of external links or backlinks pointing to that site shows the popularity of the site and pushes it to the top of the rankings.
These backlinks are basically off page factors. There are several on page factors that a site owner can control via the title tag, h1 tag, the content with keywords and their synonyms on the page that are vital to the subject being discussed. Backlinks are harder to manipulate as it is mostly other sites that choose to link to a site, over whom the site owner has no control.
This transparency in Google’s algorithm led both SEO professionals and spammers and anyone in-between to manipulate the backlinks and the underlying anchor text to game Google. There came a time when the top 10 results on Google were filled with rotten spam leaving searchers frustrated.
Sometime in 2011, Google cracked the whip with the Panda update wherein thin content sites were penalized. Prolific spammy content was the order of the day rather than limited top quality content (takes time to create), to rank high on Google. This earth shaking update preserved Google’s search algorithm and penalized huge number of sites indulging in manipulative practices.
In 2012, Google launched the Penguin update to crack on the huge number of poor quality backlinks built with software, link farms that were owned by a site and pointing tons of backlinks to the parent site, blog networks and any kind of unnatural linking. This had a huge impact and lot of spam sites got booted out of the Google index.
In the process, many good sites also took a massive hit. This was because they had linked to bad neighborhoods, in many cases unintentionally. All said and done, it was major chaos on the search landscape. Ever since, Google has been ever vigilant and updating its algorithm to reward the top quality sites and keep spammers at bay.
Goal Of Search Engines
The ultimate goal of search engines is to provide relevant results to users search queries. Based on the interaction of the users, Google can understand very clearly if the #1 site really deserves its top ranking.
If a user clicks this listing, bounces off the site and comes back to Google and either refines the search phrase or clicks on search result #2, it is a clear indication to Google that the answer provided by the first listing was not really relevant to the user’s search query.
With Google Analytics, the premium yet free web analytics software owned by Google, straddling a huge majority of the sites on the web, Google gets additional user related information like number of pageviews, average time spent on the site, bounce rate, new vs repeat visitors etc. All these factor into Google’s rankings though they are not as transparent as before about these factors, for obvious reason of getting their algorithm gamed/spammed.
PageRank still plays a vital role in the final ranking as the backlink count is a major vote for a site, especially links from sites with huge domain authority and trust. It is not the numbers that matter, it is the quality of these backlinks.
Google does not discuss much about PageRank at all. In its heyday, Google even offered a Page Rank green colored toolbar as part of the Chrome browser addon that users could install. Going to a site showed the PageRank of the site, ranging from 0 to 9. All this led to the ultimate spamming of Google using thin content and backlinks as fodder, the likes of which will never occur again.
Knowing User Intent Vital To Rank At Top Of Google
You have to understand your user, the one who visits your site, spends time reading your content and ultimately buys your product or service. Once you understand your small business audience avatar, you can then do keyword research to align with the keyword phrases that users in your niche are searching for. By producing content that satisfies these requirements, you are on your way to winning on the search engines.
At Traffic Alignment, we help small business clients in their content marketing services by promoting good content on social media using a nominal ad spend and helping such sites gain a foothold in the Google SERPs. Call us today for an appointment.
Related Posts
Leave a Reply