Spiders on the Web: A brief guide on the inner workings of search engines

We often take the ability to use a search engine for granted. Google has made search so easy that we do not have to worry about finding information. Googling something we do not know has become second nature to us. We trust Google’s results to be relevant and accurate. In order to best optimize your website for Google, it is helpful to know how the search engine works.


This article contains a detailed account about the rise of Google, if dense material isn’t your thing continue reading this summary. Early search engines, like AltaVista, would compare search terms to their database and whichever page had the most similar terms would be considered the most relevant. This was problematic because the results weren’t necessarily the most relevant. For example, if you wanted to search Columbia Sportswear, Dick’s Sporting Goods might be the first result, rather than the corporate web page, because they list a lot of Columbia products. The amount of times a keyword was mentioned in the page outweighed other information in determining relevance. Search results weren’t as helpful because users had to filter through the results themselves. Yahoo was different. They used human judgement to aid their search engine’s results. Yahoo hired people to read through webpages to pull keywords and write summaries. In 1997, Google changed the game by offering better results, without clutter. Instead of looking at the text of a website, Google observed the patterns of hyperlinks, specifically, the number and type of incoming and outgoing hyperlinks. 

Search Now

When you type your search query into Google, the search engine sends out little programs called crawlers or spiders. These crawlers jump from page to page by following links. As the spiders travel, they send a copy of each page to the search engine. Google then creates an index of the words on that page. Algorithms are complex mathematical equations which are designed to find hints to understand what you are searching for. There are plenty of great videos on Youtube, like this one, that will help you visualize the search process. Going back to our example, when you type in “Columbia Sportswear” on Google, the first search result will be the manufacturer’s website, not a retailer’s (i.e. Dick’s Sporting Goods). This is because of Google’s algorithm, which gives more weight to a company’s homepage.

Why it all matters

According to this article, less than 6% of users clicked on a link on the second page of Google results.  As a small business owner, it is crucial for you to maximize the chances of your website being on that first page. The process of making your website search-engine-friendly is called Search Engine Optimization, or as it’s commonly referred to, SEO. As the name suggests, Search Engine Optimization is the process of optimizing your website’s content for Google’s crawlers. How often websites are crawled depends on how often the content on the website changes. Therefore, frequent updates to your website are essential to driving traffic.


3 thoughts on “Spiders on the Web: A brief guide on the inner workings of search engines”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: