Crawler, Algorithm & Co. – What’s behind Google Search

18 min

The basics at a glance
What are the key factors that determine which results you get?
What are Google algorithm updates and how often do they occur?
Google algorithm and search engine optimization
Short excursus: Why are there different search results on desktop and mobile?
Conclusion: This is why the Google algorithm makes us happy

The basics at a glance

In order to understand the principles of Google’s universe, there are some basic terms you should know.

Crawler, spider, robot or whatever you want to call it

A (web) crawler is more or less the synonym for spider, robot, bot, etc. It analyzes and indexes data from the internet. That means it looks at your website, reads the content there, and then stores information about it. It also examines pages that have already been indexed for updates and new content. Through this process, information is displayed to users in a more targeted manner, so that they only receive relevant content in response to search queries. The crawler is therefore the basis of every search engine and indispensable. For this reason, a bot, spider, etc. becomes important whenever you want to optimize your website for search engines.

However, not always only one bot is used to search the internet. Google uses several, for example, one only for images. The bots therefore have a clear thematic assignment, according to which they are subdivided.

This is how the indexing process works

In principle, you can imagine the crawler like a spider – that’s why Spider is one of its countless names. Similar to a spider, a bot “crawls” along the individual threads of the web. Of course, threads are only a metaphor here. In general, however, the internet works similarly: internal and external links connect individual websites like a large web.

The spider thus follows the individual links on the web pages and thus always finds further websites and their sub-pages. In addition, sitemaps can be submitted by webmasters to the Google Search Console, which are also closely scrutinized by the crawler. They then help the crawler to recognize what should end up in the index. Through these measures, more and more content is indexed and combined into a kind of continuously growing content library. All this content is then made findable for searchers on the internet. The search engine algorithm then searches this index when you enter a search term in order to display the appropriate answer.

This is how you show the crawler what should be indexed and what not

You can help the crawler to find websites. For this, internal links on pages and from the website navigation to various subpages are particularly relevant. This way you manage to make these pages visible to the crawler, as the bot can easily follow the references. Also, submitting a sitemap is useful, as it allows you to better submit updates to your content or new content to Google. Regular publications on your website are an important signal to Google and convey that your page should be crawled in continuous intervals.

Of course, you can also tell the Googlebot which content it does not need to look at or index. To exclude pages from the index, you can mark hyperlinks to the corresponding content with the noindex attribute. If you want to refine the whole thing, you can work with a robots.txt file. This is stored in the root directory of a domain and defines which URLs may be visited by the crawler. Here, you can specifically exclude the crawler from certain page areas via a disallow statement.

Limits of crawlers

Of course, there are also moments when web crawlers reach their limits. This is the case, for example, in the deep or dark web. Content here is placed in such a way that it cannot be found easily and should remain invisible to the average Internet user.

Algorithm, PageRank, ranking systems – What is it?

After the crawler has searched and found various pages of the internet, the algorithms come into play. These are used for the weighting and evaluation of the individual websites. They thus do the pre-sorting, so to speak, so that in the end you also receive the appropriate and relevant answer to your search query. The various ranking systems of a search engine scan several billion websites that are already in the index thanks to the crawler. So you get the answer to your question in seconds.

Searching the index for the keyword "search engine optimization" takes Google a little more than half a second and delivers 190 million results.
Searching the index for the keyword “search engine optimization” takes Google a little more than half a second and returns 190 million results

History of the algorithm

The birth of the algorithm was at the end of the 90s, under the name PageRank algorithm. The Google founders Larry Page and Sergey Brin invented it. It was supposed to be the basis for the success of the search engine. The aim was to be able to search the internet for websites in such a way that very specific pages were displayed in response to search queries. At the beginning, however, the weighting was only based on the linking structure – that is, the higher the number of external links, the more important the page and the higher the rating. Based on this weighting, the ranking in the SERPs was determined.

This is how the algorithm has developed until today

In the meantime, there are of course numerous factors that have an influence on the ranking of a page. Links are only one of many. The problem is: for a search query, there are often several million possible pages on the web that provide the corresponding answer. The algorithm must therefore evaluate each individual page in order to select the one that is exactly right for the user. This is an extremely complex process. There are several hundreds of these so-called ranking factors that are used to analyze a website. Most of them are not even officially known. However, the individual factors can be roughly divided into different evaluation areas, in which, in turn, different criteria are considered.

Which factors are weighted by the Google algorithm?

There are very different areas that the algorithm analyzes in order to offer you the best possible result. These include, among others:

  • Content – Here it is clear: quality > quantity. High-quality content convinces and binds your website visitors.
  • Links – External and internal links from relevant and appropriate sites give your users added value. Links from relevant sources are therefore important.
  • User Experience – If your site is not easy and intuitive to use, the stay on your site will remain relatively short. Therefore, make sure that your visitors can easily interact with the site.
  • Social Signals – The signals that the search engine receives from social media are becoming increasingly important. Again, the focus is on high-quality content that users are happy to like and share. Such interactions send positive signals to the search engine and improve rankings.
  • Brand – A firmly established and positioned brand helps you to achieve better rankings. Thus, work on your brand image and brand awareness.
  • Technology – The technical foundation of a website should always be impeccable. However, if there are errors or problems somewhere, it is important to fix them, otherwise it can have a negative impact on your positioning in the SERPs.

These criteria are part of the process that a search algorithm goes through to evaluate the relevance and quality of different content. Of course, for each individual category, there are countless other subcategories that are also analyzed in detail. In order to be able to adjust your site accordingly, Google has publicly viewable guidelines that you can use to guide you.

What are the key factors that determine which results you get?

According to Google, there are several factors that influence which results are displayed to you in the SERPs. Particularly important are the following five top categories, which build on each other step by step:

Key factors that determine what results you get
Key factors that decide which results you get

Word analysis

Semantic relationships have become particularly important when it comes to search queries. The search engine algorithm analyzes the context and meaning of the words entered in order to display the best possible results. For this purpose, there are various language models to help decipher the contexts. This word analysis is then used to find the perfect and most relevant hit in the index.

Google uses language models to identify the semantic relationships of individual words to better decipher meaning
Google uses language models to recognize the semantic connections of individual words and thus decode the meaning better

Word analysis, however, is not only about understanding the context. Rather, it is also about classifying search queries thematically in order to make the corresponding answer even more accurate. If a search contains keywords such as “opening hours”, the search engine knows directly that the user does not want to receive results on products or the history of the company, but wants to know specifically when the store opens and closes. If the search query contains the word “images,” the searcher is taken directly to Google Images. In addition, the algorithm analyzes the content for up-to-dateness in order not to show you any outdated and thus possibly no longer relevant content.

Matching the search term

If the meaning of the word or phrase is clear, the search engine looks for pages that match the (search) query. Therefore, the search index is scanned here to find matching websites. Special attention is paid to keywords that match the search query. For this reason, various parts of the text, such as titles, headings and even entire text passages, are screened for them. Once this keyword match check is done, previous interaction data is looked at to analyze how the result was received by users so far and whether it was helpful. This allows queries to be answered with even more relevant results.

Since the algorithm is a machine-learning system, it can draw conclusions from previous data and signals and apply them to future queries. In doing so, the algorithm looks at all search queries objectively, without interpreting anything into them. Therefore, Google does not consider subjective attitudes, intentions or points of view when matching the search term.

Ranking of useful pages

This section of the key factors includes the ranking factors, which have a clear influence on which results are considered good and subsequently receive a better positioning in the search results. Therefore, in addition to the usability of the page, an optimized loading speed, the quality of the content, etc., links, for example, are also an important signal for the search engine.

However, caution is advised here! Not every link has a positive effect. Make sure that your page is not linked to from spam websites. These websites often aim to deceive or mislead the search engine. Spam and the associated machinations are something that Google does not like to see. For this reason, in such a case, you can expect penalties and negative consequences rather than an improvement of your own ranking. In order to always follow the correct path and not run the risk of causing negative consequences for your website, you should follow Google’s guidelines for webmasters.

The best results

The next step is then to analyze whether the information really matches the search query. Correctly understood: The complete process so far runs in seconds in the background – so you still don’t have a result on your screen.

Often there are an infinite number of results that match a search query. So which are the ones that will be displayed to the user in the end? For this purpose, various information and contents from the index are analyzed and then the most suitable of them is displayed to you. This works because the individual ranking factors are constantly adapting to the continuously evolving web. Thus, even better search results can be delivered. For example, the following things are examined:

  • Which browser are users surfing on?
  • Which end devices are used?
  • On which display sizes are the results displayed?
  • What are the page load times?
  • What is the internet connection like?

Contextual reference

In the last step, before you are finally presented with the search result, the personal context is examined. For this, things that Google knows about you so far are included. This includes for example

  • Your current location
  • Previous searches
  • Search settings and so on.

Above all, the location is an important factor, which sometimes leads to completely different results being displayed for one and the same keyword.

The importance of the location reference in a Google search query - football describes completely different sports in America or Europe
The importance of the location reference in a Google search query – football describes completely different sports in America or Europe, © Google

In some cases, your previous search history also becomes relevant for a search query. Here, Google interprets interests in past searches and uses this knowledge to present you with results that are even better tailored to you in the future. However, if you do not want Google to use this, you can define via your Google account which data may be used to optimize search results.

In summary: What happens with search queries?

Everything starts with you typing a search query into the Google search box. Immediately afterwards, the invisible process starts, which searches for the perfect match for your query. Within a very short time (often less than a second), the algorithm searches the search index, which is a kind of library with all the previously crawled content of the internet. This “library” has already been sorted and ordered long before your search, which makes the algorithm’s work much easier. After it has understood the semantic context of your query, matched search terms and looked at the rankings of the most relevant pages, it finds you the right result and then includes important personal background information that puts the query in the right context. Once this complete process is done, you will get the visible result with the most helpful and relevant answers.

What are Google algorithm updates and how often do they occur?

According to speculations, the Google algorithm is changed or optimized about 500 to 600 times a year. In most cases, this happens silently and under the radar. Often, this is due to the fact that continuous improvements are made and therefore often only small things are changed. Nevertheless, there are often cases in which a Google update is deliberately communicated officially. This is usually the case when many webmasters are affected by it, or it involves huge changes to the algorithm that can greatly affect rankings. This is the case, for example, with the Core Web Vitals, which are inserted as a new ranking factor. Although these are not supposed to become relevant until 2021, Google has already announced them in mid-2020.

In addition, there are always adjustments to technical innovations, such as smartphones, voice search, etc., which are incorporated into the Google algorithm. Continuous improvement is important here in order to always be able to achieve the best possible search results and to meet the user’s requests.

Google Algorithm and Search Engine Optimization

Especially, these updates keep SEOs all over the world on their toes on a regular basis. In addition to the already numerous ranking factors that have to be considered in the search engine optimization process, new ones are constantly being added. But even worse, adjustments that have already been made may become irrelevant out of nowhere.

The goal of SEO is to understand the algorithm in such a way that adjustments can be made to websites that will then lead to top rankings in the search results. For this purpose, OnPage and OffPage optimizations are carried out. Both involve countless variables that impact the performance and visibility of a website. Additionally, it is important to focus on the right keyword when creating and updating website content – a keyword analysis will help you with this. If you manage to adjust your website and its content exactly to the algorithm through search engine optimization, you will achieve a better ranking.

Short excursus: Why are there different search results on desktop and mobile?

Sometimes it happens: You get different results for one and the same search query on Google. Often it is due to the personal adjustments that Google makes for each search query. In addition, there can be various causes:

  • The location where you are at the moment
  • Varying search results due to one search query to different servers
  • Using a Google page that corresponds to your language but belongs to a different country (google.de vs. google.ch or google.at)
  • Use of different browsers or operating systems
  • Influence of previous search queries
  • Influence of web protocol or previously visited pages

There is also the fact that mobile and desktop searches differ significantly. This is because Google has its own search index for mobile devices: The so-called Mobile First Index. In it, Google collects information on mobile versions of websites, which it then retrieves when a search query is made on a mobile device. In this way, as part of the “mobile revolution,” the search engine giant wants to build on the fact that in the future more smartphones and tablets will tend to be used than desktops.

Conclusion: This is why the Google algorithm makes us happy

Of course, the Google algorithm is not alone. Every search engine has a similar tool to best answer users’ search queries. Yahoo’s algorithm, for example, is called Slurp. All these systems are complex processes that often turn the SEO world upside down, especially due to regular updates. Nevertheless, search engines have long been an integral part of our lives, and we can no longer imagine life without them. Something “googling” has long been established in our vocabulary, and rarely has knowledge been obtained faster and more uncomplicated than via search engines. Google and Co. make our lives and the acquisition of knowledge easier, and all thanks to crawlers and algorithms.

FAQs

Google Algorithm – what happens during search queries?

After you have typed in your search query, the invisible process starts, which searches for the perfect match for your query. Within a very short time (often less than a second), the algorithm searches the search index, which is a kind of library with all the previously crawled content of the internet. The algorithm then displays all the results in the database that match your search query.

What are Google algorithm updates?

The Google algorithm is regularly updated and improved. Among other things, Google wants to keep it up to date and adapt it to new circumstances.

How often are changes made to the Google algorithm?

According to speculation, the Google algorithm is changed or optimized around 500 to 600 times per year. In most cases, this happens silently and under the table. Sometimes, however, it happens that Google also communicates updates. These are often core updates.

Extreme fluctuations in the SERPs: Algorithm update at Google?

If you as a website operator notice extreme fluctuations, e.g. in your visibility, this may be due to a Google algorithm update. The changes often have an impact on different metrics, which in turn can also influence your KPIs.

What does the Google algorithm have to do with SEO?

Actually, the question is rather: What does SEO have to do with the Google algorithm? Without the Google algorithm there would be no SEO, because only through Google and the processes running in the background it is at all necessary to optimize for a search engine. SEO therefore deals with various measures that help the Google algorithm to find your own content better and display it to the right users at the right time.

Contact

Just contact us

  +49 9381 5829000