How Google Search works: from crawl to ranking, the full pipeline

The pipeline at a glance

Google describes three stages. Most SEOs split them into five, because two steps that Google files under crawling, URL discovery and rendering, fail for their own reasons and need their own fixes:

1. Discovery. Google learns that a URL exists, from a link or a sitemap.
2. Crawling. Googlebot requests the URL and downloads the response.
3. Rendering. Google runs the page's JavaScript in a browser engine to see the final content.
4. Indexing. Google analyses the content, groups duplicates, picks a canonical and stores what it learned.
5. Serving and ranking. When someone searches, Google picks matching pages from the index and orders them.

The reason this model is worth learning first: every search problem belongs to the first stage where the page fails. A page that is never discovered will not be helped by a better title. A page blocked in robots.txt will not be helped by more internal links. Diagnosis always starts at stage one and moves right.

What Google officially says

Google defines the three stages like this:

• Crawling: “Google downloads text, images, and videos from pages it found on the internet with automated programs called crawlers.”
• Indexing: “Google analyzes the text, images, and video files on the page, and stores the information in the Google index, which is a large database.”
• Serving search results: “When a user searches on Google, Google returns information that's relevant to the user's query.”

Stage 1: URL discovery

Before Google can crawl anything, it needs a URL. Google is direct about the starting point: “There isn't a central registry of all web pages, so Google must constantly look for new and updated pages and add them to its list of known pages. This process is called "URL discovery".”

Google names three ways a URL becomes known:

• It was visited before. “Some pages are known because Google has already visited them.”
• A link points to it. Google extracts links from pages it already knows. Its own example is a hub page, such as a category page, linking to a new blog post. The SEO Starter Guide puts it plainly: “Google primarily finds pages through links from other pages it already crawled.” Those can be your own pages or other websites.
• A sitemap lists it. “Still other pages are discovered when you submit a list of pages (a sitemap) for Google to crawl.”

A typical case: you publish https://example.com/blog/new-post/. Google finds it because the /blog/ hub now links to it, because an older article links to it, and because your CMS added it to sitemap.xml. Each of these is a separate path in; the more of them a page has, the more reliably it is found.

Sitemaps help, but they are a supplement. About submitting one, the Starter Guide says: “However this isn't required, and you should first focus on making sure people know about your site.” In practice, a page that is only in the sitemap and has no internal links (an orphan page) is usually discovered but crawled rarely, because nothing on the site signals that it matters.

Stage 2: Crawling

Once a URL is known, Google may fetch it. The program that does this is Googlebot. It does not crawl everything at the same pace: “Googlebot uses an algorithmic process to determine which sites to crawl, how often, and how many pages to fetch from each site.”

Crawl rate and your server

Google's crawlers try not to overload a site, and the speed adapts to how the server responds: “This mechanism is based on the responses of the site (for example, HTTP 500 errors mean "slow down").” A slow or failing server therefore gets crawled less. If you see crawling drop, check server errors and response times before anything else.

Recrawling

Crawling is not a one-off event. Google's How Search Works site explains that its crawlers “learn how often content they've seen before seems to change and revisit as needed.” Pages that change often tend to be revisited more often; pages that rarely change are revisited less. The Starter Guide adds that grouping similar topics in directories can help Google learn how often each directory changes on larger sites.

Pages Googlebot will not crawl

“However, Googlebot doesn't crawl all the pages it discovered. Some pages may be disallowed for crawling by the site owner, other pages may not be accessible without logging in to the site.” Google lists the common access problems as:

• “Problems with the server handling the site”
• “Network issues”
• “robots.txt rules preventing Googlebot's access to the page”

One distinction trips up many people: robots.txt controls crawling, not indexing. A URL blocked in robots.txt can still appear in the index without its content if other pages link to it. And a noindex rule on a blocked page is never seen, because Google never fetches the page to read it.

# robots.txt on example.com — stops crawling of /cart/, not indexing
User-agent: *
Disallow: /cart/

<!-- in the page itself — stops indexing, but only if the page can be crawled -->
<meta name="robots" content="noindex">

Stage 3: Rendering

The raw HTML is often not the whole page. Google's documentation describes what happens next: “During the crawl, Google renders the page and runs any JavaScript it finds using a recent version of Chrome, similar to how your browser renders pages you visit.” And why it matters: “Rendering is important because websites often rely on JavaScript to bring content to the page, and without rendering Google might not see that content.”

Rendering needs the same resources a browser needs. The Starter Guide warns that if your site hides important components like CSS and JavaScript from Google, it might not understand your pages. Its practical advice: “To check how Google sees your page, use the URL Inspection Tool in Search Console.”

Two practical points:

• Server HTML is safer. Content, links, titles and canonicals that are present in the server response do not depend on rendering succeeding. Content that only appears after scripts run does.
• Location matters. If your pages change by visitor location, Google advises checking what its crawler sees from its own location, “which is generally the US.”

Stage 4: Indexing

After crawling and rendering, Google tries to understand the page. Indexing includes processing the text and key tags and attributes, such as the <title> element and alt attributes, plus images and videos.

Duplicates and canonical selection

“During the indexing process, Google determines if a page is a duplicate of another page on the internet or canonical.” Google groups pages with similar content into a cluster and picks the most representative one. “The canonical is the page that may be shown in search results.” The other URLs in the cluster are alternates that may be served in specific contexts, for example to mobile users or to someone looking for that exact page.

Signals collected for later

Google also stores signals about the canonical page for the serving stage. “Some signals include the language of the page, the country the content is local to, and the usability of the page.”

What may not be indexed

Google is clear that this stage is selective: “Indexing isn't guaranteed; not every page that Google processes will be indexed.” The common reasons it gives:

• “The quality of the content on page is low”
• “Robots meta rules disallow indexing”
• “The design of the website might make indexing difficult”

Add to that the duplicate case: a URL that Google treats as an alternate of another canonical is not indexed on its own. That is normal and usually fine, as long as the canonical is the URL you want.

Stage 5: Serving and ranking

When someone types a query, Google searches its index for matching pages and returns the results it believes are the highest quality and most relevant. Two statements frame this stage: “Google doesn't accept payment to rank pages higher, and ranking is done programmatically.” And: “Relevancy is determined by hundreds of factors, which could include information such as the user's location, language, and device (desktop or phone).”

Google's How Search Works site groups the key signals into five areas:

What Google says about each, in its own words:

• Relevance: “The most basic signal that information is relevant is when content contains the same keywords as your search query.” Google also says it uses aggregated and anonymised interaction data to assess relevance.
• Quality: Google's systems look for signals of expertise, authoritativeness and trustworthiness; one example it gives is whether other prominent websites link or refer to the content.
• Usability: “When all other signals are relatively equal, content that people will find more accessible may perform better.”
• Weighting: “The weight applied to each factor varies depending on the nature of your query.” For current news, freshness matters more than it does for a dictionary definition.

The layout of the results page also changes with the query. Google's example: a search for bicycle repair shops will likely show local results and no image results, while a search for a modern bicycle is more likely to show images. Which features appear, including AI Overviews, is covered in the next lesson of this module.

Indexed but not visible

Search Console may say a page is indexed while you never see it in results. Google gives three reasons:

• “The content on the page is irrelevant to users' queries”
• “The quality of the content is low”
• “Robots meta rules prevent serving”

What Google does not do

A lot of bad advice comes from misunderstanding these limits. Google states each of them directly.

• ✕It does not sell crawling or rankings.“We don't charge anyone to be in our search index, and we don't accept payment for search result rankings.”
• ✕Ads do not change organic results.“The results you see on Search are solely based on your query, and are not influenced by whether a business, individual, or organization buys ads.” Ads are labelled Sponsored or Ad and shown separately.
• ✕It does not guarantee inclusion.Following the rules makes a page eligible, not indexed.
• ✕It does not need manual submission.Most pages are found automatically through links. Indexing requests and sitemaps help, but they are not an entry ticket.
• ✕It does not read meta keywords.“Google Search doesn't use the keywords meta tag.”
• ✕There is no word-count target.The Starter Guide says content length alone does not matter for ranking.
• ✕There is no duplicate content penalty.Duplicates on your own site are inefficient and get consolidated, but they do not cause a manual action. Copying other sites' content is a different matter.
• ✕E-E-A-T is not a ranking factor in itself.Asked whether it is, the Starter Guide answers: “No, it's not.” It describes the kind of content the quality signals aim to reward, not a single score.

Changes are also not instant. The Starter Guide says: “Some changes might take effect in a few hours, others could take several months.” Its advice is to wait a few weeks before judging whether a change worked.

Where each stage shows up in Search Console

Every stage leaves a trace in Search Console. Knowing which report belongs to which stage turns a vague complaint (my page is not on Google) into a precise question.

A quick outside check is the site: operator, for example site:example.com/blog/. The Starter Guide suggests it to see whether Google has found your content at all. Treat it as a rough sign of presence, not as an index count; URL Inspection is the reliable answer for a single URL.

What can stop a page at each stage

When a page is missing from search, go down this list in order and stop at the first stage that fails.

Common mistakes

• ✕Treating crawled as indexedA crawl is a visit, not a decision. Crawled pages can still be left out of the index.
• ✕Using robots.txt to remove pages from GoogleIt stops crawling, not indexing, and it hides any noindex on the page.
• ✕Relying on the sitemap aloneA sitemap gets a URL discovered; internal links tell Google the page matters.
• ✕Putting key content only in JavaScriptIf rendering fails or a resource is blocked, Google may not see it.
• ✕Fixing rankings when the page is not indexedTitle and content tweaks do nothing for a page stuck at an earlier stage.
• ✕Paying for faster indexing or guaranteed rankingsGoogle does not sell either. Anyone promising them is wrong, in Google's words.
• ✕Judging a change after one daySome changes take hours, others months. Give it a few weeks.

Where this course goes deeper

This lesson is the map; each stage has its own lessons later in the course:

FAQ

Sources

• Google Search Central — In-depth guide to how Google Search works
• Google Search Central — SEO Starter Guide
• Google — How Search Works
• Google — How Search organizes information
• Google — How Google determines ranking results

Quotes checked against the official pages on 30 October 2026.