Pagination and filters: crawlable lists without a flood of filter URLs

Why lists, pagination and filters are an SEO problem

Long lists are everywhere: category pages in a shop, blog archives, internal search results, product reviews, comments under an article. Showing everything at once is slow, so sites split lists into pages, load more items on request or let users narrow the list with filters. Each of these choices has SEO consequences, and they fall into two separate problems.

Problem one: content Google cannot reach. If page 2 of a category is only reachable through a JavaScript button, or if items load only when someone scrolls, Googlebot may see the first 24 products and nothing else. Products and posts that are linked only from deep in the list are discovered late or not at all.

Problem two: URLs that multiply. Every filter, sort order and view option can create a new URL that shows almost the same list. A shop with a handful of filters can produce thousands of such URLs from one category. Google crawls them before it can tell they are useless, which costs your server resources and delays the crawling of pages that matter. Many of them also end up as duplicates or near-empty pages in the index.

Pagination needs more crawlable links. Faceted navigation needs fewer. Most mistakes come from confusing the two.

What Google officially says

Google describes the cost of faceted navigation in two points. Each is a direct quote:

• “Overcrawling: Because the URLs created for the faceted navigation seem to be novel and crawlers can't determine whether the URLs are going to be useful without crawling first, the crawlers will typically access a very large number of faceted navigation URLs before the crawlers' processes determine the URLs are in fact useless.”
• “Slower discovery crawls: Stemming from the previous point, if crawling is spent on useless URLs, the crawlers have less time to spend on new, useful URLs.”

On rel="prev" and rel="next", which many guides still recommend, Google is direct: “Google no longer uses these tags, although these links may still be used by other search engines.”

Pagination, load more or infinite scroll

Google describes three ways to show part of a long list. Pagination uses "next", "previous" and page-number links, one page of results at a time. Load more is a button that adds the next batch to the current page. Infinite scroll loads more content when the user reaches the end of the page.

Whichever pattern you choose, Google's crawling rule is the same: it follows links found in the href of <a> elements. For a shop, Google also suggests a second route: “consider using a sitemap file or a Google Merchant Center feed to help Google find all of the products on your site.” A sitemap helps discovery, but it does not replace crawlable links inside the list.

Pagination best practice

Give each page a unique URL

Google's instruction is “Give each page a unique URL.” It suggests a query parameter: “include a ?page=n query parameter, as URLs in a paginated sequence are treated as separate pages by Google.” A path such as /shoes/page/2/ works just as well. Pick one format and use it everywhere: in links, canonicals and the sitemap.

Link pages sequentially

Google recommends that you “include links from each page to the following page using <a href> tags. This can help Googlebot (the Google web crawler) find subsequent pages.” The pager must be in the HTML the server sends, as normal links. A pager built from <span onclick> elements, or one that only appears after a click, is invisible to the crawler.

Numbered links (1, 2, 3 … 9) are better than "next" alone, because they shorten the path to deep pages. With only "next", page 20 is twenty clicks away from the category.

Link back to the first page

Google adds: “consider linking from all individual pages in a collection back to the first page of the collection to emphasize the start of the collection to Google.” “This can give Google a hint that the first page of a collection might be a better landing page than other pages in the collection.” A "1" link in the pager or a breadcrumb to the category does this.

Self-referencing canonical on every page

Page 2 shows different items from page 1, so it is not a duplicate of page 1. A canonical from page 2 to page 1 asks Google to treat them as the same page, and the items linked only from page 2 lose their crawl path. Give each page a canonical to itself:

<!-- on https://example.com/shoes/?page=2 -->
<link rel="canonical" href="https://example.com/shoes/?page=2">

No # fragments for page numbers

“Don't use URL fragment identifiers (the text after a # in a URL) for page numbers in a collection.” The reason, in Google's words: “If Googlebot sees a URL to the next page that only differs by the text after the #, it may not follow the link, thinking it has already retrieved the page.” /shoes/#page=2 is the same URL as /shoes/ for Google.

rel="prev" and rel="next"

Google no longer uses them. You do not need to add them, and removing them changes nothing in Google. Other search engines may still read them, so leaving existing tags in place does no harm. What Google relies on is the visible links between pages.

Titles and descriptions of paginated pages

Google normally recommends distinct titles, but makes an exception here: “pages in a paginated sequence don't need to follow this recommendation. You can use the same titles and descriptions for all pages in the sequence. Google tries to recognize pages in a sequence and index them accordingly.” Adding "– Page 2" to the title is still a common, harmless habit that helps users who have several tabs open. Keep the category introduction text on page 1 only; repeating it on every page adds nothing.

The page 1 duplicate

Many CMSs make the first page available at two URLs: /shoes/ and /shoes/page/1/ (or /shoes/?page=1). That is a plain duplicate. Page 1 should live only at the clean category URL. Redirect /page/1/ and ?page=1 with a 301 to https://example.com/shoes/, and make the "1" link in the pager and every "back to page 1" link point to the clean URL.

Page numbers that do not exist

If the category has nine pages, /shoes/?page=50 should return HTTP 404, not an empty list with status 200 and not a redirect to page 1. Google lists “nonexistent pagination URLs” among the URLs that should return 404. The same applies when products sell out and the last pages disappear.

Speed between pages

Google suggests: “Consider using preload, preconnect, or prefetch to optimize the performance for a user moving to the next page.” This is a user experience improvement; it does not change how Google crawls the series.

<!-- on /shoes/?page=2 -->
<link rel="canonical"
      href="https://example.com/shoes/">

<a href="#page=3">3</a>
<button onclick="loadPage(4)">
  Next</button>
<a href="/shoes/page/1/">1</a>

<!-- on /shoes/?page=2 -->
<link rel="canonical"
      href="https://example.com/shoes/?page=2">

<a href="/shoes/">1</a>
<a href="/shoes/?page=3">3</a>
<a href="/shoes/?page=3">Next</a>

Should deep paginated pages be noindex?

Some sites add noindex to page 2 and beyond, so that only page 1 appears in search. It works for its narrow purpose, but it has a cost that is easy to miss.

• What you gain: deep list pages stop appearing in search results. They rarely rank anyway, because page 1 is the better landing page.
• What you risk: a noindex page still has to be crawled for Google to see the rule, so it saves no crawling. In practice, pages that stay noindex for a long time tend to be crawled less often, and their links tend to be treated like nofollow over time. Products and posts that are linked only from deep pages can then become harder to discover.

The safer default is to keep paginated pages indexable with self-referencing canonicals. If deep pages are a problem, fix the structure instead:

• show more items per page, so the series is shorter;
• split very large categories into subcategories;
• link to important items from other places too: related products, guides, the home page;
• list every product or post URL in the XML sitemap.

A "view all" page is an option only for small lists that load quickly. For large categories it brings back the performance problem pagination was meant to solve.

Keep noindex for list variations that are genuinely useless in search: sort orders, view modes, items-per-page options. Google names this use itself: “To avoid indexing variations of the same list of results, block unwanted URLs from being indexed with the noindex robots meta tag or discourage crawling of particular URL patterns with a robots.txt file.”

Infinite scroll and load more done crawlably

The rule for both patterns is the same: build normal pagination underneath, then add the scrolling or the button on top with JavaScript. Google puts it this way: “Google Search does not interact with your page.” What users get as one long page, Google must be able to reach as a series of ordinary URLs. Google's checklist for infinite scroll:

1. Persistent URLs. “Give each chunk its own persistent, unique URL.”
2. Stable content per URL. “Ensure that the content shown on each URL remains the same every time it's loaded in a browser.” Use absolute page numbers such as ?page=12. “Avoid using relative elements like ?date=yesterday in these URLs.”
3. Sequential links. “Link sequentially to the individual URLs so that search engines can discover the URLs in a paginated set.” Keep these links visible and usable, for example a pager at the bottom of the page.
4. An updated address bar. When a new chunk becomes the main visible part of the page, “update the displayed URL using the History API.” Users can then refresh, share and link to the part of the list they are looking at.

// when chunk 3 scrolls into view
history.pushState(null, "", "/shoes/?page=3");

For a "Load more" button, the simplest crawlable version is a real link that JavaScript enhances. Crawlers see a link to page 2; users who click get the next batch added in place:

<a href="/shoes/?page=2" class="load-more">Load more shoes</a>
<!-- JavaScript intercepts the click, fetches page 2 and appends it -->

Then test it. Open the page in URL Inspection in Search Console, view the rendered HTML and check that the links to the next page are there. Then open /shoes/?page=2 directly in a browser: it must work as a standalone page with its own items.

How faceted navigation explodes into URLs

Faceted navigation is the set of filters on a listing page: colour, size, brand, price, rating and so on. Google calls it “a popular and useful feature” for visitors, but notes that its most common implementation, which is based on URL parameters, “can generate infinite URL spaces”. A typical filter URL looks like this:

https://example.com/shoes/?color=red&size=9&brand=acme

Change any value and the URL changes. The number of URLs is the product of all the options, not the sum.

The same list also reappears at new URLs through parameters that are not filters at all: sort orders (?sort=price), view modes (?view=grid), items per page (?limit=48) and tracking tags. None of these should ever be indexable.

The damage is the one Google describes: overcrawling and slower discovery. Google also warns that “crawling faceted URLs tends to cost sites large amounts of computing resources due to the sheer amount of URLs and operations needed to render those pages.” On top of that, many combinations show the same few products or none at all, so the index fills with near-duplicate and empty pages. There is no penalty for this, but signals get split and the pages you care about are crawled later.

One thing has changed for good: Search Console's URL Parameters tool has been retired. You cannot tell Google how to treat parameters from the outside any more. Control has to happen on the site itself.

Your options for filter URLs

Google first asks one question: do you need these URLs in search at all? If not, it recommends preventing crawling, and gives two main ways to do it. It then mentions two weaker signals, and its pagination guide adds noindex for list variations.

robots.txt disallow for filter parameters

Google's view: “Oftentimes there's no good reason to allow crawling of filtered items, as it consumes server resources for no or negligible benefit; instead, allow crawling of just the individual items' pages along with a dedicated listing page that shows all products without filters applied.” Its example file:

user-agent: Googlebot
disallow: /*?*products=
disallow: /*?*color=
disallow: /*?*size=
allow: /*?products=all$

The /*?*color= pattern matches the parameter anywhere in the query string, whatever its position. Things to know before you use it:

• robots.txt stops crawling, not indexing. Google warns that it “may still index URLs that are disallowed in robots.txt without their content.” Such URLs usually stay rare if nothing links to them, but they can appear.
• A blocked URL is never fetched, so Google never sees a noindex or canonical on it. Do not combine robots.txt with those signals on the same URL pattern.
• Blocking does not clean up URLs that are already indexed. If thousands of filter URLs are indexed today, let Google recrawl them with noindex or a canonical first, and add the robots.txt rule once they have dropped out.
• Make sure the rules do not catch URLs you want crawled, such as pagination (?page=) or the facet landing pages you chose to index.

Filters in URL fragments

“Google Search generally doesn't support URL fragments in crawling and indexing.” So if your filters change only the part after #, for example https://example.com/shoes/#color=red&size=9, Google sees just /shoes/. In Google's words, a fragment-based filter “will have no impact on crawling (positive or negative).” This is clean for filters you never want in search, but it also means none of those filter states can ever rank.

rel="canonical" to the unfiltered page

A filtered URL can point its canonical to a broader version, for example from ?color=red&size=9 back to the category. According to Google, this “may, over time, decrease the crawl volume of non-canonical versions of those URLs.” The words "may" and "over time" matter: Google still has to crawl the filter URLs to read the tag, and the canonical is a hint. It also only makes sense where the filtered page is close enough to the target to count as a duplicate. It is a good choice when the filtered URLs have external links or traffic you want to consolidate.

rel="nofollow" on filter links

Adding nofollow to filter links can help, but Google warns that “every anchor pointing to a specific URL must have the rel="nofollow" attribute in order for it to be effective.” One link without it, in a breadcrumb, a sitemap or another site, is enough for the URL to be crawled.

Google sums up both: “these methods are generally less effective in the long term than the previously mentioned methods.”

noindex robots meta tag

Google's pagination guide lists noindex for filtered and sorted versions of a list. It reliably keeps those URLs out of search results. It does not reduce crawling, because Google has to fetch each page to see the tag. Use it when the URLs must stay crawlable, for example while you clean up indexed filter URLs, or when you cannot express the pattern in robots.txt.

A practical combination for most shops: filter links that should not be indexed point to parameter URLs blocked in robots.txt (or use fragments), sort and view options never appear as crawlable links, and a small set of chosen facet pages get clean, indexable URLs.

Which facets deserve indexable landing pages

Some filter combinations match what people actually search for: "red running shoes", "nike running shoes", "waterproof hiking boots". Blocking all facets throws that demand away. The trick is to choose these pages deliberately instead of letting the filter system decide.

For each combination that passes, build it like a real category page:

1. Confirm demand in keyword research or in Search Console queries for the parent category.
2. Give it a clean, static URL, such as https://example.com/running-shoes/red/, instead of ?color=red. A separate URL format also makes it easy to keep the robots.txt rules for parameters intact.
3. Write a unique title, H1 and short intro for that combination ("Red running shoes"), not the parent category text.
4. Use a self-referencing canonical and list the URL in the XML sitemap.
5. Link to it internally from the parent category, menus or related guides, with plain <a href> links.
6. Paginate it like any category if it has more products than fit on one page.

Everything else stays non-indexable. The most effective way is not to produce crawlable links to those combinations in the first place.

Rules for facet URLs you let Google crawl

If you decide that some filter URLs may be crawled and indexed, Google lists rules to keep the damage small. Remember its warning: “Keep in mind that having these URLs crawled means an increased resource usage on your server and, potentially, slower discovery of new URLs on your site.”

Google notes one exception: on a single-page app, returning a real 404 for each combination might not be possible, and the separate best practices for single-page apps apply.

Common mistakes

• ✕Canonical from every page to page 1Page 2 is not a duplicate of page 1. Each page canonicalises to itself.
• ✕Pager or "Load more" without real linksButtons and click handlers are invisible to Googlebot. Use <a href>.
• ✕Page numbers in # fragmentsGoogle ignores the fragment and may not follow the link.
• ✕Relying on rel="prev" and rel="next"Google no longer uses them. Only the links between pages count.
• ✕/page/1/ left as a live duplicate301 it to the clean category URL and link to the clean URL.
• ✕Empty pages with status 200Filters with no results and page numbers past the end should return 404.
• ✕robots.txt plus noindex on the same URLsA blocked URL is never fetched, so its noindex is never seen.
• ✕Crawlable links to every combinationFilter panels that print hundreds of followable links on every category page.
• ✕Indexable sort and view parameters?sort=, ?view= and ?limit= show the same list at new URLs.
• ✕Parameter order that variesThe same filter set reachable as ?color=red&size=9 and ?size=9&color=red.
• ✕Blocking a pattern that holds valuable pagesA broad robots.txt rule that also catches pagination or chosen facet landing pages.

How to audit pagination and filters step by step

1. Crawl the site and group the URLs found by parameter (page, color, sort and so on). An unexpectedly large group is your first lead.
2. Check Search Console. In the Crawl stats report, look at which URLs Googlebot spends its requests on. In the Page indexing report, look for parameter URLs under statuses such as "Duplicate without user-selected canonical" and "Crawled - currently not indexed".
3. Check paginated pages in the raw HTML: self-referencing canonical, pager built from <a href> links, no page numbers in fragments.
4. Request /page/1/ and ?page=1 and confirm they 301 to the clean category URL.
5. Request a page number past the end and a filter combination with no results. Both should return 404 at that URL, not a redirect.
6. Test load-more and infinite-scroll pages in URL Inspection: the rendered HTML should contain links to the next page, and each page URL should work on its own.
7. Compare robots.txt with your parameter list. Every filter, sort and view parameter you do not want crawled is covered; pagination and chosen landing pages are not.
8. Check the sitemap: it should list canonical category pages, chosen facet landing pages and item pages, and no sort, view or random filter URLs.

OurSEO Tools can cover part of this in one run. Its 69-point checklist includes indexation, URLs and duplicates, redirects and internal linking, and its technical crawl of up to 1,000 pages records the status code, canonical, indexability and load time of every URL, which makes parameter URLs, wrong canonicals on paginated pages and empty pages returning 200 easy to spot.

FAQ

Sources

• Google Search Central — Pagination, incremental page loading, and their impact on Google Search
• Google Search Central — Fix lazy-loaded content (infinite scroll)
• Google Search Central — Managing crawling of faceted navigation URLs
• Google Search Central — How to specify a canonical URL

Quotes checked against the official pages on 30 October 2026.