Brief IA
›

Google: robots.txt Prioritizes Specific Rules

AI Tools·Tom Levy·

Google: robots.txt Prioritizes Specific Rules

Google: robots.txt Prioritizes Specific Rules
Key Takeaways
1A specific group for a robot replaces the general directives of robots.txt.
2To remove a PDF from search results using X-Robots-Tag: noindex, Google must be able to crawl the document.
3Google distinguishes between useful SEO filters and undesirable combinations.
Why it matters : a poorly targeted restriction can prevent access to content intended for search engines or large language models.

In robots.txt, Google applies the rule corresponding to the longest URL and favors permission in case of a tie. This file regulates the crawling of sites by search engine bots and AI agents. A poorly targeted restriction can exclude content from search results or large language models.

Google Must Read the HTTP Header X-Robots-Tag: noindex

Preventing a bot from accessing a PDF does not serve the same purpose as requesting its exclusion from results. For the latter operation, Google must be able to read the HTTP header X-Robots-Tag: noindex: the document must therefore remain accessible for crawling. This distinction particularly concerns commercial guides, brochures, and other files offered for download.

The Disallow: /*.pdf$ directive targets addresses ending in .pdf, regardless of the directory. However, it allows /guide.pdf?download=1 as well as extensions written as .PDF. Each variant requires a separate check.

In WordPress, Disallow: /wp-content/uploads/*.pdf$ targets documents in the upload folder. An exception such as Allow: /wp-content/uploads/2024/09/allowed-document.pdf$ allows this specific document. Its more detailed path overrides the general prohibition.

Rules Applicable to Each Bot

Located at the root of the site, robots.txt is a text file intended to regulate visits from bots, now including both search engines and AI agents. User-agent refers to the recipient of the instructions; the group marked with an asterisk serves as a default when no specific group matches. Disallow instructs compliant bots to avoid certain paths, while Allow provides permissions, even under a restricted folder. The Sitemap directive provides the address of the XML sitemap to help engines find it.

A bot-specific group replaces general instructions rather than complementing them. Googlebot can thus visit /private/ if no prohibition is listed in its group, even if that path is blocked elsewhere: the restriction must be placed in the corresponding group. For AI training bots, the choice of prohibitions must consider documented usage; accesses triggered at user requests should be examined separately.

Within the same group, Allow: /downloads/free/ preserves access to this subfolder despite Disallow: /downloads/, which continues to target the rest. If the Allow and Disallow directives both apply exactly to /downloads/, Google permits the visit. The precision of the path thus distinguishes the rules before their more or less restrictive nature.

The syntax does not offer all the possibilities of regular expressions: only two wildcard characters are available. The asterisk accepts zero or more characters; the dollar sign marks the end of the address. For paths, filter= differs from Filter=, unlike directive names and agent identifiers, which are case insensitive.

La lettre du soirGratuit · 19 h

Le brief IA que les pros lisent chaque soir

Les 7 actus IA du jour, décryptées en 5 min. Gratuit.

Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.

Choisissez votre rythme

Gratuit · Pas de spam · Désabonnement en 1 clic

Limiting Unnecessary Visits and Protecting Accounts

On a commercial catalog featuring hundreds of thousands of pages, filters can endlessly multiply variants without unique content. Without restrictions, Google may attempt parameter combinations even on non-existent addresses, leading to spikes in requests and unnecessary consumption of server and bot resources. Gary Illyes from Google has repeatedly warned of this risk for action addresses. In 2024, Google updated its guidelines regarding, among other things, cart additions or wish lists.

Limiting visits to pages without sought visibility, such as a privacy policy, can promote the crawling of useful products and services. However, Google clarifies that saved resources are not automatically redirected elsewhere unless the site has already reached its crawling capacity limit. Avoiding superfluous visits also reduces the calculations needed by engines, benefiting sustainability.

The selection depends on the usefulness of the pages and indexing goals: internal searches, sorting without SEO objectives, and commercial actions do not all justify a visit. Account spaces that are not relevant to results can also be restricted, but authentication remains necessary to protect private information. JavaScript scripts that do not play a role in content or rendering, including certain tracking tools, may also be affected.

A form can, for example, submit its data to an API endpoint with the action attribute action="/form/submissions/". If a bot discovers this address, Disallow: /form/ requests that it not be crawled. The restriction applies to pages under this directory without the need for an asterisk.

Choosing Which Filters to Keep Accessible

Not all commercial facets are intended to be blocked. Zalando, for instance, has pages by color targeting "gray t-shirt," utilizing filters aimed at general product searches. Google distinguishes these useful destinations from undesirable combinations: the former should remain crawlable. Conversely, filters limited to product sorting can generate dozens of duplicate pages.

As an address can contain multiple parameters, each criterion to be restricted must be recognized at different locations. For sortby, the patterns are /?sortby= and /?*&sortby=; the same principle applies to color and price. Only truly undesirable criteria should be added to the prohibitions, with tests conducted on both addresses to keep and those to exclude.

UTM parameters serve a different function: tracking. John Mueller discussed tracking parameters present in external links on Reddit, a situation distinct from filters and product searches conducted by the site itself. A string of parameters alone does not justify a block: its usage on the site matters. Internal links should, in turn, point to the addresses whose visits are desired.

Restricting Internal Searches on WordPress

Pages generated from the internal search engine can be restricted when they are not intended to appear in external search results. In WordPress, this search typically relies on s, as in https://www.example.com/?s=google. These results are a frequent source of unnecessary visits.

To target them, Disallow: /?s= recognizes the initial parameter, while Disallow: /?*&s= recognizes it after another. These paths start with / and avoid targeting a different name whose only last letter would be s. They do not recognize /?S=: the check must focus on the spelling and order of the parameters actually used.

Brief IA — L'actualité IA en français

L'essentiel de l'actualité de l'intelligence artificielle, décrypté et expliqué chaque jour.