Short answer. Paste the robots.txt text of the host that carries your backlink, give the URL or path of the page with the link, and pick a user-agent token. The tester applies Google's documented group selection and longest-match precedence and shows the winning line. It runs in your browser and makes no network request.
Updated 2026-10-29
One cause of a backlink that never gets crawled is a robots.txt rule on the site that carries it. Paste that file here and see whether the exact URL is allowed or blocked for the crawler you select.
By Michael Lip. The sources on this page were read on 29 October 2026.
Everything runs in this browser tab. The tester sends nothing over the network, so it cannot fetch a robots.txt for you: use the curl command below to get the file first.
How the tester groups rules
The tester follows the grouping that Google documents in its robots.txt specification. One or more consecutive user-agent lines open a group, and the allow and disallow lines after them belong to that group. When the same user-agent token appears again later in the file, Google combines the two groups into one, so the tester merges them. If no group names your token or its second token, the group for * applies. Google's special-case crawlers APIs-Google, AdsBot-Google, AdsBot-Google-Mobile and Mediapartners-Google ignore the * group, and the tester does the same. Lines that are not user-agent, allow or disallow do not end a group. Source: Google's robots.txt specification.
The tester picks the most specific group that matches your token. Google ignores the text after the token, so googlebot/1.2 and googlebot* both match googlebot. Google gives Googlebot-Image, Googlebot-Video, Googlebot-News, Google-InspectionTool and Google-CloudVertexBot a second token, Googlebot, and gives GoogleOther-Image and GoogleOther-Video a second token, GoogleOther. These crawlers use that group when no group names them. User-agent tokens are compared case-insensitively, per RFC 9309. Path matching, on the other hand, is case-sensitive: Allow: /Page does not match /page.
Which rule wins
Inside the selected group, the tester compares every rule whose path matches the URL path. Rules are ranked by the length of the rule path itself; the longest rule path wins. On an equal-length match, the least restrictive rule wins, so an allow beats a disallow. Google's specification states this rule. RFC 9309 section 2.2.2 says that an allow rule wins over an equivalent disallow rule. Google's own samples for allow: /folder with disallow: /folder and for allow: /page with disallow: /*.ph both resolve to allow because “in case of conflicting rules, Google uses the least restrictive rule”. Source: Google's robots.txt specification, and RFC 9309.
Two wildcards work in paths: * stands for any sequence of characters, and $ marks the end of the URL, so /*.php$ does not match /filename.php?parameters.
Google compares rules and URLs in their percent-encoded forms, as RFC 9309 section 2.2.2 defines, so Disallow: /foo/bar/ツ and Disallow: /foo/bar/%E3%83%84 are the same rule. The tester does the same, and the Characters column counts the rule path in that form. RFC 9309 also says that the /robots.txt URI is implicitly allowed, so the tester always shows Allowed for that path.
Built-in sample situations
Google's specification shows six sample situations for the precedence rules and one example of merged groups. The tester has all of them built in. Each button fills the inputs and evaluates the path. The group example from the specification uses these three groups:
Each case uses the rules allow: /p and disallow: /, or the variants shown on the button. Run one at a time.
Group selection cases
Case 5 (Googlebot): no group names googlebot, so the crawler uses the * group, disallow: /carrots. That rule does not match /shrimp/a, so the result is Allowed. Case 6 (Googlebot-News): Google merges the two googlebot-news groups into disallow: /fish plus disallow: /shrimp. The rule /shrimp matches, so the result is Blocked.
What a block means and does not mean
Blocking the page that carries your backlink stops Googlebot from fetching it. Google's own documentation says that if other pages point to a page with descriptive text, Google could still index the URL without visiting the page. Such a result shows no description. Source: Google's robots.txt introduction.
The Search Console robots.txt report shows the robots.txt files of your own Search Console property, and Google says that the report is available only for properties at the domain level. URL Inspection works only for a URL in your own property. Google says that the URL must be in the current property. A backlink lives on a host that you do not own, so you fetch its robots.txt yourself. The curl command above does this.
Two limits of the file format itself, from Google's specification: if a server returns a 4xx status for robots.txt (except 429), Google treats the site as having no robots.txt, so every URL is allowed; and Google ignores the content of a robots.txt after the first 500 KiB.
Questions people ask
Does the tester fetch robots.txt from the internet
No. The tester makes no network request, and a browser lets a page read a file on another host only when that host allows it. So the tester evaluates the text you paste. Copy the file with the curl command that the tester shows you.
Which crawler does the result apply to
To the user-agent token you select, applied with Google's documented rules. The tester does not claim to show how any other search engine reads the file.
Is my backlink page blocked if robots.txt returns an error
If the host returns a 4xx status (except 429) for robots.txt, Google treats the site as having no robots.txt, so nothing is blocked by that file. A 5xx status is different. For the first 12 hours, Google stops crawling the site. Then, for up to 30 days, Google uses the last good copy of the file. With no copy, Google assumes that there are no crawl restrictions. If the errors continue after 30 days and the site is generally available to Google, Google behaves as if there is no robots.txt file.
Where these rules come from
Each rule that the tester applies comes from one of these sources, which were fetched on 29 October 2026:
Google's robots.txt specification, group selection, combined groups for the same agent, the * fallback group, lines that do not end a group, longest-match precedence with allow on an equal match, the * and $ wildcards, case-sensitive path matching, the 500 KiB limit, and 4xx/5xx handling. It also contains the sample situations that the built-in cases reproduce.
RFC 9309, section 2.2.2 states the longest-match rule, that an allow rule wins over an equivalent disallow rule, how paths are percent-encoded before comparison, and that the /robots.txt URI is implicitly allowed; user-agent tokens are case-insensitive.
Google's robots.txt introduction, the URL of a page blocked by robots.txt can still appear in search results without a description, and Google can index a disallowed URL that is linked from other places on the web; robots.txt is not a mechanism for keeping a web page out of Google.
Google's list of common crawlers, the user-agent tokens in the token list, the second token Googlebot of Googlebot-Image, Googlebot-Video, Googlebot-News, Google-InspectionTool and Google-CloudVertexBot, and the second token GoogleOther of GoogleOther-Image and GoogleOther-Video.
This page summarizes those published documents in our own words. It has no relationship with Google. It does not promise a crawling or indexing result.