robots.txt: block everything except a few folders?

Only /blog/ and /guides/ should be crawlable, everything else blocked. If I put Disallow: / and then Allow lines for those two folders, does google go by the order of the lines or does the most specific rule win?
2

Leave A Reply

Log in to post an answer.

2 Answers

johnny_serp Nov 09, 2026
jasmins
Keep in mind robots.txt stops crawling, not indexing. The blocked URLs can still show up in Google with no description if other pages link to them. If you need them out of the results completely, that's a noindex job, and Google has to be able to crawl a page to see the noindex, so don't block it in robots.txt at the same time. Test the file with the robots.txt report in Search Console after you change it.
kate_pratt Oct 14, 2026
jasmins
User-agent: * Disallow: / Allow: /blog/ Allow: /products/ Google goes with the most specific (longest) matching rule, so the Allow lines win for those folders. Remember your CSS and JS folders too, or Google can't render the pages it is allowed to see.