Robots.txt gives crawlers rules about which URLs they may fetch. It is not access security and does not guarantee that a URL disappears from search results.[1]
Use it to limit unnecessary crawling, such as endless filter combinations. Protect private information with actual access controls. Preventing indexing requires a different approach from merely prohibiting fetching.
Content updated:
Sources
Compiled with AI assistance using public sources. General explanation, not individual legal or tax advice. Describing a concept does not mean Zoloo offers it as a feature.