robots.txt
robots.txt is a plain text file that lives at the root of a website — for example, example.com/robots.txt — where the site owner tells automated crawlers which parts of the site they would prefer bots not visit. It is part of a long-standing convention called the Robots Exclusion Protocol.
The file lists user agents and the paths they should or should not request. A search-engine crawler like Googlebot reads it before indexing and honors the rules, which is how sites keep certain pages out of search results or steer crawlers away from heavy or sensitive sections.
An important nuance: robots.txt is a request, not a lock. It is a set of instructions well-behaved bots choose to follow, but it does not technically prevent access, and it is not a security or legal boundary on its own. It signals a site's preferences about automated crawling — and most reputable crawlers respect it as a matter of good conduct. It is worth reading a source's robots.txt to understand what the owner is asking of automated visitors, keeping in mind that the file expresses crawling preferences rather than what is or is not lawful.
What it means for 1Scrape users
robots.txt is one signal in the wider picture of scraping responsibly and staying mindful of a site's wishes. 1Scrape focuses on collecting public data — the same posts and listings any visitor can see — and you are responsible for using what you gather lawfully and respectfully. It is a good habit to know a source's stated preferences. This is general information, not legal advice.
Frequently asked questions
Does robots.txt make scraping illegal?
No. robots.txt expresses a site's preferences for automated crawlers, but it is not a law and not a technical barrier by itself. Whether any data collection is lawful depends on the data and how it is used, not on that file alone. General information, not legal advice.