Web Crawler
A web crawler — also called a spider or bot — is a program that systematically browses the web, moving from page to page by following the links it finds. Its job is discovery: mapping what pages exist and where they live. Search engines run enormous crawlers to find and index the web so results can be served in a fraction of a second.
Crawling and scraping are related but not the same. Crawling is about navigation — finding and visiting pages; scraping is about extraction — pulling specific data off the pages once you are on them. Many real-world tools do both in sequence: crawl a list of pages to discover them, then scrape each one for the fields you actually want. In short, crawling builds the list of pages and scraping reads them.
A well-behaved crawler stays on public pages, follows sensible limits so it does not overload a site, and respects the rules a site publishes for automated visitors. That restraint is part of what separates responsible data collection from abuse, and it applies whether the crawler belongs to a search engine or a lead-gen tool.
How it relates to 1Scrape
When you run 1Scrape, a crawl-then-scrape flow happens under the hood so you never have to think about it. Point it at a Google Maps search and it works through the matching listings — the crawling part — then extracts the name, address, phone, website, and rating from each — the scraping part. You just see the count climb and download the finished CSV.
Frequently asked questions
What is the difference between a web crawler and a scraper?
A crawler discovers and visits pages by following links; a scraper extracts specific data from the pages it lands on. They often work together — crawl to find the pages, scrape to pull the data — but the crawler is about navigation and the scraper is about extraction.
Are web crawlers legal?
Crawling public pages is common and widely done — search engines depend on it. The considerations are the same as any scraping: stay on public content, do not overload the site, respect its published rules for bots, and mind privacy laws when personal data is involved.