Design a Web Crawler · System Design Interview

Design a Web CrawlerDesign a crawler that walks the public web from a list of seed URLs and stores each page it fetches.What it has to do

Fetch pages starting from a seed list and follow the links found in them

Never fetch the same URL twice within one crawl cycle

Respect each site’s stated crawl rules and rate

Scale

Pages fetched per day: 1 billion

Distinct hosts: 200 million

Out of scope

Building the search index

Ranking

Rendering JavaScript

Image and video crawling

LetsDiscuss

Loading your session…