Design Google’s Crawl Budget · System Design Interview

Design Google’s Crawl BudgetDesign how a web-scale crawler decides what to fetch next: the machines exist, the fetching works, and the question is which of hundreds of billions of URLs are worth the fetches you have. Prioritisation, freshness and politeness are the whole exercise.What it has to do

Choose which URLs to fetch next from a frontier far larger than the capacity to fetch it

A page that changes often is refetched more often than one that never changes

No site is fetched harder than it can take, whatever its priority

Scale

URLs known: 500 billion

Pages fetched per day: 20 billion

Hosts: 300 million

Out of scope

The fetcher itself and its connection handling

Parsing and indexing

Ranking

Rendering JavaScript

Storage of fetched content

Practise this challengeAll system design challenges

LetsDiscuss

Loading your session…