Web Crawler

Functional requirements

• Start from a set of seed URLs and discover new pages by following links
• Fetch and store the raw content of each page for downstream indexing
• Obey each site's robots.txt rules
• Revisit pages periodically, more often for pages that change often
• Skip pages whose content is identical to one already stored

Non-functional requirements

• Never send a single host more than about one request per second
• Throughput should scale roughly linearly with the number of fetchers
• A crashed fetcher must not lose its share of the work
• Avoid crawler traps such as endless calendar or session-id URLs

Scale assumptions

Worked estimates for this brief, not measurements of any real system. Disagree with them out loud — that is the exercise.

Target

1 billion pages per month, as a planning assumption

Page size

~100 KB average HTML, so ~40 MB/second (~320 Mbit/second) inbound

URLs seen

~20 links per page, mostly repeats; assume ~10 billion distinct URLs

Fetch rate

1 billion ÷ ~2.6 million seconds ≈ 400 pages/second average

Raw storage

1 billion × 100 KB = 100 TB/month, ~25 TB if compressed 4:1

Seen-set memory

10 billion URLs in a Bloom filter at 1% false positives ≈ 12 GB

Suggested API surface

• POST/v1/seeds

  Add seed URLs to the frontier

• internalfrontier.next(fetcherId)

  Lease the next batch of URLs whose host is due

• internalfrontier.complete(urls, links)

  Report results and newly discovered links

• GET/v1/pages/{urlHash}

  Fetch stored content and crawl metadata for one URL

Decisions you should be able to defend

• How politeness is enforced
• Remembering which URLs were seen
• Crawl order
• Duplicate content