Crawling a billion web pages in just over 24 hours