menu
Get a daily delivery of PSFK
Subscribe to get a daily digest of new ideas and discoveries and to find out about upcoming events

Creating The Next Google Just Became More Affordable

Creating The Next Google Just Became More Affordable

The Common Crawl Foundation has made 5 billion indexed web pages readily available for free.

Yi Chen


The Common Crawl Foundation has indexed 5 billion web pages and has made the data readily available to anyone for free on the Amazon EC2/S3 cloud computing infrastructure. What this essentially means is that tech innovators looking to challenge Google and create the next best search engine can do so more easily, quicker and cheaper.

To access the information, users will need to setup their own Amazon EC2 Hadoop cluster and pay for the time they use it. There are no upfront costs to use the Amazon EC2 Hadoop and it charges cost per instance hour.

Common Crawl Foundation

{{post.author_display_name}}
  • {{post.date_formated}}
{{post.author_display_name}}
  • {{post.date_formated}}
Read More Tap to Expand
PSFK Writer {{post.author_display_name}}
  • {{post.date_formated}}
Get a daily delivery of PSFK
TREND REPORT

PSFK Labs Presents
The Future of Connected Life

Live Work Play Better:
The New Consumer and Their Journey to Effectiveness, Balance & Personal Growth