Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I've been thinking about this for a while. It's infeasible for any one person to run a Google-scale web crawler -- but if millions of people join in?

I've used file sharing software which has the same "tree style" query forwarding. I think latency would be the big issue though, we're used to getting a result instantly -- and the right result too (Google does a lot of extra work knowing what you actually mean, not just keyword search).



Common Crawl is a good resource if having crawl data potentially being from a month ago is OK for your application. Donate and use their crawl data.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: