On Tue, 1 Nov 2011 18:32:01 -0700 Paul Nathan pnathan.software@gmail.com wrote:
One interesting application might be an 'archiving browser', which caches all/most of the sites you visit. Instead of rummaging through google trying to figure out what the search terms were to hit that one site (if it's still indexed by google and if it's still up), you can instead run a query on your local application.
As a personal project, I have been contemplating putting together a web spider/index for better web searching; it would be nice to contribute components from that to a larger project relating to web storage & archiving.
I really like this idea. There exist a few distributed spider+search engine projects which could perhaps one day with enough participants allow to replace commercial search engines, while permitting unrestricted searches (ever noticed how the public google search interface used to be more powerful, but was "censored" since?). Unfortunately, those projects are yet unpopular and could not at all compete at current time.
A distributed archiving system could also embed such a distributed search engine...