Design alternatives
Consider we have a number of internet nodes serving fixed size (say
10k) blobs of encrypted data via http. Each blobs has a unique
blob-id.
There seem to me to be 3 basic approaches to locating information:
-
distribute the data according to blob-id prefix. Number the
servers 0..n-1, place blob-ids at server (blob-id mod n). Have some
redundancy so there are multiple servers with the same server-id.
Then locating data is simple, just pick a server at random from the
list of servers sharing server-id.
Apparently some smart caches and high performance proxies do something
like this based on document hash.
- Have a URL server which maps virtual document URLs to server-id and
blob-ids. Either user distributes data as they see fit, or servers
self organise and keep random data, or use LRU cache replacement
policy.
Ian Goldberg & David Wagner's TAZ does something like this -- the TAZ
is a virtual URL database. (Their rewebber is chained encrypted
URLs).
- Have a web of servers which will request information from each
other, where the web is organised however the operators see fit, or
using some agreed network structure. You make a request from any node
and it returns it if it has it
Freenet proposes to do something like this. Not much activity on
freenet project at present.
Obviously there are lots of design permutations that could be tried.