Some time ago I asked how much longer before I can have the Web in my pocket. Let’s try a quick back-of-the-envelope calculation:

A paper from January 2005 calculates the publicly indexable Web (the part easily accessible to search engine web-crawlers) as being around 11.5 billion pages. Estimates on average webpage size seem to be all over the map, but let’s figure around 100 KB per page, for a total of around a petabyte (one million Gig) for today’s indexed web. (I’m assuming text and images, but ignoring other media.)

Disk these days is going for less than 50 cents per Gig, so enough disk to store your own personal Google (and then some) costs around $500,000. With compression you can probably cut that in half. The price of disk is also falling by a factor of two every 12 months, so assuming no major jumps or snags in the disk-price curve, in a little less than a decade we can expect to hold the equivalent of today’s indexed web for less than $1000.

Now of course, in that time the web will continue to grow, so we may no longer be satisfied with our measly petabyte-on-the-desk, but I figure the amount of human-generated Web content has a much slower growth rate than our disk-space curve. The number of web sites actually shrank between 2001 and 2002, and though it now seems to be growing again there’s only so much content that human beings can create in a day. The real question I have is whether in a decade anyone will see having access to the whole web as being all that interesting — I could easily see the majority of people losing interest in the surface web in favor of personal deep-web niches. The only reason I want the whole web in my pocket is because it’s too hard for me to filter out in advance the 99.99% of the web that’ll never be of interest to me — the closer we get to that kind of pruning, the less disk we need and the higher-quality the experience will be.

Update 8/2/05: doing a different back-of-the-envelope estimate leads to being able to store a compressed-HTML cache (no images) on less than $1000 worth of disk within 3 years…

…and Photoshop takes them right back off.

(Thanks to Spider for the link…)

Microsoft Research has announced a Request for Proposals for projects in relating to their Digital Memories (Memex) research kit, in the context of “personal lifetime storage.” Microsoft’s inspiration (and probably the inspiration for everyone else working in this area too, at least indirectly) is Vannevar Bush’s 1945 article As We May Think, in which he famously described a kind of personal library-in-a-desk he called the memex:

Consider a future device for individual use, which is a sort of mechanized private file and library. It needs a name, and to coin one at random, “memex” will do. A memex is a device in which an individual stores all his books, records, and communications, and which is mechanized so that it may be consulted with exceeding speed and flexibility. It is an enlarged intimate supplement to his memory.

MSR expects to give 6-9 awards to college and university projects, up to a max of $50,000 per award, and recipients would also be given a SenseCam wearable camera and software from the MyLifeBits, VIBE and Phlat research projects at Microsoft Research. Strings are minimal — they expect semiannual progress reports, want it presented at at least one of their workshops and expect the project to be either dedicated to the public domain or released under an open license such as the BSD license.

Yesterday a consortium of the major movie studios announced final specs for a new standard digital format for movie theaters. The specification uses JPEG 2000 video compression, which (though it happened before I started working there) I’m proud to say largely came out of work performed at my lab.

The big advantage of JPEG 2000 is that you can “pull out” bits from a code stream to get different resolutions — in this case a 4K distribution (1,302,083 bytes per frame at 48FPS) and a 2K distribution (651,041 bytes per frame at 48 FPS) can both be generated on-the-fly from the same file, just by discarding segments of the stream.

(Thanks to Mike for the link.)

I’m a little late on this news, but last week Fujitsu announced a new bendable e-paper technology. EE Times has the most complete technical description I’ve seen on it:

The display is a passive-matrix, reflective type cholesteric liquid crystal display. Two 3.8-inch diagonal QVGA prototypes, a monochrome display and a color version able to display 512 colors, were shown.

…

Differing from widely used flat displays that have color filters consisting of red, green and blue pixels, the paper display has a three layered structure in total about 0.8 mm thick. One layer consists of two 0.125 mm-thick films sandwiching liquid crystal. Cholesteric crystals in each layer are twisted in a certain pitch to reflect only red, green or blue light respectively.

Images on the screen can be changed with 10-milliwatts to 100-milliwatts depending on scanning speed.

(Thanks to John for the link…)