Participatory Culture Foundation has released DTV, an open source (GPL) video player that combines an RSS aggregator with a BitTorrent video player. Currently in beta for OSX, Windows and Linux coming soon. Combine this with their open source Broadcast Machine to create your own channels, or use del.icio.us to add videos you find on the web to your own published channel.

(Props to Noel for the link.)

Chris Schmandt, the head of MIT Media Lab’s Speech Interface Group, has just made a PDF of his now out-of-print 1994 book Voice Communications With Computers: Conversational Systems available for download in PDF form for free off his website.

Chris was one of the readers for my Generals Exams, and naturally this was one of the books on my reading list. It’s 12 years old at this point, but most of the issues he talks about are inherent in speech communications regardless of the technology. Highly recommended.

(Thanks to Thad for the link!)

Another back-of-the-envelope calculation, inspired by a comment by my friend Beemer:

Atoms in the Universe: 10⁷⁹
Bits on this year’s largest hard drive: 500GB = 4 x 10¹² bits
Doubling time for hard drives: 1 year

Years before a single hard drive will store 1 bit for every atom in the Universe at current doubling rates: 222

Warning: past performance is not necessarily indicative of future results.

I just read a great review of Mac OS X 10.4 (Tiger) over at ars technica. If you’re into Macs or finer points of geekery like how file systems should work, especially check out the long discussion on metadata in Tiger.

Yesterday I said that within a decade disk space should be cheap enough to put the entire visible web on your desk for under $1000. I think that’s actually a pretty conservative estimate, since it assumes a 100 KB average page size, up to an order of magnitude higher than some estimates.

Here’s another back-of-the envelope: let’s say we wanted the equivalent of Google’s webcache on your desktop (that is, all the HTML but no images). Another way to calculate it starts with the fact that the 2003 update to Berkeley’s How Much Info? study estimated that in 2002 the web was only 167 Terabytes total, with only 30 TB as HTML (69 TB when you include images). Assuming 75% compression, that’s just around 8 TB. That same year a 2002 OCLC study calculated that the total number of web pages was only increasing by about 5% per year (with the number of sites actually shrinking, but the number of pages per site growing). That rate had been decreasing ever since the explosion in the mid ’90s, but let’s assume growth became a steady 5% and will stay at that rate for the next few years. (There are a lot of assumptions going on here, but the nice thing about these kinds of curves is that even if my numbers are off by a factor of two somewhere, so long as disk keeps increasing at the same rate that crossover point only changes by one year.)

Now we’ve got two trends, and just need to find the intersection point for the price we want:

So given a few assumptions, we’ll be able to cache all the raw text on the public web for under $1000 (disk cost) within 3 years!

Some time ago I asked how much longer before I can have the Web in my pocket. Let’s try a quick back-of-the-envelope calculation:

A paper from January 2005 calculates the publicly indexable Web (the part easily accessible to search engine web-crawlers) as being around 11.5 billion pages. Estimates on average webpage size seem to be all over the map, but let’s figure around 100 KB per page, for a total of around a petabyte (one million Gig) for today’s indexed web. (I’m assuming text and images, but ignoring other media.)

Disk these days is going for less than 50 cents per Gig, so enough disk to store your own personal Google (and then some) costs around $500,000. With compression you can probably cut that in half. The price of disk is also falling by a factor of two every 12 months, so assuming no major jumps or snags in the disk-price curve, in a little less than a decade we can expect to hold the equivalent of today’s indexed web for less than $1000.

Now of course, in that time the web will continue to grow, so we may no longer be satisfied with our measly petabyte-on-the-desk, but I figure the amount of human-generated Web content has a much slower growth rate than our disk-space curve. The number of web sites actually shrank between 2001 and 2002, and though it now seems to be growing again there’s only so much content that human beings can create in a day. The real question I have is whether in a decade anyone will see having access to the whole web as being all that interesting — I could easily see the majority of people losing interest in the surface web in favor of personal deep-web niches. The only reason I want the whole web in my pocket is because it’s too hard for me to filter out in advance the 99.99% of the web that’ll never be of interest to me — the closer we get to that kind of pruning, the less disk we need and the higher-quality the experience will be.

Update 8/2/05: doing a different back-of-the-envelope estimate leads to being able to store a compressed-HTML cache (no images) on less than $1000 worth of disk within 3 years…