News, music, movies & restaurants from the editors of the Silicon Valley's #1 weekly newspaper.
Serving San Jose, Palo Alto, Los Gatos, Campbell, Sunnyvale, Mountain View, Fremont & nearby cities.

April 4-10, 2007

home | metro silicon valley index | columns | technology news

Technology News - Annalee Newitz

Technology News

Smoking Yahoo's Pipes

By Annalee Newitz

I HAVE BEEN playing around with Yahoo!'s latest technological experiment on the web. It's called Pipes, and it's a system designed to help web-savvy people write simple programs without ever having to read a book about Java. If you visit, you can take a peek.

Visitors to the site are presented with a sheet of virtual graph paper and a list of "modules" that can be dragged onto the paper and connected with "pipes." In this early stage, the modules mostly allow users to build a really customized news feed or an online research tool.

You can tell a "source" module to pull information from, say, a Google search for "Windows Vista" or the RSS feed of your favorite newspaper. Then you pipe that information to an "operator" module, which allows you to filter it, list it by date, translate it into another language and more.

Other modules let you do more complicated things, such as annotating each piece of data with geographical information or merging the RSS feeds from several sites so that you get one big "daily news" feed instead of 20 from various progressive blogs. Just think: You could mix the latest wankery from porno news site Fleshbot with the latest wonkery from Talking Points Memo! That's the beauty of a customized news feed.

Pipes isn't for everyone—it's too complicated for casual web surfers, who may not be familiar with the inner workings of RSS feeds and search queries. But a quick Google search reveals some excellent tutorials which will aid even the most RSS-clueless person in creating a Pipe of their own. Plus you can "clone" other people's pipes—so if you want a customized news feed, you can just use one that already exists, fill in your own news sources of choice and save it to your own account. There are hundreds of cool Pipes available on the site, and they're all clonable.

One of the best uses for Pipes, in my experience, is as an educational tool. Building a pipe helps you mentally deconstruct the way web applications work together, giving you a better picture of what's going on when you read an RSS feed or search for pictures of fish on Flickr.

Now, I sound like a cheerleader for Pipes, which I'm not. In fact, I recently spent an evening making fun of Pipes with one of the creators of the RSS standard (no, it wasn't Dave Winer). Our mockery was inspired by two things: One, Pipes could be an overhyped proof-of-concept that nobody will ever use; and two, it could actually limit people's control over data.

How could a tool designed to help you manipulate all kinds of information actually limit your control? To answer that, we need to delve briefly into the origin of the "pipes" idea. The name comes from a powerful command in UNIX, one of the first operating systems, which converts the output of one function into the input for another. It's hard to convey how utterly awesome and time-saving this command was when it was invented. It meant that data could be crunched, sorted, alphabetized, merged and recombined more easily than ever before.

Yahoo Pipes aims to do the same thing, only the data you use is what's available publicly on the web. So if you want to use Pipes to organize or sort your personal data, you'll have to publish it online. This is obviously quite different from the UNIX pipe, which is so powerful in part because you can use it on private stuff like passwords and financial documents.

Yahoo Pipes treats the web as if it were the hard drive of your UNIX box; you can pipe data from Google into a sorting program or pipe The New York Times RSS feed into a filter that will remove all stories that refer to "Yahoo Pipes." It's marvelously cool, but I worry that it will inspire people to put sensitive data online just because it's more convenient to crunch via Pipes.

At this point, my fears are probably unjustified. Pipes is in beta, and it may not catch on with the general public. More likely, a user-friendly version of Pipes will come along and get widely adopted in a couple of years. It will become just one more way we're being seduced into dumping all our personal stuff online. I like the idea of turning all the data on the web into my raw materials, to do with what I please. That's the beauty part of Pipes. Still, the more data we deposit into the hive mind of the web, the less power we have over it.

Annalee Newitz ([email protected]) is a surly media nerd who still hears the voice of her UNIX teacher in her head saying, 'Now Pipe it to MORE.'

Send a letter to the editor about this story.