Nano2Sol by Marc Boucher

A lot has been written about cloud computing in the last year and each day seems to bring news of a new player in the cloud arena. So what does the cloud have to offer search engine companies like Hyperix? Well that depends on how deep our pockets are. After all, we need a lot of bandwidth, processing power and data storage to run any real search engine. And as we don’t have deep pockets, nor an angel or venture firm backing us we’ve had to be find creative solutions and innovate where possible.

Up to this point we’ve been focusing solely on the technology we’re using that will differentiate ourselves from any other vertical search platform entities out there. We’ve got our own small web crawling cluster setup which we’ve used for some time to test different web crawlers, collect and parse data and measure a variety web crawler values which determine how many CPU cycles, RAM, bandwidth, and storage is necessary to create the vertical search indexes we want. We’ve also been focusing on the quality of the data we’re crawling, the algorithm which ranks the pages crawled, the parsing engines, and the results pages.

(more…)

ReadWriteWeb has a good article on cloud computing today.

The first, Reaching for the Sky Through The Compute Clouds, is written the Amazon Web Services outage last Friday fresh in our minds. I’m a big proponent of cloud computing as it’s the only way in my opinion to truly scale large data driven applications such as search which is what I’m working on.

“So is it really true - is cloud computing a bad idea? Of course not. It is a wonderful, powerful idea. In this post, we explore the ideas behind cloud computing and argue that it will be an integral part of our future.”

“Do Clouds Really Work?

You bet! The best example is Google. The king of the web is reigning with a farm of hundreds of thousands, if not millions of boxes. To race along with the web, Google constantly increases the size of its cloud, incorporating new web sites, and expanding its index.

Of course, Google isn’t the only one operating in a cloud. All major web players including Amazon, eBay, Yahoo! and Facebook are running some sort of massive computing cloud.”