April 2006 Archives

The afternoon sessions started with the awarding of the first Everett Brenner Award award for the Best Contribution to Knowledge at the 2006 Search Engine Meeting. The winner was Stavros Macrakis formerly of Lycos and now with FAST and who ironically is scheduled to be the last speaker this afternoon and of the conference.

The sessions this afternoon dealt with web and intelligent tools. The first speaker was Paul Thompson of Dartmouth College. His talk "Search and Misinformation in Intelligence and Security Informatics" was quite interesting in that it deals with a relative little researched area. He said a what was needed was a new science along the lines of bioinformatics. Fraud was increasing and he cited one prominent journal which reported that at least 20% of accepted manuscripts, let alone those not accepted, contained at least one occurrence of fraud. He went into some detail of his research done over the last few years. His paper will be online Friday at the conference web site for those interested in this subject.

After the morning break we reconvened for the panel discussion, Search: The Next Decade. Participants included;

- Susan Feldman (Moderator), IDC
- Suranga Chandratillake, blinkx
- Josh Jacobs, X1
- Andrew McKay, FAST

The panel discussion lasted an hour and half. However it was not so much a panel but rather a forum for the speakers to posit their ideas on the future. They spent nearly an hour and 15 minutes on this leaving barely 15 minutes from other panel members or the audience for questions.

One recurring theme is that managing information for the average user is time consuming.

To illustrate this a slide was shown which listed a breakdown of much time we spend on different tasks. Here's the breakdown for businesses.

- We spend an average 14.5 hours a week on email which costs the company $21k a year.
- We spend an average 13.3 hours a week creating documents costing $19K a year
- We spend an average 9.9 hours a week analizing docs costing $17k a year
- We spend an average of 9.5 hours a week searching costing $14k a year

and there was much more.

Another item mentioned is that Google is not the end of search, there's lot's of innovation and growth to come.

An interesting post but I don't think so. SEO's will adapt to the shifting market as they always do.

Could vertical search supplant SEO?: "I'm at The Search Engine Meeting in Boston listening to Vivisimo's Raul Valdes-Perez promoting vertical search. In his vision, companies will aggregate their own information universes, discreetly putting their own information first. This can turn them into go-to sites for..."

(Via BusinessWeek Online -- Blogspotting.)

Well we're in a break right now and the three morning talks were all interesting, especially Stephen Arnold of AIT. Arnold is a long time information knowledge technophile and pulls no punches when discussing the industry, and that's a good thing, as we need perspectives from all angles on search. So his talk "Google: The Erosion of Relevance" is one I've been anticipating. And he did not disappoint.

He started by calling Google, Googzilla. This drew some laughter from the crowd. He pointed out that all the talks from the previous day touched upon Google in some way. We're obsessed with Google, we love Google. And since everyone loves Google we seem to be missing something. And that something is that search relevance is being eroded. And he wasn't just critical of Google. He said all the major search engines, Microsoft and Yahoo included, and everyone else is eroding relevance. So exactly what does he mean by this?

His basic premise is that content is being steered, thus its relevance is being eroded. Search engine results are being skewed as people learn how to manipulate them. This erodes relevance. He cites emerging social tools, that, while people find cool, actually are doing us all a disservice by skewing the results. Some examples include del.icio.us which he says poses big problems, as we are overlooking what happens when humans use random terms to classify links. He cited Flickr who use "word" tags which erode relevance. He also cited digg, and how it appeared to be more popular then Slashdot but has recently run into problems in that some users had figured out to skew the results so their posts ranked at the top.

What about the search engine themselves? Are they allowing relevance to be eroded? His answer is not a simple one. A part of him says yes while another says no. Are they doing it intentionnally, no. However users are learning how to manipulate the results. To me it's an ongoing game between the search engines and the users who want to rank high. He also notes that current search compnies are not focusing on the problems of search. Not everyone would agree with him on this point.

A last note about the morning sessions. Mike Moran of IBM has put his talk, "Don't just change the search engine (powerpoint)" on his blog.

The afternoon sessions turned out to be less interesing and useful to me than I had hoped except for a couple of talks.

The first talk was by Claude Vogel of Convera. Convera, formely Excalibur, has been providing enterprise class search for many year now and their products have matured well over that time. Claude spoke about speeding search using faceting, and as later speaker defined it, "a facet is a certain classifiable characteristic of the resource -- a way to classify something." To me the talk seemed more of a marketing speech for their product Excalibur which is not surprising.

Next up was Abe Lederman of Deep Web Technologies who spoke of challenges in scaling federated (metasearch) searches. He characterized the challenges in searching thousands of sources as follows:

- Determining the sources to search
- Retrieving searches from cache and how often to update the cache
- Performing many searches in parallel
- The need to bring the best documents back

And of course how to rank the results. He touched upon the methods they used including;

- Multi-user Relevance Ranking which includes QuickRank - occurence of search terms, MetaRank - custom algorithm applied to metadata, DeepRank - indexing fulltext documents
- User-driven Ranking
- Clustering

Yahoo has updated its search index which of course is relevant to any online business. So I had a quick look to see how SpaceRef did in this update. For our most important keyword/phrase we stayed the same, in the top ten results on page 1. For a couple of secondary keywords we went down a bit and up bit. In other words status quo. Overall I'm happy with this. Of course it's only recently that I've started paying more attention to this and actually working on improving our rankings. It's all about available time. Of note, our competitors remained pretty much the same as before except for one who fell out of the top 50 for what I consider the top keyword in our business, ouch.

Weather Report: Yahoo! Search Index Update:

We rolled out an index update last night. As usual, you may see some changes in ranking as well as some shuffling of the pages that are included in the index. Those who follow these weather updates may have noticed that they are occurring more frequently; this is the result of improvements to the indexing system.

After the morning break there were four more talks in Session One: Searchers and Search Behaviours. The first up was Tony Gentile of Healthline. His talk dealt with medically guided search. In his own words "Healthline connects consumer and medical vocabulary and provides medically guided search". The talk focused on this vertical search and how to make the results relevant to them. A couple of the points I found interesting were:

- users don't know what they don't know
- there's a difference between search for something and researching something
- really know your audience, almost 30% of their searches are done on behalf of someone else
- make the user feel confident in your product, if they feel confident about the results they get, they'll be back

Overall a good talk on vertical search product. Vertical search is something aTerra (my company) has been working on for some time, primarily in the biotech and space sector. The morning sessions have given me some new ideas which I'd like to develop for these and other niches.

The next talk was by Max Copperman of Knova Software. Max talked dealt with WYSISWIG search crafting. His company makes makes call center applications and call service applications. It was a more technical talk dealing with what he calls "tuning". Tuning, as in tuning all facets of your search engine.

I'm currently in Boston attending the Search Engine Meeting conference. I'm here to learn what new technologies are emerging, who's doing what, and make some new contacts.

The first speaker today was Dave Girouard of Google who spoke about Google's new OneBox. While the OneBox may shake the industry, I found Dave's talk uninspiring. He spent his 25 minutes leading up to the introduction of OneBox. Some of his key points were:

- The enterprise market is underutilized
- not enough value to end user
- employees are consumers are the same people

Update: I forgot to mention that Dave mentioned that the enterprise search market is about 1/10th smaller than the consumer market and there's a lot of room for growth and competition.