23 Musings

So, yesterday Bill Gates hinted in his keynote at CES that Microsoft would improve search. It's an area where they haven't really shown any improvement in recent years. I thought to myself at the time, maybe they've finally realised that it's all about the algorithm and the quality of results you deliver to users, not in the interface and fancy AJAX tools.

So I watched, and waited, and lo and behold they announce an offer to buy FAST Search and Transfer!

The offer comes to approx $1.2B which is a fair valuation. Rumours are that the shareholders have already approved this and it's all a formality and will go ahead pretty quickly.

Top move Bill! FAST is a great piece of search technology with many applications. It has it's heart rooted in providing good algorithms and tailorable search tools which is exactly where Microsoft should be pushing it's Live Search offering.

Now you just have to integrate this with Adcentre and deliver a Google beating search engine, no small task, that should make the next year very interesting!

Eye tracking company Enquiro Research has produced a report (sponsored by Google no less) which shows the importance for brands to be well represented at the top of both natural and paid search listings.

They say 'There is significant correlation between brands’ appearing in the top organic search and sponsored placements and consumer brand affinity, recall and purchase intent'. Well that's pretty obvious, but it's always nice to have these concepts validated by eye tracking (something that I place great value in as an insight and usability tool).

The most interesting findings are:

This shows that there is definite brand affinity and recall to be gained by being well positioned and using the brand name in your listings. Good advice for travel companies with January coming, I see so many travel listings that are just a destination name and no brand mention at all.

The full report can be found here.

Less is more it would seem when it comes to search interfaces. Just look at Google to see the less-is-more approach at it's best. Their minimal approach to the search homepage works really well, although I do think a big reason for that is that their algorithm is much better at returning relevant results with minimal input from the user.

Prof. John Maeda of MIT Media Lab had put together an interesting image showing the development of both Google and Yahoo's homepage over time. It's really interesting to see how Yahoo lost their way and Google stayed true to the minimal approach. Click the image below to see a full size version:

In customer satisfaction it would seem.

JD Power has completed a survey of internet users looking at their satisfaction with various online services. In use of search, Dogpile has come top for the second year running with Google coming in second.

The study, now in its fourth year, examines consumer behavior, experiences and satisfaction of Internet service subscribers with the most frequently used online services. These include search engine/functions, web portal, instant messaging, social networking, music downloading services and online console gaming systems.

The search rankings are below:
Dogpile is a meta-search engine which scrapes all the major search providers and presents the results back to the user. This provides a one-stop search experience for it's users. Another factor that helps it is the companies limit they place on advertising next to the results. Paid search adverts are integrated into the results listings and mentioned as sponsored, this isn't particularly obvious so perhaps users just miss the fact that they click on ads.

I think a lot of this is down to misconception by users. I believe they probably think they are getting a better service as Dogpile promises to search all the best engines in one go. What the users probably don't notice is that the relevance suffers as a result of this. There can be pages and pages of similar results, many links to the same websites and definitely more advanced searchers won't get the same valuable results they would from Google. Could this be a case of lower numbers of users and less sophisticated users?

Dogpile beat Google on satisfaction, functionality, ease of use and results quality in the survey. I am going to trial Dogpile for a while to see if it could ever replace Google for me, I reckon it won't but am always willing to give something different a try.

I posted the other day surmising whether Yahoo is catching up in search based on some new metrics from Compete. The general feel from that post was that they weren't really and the numbers were questionable in value.

So, what should I see this morning? New, useful features in Yahoo's main search interface, the kind of features that make search a whole lot less painful for users and add a lot of value, helping users to target searches more effectively.

The first thing I noticed was that they have integrated images from Flickr and also playable videos, kind of like Google's universal search idea and equally as useful to the user. So if you search for a music artist (eg. Unkle) you should get images and video in the results along with useful additions such as links to albums, lyrics, photos and more videos. Next I did a search for a hotel (eg. 'hudson hotel new york') in New York (being the online travel buff that I am) and the top result in this case was a Yahoo Local listing for the hotel complete with map links, again extremely useful (although possibly doing themselves out of a small amount of ad revenue here). Another cool feature is the inclusion of custom results for searches such as health related (eg. 'lyme disease'), quick easy access to relevant info is the killer in search and here Yahoo have it spot on.

And then there's the biggest and most useful addition, and it's an addition that Google haven't yet implemented... It's an AJAX based search assistant panel that appears if it senses you hesitate while typing a search query. It works as an auto-complete assistant and also a guided search tool as it will both try to guess what you were typing and give you suggestions as well. This is extremely powerful and really makes search easier for the user. It also adds a lot of value to advertisers as it should mean more qualified clicks on paid search results as the searches are better informed.

I'm actually really surprised this has come out of Yahoo first, I've been waiting for Google to implement something like this for a while now but for once Yahoo have the upper hand. Now all they need to do is sort out the problems with their paid search results (relevancy, gaming etc), fix their algorithms for natural search and improve the interface and they could become my search engine of choice!

Okay, so Google are still miles ahead on market share in the search world. Below is the latest graph from Compete.com showing just how commanding a lead they have.
However, Compete have some interesting insight into quality rather than reach. Quality in search is rarely discussed and whenever it has been Google has always been assumed to be in the lead there as well due to their massive research and development capabilities.

That may be the wrong assumption though!

Compete have looked at a metric they call search fulfillment. They came up with this because even though there are many searches taking place on the major engines, not all result in a click on a result and a referral. In fact, according to their data out of approx 7.5 billion monthly searches only 5 billion result in a referral.

So, if Google looks like the leader from a search volume point of view, how about from a fulfillment point of view. The graph below shows an interesting picture...
So Google is not getting the referrals the volume suggests it should.

Yahoo seems to do really well from this which I find strange as the relevancy of results in Yahoo never seems as good to me as Google. Of course there are many other possible reasons for this, a couple being that the figures may exclude clicks on paid links and that Google returns much more useful snippets in the results than any other engine often negating the need to click through.

Of course Google's figures could be vastly inflated by all the agency types, SEO's and webmasters out there who perform daily searches to check their sites rankings without ever clicking on anything. It would be really interesting to understand those volumes!

Japanese authorities have finally recognised the fact that their dominance in all things device and hardware is slipping and that they have missed a trick by not moving into the service side of things such as search.

The Financial Times carries a story about a new initiative to move into researching search and ways to personalise devices such as in-car navigation using search as the building block.

I'd love to think that a state funded program could compete with Google (GOOG), but I fear it will take all of the Japanese inventiveness to come up with anything that comes close. I'd say they'd be better off investing the money into research in new network protocols and ways to usher in a new era of web connectivity. That is after all something the Japanese have always been very very good at.

...search, advertising and everything else in between!

During a keynote speech at the Search Engine Strategies conference in California this week, Marissa Mayer, VP of Search Products and User Experience at Google talked about personalisation as key to Googles future strategy for search and most other avenues. She said that within 10-15 years time search sites will understand much more about their users such as location and personal preferences.

She also said it's important that the ads are personalised too. Now that could have an impact for Adwords advertisers in the future, some advertisers won't want to limit their coverage by being restricted by individual searchers histories and preferences. Although other advertisers will be getting excited by the prospects of better targeting.

Interestingly she mentioned that Google is looking at changing the presentation of its universal search page "to guide users' eyes" so they can see the results and the advertisements, her philosophy being that the ads and the search results should match.

She also said that the way Facebook aggregates data about relationships between people, including when they met and how they know each other is interesting. She said "The type of information they're building about the social graph between people is something that is intelligent and will be particularly useful in the future". It certainly looks like Facebook have noticed this...

For the second consecutive year, Google has slipped in the rankings of a US consumer satisfaction survey. This year they've slipped so far they actually come below Yahoo!

According to the American Consumer Satisfaction Index, Yahoo's customer satisfaction score rose by 4% to 79, while Google fell by 3.7% to 78. Other search engines measured in the rankings were Ask.com, which posted the biggest increase to score 75, making it equal with MSN.com. AOL dropped by 9% to record a satisfaction score of 67, making it the biggest loser in the survey.

Foresee Results who run the survey said that it is an indicator of the financial success of a business in the wider marketplace and reckon that the next year could show a turnaround for Yahoo. Music to Jerry Yang's ears I'm sure, they could do with some new found success!

In a rare move, CNet News has managed to get frank explanations of privacy policies from the major search engines AOL, Ask.com, Google, Yahoo and Microsoft.

CNet sent the engines a survey and the responses are published here.

A new search engine has launched to not much fanfare and pretty stealthily called iRazoo. It's interesting to me as it's doing a similar thing to Mahalo in that it's using the input of users to qualify search results.

Rather than go down the Mahalo route where specialists are paid to put together search results, iRazoo is allowing users to recommend a search result while they are surfing. When users click a search result it's opened in a new window, this could annoy a lot of users as pop-ups (even user activated) are generally a no-no these days. That's purely a user interface issue though and I'm sure they will change it if they get enough feedback requesting it. There's a bar across the top of the new window asking whether you recommend the site and allowing comments to be left as well.

As an incentive users are able to earn points for their recommendations. However you need many thousands of points to get any meaningful prizes (camera or mp3 player) and so it could take quite some time.

It's an interesting concept and one that could work quite well if it was designed with the user in mind. At the moment it's a fairly clunky and unwieldy process with the pop-up windows and the bar across the resulting websites. A few small changes could fix that though and a bit of usability testing, and they may need to do this to gain traction with users.

Of course, it is possible to game the site a little by recommending your own websites, but I'm sure (or at least hope) they will have thought of that. If they can gain the much needed traffic and signed up members then it should even itself out anyway.

The idea of the 'crowd' providing intelligence to search results is a great one and in my opinion beats the Mahalo idea of specialists. Crowd wisdom is a far better way to measure the value of something and as such this could have potential with a little better execution.

So Compete is saying that Microsoft Live search has grown massively in the last month in the number of searches and users it's accommodating. They have increased the volume of queries by 67% from May to June and by 48% from the same time last year. Good going! Pretty impressive growth rates I hear you cry.

I thought that until I dug a little deeper and discovered via Marketing Charts that Live search has been running a promotion where users play games in return for tickets they can collect towards free gifts. All the games played involve the use of the Live.com search engine so it's easy to see how this growth has occurred.

The prizes are worth having too, and with reports of users running bots against the games to speed up the winning of prizes it does make me wonder if this is a last gasp effort by Microsoft to get some revenue from search marketing (both for themselves and for advertisers) to try to impress.

I can't see Google or even Yahoo quaking in their boots about the growth figures now...

Human powered search engines are all the rage with the launch of Mahalo one of the most talked about launches this year. However it turns out that the South Koreans have been doing this right for some time!

Naver a South Korean based search engine is storming along and has claimed to be handling 77% of all South Korean web searches. What really puts that figure into perspective is the fact that Google only handle 1.7%. Now of course the reason Google is so low is mainly to do with the lack of Korean language results, but there is another reason Naver does so well.

Relevancy and accuracy. Two things that Google desperately tries to do but can never guarantee as it's machine led intelligence driving the thing.

Naver is human lead. It works in a similar way to Yahoo Answers where humans answer your search queries. This must bode well for Mahalo as their business model is very similar.

Great article in the New York Times on Naver today here.

Google's Webmaster Central blog has an interesting post detailing their position with regards to the practice of creating startpages. Startpages are webpages with a lot of links about a specific topic. The startpages are hosted on a startpage domain and each separate startpage is maintained by an individual webmaster. The links on startpages are usually ordered by categories related to the topic of the page.

Great! They're useful starting points on the web containing a load of links and content relevant to what you are looking for.

But, can this practice be misused to spam search engines? Yes, of course it can. Create yourself a load of startpages with links into your own websites and services, make them keyword rich and highly optimised for search engine crawlers. There you go, a load of doorway pages which don't sit on your own domain and therefore don't get looked on as SEO spam.

A winning formula for unethical SEO's everywhere. This practice already exists and I'm amazed that Google hasn't taken the opportunity to frown upon this practice in this article on their blog! They do mention that link farms are against their guidelines, but no mention of whether they'll be looking to filter them out or not.

Most savvy web folk will know better than to try that, but there are bound to be some who will see Google's endorsement of startpages as a sign that they can try some less ethical practices to gain traffic and SEO link equity.

Madness I tell you!

Google has had a defamation action raised against it by a London based businessman who owns a domain registrar. This landmark legal action seeks to hold Google liable for publishing inaccurate, malicious or damaging material on the web.

It's the first case of it's kind in the UK that seeks to make a search engine responsible for the content on the web. If successful it could trigger massive restrictions on the freedom of information online.

The case states that the search engine directed users to web pages that the domain registrar claims contain "deeply offensive and commercially damaging" material about their business.

Various postings on forums etc accuse the registrar dotWorlds of cashing in on the September 11th attacks by offering free registration of domain names to U.S. businesses and users in a way that took advantage of the patriotism at the time. In other postings, they are accused of conducting fraudulent business.

Since 2003, the owner of the registrar claims that Google has removed various posts to its own discussion groups at his request, but its search engine continues to turn up links to offensive third-party sites. If Google doesn't give him a written promise to permanently remove the links, he plans to sue them.

Google would be protected by the first amendment if the action was in the U.S. but in the UK it's not yet been seen if internet companies have the same protection.

If the the suit is successful, it could have widespread consequences - not just for search engines but for all sorts of other internet service providers that link to third-party content.

Crazy! I can understand Google having to remove content on it's own groups etc but to have to police the content of web pages that they have control over is madness. Imagine the filtering they'd have to put in place and after everyone else with a libel issue filed suit it would become an unmanageable task.

I sincerely hope this isn't successful. The domain registrar in question should seek to prove the claims as inaccurate and use that to their advantage to quash all libellous talk rather than trying to ruin the freedom that the internet gives us all.

Google has retained its status as the top search engine in the US, logging more than half of all search queries last month.

comScore recorded 7.6 billion search queries in May, 3.9 billion of which were made from Google, giving the search engine a 50.7% market share.

That's a one per cent increase from April, when Google claimed a 49.7% market share, and an increase of more than three per cent since the start of 2007.

Google was also the only search engine among the top five to gain market share last month. Yahoo saw its market share drop from 26.8% to 26.4%. Microsoft remained static at 10.3%

Okay, take a look at the posts below and you'll see I haven't gone a day without... Altsearchengines, a blog aiming to encourage users to step outside the top 5 engines comfort zone is encouraging users not to use the big 5 today.

Their rules:

1. All day Tuesday, June 12th, don’t use any of the 5 major search engines.

2. Avoid Meta search engines, since most of them include the major search engines. (for this day only! Meta search engines are important; see the Great Debate Tuesday night!)

3. Likewise, the specialized vertical search engines may be too narrowly focused. (for this day only. It’s the vertical search engines that usually search the best; within their niche.)

4. Consider changing your homepage or downloading their toolbar. You can always uninstall everything and change back on Wednesday.

5. On Wednesday, leave a detailed comment under this post and share your experience with the rest of us. Which alt search engine did you chose? How would you rate the experience? (if you want to comment go here)

So if you're looking for an alternative here's the top 100:

50matches.com [HM] www.50matches.com
Accoona www.accoona.com
AfterVote {SEM} www.aftervote.com
Agent 55 www.agent55.com
Allth.at www.allth.at
Answers.com www.answers.com
Audiobaba www.audiobaba.com
Blabline www.blabline.com
blinkx www.blinkx.com
Blogdigger [HM] www.blogdigger.com
Bookmach www.bookmach.com
btbot www.btbot.com
CatchTomorrow www.catchtomorrow.com
ChaCha {#1 2006} www.chacha.com
ClipBlast! www.clipblast.com
Cognitionsearch www.cognitionsearch.com
Collarity www.collarity.com
Compete www.compete.com
Congoo www.congoo.com
CrossEngine www.crossengine.com
Cydral en.cydral.com
Decipho www.decipho.com
Deligio [HM] www.deligio.com
eTools.ch www.etools.ch
Exalead www.exalead.com
Factbites www.factbites.com
Faroo faroo.com
FeedMiner www.feedminer.com
Feedster ww.feedster.com
FindSounds [HM] www.findsounds.com
Fisssh! www.fisssh.com
FyberSearch www.fybersearch.com
GameSkoot www.gameskoot.com
GenieKnows (Games) www.genieknows.com
Gigablast www.gigablast.com
GoPubMed www.gopubmed.com
GoshMe {SEM} www.goshme.com
gravee www.gravee.com
Grokker www.grokker.com
Hakia www.hakia.com
Healthline www.healthline.com
iBoogie www.iboogie.com
Icerocket www.icerocket.com
indeed www.indeed.com
ixquick [HM] www.ixquick.com
KartOO *SEM* (tie) www.kartoo.com
Knuru www.knuru.com
KoolTorch {SEM} www.kooltorch.com
Kosmix [HM] www.kosmix.com
Krugle www.krugle.com
Ktorrents www.ktorrents.com
lijit [HM] www.lijit.com
Like www.like.com
LivePlasma www.liveplasma.com
Mojeek www.mojeek.com
MP3Realm mp3realm.org
Ms. Freckles www.msfreckles.com/?lang=en
Nayio www.nayio.com
nnseek www.nnseek.com
Nutshell www.nutshell.com
Omgili www.omgili.com
Pagebull www.pagebull.com
Picsearch www.picsearch.com
Pipl pipl.com
Pixsy www.pixsy.com
Pluggd www.pluggd.com
Podnova www.podnova.com
Podzinger www.podzinger.com
Purevideo www.purevideo.com
Quintura *SEM* (tie) www.quintura.com
Quintura Kids kids.quintura.com
Revolutionhealth www.revolutionhealth.com
Searchbots www.searchbots.net
SearchKindly www.searchkindly.com
Searchles www.searchles.com
SearchTheWeb2 www.searchtheweb2.com
SeeIt [HM] www.seeit.com
Serph www.serph.com
Sidekiq www.sidekiq.com
Simply Google www.simplygoogle.com
Simplyhired www.simplyhired.com
Slifter www.slifter.com
Sphere www.sphere.com
Sproose [HM] www.sproose.com
Srchr www.srchr.com
Sugarcode www.sugarcode.com
Surfwax www.surfwax.com
Swamii www.swamii.com
TheFind.com www.thefind.com
Trexy www.trexy.com
Turboscout www.turboscout.com
Twerq www.twerq.com
ViewFour www.viewfour.com
WasaLive en.wasalive.com
Wazap! www.wazap.com
Whonu? [HM] www.whonu.com
WiseNut www.wisenut.com
Wize wize.com
Yoople! www.yoople.com
Zuula www.zuula.com

In a concession to the EU commission over data protection, Google has announced that they will only hold customer data for 18 months, after that period the server logs will become anonymous. The policy shift was flagged up in a letter sent to the Article 29 Data Protection Working Party in Brussels on Sunday by Peter Fleischer, Google's global privacy counsel.

It's a quick response that won't hurt Googles ability to provide innovative tools and services and hopefully will help to put the privacy issue to bed for a while.

Apparently search engines are getting better at keeping users away from unsafe websites. Red sites failed the test by McAfee where as yellow should be warning users about the slightly dubious practices they employ.