Will Algorithms Make Human Editors Obselete? Not If Journalists Collaborate

by Scott Karp October 9th, 2008

That's the brilliance of Google - it's actually driven by human judgment, by the judgment that someone producing a website makes every time they link to something. Rather than replacing human judgment, Google is actually co-opting it. But Google isn't co-opting the judgment of most journalists and news orgs - because so many of them still don't link to anything. (Notable exceptions notwithstanding.)

I love the title of this post. This is where I got it:

http://www.scribemedia.org/2008/08/13/is-bigger-better-not-in-a-digital-...

Big companies talk the talk when it comes to adapting to the digital age. But do they walk the talk? It's difficult to reconcile their desire to be nimbler - and more responsive to their customers - with the fact that at big companies budgets are generally scheduled a year in advance.

What's more, when consumers are increasingly driving the dialogue online about products and services it may be problematic for big companies to be part of the conversation when they deploy a top-down strategy. Athur Ceria, founder and chief creative officer of CreativeFeed Network, says these sorts of business practices are fast becoming antiquated and may be deterring large enterprises from capitalizing on digital technologies.

Ceria has blogged about the need for big companies to think small if they want success in the digital world. He stresses that, for many large companies, process has started to infringe on creativity.

Big companies grappling with the Internet need to embrace a "sense of discovery," said Ceria, who has worked with Cisco, Intel and Yahoo, among other major brands. A sense of mission and a sense of awareness are also crucial if big companies want to take advantage of the Web. I recently spoke with Ceria about why bigger is not better in the digital age - and how large enterprises need to change if they want to stay in the game.

I also chatted with Ceria about trends in Web design. Ceria, who has an MFA from Yale University with a focus on graphic design, says far too many companies still treat their Web sites like a "brochure," rather than a living, breathing "organism" that is part of the company's DNA.

Enjoy.

I would speculate that there's so much information out there that no other organizational tool can manage the volumes of content that people online "consume." Search has become an essential utility for almost all internet users.

Pew: Daily Search Usage Approaching Email Levels

According to a new report from The Pew Internet & American Life Project, daily use of search engines is growing and starting to approach email usage levels. New research conducted by telephone among 2,251 US adults, age 18 and older, found that 49 percent of internet users use a search engine on a typical day, compared with 60 percent for email.

In 2002, Pew's data showed that about 30 percent of people online used search daily.

What about sites like www.chacha.com ? Interesting that he didn't include these in this article.

A post on TechCrunch yesterday suggested ChaCha was cutting the pay rate of its human guides to save costs as a prelude to "implosion." When I had last spoken to ChaCha the company had presented a very different picture so I decided to investigate and contacted co-founder Brad Bostic.

Some of the stats below are impressive... (CLICK HERE TO READ THE ENTIRE POST).

Here's a range of information that www.ChaCha.com provided to me:

* Millions of queries per month

* 90% month on month query growth from Jan-June

* Over 10,000 new users per day

* Over 25,000 guides in the system Advertising:

* Early advertising trials showed 5.2% conversion rates in mobile

* ChaCha also said that Nielsen Mobile found that Google SMS has approximately 37 million queries per month. This was achieved over a period of several years

* ChaCha's "mobile answers" service launched in January this year and the company said it expects to hit 30 million queries by December, this year User profile:

* 53% are repeat users

* 83% say they consider mobile ChaCha to be "very valuable"

* Average usage now more than 30 times per month. Compare average mobile search volume is nine times per month

* 88% hear about the service from a friend

* Users are 18-34; use it socially and for utilitarian purposes

Drilling deeper: Niche 'vertical sites' refine Web searching

Two years after Merriam-Webster formally recognized "google" as a transitive verb, consumers are recognizing the limitations of Google, Yahoo, Microsoft or Ask.com, whose "horizontal" searches deliver vast breadth of information but not much depth. Increasingly, people are "drilling down" into highly detailed and structured sites like Retrevo for consumer electronics; Trulia and Zillow for real estate; SimplyHired and Indeed for jobs; Kayak, Farechase and Farecast for travel, and so on... more

You Get What You Pay For With Online Q & A Sites, Study Shows

ScienceDaily (Apr. 11, 2008) - A new study by University of Minnesota computer science and engineering researchers revealed that the answer quality provided by online question-and-answer Web sites, such as Yahoo! Answers and Google Answers, depends on two factors--how much you pay and how many people contribute to your answer.

The University of Minnesota study posed 126 questions across a variety of Q&A sites and found that paying $10 or more to get your question answered at the now-closed Google Answers site yielded the highest-quality answers as judged by a panel of evaluators. These answers were long and detailed, with many links to source material.

Surprisingly the Web site Yahoo! Answers, which provides answers for free, performed as well as Google Answers when the fee was low ($3) and outperformed reference librarians and an "ask-an-expert" site. Researchers attributed this success to the large online community that contributes to that site's answers.

University of Minnesota researchers involved in the study include computer science and engineering professor Joseph Konstan and graduate student Max Harper. Their study examined a variety of questions eliciting facts, opinions and advice on topics spanning entertainment, technology and business. Examples of some of the questions posted are:

• Which actress has the first female line in a talking movie" I found on Wikipedia that Al Jolson had the male line, but I can't find any record of which female had the first line.
• What is the best technique for making BBQ spare ribs" I'll cook it if you can find me a recipe that's really worth the time and effort.

Of the findings, Konstan said, "Solutions that simply direct questions to a single individual don't achieve results as well as those that open the question and answers to a larger community."

The results of the study are reported in the academic research paper titled "Predictors of Answer Quality in Online Q&A Sites." The paper was published in the Association for Computing Machinery's 2008 Conference on Human Factors in Computing Systems (CHI 2008) proceedings, and presented at the conference in Florence, Italy, April 8.

Dec. 18, 2006 at 11:13pm Eastern by Bill Slawski

Why Do People Google Google? Understanding User Data to Measure Searcher Intent

What are people looking for when they type "Google" into Google? What do they want to see when they use "eBay" as a query? How does a Google or Yahoo learn from their log files, and other user information? What does it tell them about user intent?

In an interview posted earlier today with Luke Wroblewski, Yahoo's Principal Designer for Social Media, we're told that the amount of user data that Yahoo has to work with while designing may be almost overwhelming. A point we haven't seen made much by the search engines, it's the second time today I've heard it. The first was during a presentation from Google senior research scientist Dan Russell.

Last Tuesday, Keri Morgret went to the December BayCHI, where Dan Russell spoke about How People Use Search Engines. She reports on some of her thoughts as well as some interesting observations about eye-tracking studies at Google. In her post, she includes a link to a video of an earlier version of the speech, How do Google searchers behave? Improving search by divining intent. The questions I started this post with are ones he asks in his presentation. He's one of the guys who tries to find answers to questions like that.

Dan Russell joined Google from the IBM Almaden Research Center, and also worked for Xerox PARC and Apple's Advanced Technology Group, as well as teaching at both Stanford and Santa Clara Universities. His homepage includes three sets of slides used during different versions of this presentation. The video weighs in at almost an hour-and-a-half, so if you want the shortened version, or want to read along, you may want to download one of those big PDF files.

The presentation is long, but it provides some nice glimpses into how Google works. I tried to find out when Dan Russell started working at Google when he give the presentation at Stanford linked to above, but it wasn't easy to locate a date. It wasn't long enough for him to not discuss some of the insights he had when he first started looking at searcher intent while at Google. His thoughts:

1. Intuitions are terrible when trying to figure out what people are searching for.

2. In particular, your intuitions are terrible.

3. That's why Google does studies.

4. Fallacy 1: "I do it this way" so others do, too.

5. Fallacy 2 "My Mom does it this way" so others do, too.

6. Deep truth: You are statistically insignificant.

7. Deeper truth: (As a computer scientist/student/audience member) you are a couple of sigma from the norm

8. So are your friends

Understanding User Behavior and Types of Queries

While Google focuses upon search as its core mission, we're told that the effort is of little use if we "can't understand what the question is." What are users looking for? That's true not only for organic search, but also for things like images, Google Earth, print, and video, which don't have the benefit of pagerank and link structure to index. Most of these services are in beta, and they can get it wrong now. Applications can be changed on a very quick basis, or as Dan Russell calls it, rapid prototyping on a delivery model. Especially if they can look at the usage data of those services and somehow understand it.

An example is Google Video, which at first provided the most popular results, with snippets. But it wasn't getting the clickthroughs that they expected. So they quickly changed it to a richer display and the clickthroughs increased dramatically.

Another example was Google Maps - A couple of months after starting, the links were on the right side. After looking at log files, they decided to move them to the left, use a larger font, and add a tab for more details. They found that user behavior is influenced by small changes - and saw significantly differences in clickthrough rates for things like minor size changes in fonts. We're told that measuring millions of clickthroughs provides interesting results.

Another issue arises when making changes to something like Maps Local. Those changes need to be echoed in places like Maps Local for Mobile. Another challenge that they face is training a culture to use a new interface - smaller changes are easier to get people used to using them.

It's not unusual to see queries broken down into three different types, as described by Andrei Broder in his paper, A taxonomy of web search. We get to see some percentages, and a greater breakdown of query types in this presentation describing what people are searching for:

Navigational - 15 %

Transactional - 22%
- Obtain 8%
- Interact 6%
- Entertain 4%
- Download 4%

Informational - 63%
- List 3%
- Locate 24%
- Advice 2%
- Undirected 31%
- Directed 3%

Search Patterns

What patterns might emerge when people search?

We're told that it is usually a two step process:

1. Searchers find a good site, and;
2. Look for information there.

Another strategy is teleporting, or going directly to somewhere else.

The reasons for teleporting:

- Users don't realize they can search directly for the information
- Difficulties in formulating a query
- The user trusts the source that they are going to

Presumably, that two step process can be a good strategy if you know something about the resource, use its search engine if it has one, and understand the structure of that site. A video of a user session is shown at this point, illustrating someone exploring a site search, snippets, and the possibility of refining their query back at the search engine.

Information about Query Sessions

Instead of just looking at individual searches, considering user sessions are an important part of the analysis of searches.

How often do people do query reformation during a user session, and what do they do when they reform those queries?

1. Spell correct helps lots of people, and shortens their sessions.
2. People often make a minor change to a word, or add a word, which may not provide the best results (people often get stuck in inefficient queries, and don't change those much).

Some other things that they see:

1. The more words used in a query, the longer sessions tend to last. Their assumption here is that more sophisticated queries involve people spending more time searching.

2. People use longer sessions on weekends.

3. The longer the sessions, the more often they see multi-tasking (multiple search subjects in a session) and interruptions.

Advanced Searchers

Advanced searchers make up an extremely tiny fraction of the folks who search. Some of the characteristics of an advanced searcher are that they:

Have lots of meta-knowledge about content and sites
Take notes (on the machine, or paper, or bookmarks)
Try alternative word sequences
Use quotations correctly

While they may take advantage of these things, they don't do a lot of it.

The Challenges of Analysis of User Data

I started this post mentioning the problem of having an almost overwhelming amount of user data. Dan Russell shifts gears at this point of the presentation, and starts talking about how to analyze and reduce that data to manageable levels, so that instead of relying upon intuition, they are making meaningful use of the information.

There are two parts to meeting that challenge. The first is building a scalable data analysis system, where a portion of the data can be looked at, and parallel systems can be used to analyze the rest of the records. That type of analysis is described in a Google paper - Interpreting the Data: Parallel Analysis with Sawzall.

The second part of the analysis involves usability. The importance of field studies and lab-based usability tests of prototypes is covered in detail.

For instance, a user would perform a realistic task with a prototype, while thinking aloud. Researchers would watch to see:

- Where they have problems.

- Fail to complete a task, or take too long.

- Make an important mistake and don't realize it.

- Misunderstand an important part of the UI.

Efforts are made to avoid helping or influencing the user, and focusing upon their actions rather than their opinions. Eyetracking is often used in this type of testing.

Field Studies consisting of interviews at the places where people actually use their computers are conducted, as well as diary and ethnographic studies.

We're told that a large percentage of people who use search engines have very different mental images of how search engines work than people who work on search engines. An analogy used - someone opens the hood of your car, and points out a part, and asks you what it does. How likely are you to know?

The presentation describes a number of the issues they see when conducting field studies, and how they try to act upon them. The bigger issue here is how to take these types of studies and perform them in a manner which might be statistically significant. How useful might it be to get a greater sense of demographics involving different user segments and different cultures?

Some Questions and Conclusions

Some good questions that aren't necessarily answered from this presentation:

Is one click good? Better than two clicks?

Is no click better than one click - such as when the answer is provided in the snippet?

What's the best way to help searchers avoid distractions?

Should some diversity be mixed into results for breadth?

Why are SERPs so boring? Or are they?

How did a standard evolve across the major search engines in how search results pages look?

We're told that Google's focus in understanding how well they are doing in meeting searchers' intentions has transformed from a static IR-styled analysis of query results towards longer-run, session analysis of how users interact with the search engine. This approach involves incorporating data from many kinds of studies, and using many different approaches instead of looking at a single point of data.

They don't want to make decisions without lots of testing, they don't want to rely upon intuition, and upon a world view centered around Silicon Valley and Stanford.

Mosio the mobile questions and answers service I wrote about last year has a really cool and useful new feature today called Twitter Answers. Mosio users simply need to befriend the Mosio Twitter bot, and ask it any questions using Twitter's direct messaging feature. Other users who have befriended the bot will get your question (syndicated from the bot), and up to four of them can directly reply to you.

While the entire thread is somewhat meaningless with Twitter's lack of message threading, hopping over to Mosio you can see the entire exchange in its correct order. Better yet, if any regular Mosio members are able to answer your question, those answers will be shared over to Twitter.

For Mosio users who want to avoid using their phone's keypad, there is some reprieve. Last month Mosio partnered with Jott to set up a voice-to-text system that lets people ask questions using a standard phone call. Jott will then convert their voice question over to text and post it to the service, while sending any replies back in the form of SMS messages. Other users are able to follow along on Mosio's main page, their phone, and now on Twitter.

I still think one of the best uses of Mosio is to help sort out bar bets or random questions while out and about. Mobile users with a data plan can turn to Google or some other search engine to find out what they're looking for, but services like Mosio, Fluther, and Yahoo Answers provide a human touch in many areas that search engines cannot.

On a side note, Mosio was at SF Beta last night showing this off, and they had by far the most inventive Web swag I've seen in a long time. Witness, the adhesive mustache. How can you not remember this?

Information Searches That Solve Problems http://www.pewinternet.org/PPF/r/231/report_display.asp

12/30/2007 | Leigh Estabrook Evans Witt Lee Rainie

There are several major findings in this report. One is this: For help with a variety of common problems, more people turn to the internet than consult experts or family members to provide information and resources.

Another key insight is that members of Gen Y are the leading users of libraries for help solving problems and in more general patronage.

In a national phone survey, respondents were asked whether they had encountered 10 possible problems in the previous two years, all of which had a potential connection to the government or government-provided information. Those who had dealt with the problems were asked where they went for help and the internet topped the list:

• Search Engine Roundtable Forums
• WebmasterWorld
• Cre8asite Forums
• DigitalPoint Forums
• Sphinn
• Search Engine Forums
• High Rankings Forums
• V7N Forums
• I Help You Forums
• SitePoint Forums
• SEO Refugee Forums

In a day and age where services like Mahalo and Wikipedia are succeeding because of the human component of question answering (and some or all of those human components are receiving some sort of compensation), for Google to flout that makes for somewhat of a head-scratcher. Mashable

Google Answers to Relaunch as Google Q&A

Google intends to relaunch the question-answering service Google Answers, which was closed last year. In Google Answers, "users could post a question (...) and specify how much they were willing to pay for an answer. A researcher then searched for the information they wanted and posted it to Google Answers." Some of the former Google Answers researchers built a similar service at Uclue.com.

Google Q&A, code-named Confucius, no longer has paid experts and works in a similar way with Yahoo Answers. Google Q&A was launched in Russia in June and in China, two months later.

Here's a message from Google's translation console:

"Q&A - This message is a name of successor for Google Answers. We will use it in OneGoogle toolbar, which you see on top of google.com page in the more.. section. Also, please use full name to translate it. That is, Questions and Answers. Abbreviation should be used only for English. URL showing this message: toolbar on top of http://www.google.com."


It's interesting to note that Google Q&A is also the name of a feature that displays answers to simple questions at the top of Google's search results page. Maybe Google will combine the facts automatically extracted from web pages with the explicit answers from the new service.

Google Seeks To Turn College Students Into Local SEMs

You've got to hand it to Google for creativity. It created the local business referral representative program to enlist stay-at-home moms, college students and others to get better data about local businesses and build AdWords awareness. (I was told by Google that the response to the program had been strong.) Now Google has developed the "Google Online Marketing Challenge." According to the site, "student groups will receive US$200 of free online advertising and then work with local businesses to devise effective online marketing campaigns."

As a practical matter it seeks to create a global pool of college students (or MBA students) and turn them temporarily into search engine marketers for local businesses. It will be interesting to see how big the response is and whether it helps build Google AdWords awareness among local businesses in those markets where the competition takes off. The competition begins in February.

The program will potentially also create a great group of local marketing case studies that Google can then use in myriad ways for education or sales or training.

Google also said that an even simpler version of AdWords is coming next year: SimpleAds. Aimed at local businesses and resellers, it will seek to radically simplify AdWords adoption.

"From my office, what do I see?" Answers.com

From my office, what do I see? For starters, I am grateful just watching an intense, creative and colorful team radiating energy at our Answers.com headquarters. With our two main products constantly growing, it's a lot to be proud of.

The world of information has evolved from telephones to radios to televisions to the internet… Information has never been more available or expansive. But we find ourselves wading through facts and figures and volumes of webpages overloaded with content… It can be overwhelming, and search engines have certainly come to the rescue for narrowing the web. But if we want to narrow it further - enough to get that one answer we're seeking in any given search - then hasn't the time come for an answer machine?

Our goal at Answers Corp is build the web's ultimate answer engine. The typical search engine is great for getting back thousands of results for deep search. When you want fast info and quick answers to questions, wouldn't it serve you better to visit an answesengine? That's what we are building with our two main products, Answers.com and WikiAnswers. Both sites are designed to give you straightforward content, packed with information from either reference sources or the wider human Q&A community.

Our flagship, Answers.com, is the world's greatest - and only - encyclodictionalmanacapedia; it's one place to get answers about over 4 million topics, spanning technology, sports, entertainment, health and much more. Go ahead, try looking up arthritis, halvah, Casablanca - they all have one thing in common, and that's full definitions and explanations, conveniently accessible on one page.

On the flip side, our alter-ego, WikiAnswers, is the leading community-driven wiki-based Q&A site - and it's one of the fastest growing sites around. In fact, we were the second fastest growing site among the top 1,500 US Web sites, based on ComScore's count of unique monthly visitors, for the first nine months of 2007. On WikiAnswers, anyone can ask questions covering any topic under the sun, while anyone else with a bit of knowledge, wisdom or passion can refine their answers. Because it's a collaborative effort, the community is encouraged to not only answer questions, but also improve answers. Want to learn new dessert recipes? Interested in learning more about African history? Or maybe you're an expert in American football and you'd like to share with the rest of us… You can answer others' questions on children's literature, environmental issues, pet care, and dozens of other topics.

There are also many ways to access answers via our free tools and products, whether you're into getting information through Google, RSS feeds, Firefox or your Mobile phone. Bloggers and webmasters can choose from an array of widgets designed to bring answers to their visitors.

We're proud of how far we've come at Answers Corp, from offering licensed content at our one-stop reference source and then growing to include community-driven Q&A. And this is only the beginning for us.

My vision of the ultimate answer engine is right here, visible from my 'corner office' and accessible across the web and on mobile devices.

Now you know!

Bob Rosenschein

CEO Answers.com & WikiAnswers

Oh yes, we're proud to be 1 of the 10 Search Engine of the Year finalists; please vote for us here.

The dream team of search

In his article Search in the Year 2010, Gord Hotchkiss describes his search Dream Team:

If I ever had to build a search engine, or more precisely, the interface of a search engine, this would be the team I would want to bring together. When I came up with the idea of looking forward three years and speculating on what the search results page may look like in 2010, these are the names that immediately came to mind:

• Jakob Nielsen, the Web's best-known usability guru
• Marissa Mayer, Google's VP of user experience and interface design
• Michael Ferguson, one of the architects of Ask's unique user experience
• Larry Cornett, the VP of search experience at Yahoo!
• Justin Osmer, Product Manager for Microsoft Live search
• Chris Sherman, Executive Editor of Searchengineland and always thoughtful industry observer
• Greg Sterling, another industry analyst who always has interesting insights, particularly in the local and mobile world
• Danny Sullivan, the Go To Guy of search

Gord interviews the Starting 8 to get their predictions on what the search results page will look like in 2010. The resulting article is absolutely fascinating read and I recommend it. But I also like what I found in the comments section:

Successful new ideas are most likely to come from people that aren't subjected to limitiing beliefs. A Part 2 from smaller rising players would be an interesting read though!

Comment by DavidDalka | August 11, 2007 10:30 AM

I don't know if Gord is planning a 'Rookie Dream Team' followup to this, so I'm going to kick it off myself.

This is definitely an idea my buddy Charles Knight will go for. Right now, he's got lots of smart people busy thinking about a Unifying Interface for search. They're wondering what the best way is to allow users to easily access the massive power of more than 1,000 alternative search engines through a single interface.

I'd like to put these bright minds to work on Gord's question:

What will the search results page look like in 2010?

In contrast to Gord's piece, I'm not going to pre-select my Dream Team. Anyone who has a personal opinion on this topic is eligible to submit ideas. If you are a rising star in the search industry and would like to participate, please email me at kaila @ vortexdna . com. If you know a rising star you think should participate, either have said star email me or email me with the info of said star and I will follow up.

I will compile the answers and publish them in a post or series of posts, with acknowledgments and links as appropriate.

Let's see how creative fresh minds can be relative to the established industry experts!

SOCIAL SEARCH GUIDE: 40+ Social Search Engines

August 27, 2007 - 11:57 PM PDT - by Daksh Sharma

Google got Facebook application today, allowing you to share search results with friends. In fact, search is becoming increasingly social, with both the top search engines and startups incorporating users into the process of generating the best possible results. In our social search guide, we look at more than 40 search engines that either include user input in generating results or allow you to find information about people across a multitude of social networks and public records.

People Powered Search Engines

http://www.iprospect.com/about/researchstudy_2007_offlinechannelinfluenc...

Great study on why people use search engines.

In June 2007, iProspect partnered with JupiterResearch to develop and field a survey to U.S. search engine users (heretofore referred to as "online search users") to gain a better understanding of how exposure to offline channels influences their search and purchase behavior. Because iProspect works with many search marketing clients who coordinate and integrate their offline marketing channels programs with their search marketing initiatives, we hypothesized that a significant percentage of individuals are driven to conduct an online search as a direct result of exposure to offline messaging. Further, we also theorized that a significant percentage of those driven to search by these means eventually make a purchase from the company whose website is the object of their search. The questions that comprised the survey that lead to this study sought to quantify this offline influence on online search behavior.

MIT Convergence Culture
I saw a short news note from Daisy Whitney at TelevisionWeek yesterday, noting that NBC has said that a third of its Web site traffic comes from search engines.

This doesn't sound like news to me, but it indicates something fundamental that I think media companies have been missing for a while. As the technology for the Web has spread, media properties have competed with one another by who could create the most aesthetically pleasing site that technology allows for.

We have some of the best Flash animations, the slickest graphics, the coolest interactive features one could imagine for a site, yet many people are finding content through a search engine instead of coming to the main page of the site and clicking through. I hypothesize it might have something to do with that ugly "U" word: utility.

http://www.designobserver.com/archives/025809.html

Joseph Anthony Bartolo created a font called "Hebrewish" because, as he explains it, "the only Hebrew Latino font I have ever seen didn't really live-up to my expectations." Of course, if Hebrewish does't live up to your expectations, there is actually a real font called Faux Hebrew, and it's not the one on my desk.

ReadWrite

Powerlabs The goal and end point are not dissimilar from those of Hakia and some others, but the technology and "middleware" will be vastly different, as we shall see. Hopefully our coverage of the Powerlabs program will enable us to differentiate Powerset from other search endeavors.

What's the Big Deal With Social Search?
› › › SearchDay

By Chris Sherman | August 15, 2006 Social search is garnering a lot of attention these days, but despite all the hoopla it's not likely to displace traditional algorithmic search any time soon.

A Day Without Google
http://blogs.business2.com/business2blog/2007/06/a_day_without_g.html