The fall (and rise?) of Yahoo: How the web giant crumbled and built some great tech in the process

At one point not so long ago, Yahoo (s yhoo) was one of the top technology companies in the world. “The only exception [was] Google,” Bassel Ojjeh, a former senior vice president of data technologies and products at Yahoo, said in a recent interview. “… That’s where big data came from. … It didn’t come from Walmart figuring out the connection between diapers and beer.”

Indeed, Yahoo did help shape the modern, data-obsessed web. It spearheaded projects like Hadoop that now run inside nearly every web company around, it eventually pushed Google (s goog) to its limits on search and it showed what the future of content would look like with major efforts around personalization. Companies like Twitter (s twtr), Facebook (s fb) and Netflix (s nflx) would have existed without Yahoo, but their services might look very different.

But in the end, the bigger and badder Google brought Yahoo’s business to its knees. The company isn’t dead, but its body has been prematurely autopsied countless times by technology journalists and analysts trying to diagnose what went wrong and how the next great CEO will fare. “Yahoo is the best-known company that people don’t like,” said Raymie Stata, former Yahoo CTO and current founder and CEO of a Hadoop startup. “… [A] well-known failure.”

Usama Fayyad, a former executive vice president and chief data officer at Yahoo, thinks that as late as 2006 Yahoo was still in the best position to take on Google and the rest of the emerging web companies. It had the technologies, the users, the data, the brand, a mature business — what more could you want? Looking at Yahoo now, after it squandered all that and is still fighting for relevancy, he said, “My expression would be ‘How the heck could this happen?’”

Here’s how it did happen, according to some of Yahoo’s top technology executives during its last hurrah as a legitimate competitor to Google, Microsoft and Facebook. The years between 2004 and 2009 were a time of great disappointment and missed opportunities, but also prescient ideas and impressive technological innovation that probably helped keep Yahoo afloat and might still help it rise again.

Turning Yahoo’s cash cow into a gold mine

If there’s one thing Yahoo never had a problem getting, it was users. And during the company’s heyday, a lot of users meant a lot of people seeing the display ads on sites; perhaps a few of them even clicked. For about a decade, it dominated in display advertising revenue and innovation. This meant a lot of money for Yahoo and — almost as important — a lot of data.

Fayyad used all that data to lure computer scientists into his newly formed Yahoo Research division. It wasn’t always easy — many engineers thought of Yahoo more as an advertising operation than a place concerned about real technology, much less real computer science research — so it was Fayyad’s job to help change that image.

Over the course of a few years, though, “we were systematically winning” against Microsoft and Google, Fayyad said, even though Yahoo often offered researchers significantly less money than the other guys were offering. (Although, he acknowledged, Research was decimated over the years and most of his hires have since left for Microsoft and Google.) Yahoo Research was able to attract talent because it had much more, and more interesting, data for researchers to experiment with than did Microsoft (this was pre-Bing) and would give researchers more opportunities to publish than would Google.

The influx of brainpower within research and elsewhere paid dividends. Google went public in 2004, the same year Fayyad arrived at Yahoo, and was already becoming a juggernaut, but banner ads were still ringing the cash register at Yahoo, so it spent a lot of time trying to make them even more profitable via targeting. The traditional model was just selling ad space and relying on the total number of page views to goose buyers’ excitement.

The space Yahoo used for targeted ads actually represented a small amount of its total inventory, but it was able to get advertisers to pay a premium by showing specific ads to specific demographics, Fayyad said. “It was free money.”

When Fayyad got to Yahoo, he said, ad targeting was responsible for about $20 million in revenue a year. When he left, that amount had swelled to about $500 million. Bassel Ojjeh, whose team was charged with building a lot of the data products Yahoo sold to advertisers, said about $700 million a year was flowing through the data systems by the time he left in 2009.

“In many cases, we were used as a sales tool,” he explained. At least once a week, his team was in front of customers and partners explaining why its work would result in a better return on investment. “You can only talk so much about the new features of Yahoo Mail,” he added, an allusion to the reality that simply placing ads in front of people (and on arguably subpar products) didn’t necessarily result in clicks.

Overcoming search engine apathy

On the surface, a focus on display ads doesn’t seem like such a bad idea as long as the company can keep its revenue from them flowing. For Yahoo, though, the necessity of selling display ads arguably meant it needed to act like a media company — up until then the only other type of company that has ever made money based almost solely from advertisements. So Yahoo’s mission was to create more content — articles, games, email, search engines — to which it could draw users and place ads.

“What I think happened was there was an identity crisis in the exec ranks,” said David Ku, who was at Yahoo from 2004 through 2010 (most recently as senior vice president of advertising products) and is currently corporate vice president in Microsoft’s Online Services Division. No one was sure if Yahoo was a media company, a search company, a technology company or whatever, and that “back and forth” caused a lack of cohesion among the teams and resulted in an identity crisis.

In hindsight, many argue, Yahoo should have put a lot more resources into search a lot earlier. It wasn’t that Yahoo didn’t understand the value of being able to search the web — it has always had a search engine, after all — it was just that it didn’t understand the value of search. “Search was always something that a few folks felt strongly and passionate about in Yahoo but a few folks, starting at the CEO, couldn’t grok,” Ku said.

Well, its first CEO, at least. Timothy Koogle, who was the company’s chief executive from 1995 until 2001, actually outsourced Yahoo search to Google in 2000. Paul Graham, the Y Combinator creator, wrote a scathing blog post in 2010 that laid out some of the problems he saw while in Yahoo’s employ after it acquired his startup Viaweb in 1998. About Yahoo’s apathetic attitude toward search, he wrote:

“I remember telling [Yahoo co-founder] David Filo in late 1998 or early 1999 that Yahoo should buy Google, because I and most of the other programmers in the company were using it instead of Yahoo for search. He told me that it wasn’t worth worrying about. Search was only 6% of our traffic, and we were growing at 10% a month. It wasn’t worth doing better.”

(Yahoo CEO Terry Semel, who served from 2001 through 2007, famously failed to buy Google for $3 billion in 2002, and later lost a deal to buy Facebook for $1 billion in 2006.)

Fast-forward a few years and it was pretty clear that Google was onto something with its idea of search as the product. “Oh my goodness, the portal isn’t necessarily the main attraction,” Ku said, explaining Yahoo’s reaction to the fact that someone might treat search as more than a commodity feature. “There’s a desire that search become the starting point.”

So, around 2003, Yahoo did begin to embrace search — kind of. Then-CEO Semel acquired another struggling search company, Inktomi, and tasked it to build out Yahoo’s search capabilities. By 2005, the company had actually built a very respectable search engine (more on that later), achieving in just two years what took Google almost 10 years, Usama Fayyad said.

“We didn’t lose users because we had a sucky search product,” former CTO Stata said.

The business of search proves elusive

Yahoo lost users because it didn’t evolve its business model to really embrace the search experience. If you’re trying to sell ad space, search is a great way to make money because users tell you exactly what they’re looking for. “What is most valuable [in targeted advertising] is understanding users’ needs and intent,” Ku explained. “… Intent is money.”

In search, advertising begins with sponsored search — those paid results that appear above and to the side of the real results. Until 2007, Yahoo’s method for filling sponsored results was, essentially, just giving them to the highest bidder (thanks in part to its $1.63 billion acquisition of Overture). Google’s approach, which Stata called “just fundamentally better,” valued bid price as well as the statistical likelihood that users would actually click on the link.

Google’s algorithmic approach to sponsored search drove up clickthrough rates on its ads, resulting in happy users, happy advertisers and boatloads of ad revenue. Revenue bought distribution, including a $2 billion toolbar deal with Dell and the ability to pump hundreds of millions into the Mozilla Foundation and become the default search option in Firefox. That was the time when Firefox went mainstream “and they took Google with them,” Stata said.

Yahoo was aware that toolbar deals were important because it had actually conducted a study that proved people will use whatever search bar is in front of them, regardless the quality of the results, said Bassel Ojjeh. His team shared the study “and the executives did nothing with it,” he added. “… At that point it was almost game over.”

To make matters worse, Yahoo’s sponsored search used to run on a massive Oracle database that would go down for 36 hours at a time and cost the company tens of millions of dollars every time it did, Ojjeh said. “And it went down frequently,” he added.

Semel’s answer to Google was an epic project called Panama that aimed to rebuild Yahoo’s sponsored search in Google’s image. The system finally rolled out later than anticipated, in 2007, and did improve search revenue but not nearly enough to really dent Google’s sizable lead. It also consumed the resources of about 2,000 employees at its peak.

“There were a couple of systems that pretty much consumed the entire energy of the whole company … [and] ended up paralyzing its ability to innovate anywhere else,” Ojjeh said.

One was of them was Panama. Ojjeh doesn’t look back on Panama as glowingly as some other Yahoo veterans because it represented an attempt to play catch up in a field where Yahoo was clearly No. 2. He thinks Yahoo should have focused instead on growing its lead in display advertising, where it ultimately ended up playing catchup, too, after Google bought DoubleClick (another company Semel could have acquired, but didn’t, as Yahoo CEO.) The other system Ojjeh referenced was Apex, an automated display advertising system that involved parts of its pricey Right Media acquisition and was meant to put Yahoo on equal footing with DoubleClick.

Microsoft, Jerry Yang’s vision and what could have been

One could argue Ojjeh was right. Microsoft famously offered nearly $50 billion for Yahoo in 2008 and Yahoo, under newly installed CEO (and co-founder) Jerry Yang turned down the deal. The Microsoft offer was “hugely distracting,” Usama Fayyad said. “It almost froze the company for six months.”

And at a critical time, too. Fayyad was already planning his exit when Yang took the CEO reins and laid out a “reinvigorating” strategy — complete with a presentation by late Apple CEO Steve Jobs for the company’s top executives — around focusing on Yahoo’s core value to users and advertisers and shedding the businesses that didn’t fit that mold. Search was definitely part of the future as were some newer attempts to capitalize on Yahoo’s data expertise.

Fayyad thought the results would have been “incredible,” but the company was too distracted by other things — its myriad businesses, the day-to-day stock price and the Microsoft offer — to ever begin executing. Shortly after Yahoo’s board rejected Microsoft’s final offer, he and a handful of other key Yahoo executives including Qi Lu (who’s now executive vice president of Microsoft Applications and Services Group) and Jeff Weiner (who’s now CEO of LinkedIn) made for the door. Some had been planning their departures for a while and only stuck around because they didn’t want to negatively effect the company’s value or image during that negotiation, Fayyad said.

In 2009, with Panama not delivering the results the company was hoping for, Yahoo’s search business all but vanished when it signed the infamous 10-year deal with Microsoft that currently has Bing powering Yahoo’s search engine. “It’s unthinkable to me where you go from a place where Microsoft would have paid billions for a business to Microsoft taking that business for free,” Fayyad said.

Through it all, lots of good technology and near hits

However, anyone looking at Yahoo only as an unfocused business might miss out on all the innovation — and occasionally even good product ideas — going on inside the company. Take, for example, search — not the sponsored part but the actual blue links representing query results. Stata, who spent years working on search before taking the CTO role, said people at Google used to tell him how amazed they were that Yahoo’s relatively small group — largely without the requisite Stanford degrees — was able to keep up with Google and actually push the envelope in some places.

Much of that work was done before he arrived in 2004 (the result of the Inktomi acquisition) but one of his proudest moments came after he arrived: when Google stopped displaying the number of URLs it had indexed because, he claimed, it realized it wasn’t pulling away from Yahoo on this front. One thing lost in all the back-and-forth surrounding the eventual Microsoft deal is that Yahoo almost made a similar deal with Google until the government shut it down. Stata believes the only reason a Google deal was ever a possibility is because Yahoo was so relentless on search. (And if you look at Yahoo’s search engine today, he noted, it’s still responsible for the page design and all the additional information displayed on the sidebars.)

Yahoo tried to be a leader in social search as early as 2005, which is why it bought Flickr and Delicio.us and launched new services like Yahoo Answers. The company was actually thinking about ranking results based on users’ social networks, Stata said, but it ultimately stuck with content because it never generated enough of a social graph to get the network-based scoring off the ground. (Although that might have been a different story had Yahoo 360 not been built completely separately from the search team and actually leveraged the popular Yahoo Messenger service, he suggested.)

“If Wikipedia didn’t exist, we probably would have had a better time of it, particularly with Answers,” he added.

Yahoo’s efforts to personalize front page content for users were pretty impressive — at least in theory — even if they underscored its media-company mentality. Yahoo had gathered lots of data about users by that point, as well as data about the types of articles that were popular on Yahoo. The company’s search experts assumed they could automate front-page curation by developing machine learning algorithms that would predict what articles would be popular among which readers and target them accordingly.

However, Stata explained, the company ultimately had to settle on a method by which human editors and targeting algorithms worked together to select the best pages for the Yahoo front page and to place them in front of the right readers. Sometimes, they’d simply A-B test articles and give wider distribution to those that the test audience clicked on.

“We tried six ways to Christmas to try and create that classifier and we couldn’t,” Stata said. “… What’s popular is popular.”

David Ku thinks the hit-or-miss results on content optimization might come back to Yahoo’s relative failure in search. There are known techniques for personalizing content but like with advertising it helps to know what someone is looking for at that particular time, or to have some purpose to the personalization such as Facebook’s focus on what’s popular among your friends.

“If it’s just for me,” Ku said, “then the personalization had better be damn good.”

Powering it all with Hadoop — for better or worse

Yahoo’s most-successful — and most-famous — technological endeavor is probably Hadoop, the open source big data platform that Yahoo helped nurture and mature into a full-fledged enterprise technology by 2011. (For more on the creation of Hadoop, Yahoo’s very important role in its evolution and Hadoop’s ultimate status as an entire enterprise IT market, check out our February 2013 feature on the technology.)

Yahoo poured millions of dollars (possibly billions) into Hadoop’s development, and it famously acquired the distinction of being “behind every click” at Yahoo. That’s largely true, at least to the point that those clicks involved analytics, targeted content or search results. Yahoo brought in Hadoop in 2006 on to help index web pages so it could compete with Google on search (Hadoop finally began powering search a few years later), but it proved useful enough to underpin later iterations of Panama, analytic tools that Yahoo presented to advertisers, content optimization, targeted advertising and a litany of other uses.

Hadoop spurred a number of changes in Yahoo’s infrastructure and led to new ways of building an online experience that was rich in data and design, Ku said. Across the company, there was a continuous loop between serving content and analyzing data that resulted in better results for users, advertisers and Yahoo.

Widespread adoption wasn’t a guarantee, though, because Yahoo’s divisions often were running in isolation from each other and had different ideas about the right technologies, Stata said. In fact, he acknowledged, the search team was as guilty of this as anybody else — it had taken on a “bunker mentality” as it focused its energy on battling Google. Hadoop was the first thing that came out of search and flowed in the other direction and was seen as useful.

“It did help bring the company together technically as this Yahoo thing,” Stata said.

Usama Fayyad, who departed in 2008 before Hadoop had really reach its full stride inside Yahoo, acknowledged it was a savior in some ways and was perfectly suited to some jobs, but also identified what he called “an overcommitment to it.” “It’s almost like a religion,” he said.

Before he left, Yahoo was investing heavily in relational databases, NoSQL databases and even a columnar analytic database called Everest that was designed for querying big data related to targeted advertising. He views Yahoo’s decision to port so many workloads to Hadoop — in part because of a fascination with the technology, in part because of a concerted effort to focus energy around fewer things — as akin to using a bulldozer to push a stone, in some cases.

Indeed, Bassel Ojjeh’s new company, nPario, actually benefited from the decision to focus on Hadoop, which took longer to get up to production quality than initially hoped for. nPario utilizes the Everest technology as part of its analytic engine (it now runs the Everest technology on top of Hadoop) and for a while counted Yahoo as a customer.

When you’re talking about such a large-scale system, Fayyad explained, it ultimately becomes a business question as much as a technology question: “Is the cost of maintaining that [Hadoop cluster] justified by the value of the computation being made?” For certain workloads at Yahoo and many other companies that have went down the same path, Fayyad said, the answer is probably “No.”

He suggested Yahoo lost a number of its best data scientists because of Hadoop, who saw opportunities to do certain computations on more-appropriate technologies elsewhere. (I have argued before that Yahoo’s business failings actually resulted in a double whammy, as they made it possible for companies like Facebook to make hay with Hadoop at Yahoo’s expense.)

In the end, users still need a reason to visit

Really, though, any debate over the wisdom of committing to Hadoop, investing so heavily in Panama or Yahoo’s myriad acquisitions (including those under new CEO Marissa Mayer), would be largely moot had the company recognized its situation earlier on and committed to being something — anything — other than a portal. Whether it was Google with search or, later, Facebook with social, other companies just had clearer visions of how consumers could use them to access the web and they became people’s preferred access points.

You need “some reason why you want to have that service be part of your life,” David Ku said. An unfocused Yahoo that wasn’t hooking you in with information or friends, but rather with an amalgamation of (mediocre) services and content eventually wasn’t giving users much of a reason.

“Yahoo was not addressing the next generation of users,” Ojjeh said. “Yahoo was really for people my age.”

Ku thinks Mayer has been a boon for Yahoo so far, at least insofar as she has injected a sense of energy and momentum into a once-stagnant company and genuinely seems concerned about improving the user experience. However, she still must find the company’s core business (hey — maybe it’s media, after all!) and a way to integrate it into users’ lives. Microsoft and Google are already neck-deep into the world of devices and using them as another consumer access point, he noted.

And even though Yahoo has fallen from grace over the years and its ranks have been somewhat decimated, Mayer didn’t walk into a kitchen where the cupboards are bare. (Our own Om Malik might disagree with that sentiment.) The company still has some great technology (if it wasn’t for the stuff he and his peers put in place, Stata said, the company might already be dead), great technologists and an institutional knowledge about data. And hundreds of millions users and visitors across its various services.

“I’m still in the startup business … and we’re starting from scratch,” Fayyad said. “Imagine if you give me a base and you say, ‘OK, Usama, you can start from 700 million users. What can you do?’ I’ll tell you, it’s amazing what you can do.”

Author: Derrick Harris

View Online / PDF