From the Abstract:

The battle between proponents of the Efficient Markets Hypothesis and champions of behavioral finance has never been more pitched, and little consensus exists as to which side is winning or the implications for investment management and consulting. In this article, I review the case for and against the Efficient Markets Hypothesis and describe a new framework—the Adaptive Markets Hypothesis—in which the traditional models of modern financial economics can coexist alongside behavioral models in an intellectually consistent manner. Based on evolutionary principles, the Adaptive Markets Hypothesis implies that the degree of market efficiency is related to environmental factors characterizing market ecology such as the number of competitors in the market, the magnitude of profit opportunities available, and the adaptability of the market participants. Many of the examples that behavioralists cite as violations of rationality that are inconsistent with market efficiency—loss aversion, overconfidence, overreaction, mental accounting, and other behavioral biases—are, in fact, consistent with an evolutionary model of individuals adapting to a changing environment via simple heuristics. Despite the qualitative nature of this new paradigm, I show that the Adaptive Markets Hypothesis yields a number of surprisingly concrete applications for both investment managers and consultants.

Read the paper here

Here’s the most important lesson from this paper (via ssrn):

“This suggests another fundamental difference between rational expectations investors and trusting investors. Where the former look to “the shadow of the future,” the latter care about “the shadow of the past.” Put differently, a rational expectations investor expects others to exploit her whenever possible. Accordingly she will always be forward-looking, trying, just as a chess player might, to anticipate other players’ opportunistic future moves. In contrast, trusting investors look to the past. If someone or something has always behaved in a particular way in the past, trusting investors assume that that person or thing will continue to behave similarly in the future, without worrying too much about understanding what drives the behavior in question.”

Click Here To Learn About The Role Of Trust In Securities Markets

Abstract (Via SSRN)
Evidence is accumulating that in making investment decisions, many investors do not employ a ‘rational expectations’ approach in which they anticipate others’ future behavior by analyzing their incentives and constraints. Rather, many investors rely on trust. Indeed, trust may be essential to a well-developed securities market. A growing empirical literature investigates why and when people trust, and this literature offers several useful lessons. In particular, most people seem surprisingly willing to trust other people, and even institutions like ‘the market,’ in novel situations. Trust behavior, however, is subject to history effects. When trust is not met by trustworthiness but instead is abused, trust tends to disappear. These lessons carry significant implications for our understanding of modern securities markets.

Great Introduction (Via SSRN)

Burt Ross graduated from Harvard University in 1965. After working several years as a stockbroker, he ran for and was elected mayor of Fort Lee, New Jersey. Then Ross turned to commercial real estate. In 2003, he decided to sell some of his buildings and invest the proceeds, which amounted to more than five million dollars. Ross thought he was prepared for retirement. At least, he thought he was prepared until December 11, 2008, when he learned that his nest egg–which he had invested almost entirely in funds managed by the now-infamous Ponzi schemer Bernard Madoff– was gone. (Pulliam, 2008)

Read the Paper Here

(Cassidy’s analysis) asks us to ignore the repercussions of our behaviors. We can rationalize short term gains at the expense of long term losses, because we need to obtain quarterly profits regardless. Apparently, when it bankrupts the company, only then with the benefit of hindsight can we see what went wrong.

I am terribly sorry, but that is precisely the sort of thinking that led to the crisis in the first place. Making loans to people who cannot pay them back is not rational when its profitable — its NEVER rational.

Goldman Sachs avoided most of the credit debacle — were they being irrational when they forewent short term profits for a few years — but avoided the worst of the sub-prime debacle? And what about hedge fund manager John Paulson? His fund bet against all of these other players, netting several billions in profits while others suffered from their “Rational Irrationality.” How irrational was Paulson’s investment posture?

On a risk adjusted basis, the behaviors of Citi, Bear, Lehman, New Century and others was hardly rational. Call it whatever you want, but do not forget this simple fact: It was the sort of narrow, risk-ignoring thinking that is ALWAYS rewarded in the short term, and ALWAYS punished in the long term.

Great stuff from BR! Read the full post here

The problem is that financial breakdowns come with low frequency. Since flaws in the financial system may appear decades apart, it is hard to figure out how some new financial device will behave. Moreover, because of the low frequency of crises, people who use financial instruments often have little or no personal experience with the crises and so trust is harder to establish.

When people invest for their children’s education or their retirement, they are concerned about risks that will not become visible for years. They may not be able to rebound from mistaken purchases of faulty financial devices and they may suffer if circumstances develop that create risks that could have been protected against.

Thus, to facilitate financial progress, we need regulators who ensure trust in sophisticated products. They must work towards clearing the way to widespread use of better products, concerning themselves with both safety and creative ideas. They must not simply be law enforcers against the shenanigans of cynical promoters, but also be open to making complex ideas work that have the potential to improve public welfare. Unfortunately, the crisis has sharply reduced trust in our financial system.

…Unfortunately, people do not trust some good innovations that could protect them better. The innovations in mortgages in recent years (involving such things as option-adjustable rate mortgages) are not products of sophisticated financial theory. I have proposed the idea of “continuous workout mortgages”, motivated by basic principles of risk management. The privately issued mortgage would protect against exigencies such as recessions or drops in home prices. Had such mortgages been offered before this crisis, we would not have the rash of foreclosures. Yet, even after the crisis, regulators seem to be assuming a plain vanilla mortgage is just what we need for the future.

Another example of a potentially useful innovation is the target-date fund (also called life-cycle fund) that invests money for people’s retirement in a way that is specifically tailored for people their age. Such a fund plans for young people to take greater risks and for older people to invest more conservatively. Target-date funds, first introduced by Wells Fargo and BGI in the 1990s, are growing in importance, but few people commit the bulk of their portfolio to such funds, or make use of target-date funds that might make adjustments for their other investments. It appears that people do not fully trust that these funds are designed correctly, or would protect them from crises.

…It is critical that we take the opportunity of the crisis to promote innovation-enhancing financial regulation and not let this be eclipsed by superficially popular issues. Despite the apparent improvement in the economy, the crisis is not over and so the public continues to support government-led interventions. Doing this means encouraging better dialogue between private-sector innovators and regulators. My experience with regulators suggests that they are intelligent and well-meaning but often bogged down in bureaucracy. Regulatory agencies need to be given a stronger mission of encouraging innovation. They must hire enough qualified staff to understand the complexity of the innovative process and talk to innovators with less of a disapprove-by-the-rules stance and more that of a contributor to a complex creative process.

Read the full article here

One of the most provocative sessions at Schwab Impact conference was given by Dan Ariely, who deftly summarized his current research in the important field of behavioral finance.  Ariely is a professor of economics at Duke University and a visiting professor at MIT’s Media Laboratory.  He is also the author of the popular book, Predictably Irrational.

Ariely’s message was that, no matter how good their intentions or how deep their experience, people – investors specifically – consistently make the wrong decisions.  They behave irrationally, and predictably so.

Advisors who understand the natural biases in individual behavior can frame questions that will steer their decision-making process in a more rational – and economically better – direction.

Read the full article here

The widespread failure of economists to forecast the financial crisis that erupted last year has much to do with faulty models. This lack of sound models meant that economic policymakers and central bankers received no warning of what was to come.

As George Akerlof and I argue in our recent book Animal Spirits, the current financial crisis was driven by speculative bubbles in the housing market, the stock market, energy and other commodities markets. Bubbles are caused by feedback loops: rising speculative prices encourage optimism, which encourages more buying and hence further speculative price increases — until the crash comes.

You won’t find the word “bubble,” however, in most economics treatises or textbooks. Likewise, a search of working papers produced by central banks and economics departments in recent years yields few instances of “bubbles” even being mentioned. Indeed, the idea that bubbles exist has become so disreputable in much of the economics and finance profession that bringing them up in an economics seminar is like bringing up astrology to a group of astronomers.

The fundamental problem is that a generation of mainstream macroeconomic theorists has come to accept a theory that has an error at its very core — the axiom that people are fully rational. As the statistician Leonard “Jimmie” Savage showed in 1954, if people follow certain axioms of rationality, they must behave as if they knew all the probabilities and did all the appropriate calculations.

So economists assume that people do indeed use all publicly available information and know, or behave as if they know, the probabilities of all conceivable future events. They are not influenced by anything but the facts, and probabilities are taken as facts. They update these probabilities as soon as new information becomes available and so any change in their behavior must be attributable to their rational response to genuinely new information. If economic actors are always rational, then no bubbles — irrational market responses — are allowed.

Abundant psychological evidence, however, has now shown that people do not satisfy Savage’s axioms of rationality.

Read the full article here

Anchoring
Anchoring is the act of latching on to a given piece of information and using that as a point of reference for making decisions. Unfortunately, many investors anchor on things that are irrelevant to a business’s value, such as their own personal cost basis in a given stock or the 52-week trading high. Rather, we should focus on the thing that matters the most, the estimated future cash flow of a company.

Availability Bias
This mental shortcut concerns the relative importance of information. The importance our minds attach to information is correlated to how often we see the information. If we see and think about something often, our brains attach greater importance to it.

Endowment Effect
People place a higher value on things that they already own than things they do not own. Meaning, we would sell our possessions at a much higher price than at which we would buy the very same possessions if we did not already own them.

Sunk Cost Aversion
Sunk costs are costs that cannot be recovered once incurred. Once something is paid for in either time or money, our instinct is that we must soldier on and get some benefit for the expense, lest we feel like we are wasting resources. This is a variation of loss aversion, which is a concept that says people feel the pain of a loss at double the magnitude they feel the pleasure of a gain of the same amount. Two tips here. First, if a stock is clearly worth far less than what we originally paid for it, we should be willing to sell if today’s price is above our estimate of current value; that we are realizing a loss should be irrelevant to the decision. Second, if we spend several hours to research a given opportunity, we should still be willing to walk away. Our instinct will be to like the opportunity since we just spent time on it, but our goal should be to have rationality outweigh instinct.

Herd Behavior
Our deepest instincts tell us that there is safety in numbers. Beyond having a desire to do what is perceived to be socially acceptable, we often believe others have useful information from which we can take cues. After all, we all like to be liked, and the bigger group may know something we don’t. Simply, if “everybody’s doing it,” we feel the pressure to take that same action, whatever it may be. Plus, with investing being an activity where having incomplete information is the norm, this instinct to take cues from others can be amplified.

Recency Bias
We live in the here and now, and the ability to contemplate things far in the past and/or future is a uniquely human ability that requires higher cognitive functions. Yet our instincts can still get the better of us on occasion. Recency is the tendency to weigh recent events much more heavily into our decision-making than more distant events. It is a sort of mental short-sightedness where we think much more about our current situation than the much broader historical perspective. This can cause us to assume that the current state of the world–good or bad–persists into the future, rather than reverting to a long-run mean.

Confirmation Bias
Our brains inherently do not like conflict; they prefer to have a consistent, harmonious view of the world. They are wired to avoid cognitive dissonance–having two different ideas that are incompatible with each other. Our instinct is to search out information that confirms our existing views, accepting data that plug neatly into our preconceived biases, while rejecting data that do not support what we already think. Information that is consistent is processed more easily and does not increase stress.

Overconfidence
It’s an unfortunate fact that people tend to believe that their skill level is much higher than what it is in reality. For instance, the vast majority of drivers believe their driving ability is above average, even though this is statistically impossible. Unfortunately, Lake Wobegon is but fiction, and there is not a place where “all the women are strong, all the men are good-looking, and all the children are above average.”

This positive illusion we carry about ourselves allows us to be, as the famous book is titled, “Fooled by Randomness,” and attribute positive outcomes to our personal skills rather than luck or a trend over which we really had no control. Overconfidence can help us get through the stresses of our lives, but it can be deadly in the world of finance by causing one to overplay his or her hand.

Read the full article to learn more about each emotion

Related: Why Smart People Make Big Money Mistakes And How To Correct Them: Lessons From The New Science Of Behavioral Economics

IN the aftermath of the great meltdown of 2008, Wall Street’s quants have been cast as the financial engineers of profit-driven innovation run amok. They, after all, invented the exotic securities that proved so troublesome.

But the real failure, according to finance experts and economists, was in the quants’ mathematical models of risk that suggested the arcane stuff was safe.

The risk models proved myopic, they say, because they were too simple-minded. They focused mainly on figures like the expected returns and the default risk of financial instruments. What they didn’t sufficiently take into account was human behavior, specifically the potential for widespread panic. When lots of investors got too scared to buy or sell, markets seized up and the models failed.

That failure suggests new frontiers for financial engineering and risk management, including trying to model the mechanics of panic and the patterns of human behavior.

“What wasn’t recognized was the importance of a different species of risk — liquidity risk,” said Stephen Figlewski, a professor of finance at the Leonard N. Stern School of Business at New York University. “When trust in counterparties is lost, and markets freeze up so there are no prices,” he said, it “really showed how different the real world was from our models.”

In the future, experts say, models need to be opened up to accommodate more variables and more dimensions of uncertainty.

The drive to measure, model and perhaps even predict waves of group behavior is an emerging field of research that can be applied in fields well beyond finance.

Financial markets, like online communities, are social networks. Researchers are looking at whether the mechanisms and models being developed to explore collective behavior on the Web can be applied to financial markets. A team of six economists, finance experts and computer scientists at Cornell was recently awarded a grant from the National Science Foundation to pursue that goal.

“The hope is to take this understanding of contagion and use it as a perspective on how rapid changes of behavior can spread through complex networks at work in financial markets,” explained Jon M. Kleinberg, a computer scientist and social network researcher at Cornell.

Read the full article here

From the summary:

How can behavioural finance inform the investment process? We have taken a hypothetical ‘typical’ large fund management house and analysed their process. This collection of notes tries to explore some of the areas in which understanding psychology could radically alter the way they structure their businesses. The results may challenge some of your most deeply held beliefs.

► This collection of notes aims to explore some of the more obvious behavioural weaknesses inherent in the ‘average’ investment process.

► Seven sins (common mistakes) were identified. The first was placing forecasting at the very heart of the investment process. An enormous amount of evidence suggests that investors are generally hopeless at forecasting. So using forecasts as an integral part of the investment process is like tying one hand behind your back before you start.

► Secondly, investors seem to be obsessed with information. Instead of focusing on a few important factors (such as valuations and earnings quality), many investors spend countless hours trying to become experts about almost everything. The evidence suggests that in general more information just makes us increasingly over-confident rather than better at making decisions.

► Thirdly, the insistence of spending hours meeting company managements strikes us as bizarre from a psychological standpoint. We aren’t good at looking for information that will prove us to be wrong. So most of the time, these meetings are likely to be mutual love ins. Our ability to spot deception is also very poor, so we won’t even spot who is lying.

► Fourthly, many investors spend their time trying to ‘beat the gun’ as Keynes put it. Effectively, everyone thinks they can get in at the bottom and out at the top. However, this seems to be remarkably hubristic.

► Fifthly, many investors seem to end up trying to perform on very short time horizons and overtrade as a consequence. The average holding period for a stock on the NYSE is 11 months! This has nothing to do with investment, it is speculation, pure and simple.

► Penultimately, we all appear to be hardwired to accept stories. However, stories can be very misleading. Investors would be better served by looking at the facts, rather than getting sucked into a great (but often hollow) tale.

► And finally, many of the decisions taken by investors are the result of group interaction. Unfortunately groups are far more a behavioural panacea. In general, they amplify rather than alleviate the problems of decision making.

► Each of these sins seems to be a largely self imposed handicap when it comes to trying to outperform. Identifying the psychological flaws in the ‘average’ investment process is an important first step in trying to design a superior version that might just be more robust to behavioural biases.

You can read the paper here.

Is The Market Rational? No, say the experts. But neither are you–so don’t go thinking you can outsmart it.
By Justin Fox
December 9, 2002

Buy and hold. Diversify. Put your money in index funds. Pay attention to the one thing you can control–costs–and keep them as low as possible. Today that is pretty standard, if often unheeded, investment advice. Forty years ago it was revolutionary. The revolution started on college campuses, in particular at the University of Chicago, and it went by the unrevolutionary-sounding name “efficient markets.”

“In an efficient market,” wrote Chicago professor Eugene Fama in a landmark paper he delivered at the 1969 annual meeting of the American Finance Association, “prices ‘fully reflect’ available information.” That is, in an efficient market you can’t beat the market unless you have inside information. So why bother trying?

That logic led, among other things, to the creation of index funds that aim to mimic, not beat, the likes of the S&P 500 and the Wilshire 5000. Today such funds account for about 10% of total U.S. stock market capitalization, as well as 60% of what little money has flowed into equity mutual funds so far this year. But millions of small investors have continued to ignore the advice derived from efficient-markets theory, preferring instead to trade stocks and pile in and out of mutual funds in search of elusive market-beating returns (blowing much of their money on fees and commissions in the process).

Meanwhile, back on campus, a new generation of finance professors has been ripping Fama’s teachings to shreds. The organizing principle for this new breed of scholars is not efficient markets but something called behavioral finance. Behavioral finance teaches that stock market investors are irrational, that future stock price movements are at least partly predictable from past behavior, and that careful analysis of past trends and financial reports can pay off. Which happens to be the way most investors see the market already.

Over the next few pages we’re going to take you on a journey through the academic battles that have brought us to this point. This is FORTUNE’s annual Investor Guide, not The Chronicle of Higher Education, so we wouldn’t tell this story if we didn’t think it had relevance for investors battered by the experience of the past few years. The message that the behavioral finance guys have for investors is that yes, you can beat the market, but–for reasons that are essential to the whole behavioralist case–you almost certainly won’t. As a result, they end up offering much of the same investment advice that the efficient markets folks do. Only this time we might actually listen.

Gene Fama, it must be said, doesn’t buy any of this. It’s a cold, gray mid-autumn day in Chicago, but Fama is wearing a loud shirt with too-short sleeves that prove him to be probably the buffest 63-year-old finance professor on the planet. With the hint of a Boston accent that remains after two-thirds of a lifetime in Chicago, he’s talking the academic version of trash. “I don’t know that it’s progressed much beyond the level of curiosity items,” Fama says of the research done by the behavioralists. “They’ve got lots of interesting curiosity items.”

For years Fama was more than tolerant of this search for “curiosity items.” Unlike his longtime Chicago colleague Merton Miller, who saw the behavioralists’ work as an ideological assault on free markets (Miller died in 2000), Fama always encouraged empirical research, even research that revealed seemingly inefficient market behavior. In a 1991 sequel to his famous “Efficient Capital Markets” paper of 1969, he acknowledged that reality had in fact turned out a lot messier than he and other efficient-markets theorists had envisioned–although not so messy that the theories couldn’t accommodate it.

In 1997, though, Fama wrote still another paper, one that argued that a lot of the purported market anomalies discovered by the behavioralists were due to bad statistical work, and that the behavioralists’ attempts at building a theoretical alternative to the efficient-markets hypothesis had so far been “embarrassing.” The paper was something of a sensation–until interest in the D.C.-area sniper sent a piece called “Multiple-Victim Public Shootings” to the top of the charts in October, it was the most downloaded work in the almost six-year history of the Social Science Research Network, a leading academic website. Fama had no problem with behavioral finance as a critique of efficient-markets theory. But he thinks it’s a disaster as a replacement. “I don’t know what asset pricing would look like in a world that really took behavioral finance seriously,” he says. “If you really think prices are incorrect, what are you going to tell me about the cost of capital?”

We’ll get back to that question. But first let us document the behavioralists’ triumph. Half of this year’s economics Nobel went to their patron saint, Princeton psychologist Daniel Kahneman (the other half went to Vernon Smith of George Mason, whose economic experiments have also shot holes in efficient-markets dogma). Then there’s the John Bates Clark Medal, awarded by the American Economic Association every two years to the most important U.S. economist under 40: The 1999 and 2001 editions both went to behavioralists. On the pop-culture front, Yale efficient-markets skeptic Robert Shiller’s 2000 bestseller Irrational Exuberance was the most talked-about book by an economist in years.

The most dramatic development of all, though, may be that the office directly below Fama’s at Chicago’s Graduate School of Business now belongs to behavioralist pioneer Richard Thaler, 57. A magazine profile last year characterized Thaler, to the undying amusement of his students, as “thick-set,” but that’s not quite fair. He is not the jock that his upstairs neighbor is–Fama beats him at tennis. But Thaler, who arrived in Chicago in 1995 after years in the relative academic wilderness of Cornell University, appears to have eclipsed Fama as the most influential faculty member at the business school that has had more influence on the study of finance than any other.

For decades the University of Chicago was the temple for those who believed that markets always got things right. Thaler’s intellectual journey, in fact, began in the early 1970s on a campus that followed Chicago’s example to almost absurd lengths: the University of Rochester. “I can remember my professors yelling, ‘The price is right! The price is right!’” recalls Dartmouth finance professor Kenneth French, who arrived at Rochester just after Thaler left. “It was like a bad game show.”

Rochester was an extreme version of the direction the entire economics profession–not just its finance offshoot–had taken. The idea that economic activity can be explained as rational individuals trying to maximize their wealth goes back at least to 18th-century Scotsman Adam Smith, and the thought that all this was best expressed mathematically occurred to economists as long ago as the 1870s. But the approach didn’t really take off until the 1947 publication of MIT professor Paul Samuelson’s Foundations of Economic Analyses, which recast the principles of economics in the language of Newtonian calculus. The mathematization of economics that followed swept all before it. While academic economics had previously allowed room for multiple streams of thought, it took just a couple of decades before mathematical models built on the assumption of rationality were the only game on campus.

One of the most compelling of those models, and the one that seemed most closely to fit real-world data, was the efficient-markets hypothesis. It had its roots in empirical research that appeared to show stocks moving in a random walk–albeit with an upward trajectory determined by rising corporate earnings. The theoretical explanation, first proffered by Samuelson in 1965 and soon elaborated by Fama (who got his Ph.D. at Chicago in 1964) and others, was that stock prices fluctuate randomly because all knowable information about the value of a stock is already discounted in the price. That is, prices change only in reaction to news, which is by definition unpredictable.

For this view of price movements to work, the market has to behave rationally. That doesn’t mean every last investor has to be rational; it’s enough to assume that the hordes of irrational investors are irrational in different ways, thus canceling each other out. Or failing that–and here the theory begins to wobble–rational investors would be able to take advantage of the market’s temporary insanity to make a killing and push prices to where they belong.

That brings us back to Thaler, who was working on a Ph.D. in economics at Rochester in the early 1970s. His dissertation was an attempt to put a value on human life by looking at how much more people were paid to work in risky fields like mining and logging. He was working on the assumption, of course, that people rationally weighed the risk of death in their decision to accept a job.

Along the way Thaler decided to ask a few friends how much they’d be willing to pay to eliminate a one-in-1,000 chance of immediate death and how much they would have to be paid to willingly accept an extra one-in-1,000 chance of immediate death. What he found was that they wouldn’t pay much for the extra margin of safety but demanded huge sums to accept added risk–which isn’t, strictly speaking, rational. “I came to two conclusions about these answers,” Thaler wrote years later. “(1) I had better get back to running regressions if I want to graduate, and (2) the disparity between buying and selling prices was very interesting.”

Thaler did discuss his subversive thoughts with a few trusted colleagues and people from other disciplines. One of those people happened to be a newly minted psychology Ph.D., who sent Thaler a copy of a 1974 article by Israeli psychology professors Amos Tversky and Daniel Kahneman. (Tversky died in 1996; if he were still around, he surely would have shared in this year’s Nobel.) The article argued that in making decisions involving probability and risk, people rely on mental shortcuts that “are highly economical and usually effective but … lead to systematic and predictable errors.”

It was that last part that was so significant. That people make judgment errors wasn’t news, but if those errors were “systematic and predictable,” well, that was something an equation-wielding economist could get up and run with. (And making decisions involving probability and risk is what investing is all about, although Thaler wasn’t really thinking about that at the time.) Thaler wangled a short-term research gig at Stanford University when Tversky and Kahneman were visiting professors there in 1978 and ended up staying for 15 months. Then he played a key role in unleashing Tversky and Kahneman’s ideas on the economics profession.

The profession did not immediately respond with great enthusiasm. But over the years Thaler began to collect a few allies. Some were economists who shared his fascination with psychology. Others were number-crunching finance professors who had stumbled across seemingly irrational market phenomena–from short-lived ones like the “January effect” of rising stock prices in the first weeks of the year, which disappeared not long after people started writing about it, to the persistent tendency of “value” stocks with low price-to-book ratios to outperform the market. Throughout the 1980s the ranks of the doubters grew, but they remained a fringe element.

The efficient-markets guys, meanwhile, not only had come to occupy the academic mainstream but also had moved in on Wall Street. Not surprisingly, their initial relations with the Street had been hostile. What the professors were saying, after all, was that highly paid fund managers and analysts were not worth a dime. Some of the professors clearly reveled in that: In one famous mid-1960s exchange, a money manager asked MIT’s Paul Cootner, “If you’re so smart, why aren’t you rich?” To which Cootner replied, “If you’re so rich, why aren’t you smart?”

The answer to that second question was that people on Wall Street didn’t have to be smart to get rich, since they could make money off fees and brokerage commissions even when their market calls stank. But the devastating bear market of the 1970s caused some investors to question whether the people with whom they’d entrusted their money really were worth the expense. One logical result of such thinking was the index fund, which instead of trying to outsmart the market simply tried to imitate it while charging much lower fees than actively managed funds do. The first index fund for institutional investors was started in 1971 by Wells Fargo Investment Advisors (now Barclays Global Investors) in San Francisco. The first such fund for retail investors–the Vanguard Index Trust–was launched five years later.

Meanwhile a few finance scholars of a more diplomatic bent than Cootner began spreading their ideas of risk and return in the real world. Princeton economist Burton Malkiel’s A Random Walk Down Wall Street, published in 1973, probably played the biggest role in bringing efficient-markets thinking to the retail investing masses. But on Wall Street itself, the most important messenger was William Sharpe.

Sharpe, now 68, grew up in Southern California and learned his economics at UCLA. He was of the efficient-markets school, but his work (for which he won the economics Nobel in 1990) appealed even to those who still hoped to beat the market. In an efficient market the only way to outperform the market is to take on more risk. Sharpe devised a simple measure of risk based on past volatility, called “beta,” that could be used to build balanced portfolios–and to measure whether active money managers were actually beating the market or just taking on extra risk.

Sharpe wasn’t content to make his point merely in academic journals. He wrote textbooks on investments and finance and did so much consulting for Wall Street firms and pension funds that he gave up full-time teaching at Stanford in the mid-1980s. In 1996 he even launched a dot-com, Financial Engines, to make his advice available to small investors. So while Sharpe believes in efficient markets, he has also spent much of his career helping investors make choices. That, it turns out, makes him a big fan of behavioral finance. “As a practical matter, I still think it’s prudent to assume that the market is pretty close to efficient in terms of pricing and risk and return and all that,” Sharpe says. “On the other hand, we’ve certainly learned from cognitive psychology that ordinary human beings need to have alternatives framed in ways that can help them make right decisions rather than wrong decisions.”

Most of the wrong decisions investors make, behavioral research has shown, stem from overconfidence. That is, we think we know more than we do. We trade too much, we don’t diversify enough, and we extrapolate from the recent past to make assumptions about what will happen next.

As a result, much of what the behavioralists have to offer in terms of advice has to do with protecting retail investors from themselves. That’s why Thaler spends a lot of his time thinking about how best to design 401(k) plans. It’s why Sharpe incorporates behavioralist research into the advice Financial Engines doles out. And it’s almost certainly why Daniel Kahneman, when asked by a CNBC anchorman the day after his Nobel was announced in October what investment tips he had for viewers, responded, “Buy and hold.”

When I recount Kahneman’s words a few weeks later to Fama, he reacts with glee. “That means I won!” he shouts. It is, on one level, an absurd claim. The behavioralists are now clearly the dominant stream in academic finance, having made the leap from outsider status during the 1990s as a new generation of professors rose to positions of prominence. But the real-world phenomenon that cemented the behavioralists’ victory also illustrates why, when it comes to actual investing advice, they sound so much like Fama and Sharpe.

That real-world phenomenon was the stock market bubble of the late 1990s. According to strict efficient-markets thinking, there must be a rational explanation for what happened. Fama describes those sky-high Internet stock valuations as a risky but not crazy bet that one or two of those money-losing Net companies would end up as big as Microsoft. But he’s almost all alone on this one. “We have just lived through the biggest bubble of all time,” says Malkiel, who now calls himself a “random walker with a crutch.” Fama’s favorite collaborator, Dartmouth’s French, is on the verge of using the b-word as well when he stops himself. “I work very closely with Gene,” he says. “He would be very upset if I used that word in print.”

Yale economist Robert Shiller has no such compunctions about ticking off Gene Fama. In 1984 he declared that the logical leap from observing that stock price movements were unpredictable to concluding that the prices are in fact right “represents one of the most remarkable errors in the history of economic thought.” That was Shiller’s first brush with fame. He got more popular attention after the 1987 stock market crash, which the efficient-markets professors had trouble explaining. (“It’s weird,” Sharpe told a reporter at the time. Later his mother called to berate him: “Fifteen years of education, three advanced degrees, and all you can say is, ‘It’s weird’?”)

Shiller is 56 and did his economics training under Samuelson at MIT. He and Thaler have long been allies, but Shiller seems less interested than many of the other behavioralists in assembling the cognitive-psychology building blocks of a market bubble (which would involve that persistent flaw of extrapolating from the recent past to make assumptions about what will happen next). Instead he’s perfectly willing to accept at face value the conventional wisdom that markets are sometimes taken over by fads and mass hysterias. By the mid-1990s Shiller had become convinced that we were entering into one of those mass hysterias. His evidence was straightforward: Price/earn-ings ratios were really high. He began sounding the alarm wherever he could, including the offices of the Federal Reserve Board. Then he wrote Irrational Exuberance, which hit bookstores in March 2000, just as the market peaked.

The book’s perfect timing was dumb luck, Shiller himself says. And while he took most of his own money out of the stock market in the 1990s, his advice to investors now is to “diversify completely” and not try to beat the market. This happens to be what Sharpe would tell you. Or Fama. Or Thaler. The dirty little secret of the behavioralists is that, for all their work on investor irrationality and market anomalies, they still believe that markets work pretty well and that trying to outguess the collective wisdom of millions of investors is usually futile. In answer to Fama’s question of how they plan to calculate the cost of capital in a world where prices are incorrect, the behavioralists say that for the purposes of such calculations, they’ll just assume that prices are right.

But efficient-markets theory has a dirty little secret, too, which is that for the market to remain efficient, there have to be lots of rational investors who believe enough in the market’s inefficiency to spend their careers trying to beat it. Behavioralist theory, of course, has no problem accommodating the belief that some investors can beat the market. In fact, several behavioralist professors, Thaler included, have money-management firms that try to take advantage of the anomalies they discover in their research.

But there’s a limit to the riches that can be dredged from market anomalies. That’s because “markets can remain irrational longer than you can remain solvent.” This aphorism is usually attributed to economist and speculator John Maynard Keynes, and there are those who contend that the whole of the behavioralist case is contained in chapter 12 of Keynes’s 1936 General Theory, with its wonderful depiction of investing as a game of musical chairs. But the argument of modern behavioralists includes a crucial observation that wasn’t in Keynes–that professional investors are now under so much pressure from their customers that they cannot make the kind of long-term bets that might beat the market. If they do, as was the case with a lot of value-oriented mutual funds in the late 1990s, they can soon find themselves without any customers’ money to invest.

That gets us to a world in which an investor with enough staying power and contrarian gumption can beat the market, but the vast majority of mutual funds and hedge funds don’t. In other words, the behavioralists have reconciled the success of a Warren Buffett (which efficient-markets purists have absurdly termed dumb luck) with the overwhelmingly empirical evidence that most professional money managers fail to beat the market.

This is, we posit, a major intellectual accomplishment. What does it mean for you? That’s easy: Buy and hold. Diversify. Put your money in index funds. Pay attention to the one thing you can control–costs–and keep them as low as possible.