Note: this post is a revised version of an earlier one on Marketocracy (which has been removed), after talking to the founders who alerted us to a few factual errors.

Marketocracy is on a mission to find the best investors in the world by tracking, analyzing and evaluating their performance in managing virtual portfolios on the site. The top 100, based on a proprietary ranking system, are used to create an equally weighted virtual m100 index, which is then used to provide the basis for a real mutual fund, called the Marketocracy Masters 100. Beyond the recognition these top investors get, they can also receive financial rewards for participating. All told, it’s a great framework to harness the power of mass collaboration.

But here’s where it gets tricky. If you go to the Marketocracy site, there is a graph showing a 5-year, 110% cumulative return (about 16% compounded annually, beta of 0.84) for the m100 virtual index, next to the S&P returning about 35% (about 6% compounded annually, beta of 1). This looks good, and appears to support the statement beside it of “higher returns with lower risk.” But then you look at the real fund, and it reports a 10.85% annualized return (about 65% cumulative), which Morningstar says is a below average return, while indicating the fund took on above average risk. Huh?

In trying to reconcile this suspicious looking performance and reporting gap, I originally surmised that the discrepancy indicates the use of hindsight in the virtual portfolio. In short, you kick out underperforming managers and replace them with better ones, and exclude the former groups past results, while the real portfolio doesn’t get the benefit of taking mulligans. After all, the difference between 110% and 65% is hardly negligible, and the big issue seems to appear when the real portfolio drops dramatically while the virtual one does not. But it appears this logic was off the mark.

While the components of the m100 virtual portfolio can be adjusted monthly, Marketocracy indicates the bad results from the past are still counted. And each of these virtual managers are assessed realistic transaction fees for each trade, and charge a virtual 1.95% management fee, so it should track reality fairly closely.

Marketocracy indicates the issue was that in 2004, when the real portfolio dropped dramatically, there was an extraordinary amount of movement in the virtual portfolios due to a tumultuous investing environment, and that for a variety of reasons the real fund (size, transaction costs, tools still in development, etc.) was unable to replicate them in the real world. At the same time, in a situation many funds experience, when people start selling a stampede to the door can start, driving the fund price down further in an often deadly spiral.

Marketocracy also indicates they believe they have corrected these issues, have gotten better at picking top investors, and point to how the real fund is now not only tracking the virtual one again, they are actually beating it (over the last year or so). The key reason given is that weightings in the real fund don’t have to be equal, so they can use their improving assessment tools to skew towards the best performers. In essence, they are getting smarter with each passing day and passing on the benefits to fund holders.

The more skeptical argument in regards to what happened would tie to Nassim Taleb’s book Fooled by Randomness. Marketocracy had up to 65,000 model portfolios going, from which they selected 100 (or 0.15%) to base the real fund on, with only a short time period of data to draw from. Those that did very well early may well have just been the lucky beneficiaries of random distribution – a million monkeys on a million typewriters kind of thing. When the luck ran out, the performance dropped off dramatically. In turn, or so a skeptic would argue, there is a distinct possibility that the recent success (buoyed by a new approach) might follow the same pattern given enough time. Only time will tell.

There is also another layer of complexity in here due to comparables. While the chart is shown against the S&P 500 (and the company feels the broader Wilshire index is probably the best benchmark), Morningstar put it in the “Small Blend” category based on what they perceive the apparent strategy to be. Other investment funds pursuing a similar strategy were seen to have earned higher returns, at lower risk, than the Marketocracy fund did, which is where their negative ratings come from. In short, Marketocracy benefited from being in the right part of the market, but others doing the same benefited more.

Marketocracy’s response is that they don’t belong in a category like that, because over time, their fund is designed to shift into whatever strategy is best, as their combination of fund managers, algorithms and various tools predict. So if large caps roar back into favor (or anything else for that matter), and you were with one of those small blend funds you’d miss the boat – while if the marketocracy approach works correctly you’ll automatically shift over. Again, only time will tell if this is true, particularly since the last time there was a major market shift is what caused many of the earlier problems.

Confused yet? The jury will remain out until we have a longer period of data and results to draw on, and probably long after that as well. In fairness to Marketocracy, making predictions about the future of the stock market is notoriously difficult, and there are regularly periods of three, five, or many more years that history looks back on as irrational and speculative. And the big question hovering above all of it depends on whether you believe markets are efficient or not, and whether there is any way to predict who will beat them.

Even Warren Buffett, sitting on an enviable track record and one of the richest men in the world, isn’t immune from speculation that he is a by product of randomness himself. Perhaps more to the point, almost the entire skeptical argument one can put against Marketocracy can also be applied to many mutual fund companies, which also tend to be far, far less transparent as they trumpet their winners loudly and hide the losers.

But if you do believe there are people out there that can beat the market based on skill, and you do believe there is some way to find them early based on results, think about how amazing Marketocracy is for them. The traditional path to Portfolio Manager has long been mired by a lack of comparable data to draw on, word of mouth recruiting, and many other things including lots of, well, luck. The vast majority of these people, in time, are proven to under perform the markets they are in. While there still might be some issues to iron out, the potential here is obvious, as Marketocracy sets out to create – at minimum – the farm team for the investment world, leveraging measurable data on every single decision they personally make.

And if Marketocracy can pull this off, they’ll have something that Warren Buffett himself would covet in his businesses – a big, wide moat of a competitive advantage. While many things can be quickly replicated in this world, a 5+ years and counting database of trading activity from thousands of virtual portfolios managers, with a lot of predictive modeling testing behavior along the way, cannot be copied easily.