In this module we're going to discuss statistical biases in performance evaluation. The goal here is not to build realistic models but rather stylized models, so simpler models, but models that nonetheless will help explain to us that biases do arise, and that biases can be significant. When we are evaluating the performance of fund managers. Let's get started. Some fund managers claim to have special skills. What do I mean by special skills? Well for example they might claim to be very good at picking stocks or they might claim to be very good at timing the market. And what do I mean by timing the market? I mean the following. Maybe they know a good time to enter into the market, in other words when to invest in the S&P 500 or the Euro stocks, or whatever. And also they might claim that they know a good time when to get out of the market, when is a good time to sell. So, that's what timing the market means. This skill is often referred to generically as alpha. And the term comes from the capital asset pricing model. If you recall the cap end that you'ld seen in the previous module. We know that the expected return on an asset is equal to the risk free rate plus beta times the expected return on the market minus the risk free rate. I've added in here this additional term here alpha. So this can be viewed in the context of the capital asset pricing model as the excess returner, the skill if you like, that a manager can earn on an investment. I don't want you to think that alpha is necessarily associated with account fund. It's a generic term used to evaluate skill, whereby a manager can earn excess. Risk adjusted returns. Why is this important? Well, a fund manager who has alpha, or who has skill, can often charge substantial management fees. And so what we want to know, is whether or not the skill is real. In general, it's very hard to tell, but you can still do some interesting analysis, and that's what we're going to do in this module. We're actually going to look at this question from three different perspectives. The first two perspectives, will require the binomial distribution, and so we're going to review a little bit about the binomial distribution now. If you recall, we say that x is a binomial distribution. Or we write x tilde Bin n p, if the probability that x equals r equals n choose r, times p to the r, times one minus p to the n minus r. And so x might represent for example, the number of heads in n independent coin tosses, where p is equal to the probability of a head. The mean and variance of the binomial distribution are given by these quantities here. The expected value of x is n times p, and the variance of x is n p times 1 minus p. You can also see from one that the probability that x is greater than or equal to r is equal to the sum of the probabilities that x equals r, x equals r plus 1 up to x equals n. And that's given to us by this summation here on the right hand side. So, to see our first perspective on this problem, consider the following situation. Suppose a fund manager has a track record of ten years and that this fund manager has outperformed the market in nine of the ten years. The fund manager claims to have great skill and that his fees should reflect this. What we want to know is, well how can we assess his claims? Does he really have great skill? So first analysis might assume the following, in a given year he outperforms the probability p and under performs with probability 1 minus p. And by the way, when I say outperform, I mean outperform relative to the risk the manager is taking on. So I'm talking about risk adjusted returns here. Outperformance or underperformance is assumed to be independent across years. If the manager has skill, then p is greater than a half. Otherwise p is less than or equal to a half, and the manager has no skill. So the first question that now comes to mind is the following. How likely is such a track record if the fund manager had no skill? Well to answer this, let x be the number of outperforming years. If the fund manager has no skill, then x is going to be binomial, with n equals 10, and p equals a half. We said that the fund manager outperformed in 9 of the 10 years. So, a track record as good as this would corresponds to x being equal to 9 or indeed x equals 10. So we want to compute the probability that x is greater than or equal to 9. Assuming the manager had no skill. In other words, assuming p is equal to a half. That is given to us by our binomial probability we saw in the previous slide. We can evaluate this easily in Excel or some other piece of software, and we find the answer of 0.0107. So therefore, if the fund manager has no skill, then the probability of having a track record as good as his, or better, is only 0.0107. So at this point, you might think it's fair to conclude, that the fund manager does indeed have skill. After all, you could think of this in a statistic setting, for those of you who are familiar with the statistics, you'd be aware of the concept of a p value. And usually a p value that's less than or equal to 0.05 would be assumed to be significant. So we've got a p value here if you like, of 0.0107, this seems significant. It seems unlikely in this case that the fund manager is no skill. Given the track record of 9 successful years out of 10. So that's the first perspective we're going to take. Here is a second perspective, suppose instead that there are M fund managers and that the manager who claims to have skill has the best track record of these managers. The questions that now arises is, does this change anything in our analysis? Should it change something in our analysis? To answer this suppose we start with the hypothesis that none of the fund managers have skill and that track records of fund managers are independent. There are two possible questions now that arise. The first question is, how likely is the third manager to have such a track record if all fund managers have no skill. Well, in this case, if we've identified the third manager in advance, then the first perspective gives us the answer. It would actually. Actually be 0.0107, as we calculated beforehand. The second question or possible question to, to ask, is how likely is the best manager to have such a track record if all fund managers have no skill? And now, what is the appropriate question here? Which of these two questions is more appropriate? This actually depends on our prior hypothesis. What do I mean by that? Well let's come back to this slide here. We've actually identified this manager here, this is the third manager. And this is the best performing manager of the M managers. And so what I mean by prior hypothesis is the following. Did we identify this third manager in advance? Maybe this third manager was our friend, or a cousin, or somebody else we specified in advance before we saw any data. If that is the case, then this question is the more appropriate question. We've seen the manager in advance, we're interested in his performance, not because he was the best performing manager, but because he was the third manager or some manager we've seen in advance. However, if we're interested in the third manager because he was the best performing manager, and that's why his track record has come to our attention, then the appropriate question is actually the second one. How likely is the best manager to have such a record, if all fund managers have no skill? And this is because the reason we're interested in the third manager is not because he was number three in the list. But because he was the best performing out of all m managers. And that's why he came to our attention. So, it really depends on our hypothesis. Why are focusing on this manager? Is it because he was the third manager, we'd pre-selected him in advance? Or is it because he was the best manager, and that's why the manager's track record has come to our attention. Depending on which hypothesis is correct, we get a very different answer. With the first hypothesis, we see it's 0.0107. With the second hypothesis, for our interest in the manager is because he's the best performing manager out of M, the second hypothesis is the more appropriate. So let's answer this second question. We're going to assume none of the managers have skill. We're going to let z i be the event that the ith manager out-performs in r years or more out of the n years. We're going to let v be the event that the best manager outperforms in r years that event, or more. Then what we're interested in is the probability of v. The probability of v is equal to 1 minus the probability of zed 1 bar up to zed m bar. Now what is zed i bar? Zed i bar is the complement of zed i. So zed i is the event that the ith manager performs in r years out of n, so zed i bar is the event that the ith manager outperforms. In less than r years out of n, so it's the complement of zed i. So therefore just to make sense of this, if the best manager is to outperform in r years, the event that the best manager doesn't outperform in r years and more. Is equivalent to all managers. All m of them outperforming in less than r years out of n. And that's this quantity over here. Because we're assumption that these z, zed i's are independent, and IID. I can write this as just the probability of zed 1 bar. Times the probability of zed q bar, up to the probability of zed m bar. Just keeping these separate. And these are all IIDs, so this is, there's m of them. And so I get the probability of zed one bar to the power of m. Now, recall, the probability of zed 1 bar. Zed 1 bar is the event that's the compliment of zed 1. So therefore, the probability of zed 1 bar is equal to 1 minus the probability of zed 1. And therefore, I can plug in the numbers. I'm assuming, in this case, I believe, that m is equal to 20. And I know that the probability of zed 1 is 0.0107. Because I actually calculated this. Onto our first perspective. So this is for a fixed single fund manager. The probability of zed 1, we found to be 0.0107. So we need the probability of zed 1 bar, here, which is 1 minus that. Take it to the power of m, we get this number here, 0.1942. And actually you can easily see what happens. As m gets bigger, as we increase m, the number of fund managers in the marketplace. Place then actually probability of v will actually increase to one as m gets very large. What are our conclusions. Well our conclusions are as follows. If there are a lot of fund managers in the market, then the, the fact that one particular fund manager has a fantastic record, does not, in and of itself constitute evidence that, that fund manager has skill. After all, we have shown here, that with just 20 fund managers we would expect the best manager to have a record of outperforming in 9 years. Or more out of 10 to recover probability 0.1942. And that's not a very small number. If we take m equal to 30 or 40, this number will get bigger. And so even though we've assumed all the fund managers have no skill here. The fact that we're focusing on the best manager's track record. Suggests that the best track record, will be good. So we haven't established evidence to suggest that a particular fund manager has skill just because he's got a great track record. It all depends on the hypothesis. How did we identify this manager? I should also point out that while we have focused on the best manager here, we could also have focused on the second best manager or the third best. Manager and so on. If n gets sufficiently large, we're not going to just expect the best manager to do well. The second best manager will also have a very good track record. The third best manager will have a very good track record. And so on. As long as m gets sufficiently large. And this is true even if all of the fund managers have no skill. Here's another perspective. This is our third perspective. Suppose again that all fun, fund managers have no skill. At the end of every year, fund managers who have out performed the market that year, survive. And fund managers who have under performed in the market that year, get fired. Here's a question. After one year of this experiment, what would be the average track record of fund managers in the market? Well. It's got to be perfect. Only the fund managers that have a good year survive, so they will have a perfect track record. The fund managers that have an imperfect track record, in other words they under performed that first year, well they've been fired, so they're no longer available, they're no longer in the marketplace so we won't see them. And so their performance does not enter into our calculation of the average track record. And indeed you can look at this after 2 years, or 3 years you'll only see perfect fund managers in the market. So only perfect fund managers survive, and they appear to have a perfect track record. And yet it's clear that in fact, that these fund managers have no skill. You could also generalize this in fact, as follows: you could imagine that fund managers who've outperformed in a given year will stay in the market with some probability p. And that fund managers who've underperformed in the market will be fired with a different probability. And we can assume that fund managers who've under-performed get fired more frequently than fires, than managers who've over performed. You could do that type of experiment, you could simulate that kind of system and what you'll still see is that the average track record of fund managers in the market would actually be greater than a half. So actually they will appear to be a level of skill in the market, even though that skill isn't really there. This is an example of what is called survivorship bias. The reason being that some people have survived, they're the people who have outperformed in the market, and they actually make the market look better. They look like the, the collection of fund managers look better than they really are. They've just survived because the poor and unlucky fund managers have actually Under performed and lost their, and lost their jobs. So this is an example called survivor bias or survivorship bias. Its a very common phenomenon in finance and beyond and we're actually going to see some more examples of survivor, survivorship bias very soon. Now some final thoughts here. I've been assuming up until now that the fund manager or the fund managers have no skill. And you might ask the question, well, is that fair? Is it fair to assume to begin with that fund managers have no skill? After all, in practice there are some managers with skill. And that's absolutely true. There are some managers in practice with skill. I don't there are too many of them and certainly lots of fund managers may think they have the skill but the reality is quite different. Nonetheless it is true that there are some fund managers out there with skill. That having been said, I don't think we're being unfair in assuming to begin with that fund managers have no skill. And that's because it is the responsibility of the fund manager to convince us that they have skill. After all, fund managers with skill try to charge fees to manage our money. So it's up to them to convince us that they have skill. It's not up to us to give them the benefit of the doubt and assume they have skill. So it's perfectly reasonable to start off with the assumption that they've no skill, and then to see where that takes us. And if we can find such a manager, is the resulting out-performance sufficient to justify the management fees? That is not clear as well. If the management fees are too large, then those fees are going to dominate the out-performance. That the manager provides and so in that situation we wouldn't want to invest with the manager anyway.