In this lecture we'll discuss some of the risks with AI. I'll begin with a simple statistical risk which has significant managerial implications. And then I'll talk about social and ethical risks. So, the first risk I want to discuss is the risk of overfitting. Now many of the complex and advanced machine learning models, such as neural networks great and boosting and others can easily overfit the data, meaning that they tend to fit the historical data almost too well, but they fail in realistic situations elsewhere. This happens because these models often have too many parameters and they can fit very complex shapes and they tend to fit historical data just too well. But then they don't generalize outside of that historical data. Now, if we don't understand what is helping the model perform well, then this poses a significant risk when we deploy these models in the real world, there are many operational risks from using models that have been overfit. For example, suppose you have a trading algorithm that is based on machine learning and that is making stock trading decisions that are direct financial risks from it. They're also customer perception and reputational risks. For example, you have a chat part that is interacting with customers or you have a personalization algorithm that is personalizing user experience and that is not working well in practice. Then it hurts customer perception and retention over the long run. So, in short while there are many of these statistical risks, they can actually be tested. And so it is very important that machine learning models go through a very significant stress testing exercise, which includes doing validation, as I mentioned previously. But it also includes doing a number of other stress tests, which we'll discuss in a later lecture. A second kind of risk faced with machine learning algorithms relates to social and ethical risks. To illustrate that I'll provide you a few examples that I discovered as I was doing my research on my book. Now one of the people I interviewed as part of my research was a 22 year old biotech professional in China by the name of Yuan Zhang. Now Yuan had a very interesting routine which was that every night before she went to bed, she chatted with the social media celebrity in China called Xiaobing and she engaged in these fun and playful conversations. Xiaobing is a teenage girl who has 40 million followers in China and it's interesting that Xiaobing was able to engage in these conversations with so many of her followers. Now of course, as I was digging deeper into it, I realized that Xiaobing is not a human being. Xiaobing in fact was a chat part that was created by Microsoft Research and was actually very successful in China. Now the same company actually rolled out a chat part in the US much later, it was called Microsoft Tay. And Tay unfortunately engaged in sexist, racist and fascist conversations with many people and had to be shut down within 24 hours of being launched. Well, it's interesting to notice that too similar chat parts launched by the same company had such different outcomes. And that speaks to a little bit about some of the challenges with machine learning algorithms and how they need to be stress tested significantly before they're deployed. Another example of a challenge with machine learning is its application and resume screening. Now large companies like Amazon might receive hundreds of thousands or even millions of resumes in any given year, they have to sift through these millions of resumes and figure out with subset of these applicants to invite for a job interview. It's very hard to do this at scale with human beings. So, a lot of large companies are experimenting with the use of machine learning in order to screen job applicants. In a recent new story by Reuters, it was reported that Amazon discovered that their initial resume screening algorithms had a gender bias. Fortunately, it was discovered by folks at Amazon and that algorithm is no longer in use. But it's interesting to know that even a large company like Amazon had to face this issue where an algorithm that designed based on very cutting edge advanced machine learning algorithms had a gender bias. There's also a new story a few years back that was reported by ProPublica about algorithms used in courtrooms in the US To help judges and parole officers in making bail sentencing and parole decisions. These algorithms look at defendants histories and predict the likelihood that a defendant is likely to re-offend, based on these predictions judges can then make sentencing decisions. The investigation found that the algorithm was twice as likely to falsely predict future criminality in black defendants than white defendants. And this is an example where the algorithm developed a race bias even though such a bias was not programmed in by any developer. Now, the question that obviously arises is how come these kinds of biases are emerging? Why are some resume screening algorithms showing gender bias? Why are some sentencing algorithms racist? Why are some chatbots racist? Notice here that when we talked about design of AI, we talked about how there are rule based approaches to designing AI. But there's also machine learning based approaches to designing AI. So, if you look at what drives the behavior of AI systems, it's partly driven by the logic that the programmer has are the rules that the program is provided with expert systems. They're mostly rule based or the traditional software. They're almost entirely rule based. Now with machine learning, there are rules. But then there's also data and a lot of what is learned is learned from data. So, with human behaviors, we think of human behavior being driven by our nature and our nurture, our nature is our genetic code and that drives some of our behavior. Nurture is our environment and we learn from that environment and that drives some of our behavior, psychologists have attributed problematic behaviors like, let's say alcoholism partly to nature and partly to nurture with the AI, it's no different. If you look at problematic behavior, nature and nurture again plays a role. Nature are the rules that the programmer has created for the AI. Nurture is essentially the data from which the islands if there are biases in the data, then the AI system can pick it up as well. And so in other words, a lot of these biases might exist in the data. When we say a resume screening algorithm has a gender bias. What is actually happening probably, is that it is learning from past data and this past data is based on past decisions made by human beings. Hundreds of thousand people, hundreds of thousands of people have applied for jobs at an organization. People decided who to invite for job interviews. We then look at which of them got a job offer. We then look at which of them got promoted. These are the kinds of people that the AI system is trying to invite for job interviews. If there was a gender bias in the past, then that might have been now captured in the data and in turn is now captured in the AI as well. So, when we think about AI based decisions and some of the risks associated with them, it is often coming from biases in the data. Now, what are some of the risks that are created? First of all, there are many risks to society, especially when automated decisions made by AI based systems can result in disadvantaged minorities continuing to be left behind. The AI Now institute classifies these risks into two groups. The first is harms of allocation, the second is harms of representation. Harms of allocation are essentially about situations where a scarce resource has to be allocated to people. For example, loan approval decisions are job decisions where multiple people apply for a job. But only a few people are going to get it and a resume screening algorithm decides which of the many applicants will get allocated that scarce resource. Harms of representation refers to situation when a system represents a group in an unfavorable way. So for example, if you have a screening system at an airport that is looking at people's facial expressions and other such factors to figure out who needs to be screened. If it has a bias against minorities, then it's a harm of representation. Now, both these are very important harms and we should be worried about them. Now, these are not just risk to societies, but they're also risk to companies because ultimately the social risks also create reputational legal and regulatory risks for organizations. The reputational risk comes from being perceived as a biased or prejudiced company and the PR backlash can result in customers leaving the organization. The legal risk comes from being sued by customers or other such folks for discriminatory practices. And regulatory risks come in when regulators feel that your algorithms are actually discriminating or creating social risks and they now put in a lot of regulation and this creates a cost of compliance. For example, if you look at EU's GDPR regulation which is focused primarily on privacy. It does have some clause that relate to automated decisions. One of them is a right to explanation, and that certainly is one which is a very valuable right that consumers can have and should have. But it also creates compliance risks for companies. So, in short, AI does pose a number of risks. There are risks to society, which is harms of allocation and harms of representation. And in turn, these risks to society create a number of reputational, legal and regulatory risks for companies. The question then, is how do we manage those risks? We will explore that in the next lecture.
