So, we started this entire course printing hello world and I just said "Hello world," and out comes hello world. It'd be nice if that was super simple. In 1970, it was simple because there was pretty much one character set. Even in 1970, when I started, we didn't even have a lowercase characters. We just had uppercase characters, and I'll tell you we were happy when we just had uppercase characters. You kids these days with your lowercase characters, and numbers, and slashes, and stuff. So, the problem that computers have is they have to come up with a way, I mean, computers don't understand letters actually what computers understand his numbers. So, we had to come up with a mapping between letters and numbers. So, we came up with a mapping, and there's been many mappings historically. The one that is the most common mapping of the 1980s, is this mapping called ASCII, the American Standard Code for Information Interchange and it says basically this number equals this letter. So for example, the number for Hello World, for capital H the number is 72. Somebody just decided that the capital H was going to be 72 lowercase e, number is 101 and new line is 10. So if you were really and truly going to look at what was going on inside the computer, it's storing these numbers but the problem is, is there are 128 of these, which means that you can't put every character into a 0-128. So, in the early days, we just dealt with whatever characters are possible. Like I said, when I started you could only do uppercase, you couldn't even do lowercase. So, there is this function as long as you're dealing with simple values that you can say, "Hey, what is the actual value for the letter H?" and it's called ord which stands for ordinal. What's the ordinal? What does the number corresponding to H? That's 72. What's the number corresponding to lowercase e? It's 101, and what's a number corresponding to new line? That is a 10. Remember, new line is a single character. This also explains why the lowercase letters are all greater than the uppercase letters because they're ordinal. For ASCII now, there's so many characters sites but just for the default old school 128 characters that we could represent with ASCII, the uppercase letters were had a lower ordinal than lowercase letters. So, Hi is less than z, z, z, all lowercase and that's because all lowercase letters are less. I mean all uppercase letters are less than all lowercase letters actually this could be a, a, a. That's what I should've said there, okay. So, don't worry about that just know that they are all numbers and in the early days, life was simple. We would store every character in a byte of memory otherwise known as 8-bits of memory, it's the same thing when you say I have a many gigabyte USB stick, that's a 16 gigabyte USB stick that means there are 16 billion bytes of memory on there which means we could put 16 million characters down here in the old days. Okay? So, the problem is, is the old days, we just had so few characters that we could put one character in a byte. So, the ord function tells us the numeric value of a simple ASCII character. So, like I said, if you take a look at this, the e is 101 and H, capital H is 72 and then the new line which is here at line feed which is 10. Now, we can represent these in hexadecimal which is base 16, or octal which is base eight, or actual binary which is what's really going on which has nothing but zeroes and ones, but these are just this is the binary for 10, 0001010 and so these are just, these three are just alternate versions of these numbers. The numbers go up to 127, and if you look at the binary, you can see in this, this is actually seven bits or binary, you can see that it's all one. So, it starts at all zeros goes into all ones. So, it's like zeros and ones are what the computer's always do. If you go all the way back to the hardware, the little wires and stuff, the wires or character are taking zeros and ones. So, this is what we did in the 60s and 70s we just said whatever we're capable of squeezing in, we're just totally happy, we're not going to have anything tricky and like I said, halfway you know early in my undergraduate career, I started to see lowercase letters I'm like, "Oh that's really beautiful." lowercase letters. Now, the real world is nothing like this. There are all kinds of characters and they had to come up with a scheme by which we could map these characters and for awhile there were a whole bunch of incompatible ways to represent characters other than these ASCII, also known as Latin character sets, also known as Arabic character sets. These other character sets just completely invented their own way of representing and so, you have these situations where you know Japanese computers pretty much couldn't talk to American computers or European computers at all. I mean the Japanese computers just had their own way of representing characters and American computers had their own way of representing characters and they just couldn't talk, but they invented this thing called Unicode. So, Unicode is this universal code for hundreds of millions of different characters and hundreds of different character sets, so that instead of saying, "Oh sorry. You don't fit with your language from some South Sea Island." it's okay we've got space in Unicode for that. So, Unicode has lots and lots of character not 128, lots and lots of characters. So, there was a time like I said in the 70s or the 80s where everyone has something different and even like in the early 2000s as the Internet, what happened was as the Internet came out, it became unimportant issue to have as a way to exchange data. So we had to say, "Oh well, it's not sufficient for Japanese computers to talk to Japanese computers and American computers to talk to America computers when Japanese and American commuters to exchange data.". So, they built these character sets and so there is Unicode which is this abstraction of all different possible characters and there are different ways of representing them inside of computers. So, there's a couple of simple things that you might think are good ideas that turn out to be not such good ideas, although they're used. So the first thing we did is these UTF-16, UTF-32 and UTF-8 are basically ways of representing a larger set of characters. Now the gigantic one is 32-bits which is four bytes, it's four times as much data for a single character and so that's quite a lot of data. So, you're dividing the number of characters by four so if this is 16 gigabytes it can only still, it can only handle four billion characters or something, divided by four, right? Four bytes per character and so that's not so efficient and then some there's a compromise or like two bytes but then you have to pick this can do all the characters, this can do lots of single lots of character sets, but it turns out that even though you might instinctively think that like UTF-32 is better than UTF-16 and UTF-8 is the worst. It turns out that UTF-8 is the best. So UTF-8 basically says, it's either going to be one, two, three or four characters and there's little marks that tell it when to go from one to four. The nice thing about it is that UTF overlaps with ASCII. Right? So, if the only characters you're putting in are the original ASCII or Latin-1 character set, then UTF-8 and ASCII are literally the same thing. Then, use a special character that's not part of ASCII to indicate flipping from one byte characters to two byte characters, or three byte characters, or four byte. So, it's a variable length. So, you can automatically detect, you can just be reading through a string and say, "Whoa, I just saw this weird marker character, I must be in UTF-8." Then if I'm in UTF-8, than I can expand this, and find represent all those character sets, and all those characters in those character sets. So, what happened was, is they went through all these things, and as you can see from this graph, the graph doesn't really say much other than the fact that UTF-8 is awesome, and getting awesomer, and every other way of representing data is becoming less awesome, right? This is 2012, so that's a long time ago. So, this is like UTF-8 rocks. That's really because as soon as these ideas came out, it was really clear that UTF-8 is the best practice for encoding data moving between systems, and that's why we're talking about this right now. Finally, with this network, we're doing sockets were moving data between systems. So, you're American computer might be talking to a computer in Japan, and you got to know a character sets coming out, right? You might be getting Japanese characters even though everything I've shown you is non-Japanese characters or orient our Asian characters or whatever, right? So, UTF-8 turns out to be the best practice. If you're moving a file between ASCII systems or if you're moving network data between two systems, the world recommends UTF-8, okay? So, if you think about your computer, inside your computer, the strings that are inside your Python like x equals hello world, we don't really care what their syntax is. If there is a file, usually the Python running on the computer in the file had the same character set, they might be UTF-8 inside Python, it might be UTF-8 inside. But we don't care, you open a file, and that's why we didn't have to talk about this we're opening files even though you might someday encounter a file that's different than your normal character set, it's rare. So, files are inside the computer, strings are inside the computer, but network connections are not inside the computer, and we get databases, we're going to see they're not inside the computer either. So, this is also something that's changed from Python two to Python three, it was actually a big deal, a big thing. Most people think it's great, I actually think it's great. Some people are grumpy about it, but I think those people just are people that fear change. So, there were two strings in Python. There were a normal old string and a Unicode string. So, you could see that Python two would be able to make a string constant, and that's type string, and it would make a Unicode constant by prefixing U before the quote. That's a separate thing, and then you had to convert back and forth between Unicode and strings. What we've done in Python three is, this is a regular string, and this is a Unicode string, but you'll notice they're both strings. So, it means that inside the world of Python, if we're pulling stuff and you might have to convert it, but inside Python everything is Unicode. You don't have to worry about it every string is the same, whether it has Asian characters, or Latin characters, or Spanish characters, or French characters, is just fine. So, this simplifies this. But, then there are certain things that we're going to have to be responsible for. So, the one string that we haven't used yet, but becomes important, and it's present in both Python two and Python three. Remember how I said in the old days a character, and a byte are the same thing. So, there's always been a thing like a byte string, and I knew this by prefixing the b, and that says, "This is a string of bytes" that mean this character. If you look at the byte string in Python two, and then you look at a regular string in Python two they're both type string the bytes are the same as string and a Unicode is different. So, these two are the same in Python two and these two are different in Python two. I am not doing a very good picture of that. So, the byte string and the regular string are the same, and the regular string and the Unicode string are different. So, what happened is in Python three, the regular string and the Unicode string are the same, and now, the byte string and the regular string are different, okay? So, bytes turns out to be wrong on encoded, that might be UTF-8 might be UTF-16, it might be ASCII. We don't know what it is, it we don't know what it's encoding is. So, it turns out that this is the thing we have to manage when dealing with data from the outside. So, in Python three, all the strings internally are unicode, not UTF-8, not UTF-16 not UTF-32, and if you just open a file pretty much usually works, if you talk to a network now, we have to understand. Now, the key thing is is we have to decode this stuff, we have to realize what is the character set of the stuff we're pulling in. Now, the beauty is, it's because 99 percent or maybe a 100 percent of stuff you're ever going to run across just uses UTF-8, it turns out to be relatively simple. So, there's this little decode operation. So, if you look at this code right here, when we talked to an external resource, we get a byte array back like the socket gives us an array of bytes which are characters, but they need to be decoded. So, we know if you have UTF-8, UTF-16 or ASCII. So, there is this function that's part of byte arrays, so data.decode says, "Figure this thing out" and the nice thing is as you can tell it what character set it is, but by default it assumes UTF-8 or ASCII dynamically because ASCII and UTF-8 are up it's compatible with one another. So, if it's like old data you're probably getting ASCII if it's newer data your pride getting UTF-8. Literally, it's a law of diminishing returns like it's one, and it's very rare that you get anything other than those two. So, you just almost never have to tell it what it is, right? So, you just say decode it, look at it, it might be ASCII, it might be UTF-8, but whatever it is, by the time it's done with that, it's a string, it's all Unicode inside of this. So, this is bytes, and this is Unicode. So, decode goes from bytes to Unicode. You also can see when we're looking at the sending of the data, we're going to turn it into bytes. So, encode takes this string, and makes it into bytes. So, this is going to be bytes that are properly encoded in UTF-8. Again, you could have put a thing here UTF-8, but it just assumes UTF-8, and this is all ASCII. So, it actually doesn't do anything. So, but that's, okay. Then, we're sending the bytes out the commands. So, we have we have to send the stuff out, then we receive it, we decode it, we send it, we encode it. Now, in this world is where the UTF-8 is. Here, we just have Unicode, and so before we do the send, and before we receive, we have to encode, and decode this stuff, so that it works out and works out correctly. So you can look at the documentation for both the encoder then decode. Decode is a method in a bytes class, and it says, "You can see that the encoding" We're telling it you can say, "It's not UTF-8". Asking UTF-8 or in the same thing the default to UTF-8, which is probably all you're ever going to use, and the same thing is through strings can be encoded using UTF-8 into a byte array, and then we send that byte array out to the outside world. It sounds more complex than it is. So, after all that, think of it this way. On the way out, we have an internal string before we send it. We have to encode it, and then we send it. Getting stuff back, we receive it. It comes back as bytes, we happen to know it's UTF-8 or we're letting it automatically detect UTF-8, and decode it, and now we have a string. Now, internally inside of Python, we can write files, we can do all stuff in and out of this stuff. In a solar works all together it's just that this is UTF-8 question mark, question mark this is the outside world. So, you have to look at your program and say okay, "When am I talking to the outside world?" Well, in this case, it's when I'm talking to a socket, right? I'm talking to a socket, so, I have to know enough to encode and decode as I go in and out of the socket. So, it looks weird when you all started start seeing these n-codes and decodes, but they actually make sense. They're like this barrier between this outside world, and our inside world. So, that inside our data is all completely consistent, and we can mix strings from various sources without regard to the character set of those strings. So, now we're going to do is, we're going to rewrite that program. It's a short program, but we're going to make it even shorter.
