Representing text

A computer can only store bits. The letter H is a shape on a screen. So before a computer can store H, somebody has to decide a number for it.

1. Give every character a number. H = 72. i = 105. Space = 32. That number is the character code.

2. Store the number in binary. 72 becomes 01001000. You already know how to do that from Topic 3.

3. Write the list down and share it. The full list of characters and their numbers is a character set. It is a lookup table: find the character, read off its number. Or the other way round.

ASCII (said "ass-key") is the American Standard Code for Information Interchange. It was first published in 1963. Lower case letters were added in 1967.

1. 7 bits per character. 7 bits make 2⁷ = 128 patterns, so ASCII has 128 characters, numbered 0 to 127.

2. What's in it: A–Z, a–z, the digits 0–9, punctuation, the space, and 33 invisible control characters.

3. Control characters are instructions, not shapes. 10 = new line. 9 = tab. 8 = backspace. 7 = ring a bell (old teleprinters had real bells).

4. Stored in a byte. Computers store data in whole bytes, so each 7-bit code sits in 8 bits with a 0 in front: A = 01000001.

Why 7 bits gives exactly 128

It's the patterns rule from Topic 2. Each extra bit doubles the number of patterns. So the number of characters a set can hold is 2 to the power of the bits.

1. 1 bit → 2 patterns. 2 bits → 4. 3 bits → 8. 4 bits → 16.

2. 5 bits → 32. Enough for A–Z (26), but not lower case too.

3. 6 bits → 64. Enough for A–Z, a–z (52) and digits (10) = 62. No room for punctuation.

4. 7 bits → 128. Room for everything English needs (about 95 visible characters) plus 33 control codes.

5. 8 bits → 256. Extended ASCII used the extra 128 for é, ñ, £ and box-drawing shapes (Step 6).

ASCII's order was chosen with care. Capitals start at 65. Lower case starts at 97. The gap is exactly 32, and 32 is one binary column.

1. Letters run in order. A = 65, B = 66, C = 67 … Z = 90. So if you know one code you can count to any other letter. D = 65 + 3 = 68.

2. That's how computers sort words. They compare the codes. "cat" comes before "dog" because 99 < 100.

3. The 32 trick. Lower case = capital + 32. In binary that means: switch the 32s bit on. One tiny circuit can change case.

print(ord("A"))          # 65
print(chr(97))           # a
print(chr(ord("g") - 32))  # G: take the 32 away
print("Zebra" < "apple")   # True! Z (90) is smaller than a (97)

The digit you type on a keyboard is a character, like a letter. Its code is not its value. The character '7' has code 55.

1. You type 7 on the keyboard. The computer receives the character '7', code 55.

2. To get the number, take away the code of '0': 55 − 48 = 7.

3. That's the trick: '7' − '0' = 7. It works for every digit, because the digits run in order.

4. In Python, int("7") does this for you. So does ord("7") - ord("0").

print("7" + "7")

print(7 + 7)

To store text, the computer looks up each character, writes its code in binary, and puts the bytes one after another. Spaces and punctuation are characters too.

1. The word is Hi. Look up each character: H → 72, i → 105.

2. 72 in binary: 64 + 8 = 72 → 01001000.

3. 105 in binary: 64 + 32 + 8 + 1 = 105 → 01101001.

4. The computer stores the bytes side by side: 01001000 01101001.

5. Size: 2 characters × 8 bits = 16 bits = 2 bytes.

The text-size formula

Every character takes the same number of bits in ASCII. So the size is just characters × bits each.

1. Count every character. "Hello world" = H, e, l, l, o, space, w, o, r, l, d = 11. The space counts.

2. Bits each: 7 for pure ASCII, 8 when stored in bytes (extended ASCII, and how files really store ASCII).

3. Multiply: 11 × 8 = 88 bits.

4. Into bytes: 88 ÷ 8 = 11 bytes. With 8-bit codes, bytes = characters. Handy!

5. 7-bit version: 11 × 7 = 77 bits. Exams sometimes ask this to check you know ASCII is 7-bit.

Look at the ASCII map again. There's no ñ, no á, no ¿, no €. 7-bit ASCII was made for American English. Spanish can't be written in it.

1. The fix: use the 8th bit. 8 bits give 256 codes. 0–127 stay ASCII. 128–255 are 128 new spaces. This is extended ASCII.

2. The new problem: there was no single "extended ASCII". Every region filled 128–255 with different characters. Each version is called a code page.

3. So one byte had many meanings. Byte 241 is ñ in Western Europe, ± on an old IBM PC, and ё in Russia.

Unicode is one giant character set for every writing system, plus symbols and emoji. Version 1.0 came out in 1991. A new version comes out every year.

1. Every character gets one number, called its code point. It's written U+ followed by hex: ñ is U+00F1.

2. Hex, from Topic 3. F1 in hex = 15 × 16 + 1 = 241. Same number as Latin-1. Unicode copied Latin-1 for its first 256 codes.

3. The first 128 are exactly ASCII. A is still 65 (U+0041). Old English text is already valid Unicode.

4. Room for 1,114,112 code points (U+0000 to U+10FFFF). About 160,000 are used so far. Lots of room left.

How many bits would a fixed size need?

The biggest code point is U+10FFFF = 1,114,111. Using the patterns rule: 2²⁰ = 1,048,576 is too small. 2²¹ = 2,097,152 is enough. So a code point needs 21 bits. Computers like whole bytes, so a fixed-size version (UTF-32) uses 4 bytes for every character.

UTF-8 stores small code points in 1 byte and bigger ones in 2, 3 or 4. English text stays as small as ASCII. Every other script still fits.

1. Starts with 0? It's a 1-byte character: plain ASCII. So ASCII files are already UTF-8.

2. Starts with 110, 1110 or 11110? It's the first byte of a 2, 3 or 4-byte character. Count the 1s at the start: that's how many bytes.

3. Starts with 10? It's a continuation byte: the middle or end of a longer character. Never a start.

4. Why this design is clever: if a byte is lost, the reader skips forward to the next byte that doesn't start with 10, and carries on. One damaged character, not a ruined file.

In ASCII, size = characters × 8 bits. In UTF-8 characters have different sizes, so you add them up instead.

1. Text: España. Split it into characters: E, s, p, a, ñ, a.

2. For each one, find its code point and its UTF-8 size. E, s, p, a, a are ASCII (under 128): 1 byte each.

3. ñ is 241. That's above 127 but below 2,048: 2 bytes.

4. Add: 1 + 1 + 1 + 1 + 2 + 1 = 7 bytes = 7 × 8 = 56 bits.

5. Compare: if the formula S = c × b were used with b = 8, you'd say 6 bytes. It's 1 byte short, because ñ needs 2.

One byte, eight LEDs. Light the LEDs for a character's code and you can see a letter in binary. Flip just the 32s LED and the case changes.

The mistakes nearly everyone makes the first time. Read the left side, then the right.

Quick questions first. Then endless practice: a wrong answer shows you the working, step by step.