Codectionary / Developer documentation / Python

Bytes and text encoding

Convert between Unicode text and byte sequences deliberately at file and network boundaries.

How it works

str represents Unicode text; bytes represents immutable integers from zero to 255. An encoding defines how characters map to bytes. Encoding converts text to bytes, while decoding interprets bytes as text.

A character can occupy more than one UTF-8 byte. Therefore the length of text and the length of its encoded representation need not match. bytearray is a mutable byte sequence, not a mutable Unicode string.

Choices and common mistakes

Use the encoding required by the format or protocol. Guessing or silently ignoring decoding errors can lose data.

Do not use str(payload) to decode bytes: it produces a representation such as b'...'. Use decode with the correct encoding.

Try it yourself

Encode a string containing an accented character, compare the two lengths and decode it again. Then deliberately decode invalid UTF-8 and handle the error.

Syntax

payload = text.encode("utf-8")
text = payload.decode("utf-8")

Examples

Round-trip Unicode text

Show why byte counts differ from character counts.

text = "café"
payload = text.encode("utf-8")
print(len(text), len(payload))
print(payload.decode("utf-8"))
  1. The final character uses two bytes in UTF-8, so four Unicode code points become five bytes.
  2. decode applies the same encoding and reconstructs the original text. Neither operation changes text in place.
  3. Use byte lengths for network payload limits and text lengths only when they match the application’s character-counting rules.
  4. Expected output: 4 5 café

Handle invalid input

Reject bytes that are not valid under the chosen encoding.

payload = bytes([255])
try:
    print(payload.decode("utf-8"))
except UnicodeDecodeError:
    print("Invalid UTF-8")
  1. bytes([255]) creates a one-byte sequence. That byte cannot stand alone as valid UTF-8.
  2. decode raises before print receives text. The handler reports the decoding failure without silently dropping the byte.
  3. At an import boundary, report the problem or request the correct encoding instead of pretending the content was decoded successfully.
  4. Expected output: Invalid UTF-8

Edit a binary buffer

Use bytearray when bytes need to be changed.

buffer = bytearray(b"cat")
buffer[0] = ord("b")
print(bytes(buffer))
print(buffer.decode("ascii"))
  1. Index assignment requires an integer byte value. ord supplies the ASCII value for b, replacing the first byte.
  2. bytes creates an immutable copy; decode turns the buffer into text. The two printed representations have different types.
  3. Use a mutable buffer for binary processing, and decode only when the byte layout actually represents text.
  4. Expected output: b'bat' bat

Best practices

  • Use the encoding required by the format or protocol. Guessing or silently ignoring decoding errors can lose data.
  • Do not use str(payload) to decode bytes: it produces a representation such as b'...'. Use decode with the correct encoding.

Runtime & versions

Python interpreter. Feature availability depends on the language and runtime version used by your project; consult the official reference below.

Python language reference

Specifications & further reading