How to Resolve "UnicodeDecodeError: 'utf-8' Codec Can't Decode Byte"

Opening or reading a file crashes with Python complaining it can't decode a specific byte as UTF-8. This means the file isn't actually UTF-8 encoded — it's something else (commonly Windows-1252, Latin-1, or UTF-16), and Python's default assumption of UTF-8 doesn't match the file's real encoding.

The Problem

A file that opens fine in most text editors fails when read in Python:

>>> with open("data.csv") as f:
...     content = f.read()
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/usr/lib/python3.11/codecs.py", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 1847: invalid start byte

It's especially common with files exported from Excel or older Windows applications, which frequently use different default encodings than the modern web-standard UTF-8:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 302: invalid continuation byte

Why It Happens

Text files don't carry explicit metadata about their own encoding (with rare exceptions like a UTF-8 BOM marker) — Python has to be told, or has to guess, what encoding to use when converting the raw bytes into a string. When Python's assumed encoding doesn't match the file's actual encoding, specific byte sequences that are invalid in the assumed encoding trigger this error. Common causes:

• The file was saved with Windows-1252 or Latin-1 encoding — common for files exported from older Windows software, or from Excel's default CSV export on many locales, which use single-byte encodings that overlap with but aren't identical to UTF-8.
• The file is UTF-16, which uses a completely different byte layout (2 bytes per character, plus a byte-order mark) that UTF-8 decoding logic will reject almost immediately.
• The file has mixed encodings — data concatenated or appended from multiple sources over time, each saved with a different tool or encoding assumption.
• Genuinely corrupted data — a truncated download or a file damaged in transit, though this is less common than a simple encoding mismatch.

The Fix

First, try to identify the file's actual encoding rather than guessing randomly. The file command-line tool on Linux/macOS often makes a reasonable guess:

file -i data.csv

data.csv: text/plain; charset=iso-8859-1

For a more reliable detection from within Python, use the chardet or charset-normalizer library, which analyzes the byte patterns statistically:

pip install charset-normalizer

from charset_normalizer import from_path

result = from_path("data.csv").best()
print(result.encoding)

Windows-1252

Once you know the actual encoding, specify it explicitly when opening the file:

with open("data.csv", encoding="windows-1252") as f:
    content = f.read()

If you need the content specifically as UTF-8 going forward (for consistency with the rest of your pipeline), read with the detected encoding and re-save as UTF-8:

with open("data.csv", encoding="windows-1252") as f:
    content = f.read()

with open("data_utf8.csv", "w", encoding="utf-8") as f:
    f.write(content)

If you're processing many files with inconsistent or unknown encodings and can tolerate some data loss on genuinely invalid bytes, an error-handling strategy can get you a usable result without crashing, though this should be a deliberate, visible choice rather than a silent default:

# Replace unreadable bytes with a placeholder character instead of crashing
with open("data.csv", encoding="utf-8", errors="replace") as f:
    content = f.read()

# Or silently drop unreadable bytes entirely
with open("data.csv", encoding="utf-8", errors="ignore") as f:
    content = f.read()

Use errors="replace" or errors="ignore" cautiously — they can silently corrupt or lose data rather than actually fixing the underlying encoding mismatch, and are best reserved for cases where losing a small number of problematic characters is genuinely acceptable for your use case.

Still Not Working?

If detection tools can't confidently identify the encoding, or the file appears to mix multiple encodings within itself, check whether it has a byte-order mark (BOM) indicating UTF-16 or UTF-8-with-BOM specifically, since these need distinct handling from plain UTF-8:

with open("data.csv", "rb") as f:
    print(f.read(4))

b'\xff\xfe...'  # UTF-16 little-endian BOM
b'\xef\xbb\xbf...'  # UTF-8 BOM

If you see a UTF-16 BOM, open explicitly with that encoding rather than UTF-8:

with open("data.csv", encoding="utf-16") as f:
    content = f.read()