How to Fix "MemoryError" When Processing Large Files in pandas/Python

Loading a large CSV or dataset into pandas crashes the process instead of completing, because the entire file is being loaded into RAM at once and there simply isn't enough available. The fix is almost always about processing the data in smaller pieces rather than requiring a bigger machine.

The Problem

A straightforward read that works fine on smaller files fails on a larger one:

>>> import pandas as pd
>>> df = pd.read_csv("large_dataset.csv")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
MemoryError

Sometimes the process gets killed by the OS instead of raising a clean Python exception at all, especially in memory-constrained environments like containers:

Killed

$ dmesg | grep -i oom
Out of memory: Killed process 8821 (python3) total-vm:8234012kB, anon-rss:7921456kB

Why It Happens

By default, pd.read_csv() and similar functions load the entire dataset into memory as one DataFrame before you can do anything with it, and pandas' in-memory representation is often significantly larger than the file's on-disk size — sometimes 2-5x larger, depending on data types. This error appears when:

• The file is simply too large for available RAM to hold as one complete DataFrame, especially common on smaller cloud instances or local development machines.
• Column data types are inefficient — pandas defaults to 64-bit integers and floats, and generic object dtype for strings, all of which use significantly more memory than more specific, appropriately-sized types would.
• Multiple copies of the data exist simultaneously in memory during processing — an intermediate transformation step that creates a new DataFrame without releasing the old one, effectively doubling memory usage temporarily.
• Other processes on the same machine are also competing for RAM, leaving less available than the total system memory might suggest.

The Fix

The most direct fix is reading the file in chunks instead of all at once, processing each chunk and discarding it before moving to the next:

chunk_size = 100_000
results = []

for chunk in pd.read_csv("large_dataset.csv", chunksize=chunk_size):
    processed = chunk[chunk["amount"] > 0].groupby("category")["amount"].sum()
    results.append(processed)

final_result = pd.concat(results).groupby(level=0).sum()

This keeps memory usage bounded to roughly one chunk's worth of data at a time, regardless of how large the total file is. If you only need specific columns, tell pandas to skip loading the rest entirely, which reduces memory usage proportionally:

df = pd.read_csv("large_dataset.csv", usecols=["date", "category", "amount"])

Specify more memory-efficient data types explicitly instead of relying on pandas' generous defaults:

dtypes = {
    "category": "category",   # much smaller than generic object/string for repeated values
    "amount": "float32",       # half the size of the default float64
    "quantity": "int16",       # smaller than the default int64 if values fit
}
df = pd.read_csv("large_dataset.csv", dtype=dtypes)

The category dtype in particular is highly effective for columns with a limited number of repeated string values (like a status field or a country code), since pandas stores each unique value once and references it by a small integer internally rather than repeating the full string for every row.

For genuinely large datasets that don't fit comfortably in memory even with these optimizations, consider a library designed for out-of-core or distributed processing instead of forcing everything through pandas' in-memory model:

# Dask mirrors much of the pandas API but processes data lazily, in parallel chunks
import dask.dataframe as dd

ddf = dd.read_csv("large_dataset.csv")
result = ddf[ddf["amount"] > 0].groupby("category")["amount"].sum().compute()

Still Not Working?

If you've applied chunking and efficient dtypes but still hit memory limits, check whether an earlier step in your pipeline is holding onto references to old DataFrames that should have been released, preventing Python's garbage collector from freeing that memory:

import gc

del df  # remove the reference explicitly
gc.collect()  # force garbage collection

Also profile actual memory usage per step rather than guessing where the peak is occurring, since the real bottleneck is sometimes in an unexpected transformation step rather than the initial file read:

pip install memory_profiler

from memory_profiler import profile

@profile
def process_data():
    df = pd.read_csv("large_dataset.csv")
    ...

Running this reports memory usage line by line, which makes it far easier to pinpoint exactly which operation is responsible for the largest memory spike rather than assuming it's simply the file size itself.