Pandas

655 guides

keyboard_arrow_down

Other math topics

Dagster

Pandas

NumPy

Matplotlib

PySpark

MySQL

chevron_leftCreating DataFrames Cookbook

Combining multiple Series into a DataFrame Combining multiple Series to form a DataFrame Converting a Series to a DataFrame Converting list of lists into DataFrame Converting list to DataFrame Converting percent string into a numeric for read_csv Converting scikit-learn dataset to Pandas DataFrame Converting string data into a DataFrame Creating a DataFrame from a string Creating a DataFrame using lists Creating a DataFrame with different type for each column Creating a DataFrame with empty values Creating a DataFrame with missing values Creating a DataFrame with random numbers Creating a DataFrame with zeros Creating a MultiIndex DataFrame Creating a Pandas DataFrame Creating a single DataFrame from multiple files Creating empty DataFrame with only column labels Filling missing values when using read_csv Importing Dataset Importing tables from PostgreSQL as Pandas DataFrames Initialising a DataFrame using a constant Initialising a DataFrame using a dictionary Initialising a DataFrame using a list of dictionaries Inserting lists into a DataFrame cell Keeping leading zeroes when using read_csv Parsing dates when using read_csv Preventing strings from getting parsed as NaN for read_csv Reading data from GitHub Reading file without header Reading large CSV files in chunks Reading n random lines using read_csv Reading space-delimited files Reading specific columns from file Reading tab-delimited files Reading the first few lines of a file to create DataFrame Reading the last n lines of a file Reading URL using read_csv Reading zipped csv file as a DataFrame Removing Unnamed:0 column Resolving ParserError: Error tokenizing data Saving DataFrame as zipped csv Skipping rows without skipping header for read_csv Specifying data type for read_csv Treating missing values as empty strings rather than NaN for read_csv

check_circle

Mark as learned

thumb_up

thumb_down

chat_bubble_outline

Comment

auto_stories Bi-column layout

settings

Reading large CSV files in chunks in Pandas

schedule Aug 11, 2023

Last updated

local_offer

Python●Pandas

Example

Consider the following sample.txt file:

To read this file in chunks of two rows, set chunksize like so:


        
        
            
                
                
                    for chunk in pd.read_csv("sample.txt", chunksize=2):
    print(chunk)
    print("-----")
                
            
               A  B
0  1  2
1  3  4
-----
   A  B
2  5  6
3  7  8
-----
   A   B
4  9  10
-----

Each chunk is a DataFrame, allowing you to work with the dataset piece by piece if you do not need the whole dataset in memory at one time.

Published by Arthur Yanagisawa

Edited by 0 others

Did you find this page useful?

thumb_up

thumb_down

Comment

Citation

Ask a question or leave a feedback...

thumb_up

thumb_down

chat_bubble_outline

settings

Enjoy our search

Hit / to insta-search docs and recipes!