Search
 
SCRIPT & CODE EXAMPLE
 

PYTHON

pandas read chunk of csv

import pandas as pd 
import pickle

in_path = "" #Path where the large file is
out_path = "" #Path to save the pickle files to
chunk_size = 400000 #size of chunks relies on your available memory
separator = "~"

reader = pd.read_csv(in_path,sep=separator,chunksize=chunk_size, 
                    low_memory=False)    


for i, chunk in enumerate(reader):
    out_file = out_path + "/data_{}.pkl".format(i+1)
    with open(out_file, "wb") as f:
        pickle.dump(chunk,f,pickle.HIGHEST_PROTOCOL)
Comment
PREVIOUS NEXT
Code Example
Python :: python link to jpg 
Python :: make first row column names pandas 
Python :: python ssh library 
Python :: pandas convert float to int with nan null value 
Python :: round godot 
Python :: python download s3 image 
Python :: python dedent 
Python :: sqlalchemy lock row 
Python :: enumerate in python 
Python :: jsonresponse status code django 
Python :: pandas plot histogram 
Python :: selenium zoom out python 
Python :: language detection python 
Python :: telethon get all channels 
Python :: python write to file csv 
Python :: how to reverse a list in python 
Python :: how to rotate plot in jupyter 
Python :: how to import iris dataset 
Python :: python remove n random elements from a list 
Python :: numpy compute mad 
Python :: get classification report sklearn 
Python :: python isprime 
Python :: decode html python 
Python :: get columns containing string 
Python :: get os information python 
Python :: how to write your first python program 
Python :: convert video to text python 
Python :: list to excel python 
Python :: python replace first 
Python :: sort dictionary 
ADD CONTENT
Topic
Content
Source link
Name
6+6 =