Python

Search

remove duplicate rows in csv file python

# credit to the Stack Overflow user in the source link

with open('file_with_duplicates.csv','r') as in_file, open('ouput.csv','w') as out_file:
  
    seen = set() # set for fast O(1) amortized lookup
    
    for line in in_file:
        if line in seen: 
          continue # skip duplicate

        seen.add(line)
        out_file.write(line)

Comment

PREVIOUS	NEXT

Code Example
Python :: django models distinct
Python :: pandas string does not contain
Python :: python code to plot pretty figures
Python :: all combination of params
Python :: get rid of n in string python
Python :: ready command discord.py
Python :: find duplicate in dataset python
Python :: get request header flask
Python :: python clock
Python :: jupyter notebook change default directory
Python :: python speech recognition module
Python :: set text and background color in pandas table
Python :: python turtle shooting game
Python :: run django server
Python :: how to remove duplicate files from folder with python
Python :: matplotlib rc params
Python :: from sklearn.externals import joblib instead use.....
Python :: resample python numpy
Python :: how to draw a bar graph in python
Python :: https flask
Python :: python module with alphabet list
Python :: pandas absolute value
Python :: print type(x) in python
Python :: python get nth letter of alphabet
Python :: identify the common columns between two dataframes pandas python
Python :: how to delete nan values in python
Python :: Local to ISO 8601 with TimeZone information (Python 3):
Python :: python 1 to 01
Python :: python insert object into list
Python :: binary string to hex python

Search

PYTHON

remove duplicate rows in csv file python

ADD CONTENT