Python

Search

how to find duplicates in csv file using python

import csv
import collections

with open(r"T:DataDumpBook1.csv") as f:
    csv_data = csv.reader(f,delimiter=",")

    next(csv_data)  # skip title line

    count = collections.Counter()

    # first pass: read the file
    for row in csv_data:
        address = row[6]
        count[address] += 1

    # second pass: display duplicate info & compute total
    total_dups = 0
    for address,nb in count.items():
        if nb>1:
            total_dups += nb
            print('{} is a duplicate address, seen {} times'.format(address,nb))
        else:
            print('{} is a unique address'.format(address))
    print("Total duplicate addresses {}".format(toal_dups))

Comment

PREVIOUS	NEXT

Code Example
Python :: bash escape double quote windows batch
Python :: how to add hyperlink in jupyter notebook
Python :: Proj 4.9.0 must be installed.
Python :: python dataframe sort by column name
Python :: best python books python 3
Python :: python dataframe show row
Python :: pandas loop over chunk of rows
Python :: django model choice field from another model
Python :: filter field set in django formds
Python :: value list in django
Python :: check if a word is a noun python
Python :: invalid syntax python else
Python :: Write a Python program to remove a key from a dictionary.
Python :: deleting a tuple in python
Python :: keras load model with custom objects
Python :: time zone
Python :: pygame rect
Python :: python class example
Python :: pandas read columns as list
Python :: keras.callbacks.History
Python :: variable globale python
Python :: NumPy flipud Example
Python :: swap two lists without using third variable python
Python :: python compiler online
Python :: how to get cpu model in python
Python :: smooth interpolation python
Python :: // in python
Python :: how to sum all the values in a list in python
Python :: Python DateTime Class Syntax
Python :: python poetry

Search

PYTHON

how to find duplicates in csv file using python

ADD CONTENT