CSV package in python loads only first 34739332 rows and not beyond that?

Question

I am working on a file with 156371992 rows and using the CSV package of python. But it always loads only the first 34739332. It is not throwing any error which I suppose is because the reader is believing to have reached the end of the file which is far from it. I couldn't find anything docs, am adding the code snippet too

has_header = csv.Sniffer().has_header(csvfile.read(1024))
csvfile.seek(0)
reader = csv.reader(csvfile)
if has_header:
    next(reader)
print("len of reader", len(list(reader)))

Which always gives 34739332 as value. Any explanations?

question from:https://stackoverflow.com/questions/65850855/csv-package-in-python-loads-only-first-34739332-rows-and-not-beyond-that

Answer 1

For datasets, It might be better to import read_csv from pandas rather than use the csv library. Try

from pandas import read_csv
dataset = read_csv(csvfile)

This will create a pandas Dataframe. If you need to manipulate it, the pandas library functions should be adequate. If not, you can import numpy and use dataset = numpy.array(dataset).

If that doesn't work, try importing NumPy and using genfromtxt instead.