performance - Python copy larger file too slow

Question

I am trying to copy a large file (> 1 GB) from hard disk to usb drive using shutil.copy. A simple script depicting what I am trying to do is:-

import shutil
src_file = "sourceolargefile"
dest = "destinationdirectory"
shutil.copy(src_file, dest)

It takes only 2-3 min on linux. But the same file copy on same file takes more that 10-15 min under Windows. Can somebody explain why and give some solution, preferably using python code?

Update 1

Saved the file as test.pySource file size is 1 GB. Destinantion directory is in USB drive. Calculated file copy time with ptime. Result is here:-

ptime.exe test.py

ptime 1.0 for Win32, Freeware - http://www.
Copyright(C) 2002, Jem Berkes <jberkes@pc-t

===  test.py ===

Execution time: 542.479 s

542.479 s == 9 min. I don't think shutil.copy should take 9 min for copying 1 GB file.

Update 2

Health of the USB is good as same script works well under Linux. Calculated time with same file under windows native xcopy.Here is the result.

ptime 1.0 for Win32, Freeware - http://www.pc-tools.net/
Copyright(C) 2002, Jem Berkes <[email protected]>

===  xcopy F:est.iso L:usbest.iso
1 File(s) copied

Execution time: 128.144 s

128.144 s == 2.13 min. I have 1.7 GB free space even after copying test file.

Answer 1

Just to add some interesting information: WIndows does not like the tiny buffer used on the internals of the shutil implementation.

I've quick tried the following:

• Copied the original shutil.py file to the example script folder and renamed it to myshutil.py
• Changed the 1st line to import myshutil
• Edited the myshutil.py file and changed the copyfileobj from

def copyfileobj(fsrc, fdst, length=16*1024):

to

def copyfileobj(fsrc, fdst, length=16*1024*1024):

Using a 16 MB buffer instead of 16 KB caused a huge performance improvement.

Maybe Python needs some tuning targeting Windows internal filesystem characteristics?

Edit:

Came to a better solution here. At the start of your file, add the following:

import shutil

def _copyfileobj_patched(fsrc, fdst, length=16*1024*1024):
    """Patches shutil method to hugely improve copy speed"""
    while 1:
        buf = fsrc.read(length)
        if not buf:
            break
        fdst.write(buf)
shutil.copyfileobj = _copyfileobj_patched

This is a simple patch to the current implementation and worked flawlessly here.

Python 3.8+: Python 3.8 includes a major overhaul of shutil, including increasing the windows buffer from 16KB to 1MB (still less than the 16MB suggested in this ticket). See ticket , diff