python - Why does ReplaceFile fail with ERROR_SHARING_VIOLATION?

Question

While the documentation is vague, based on this question and comments and this answer, I expected that ReplaceFile called with the third argument (backup filename) should succeed even if there are handles to source and destination files open in other processes without FILE_SHARE_DELETE flag. It's supposed to be overcome the lock by changing just the file metadata (= directory entry), which is not controlled by the lock. (All three files are on the same disk drive, so changing metadata is enough to rename them.)

However, the code below fails with ERROR_SHARING_VIOLATION. This is not my use case, but just a demonstration of the failure. The use case is that I'm trying to rename files that are occasionally (and unpredictably) open in other processes on the system, such as antivirus or backup programs, which didn't bother to use FILE_SHARE_DELETE flag.

# python 3
import os
import ctypes

fname1 = 'test1.txt'
fname2 = 'test2.txt'
f1 = open(fname1, 'w')
f1.write(fname1)
f2 = open(fname2, 'w')
f2.write(fname2)

# tmp123 does not exist when the program is started
ctypes.windll.kernel32.ReplaceFileW(fname2, fname1, 'tmp123', 0, None, None) # 0
ctypes.GetLastError() # ERROR_SHARING_VIOLATION

# if we close file handles, it works as expected
f1.close()
f2.close()
ctypes.windll.kernel32.ReplaceFileW(fname2, fname1, 'tmp123', 0, None, None) # 1

Why?

Answer 1

I had experienced several times similar issues using Windows DLLs.

For performances optimization, security purposes and a bounch of other low level but nonetheless important details I didn't dig into, DLLs in Windows are loaded once (per platform/version/culture/...). So when you refere them, as you did in your code, if they have been already loaded by some other process they have also already loaded their data, security context and so on.

In this case the call to the DLL even if it's performed by your process has a different context: here is where the sharing thing starts making sense.

As you already know, Python is a general purpose, multiplatform programming language, so it's main aim is to give you an interface to the underlying OS features indipendently of how it exposes them.

Well, it turns out that the windows CreateFile function as a significant high number of options that can be delivered to it and does not apply to other OS, one of which is dwShareMode that dictates how the file is shared with other processes. (Here the ufficial documentation).

If this parameter is zero and CreateFile succeeds, the file or device cannot be shared and cannot be opened again until the handle to the file or device is closed.

You cannot request a sharing mode that conflicts with the access mode that is specified in an existing request that has an open handle. CreateFile would fail and the GetLastError function would return ERROR_SHARING_VIOLATION.

I think core python developers, if using that function at all (not the posix one if supported by windows), had left the value of that variable to 0 leading to the behaviour you're experiencing.

In order for your process to be able to call the DLLs functions that deals with the files created by your own code you have to be sure you do not have "non shared" descriptors open to them.

You have at least two options:

1. Create the file calling directly the Windows API and specifying the sharing behaviour you desire.
2. Opening file through context manager

Solution 2 is easier and more "pythonic" so I'll go with it:

import os
import ctypes

fname1 = 'test1.txt'
fname2 = 'test2.txt'

with open(fname1, 'w') as f1:
   f1.write(fname1)

with open(fname2, 'w') as f2:
   f2.write(fname2)

ctypes.windll.kernel32.ReplaceFileW(fname2, fname1, 'tmp123', 0, None, None)