Powershell: Get-ChildItem performance to deal with bulk files

Question

The scenario is in a remote server, a folder is shared for ppl to access the log files. The log files will be kept for around 30 days before they got aged, and each day, around 1000 log files will be generated. For problem analysis, I need to copy log files to my own machine, according to the file timestamp.

My previous strategy is:

1. Use dir /OD command to get the list of the files, to my local PC, into a file
2. Open the file, find the timestamp, get the list of the files I need to copy
3. Use copy command to copy the actual log files

It works but needs some manual work, ie step2 I use notepad++ and regular expression to filter the timestamp

I tried to use powershell as:

Get-ChildItem -Path $remotedir | where-object {$_.lastwritetime -gt $starttime -and $_lastwritetime -lt $endtime } |foreach {copy-item $_.fullname -destination .}

However using this approach it took hours and hours and no file has been copied, while compared with the dir solution it took around 7-8 minutes to generate the list of the file than copy itself took sometime but not hours

I guess most of the time spent on the filter file. I'm not quite sure why the get-childitem's performance is so poor.

Can you please advise if there's anything i can change? Thanks

question from:https://stackoverflow.com/questions/65834866/powershell-get-childitem-performance-to-deal-with-bulk-files

Answer 1

For directories with a lot of files, Get-ChildItem is too slow. It looks like most of the time is spent enumerating the directory, then filtering through 'where', then copying each file.

Use .net directly, particularly [io.directoryinfo] with the GetFileSystemInfos() method.

e.g.

$remotedir   = [io.directoryinfo]'\servershare'
$destination = '.'
$filemask    = '*.*'
$starttime   = [datetime]'jan-21-2021 1:23pm'
$endtime     = [datetime]'jan-21-2021 4:56pm'

$remotedir.GetFileSystemInfos($filemask, [System.IO.SearchOption]::TopDirectoryOnly) | % {
    if ($_.lastwritetime -gt $starttime -and $_.lastwritetime -lt $endtime){
        Copy-Item -Path $_.fullname -Destination $destination
    }
}