loops - Moving all .csv files which match specific text pattern from multiple subfolders to new folder [Python]

Question

I see many answers on here to similar questions but I cannot seem to adapt it quite yet due to my budding Python skill. I'd like to save the time of individually grabbing the data sets that contain what I need for analysis in R, but my scripts either don't run or seem to do what I need.

I need to 1) loop through a sea of subfolders in a parent folder, 2) loop through the bagillion .csv files in those subs and pick out the 1 that matters (matching text below) and 3) copy it over to a new clean folder with only what I want.

What I have tried:

1)

import os, shutil, glob

src_fldr = 'C:/project_folder_withsubfolders'; 

dst_fldr = 'C:/project_folder_withsubfolders/subfolder_to_dump_into'; 

try:
  os.makedirs(dst_fldr); ## it creates the destination folder
except:
  print ("Folder already exist or some error");

for csv_file in glob.glob(src_fldr+'*statistics_Intensity_Sum_Ch=3_Img=1.csv*'):
    shutil.copy2(csv_file, dst_fldr);

where the text statistics_Intensity_Sum etc is the exact pattern I need for the file to copy over

• this didn't actually copy anything over

2. Making a function that will do this:

srcDir = 'C:/project_folder_withsubfolders'
dstDir = 'C:/project_folder_withsubfolders/subfolder_to_dump_into'
def moveAllFilesinDir(srcDir, dstDir):
    files = os.listdir(srcDir)
    for f in files:
        if f.find("statistics_Intensity_Sum_Ch=3_Img=1"):
            shutil.move(f, dstDir)
        else:
            shutil.move(f, srcDir)
moveAlllFilesinDir(srcDir, dstDir)

• This returned the following error:

  File "C:Usersjbla12AppDataLocalProgramsPythonPython39libshutil.py", line 806, in move
    os.rename(src, real_dst)
FileNotFoundError: [WinError 2] The system cannot find the file specified: 'F1 converted' -> 'C:/Users/jbla12/Desktop/R Analyses/p65_project/sum_files\F1 converted' 

That's because that's a sub-folder I want it to go through! I've tried other methods but don't have record of them in my scripts.

question from:https://stackoverflow.com/questions/65940295/moving-all-csv-files-which-match-specific-text-pattern-from-multiple-subfolders

Answer 1

SOLVED: Special thanks to "Automate the Boring Stuff"

import shutil
import os

dest = 'C:/Users/jbla12/Desktop/R Analyses/p65_project/sum_files/'
src = 'C:/Users/jbla12/Desktop/R Analyses/p65_project/'
txt_ID = 'statistics_Intensity_Sum_Ch=3_Img=1.csv'
def moveSpecFiles(txt_ID, src, dest):
    #src is the original file(s) destination
    #dest is the destination for the files to end up in
    #spec_txt is what the files end with that you want to ID
    for foldername, subfolders, filenames in os.walk(src):
        for file in filenames:
            if file.endswith(txt_ID):
                shutil.copy(os.path.join(foldername, file), dest)
    print('Your files are ready sir/madam!')


moveSpecFiles(txt_ID, src, dest)