Python: Issues downloading file from google drive api, getting daily limit for unauthenticated use exceeded

Question

Problem Statement:

I am being plagued with a single issue when trying to download a file from the google drive api for my python app. I just started working with it today so I am probably doing something stupid so bear with me :).

The issue I am getting is that after authenticating my app with the drive-api, I can read files and metadata from google drive but I can't download them. The relevant code is posted below for reference.

Steps Followed:

I first followed the directions to enable OAuth2, and get the credentials.json/client_secrets.json. I made sure that the scope was correct in terms of the permissions and then generated my pickle file. From there, I used the code from the docs and quickstart guides to make my code. From there, I couldn't make any progress. I thought that I might have to require a security assessment for my app but I am not planning on publishing it so I thought this level of permissions would be fine for developers. I see many other stack overflow posts about this but I am not getting anything helpful (I thought I followed all the same steps to authenticate and enable drive-api permissions for my app).

I also haven't been able to download a single file yet so I don't think I am hitting the daily limits. I think I am not doing the request authentication correctly but I can't find any documentation on that. I'd appreciate any help, thank you.

Reference Code:

All the code I have is taken straight from the docs. Here are the relevant links:

1. authorization & list files: https://developers.google.com/drive/api/v3/quickstart/python
2. download files: https://developers.google.com/drive/api/v3/manage-downloads
3. delete_file: https://developers.google.com/drive/api/v3/reference/files/delete#auth

Here is the initialization code:

# packages
from __future__ import print_function
import pickle
from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
from apiclient import errors
from googleapiclient.http import MediaIoBaseDownload

# If modifying these scopes, delete the file token.pickle.
SCOPES = ['https://www.googleapis.com/auth/drive'] # set permisions to read/write/delete
creds = None
if os.path.exists('token.pickle'):
    with open('token.pickle', 'rb') as token:
        creds = pickle.load(token)
if not creds or not creds.valid:
    if creds and creds.expired and creds.refresh_token:
        creds.refresh(Request())
    else:
        flow = InstalledAppFlow.from_client_secrets_file(
             'client_secrets.json', SCOPES)
        creds = flow.run_local_server(port=0)
    with open('token.pickle', 'wb') as token:
        pickle.dump(creds, token)
drive = build('drive', 'v3', credentials=creds)

And here is my drive download function:

def download_from_drive_to_local(drive, fname):
    # search for image in drive
    file_search = drive.files().list(
        q=f"name = '{fname}'", 
        spaces='drive', 
        fields="nextPageToken, files(id, name)").execute()
    items = file_search.get('files', [])
    print('Files retrieved: ', items)
    
    # download retrieved image from drive
    item_ids = [i['id'] for i in items]
    if len(item_ids) > 1: print("Warning: multiple files exist with the same name. Using first file found.")
    for i in items:
        request = drive.files().get_media(fileId=i)
        fh = io.BytesIO()
        downloader = MediaIoBaseDownload(fh, request)
        done = False
        while done is False:
            status, done = downloader.next_chunk()
        break

    # delete retrieved image from drive
    for i in items:
        try:
            drive.files().delete(fileId=i).execute()
        except errors.HttpError as error:
            print(f'An error occured deleting file id <{i}>: {error}.')
    # write bytearray to file
    with open(os.path.join(self.download_dir, f'{fname}.tif'), 'wb') as file: file.write(fh)

And my erroneous result:

Clicking the link gives me this:

Answer 1

Modification points:

• When I saw your script, I thought that items of for i in items: is from items = file_search.get('files', []). In this case, i is like {'name': '###', 'id': '###'}. And when this is used as i of request = drive.files().get_media(fileId=i), the file is is used as {'name': '###', 'id': '###'}. I thought that this is the reason of your issue.

• And, when you want to save the downloaded data as a file using fh = io.BytesIO(), the script for saving it is as follows.

    with io.open(filename, 'wb') as f:
        fh.seek(0)
        f.write(fh.read())
  

• When the length of item_ids is 0, an error occurs.

When above points are reflected to your script, it becomes as follows.

Modified script:

From:

# download retrieved image from drive
item_ids = [i['id'] for i in items]
if len(item_ids) > 1: print("Warning: multiple files exist with the same name. Using first file found.")
for i in items:
    request = drive.files().get_media(fileId=i)
    fh = io.BytesIO()
    downloader = MediaIoBaseDownload(fh, request)
    done = False
    while done is False:
        status, done = downloader.next_chunk()
    break

# delete retrieved image from drive
for i in items:
    try:
        drive.files().delete(fileId=i).execute()
    except errors.HttpError as error:
        print(f'An error occured deleting file id <{i}>: {error}.')
# write bytearray to file
with open(os.path.join(self.download_dir, f'{fname}.tif'), 'wb') as file: file.write(fh)

To:

# download retrieved image from drive
item_ids = [i['id'] for i in items]
if len(item_ids) > 1:
    print("Warning: multiple files exist with the same name. Using first file found.")
for i in item_ids:
    request = drive.files().get_media(fileId=i)
    fh = io.BytesIO()
    downloader = MediaIoBaseDownload(fh, request)
    done = False
    while done is False:
        status, done = downloader.next_chunk()
    break

# delete retrieved image from drive
for i in item_ids:
    try:
        drive.files().delete(fileId=i).execute()
    except errors.HttpError as error:
        print(f'An error occured deleting file id <{i}>: {error}.')

# write bytearray to file
if item_ids != []:
    with io.open(os.path.join(self.download_dir, f'{fname}.tif'), 'wb') as f:
        fh.seek(0)
        f.write(fh.read())