python - Youtube Data Api Storing Comments URL

Question

I'm trying to store youtube comments profile URLs via the data API. Everything works fine but I'm trying to go through all the comments with nextPageToken. I'm able to set a value to it at first but then it won't change and the code goes into an endless loop. Printing the pageToken in the loop just gives out the same value all the time.

API_ENDPOINT = 'https://www.googleapis.com/youtube/v3/commentThreads?key=MYKEY&textFormat=plainText&part=snippet&videoId=0DmA_MCIjy8&maxResults=2&order=time'
r = requests.get(url = API_ENDPOINT)
data = r.json()
for items in data['items']:
    store = str(items['snippet']['topLevelComment']['snippet']['authorChannelUrl'])
    channelUrls.append(store)

pageToken = str(data['nextPageToken'])

while pageToken:
    API_ENDPOINT = 'https://www.googleapis.com/youtube/v3/commentThreads?key=MYKEY&textFormat=plainText&part=snippet&videoId=0DmA_MCIjy8&maxResults=2&&pageToken=' + pageToken

    pageToken= str(data['nextPageToken'])
    print(len(channelUrls))
    for items in data['items']:
        store = str(items['snippet']['topLevelComment']['snippet']['authorChannelUrl'])
        channelUrls.append(store)

question from:https://stackoverflow.com/questions/65846428/youtube-data-api-storing-comments-url

Answer 1

Your code goes into an endless loop because the loop while pageToken: is missing a call to requests.get(url = API_ENDPOINT):

while pageToken:
        API_ENDPOINT = 'https://www.googleapis.com/youtube/v3/commentThreads?key=MYKEY&textFormat=plainText&part=snippet&videoId=0DmA_MCIjy8&maxResults=2&&pageToken=' + pageToken
        r = requests.get(url = API_ENDPOINT)
        data = r.json()
   
        pageToken = data.get('nextPageToken')
        print(len(channelUrls))
        for items in data['items']:
            store = str(items['snippet']['topLevelComment']['snippet']['authorChannelUrl'])
            channelUrls.append(store)

By the way, you could better write your API result set pagination this way:

API_ENDPOINT = 'https://www.googleapis.com/youtube/v3/commentThreads?key=MYKEY&textFormat=plainText&part=snippet&videoId=0DmA_MCIjy8&maxResults=2&order=time'
pageToken = None
channelUrls = []

while True:
    url = API_ENDPOINT
    if pageToken: url += "&pageToken=" + pageToken
    data = requests.get(url = url).json()
    for item in data.get('items', []):
        store = item['snippet']['topLevelComment']['snippet']['authorChannelUrl']
        channelUrls.append(store)
    pageToken = data.get('nextPageToken')
    if not pageToken: break

Also note that your API_ENDPOINT URL may well contain a fields request parameter of the form:

API_ENDPOINT = '...&fields=nextPageToken,items/snippet/topLevelComment/snippet/authorChannelUrl'

Always is a good practice to ask from the API only the info that's of actual use.