Issue with Python - KeyError

Question

I'm trying to run a Python code where I'm calling an API to extract the data and upload to a CSV file. The CSV file is extracting fine, but throws an error. Can someone please let me know what I might be dong wrong here?

Error Message:

Code:

import http.client
import json
import csv
import os

conn = http.client.HTTPSConnection("api.betterimpact.com")
conn1 = http.client.HTTPSConnection("api.betterimpact.com")

if os.path.exists("CSVVolunteerOutput.csv"):
  os.remove("CSVVolunteerOutput.csv") 

headers = {
    'Authorization': 'Basic XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
    'Cookie': '; TrackingClientId=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'
}

conn.request("GET", "/v1/enterprise/users/?volunteer_status=accepted&include_custom_fields=false&include_qualifications=false&page_Size=250&include_verified_volunteers_background_check_results=false", headers=headers)
res = conn.getresponse()
data = json.load(res)

conn1.request("GET", "/v1/enterprise/users/?volunteer_status=accepted&include_custom_fields=false&include_qualifications=false&page_Size=250&include_verified_volunteers_background_check_results=false&page_number=0", headers = headers)
res1 = conn1.getresponse()
data1 = json.load(res1)

if data == None or data == "" or len(data) == 0:
    print("Check API Credentials..")
    exit()


volunteer_status = "Accepted"
pageNum = 0
_page_count = data1['header']['page_count']

while True:
    pageNum+=1
    with open('CSVVolunteerOutput.csv', 'a', newline='') as file:
        writer = csv.writer(file)

        writer.writerow(["SyncID", "FirstName", "LastName", "Jobtitle", "Division", "BusinessUnit", "RegionArea", "WorkLocation", "Department", "WorkEmailAddressPrimary",
                         "PersonalEmailAddress", "PersonalMobilePhonePrimary", "WorkCountry"])

        for user in data['users']:
            _id = user['user_id']
            _firstName = user['first_name']
            _surName = user['last_name']
            _emailAddress = user['email_address']
            _emailAddressSec = user['secondary_email_address']
            _cellPhone = user['cell_phone']
            _country = user['country']
            for details in user['memberships']:
                _orgName = details['organization_name']
                _volunteerStatus = details['volunteer_status']
                if volunteer_status == _volunteerStatus:
                    writer.writerow([_id, _firstName, _surName, "Volunteer", "", "", "", _orgName, "", _emailAddress,
                                _emailAddressSec, _cellPhone, _country])

        if pageNum > int(_page_count):
            break
        else:
            conn.request("GET", "/v1/enterprise/users/?volunteer_status=accepted&include_custom_fields=false&include_qualifications=false&page_Size=250&include_verified_volunteers_background_check_results=false&page_number="+str(pageNum), headers=headers)
            res = conn.getresponse()
            data = json.load(res)  

print("CSV file created successfully")

API Documentation is here: https://www.betterimpact.com/volunteer-impact-help/it-api/

Thanks.

question from:https://stackoverflow.com/questions/65643016/issue-with-python-keyerror

Answer 1

I can't run code so I'm guessing.

---

You have data = ... in two places:

• before while-loop
• inside while-loop

like this

# --- first `data` ---

conn.request(...)
res = conn.getresponse()
data = json.load(res)

# ... code ...

while True:
    
    # ... code ...
    
        for user in data['users']:

            # ... code ...

        if pageNum > int(_page_count):
            break
        else:

            # --- second `data` ---

            conn.request(...)
            res = conn.getresponse()
            data = json.load(res)  

It seems you checked users in data only after first data = ... but you didn't check it for second data = ... but it can gives you data without users.

You could check both in one place

 if "users" in data:
      for user in data['users']:

          # ... code ...

---

By the way: if you want to append to file in loop then better append headers before loop. I current version you add headers before every row with data. OR you could append all data to list and after loop write all at once