python - Convert a JSON buffer into .gz

Question

I have the following code saving a pandas data-frame to json in memory and then loading it to AWS S3:

json_buffer = StringIO()
df.to_json(json_buffer, orient='records', date_format='iso')
json_file_name = file_to_load.split(".")[0] + ".json"
s3_conn.put_object(Body=json_buffer.getvalue(), Bucket=s3_bucket, Key=f"{target_path}{json_file_name}")

What I am trying to do is to archive the json file in gzip (.gz) format just before I put it in the S3 bucket. Do you have any ideas on how to achieve that?

Thank you!

question from:https://stackoverflow.com/questions/65897170/convert-a-json-buffer-into-gz

Answer 1

You can use the following using gzip:

with gzip.open(json_file_name, 'wt', encoding='UTF-8') as zipfile:
    json.dump(data, zipfile)

This will save the json file in gzip format. So call is before passing to the S3 buckt.