google cloud platform - Big query EXPORT DATA statement creating mutiple files with no data and just header record

Question

I have read similar issue here but not able to understand if this is fixed.

Google bigquery export table to multiple files in Google Cloud storage and sometimes one single file

I am using below big query EXPORT DATA OPTIONS to export the data from 2 tables in a file. I have written select query for the same.

EXPORT DATA OPTIONS(
uri='gs://whr-asia-datalake-dev-standard/outbound/Adobe/Customer_Master_'||CURRENT_DATE()||'*.csv',
format='CSV',
overwrite=true,
header=true,
field_delimiter='|') AS     
SELECT

I have only 2 rows returning from my select query and I assume that only one file should be getting created in google cloud storage. Multiple files are created only when data is more than 1 GB. thats what I understand.

However, 3 files got created in cloud storage where 2 files just had the header record and the third file has 3 records(one header and 2 actual data record)

radhika_sharma_ibm@cloudshell:~ (whr-asia-datalake-nonprod)$ gsutil ls gs://whr-asia-datalake-dev-standard/outbound/Adobe/
gs://whr-asia-datalake-dev-standard/outbound/Adobe/
gs://whr-asia-datalake-dev-standard/outbound/Adobe/Customer_Master_2021-02-04000000000000.csv
gs://whr-asia-datalake-dev-standard/outbound/Adobe/Customer_Master_2021-02-04000000000001.csv
gs://whr-asia-datalake-dev-standard/outbound/Adobe/Customer_Master_2021-02-04000000000002.csv

Why empty files are getting created? Can anyone please help? We don't want to create empty files. I believe only one file should be created when it is 1 GB. more than 1 GB, we should have multiple files but NOT empty.

question from:https://stackoverflow.com/questions/66045942/big-query-export-data-statement-creating-mutiple-files-with-no-data-and-just-hea

Answer 1

It turns out you need to enforce multiple files, wildcard syntax. Either a file for CSV or folder for other like AVRO.

The uri option must be a single-wildcard URI as described

https://cloud.google.com/bigquery/docs/reference/standard-sql/other-statements