sql - How to optimise google BigQuery with 17+ tables which contains approx. 55 GB of data?

Question

I have a huge amount of data store which contains almost 20+ tables. all tables contain data in GB.

So basically I'm exporting all data into CSV for analysis. I have 17+ tables in join query which almost process billions of records. Google says it will process 10 GB data.

Now the problem is query taking too much time & resources, sometimes query fails with resource limit. how can I optimize such a query?

FYI: I'm using LEFT JOIN

question from:https://stackoverflow.com/questions/65920807/how-to-optimise-google-bigquery-with-17-tables-which-contains-approx-55-gb-of

Answer 1

Best way to optimize your query is implement Partitioning & Clustering. Best solution is to implement partitioning and Clustering on fields over which Joining conditions are done.