google cloud platform - Re-run the "program" of an external data source on every plan (refresh state) terraform

Question

I'm using an external data source in order to perform a cURL command using the program argument :

data "external" "curl_zip" {
    program = ["bash", "-c", "curl", ...]
}

I'm running the Terraform in a pipeline so I need to retrieve the data on every terraform plan.

It seemed to work well until I created a new resource requiring the curl to be performed. But it looks like Terraform is only refreshing and so doesn't do the program command after the first plan:

data.external.curl_zip["something.json"]: Refreshing state... [id=-]

My question is : is there a way to re-run the program argument on every plan even during refresh ?

PS : I already tried to use a null_resource instead with a local-exec, turned out to not be the solution here because (for some reason) I also need to use a archive_file data source to create zips files so my GCP app engines resource can read them, and the local-exec is being executed after the terraform apply, which doesn't work since the data source is being refreshed or created during the plan.

question from:https://stackoverflow.com/questions/65836439/re-run-the-program-of-an-external-data-source-on-every-plan-refresh-state-te

Answer 1

It seems to me like you're over-complicating things. What, exactly, are you trying to achieve? With all things Terraform it's usually better to ask how can I achieve so and so? than how can I get my Terraform code to work?.

Is the following what you're after?

data "external" "hello" {
    program = ["bash", "-c", "echo 'Hello World!' > helloworld.txt; echo -n '{"hello":"world!"}'"]
}

resource "null_resource" "world" {
  provisioner "local-exec" {
    command = "echo '${data.external.hello.result["hello"]}'"
  }
}

As you can see from the timestamps in the following output, helloworld.txt is being generated five times, once every time Terraform plan is invoked:

jdsalaro$ for i in {1..5} ;do terraform plan; ls -lah --full-time helloworld.txt ;done 
| grep helloworld.txt | cut -d ' ' -f 7,9

00:04:18.610304219 helloworld.txt
00:04:19.902246088 helloworld.txt
00:04:21.226186506 helloworld.txt
00:04:22.574125835 helloworld.txt
00:04:23.886066774 helloworld.txt

I uploaded the whole example here just in case.