Use both regular and preemptible machines in Nextflow and Google Lifesciences

Question

Consider having three processes, that run after one another. The first one must not be preempted, the second one can be preempted and the third one must not be preempted.

My config looks like this:

google {
    project = "cool-name"
    region = "cool-region"
    
    lifeSciences {
        bootDiskSize = "200 GB"
        debug = true
        preemptible = false
    }
}

Using preemptible = false like this, all of the three processes cannot be preempted. Is it somehow possible to make the second process preemptible?

question from:https://stackoverflow.com/questions/66047246/use-both-regular-and-preemptible-machines-in-nextflow-and-google-lifesciences

Answer 1

I think you should be able to use a process selector (i.e. withName or withLabel) to specify the required config option(s) for your process:

process {
    withName: 'second_process' {
        google.lifeSciences.preemptible = true
    }
}

This should override the 'default' value you specified in your config for your second process only.

---

Here's an example, just using the echo directive:

Contents of test.nf:

nextflow.enable.dsl=2

process second_process {

    script:
    "echo HelloWorld"
}

workflow {

    second_process()
}

Contents of nextflow.config:

process {
    withName: 'second_process' {
        echo = true
    }
}

Run using:

nextflow run -ansi-log false test.nf

Example output:

N E X T F L O W  ~  version 20.10.0
Launching `test.nf` [nauseous_linnaeus] - revision: 6c35b71798
[78/cc161d] Submitted process > second_process
HelloWorld