compilation - If a Julia script is run from the command line, does it need to be re-compiled every time?

Question

I've read through quite some documentation and questions but I'm still confused about this.

In the Profiling section of the documentation it's suggested to first run the target function in the REPL once, so that it's already compiled before being profiled. However, what if the script is fairly complicated and is inteded to be run in the command line, taking arguments? When the julia process finishes and I run the script the second time, is the compilation performed again? Posts like https://stackoverflow.com/a/42040763/1460448, Julia compiles the script every time? give conflicting answers. They also seem to be old while Julia is constantly evolving.

It seems to me that the second run takes exactly as much time as the first run in my experience. The startup time is quite long. How should I optimize such a program? Adding __precompile__() doesn't seem to have changed the execution time at all.

Also, what should I do when I want to profile such a program? All resources on profiling talk about doing so in the REPL.

Answer 1

I disagree somewhat with my colleagues. There are absolutely valid scenarios where one would rely on running julia scripts. E.g. when you have a pipeline of scripts (e.g. matlab, python, etc) and you need to plug in a julia script somewhere in the middle of all that, and control the overall pipeline from a shell script. But, whatever the use case, saying "just use the REPL" isn't a proper answer to this question, and even if one couldn't come up with "valid" scenarios, it is still a question worth answering directly rather than with a workaround.

What I do agree on is that the solution to having appropriate code is to wrap everything critical that needs to be precompiled into modules, and only leave all but the most external commands at the script top-level. This is not too dissimilar to the matlab or C++ world anyway, where you're expected to write thorough functions, and only treat your script / main function as some sort of very brief, top-level entry point whose job is to simply prepare the initial environment, and then run those more specialised functions accordingly.

Here's an example of what I mean:

# in file 'myscript.jl'
push!( LOAD_PATH, "./" )
import MyPrecompiledModule
println( "Hello from the script. The arguments passed into it were $ARGS" )
MyPrecompiledModule.exportedfun()

# in file 'MyPrecompiledModule.jl' (e.g. in the same directory as myscript.jl)
__precompile__()
module MyPrecompiledModule
  export exportedfun;
  function innerfun()
    println("Hello from MyPrecompiledModule.innerfun");
  end

  function exportedfun()
    innerfun()
    print("Hello from MyPrecompiledModule.exportedfun");
  end
end

In the above scenario, the compiled version of the MyPrecompiledModule will be used in the script (and if one does not exist, one will be compiled the first time you run the script), therefore any optimisations from compiling will not be lost at the end of the script, but you still end up with a standalone julia script you can use as part of a bash shell script pipeline process, that you can also pass arguments to. The myscript.jl script then only has to pass these on to the imported module functions if necessary, and perform any other commands that you don't particularly care about them being compiled / optimised or not, such as perform benchmarks, provide script usage instructions, etc.