How do I call a Perl script in an SBATCH script for SLURM submissions?

Question

I received a Perl script that apparently is called from an SBATCH script to be submitted as a job to a computer cluster managed by SLURM. The script is old and I am yet to become more familiar with Perl. Additionally, the Perl script is being used as wrapper to call an executable with mpiexec_mpt.

But whenever I do sbatch sbatch_submission, the Perl script is executed by the computer node but I don't obtain any output or execution of the system() method - or I do but I don't know where it is.

I know Perl is executed by SBATCH because I got an error that it couldn't find a module so I manually pointed Perl to the library path using the -l flag as shown below. But after that I don't get any output.

The SBATCH script and the perl script are below:

SBATCH SCRIPT

  1 #!/bin/bash
  2 #SBATCH --job-name=job_submission
  3 #SBATCH --output=output_perl.run
  4 #SBATCH --error=error_perl.run
  5 #SBATCH -n 2 # request 2 cores
  6 #SBATCH --constraint=intel
  7
  8 # Load Needed Modules:
  9 module load mpt
 10
 11 # Set-up environment for perl:
 12 
 13 
 14
 15 # Running perl script:
 16 echo "Calling simple hello_world.c with perl (sbatch)"
 17
 18 perl input_perl.pl 1> perl_in.stdout 2> perl_in.stderr # edit after 
                                                           # suggestions
 19 echo "Done with perl script (sbatch)"
 20

PERL INPUT

  1 #!/usr/bin/perl -w
  2 use strict;
  3 use warnings;
  4 use diagnostics;
  5 use List::MoreUtils qw(indexes); ## edit after suggestions
  6 system("echo this is your hostname:");
  7 system("hostname");
  8 system("mpiexec_mpt -np 2 hello_world");
  9 print "Done executing hello world! from within perl script!
"

OUTPUT FROM STDERR

  1 Can't locate List/MoreUtils.pm in @INC (@INC contains: /usr/lib64/perl5/vendor_perl/List /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /    usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at input_perl.pl line 5.
  

Aside from the output above the output files: perl.output and output_perl.run are empty.

I suspect I am missing something regarding the applicability of the system() method in Perl, as well as how to tell Perl where to send it's output when working with slurm. I have also tried generating a .txt file with the Perl script, but when I run it with SBATCH the .txt file is not generated. I have no issues running the perl_input.pl without using the SBATCH script as wrapper: e.g: perl perl_input.pl.

Additional info, the hello_world executable has been written in C and I have tested it independently and it runs. It is a simple MPI program that lists ranks and size. I don't think that's the issue though.

Independently and running locally the Perl and C scripts run, it's when I use SBATCH that the issues arise.

question from:https://stackoverflow.com/questions/65602693/how-do-i-call-a-perl-script-in-an-sbatch-script-for-slurm-submissions

Answer 1

I am not sure this is a solution but this is what worked for me.

1. Ended up re-installing perl using the instructions here:

https://learn.perl.org/installing/unix_linux.html

I work from a computer cluster so there might be something going on with the perl installation I was using. The module call that was causing the problem seems to be installed but perl can't find it when I call the script from SBATCH.

2. I installed the module I needed: List::MoreUtils as follows:

perl -MCPAN -Mlocal::lib -e 'install List::MoreUtils'

Link: https://www.perlmonks.org/?node_id=1117597

3. Logged out and then started a new session. The new perl installation sources a bashrc file that automatically updates the perl version to the one that was just installed. So whenever you run a perl script you don't use the old installation.

4. After that, I set the PERL5LIB variable to the path where MoreUtils is located and everything worked.