python - How to I print the number of jobs scheduled by scikit learn?

Question

I plan to run the following code on AWS

from sklearn.metrics import pairwise_distances
pw_distance = pairwise_distances(model_in, model_out, n_jobs=-1)

How do I print the number of jobs scheduled with n_jobs=-1? I'm aware that I can check monitor CPU usage with AWS but I also would like a direct readout.

question from:https://stackoverflow.com/questions/65864663/how-to-i-print-the-number-of-jobs-scheduled-by-scikit-learn

Answer 1

Like this:

from joblib import effective_n_jobs
print(effective_n_jobs(-1))

pairwise_distances uses Parallel under the hood; you can test it if you like:

from time import sleep
from joblib import Parallel, delayed
Parallel(n_jobs=-1, verbose=10)(delayed(sleep)(.2) for _ in range(10))

Output:

[Parallel(n_jobs=-1)]: Using backend LokyBackend with 12 concurrent workers.
[Parallel(n_jobs=-1)]: Done   3 out of  10 | elapsed:    0.6s remaining:    1.3s
[Parallel(n_jobs=-1)]: Done   5 out of  10 | elapsed:    0.6s remaining:    0.6s
[Parallel(n_jobs=-1)]: Done   7 out of  10 | elapsed:    0.6s remaining:    0.3s
[Parallel(n_jobs=-1)]: Done  10 out of  10 | elapsed:    0.6s finished