pickle - How to "stop" and "resume" long time running Python script?

Question

I wrote Python script that processes big number of large text files and may run a lot of time. Sometimes, there is a need to stop the running script and to resume it later. The possible reasons to stop the script are program crash, disk 'out of space' situation or many others when you have to do it. I want to implement kind of "stop/resume" mechanism for the script.

• On stop: the script quits & saves its current state.
• On resume: the script starts, but continues from the latest saved state

I'm going to implement it using the pickle and the signal modules.

I'll be glad to hear how to do it in pythonic way.

Thank you!

Answer 1

Here is something simple that hopefully can help you:

import time
import pickle


REGISTRY = None


def main(start=0):
    """Do some heavy work ..."""

    global REGISTRY

    a = start
    while 1:
        time.sleep(1)
        a += 1
        print a
        REGISTRY = pickle.dumps(a)


if __name__ == '__main__':
    print "To stop the script execution type CTRL-C"
    while 1:
       start = pickle.loads(REGISTRY) if REGISTRY else 0
        try:
            main(start=start)
        except KeyboardInterrupt:
            resume = raw_input('If you want to continue type the letter c:')
            if resume != 'c':
                break

Example of running:

$ python test.py
To stop the script execution type CTRL-C
1
2
3
^CIf you want to continue type the letter c:c
4
5
6
7
8
9
^CIf you want to continue type the letter c:
$ python test.py