iterator - Why does Python's itertools.cycle need to create a copy of the iterable?

Question

The documentation for Python's itertools.cycle() gives a pseudo-code implementation as:

def cycle(iterable):
    # cycle('ABCD') --> A B C D A B C D A B C D ...
    saved = []
    for element in iterable:
        yield element
        saved.append(element)
    while saved:
        for element in saved:
              yield element

Below, it states: "Note, this member of the toolkit may require significant auxiliary storage (depending on the length of the iterable)."

I basically was going down this path, except I did this, which does not require creating a copy of the iterable:

def loop(iterable):
    it = iterable.__iter__()

    while True:
        try:
            yield it.next()
        except StopIteration:
            it = iterable.__iter__()
            yield it.next()

x = {1, 2, 3}

hard_limit = 6
for i in loop(x):
    if hard_limit <= 0:
        break

    print i
    hard_limit -= 1

prints:

1
2
3
1
2
3

Yes, I realize my implementation wouldn't work for str's, but it could be made to. I'm more curious as to why it creates another copy. I have a feeling it has to do with garbage collection, but I'm not well studied in this area of Python.

Thanks!

Answer 1

Iterables can only be iterated over once.

You create a new iterable in your loop instead. Cycle cannot do that, it has to work with whatever you passed in. cycle cannot simply recreate the iterable. It thus is forced to store all the elements the original iterator produces.

If you were to pass in the following generator instead, your loop() fails:

def finite_generator(source=[3, 2, 1]):
    while source:
        yield source.pop()

Now your loop() produces:

>>> hard_limit = 6
>>> for i in loop(finite_generator()):
...     if hard_limit <= 0:
...         break
...     print i
...     hard_limit -= 1
... 
1
2
3

Your code would only work for sequences, for which using cycle() would be overkill; you don't need the storage burden of cycle() in that case. Simplify it down to:

def loop_sequence(seq):
    while True:
        for elem in seq:
            yield elem