performance - Python: Why is list comprehension slower than for loop

Question

Essentially these are the same functions - except list comprehension uses sum instead of x=0; x+= since the later is not supported. Why is list comprehension compiled to something 40% slower?

#list comprehension
def movingAverage(samples, n=3): 
    return [float(sum(samples[i-j] for j in range(n)))/n for i in range(n-1, len(samples))]

#regular
def moving_average(samples, n=3):
    l =[]
    for i in range(n-1, len(samples)):
        x= 0
        for j in range(n): 
            x+= samples[i-j]
        l.append((float(x)/n))
    return l

For timing the sample inputs I used variations on [i*random.random() for i in range(x)]

Answer 1

You are using a generator expression in your list comprehension:

sum(samples[i-j] for j in range(n))

Generator expressions require a new frame to be created each time you run one, just like a function call. This is relatively expensive.

You don't need to use a generator expression at all; you only need to slice the samples list:

sum(samples[i - n + 1:i + 1])

You can specify a second argument, a start value for the sum() function; set it to 0.0 to get a float result:

sum(samples[i - n + 1:i + 1], 0.0)

Together these changes make all the difference:

>>> from timeit import timeit
>>> import random
>>> testdata = [i*random.random() for i in range(1000)]
>>> def slow_moving_average(samples, n=3):
...     return [float(sum(samples[i-j] for j in range(n)))/n for i in range(n-1, len(samples))]
... 
>>> def fast_moving_average(samples, n=3):
...     return [sum(samples[i - n + 1:i + 1], 0.0) / n for i in range(n-1, len(samples))]
... 
>>> def verbose_moving_average(samples, n=3):
...     l =[]
...     for i in range(n-1, len(samples)):
...         x = 0.0
...         for j in range(n): 
...             x+= samples[i-j]
...         l.append(x / n)
...     return l
... 
>>> timeit('f(s)', 'from __main__ import verbose_moving_average as f, testdata as s', number=1000)
0.9375386269966839
>>> timeit('f(s)', 'from __main__ import slow_moving_average as f, testdata as s', number=1000)
1.9631599469939829
>>> timeit('f(s)', 'from __main__ import fast_moving_average as f, testdata as s', number=1000)
0.5647804250038462