python - Pandas Dataframe - for each row, return count of other rows with overlapping dates

Question

I've got a dataframe with projects, start dates, and end dates. For each row I would like to return the number of other projects in process when the project started. How do you nest loops when using df.apply()? I've tried using a for loop but my dataframe is large and it takes way too long.

import datetime as dt

data = {'project' :['A', 'B', 'C'],
        'pr_start_date':[dt.datetime(2018, 9, 1), dt.datetime(2019, 4, 1), dt.datetime(2019, 6, 8)],
        'pr_end_date': [dt.datetime(2019, 6, 15), dt.datetime(2019, 12, 1), dt.datetime(2019, 8, 1)]}

df = pd.DataFrame(data)

def cons_overlap(start):
    overlaps = 0
    for i in df.index:
        other_start = df.loc[i, 'pr_start_date']
        other_end = df.loc[i, 'pr_end_date']
        if (start > other_start) & (start < other_end):
            overlaps += 1

    return overlaps

df['overlap'] = df.apply(lambda row: cons_overlap(row['pr_start_date']), axis=1)

This is the output I'm looking for:

    pr  pr_start_date pr_end_date   overlap
0   A   2018-09-01    2019-06-15    0
1   B   2019-04-01    2019-12-01    1
2   C   2019-06-08    2019-08-01    2

Answer 1

I suggest you take advantage of numpy broadcasting:

ends = df.pr_start_date.values < df.pr_end_date.values[:, None]
starts = df.pr_start_date.values > df.pr_start_date.values[:, None]
df['overlap'] = (ends & starts).sum(0)
print(df)

Output

  project pr_start_date pr_end_date  overlap
0       A    2018-09-01  2019-06-15        0
1       B    2019-04-01  2019-12-01        1
2       C    2019-06-08  2019-08-01        2

Both ends and starts are matrices of 3x3 that are truth when the condition is met:

# ends   
[[ True  True  True]  
 [ True  True  True]
 [ True  True  True]]

# starts
[[False  True  True]
 [False False  True]
 [False False False]]

Then find the intersection with the logical & and sum across columns (sum(0)).