Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Ask a Question
Welcome To Ask or Share your Answers For Others

Categories

regex - Python/Pandas: How to Match List of Strings with a DataFrame column

0 votes
801 views
in Technique[技术] by (71.8m points)

regex - Python/Pandas: How to Match List of Strings with a DataFrame column

I want to compare two columns: Description and Employer. I want to see if any keywords from Employer are found in the Description column. I have broken the Employer column down to words and converted to a list. Now I want to see if any of those words are in the corresponding Description column.

Sample input:

print(df.head(25))


          Date           Description   Amount  AutoNumber  
0    3/17/2015  WW120 TFR?FR xxx8690   140.00       49246   
2    3/13/2015  JX154 TFR?FR xxx8690   150.00       49246   
5     3/6/2015   CANSEL SURVEY E PAY  1182.08       49246   
9     3/2/2015  UE200 TFR?FR xxx8690   180.00       49246   
10   2/27/2015  JH401 TFR?FR xxx8690   400.00       49246   
11   2/27/2015   CANSEL SURVEY E PAY   555.62       49246   
12   2/25/2015  HU204 TFR?FR xxx8690   200.00       49246   
13   2/23/2015  UQ263 TFR?FR xxx8690   102.00       49246   
14   2/23/2015  UT460 TFR?FR xxx8690   200.00       49246   
15   2/20/2015   CANSEL SURVEY E PAY  1222.05       49246   
17   2/17/2015  UO414 TFR?FR xxx8690   250.00       49246   
19   2/11/2015  HI540 TFR?FR xxx8690   130.00       49246   
20   2/11/2015  HQ010 TFR?FR xxx8690   177.00       49246   
21   2/10/2015  WU455 TFR?FR xxx8690   200.00       49246   
22    2/6/2015  JJ500 TFR?FR xxx8690   301.00       49246   
23    2/6/2015   CANSEL SURVEY E PAY  1182.08       49246   
24    2/5/2015  IR453 TFR?FR xxx8690   168.56       49246   
26    2/2/2015  RQ574 TFR?FR xxx8690   500.00       49246   
27    2/2/2015  UT022 TFR?FR xxx8690   850.00       49246   
28  12/31/2014  HU521 TFR?FR xxx8690   950.17       49246   

                   Employer  
0   Cansel Survey Equipment  
2   Cansel Survey Equipment  
5   Cansel Survey Equipment  
9   Cansel Survey Equipment  
10  Cansel Survey Equipment  
11  Cansel Survey Equipment  
12  Cansel Survey Equipment  
13  Cansel Survey Equipment  
14  Cansel Survey Equipment  
15  Cansel Survey Equipment  
17  Cansel Survey Equipment  
19  Cansel Survey Equipment  
20  Cansel Survey Equipment  
21  Cansel Survey Equipment  
22  Cansel Survey Equipment  
23  Cansel Survey Equipment  
24  Cansel Survey Equipment  
26  Cansel Survey Equipment  
27  Cansel Survey Equipment  
28  Cansel Survey Equipment

I tried something like this, but it doesn't seem to work.:

df['Text_Search'] = df['Employer'].apply(lambda x: x.split(" "))
df['Match'] = np.where(df['Description'].str.contains("|".join(df['Text_Search'])), "Yes", "No")

My desired output would be as shown below:

          Date           Description   Amount  AutoNumber  
0    3/17/2015  WW120 TFR?FR xxx8690   140.00       49246   
2    3/13/2015  JX154 TFR?FR xxx8690   150.00       49246   
5     3/6/2015   CANSEL SURVEY E PAY  1182.08       49246   
9     3/2/2015  UE200 TFR?FR xxx8690   180.00       49246   
10   2/27/2015  JH401 TFR?FR xxx8690   400.00       49246   
11   2/27/2015   CANSEL SURVEY E PAY   555.62       49246   
12   2/25/2015  HU204 TFR?FR xxx8690   200.00       49246   
13   2/23/2015  UQ263 TFR?FR xxx8690   102.00       49246   
14   2/23/2015  UT460 TFR?FR xxx8690   200.00       49246   
15   2/20/2015   CANSEL SURVEY E PAY  1222.05       49246   
17   2/17/2015  UO414 TFR?FR xxx8690   250.00       49246   
19   2/11/2015  HI540 TFR?FR xxx8690   130.00       49246   
20   2/11/2015  HQ010 TFR?FR xxx8690   177.00       49246   
21   2/10/2015  WU455 TFR?FR xxx8690   200.00       49246   
22    2/6/2015  JJ500 TFR?FR xxx8690   301.00       49246   
23    2/6/2015   CANSEL SURVEY E PAY  1182.08       49246   
24    2/5/2015  IR453 TFR?FR xxx8690   168.56       49246   
26    2/2/2015  RQ574 TFR?FR xxx8690   500.00       49246   
27    2/2/2015  UT022 TFR?FR xxx8690   850.00       49246   
28  12/31/2014  HU521 TFR?FR xxx8690   950.17       49246   
29  12/30/2014  WZ553 TFR?FR xxx8690   200.00       49246   
32  12/29/2014  JW173 TFR?FR xxx8690   300.00       49246   
33  12/24/2014   CANSEL SURVEY E PAY  1219.21       49246   
34  12/24/2014   CANSEL SURVEY E PAY   434.84       49246   
36  12/23/2014  WT002 TFR?FR xxx8690   160.00       49246   

                   Employer                  Text_Search Match  
0   Cansel Survey Equipment  [Cansel, Survey, Equipment]    No  
2   Cansel Survey Equipment  [Cansel, Survey, Equipment]    No  
5   Cansel Survey Equipment  [Cansel, Survey, Equipment]    Yes 
9   Cansel Survey Equipment  [Cansel, Survey, Equipment]    No  
10  Cansel Survey Equipment  [Cansel, Survey, Equipment]    No  
11  Cansel Survey Equipment  [Cansel, Survey, Equipment]    Yes  
12  Cansel Survey Equipment  [Cansel, Survey, Equipment]    No  
13  Cansel Survey Equipment  [Cansel, Survey, Equipment]    No  
14  Cansel Survey Equipment  [Cansel, Survey, Equipment]    No  
15  Cansel Survey Equipment  [Cansel, Survey, Equipment]    Yes  
17  Cansel Survey Equipment  [Cansel, Survey, Equipment]    No  
19  Cansel Survey Equipment  [Cansel, Survey, Equipment]    No  
20  Cansel Survey Equipment  [Cansel, Survey, Equipment]    No  
21  Cansel Survey Equipment  [Cansel, Survey, Equipment]    No  
22  Cansel Survey Equipment  [Cansel, Survey, Equipment]    No  
23  Cansel Survey Equipment  [Cansel, Survey, Equipment]    Yes  
24  Cansel Survey Equipment  [Cansel, Survey, Equipment]    No  
26  Cansel Survey Equipment  [Cansel, Survey, Equipment]    No  
27  Cansel Survey Equipment  [Cansel, Survey, Equipment]    No  
28  Cansel Survey Equipment  [Cansel, Survey, Equipment]    No  
29  Cansel Survey Equipment  [Cansel, Survey, Equipment]    No  
32  Cansel Survey Equipment  [Cansel, Survey, Equipment]    No  
33  Cansel Survey Equipment  [Cansel, Survey, Equipment]    Yes  
34  Cansel Survey Equipment  [Cansel, Survey, Equipment]    Yes  
36  Cansel Survey Equipment  [Cansel, Survey, Equipment]    No
See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙；凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Answer

0 votes
by (71.8m points)

Here is fast and memory-saving vectrorized solution, which uses sklearn.feature_extraction.text.CountVectorizer method:

from sklearn.feature_extraction.text import CountVectorizer
vect = CountVectorizer(min_df=1, lowercase=True)

X = vect.fit_transform(df['Employer'])
cols_emp = vect.get_feature_names()

X = vect.fit_transform(df['Description'])
cols_desc = vect.get_feature_names()

common_cols_idx = [i for i,col in enumerate(cols_desc) if col in cols_emp]

df['Match'] = (X.toarray()[:, common_cols_idx] == 1).any(1)

Source DF:

In [259]: df
Out[259]:
          Date           Description   Amount  AutoNumber                 Employer
0    3/17/2015  WW120 TFR?FR xxx8690   140.00       49246  Cansel Survey Equipment
2    3/13/2015  JX154 TFR?FR xxx8690   150.00       49246  Cansel Survey Equipment
5     3/6/2015   CANSEL SURVEY E PAY  1182.08       49246  Cansel Survey Equipment
9     3/2/2015  UE200 TFR?FR xxx8690   180.00       49246  Cansel Survey Equipment
10   2/27/2015  JH401 TFR?FR xxx8690   400.00       49246  Cansel Survey Equipment
11   2/27/2015   CANSEL SURVEY E PAY   555.62       49246  Cansel Survey Equipment
12   2/25/2015  HU204 TFR?FR xxx8690   200.00       49246  Cansel Survey Equipment
13   2/23/2015  UQ263 TFR?FR xxx8690   102.00       49246  Cansel Survey Equipment
14   2/23/2015  UT460 TFR?FR xxx8690   200.00       49246  Cansel Survey Equipment
15   2/20/2015   CANSEL SURVEY E PAY  1222.05       49246  Cansel Survey Equipment
17   2/17/2015  UO414 TFR?FR xxx8690   250.00       49246  Cansel Survey Equipment
19   2/11/2015  HI540 TFR?FR xxx8690   130.00       49246  Cansel Survey Equipment
20   2/11/2015  HQ010 TFR?FR xxx8690   177.00       49246  Cansel Survey Equipment
21   2/10/2015  WU455 TFR?FR xxx8690   200.00       49246  Cansel Survey Equipment
22    2/6/2015  JJ500 TFR?FR xxx8690   301.00       49246  Cansel Survey Equipment
23    2/6/2015   CANSEL SURVEY E PAY  1182.08       49246  Cansel Survey Equipment
24    2/5/2015  IR453 TFR?FR xxx8690   168.56       49246             Cansel IR453
26    2/2/2015  RQ574 TFR?FR xxx8690   500.00       49246  Cansel Survey Equipment
27    2/2/2015  UT022 TFR?FR xxx8690   850.00       49246  Cansel Survey Equipment
28  12/31/2014  HU521 TFR?FR xxx8690   950.17       49246      Cansel Survey HU521

Result:

In [261]: df
Out[261]:
          Date           Description   Amount  AutoNumber                 Employer  Match
0    3/17/2015  WW120 TFR?FR xxx8690   140.00       49246  Cansel Survey Equipment  False
2    3/13/2015  JX154 TFR?FR xxx8690   150.00       49246  Cansel Survey Equipment  False
5     3/6/2015   CANSEL SURVEY E PAY  1182.08       49246  Cansel Survey Equipment   True
9     3/2/2015  UE200 TFR?FR xxx8690   180.00       49246  Cansel Survey Equipment  False
10   2/27/2015  JH401 TFR?FR xxx8690   400.00       49246  Cansel Survey Equipment  False
11   2/27/2015   CANSEL SURVEY E PAY   555.62       49246  Cansel Survey Equipment   True
12   2/25/2015  HU204 TFR?FR xxx8690   200.00       49246  Cansel Survey Equipment  False
13   2/23/2015  UQ263 TFR?FR xxx8690   102.00       49246  Cansel Survey Equipment  False
14   2/23/2015  UT460 TFR?FR xxx8690   200.00       49246  Cansel Survey Equipment  False
15   2/20/2015   CANSEL SURVEY E PAY  1222.05       49246  Cansel Survey Equipment   True
17   2/17/2015  UO414 TFR?FR xxx8690   250.00       49246  Cansel Survey Equipment  False
19   2/11/2015  HI540 TFR?FR xxx8690   130.00       49246  Cansel Survey Equipment  False
20   2/11/2015  HQ010 TFR?FR xxx8690   177.00       49246  Cansel Survey Equipment  False
21   2/10/2015  WU455 TFR?FR xxx8690   200.00       49246  Cansel Survey Equipment  False
22    2/6/2015  JJ500 TFR?FR xxx8690   301.00       49246  Cansel Survey Equipment  False
23    2/6/2015   CANSEL SURVEY E PAY  1182.08       49246  Cansel Survey Equipment   True
24    2/5/2015  IR453 TFR?FR xxx8690   168.56       49246             Cansel IR453   True
26    2/2/2015  RQ574 TFR?FR xxx8690   500.00       49246  Cansel Survey Equipment  False
27    2/2/2015  UT022 TFR?FR xxx8690   850.00       49246  Cansel Survey Equipment  False
28  12/31/2014  HU521 TFR?FR xxx8690   950.17       49246      Cansel Survey HU521   True

Some explanations:

In [266]: cols_desc
Out[266]:
['cansel',
 'fr',
 'hi540',
 'hq010',
 'hu204',
 'hu521',
 'ir453',
 'jh401',
 'jj500',
 'jx154',
 'pay',
 'rq574',
 'survey',
 'tfr',
 'ue200',
 'uo414',
 'uq263',
 'ut022',
 'ut460',
 'wu455',
 'ww120',
 'xxx8690']

In [267]: cols_emp
Out[267]: ['cansel', 'equipment', 'hu521', 'ir453', 'survey']

In [268]: common_cols_idx = [i for i,col in enumerate(cols_desc) if col in cols_emp]

In [269]: common_cols_idx
Out[269]: [0, 5, 6, 12]

In [270]: X.toarray()
Out[270]:
array([[0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1],
       [0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1],
       [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1],
       [0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1],
       [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1],
       [0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1],
       [0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1],
       [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1],
       [0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1],
       [0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1],
       [0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1],
       [0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1],
       [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1],
       [0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1],
       [0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1],
       [0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1]], dtype=int64)

In [271]: X.toarray()[:, common_cols_idx]
Out[271]:
array([[0, 0, 0, 0],
       [0, 0, 0, 0],
       [1, 0, 0, 1],
       [0, 0, 0, 0],
       [0, 0, 0, 0],
       [1, 0, 0, 1],
       [0, 0, 0, 0],
       [0, 0, 0, 0],
       [0, 0, 0, 0],
       [1, 0, 0, 1],
       [0, 0, 0, 0],
       [0, 0, 0, 0],
       [0, 0, 0, 0],
       [0, 0, 0, 0],
       [0, 0, 0, 0],
       [1, 0, 0, 1],
       [0, 0, 1, 0],
       [0, 0, 0, 0],
       [0, 0, 0, 0],
       [0, 1, 0, 0]], dtype=int64)

In [272]: X.toarray()[:, common_cols_idx] == 1
Out[272]:
array([[False, False, False, False],
       [False, False, False, False],
       [ True, False, False,  True],
       [False, False, False, False],
       [False, False, False, False],
       [ True, False, False,  True],
       [False, False, False, False],
       [False, False, False, False],
       [False, False, False, False],
       [ True, False, False,  True],
       [False, False, False, False],
       [False, False, False, False],
       [False, False, False, False],
       [False, False, False, False],
       [False, False, False, False],
       [ True, False, False,  True],
       [False, False,  True, False],
       [False, False, False, False],
       [False, False, False, False],
       [False,  True, False, False]], dtype=bool)

In [273]: (X.toarray()[:, common_cols_idx] == 1).any(1)
Out[273]: array([False, False,  True, False, False,  True, False, False, False,  True, False, False, False, False, False,  True,  True, Fals
e, False,  True], dtype=bool)

与恶龙缠斗过久,自身亦成为恶龙；凝视深渊过久,深渊将回以凝视…
Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

Just Browsing Browsing

2.1m questions

2.1m answers

60 comments

57.0k users

Most popular tags

Powered by Question2Answer
Donate
Theme by Q2A Market&&OStack.cn
...