python - Pandas remove rows where several columns are not nan

Question

I have a dataframe that looks like this:

   A   B   C    D    E
0  P  10 NaN  5.0  9.0
1  Q  19 NaN  NaN  4.0
2  R   8 NaN  3.0  7.0
3  S  20 NaN  3.0  7.0
4  T   4 NaN  2.0  NaN

And I have a list: [['A', 'B', 'D', 'E'], ['A', 'B', 'D'], ['A', 'B', 'E']]

I am iterating over the list and getting only those rows from the dataframe, for which the columns specified by the list are not empty.

I have tried with the following code:

test_df = pd.DataFrame([['P', 10, np.nan, 5, 9], ['Q', 19, np.nan, np.nan, 4], ['R', 8, np.nan, 3, 7],
                        ['S', 20, np.nan, 3, 7], ['T', 4, np.nan, 2, np.nan]], columns=list('ABCDE'))
priority_list = [list('ABDE'), list('ABD'), list('ABE')]
for elem in priority_list:
    test_df = test_df.loc[test_df[elem].notna()]
    print(test_df)

But this is throwing the following error:

File "C:Python37libsite-packagespandascoreindexing.py", line 879, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
  File "C:Python37libsite-packagespandascoreindexing.py", line 1097, in _getitem_axis
    raise ValueError("Cannot index with multidimensional key")
ValueError: Cannot index with multidimensional key

How to overcome this issue and check for multiple columns for non-na values in the dataframe?

question from:https://stackoverflow.com/questions/66061955/pandas-remove-rows-where-several-columns-are-not-nan

Answer 1

Use DataFrame.all for test if all selected values are Trues:

priority_list = [list('ABDE'), list('ABD'), list('ABE')]
for elem in priority_list:
    test_df = test_df.loc[test_df[elem].notna().all(axis=1)]
    print(test_df)