Check for Not Null Value in Pandas Python

In python, we sometimes need to filter not null and null values. In this article, we will discuss different ways to check for not null in pandas using examples.

We can check for not null in pandas using the notna() function and the notnull() function. Let us discuss each function one by one.

Check for Not Null in Pandas Using the notna() Method

As the name suggests, the notna() method works as a negation of the isna() method. The isna() method is used to check for nan values in pandas.  The notna() function has the following syntax.

pandas.notna(object)

Here, the object can be a single python object or a collection of objects such as a python list or tuple.

If we pass a single python object to the notna() method as an input argument, it returns False if the python object is None, pd.NA or np.NaN object. For python objects that are not null, the notna() function returns True. You can observe this in the following example.

import pandas as pd
import numpy as np
x=pd.NA
print("The value is:",x)
output=pd.notna(x)
print("Is the value not Null:",output)

Output:

The value is: <NA>
Is the value not Null: False

In the above example, we have passed the pandas.NA object to the notna() function. Hence, it returns False.

When we pass a list or numpy array of elements to the notna() function, the notna() function is executed with each element of the array. After execution, it returns a list or array containing True and False values. The True values of the output array correspond to all the values that are not NA, NaN, or None at the same position in the input list or array. The False values in the output array correspond to all the NA, NaN, or None values at the same position in the input list or array. You can observe this in the following example.

import pandas as pd
import numpy as np
x=[1,2,pd.NA,4,5,None, 6,7,np.nan]
print("The values are:",x)
output=pd.notna(x)
print("Are the values not Null:",output)

Output:

The values are: [1, 2, <NA>, 4, 5, None, 6, 7, nan]
Are the values not Null: [ True  True False  True  True False  True  True False]

In this example, we have passed a list containing 9 elements to the notna() function. After execution, the notna() function returns a list of 9 boolean values. Each element in the output list is associated with the element at the same index in the input list given to the notna() function. At the indices where the input list does not contain Null values, the output list contains True. Similarly, at indices where the input list contains null values, the output list contains False.

Check for Not NA in a Pandas Dataframe Using notna() Method

Along with the notna() function, python also provides us with the notna() method to check for not null values in pandas dataframes and series objects.

The notna() method, when invoked on a pandas dataframe, returns another dataframe containing True and False values. True values of the output dataframe correspond to all the values that are not NA, NaN, or None at the same position in the input dataframe. The False values in the output dataframe correspond to all the NA, NaN, or None values at the same position in the input dataframe. You can observe this in the following example.

import pandas as pd
import numpy as np
df=pd.read_csv("grade.csv")
print("The dataframe is:")
print(df)
output=df.notna()
print("Are the values not Null:")
print(output)

Output:

The dataframe is:
    Class  Roll        Name  Marks Grade
0       1    11      Aditya   85.0     A
1       1    12       Chris    NaN     A
2       1    14         Sam   75.0     B
3       1    15       Harry    NaN   NaN
4       2    22         Tom   73.0     B
5       2    15        Golu   79.0     B
6       2    27       Harsh   55.0     C
7       2    23       Clara    NaN     B
8       3    34         Amy   88.0     A
9       3    15    Prashant    NaN     B
10      3    27      Aditya   55.0     C
11      3    23  Radheshyam    NaN   NaN
Are the values not Null:
    Class  Roll  Name  Marks  Grade
0    True  True  True   True   True
1    True  True  True  False   True
2    True  True  True   True   True
3    True  True  True  False  False
4    True  True  True   True   True
5    True  True  True   True   True
6    True  True  True   True   True
7    True  True  True  False   True
8    True  True  True   True   True
9    True  True  True  False   True
10   True  True  True   True   True
11   True  True  True  False  False

In the above example, we have invoked the notna() method on a dataframe containing NaN values along with other values. The notna() method returns a dataframe containing boolean values. Here, False values of the output dataframe correspond to all the values that are NA, NaN, or None at the same position in the input dataframe. The True values in the output dataframe correspond to all the not null values at the same position in the input dataframe.

Check for Not Null Values in a Column in Pandas Dataframe

Instead of the entire dataframe, you can also check for not null values in a column of a pandas dataframe. For this, you just need to invoke the notna() method on the particular column as shown below.

import pandas as pd
import numpy as np
df=pd.read_csv("grade.csv")
print("The dataframe column is:")
print(df["Marks"])
output=df["Marks"].notna()
print("Are the values not Null:")
print(output)

Output:

The dataframe column is:
0     85.0
1      NaN
2     75.0
3      NaN
4     73.0
5     79.0
6     55.0
7      NaN
8     88.0
9      NaN
10    55.0
11     NaN
Name: Marks, dtype: float64
Are the values not Null:
0      True
1     False
2      True
3     False
4      True
5      True
6      True
7     False
8      True
9     False
10     True
11    False
Name: Marks, dtype: bool

Check for Not NA in a Pandas Series Using notna() Method

Like a dataframe, we can also invoke the notna() method on a pandas Series object. In this case, the notna() method returns a Series containing True and False values. You can observe this in the following example.

import pandas as pd
import numpy as np
x=pd.Series([1,2,pd.NA,4,5,None, 6,7,np.nan])
print("The series is:")
print(x)
output=pd.notna(x)
print("Are the values not Null:")
print(output)

Output:

The series is:
0       1
1       2
2    <NA>
3       4
4       5
5    None
6       6
7       7
8     NaN
dtype: object
Are the values not Null:
0     True
1     True
2    False
3     True
4     True
5    False
6     True
7     True
8    False
dtype: bool

In this example, we have invoked the notna() method on a pandas series. The notna() method returns a Series of boolean values after execution. Here, False values of the output series correspond to all the values that are NA, NaN, or None at the same position in the input series. The True values in the output series correspond to all the not null values at the same position in the input series.

Check for Not Null in Pandas Using the notnull() Method

The notnull() method is an alias of the notna() method. Hence, it works exactly the same as the notna() method.

When we pass a NaN value, pandas.NA value, pandas.NaT value, or None object to the notnull() function, it returns False. 

import pandas as pd
import numpy as np
x=pd.NA
print("The value is:",x)
output=pd.notnull(x)
print("Is the value not Null:",output)

Output:

The value is: <NA>
Is the value not Null: False

In the above example, we have passed pandas.NA value to the notnull() function. Hence, it returns False.

When we pass any other python object to the notnull() function, it returns True as shown below.

import pandas as pd
import numpy as np
x=1117
print("The value is:",x)
output=pd.notnull(x)
print("Is the value not Null:",output)

Output:

The value is: 1117
Is the value not Null: True

In this example, we passed the value 1117 to the notnull() function. Hence, it returns True showing that the value is not a null value.

When we pass a list or numpy array to the notnull() function, it returns a numpy array containing True and False values. You can observe this in the following example.

import pandas as pd
import numpy as np
x=[1,2,pd.NA,4,5,None, 6,7,np.nan]
print("The values are:",x)
output=pd.notnull(x)
print("Are the values not Null:",output)

Output:

The values are: [1, 2, <NA>, 4, 5, None, 6, 7, nan]
Are the values not Null: [ True  True False  True  True False  True  True False]

In this example, we have passed a list to the notnull() function. After execution, the notnull() function returns a list of boolean values. Each element in the output list is associated with the element at the same index in the input list given to the notnull() function. At the indices where the input list contains Null values, the output list contains False. Similarly, at indices where the input list contains integers, the output list contains True.

Check for Not Null in a Pandas Dataframe Using the notnull() Method

You can also invoke the notnull() method on a pandas dataframe to check for nan values as shown below.

import pandas as pd
import numpy as np
df=pd.read_csv("grade.csv")
print("The dataframe is:")
print(df)
output=df.notnull()
print("Are the values not Null:")
print(output)

Output:

The dataframe is:
    Class  Roll        Name  Marks Grade
0       1    11      Aditya   85.0     A
1       1    12       Chris    NaN     A
2       1    14         Sam   75.0     B
3       1    15       Harry    NaN   NaN
4       2    22         Tom   73.0     B
5       2    15        Golu   79.0     B
6       2    27       Harsh   55.0     C
7       2    23       Clara    NaN     B
8       3    34         Amy   88.0     A
9       3    15    Prashant    NaN     B
10      3    27      Aditya   55.0     C
11      3    23  Radheshyam    NaN   NaN
Are the values not Null:
    Class  Roll  Name  Marks  Grade
0    True  True  True   True   True
1    True  True  True  False   True
2    True  True  True   True   True
3    True  True  True  False  False
4    True  True  True   True   True
5    True  True  True   True   True
6    True  True  True   True   True
7    True  True  True  False   True
8    True  True  True   True   True
9    True  True  True  False   True
10   True  True  True   True   True
11   True  True  True  False  False

In the output, you can observe that the notnull() method behaves in exactly the same manner as the notna() method.

Instead of the entire dataframe, you can also use the notnull() method to check for not nan values in a column as shown in the following example.

import pandas as pd
import numpy as np
df=pd.read_csv("grade.csv")
print("The dataframe column is:")
print(df["Marks"])
output=df["Marks"].notnull()
print("Are the values not Null:")
print(output)

Output:

The dataframe column is:
0     85.0
1      NaN
2     75.0
3      NaN
4     73.0
5     79.0
6     55.0
7      NaN
8     88.0
9      NaN
10    55.0
11     NaN
Name: Marks, dtype: float64
Are the values not Null:
0      True
1     False
2      True
3     False
4      True
5      True
6      True
7     False
8      True
9     False
10     True
11    False
Name: Marks, dtype: bool

In a similar manner, you can invoke the notnull() method on a pandas series as shown below.

import pandas as pd
import numpy as np
x=pd.Series([1,2,pd.NA,4,5,None, 6,7,np.nan])
print("The series is:")
print(x)
output=pd.notnull(x)
print("Are the values not Null:")
print(output)

Output:

The series is:
0       1
1       2
2    <NA>
3       4
4       5
5    None
6       6
7       7
8     NaN
dtype: object
Are the values not Null:
0     True
1     True
2    False
3     True
4     True
5    False
6     True
7     True
8    False
dtype: bool

In the above example, we have invoked the notnull() method on a series. The notnull() method returns a Series of boolean values after execution. Here, the True values of the output series correspond to all the values that are not NA, NaN, or None at the same position in the input series. The False values in the output series correspond to all the NA, NaN, or None values at the same position in the input series.

Conclusion

In this article, we have discussed different ways to check for not null values in pandas. To learn more about python programming, you can read this article on how to sort a pandas dataframe. You might also like this article on how to drop columns from a pandas dataframe.

I hope you enjoyed reading this article. Stay tuned for more informative articles.

Happy Learning!