Check for NaN Values in Pandas Python

While working with data in python, we often encounter null values or NaN values. In this article, we will discuss different ways to check for nan values or null values in a pandas dataframe or series.

The isna() Function

The isna() function in pandas is used to check for NaN values. It has the following syntax.

pandas.isna(object)

Here, the object can be a single python object or a list/array of python objects.

If we pass a single python object to the isna() method as an input argument, it returns True if the python object is None, pd.NA or np.NaN object. You can observe this in the following example.

import pandas as pd
import numpy as np
x=pd.NA
print("The value is:",x)
output=pd.isna(x)
print("Is the value Null:",output)

Output:

The value is: <NA>
Is the value Null: True

In the above example, we have passed the pandas.NA object to the isna() function. After execution, the function returns True.

When we pass a list or numpy array of elements to the isna() function, the isna() function is executed with each element of the array.

After execution, it returns a list or array containing True and False values. The False values of the output array correspond to all the values that are not NA, NaN, or None at the same position in the input list or array. The True values in the output array correspond to all the NA, NaN, or None values at the same position in the input list or array. You can observe this in the following example.

import pandas as pd
import numpy as np
x=[1,2,pd.NA,4,5,None, 6,7,np.nan]
print("The values are:",x)
output=pd.isna(x)
print("Are the values Null:",output)

Output:

The values are: [1, 2, <NA>, 4, 5, None, 6, 7, nan]
Are the values Null: [False False  True False False  True False False  True]

In this example, we have passed a list containing 9 elements to the isna() function. After execution, the isna() method returns a list of 9 boolean values. Each element in the output list is associated with the element at the same index in the input list given to the isna() function. At the indices where the input list contains Null values, the output list contains True. Similarly, at indices where the input list contains integers, the output list contains False.

Check for NaN Values in a Pandas Dataframe Using The isna() Method

Along with the isna() function, the pandas module also has the isna() method at the dataframe level. You can directly invoke the isna() method on the pandas dataframe to check for nan values.

The isna() method, when invoked on a pandas dataframe, returns another dataframe containing True and False values. You can observe this in the following example.

import pandas as pd
import numpy as np
df=pd.read_csv("grade.csv")
print("The dataframe is:")
print(df)
output=df.isna()
print("Are the values Null:")
print(output)

Output:

The dataframe is:
    Class  Roll        Name  Marks Grade
0       1    11      Aditya   85.0     A
1       1    12       Chris    NaN     A
2       1    14         Sam   75.0     B
3       1    15       Harry    NaN   NaN
4       2    22         Tom   73.0     B
5       2    15        Golu   79.0     B
6       2    27       Harsh   55.0     C
7       2    23       Clara    NaN     B
8       3    34         Amy   88.0     A
9       3    15    Prashant    NaN     B
10      3    27      Aditya   55.0     C
11      3    23  Radheshyam    NaN   NaN
Are the values Null:
    Class   Roll   Name  Marks  Grade
0   False  False  False  False  False
1   False  False  False   True  False
2   False  False  False  False  False
3   False  False  False   True   True
4   False  False  False  False  False
5   False  False  False  False  False
6   False  False  False  False  False
7   False  False  False   True  False
8   False  False  False  False  False
9   False  False  False   True  False
10  False  False  False  False  False
11  False  False  False   True   True

In the above example, we have passed a dataframe containing NaN values along with other values. The isna() method returns a dataframe containing boolean values. Here, False values of the output dataframe correspond to all the values that are not NA, NaN, or None at the same position in the input dataframe. The True values in the output dataframe correspond to all the NA, NaN, or None values at the same position in the input dataframe.

Check for Nan Values in a Column in Pandas Dataframe

Instead of the entire dataframe, you can also check for nan values in a column of a pandas dataframe. For this, you just need to invoke the isna() method on the particular column as shown below.

import pandas as pd
import numpy as np
df=pd.read_csv("grade.csv")
print("The dataframe column is:")
print(df["Marks"])
output=df["Marks"].isna()
print("Are the values Null:")
print(output)

Output:

The dataframe column is:
0     85.0
1      NaN
2     75.0
3      NaN
4     73.0
5     79.0
6     55.0
7      NaN
8     88.0
9      NaN
10    55.0
11     NaN
Name: Marks, dtype: float64
Are the values Null:
0     False
1      True
2     False
3      True
4     False
5     False
6     False
7      True
8     False
9      True
10    False
11     True
Name: Marks, dtype: bool

Check for Nan Values in a Pandas Series Using The isna() Method

Like a dataframe, we can also invoke the isna() method on a Series object in pandas. In this case, the isna() method returns a Series containing True and False values. You can observe this in the following example.

import pandas as pd
import numpy as np
x=pd.Series([1,2,pd.NA,4,5,None, 6,7,np.nan])
print("The series is:")
print(x)
output=pd.isna(x)
print("Are the values Null:")
print(output)

Output:

The series is:
0       1
1       2
2    <NA>
3       4
4       5
5    None
6       6
7       7
8     NaN
dtype: object
Are the values Null:
0    False
1    False
2     True
3    False
4    False
5     True
6    False
7    False
8     True
dtype: bool

In this example, we have invoked the isna() method on a pandas series. The isna() method returns a Series of boolean values after execution. Here, False values of the output series correspond to all the values that are not NA, NaN, or None at the same position in the input series. The True values in the output series correspond to all the NA, NaN, or None values at the same position in the input series.

Check for NaN Values in Pandas Using the isnull() Method

The isnull() function is an alias of the isna() function. Hence, it works exactly the same as the isna() function.

When we pass a NaN value, pandas.NA value, pandas.NaT value, or None object to the isnull() function, it returns True. 

import pandas as pd
import numpy as np
x=pd.NA
print("The value is:",x)
output=pd.isnull(x)
print("Is the value Null:",output)

Output:

The value is: <NA>
Is the value Null: True

In the above example, we have passed pandas.NA value to the isnull() function. Hence, it returns True.

When we pass any other python object to the isnull() function, it returns False as shown below.

import pandas as pd
import numpy as np
x=1117
print("The value is:",x)
output=pd.isnull(x)
print("Is the value Null:",output)

Output:

The value is: 1117
Is the value Null: False

In this example, we passed the value 1117 to the isnull() function. Hence, it returns False showing that the value is not a null value.

When we pass a list or numpy array to the isnull() function, it returns a numpy array containing True and False values. You can observe this in the following example.

import pandas as pd
import numpy as np
x=[1,2,pd.NA,4,5,None, 6,7,np.nan]
print("The values are:",x)
output=pd.isnull(x)
print("Are the values Null:",output)

Output:

The values are: [1, 2, <NA>, 4, 5, None, 6, 7, nan]
Are the values Null: [False False  True False False  True False False  True]

In this example, we have passed a list to the isnull() function. After execution, the isnull() function returns a list of boolean values. Each element in the output list is associated with the element at the same index in the input list given to the isnull() function. At the indices where the input list contains Null values, the output list contains True. Similarly, at indices where the input list contains integers, the output list contains False.

Check for NaN Values in a Dataframe Using the isnull() Method

You can also invoke the isnull() method on a pandas dataframe to check for nan values as shown below.

import pandas as pd
import numpy as np
df=pd.read_csv("grade.csv")
print("The dataframe is:")
print(df)
output=df.isnull()
print("Are the values Null:")
print(output)

Output:

The dataframe is:
    Class  Roll        Name  Marks Grade
0       1    11      Aditya   85.0     A
1       1    12       Chris    NaN     A
2       1    14         Sam   75.0     B
3       1    15       Harry    NaN   NaN
4       2    22         Tom   73.0     B
5       2    15        Golu   79.0     B
6       2    27       Harsh   55.0     C
7       2    23       Clara    NaN     B
8       3    34         Amy   88.0     A
9       3    15    Prashant    NaN     B
10      3    27      Aditya   55.0     C
11      3    23  Radheshyam    NaN   NaN
Are the values Null:
    Class   Roll   Name  Marks  Grade
0   False  False  False  False  False
1   False  False  False   True  False
2   False  False  False  False  False
3   False  False  False   True   True
4   False  False  False  False  False
5   False  False  False  False  False
6   False  False  False  False  False
7   False  False  False   True  False
8   False  False  False  False  False
9   False  False  False   True  False
10  False  False  False  False  False
11  False  False  False   True   True

In the output, you can observe that the isnull() method behaves in exactly the same manner as the isna() method.

Check for NaN in a Column in a Dataframe Using the isnull() Method

Instead of the entire dataframe, you can also use the isnull() method to check for nan values in a column as shown in the following example.

import pandas as pd
import numpy as np
df=pd.read_csv("grade.csv")
print("The dataframe column is:")
print(df["Marks"])
output=df["Marks"].isnull()
print("Are the values Null:")
print(output)

Output:

The dataframe column is:
0     85.0
1      NaN
2     75.0
3      NaN
4     73.0
5     79.0
6     55.0
7      NaN
8     88.0
9      NaN
10    55.0
11     NaN
Name: Marks, dtype: float64
Are the values Null:
0     False
1      True
2     False
3      True
4     False
5     False
6     False
7      True
8     False
9      True
10    False
11     True
Name: Marks, dtype: bool

In a similar manner, you can invoke the isnull() method on a pandas series as shown below.

import pandas as pd
import numpy as np
x=pd.Series([1,2,pd.NA,4,5,None, 6,7,np.nan])
print("The series is:")
print(x)
output=pd.isnull(x)
print("Are the values Null:")
print(output)

Output:

The series is:
0       1
1       2
2    <NA>
3       4
4       5
5    None
6       6
7       7
8     NaN
dtype: object
Are the values Null:
0    False
1    False
2     True
3    False
4    False
5     True
6    False
7    False
8     True
dtype: bool

In the above example, we have invoked the isnull() method on a series. The isnull() method returns a Series of boolean values after execution. Here, False values of the output series correspond to all the values that are not NA, NaN, or None at the same position in the input series. The True values in the output series correspond to all the NA, NaN, or None values at the same position in the input series.

Conclusion

In this article, we have discussed different ways to check for nan values in pandas. To learn more about python programming,

you can read this article on how to sort a pandas dataframe. You might also like this article on how to drop columns from a pandas dataframe.

I hope you enjoyed reading this article. Stay tuned for more informative articles.

Happy Learning!