Custom JSON Encoder in Python

JSON objects are one of the most efficient ways to transmit information on the internet. However, we cannot directly convert all types of data into JSON. In this article, we will discuss different ways to create a custom JSON encoder to convert python objects into JSON format in Python.

This article discusses how to create a custom JSON encoder in Python. If you don’t haven’t worked with JSON files even with the default encoders, I suggest you read this article on working with JSON files in Python.

Convert Python Object to JSON Using The dump() Method

The dump() method is used to convert a python object to a JSON file. It takes the python object and the file pointer to the JSON file and saves the JSON formatted content to the JSON file as shown below.

fp=open("sample.json","w")
myDict={"Name": "Aditya", "Age": 23}
json.dump(myDict, fp)
fp.close()

Output:

Here, we have converted a python dictionary to a JSON file. We can convert almost all the primitive data types to JSON file using the dump() method. However, when we try to convert a custom python object to JSON file using the dump() method, the program runs into a TypeError exception with the message “TypeError: Object of type _class name_ is not JSON serializable”.

class Student:
    def __init__(self, name, age):
        self.Name=name
        self.Age=age
student=Student("Aditya",23)
fp=open("sample.json","w")
json.dump(student, fp)
fp.close()

Output:

TypeError: Object of type Student is not JSON serializable

In the above example, we have created a Student class. When we pass a python object of Student class to the dump() method, the program runs into TypeError exception. This is due to the reason that the dump() method doesn’t know how to encode an object of Student class to a JSON object.

To avoid this error, we can use the “default” parameter or “cls” parameter in the dump() method.

Create Custom JSON Encoder Using the “default” Parameter

When the dump() method gets a non-serializable object as input, it raises a TypeError exception. You can use the “default” parameter to handle this case. The default parameter takes a function as its input argument. The function should return a JSON-serializable version of the object. 

To create a JSON serializable object, we will convert the python object to its dictionary representation using the __dict__ attribute of the object. The function will then return a dictionary object which is JSON serializable. Hence, the object will be converted to JSON file as shown in the following example.

class Student:
    def __init__(self, name, age):
        self.Name=name
        self.Age=age
def SimpleEncoderFunction(obj):
    return obj.__dict__
student=Student("Aditya",23)
fp=open("samplestudent.json","w")
json.dump(student, fp,default=SimpleEncoderFunction)
fp.close()

Output:

In the above example, we have defined a custom encoder function named SimpleEncoderFunction(). The function takes a python object as its input argument and returns a dictionary representation of the object.

Whenever the dump() method encounters a custom python object which is non-serializable, it sends the object to the SimpleEncoderFunction. The SimpleEncoderFunction() function then converts the python object to a dictionary and returns the dictionary. The dump() method then converts the dictionary to JSON format.

Create Custom JSON Encoder Using the “cls” Parameter

Instead of using the default parameter, we can use the cls parameter to create a custom json encoder using the dump() method. The cls parameter is used when we want to convert custom python objects to JSON. For converting a custom object to JSON, we need to define a  custom JSONEncoder subclass and pass it to the cls parameter.

The subclass should override the default() method of the JSONEncoder class and return a JSON serializable version of the python object. For instance, consider the following example.

class Student:
    def __init__(self, name, age):
        self.Name=name
        self.Age=age
class SimpleEncoderClass(json.JSONEncoder):
        def default(self, obj):
            return obj.__dict__
student=Student("Aditya",23)
fp=open("samplestudent.json","w")
json.dump(student, fp,cls=SimpleEncoderClass)
fp.close()

Output:

Here, the SimpleEncoderClass is a subclass of the JSONEncoder class. It overrides the default() method. When we pass SimpleEncoderClass to the cls parameter, the dump() method successfully creates a JSON file from the python object.

While execution, Whenever the dump() method encounters a custom python object which is non-serializable, it uses the SimpleEncoderClass to encode the object. The default() method in the SimpleEncoderClass then converts the python object to a dictionary and returns the dictionary. The dump() method then converts the dictionary to JSON format.

Convert Python Objects to JSON Using the dumps() Method

If you want to convert python objects to a JSON string instead of the JSON file, you can use the dumps() method. The dumps() method takes a JSON serializable python object as its input argument and returns the JSON string as shown in the following example.

myDict={"Name": "Aditya", "Age": 23}
print("The python object is:")
print(myDict)
jsonStr=json.dumps(myDict)
print("The json string is:")
print(jsonStr)

Output:

The python object is:
{'Name': 'Aditya', 'Age': 23}
The json string is:
{"Name": "Aditya", "Age": 23}

In this example, we have created a JSON string from a python dictionary. However, when we pass a custom python object to the dumps() method, it runs into a TypeError exception as shown below.

class Student:
    def __init__(self, name, age):
        self.Name=name
        self.Age=age
student=Student("Aditya",23)
jsonStr=json.dumps(student)
print("The json string is:")
print(jsonStr)

Output:

TypeError: Object of type Student is not JSON serializable

In this example, you can observe that the dumps() method has failed to create a JSON string from a python object of class type Student.

To avoid the error, we can pass a custom JSON encoder function to the default parameter of the dumps() method as shown below.

class Student:
    def __init__(self, name, age):
        self.Name=name
        self.Age=age
def SimpleEncoderFunction(obj):
    return obj.__dict__
student=Student("Aditya",23)
jsonStr=json.dumps(student,default=SimpleEncoderFunction)
print("The json string is:")
print(jsonStr)

Output:

The json string is:
{"Name": "Aditya", "Age": 23}

Instead of the default parameter, you can use the cls parameter with a subclass of JSONEncoder class as shown below.

class Student:
    def __init__(self, name, age):
        self.Name=name
        self.Age=age
class SimpleEncoderClass(json.JSONEncoder):
        def default(self, obj):
            return obj.__dict__
student=Student("Aditya",23)
jsonStr=json.dumps(student,cls=SimpleEncoderClass)
print("The json string is:")
print(jsonStr)

Output:

The json string is:
{"Name": "Aditya", "Age": 23}

In the above examples, the SimpleEncoderClass and SimpleEncoderFunction works with the dumps() method the same way it worked with the dump() method.

Convert Python Object to JSON Using the JSONEncoder Class

Instead of the the dumps() method, you can use the JSONEncoder class to convert a python object to JSON string. The JSONEncoder class constructor has the following syntax.

json.JSONEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)

Here, 

• A key in a python object can be of any data type. However, all the data types cannot be converted to JSON format. When we try to encode a python object or a dictionary with keys having data types other than str, int, float, bool, and None, the encoder raises a TypeError exception. The skipkeys parameter helps us handle data in such situations. When we set skipkeys to True, the encoder skips the keys having incompatible data types instead of running into a TypeError exception.
• The ensure_ascii parameter is used to make sure that all the characters in the output JSON file are ASCII characters. When ensure_ascii is set to True, all the non-ASCII characters are skipped while encoding. If it is set to False, the non-ASCII characters are saved to the JSON file as is while encoding.
• The check_circular parameter is used to make sure that the encoder performs circular reference checks for container types. If check_circular is set to False, the circular reference check is skipped. In this case, a circular reference will cause the program to run into RecursionError.
• The allow_nan parameter is used to convert NaN and infinity values to JSON format while encoding. When allow_nan is set to True, the encoder converts NaN, +inf, and -inf to JavaScript NaN, Infinity, -Infinity respectively. When allow_nan is set to False, the encoder raises a ValueError exception when it finds NaN, +inf, or -inf while encoding python objects to JSON.
• The sort_keys parameter is used in the JSONEncoder class if you want the keys of the JSON object in a sorted manner. If the sort_keys parameter is set to True, the keys of the output JSON object are sorted in a lexicographic manner. 
• The indent parameter is used to specify the indentation in the JSON object. When the indent parameter is set to None or a negative integer, the JSON object is in the most compact representation. When the indent parameter is set to a positive integer value, it indents that many spaces per level in the JSON object created while encoding. When the indent is set to a string, the string is used as the indentation character. When the indent is set to 0 or an empty string, a new line is introduced for each level of indentation.
• By default, the key-value pairs of a JSON object are separated by a comma and the key is separated from the values using a colon character. To specify a new separator for the keys and items while encoding a python object to JSON,  you can pass a tuple containing two characters to the separators parameter. The first character of the tuple becomes the separator for the key and value. The second element of the tuple becomes a separator for different items. When the indent parameter is set to None, the default value for the separators parameter is (', ', ': '). Otherwise, the default value for the separators parameter is (',', ': '). To get the most compact JSON object, you should remove whitespaces from the separators and use (',', ':') as an input argument to the separators parameter.
• When the encoder gets a non-serializable object in the python object, it raises a TypeError exception. You can use the default parameter to handle this case. The default parameter takes a function as its input argument. The function should return a JSON-encodable version of the object or raise a TypeError. It helps encode python objects that can’t otherwise be serialized.

To convert a python object to JSON, we can first create a JSONEncoder object using the JSONEncoder() constructor. After that, we can invoke the encode() method on the JSONEncoder object. The encode() method takes the python object and returns a JSON string as shown below.

myDict={"Name": "Aditya", "Age": 23}
print("The python object is:")
print(myDict)
jsonStr=json.JSONEncoder().encode(myDict)
print("The json string is:")
print(jsonStr)

Output:

The python object is:
{'Name': 'Aditya', 'Age': 23}
The json string is:
{"Name": "Aditya", "Age": 23}

In the above example, we have converted a python dictionary to JSON string using the JSONEncoder class and the encode() method.

When we pass a non-json serializable object to the encode() method, the program runs into a TypeError exception as shown below.

class Student:
    def __init__(self, name, age):
        self.Name=name
        self.Age=age
student=Student("Aditya",23)
jsonStr=json.JSONEncoder().encode(student)
print("The json string is:")
print(jsonStr)

Output:

TypeError: Object of type Student is not JSON serializable

Here, we have tried to create a JSON string from a custom python object of class Student. As the python object is non-json serializable, the program runs into TypeError exception.

To avoid the error, we can pass the SimpleEncoderFunction to the default parameter of the JSONEncoder() constructor as shown below.

class Student:
    def __init__(self, name, age):
        self.Name=name
        self.Age=age
def SimpleEncoderFunction(obj):
    return obj.__dict__
student=Student("Aditya",23)
jsonStr=json.JSONEncoder(default=SimpleEncoderFunction).encode(student)
print("The json string is:")
print(jsonStr)

Output:

The json string is:
{"Name": "Aditya", "Age": 23}

Here, the JSONEncoder class uses the SimpleEncoderFunction instead of the predefined default() method to encode a python object to JSON.

In the above code, whenever the encode() method encounters a custom python object which is non-serializable, it sends the object to the SimpleEncoderFunction. The SimpleEncoderFunction() function then converts the python object to a dictionary and returns the dictionary. The encode() method then converts the dictionary to JSON format.

Suggested Reading: If you are into machine learning, you can read this article on mlops for beginners. You might also like this article on clustering mixed data types in Python.

JSON Encoder for Converting Complex Python Objects to JSON

Python objects that don’t have any other custom object as their attributes can be easily converted to JSON objects using a simple JSON encoder like the SimpleEncoderFunction or SimpleEncoderClass. They can also be converted back to a python object using a simple JSON decoder as you can observe in this article on custom JSON decoder in Python. 

However, complex python objects that have another object as their attributes are not very well managed by a simple encoder and decoder. For complex python objects, we can use the SimpleEncoderClass or SimpleEncoderFunction as discussed above. You can observe this in the following example.

class Student:
    def __init__(self, name, age,details):
        self.Name=name
        self.Age=age
        self.Details=details
class Details:
    def __init__(self, height, weight):
        self.Height=height
        self.Weight=weight
def SimpleEncoderFunction(obj):
    return obj.__dict__
details=Details(160,60)
student=Student("Aditya",23,details)
jsonStr=json.JSONEncoder(default=SimpleEncoderFunction).encode(student)
print("The json string is:")
print(jsonStr)

Output:

The json string is:
{"Name": "Aditya", "Age": 23, "Details": {"Height": 160, "Weight": 60}}

In the above example, we have a Details class that has "Height" and "Weight" attribute. In the Student class, we have "Name", "Age", and "Details" attributes.

When an object of the Student class is given as input to the encode() method, it is sent to the SimpleEncoderFunction(). The SimpleEncoderFunction() while converting the Student class to a dictionary, encounters the Details class. The SimpleEncoderFunction() function then recursively calls itself with the object of Details class as input argument. Here, the function returns a dictionary representation of the Details object. Then, the SimpleEncoderFunction() returns the dictionary representation of the Student object. After this, the dictionary is converted to JSON object by the encode() method.

This type of encoding leads to an error when we try to convert the JSON object back to a python object. So, we also need to specify the type of python object in the JSON. This helps in avoiding errors while decoding the JSON object.

You can define a custom JSON encoder function and use it with the dump() method as shown below.

class Student:
    def __init__(self, name, age,details):
        self.Name=name
        self.Age=age
        self.Details=details
class Details:
    def __init__(self, height, weight):
        self.Height=height
        self.Weight=weight
def ComplexEncoderFunction(obj):
        objDict=obj.__dict__
        typeDict={"__type__":type(obj).__name__}
        return {**objDict,**typeDict}
    
details=Details(160,60)
student=Student("Aditya",23,details)
fp=open("complexjson.json","w")
json.dump(student, fp,default=ComplexEncoderFunction)
fp.close()

Output:

In the above example, we have defined a ComplexEncoderFunction() instead of SimpleEncoderFunction(). The ComplexEncoderFunction() adds a key named "__type__" with the class name of the python object as its associated value while converting the python object to dictionary, Then, the dictionary is converted to JSON by the The class name helps us convert the JSON object to custom python object with correct class type.

You can also define a custom subclass of JSONEncoder class and use it with the dump() method and the cls parameter as shown below. 

class Student:
    def __init__(self, name, age,details):
        self.Name=name
        self.Age=age
        self.Details=details
class Details:
    def __init__(self, height, weight):
        self.Height=height
        self.Weight=weight
class ComplexEncoderClass(json.JSONEncoder):
        def default(self, obj):
            objDict=obj.__dict__
            typeDict={"__type__":type(obj).__name__}
            return {**objDict,**typeDict}
    
details=Details(160,60)
student=Student("Aditya",23,details)
fp=open("complexjson.json","w")
json.dump(student, fp,cls=ComplexEncoderClass)
fp.close()

Output:

You can use the custom encoder function with the dumps() method and the default parameter as shown below.

class Student:
    def __init__(self, name, age,details):
        self.Name=name
        self.Age=age
        self.Details=details
class Details:
    def __init__(self, height, weight):
        self.Height=height
        self.Weight=weight
def ComplexEncoderFunction(obj):
        objDict=obj.__dict__
        typeDict={"__type__":type(obj).__name__}
        return {**objDict,**typeDict}
    
details=Details(160,60)
student=Student("Aditya",23,details)
jsonStr=json.dumps(student,default=ComplexEncoderFunction)
print("The json string is:")
print(jsonStr)

Output:

The json string is:
{"Name": "Aditya", "Age": 23, "Details": {"Height": 160, "Weight": 60, "__type__": "Details"}, "__type__": "Student"}

You can also use the custom subclass of JSONEncoder class with the dumps() method as shown below.

class Student:
    def __init__(self, name, age,details):
        self.Name=name
        self.Age=age
        self.Details=details
class Details:
    def __init__(self, height, weight):
        self.Height=height
        self.Weight=weight
class ComplexEncoderClass(json.JSONEncoder):
        def default(self, obj):
            objDict=obj.__dict__
            typeDict={"__type__":type(obj).__name__}
            return {**objDict,**typeDict}
    
details=Details(160,60)
student=Student("Aditya",23,details)
jsonStr=json.dumps(student,cls=ComplexEncoderClass)
print("The json string is:")
print(jsonStr)

Output:

The json string is:
{"Name": "Aditya", "Age": 23, "Details": {"Height": 160, "Weight": 60, "__type__": "Details"}, "__type__": "Student"}

Instead of using a custom JSON encoder subclass, you can use the JSON encoder function and the default parameter in the JSONEncoder constructor to create a JSON encoder and use it to convert a python object to JSON as shown in the following example.

class Student:
    def __init__(self, name, age,details):
        self.Name=name
        self.Age=age
        self.Details=details
class Details:
    def __init__(self, height, weight):
        self.Height=height
        self.Weight=weight
def ComplexEncoderFunction(obj):
        objDict=obj.__dict__
        typeDict={"__type__":type(obj).__name__}
        return {**objDict,**typeDict}
    
details=Details(160,60)
student=Student("Aditya",23,details)
jsonStr=json.JSONEncoder(default=ComplexEncoderFunction).encode(student)
print("The json string is:")
print(jsonStr)

Output

The json string is:
{"Name": "Aditya", "Age": 23, "Details": {"Height": 160, "Weight": 60, "__type__": "Details"}, "__type__": "Student"}

Conclusion

In this article, we have discussed different ways to create a custom JSON encoder to convert complex python objects to JSON. To learn more about python programming, you can read this article on how to create a chat app in Python. You might also like this article on linear regression using the sklearn module in Python.

Stay tuned for more informed articles.

Happy Learning!