The Fastest Way to Split a Text File Using Python

Python is one of the most popular programming languages in the world. One reason for its popularity is that Python makes it easy to work with data. Reading data from a text file is a routine task in Python. In this post, we’re going to look at the fastest way to read and split a text file using Python. Splitting the data will convert the text to a list, making it easier to work with. We’ll also cover some other methods for splitting text files in Python, and explain how and when these methods are useful.

Introducing The split() Method in Python

The fastest way to split text in Python is with the split() method. This is a built-in method that is useful for separating a string into its individual parts.

The split() method, when invoked on a string, takes a delimiter as its input argument. After execution, it returns a list of the substrings in the string. By default, Python uses whitespace as a delimiter to split the string, but you can provide a delimiter and specify what character(s) to use instead.

For example, a comma(,) is often used to separate string data. This is the case with Comma Separated Value (CSV) files. Whatever you choose as the separator, Python will use to split the string.

Splitting A Text File With The split() Method in Python

In our first example, we have a text file of employee data, including the names of employees, their phone numbers, and occupations. We’ll need to write a Python program that can read this randomly generated information and split the data into lists. To start with, first, save the following text into a file named employee_data.txt.

Lana Anderson 485-3094-88 Electrician
Elian Johnston 751-5845-87 Interior Designer
Henry Johnston 777-6561-52 Astronomer
Dale Johnston 248-1843-09 Journalist
Luke Owens 341-7471-63 Teacher
Amy Perry 494-3532-17 Electrician
Chloe Baker 588-7165-01 Interior Designer

We will first open the above file in read mode using Python with open statement. Here, the open() method takes the file name as its first input argument and the Python literal “r” as its second input argument. After execution, the open() method returns a file pointer that we will assign to the data_file variable. After this, we can iterate through the file’s contents using a for loop. Once the data is read, we can use the split() method to separate the text into words.

Example 1: Splitting employee data with Python

The following code shows how to split a text file using the split() method in Python. In the example, we have used the “employee_data.txt” file defined above.

with open("employee_data.txt",'r') as data_file:
    for line in data_file:
        data = line.split()
        print(data)

Output

['Lana', 'Anderson', '485-3094-88', 'Electrician']
['Elian', 'Johnston', '751-5845-87', 'Interior', 'Designer']
['Henry', 'Johnston', '777-6561-52', 'Astronomer']
['Dale', 'Johnston', '248-1843-09', 'Journalist']
['Luke', 'Owens', '341-7471-63', 'Teacher']
['Amy', 'Perry', '494-3532-17', 'Electrician']
['Chloe', 'Baker', '588-7165-01', 'Interior', 'Designer']

In the above example, you can observe that the output contains a list of words for each line in the text file. In this case, the text is split at whitespaces, which is the default behavior of the split() method.

Splitting Strings With a Comma in Python

We can also split a text file at commas using the split() method in Python. For this, we can provide the comma character as an optional separator to the split() method to specify which character to split the string with.

To discuss how to split a text file at comma using the split() method in Python, let us first create a text file containing commas as shown below.

Janet,100,50,69
Thomas,99,76,100
Kate,102,78,65

You can save the data in the above snippet as grades.txt.

Example 2: Splitting grades with a comma

To split a text file with a comma in Python, we will first open the file. Then, we will read the file line by line using a for loop. We will first remove any extra spaces or newline characters from each line using the strip() method. The strip() method, when invoked on a string, removes any spaces or newline characters from the start and end of a string.

Next, we will invoke the split() method on the string returned by the strip() method and pass the comma character “,” as its input argument. After execution, the split() method returns a list having sub-strings of the original lines in the text file separated at commas. You can observe this in the following example.

with open("grades.txt",'r') as file:
    for line in file:
        grade_data = line.strip().split(',')
        print(grade_data)

The output of the above code looks as follows.

['Janet', '100', '50', '69']
['Thomas', '99', '76', '100']
['Kate', '102', '78', '65']

Splitting a Text File With splitlines() Method in Python

The splitlines() method is used to get a list of the lines in a text file. For the next examples, we’ll pretend we run a website that’s dedicated to a theatre company. We’re reading script data from text files and pushing it to the company’s website. To proceed ahead, you can save the following lines in a text file “juliet.txt”.

O Romeo, Romeo, wherefore art thou Romeo?
Deny thy father and refuse thy name.
Or if thou wilt not, be but sworn my love
And I'll no longer be a Capulet.

We can read the text file and split it at the newline characters with the splitlines() method to obtain a list of lines. Afterward, You can use a for loop to print the contents of the list as shown below.

Example 3: Using splitlines() to read a text file

In the following example, we will read the juliet.txt file into a variable named “script”. Then, we will use the read() method to read the contents of the file. After that, we will use the splitlines() method to split the text file into a list of lines. We will store the list in the variable “speech”. Finally, we will print the list contents using a for loop as shown below.

with open("juliet.txt",'r') as script:
    speech = script.read().splitlines()
print("The list is:")
print(speech)
for line in speech:
    print(line)

When we execute the above Python code, the output looks as follows.

Using a Generator to Split a Text File in Python

In Python, a generator is a special routine that we can use to create an iterable object. A generator in Python is similar to a function that returns an iterable object, but it does so one element at a time.

Generators use the yield keyword. When Python encounters a yield statement, it stores the state of the function until later, when the generator is called again.

In the next example, we’ll use a generator to read the beginning of Romeo’s famous speech from Shakespeare’s Romeo and Juliet. Using the yield keyword ensures that the state of our while loop is saved during each iteration. This can be useful when working with large files. To proceed ahead, let us create a file named romeo.txt with the following text.

But soft, what light through yonder window breaks?
It is the east, and Juliet is the sun.
Arise, fair sun, and kill the envious moon,
Who is already sick and pale with grief
That thou, her maid, art far more fair than she.

Example 4: Splitting a text file with a generator

We can split the above text file using a generator in Python as shown in the following example.

In the following code, we first create a generator named generator_read() that takes a file name as its input. It opens the file in read mode and reads the file contents using a while loop and the readline() method. Then, it gives each line as output using the yield statement.

When executed, it returns the contents of the line by line in a generator named file_data. We can iterate over this generator to access the file contents line by line. Finally, we will use the split() method to split each line of the text in Python using a generator as shown below.

def generator_read(file_name):
    file = open(file_name,'r')
    while True:
        line = file.readline()
        if not line:
            file.close()
            break
        yield line

file_data = generator_read("romeo.txt")
print("The generator is:")
print(file_data)
print("The file contents are:")
for line in file_data:
    print(line.split())

The output of the above code looks as follows.

Reading File Data with List Comprehension

Python list comprehension provides an elegant solution for working with lists. We can take advantage of shorter syntax to write our code with list comprehension instead of the for loop to read a file. In addition, list comprehension statements are usually easier to read.

In our previous examples, we’ve had to use a for loop to read the text files. We can exchange our for loop for a single line of code using list comprehension.

For this, we will first read open a file using the open() method. Then, we will use list comprehension to create a list of lines in the file.

Once we get the data via list comprehension, we will use the split() method to split the lines and add them to a new list as shown below.

with open("romeo.txt",'r') as file:
    lines = [line.strip() for line in file]
print("The list is:")
print(lines)
print("The data after splitting is:")
for line in lines:
    print(line.split())

The output of the above code looks as follows.

The list is:
['But soft, what light through yonder window breaks?', 'It is the east, and Juliet is the sun.', 'Arise, fair sun, and kill the envious moon,', 'Who is already sick and pale with grief', 'That thou, her maid, art far more fair than she.']
The data after splitting is:
['But', 'soft,', 'what', 'light', 'through', 'yonder', 'window', 'breaks?']
['It', 'is', 'the', 'east,', 'and', 'Juliet', 'is', 'the', 'sun.']
['Arise,', 'fair', 'sun,', 'and', 'kill', 'the', 'envious', 'moon,']
['Who', 'is', 'already', 'sick', 'and', 'pale', 'with', 'grief']
['That', 'thou,', 'her', 'maid,', 'art', 'far', 'more', 'fair', 'than', 'she.']

Split a Text File into Multiple Smaller Files in Python

What if we have a large file that we’d like to split into smaller files? We split a large file in Python using for loops and slicing.

With list slicing, we tell Python we want to work with a specific range of elements from a given list. This is done by providing a start point and end point for the slice.

In Python, a list can be sliced using the indexing operator. In the following example, we’ll use list slicing to split a text file into multiple smaller files.

Split a File with List Slicing in Python

To split a text file into multiple files in python, we can use list slicing. For this, we will first open the file in read mode. Then, we will obtain a list of lines in the text file using the readlines() method. Next, the top half of the file is written to a new file called romeo_A.txt. We’ll use list slicing within this for loop to write the first half of the original file to a new file. Using a second for loop, we’ll write the rest of the text to another file. In order to perform the slice, we will also use the len() method to find the total number of lines in the original file.

You can observe this in the following example.

with open("romeo.txt",'r') as file:
    lines = file.readlines()

with open("romeo_A.txt",'w') as file:
    for line in lines[:int(len(lines)/2)]:
        file.write(line)

with open("romeo_B.txt",'w') as file:
    for line in lines[int(len(lines)/2):]:
        file.write(line)

Running this program in the same directory as romeo.txt will create the following text files.

romeo_A.txt
But soft, what light through yonder window breaks?
It is the east, and Juliet is the sun.

romeo_B.txt
Arise, fair sun, and kill the envious moon,
Who is already sick and pale with grief
That thou, her maid, art far more fair than she.

Conclusion

In this article, we discussed how to use the split() method to split a text file in Python. Additionally, our examples have shown how the split() method is used in tandem with Python generators and list comprehension to read large files more elegantly. Taking advantage of Python’s many built-in methods, such as split() and readlines(), allows us to process text files more quickly. Using these tools will save us time and effort.

If you’re serious about mastering Python, it’s a good idea to invest some time in learning how to use these methods to prepare your own solutions. If you’d like to learn more about programming with Python, you can read this article on tuple comprehension in Python. You might also like this article on unpacking in Python.

I hope you enjoyed reading this article. Stay tuned for more informative articles.

Happy learning!