Python Threading

Summary: in this tutorial, you’ll learn how to use the Python threading module to develop multi-threaded applications.

Single-threaded applications

Let’s start with a simple program:

from time import sleep, perf_counter def task(): print('Starting a task...') sleep(1) print('done') start_time = perf_counter() task() task() end_time = perf_counter() print(f'It took {end_time- start_time: 0.2f} second(s) to complete.')
Code language: Python (python)

How it works.

First, import the sleep() and perf_counter() functions from the time module:

from time import sleep, perf_counter
Code language: Python (python)

Second, define a function that takes one second to complete:

def task(): print('Starting a task...') sleep(1) print('done')
Code language: Python (python)

Third, get the value of the performance counter by calling the perf_counter() function:

start_time = perf_counter()
Code language: Python (python)

Fourth, call the task() function twice:

task() task()
Code language: Python (python)

Fifth, get the value of the performance counter calling the perf_counter() function:

end_time = perf_counter()
Code language: Python (python)

Finally, output the time that takes to complete running the task() function twice:

print(f'It took {end_time- start_time: 0.2f} second(s) to complete.')
Code language: Python (python)

Here is the output:

Starting a task... done Starting a task... done It took 2.00 second(s) to complete.
Code language: Python (python)

As you may expect, the program takes about two seconds to complete. If you call the task() function 10 times, it would take about 10 seconds to complete.

The following diagram illustrates how the program works:

First, the task() function executed and slept for one second. Then it executed a second time and also slept for another second. Finally, the program completes.

When the task() function called the sleep() function, the CPU didn’t do anything. This is not efficient.

This program has one process with a single thread, which is called the main thread.

Because the program has only one thread, it’s called the single-threaded program.

Using Python threading to develop a multi-threaded program example

To create a multi-threaded program, you need to use the Python threading module.

First, import the Thread class from the threading module:

from threading import Thread
Code language: Python (python)

Second, create a new thread by instantiating an instance of the Thread class:

new_thread = Thread(target=fn,args=args_tuple)
Code language: Python (python)

The Thread() accepts many parameters. The main ones are:

  • target: specifies a function (fn) to run in the new thread.
  • args: specifies the arguments of the function (fn). The args argument is a tuple.

Third, start the thread by calling the start() method of the Thread instance:

new_thread.start()
Code language: Python (python)

If you want to wait for the thread to complete in the main thread, you can call the join() method:

new_thread.join()
Code language: Python (python)

By calling the join() method, the main thread will wait for the second thread to complete before it is terminated.

The following program illustrates how to use the threading module:

from time import sleep, perf_counter from threading import Thread def task(): print('Starting a task...') sleep(1) print('done') start_time = perf_counter() # create two new threads t1 = Thread(target=task) t2 = Thread(target=task) # start the threads t1.start() t2.start() # wait for the threads to complete t1.join() t2.join() end_time = perf_counter() print(f'It took {end_time- start_time: 0.2f} second(s) to complete.')
Code language: Python (python)

How it works. (and we’ll focus on the threading part only)

First, create two new threads:

t1 = Thread(target=task) t2 = Thread(target=task)

Second, start both both threads by calling the start() method:

t1.start() t2.start()
Code language: Python (python)

Third, wait for both threads to complete:

t1.join() t2.join()
Code language: Python (python)

Finally, show the executing time:

print(f'It took {end_time- start_time: 0.2f} second(s) to complete.')
Code language: Python (python)

Output:

Starting a task... Starting a task... done done It took 1.00 second(s) to complete.
Code language: Python (python)

When the program executes, it’ll have three threads: the main thread is created by the Python interpreter, and two new threads created by the program.

As shown clearly from the output, the program took one second instead of two to complete.

The following diagram shows how threads execute:

Passing arguments to threads

The following program illustrates how to pass arguments to the function assigned to a thread:

from time import sleep, perf_counter from threading import Thread def task(id): print(f'Starting the task {id}...') sleep(1) print('done') start_time = perf_counter() # create and start 10 threads threads = [] for n in range(1, 11): t = Thread(target=task, args=(n,)) threads.append(t) t.start() # wait for the threads to complete for t in threads: t.join() end_time = perf_counter() print(f'It took {end_time- start_time: 0.2f} second(s) to complete.')
Code language: Python (python)

How it works.

First, define a task() function that accepts the an argument:

def task(id): print(f'Starting the task {id}...') sleep(1) print('done')
Code language: Python (python)

Second, create 10 new threads and pass an id to each. The threads list is used to keep track of all newly created threads:

threads = [] for n in range(1, 11): t = Thread(target=task, args=(n,)) threads.append(t) t.start()
Code language: Python (python)

Notice that if you call the join() method inside the loop, the program will wait for the first thread to complete before starting the next one.

Third, wait for all threads to complete by calling the join() method:

for t in threads: t.join()
Code language: Python (python)

The following shows the output of the program:

Starting the task 1... Starting the task 2... Starting the task 3... Starting the task 4... Starting the task 5... Starting the task 6... Starting the task 7... Starting the task 8... Starting the task 9... Starting the task 10... done done done done done done done done done done It took 1.05 second(s) to complete.
Code language: Python (python)

It just took 1.05 seconds to complete.

When to use Python threading

As introduced in the process and thread tutorial, there’re two main tasks:

  • I/O-bound tasks – the time spending on I/O significantly more than the time spending on computation
  • CPU-bound tasks – the time spending on computation is significantly higher than the time waiting for I/O.

Python threading is optimized for I/O bound tasks. For example, requesting remote resources, connecting a database server, or reading and writing files.

A Practical Python threading example

Suppose that you have a list of text files in a folder e.g., C:/temp/. And you want to replace a text with a new one in all the files.

The following single-threaded program shows how to replace a substring with the new one in the text files:

from time import perf_counter def replace(filename, substr, new_substr): print(f'Processing the file {filename}') # get the contents of the file with open(filename, 'r') as f: content = f.read() # replace the substr by new_substr content = content.replace(substr, new_substr) # write data into the file with open(filename, 'w') as f: f.write(content) def main(): filenames = [ 'c:/temp/test1.txt', 'c:/temp/test2.txt', 'c:/temp/test3.txt', 'c:/temp/test4.txt', 'c:/temp/test5.txt', 'c:/temp/test6.txt', 'c:/temp/test7.txt', 'c:/temp/test8.txt', 'c:/temp/test9.txt', 'c:/temp/test10.txt', ] for filename in filenames: replace(filename, 'ids', 'id') if __name__ == "__main__": start_time = perf_counter() main() end_time = perf_counter() print(f'It took {end_time- start_time :0.2f} second(s) to complete.')
Code language: Python (python)

Output:

It took 0.16 second(s) to complete.
Code language: Python (python)

The following program has the same functionality. However, it uses multiple threads instead:

from threading import Thread from time import perf_counter def replace(filename, substr, new_substr): print(f'Processing the file {filename}') # get the contents of the file with open(filename, 'r') as f: content = f.read() # replace the substr by new_substr content = content.replace(substr, new_substr) # write data into the file with open(filename, 'w') as f: f.write(content) def main(): filenames = [ 'c:/temp/test1.txt', 'c:/temp/test2.txt', 'c:/temp/test3.txt', 'c:/temp/test4.txt', 'c:/temp/test5.txt', 'c:/temp/test6.txt', 'c:/temp/test7.txt', 'c:/temp/test8.txt', 'c:/temp/test9.txt', 'c:/temp/test10.txt', ] # create threads threads = [Thread(target=replace, args=(filename, 'id', 'ids')) for filename in filenames] # start the threads for thread in threads: thread.start() # wait for the threads to complete for thread in threads: thread.join() if __name__ == "__main__": start_time = perf_counter() main() end_time = perf_counter() print(f'It took {end_time- start_time :0.2f} second(s) to complete.')
Code language: Python (python)

Output:

Processing the file c:/temp/test1.txt Processing the file c:/temp/test2.txt Processing the file c:/temp/test3.txt Processing the file c:/temp/test4.txt Processing the file c:/temp/test5.txt Processing the file c:/temp/test6.txt Processing the file c:/temp/test7.txt Processing the file c:/temp/test8.txt Processing the file c:/temp/test9.txt Processing the file c:/temp/test10.txt It took 0.02 second(s) to complete.
Code language: Python (python)

As you can see clearly from the output, the multi-thread program runs so much faster.

Summary

  • Use the threading module to create a multi-threaded application.
  • Use the Thread(function, args) to create a new thread.
  • Call the start() method of the Thread to start the thread.
  • Call the join() method o the Thread to wait for the thread to complete in the main thread.
  • Only use threading for I/O bound processing applications.
Did you find this tutorial helpful ?