Understanding Differences in Behavior of Process.join() in Windows 10 and Ubuntu 18.04 in Python 3.6

When it comes to multi-processing in Python, developers often run into differences in behavior between different operating systems. One such difference is in how Windows 10 and Ubuntu 18.04 handle the Process.join() function. In this article, we will explore this difference in behavior and understand what accounts for it.

In this code snippet, we start a subprocess and then call join() on it in the main process. The subprocess runs a thread that is not terminated.

import threading

import time

from multiprocessing import Process



def thread_f():

    print("Thread function")

    while True:

        time.sleep(1)



def subprocess_f():

    print("TOP of Subprocess function")

    thread = threading.Thread(target=thread_f)

    thread.start()

    threads = ', '.join(map(str, [(thread.name, thread.ident, "isDaemon {}".format(thread.isDaemon())) for thread in

                                  threading.enumerate()]))

    print(threads)

    print("BOTTOM of Subprocess function")



def main():

    print("TOP of main")

    process = Process(target=subprocess_f)

    process.start()

    process.join()

    print("BOTTOM of main")



if __name__ == '__main__':

    main()

In Windows 10, the join() function continues to block, and the main process runs forever. However, in Ubuntu Linux 18.04, the subprocess ends, and the main process ends as well.

This difference in behavior can be attributed to how Windows and Linux handle child processes differently. In Windows, child processes are created using the CreateProcess() API, which creates a new process using the same address space as the parent process. In contrast, Linux creates child processes using the fork() system call, which creates a new process with a new address space that is a copy of the parent process's address space.

This difference in the way processes are created affects the behavior of the join() function. When a child process is created using CreateProcess(), it inherits a copy of all the handles of the parent process, including the handles to any open files or sockets. Therefore, when the parent process calls join(), it waits for the child process to terminate and close all its handles, including any open files or sockets. If the child process never terminates, the parent process waits forever.

In contrast, when a child process is created using fork(), it creates a copy of the parent process's address space, including all the open file descriptors. However, each file descriptor has its copy-on-write flag set, so any modifications to the file descriptor are made to a copy of the underlying data. Therefore, when the child process terminates, it closes all its file descriptors, including any open files or sockets. When the parent process calls join(), it simply waits for the child process to terminate, and it does not wait for the child process to close any open files or sockets.

To fix the issue with the join() function blocking in Windows, we can set the daemon flag of the thread to True. This will make the thread a daemon thread, which means it will be automatically terminated when the main process terminates.

def thread_f():
    print("Thread function")
    while True:
        time.sleep(1)

def subprocess_f():
    print("TOP of Subprocess function")
    thread = threading.Thread(target=thread_f)
    thread.daemon = True
    thread.start()
    threads = ', '.join(map(str, [(thread.name, thread.ident, "isDaemon {}".format(thread.isDaemon())) for thread in
                                  threading.enumerate()]))

To fix this issue, you need to tell the child process to terminate its threads before it exits. This can be achieved by setting the daemon property of the thread to True. A daemon thread is a thread that runs in the background and does not prevent the main program from exiting. When a daemon process exits, all of its daemon threads are terminated automatically.

Here's the modified code:

import threading
import time
from multiprocessing import Process

def thread_f():
    print("Thread function")
    while True:
        time.sleep(1)

def subprocess_f():
    print("TOP of Subprocess function")
    thread = threading.Thread(target=thread_f)
    thread.daemon = True  # Set daemon flag
    thread.start()
    threads = ', '.join(map(str, [(thread.name, thread.ident, "isDaemon {}".format(thread.isDaemon())) for thread in
                                  threading.enumerate()]))
    print(threads)
    print("BOTTOM of Subprocess function")

def main():
    print("TOP of main")
    process = Process(target=subprocess_f)
    process.start()
    process.join()
    print("BOTTOM of main")

# In both Windows and Linux, the main process exits cleanly.
if __name__ == '__main__':
    main()

In this modified version, we set the daemon flag of the thread to True before starting it. Now, when the subprocess exits, its daemon thread is terminated automatically, and the main process is free to exit cleanly.

Conclusion

In this blog post, we've explored the difference in behavior between Windows and Linux when using the Process.join() function in Python's multiprocessing module. We've seen that on Windows, the main process continues to block even after the subprocess has finished, while on Linux, the main process exits cleanly when the subprocess finishes.

We've also explained the root cause of this behavior: the fact that the subprocess is not terminating its threads. Finally, we've provided a solution to this issue by setting the daemon flag of the thread to True.

If you're working with Python's multiprocessing module, it's important to be aware of these platform-specific differences and to write code that works correctly on both Windows and Linux. With the solution provided in this blog post, you should be able to avoid the common pitfall of blocking processes and ensure that your code exits cleanly on all platforms.