Multiprocessing on Windows

If I can parallellize a process, I will!!!

I use the python library concurrent futures. After setting it up the first time, it's a script you can carry with you on so many projects!!!

If you remember in the last post ChatGPT gave me this script:

from concurrent.futures import ProcessPoolExecutor
with ProcessPoolExecutor() as executor: 
	results = list(executor.map(process_csv, csv_files))

Yes it's that easy!!! But, of course, if that's all you do you get an error:

....An attempt has been made to start a new process before the current process has finished its bootstrapping phase....

Feed that error to ChatGPT and it will respond (I really like its tone, btw! it's very patient and encouraging):

Ah, it looks like you're running into an issue with multiprocessing in Python. This happens mainly on Windows (and sometimes macOS) when using ProcessPoolExecutor.

🔍 Why is this happening?

1. On Windows, multiprocessing needs to "spawn" new processes instead of forking them (which Linux does).

2. If a script is not inside a if name == "__main__": block, Windows fails to properly initialize new processes.
   

   ✅ How to Fix It

   You need to wrap the multiprocessing code inside:

if __name__ == "__main__":
    with ProcessPoolExecutor() as executor:
        executor.map(process_csv_group, csv_groups)

Yes it's that easy!!!

Now you're ready to go..... I will do a separate blog getting into more details about concurrent futures and its many awesome features.

But one more recommendation. From me, not from Chat: Once you have your parallel script working (test it in serial mode first, and then with a small number of cases), don't run it in your IDE, such as spyder. Open up a conda prompt, change your directory to the one of your python script and then run it at the prompt by typing:

python myscript.py

This way you do not carry the extra possible weight of your ide or other processes. And you can stop all of the parallel processes using a control-C. If you try this in spyder you are not really stopping everything and it's a mess!