Had quite the time working on a personal project recently. It changed my life forever.

• Learned the hard way that the (default on most POSIX) fork context is bad news.
• Wrote a Unix domain datagram based log infrastructure.
• Wrote an algorithm that operates kind of like concurrent.futures.as_completed(), except it has a priority queue and doesn't eagerly load the list of futures.
• Discovered that it's possible to overload concurrent.futures.ProcessPoolExecutor with futures, preventing any actual background processing after a point.
• Got TONS of practice optimizing stuff for large datasets.
• Learned that taking breaks is important for reasons than most people are aware of.
• My life was permanently altered by this project. I basically nerd-sniped myself.

#Python #programming

Doughnut Lollipop 【記録係】:blobfoxgooglymlem:

2024-09-23 03:21:07

Never, ever, use the fork context in multiprocessing. POSIX OSes (aside from macOS) have that as the default. Always use the spawn context!

Of course, unless you have an extreme edge case. (Yes, the names are funny. )

#Python #multiprocessing

multiprocessing — Process-based parallelism

Source code: Lib/multiprocessing/ Availability: not Emscripten, not WASI. This module does not work or is not available on WebAssembly platforms wasm32-emscripten and wasm32-wasi. See WebAssembly p...

^{Python documentation}