I used to love #Python, but dealing with the honestly kind of scary #multiprocessing library was a reminder of the #GIL #threading situation really hurts working with #parallelism. 😰🫠

I've started learning #Kotlin after honestly really enjoying the threading library it provides. It's so's easy to work with when you understand it! 😀 (I would prefer to stay away from some of the #Java conventions after working with them for so long. 👍) #JVM #programming

Neil E. Hodges

2024-09-24 14:40:13

Had quite the time working on a personal project recently. It changed my life forever.

• Learned the hard way that the (default on most POSIX) fork context is bad news.
• Wrote a Unix domain datagram based log infrastructure.
• Wrote an algorithm that operates kind of like concurrent.futures.as_completed(), except it has a priority queue and doesn't eagerly load the list of futures.
• Discovered that it's possible to overload concurrent.futures.ProcessPoolExecutor with futures, preventing any actual background processing after a point.
• Got TONS of practice optimizing stuff for large datasets.
• Learned that taking breaks is important for reasons than most people are aware of.
• My life was permanently altered by this project. I basically nerd-sniped myself.

#Python #programming

Had quite the time working on a personal project recently. It changed my life forever.

• Learned the hard way that the (default on most POSIX) fork context is bad news.
• Wrote a Unix domain datagram based log infrastructure.
• Wrote an algorithm that operates kind of like concurrent.futures.as_completed(), except it has a priority queue and doesn't eagerly load the list of futures.
• Discovered that it's possible to overload concurrent.futures.ProcessPoolExecutor with futures, preventing any actual background processing after a point.
• Got TONS of practice optimizing stuff for large datasets.
• Learned that taking breaks is important for reasons than most people are aware of.
• My life was permanently altered by this project. I basically nerd-sniped myself.

#Python #programming

Doughnut Lollipop 【記録係】:blobfoxgooglymlem:

2024-09-23 03:21:07

Never, ever, use the fork context in multiprocessing. POSIX OSes (aside from macOS) have that as the default. Always use the spawn context!

Of course, unless you have an extreme edge case. (Yes, the names are funny. )

#Python #multiprocessing

multiprocessing — Process-based parallelism

Source code: Lib/multiprocessing/ Availability: not Emscripten, not WASI. This module does not work or is not available on WebAssembly platforms wasm32-emscripten and wasm32-wasi. See WebAssembly p...

^{Python documentation}

botocore.exceptions.ClientError: An error occurred (BadDigest) when calling the PutObject operation (reached max retries: 4): The Content-MD5 you specified did not match what we received.

This apparently doesn't work correctly. When I tried it, a bunch of log lines were duplicated. 😞

class LzmaFileHandler(FileHandler):
    def _open(self) -> IO[str]:
        return lzma.open(
            self.baseFilename,
            self.mode,
            encoding=self.encoding,
            errors=self.errors,
        )

class CompressionAlgorithm(Enum):
    lzma = "lzma"
    gzip = "gzip"
    bzip2 = "bzip2"


def _compressed_handler(
    opener: Callable[[Path, str], IO[str]],
    filename: Path,
    mode: str = "at",
) -> logging.Handler:
    stream = opener(filename, "at")
    stream.reconfigure(write_through=True)
    atexit.register(stream.close)
    return StreamHandler(stream)
 
 
COMPRESSION_HANDLERS: Mapping[CompressionAlgorithm, Callable[[Path, str], logging.Handler]] = {
    CompressionAlgorithm.lzma: partial(_compressed_handler, lzma.open),
    CompressionAlgorithm.gzip: partial(_compressed_handler, gzip.open),
    CompressionAlgorithm.bzip2: partial(_compressed_handler, bz2.open),
}

#Python

With more than one item, the context managers are processed as if multiple with statements were nested:

with A() as a, B() as b:
    SUITE

is semantically equivalent to:

with A() as a:
    with B() as b:
        SUITE

You can also write multi-item context managers in multiple lines if the items are surrounded by parentheses. For example:

with (
    A() as a,
    B() as b,
):
    SUITE

:O #Python

  File "lib/python3.12/tarfile.py", line 690, in read
    self.fileobj.seek(offset + (self.position - start))
  File "lib/python3.12/tarfile.py", line 522, in seek
    raise StreamError("seeking backwards is not allowed")
tarfile.StreamError: seeking backwards is not allowed

PEP 703 – Making the Global Interpreter Lock Optional in CPython

CPython’s global interpreter lock (“GIL”) prevents multiple threads from executing Python code at the same time. The GIL is an obstacle to using multi-core CPUs from Python efficiently. This PEP proposes adding a build configuration (--disable-gil) to CPython to let it run Python code without the global interpreter lock and with the necessary changes needed to make the interpreter thread-safe.

#python #programming #gil

% sudo pip install python-magic
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try 'pacman -S
    python-xyz', where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Arch-packaged Python package,
    create a virtual environment using 'python -m venv path/to/venv'.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip.
    
    If you wish to install a non-Arch packaged Python application,
    it may be easiest to use 'pipx install xyz', which will manage a
    virtual environment for you. Make sure you have python-pipx
    installed via pacman.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

@contextmanager
def func(…):
    resource = …
    try:
        yield resource
    finally:
        …