Skip to main content


Personal growth is hard when you leave no room (time) for it.

Neil E. Hodges reshared this.


I used to love #Python, but dealing with the honestly kind of scary #multiprocessing library was a reminder of the #GIL #threading situation really hurts working with #parallelism. 😰🫠

I've started learning #Kotlin after honestly really enjoying the threading library it provides. It's so's easy to work with when you understand it! 😀 (I would prefer to stay away from some of the #Java conventions after working with them for so long. 👍) #JVM #programming


Had quite the time working on a personal project recently. It changed my life forever.
  • Learned the hard way that the (default on most POSIX) fork context is bad news.
  • Wrote a Unix domain datagram based log infrastructure.
  • Wrote an algorithm that operates kind of like concurrent.futures.as_completed(), except it has a priority queue and doesn't eagerly load the list of futures.
  • Discovered that it's possible to overload concurrent.futures.ProcessPoolExecutor with futures, preventing any actual background processing after a point.
  • Got TONS of practice optimizing stuff for large datasets.
  • Learned that taking breaks is important for reasons than most people are aware of.
  • My life was permanently altered by this project. I basically nerd-sniped myself.

#Python #programming


This entry was edited (1 week ago)

reshared this



Had quite the time working on a personal project recently. It changed my life forever.

  • Learned the hard way that the (default on most POSIX) fork context is bad news.
  • Wrote a Unix domain datagram based log infrastructure.
  • Wrote an algorithm that operates kind of like concurrent.futures.as_completed(), except it has a priority queue and doesn't eagerly load the list of futures.
  • Discovered that it's possible to overload concurrent.futures.ProcessPoolExecutor with futures, preventing any actual background processing after a point.
  • Got TONS of practice optimizing stuff for large datasets.
  • Learned that taking breaks is important for reasons than most people are aware of.
  • My life was permanently altered by this project. I basically nerd-sniped myself.

#Python #programming


Never, ever, use the fork context in multiprocessing. :blobfoxsweating: POSIX OSes (aside from macOS) have that as the default. :blobfoxangrylaugh: Always use the spawn context! :blobfoxdead:

Of course, unless you have an extreme edge case. :blobfoxgoogly: (Yes, the names are funny. :blobfoxgooglytrash: )

#Python #multiprocessing


Tech Cyborg reshared this.

in reply to Neil E. Hodges

This whole mess got me thinking seriously about learning #Kotlin, since it would eliminate needing to use IPC to make use of multiple CPU cores. (Yes, I actually like the #JVM, but don’t want to boilerplate myself to death with #Java. :P )

I’ll probably still look into learning Kotlin, but not for this project.


Shannon Prickett reshared this.


botocore.exceptions.ClientError: An error occurred (BadDigest) when calling the PutObject operation (reached max retries: 4): The Content-MD5 you specified did not match what we received.

The Content-MD5 header was generated by botocore (not me) and the file didn't change. What is happening? D': #Python #Amazon #AWS

reshared this



Is 2 GB too large for an #SQLite database performance-wise? 🤔 #Python

reshared this



Having to make heavy use of complicated multiprocessing libraries because of the #GIL is peak #Python. :(

Tech Cyborg reshared this.



This apparently doesn't work correctly. When I tried it, a bunch of log lines were duplicated. 😞

class LzmaFileHandler(FileHandler):
    def _open(self) -> IO[str]:
        return lzma.open(
            self.baseFilename,
            self.mode,
            encoding=self.encoding,
            errors=self.errors,
        )

I had to switch it to this to get it to work:
class CompressionAlgorithm(Enum):
    lzma = "lzma"
    gzip = "gzip"
    bzip2 = "bzip2"


def _compressed_handler(
    opener: Callable[[Path, str], IO[str]],
    filename: Path,
    mode: str = "at",
) -> logging.Handler:
    stream = opener(filename, "at")
    stream.reconfigure(write_through=True)
    atexit.register(stream.close)
    return StreamHandler(stream)
 
 
COMPRESSION_HANDLERS: Mapping[CompressionAlgorithm, Callable[[Path, str], logging.Handler]] = {
    CompressionAlgorithm.lzma: partial(_compressed_handler, lzma.open),
    CompressionAlgorithm.gzip: partial(_compressed_handler, gzip.open),
    CompressionAlgorithm.bzip2: partial(_compressed_handler, bz2.open),
}

Also, lzma buffers a ton in memory before writing, so it didn't really work for my purpose. Even bz2 didn't work, so I had to use gzip. 🙃

#Python

Tech Cyborg reshared this.



With more than one item, the context managers are processed as if multiple with statements were nested:
with A() as a, B() as b:
    SUITE

is semantically equivalent to:

with A() as a:
    with B() as b:
        SUITE

You can also write multi-item context managers in multiple lines if the items are surrounded by parentheses. For example:

with (
    A() as a,
    B() as b,
):
    SUITE

:O #Python

Tech Cyborg reshared this.

in reply to Neil E. Hodges

Non-GC languages with destructors: "Look what they need to mimic a fraction of our power!"

Sam Levine reshared this.


Anyone know how to read tar files from a pipe in #Python? I'm getting this error when using tarfile.open(path, mode="r|"):
  File "lib/python3.12/tarfile.py", line 690, in read
    self.fileobj.seek(offset + (self.position - start))
  File "lib/python3.12/tarfile.py", line 522, in seek
    raise StreamError("seeking backwards is not allowed")
tarfile.StreamError: seeking backwards is not allowed

reshared this

in reply to Neil E. Hodges

It looks like to write over a pipe, open the file first and pass the handle to tarfile. Might work for reading too?
https://stackoverflow.com/a/25333193
in reply to Neil E. Hodges

The solution ended up being replacing intar.extractfile(info.name) with intar.extractfile(info). 👍

reshared this



Tried using the CFFI library with libmtp to put together a script for grabbing photos off my phone, but libmtp didn't like #Android's custom flavor of MTP. Ended up just wrapping aft-mtp-cli instead. 👍 #Python



PEP 703 – Making the Global Interpreter Lock Optional in CPython


CPython’s global interpreter lock (“GIL”) prevents multiple threads from executing Python code at the same time. The GIL is an obstacle to using multi-core CPUs from Python efficiently. This PEP proposes adding a build configuration (--disable-gil) to CPython to let it run Python code without the global interpreter lock and with the necessary changes needed to make the interpreter thread-safe.


#python #programming #gil

Andrew Sterian reshared this.



I was originally annoyed by this, but I think it's probably for the best. #python #linux
% sudo pip install python-magic
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try 'pacman -S
    python-xyz', where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Arch-packaged Python package,
    create a virtual environment using 'python -m venv path/to/venv'.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip.
    
    If you wish to install a non-Arch packaged Python application,
    it may be easiest to use 'pipx install xyz', which will manage a
    virtual environment for you. Make sure you have python-pipx
    installed via pacman.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

in reply to Neil E. Hodges

how did we get to the point when we're surprised that a repository includes the latest version of a tool released "just" half a year ago? Now I see why people talk that much about #NixOS.
in reply to gram

@orsinium I know somebody who is very disappointed that bookworm doesn't have golang 1.20 when it was released "months ago". I think I can see why - bookworm was already partially frozen then. But explaining this doesn't help.

On the other hand it does look like golang 1.20 got into nixos 23.05, if I am reading this correctly.

@gram
in reply to gram

@orsinium

Thats quite fast for debian.
And that being fast for debian, is the mainreasen to not use debian on the desktop. (Which I did for a very long time btw.)

@gram

Neil E. Hodges reshared this.


I think I found a #Python #bug today. If you try to do a csv.writer.writerow() and one of the strings (not bytes) in the row contains a null character in it, and you are using a csv.Dialect with escapechar set to None, it will result in this condition evaluating to true when it shouldn't. That will almost immediately result in the need to escape, but no escapechar set exception being thrown. #programming

reshared this



Fixing the GIL with from concurrent.futures import ProcessPoolExecutor. #Python

Neil E. Hodges reshared this.


Learned a new #Python trick on the job:
@contextmanager
def func(…):
    resource = …
    try:
        yield resource
    finally:
        …

#programming

reshared this