I used to love #Python, but dealing with the honestly kind of scary #multiprocessing library was a reminder of the #GIL #threading situation really hurts working with #parallelism. 😰🫠
I've started learning #Kotlin after honestly really enjoying the threading library it provides. It's so's easy to work with when you understand it! 😀 (I would prefer to stay away from some of the #Java conventions after working with them for so long. 👍) #JVM #programming
like this
reshared this
Had quite the time working on a personal project recently. It changed my life forever.
- Learned the hard way that the (default on most POSIX)
fork
context is bad news. - Wrote a Unix domain datagram based log infrastructure.
- Wrote an algorithm that operates kind of like
concurrent.futures.as_completed()
, except it has a priority queue and doesn't eagerly load the list of futures. - Discovered that it's possible to overload
concurrent.futures.ProcessPoolExecutor
with futures, preventing any actual background processing after a point. - Got TONS of practice optimizing stuff for large datasets.
- Learned that taking breaks is important for reasons than most people are aware of.
- My life was permanently altered by this project. I basically nerd-sniped myself.
like this
Tech Cyborg reshared this.
botocore.exceptions.ClientError: An error occurred (BadDigest) when calling the PutObject operation (reached max retries: 4): The Content-MD5 you specified did not match what we received.
The
Content-MD5
header was generated by botocore
(not me) and the file didn't change. What is happening? D': #Python #Amazon #AWSlike this
reshared this
This apparently doesn't work correctly. When I tried it, a bunch of log lines were duplicated. 😞
class LzmaFileHandler(FileHandler):
def _open(self) -> IO[str]:
return lzma.open(
self.baseFilename,
self.mode,
encoding=self.encoding,
errors=self.errors,
)
I had to switch it to this to get it to work:
class CompressionAlgorithm(Enum):
lzma = "lzma"
gzip = "gzip"
bzip2 = "bzip2"
def _compressed_handler(
opener: Callable[[Path, str], IO[str]],
filename: Path,
mode: str = "at",
) -> logging.Handler:
stream = opener(filename, "at")
stream.reconfigure(write_through=True)
atexit.register(stream.close)
return StreamHandler(stream)
COMPRESSION_HANDLERS: Mapping[CompressionAlgorithm, Callable[[Path, str], logging.Handler]] = {
CompressionAlgorithm.lzma: partial(_compressed_handler, lzma.open),
CompressionAlgorithm.gzip: partial(_compressed_handler, gzip.open),
CompressionAlgorithm.bzip2: partial(_compressed_handler, bz2.open),
}
Also,
lzma
buffers a ton in memory before writing, so it didn't really work for my purpose. Even bz2
didn't work, so I had to use gzip
. 🙃dieter_wilhelm likes this.
Tech Cyborg reshared this.
With more than one item, the context managers are processed as if multiplewith
statements were nested:with A() as a, B() as b: SUITE
is semantically equivalent to:
with A() as a: with B() as b: SUITE
You can also write multi-item context managers in multiple lines if the items are surrounded by parentheses. For example:
with ( A() as a, B() as b, ): SUITE
:O #Python
dieter_wilhelm likes this.
Tech Cyborg reshared this.
Neil E. Hodges likes this.
tarfile.open(path, mode="r|")
: File "lib/python3.12/tarfile.py", line 690, in read
self.fileobj.seek(offset + (self.position - start))
File "lib/python3.12/tarfile.py", line 522, in seek
raise StreamError("seeking backwards is not allowed")
tarfile.StreamError: seeking backwards is not allowed
like this
reshared this
https://stackoverflow.com/a/25333193
python tarfile writes tar to pipe
I want to create a tar file and pipe it to a http upload. However, seems python tarfile module performs seek which make it impossible to pipe to the next process. Here is the code tar = tarfile....Stack Overflow
intar.extractfile(info.name)
with intar.extractfile(info)
. 👍David Zaslavsky likes this.
reshared this
PEP 703 – Making the Global Interpreter Lock Optional in CPython
CPython’s global interpreter lock (“GIL”) prevents multiple threads from executing Python code at the same time. The GIL is an obstacle to using multi-core CPUs from Python efficiently. This PEP proposes adding a build configuration (--disable-gil
) to CPython to let it run Python code without the global interpreter lock and with the necessary changes needed to make the interpreter thread-safe.
#python #programming #gil
dieter_wilhelm likes this.
Andrew Sterian reshared this.
% sudo pip install python-magic
error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try 'pacman -S
python-xyz', where xyz is the package you are trying to
install.
If you wish to install a non-Arch-packaged Python package,
create a virtual environment using 'python -m venv path/to/venv'.
Then use path/to/venv/bin/python and path/to/venv/bin/pip.
If you wish to install a non-Arch packaged Python application,
it may be easiest to use 'pipx install xyz', which will manage a
virtual environment for you. Make sure you have python-pipx
installed via pacman.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
like this
@orsinium I know somebody who is very disappointed that bookworm doesn't have golang 1.20 when it was released "months ago". I think I can see why - bookworm was already partially frozen then. But explaining this doesn't help.
On the other hand it does look like golang 1.20 got into nixos 23.05, if I am reading this correctly.
csv.writer.writerow()
and one of the strings (not bytes) in the row contains a null character in it, and you are using a csv.Dialect
with escapechar
set to None
, it will result in this condition evaluating to true when it shouldn't. That will almost immediately result in the need to escape, but no escapechar set
exception being thrown. #programminglike this
reshared this
from concurrent.futures import ProcessPoolExecutor
. #Pythondieter_wilhelm likes this.
@contextmanager
def func(…):
resource = …
try:
yield resource
finally:
…
#programming
like this
reshared this
Neil E. Hodges
in reply to Neil E. Hodges • •This whole mess got me thinking seriously about learning #Kotlin, since it would eliminate needing to use IPC to make use of multiple CPU cores. (Yes, I actually like the #JVM, but don’t want to boilerplate myself to death with #Java. :P )
I’ll probably still look into learning Kotlin, but not for this project.
Neil E. Hodges
in reply to Neil E. Hodges • •