bitmap: We'll have to replace a SSD on yamaoka, can you check if we can move services elsewhere (or afford to stop them a short time for the replacement)? I'll handle that tomorrow (that's late here, going to bed)
2025-08-06 21818, 2025
bitmap[m]
zas: I've temporarily started artwork-indexer prod and CB prod containers on selda. other non-redundant containers on yamaoka include CB beta, mbspotify, and jenkins, but downtime shouldn't be problematic for those (not 100% sure about mbspotify though)
2025-08-06 21822, 2025
bitmap[m]
apparently CB was already running on selda but wasn't added to scripts/nodes/selda.sh
2025-08-06 21810, 2025
DjSlash has quit
2025-08-06 21833, 2025
DjSlash joined the channel
2025-08-06 21856, 2025
d4rk-ph0enix has quit
2025-08-06 21818, 2025
d4rk-ph0enix joined the channel
2025-08-06 21842, 2025
d4rk-ph0enix has quit
2025-08-06 21828, 2025
d4rk-ph0enix joined the channel
2025-08-06 21856, 2025
d4rk-ph0enix has quit
2025-08-06 21823, 2025
d4rk-ph0enix joined the channel
2025-08-06 21842, 2025
d4rk-ph0enix has quit
2025-08-06 21800, 2025
d4rk-ph0enix joined the channel
2025-08-06 21804, 2025
lucifer[m]
bitmap: i was looking at all yesterday to figure out which ones were running CB and i didn't have selda in my local script so skipped it. thanks!
2025-08-06 21835, 2025
lucifer[m]
reosarevok: CB issues should now be fixed, yamaoka and selda were both running CB but on different versions and that caused all the reported issues.
2025-08-06 21812, 2025
lucifer[m]
i have stopped all CB containers on yamaoka.
2025-08-06 21807, 2025
yvanzo[m]
<aerozol[m]> "yvanzo: I think you might be..." <- aerozol: I clearly wrote that “it is open to suggestions and the script’s code is organized in a way that makes it easy to play with colors”. There is no need to split people into made-up opposite camps, thank you. Please make actual suggestions for the color scheme instead: Pick one image such the cover art archive table structure, change the colors, and we will see.
2025-08-06 21833, 2025
aerozol[m]
I wasn't splitting anyone up... I just read a different part of a very long comment that you linked to.
2025-08-06 21821, 2025
aerozol[m]
My advice is to remove the background color and leave it plain. I will take the time to change it if it isn't going to be changed back
For everyone else, I continued this conversation in private with aerozol to go beyond unjustified assumptions before having to go. To be continued soon…
yamaoka disk replacement should occur within 2-4 hours
2025-08-06 21849, 2025
petitminion joined the channel
2025-08-06 21827, 2025
zas[m]
yamaoka disk was replaced, RAID array is rebuilding (ETA ~146 minutes), it is back to work
2025-08-06 21807, 2025
lucifer[m]
mayhem: troi ratelimiting is broken because multiple concurrent requests to explore/lb-radio invoke troi in separate threads and the global dict to manage ratelimits is concurrently modified.
the del operation fails because key is already removed from the dict.
2025-08-06 21845, 2025
lucifer[m]
by another thread/request.
2025-08-06 21833, 2025
lucifer[m]
there are two ways to fix this imo, either add a lock and only make one global request for troi at a time. but that will hurt lb-radio performance significantly.
2025-08-06 21800, 2025
lucifer[m]
make the dict storing the ratelimits a thead local. each thread/request gets its own dict.
2025-08-06 21847, 2025
lucifer[m]
there is a third option to make the dict accesses safer, use get/pop with defaults but might defeat the purpose of ratelimit logic.
2025-08-06 21819, 2025
lucifer[m]
hmm, no actually that might work fine too. we just need to decide whether we want global ratelimits on a best effort basis or per-thread ratelimits.
2025-08-06 21812, 2025
mayhem[m] reads
2025-08-06 21815, 2025
mayhem[m]
the del that fails, can we just ignore the failure. (likely not that simple)
2025-08-06 21803, 2025
mayhem[m]
<lucifer[m]> "mayhem: troi ratelimiting is..." <- let me clarify -- this is separate *processes* not requests, right?
2025-08-06 21818, 2025
mayhem[m]
threads
2025-08-06 21826, 2025
lucifer[m]
I'll have to check uwsgi configuration to see whether it's doing threads or processes, likely a combination of both
2025-08-06 21830, 2025
mayhem[m]
not processes. still waking up.
2025-08-06 21849, 2025
mayhem[m]
troi doesn't use threads, let me check.
2025-08-06 21858, 2025
lucifer[m]
No troi doesn't but uwsgi might be
2025-08-06 21807, 2025
mayhem[m]
it doesn't.
2025-08-06 21812, 2025
lucifer[m]
The issues was seen in lb prod api logs.
2025-08-06 21821, 2025
mayhem[m]
its specifically multi-process, but the processes themselves may use threads.
so, this double manipulation of the dict is rather curious.
2025-08-06 21830, 2025
lucifer[m]
We have enabled thread use from what I understand
2025-08-06 21839, 2025
lucifer[m]
So multiple threads per uwsgi process
2025-08-06 21849, 2025
mayhem[m]
lucifer[m]: yes, this means threads for child processes are enabled. I believe we can take that option away, since we're not usingit.
2025-08-06 21815, 2025
lucifer[m]
I think we are.
2025-08-06 21825, 2025
lucifer[m]
I'll check uwsgi docs again later to confirm
2025-08-06 21803, 2025
mayhem[m]
from gemini: "Enabling threads in uWSGI doesn't change it into a "threading server" in the same way a single-process, multi-threaded application works. Instead, it enables threading within each of its worker processes. The primary concurrency model of uWSGI remains multiprocess."
I understood it just like you do, but that is not how it works. faster-fuzzy exists because of this. I know the pain.
2025-08-06 21813, 2025
lucifer[m]
I understand that but with sleep the control can be returned to the process to start running another thread.
2025-08-06 21844, 2025
mayhem[m]
lucifer[m]: no. uwsgi itself uses no threads.
2025-08-06 21850, 2025
lucifer[m]
asyncio does that and if there is monkey patching involved using greenlet or gevent etc it can happen here too.
2025-08-06 21802, 2025
mayhem[m]
let me try turning off threads on lb-test to see if anything breaks, ok?
2025-08-06 21807, 2025
lucifer[m]
Okay sure.
2025-08-06 21859, 2025
lucifer[m]
lucifer[m]: Basically the second answer's first comment.
2025-08-06 21824, 2025
mayhem[m]
"/usr/local/lib/python3.13/site-packages/sentry_sdk/_compat.py:86: Warning: IMPORTANT: We detected the use of uWSGI in preforking mode without thread support. This might lead to crashing workers. Please run uWSGI with both "--enable-threads" and "--py-call-uwsgi-fork-hooks" for full support." when I turn threads to zero.
2025-08-06 21831, 2025
mayhem[m]
time to read about pre-forking mode.
2025-08-06 21811, 2025
mayhem[m]
oh, I see. its not a pure threading model, but a hybrid threads inside process model.
2025-08-06 21840, 2025
mayhem[m]
in that case, putting a simple Lock around the rate limit should fix the problem nicely.
2025-08-06 21808, 2025
mayhem[m]
well, it has the potential to fix it. but I am not conviced. lets pick this up when you're near a computer.
if the dict munging theory is correct, this should fix that.
2025-08-06 21816, 2025
aerozol[m]
julian45: Another 165 spam comments on the blog, since last time we talked D: Definitely need to get Akismet set up again
2025-08-06 21832, 2025
mayhem[m]
yea, on my task list for very soon.
2025-08-06 21804, 2025
aerozol[m]
Ah thanks mayhem
2025-08-06 21818, 2025
lucifer[m]
mayhem: yes that, ` but a hybrid threads inside process model.`
2025-08-06 21800, 2025
lucifer[m]
a lock will fix it yes by blocking other requests from progressing.
2025-08-06 21858, 2025
mayhem[m]
lucifer[m]: the locks are not held for long. should be fine.
2025-08-06 21855, 2025
mayhem[m]
remember, it works in most cases, so its a race condition. and an occaisonal block for a microsecond or two won't do any harm we can notice.
2025-08-06 21855, 2025
lucifer[m]
<mayhem[m]> "the del that fails, can we..." <- yes but i think we can do without locks, by this.
2025-08-06 21825, 2025
lucifer[m]
instead of del use pop with a None default.
2025-08-06 21829, 2025
mayhem[m]
sure, lets try that.
2025-08-06 21800, 2025
lucifer[m]
one more change by combining the key in dict and dict.get check.
2025-08-06 21837, 2025
lucifer[m]
will open a new pr to try that and otherwise we can use a lock yeah.
2025-08-06 21820, 2025
lucifer[m]
just for completeness sake, one more way to do this is thread local. which is what flask uses for its g and app context.
2025-08-06 21844, 2025
d4rkie has quit
2025-08-06 21815, 2025
d4rkie joined the channel
2025-08-06 21857, 2025
petitminion has quit
2025-08-06 21830, 2025
mayhem[m]
we just received a 1000Eur donation via a bank transfer and I have no way to contact them to thank them. how odd.
2025-08-06 21847, 2025
dvirtz[m] joined the channel
2025-08-06 21848, 2025
dvirtz[m] posted a file: (19KiB) < https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/uRMLJkBpNtHhTbjtMMdnYHYA/DBDefs.pm >
2025-08-06 21848, 2025
dvirtz[m]
I moved to a new machine and now my MB dev setup stopped working. I'm using musicbrainz-docker and can see in the db logs... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/OfGEUuWvvywrcGrevlBKPukf>)