bitmap: We'll have to replace a SSD on yamaoka, can you check if we can move services elsewhere (or afford to stop them a short time for the replacement)? I'll handle that tomorrow (that's late here, going to bed)
2025-08-06 21818, 2025
bitmap[m]
zas: I've temporarily started artwork-indexer prod and CB prod containers on selda. other non-redundant containers on yamaoka include CB beta, mbspotify, and jenkins, but downtime shouldn't be problematic for those (not 100% sure about mbspotify though)
2025-08-06 21822, 2025
bitmap[m]
apparently CB was already running on selda but wasn't added to scripts/nodes/selda.sh
2025-08-06 21810, 2025
DjSlash has quit
2025-08-06 21833, 2025
DjSlash joined the channel
2025-08-06 21856, 2025
d4rk-ph0enix has quit
2025-08-06 21818, 2025
d4rk-ph0enix joined the channel
2025-08-06 21842, 2025
d4rk-ph0enix has quit
2025-08-06 21828, 2025
d4rk-ph0enix joined the channel
2025-08-06 21856, 2025
d4rk-ph0enix has quit
2025-08-06 21823, 2025
d4rk-ph0enix joined the channel
2025-08-06 21842, 2025
d4rk-ph0enix has quit
2025-08-06 21800, 2025
d4rk-ph0enix joined the channel
2025-08-06 21804, 2025
lucifer[m]
bitmap: i was looking at all yesterday to figure out which ones were running CB and i didn't have selda in my local script so skipped it. thanks!
2025-08-06 21835, 2025
lucifer[m]
reosarevok: CB issues should now be fixed, yamaoka and selda were both running CB but on different versions and that caused all the reported issues.
2025-08-06 21812, 2025
lucifer[m]
i have stopped all CB containers on yamaoka.
2025-08-06 21807, 2025
yvanzo[m]
<aerozol[m]> "yvanzo: I think you might be..." <- aerozol: I clearly wrote that “it is open to suggestions and the script’s code is organized in a way that makes it easy to play with colors”. There is no need to split people into made-up opposite camps, thank you. Please make actual suggestions for the color scheme instead: Pick one image such the cover art archive table structure, change the colors, and we will see.
2025-08-06 21833, 2025
aerozol[m]
I wasn't splitting anyone up... I just read a different part of a very long comment that you linked to.
2025-08-06 21821, 2025
aerozol[m]
My advice is to remove the background color and leave it plain. I will take the time to change it if it isn't going to be changed back
For everyone else, I continued this conversation in private with aerozol to go beyond unjustified assumptions before having to go. To be continued soon…
yamaoka disk replacement should occur within 2-4 hours
2025-08-06 21849, 2025
petitminion joined the channel
2025-08-06 21827, 2025
zas[m]
yamaoka disk was replaced, RAID array is rebuilding (ETA ~146 minutes), it is back to work
2025-08-06 21807, 2025
lucifer[m]
mayhem: troi ratelimiting is broken because multiple concurrent requests to explore/lb-radio invoke troi in separate threads and the global dict to manage ratelimits is concurrently modified.
the del operation fails because key is already removed from the dict.
2025-08-06 21845, 2025
lucifer[m]
by another thread/request.
2025-08-06 21833, 2025
lucifer[m]
there are two ways to fix this imo, either add a lock and only make one global request for troi at a time. but that will hurt lb-radio performance significantly.
2025-08-06 21800, 2025
lucifer[m]
make the dict storing the ratelimits a thead local. each thread/request gets its own dict.
2025-08-06 21847, 2025
lucifer[m]
there is a third option to make the dict accesses safer, use get/pop with defaults but might defeat the purpose of ratelimit logic.
2025-08-06 21819, 2025
lucifer[m]
hmm, no actually that might work fine too. we just need to decide whether we want global ratelimits on a best effort basis or per-thread ratelimits.
2025-08-06 21812, 2025
mayhem[m] reads
2025-08-06 21815, 2025
mayhem[m]
the del that fails, can we just ignore the failure. (likely not that simple)
2025-08-06 21803, 2025
mayhem[m]
<lucifer[m]> "mayhem: troi ratelimiting is..." <- let me clarify -- this is separate *processes* not requests, right?
2025-08-06 21818, 2025
mayhem[m]
threads
2025-08-06 21826, 2025
lucifer[m]
I'll have to check uwsgi configuration to see whether it's doing threads or processes, likely a combination of both
2025-08-06 21830, 2025
mayhem[m]
not processes. still waking up.
2025-08-06 21849, 2025
mayhem[m]
troi doesn't use threads, let me check.
2025-08-06 21858, 2025
lucifer[m]
No troi doesn't but uwsgi might be
2025-08-06 21807, 2025
mayhem[m]
it doesn't.
2025-08-06 21812, 2025
lucifer[m]
The issues was seen in lb prod api logs.
2025-08-06 21821, 2025
mayhem[m]
its specifically multi-process, but the processes themselves may use threads.
so, this double manipulation of the dict is rather curious.
2025-08-06 21830, 2025
lucifer[m]
We have enabled thread use from what I understand
2025-08-06 21839, 2025
lucifer[m]
So multiple threads per uwsgi process
2025-08-06 21849, 2025
mayhem[m]
lucifer[m]: yes, this means threads for child processes are enabled. I believe we can take that option away, since we're not usingit.
2025-08-06 21815, 2025
lucifer[m]
I think we are.
2025-08-06 21825, 2025
lucifer[m]
I'll check uwsgi docs again later to confirm
2025-08-06 21803, 2025
mayhem[m]
from gemini: "Enabling threads in uWSGI doesn't change it into a "threading server" in the same way a single-process, multi-threaded application works. Instead, it enables threading within each of its worker processes. The primary concurrency model of uWSGI remains multiprocess."
I understood it just like you do, but that is not how it works. faster-fuzzy exists because of this. I know the pain.
2025-08-06 21813, 2025
lucifer[m]
I understand that but with sleep the control can be returned to the process to start running another thread.
2025-08-06 21844, 2025
mayhem[m]
lucifer[m]: no. uwsgi itself uses no threads.
2025-08-06 21850, 2025
lucifer[m]
asyncio does that and if there is monkey patching involved using greenlet or gevent etc it can happen here too.
2025-08-06 21802, 2025
mayhem[m]
let me try turning off threads on lb-test to see if anything breaks, ok?
2025-08-06 21807, 2025
lucifer[m]
Okay sure.
2025-08-06 21859, 2025
lucifer[m]
lucifer[m]: Basically the second answer's first comment.
2025-08-06 21824, 2025
mayhem[m]
"/usr/local/lib/python3.13/site-packages/sentry_sdk/_compat.py:86: Warning: IMPORTANT: We detected the use of uWSGI in preforking mode without thread support. This might lead to crashing workers. Please run uWSGI with both "--enable-threads" and "--py-call-uwsgi-fork-hooks" for full support." when I turn threads to zero.
2025-08-06 21831, 2025
mayhem[m]
time to read about pre-forking mode.
2025-08-06 21811, 2025
mayhem[m]
oh, I see. its not a pure threading model, but a hybrid threads inside process model.
2025-08-06 21840, 2025
mayhem[m]
in that case, putting a simple Lock around the rate limit should fix the problem nicely.
2025-08-06 21808, 2025
mayhem[m]
well, it has the potential to fix it. but I am not conviced. lets pick this up when you're near a computer.
if the dict munging theory is correct, this should fix that.
2025-08-06 21816, 2025
aerozol[m]
julian45: Another 165 spam comments on the blog, since last time we talked D: Definitely need to get Akismet set up again
2025-08-06 21832, 2025
mayhem[m]
yea, on my task list for very soon.
2025-08-06 21804, 2025
aerozol[m]
Ah thanks mayhem
2025-08-06 21818, 2025
lucifer[m]
mayhem: yes that, ` but a hybrid threads inside process model.`
2025-08-06 21800, 2025
lucifer[m]
a lock will fix it yes by blocking other requests from progressing.
2025-08-06 21858, 2025
mayhem[m]
lucifer[m]: the locks are not held for long. should be fine.
2025-08-06 21855, 2025
mayhem[m]
remember, it works in most cases, so its a race condition. and an occaisonal block for a microsecond or two won't do any harm we can notice.
2025-08-06 21855, 2025
lucifer[m]
<mayhem[m]> "the del that fails, can we..." <- yes but i think we can do without locks, by this.
2025-08-06 21825, 2025
lucifer[m]
instead of del use pop with a None default.
2025-08-06 21829, 2025
mayhem[m]
sure, lets try that.
2025-08-06 21800, 2025
lucifer[m]
one more change by combining the key in dict and dict.get check.
2025-08-06 21837, 2025
lucifer[m]
will open a new pr to try that and otherwise we can use a lock yeah.
2025-08-06 21820, 2025
lucifer[m]
just for completeness sake, one more way to do this is thread local. which is what flask uses for its g and app context.
2025-08-06 21844, 2025
d4rkie has quit
2025-08-06 21815, 2025
d4rkie joined the channel
2025-08-06 21857, 2025
petitminion has quit
2025-08-06 21830, 2025
mayhem[m]
we just received a 1000Eur donation via a bank transfer and I have no way to contact them to thank them. how odd.
2025-08-06 21847, 2025
dvirtz[m] joined the channel
2025-08-06 21848, 2025
dvirtz[m] posted a file: (19KiB) < https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/uRMLJkBpNtHhTbjtMMdnYHYA/DBDefs.pm >
2025-08-06 21848, 2025
dvirtz[m]
I moved to a new machine and now my MB dev setup stopped working. I'm using musicbrainz-docker and can see in the db logs... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/OfGEUuWvvywrcGrevlBKPukf>)
dvirtz: that's odd, sounds like it's connecting to the wrong DB or as a different user (affecting the `search_path`). the connection info in DBDefs.pm comes from `%ENV` so you'd have to check the `env` variables inside the musicbrainz container
2025-08-06 21829, 2025
dvirtz[m] sent a code block: https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/KplttsORCcrsAZYnvCHvIDNX
2025-08-06 21840, 2025
dvirtz[m]
* ```... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/KTfjmpcNwhIPumhdoJPhtjnb>)
2025-08-06 21845, 2025
bitmap[m]
dvirtz: looks fine to me, just to clarify, are you are trying to access the website and it's giving an Internal Server Error?
2025-08-06 21811, 2025
bitmap[m]
does docker compose exec musicbrainz bash -c 'carton exec -- admin/psql READWRITE' work for you? if yes, then it's not a configuration issue
2025-08-06 21854, 2025
dvirtz[m]
bitmap
2025-08-06 21854, 2025
dvirtz[m]
That command results in `bash: line 1: carton: command not found`.
2025-08-06 21854, 2025
dvirtz[m]
Most of the pages work but entity edit/create give `Couldn't render template "undef error` and a long html following
2025-08-06 21802, 2025
monkey[m] has quit
2025-08-06 21807, 2025
bwebbo[m] joined the channel
2025-08-06 21807, 2025
bwebbo[m]
Hi all - I have downloaded a PostgreSQL Data Dump and using pg admin to query. When trying to identify genre or tag info, I'm finding a lot of empty tables / tables with very few rows. Please would someone be able to advise the best way to retrieve these attributes or if it sounds like there may have been an import error etc?
