#metabrainz

/

      • d4rkie has quit
      • 2025-08-06 21822, 2025

      • d4rkie joined the channel
      • 2025-08-06 21859, 2025

      • wargreen has quit
      • 2025-08-06 21851, 2025

      • wargreen joined the channel
      • 2025-08-06 21823, 2025

      • zas[m]
        bitmap: We'll have to replace a SSD on yamaoka, can you check if we can move services elsewhere (or afford to stop them a short time for the replacement)? I'll handle that tomorrow (that's late here, going to bed)
      • 2025-08-06 21818, 2025

      • bitmap[m]
        zas: I've temporarily started artwork-indexer prod and CB prod containers on selda. other non-redundant containers on yamaoka include CB beta, mbspotify, and jenkins, but downtime shouldn't be problematic for those (not 100% sure about mbspotify though)
      • 2025-08-06 21822, 2025

      • bitmap[m]
        apparently CB was already running on selda but wasn't added to scripts/nodes/selda.sh
      • 2025-08-06 21810, 2025

      • DjSlash has quit
      • 2025-08-06 21833, 2025

      • DjSlash joined the channel
      • 2025-08-06 21856, 2025

      • d4rk-ph0enix has quit
      • 2025-08-06 21818, 2025

      • d4rk-ph0enix joined the channel
      • 2025-08-06 21842, 2025

      • d4rk-ph0enix has quit
      • 2025-08-06 21828, 2025

      • d4rk-ph0enix joined the channel
      • 2025-08-06 21856, 2025

      • d4rk-ph0enix has quit
      • 2025-08-06 21823, 2025

      • d4rk-ph0enix joined the channel
      • 2025-08-06 21842, 2025

      • d4rk-ph0enix has quit
      • 2025-08-06 21800, 2025

      • d4rk-ph0enix joined the channel
      • 2025-08-06 21804, 2025

      • lucifer[m]
        bitmap: i was looking at all yesterday to figure out which ones were running CB and i didn't have selda in my local script so skipped it. thanks!
      • 2025-08-06 21835, 2025

      • lucifer[m]
        reosarevok: CB issues should now be fixed, yamaoka and selda were both running CB but on different versions and that caused all the reported issues.
      • 2025-08-06 21812, 2025

      • lucifer[m]
        i have stopped all CB containers on yamaoka.
      • 2025-08-06 21807, 2025

      • yvanzo[m]
        <aerozol[m]> "yvanzo: I think you might be..." <- aerozol: I clearly wrote that “it is open to suggestions and the script’s code is organized in a way that makes it easy to play with colors”. There is no need to split people into made-up opposite camps, thank you. Please make actual suggestions for the color scheme instead: Pick one image such the cover art archive table structure, change the colors, and we will see.
      • 2025-08-06 21833, 2025

      • aerozol[m]
        I wasn't splitting anyone up... I just read a different part of a very long comment that you linked to.
      • 2025-08-06 21821, 2025

      • aerozol[m]
        My advice is to remove the background color and leave it plain. I will take the time to change it if it isn't going to be changed back
      • 2025-08-06 21835, 2025

      • __BrainzGit
        [troi-recommendation-playground] 14amCap1712 merged pull request #174 (03main…fix-import): misc small fixes https://github.com/metabrainz/troi-recommendation…
      • 2025-08-06 21808, 2025

      • __BrainzGit
        [troi-recommendation-playground] release 03v2025.08.06.1 has been published by 14amCap1712: https://github.com/metabrainz/troi-recommendation…
      • 2025-08-06 21823, 2025

      • yvanzo[m]
        For everyone else, I continued this conversation in private with aerozol to go beyond unjustified assumptions before having to go. To be continued soon…
      • 2025-08-06 21825, 2025

      • __BrainzGit
        [listenbrainz-server] 14amCap1712 opened pull request #3335 (03master…update-troi): Update troi to 2025.8.6.1 https://github.com/metabrainz/listenbrainz-server…
      • 2025-08-06 21806, 2025

      • d4rkie has quit
      • 2025-08-06 21833, 2025

      • d4rkie joined the channel
      • 2025-08-06 21859, 2025

      • SigHunter has quit
      • 2025-08-06 21855, 2025

      • SigHunter joined the channel
      • 2025-08-06 21838, 2025

      • SigHunter has quit
      • 2025-08-06 21825, 2025

      • SigHunter joined the channel
      • 2025-08-06 21823, 2025

      • __BrainzGit
        [listenbrainz-server] 14amCap1712 merged pull request #3335 (03master…update-troi): Update troi to 2025.8.6.1 https://github.com/metabrainz/listenbrainz-server…
      • 2025-08-06 21811, 2025

      • __BrainzGit
        [listenbrainz-server] release 03v-2025-08-06.0 has been published by 14github-actions[bot]: https://github.com/metabrainz/listenbrainz-server…
      • 2025-08-06 21838, 2025

      • fettuccinae[m] has quit
      • 2025-08-06 21804, 2025

      • zas[m]
        yamaoka disk replacement should occur within 2-4 hours
      • 2025-08-06 21849, 2025

      • petitminion joined the channel
      • 2025-08-06 21827, 2025

      • zas[m]
        yamaoka disk was replaced, RAID array is rebuilding (ETA ~146 minutes), it is back to work
      • 2025-08-06 21807, 2025

      • lucifer[m]
        mayhem: troi ratelimiting is broken because multiple concurrent requests to explore/lb-radio invoke troi in separate threads and the global dict to manage ratelimits is concurrently modified.
      • 2025-08-06 21804, 2025

      • lucifer[m]
        the ratelimiting broken is not the issue this time but this: https://github.com/metabrainz/troi-recommendation…
      • 2025-08-06 21837, 2025

      • lucifer[m]
        the del operation fails because key is already removed from the dict.
      • 2025-08-06 21845, 2025

      • lucifer[m]
        by another thread/request.
      • 2025-08-06 21833, 2025

      • lucifer[m]
        there are two ways to fix this imo, either add a lock and only make one global request for troi at a time. but that will hurt lb-radio performance significantly.
      • 2025-08-06 21800, 2025

      • lucifer[m]
        make the dict storing the ratelimits a thead local. each thread/request gets its own dict.
      • 2025-08-06 21847, 2025

      • lucifer[m]
        there is a third option to make the dict accesses safer, use get/pop with defaults but might defeat the purpose of ratelimit logic.
      • 2025-08-06 21819, 2025

      • lucifer[m]
        hmm, no actually that might work fine too. we just need to decide whether we want global ratelimits on a best effort basis or per-thread ratelimits.
      • 2025-08-06 21812, 2025

      • mayhem[m] reads
      • 2025-08-06 21815, 2025

      • mayhem[m]
        the del that fails, can we just ignore the failure. (likely not that simple)
      • 2025-08-06 21803, 2025

      • mayhem[m]
        <lucifer[m]> "mayhem: troi ratelimiting is..." <- let me clarify -- this is separate *processes* not requests, right?
      • 2025-08-06 21818, 2025

      • mayhem[m]
        threads
      • 2025-08-06 21826, 2025

      • lucifer[m]
        I'll have to check uwsgi configuration to see whether it's doing threads or processes, likely a combination of both
      • 2025-08-06 21830, 2025

      • mayhem[m]
        not processes. still waking up.
      • 2025-08-06 21849, 2025

      • mayhem[m]
        troi doesn't use threads, let me check.
      • 2025-08-06 21858, 2025

      • lucifer[m]
        No troi doesn't but uwsgi might be
      • 2025-08-06 21807, 2025

      • mayhem[m]
        it doesn't.
      • 2025-08-06 21812, 2025

      • lucifer[m]
        The issues was seen in lb prod api logs.
      • 2025-08-06 21821, 2025

      • mayhem[m]
        its specifically multi-process, but the processes themselves may use threads.
      • 2025-08-06 21833, 2025

      • mayhem[m]
        (on uwsgi)
      • 2025-08-06 21813, 2025

      • lucifer[m]
      • 2025-08-06 21816, 2025

      • mayhem[m]
        so, this double manipulation of the dict is rather curious.
      • 2025-08-06 21830, 2025

      • lucifer[m]
        We have enabled thread use from what I understand
      • 2025-08-06 21839, 2025

      • lucifer[m]
        So multiple threads per uwsgi process
      • 2025-08-06 21849, 2025

      • mayhem[m]
        lucifer[m]: yes, this means threads for child processes are enabled. I believe we can take that option away, since we're not usingit.
      • 2025-08-06 21815, 2025

      • lucifer[m]
        I think we are.
      • 2025-08-06 21825, 2025

      • lucifer[m]
        I'll check uwsgi docs again later to confirm
      • 2025-08-06 21803, 2025

      • mayhem[m]
        from gemini: "Enabling threads in uWSGI doesn't change it into a "threading server" in the same way a single-process, multi-threaded application works. Instead, it enables threading within each of its worker processes. The primary concurrency model of uWSGI remains multiprocess."
      • 2025-08-06 21816, 2025

      • lucifer[m]
      • 2025-08-06 21818, 2025

      • mayhem[m]
        as I expected.
      • 2025-08-06 21837, 2025

      • mayhem[m]
        the stack overflow anwswer is shit.
      • 2025-08-06 21823, 2025

      • lucifer[m]
      • 2025-08-06 21832, 2025

      • lucifer[m]
        There are multiple ones saying the same though
      • 2025-08-06 21840, 2025

      • lucifer[m]
        lucifer[m]: My understanding has always been this
      • 2025-08-06 21856, 2025

      • lucifer[m]
        But i'll check docs when at PC later to confirm
      • 2025-08-06 21803, 2025

      • mayhem[m]
        see the second answer.
      • 2025-08-06 21842, 2025

      • mayhem[m]
        I understood it just like you do, but that is not how it works. faster-fuzzy exists because of this. I know the pain.
      • 2025-08-06 21813, 2025

      • lucifer[m]
        I understand that but with sleep the control can be returned to the process to start running another thread.
      • 2025-08-06 21844, 2025

      • mayhem[m]
        lucifer[m]: no. uwsgi itself uses no threads.
      • 2025-08-06 21850, 2025

      • lucifer[m]
        asyncio does that and if there is monkey patching involved using greenlet or gevent etc it can happen here too.
      • 2025-08-06 21802, 2025

      • mayhem[m]
        let me try turning off threads on lb-test to see if anything breaks, ok?
      • 2025-08-06 21807, 2025

      • lucifer[m]
        Okay sure.
      • 2025-08-06 21859, 2025

      • lucifer[m]
        lucifer[m]: Basically the second answer's first comment.
      • 2025-08-06 21824, 2025

      • mayhem[m]
        "/usr/local/lib/python3.13/site-packages/sentry_sdk/_compat.py:86: Warning: IMPORTANT: We detected the use of uWSGI in preforking mode without thread support. This might lead to crashing workers. Please run uWSGI with both "--enable-threads" and "--py-call-uwsgi-fork-hooks" for full support." when I turn threads to zero.
      • 2025-08-06 21831, 2025

      • mayhem[m]
        time to read about pre-forking mode.
      • 2025-08-06 21811, 2025

      • mayhem[m]
        oh, I see. its not a pure threading model, but a hybrid threads inside process model.
      • 2025-08-06 21840, 2025

      • mayhem[m]
        in that case, putting a simple Lock around the rate limit should fix the problem nicely.
      • 2025-08-06 21808, 2025

      • mayhem[m]
        well, it has the potential to fix it. but I am not conviced. lets pick this up when you're near a computer.
      • 2025-08-06 21855, 2025

      • __BrainzGit
        [troi-recommendation-playground] 14mayhem opened pull request #175 (03main…lock-ratelimit-dict): Add a lock to the rate-limit dict https://github.com/metabrainz/troi-recommendation…
      • 2025-08-06 21819, 2025

      • mayhem[m]
        if the dict munging theory is correct, this should fix that.
      • 2025-08-06 21816, 2025

      • aerozol[m]
        julian45: Another 165 spam comments on the blog, since last time we talked D: Definitely need to get Akismet set up again
      • 2025-08-06 21832, 2025

      • mayhem[m]
        yea, on my task list for very soon.
      • 2025-08-06 21804, 2025

      • aerozol[m]
        Ah thanks mayhem
      • 2025-08-06 21818, 2025

      • lucifer[m]
        mayhem: yes that, ` but a hybrid threads inside process model.`
      • 2025-08-06 21800, 2025

      • lucifer[m]
        a lock will fix it yes by blocking other requests from progressing.
      • 2025-08-06 21858, 2025

      • mayhem[m]
        lucifer[m]: the locks are not held for long. should be fine.
      • 2025-08-06 21855, 2025

      • mayhem[m]
        remember, it works in most cases, so its a race condition. and an occaisonal block for a microsecond or two won't do any harm we can notice.
      • 2025-08-06 21855, 2025

      • lucifer[m]
        <mayhem[m]> "the del that fails, can we..." <- yes but i think we can do without locks, by this.
      • 2025-08-06 21825, 2025

      • lucifer[m]
        instead of del use pop with a None default.
      • 2025-08-06 21829, 2025

      • mayhem[m]
        sure, lets try that.
      • 2025-08-06 21800, 2025

      • lucifer[m]
        one more change by combining the key in dict and dict.get check.
      • 2025-08-06 21837, 2025

      • lucifer[m]
        will open a new pr to try that and otherwise we can use a lock yeah.
      • 2025-08-06 21820, 2025

      • lucifer[m]
        just for completeness sake, one more way to do this is thread local. which is what flask uses for its g and app context.
      • 2025-08-06 21844, 2025

      • d4rkie has quit
      • 2025-08-06 21815, 2025

      • d4rkie joined the channel
      • 2025-08-06 21857, 2025

      • petitminion has quit
      • 2025-08-06 21830, 2025

      • mayhem[m]
        we just received a 1000Eur donation via a bank transfer and I have no way to contact them to thank them. how odd.
      • 2025-08-06 21847, 2025

      • dvirtz[m] joined the channel
      • 2025-08-06 21848, 2025

      • dvirtz[m] posted a file: (19KiB) < https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/uRMLJkBpNtHhTbjtMMdnYHYA/DBDefs.pm >
      • 2025-08-06 21848, 2025

      • dvirtz[m]
        I moved to a new machine and now my MB dev setup stopped working. I'm using musicbrainz-docker and can see in the db logs... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/OfGEUuWvvywrcGrevlBKPukf>)