#metabrainz

/

      • lucifer
        read as unless the community is fine with being exploited to generate unsustainable returns for investors
      • 2023-10-23 29616, 2023

      • aerozol
        bitmap: ping me for a downtime post next time, I’ll put it on our socials 👍
      • 2023-10-23 29614, 2023

      • lusciouslover has quit
      • 2023-10-23 29615, 2023

      • reosarevok
        zas, atj: is CB meant to be 500ing now? https://critiquebrainz.org/
      • 2023-10-23 29638, 2023

      • reosarevok
        Apparently it was 502 earlier
      • 2023-10-23 29601, 2023

      • atj
        reosarevok: it's trying to connect to floyd for some reason
      • 2023-10-23 29631, 2023

      • atj
        even though the config contains no references to floyd
      • 2023-10-23 29606, 2023

      • human joined the channel
      • 2023-10-23 29608, 2023

      • human has quit
      • 2023-10-23 29651, 2023

      • atj
        reosarevok: i think this is related to the postgres changes bitmap made yesterday. there is no pgbouncer container running on floyd, but critiquebranz is trying to connect to one
      • 2023-10-23 29631, 2023

      • reosarevok
        Can we bounce them elsewhere? :P
      • 2023-10-23 29617, 2023

      • human joined the channel
      • 2023-10-23 29642, 2023

      • aaditya joined the channel
      • 2023-10-23 29601, 2023

      • aaditya has quit
      • 2023-10-23 29646, 2023

      • human has quit
      • 2023-10-23 29626, 2023

      • human joined the channel
      • 2023-10-23 29602, 2023

      • mayhem
        safe travels home akshaaatt -- you're the last person to return from the summit. next time you're in barcelona we'll unravel the dark mysteries of how shower curtains work. another exciting trip, for sure!
      • 2023-10-23 29619, 2023

      • mayhem finishes mopping up his bathroom again
      • 2023-10-23 29637, 2023

      • akshaaatt
        Hahahahah XD
      • 2023-10-23 29644, 2023

      • akshaaatt
        Thanks btw!!
      • 2023-10-23 29648, 2023

      • mayhem
        np
      • 2023-10-23 29629, 2023

      • zas
        :D
      • 2023-10-23 29628, 2023

      • zas
      • 2023-10-23 29658, 2023

      • mayhem
        yeah, we should depricate that piece, and just tell people to use LB API, but I'll got restart it.
      • 2023-10-23 29631, 2023

      • atj
        reosarevok: I could probably hard code some IPs, but I don't really want to mess with stuff I don't fully understand, especially when it's database related
      • 2023-10-23 29622, 2023

      • reosarevok
        Seems ok to wait for bitmap
      • 2023-10-23 29642, 2023

      • bitmap
        mayhem fixed it already but basically you just need to restart the container if that happens
      • 2023-10-23 29615, 2023

      • mayhem
        critiquebrainz-web-prod is what I restarted
      • 2023-10-23 29602, 2023

      • bitmap
        if we have any other projcets using uwsgi it'd be nice to compare their setup and find out why cb's is borked
      • 2023-10-23 29652, 2023

      • atj
        bitmap: ffs :P
      • 2023-10-23 29610, 2023

      • atj
        i restarted uwsgi and it was still broken, so i thought a container restart was pointless
      • 2023-10-23 29643, 2023

      • bitmap
        yeah! I tried restarting the runit service at first too but it wouldn't take
      • 2023-10-23 29615, 2023

      • atj
        I guess I should trust in the old three-finger salute method more
      • 2023-10-23 29651, 2023

      • bitmap
        aerozol: thanks for the posts, was gonna ping you today but should've done it sooner :)
      • 2023-10-23 29654, 2023

      • bitmap
        yvanzo: zas: hi, so in order to move sir-prod off of floyd and onto hendrix, I understand we have to whitelist hendrix's IP somewhere. could you help with that?
      • 2023-10-23 29648, 2023

      • human has quit
      • 2023-10-23 29604, 2023

      • human joined the channel
      • 2023-10-23 29600, 2023

      • human has quit
      • 2023-10-23 29638, 2023

      • petitminion joined the channel
      • 2023-10-23 29643, 2023

      • zas
        bitmap: yes, I'll have a look right now, I guess it's https://github.com/metabrainz/mb-solr-cloud/blob/…
      • 2023-10-23 29617, 2023

      • zas
        bitmap: done, check if it works
      • 2023-10-23 29655, 2023

      • bitmap
        zas: yes, it seems to start without errors now...thank you!
      • 2023-10-23 29650, 2023

      • petitminion has quit
      • 2023-10-23 29654, 2023

      • bitmap
        processing messages too
      • 2023-10-23 29613, 2023

      • bitmap
        here's the switchover doc I'm working with - https://docs.google.com/document/d/e/2PACX-1vSq8I…
      • 2023-10-23 29649, 2023

      • human joined the channel
      • 2023-10-23 29636, 2023

      • human has quit
      • 2023-10-23 29639, 2023

      • bitmap
        guess I'll have to rewrite a good chunk of syswiki/DatabaseServer.md today 😑
      • 2023-10-23 29600, 2023

      • mayhem
        this looks interesting: https://protomaps.com/
      • 2023-10-23 29641, 2023

      • bitmap
        looks like a good alternative to mapbox which MB still depends on
      • 2023-10-23 29622, 2023

      • mayhem
        I was wondering about that.
      • 2023-10-23 29632, 2023

      • bitmap
        lucifer: mayhem: monkey: if there are any LB/BB cron jobs that access the main DB you'll want to stop them soon
      • 2023-10-23 29630, 2023

      • mayhem
        lucifer: you around? is your brain? my brain is still super slow from the cold.
      • 2023-10-23 29619, 2023

      • lucifer
        Hi! Yes, I can do that in ~5 mins.
      • 2023-10-23 29615, 2023

      • bitmap
        if they start at :00 and definitely take less than an hour you can let the next one finish, but :)
      • 2023-10-23 29656, 2023

      • monkey checks
      • 2023-10-23 29617, 2023

      • monkey
        Should be good for BB
      • 2023-10-23 29633, 2023

      • Pratha-Fish
        reosarevok / bitmap Is this all area related info that we have on musicbrainz? https://usercontent.irccloud-cdn.com/file/CBUzUKU…
      • 2023-10-23 29659, 2023

      • Pratha-Fish
        Also, Guam should be a territory right? IDK why it's area type is marked as 1 (country) in MusicBrainz
      • 2023-10-23 29640, 2023

      • lucifer
        bitmap, mayhem: stopped both cron containers.
      • 2023-10-23 29659, 2023

      • mayhem
        thx
      • 2023-10-23 29649, 2023

      • bitmap
        ++
      • 2023-10-23 29605, 2023

      • Pratha-Fish
        also, looks like Turkey isn't even classified as a country in MusicBrainz for some reason https://usercontent.irccloud-cdn.com/file/kzQf1LJ…
      • 2023-10-23 29640, 2023

      • bitmap
        Pratha-Fish: neither of those appear to be Türkiye :)
      • 2023-10-23 29632, 2023

      • bitmap
        I'm not sure about your first question, is there info you see on the website that is missing there?
      • 2023-10-23 29606, 2023

      • Pratha-Fish
        Not really. But I am not sure if I am missing anything in particular. Just verifying :)
      • 2023-10-23 29631, 2023

      • Pratha-Fish
        Just struggling with getting the area data in right shape out of the musicbrainz database
      • 2023-10-23 29655, 2023

      • Pratha-Fish
        ig I'll rewrite it a bit
      • 2023-10-23 29641, 2023

      • bitmap
        whoops, just fixed some ssh issues in the pg containers that would've caused the switchover to fail
      • 2023-10-23 29613, 2023

      • bitmap simplified some steps in the google doc
      • 2023-10-23 29613, 2023

      • bitmap
        I'll bring services down on the gateway in a few minutes
      • 2023-10-23 29613, 2023

      • bitmap
        proceeding with the switchover
      • 2023-10-23 29624, 2023

      • zas
        bitmap: tell me if you need any help
      • 2023-10-23 29630, 2023

      • bitmap
        looks good, I think we can bring services back now
      • 2023-10-23 29610, 2023

      • rozlav has quit
      • 2023-10-23 29629, 2023

      • bitmap
        we're back
      • 2023-10-23 29638, 2023

      • rozlav joined the channel
      • 2023-10-23 29601, 2023

      • zas
        that was fast
      • 2023-10-23 29615, 2023

      • bitmap
        yeah!
      • 2023-10-23 29641, 2023

      • monkey 👏
      • 2023-10-23 29647, 2023

      • reosarevok
        Pratha-Fish: our country list is basically "stuff ISO marks with an ISO 1 code"
      • 2023-10-23 29600, 2023

      • reosarevok
        Easier than making too many political decisions
      • 2023-10-23 29604, 2023

      • bitmap
        zas: welp, floyd can be retired from service now 🫡
      • 2023-10-23 29602, 2023

      • zas
        guys, if you have anything left on floyd/pink you want to back up, please do asap
      • 2023-10-23 29655, 2023

      • zas
        I just noticed hendrix isn't listed as pg server in ansible hosts config, I'll add it right now. telegraf doesn't report pg stats for it
      • 2023-10-23 29658, 2023

      • bitmap
        oh, right
      • 2023-10-23 29606, 2023

      • bitmap restarts CB containers
      • 2023-10-23 29600, 2023

      • zas
        what's the issue with CB containers?
      • 2023-10-23 29648, 2023

      • bitmap
        its consul-template config is broken or something, it can't restart uwsgi properly
      • 2023-10-23 29607, 2023

      • zas
        atj, bitmap: https://github.com/metabrainz/metabrainz-ansible/… does this still make sense?
      • 2023-10-23 29643, 2023

      • bitmap
        well, for pgbouncer, the ports changed for one, and there are now 6 pgbouncers (3 on jimmy, 3 on hendrix)
      • 2023-10-23 29620, 2023

      • zas
        also we have few obsolete container's names in https://github.com/metabrainz/metabrainz-ansible/… (-floyd / -pink)
      • 2023-10-23 29643, 2023

      • reosarevok
        bitmap: have you checked https://github.com/metabrainz/musicbrainz-server/… at all btw?
      • 2023-10-23 29646, 2023

      • bitmap
        one is for master/primary connections, one for slave/standby connections, and another for "any" (unused, but useful if we want to distribute read only queries to either the master or standby in the future)
      • 2023-10-23 29651, 2023

      • atj
        would be good to work out what this uwsgi issue is
      • 2023-10-23 29654, 2023

      • zas
        bitmap: then we should adapt so we can get stats for all pgbouncer instances
      • 2023-10-23 29611, 2023

      • reosarevok
        It says "One limitation of the approach used here is that it doesn't detect relationship changes." - maybe you have any suggestions for that :)
      • 2023-10-23 29626, 2023

      • bitmap
        reosarevok: not yet, but I can check it later today
      • 2023-10-23 29630, 2023

      • reosarevok
        Perfect
      • 2023-10-23 29641, 2023

      • reosarevok
        Would be good to put a beta out tonight or tomorrow
      • 2023-10-23 29606, 2023

      • reosarevok
        But I see you're quite busy today :D
      • 2023-10-23 29650, 2023

      • bitmap
        zas: the relevant containers now are postgres-jimmy, postgres-hendrix, pgbouncer-master, pgbouncer-slave, pgbouncer-any, and haproxy-postgres
      • 2023-10-23 29621, 2023

      • zas
        bitmap: ok, thanks, I'll add them
      • 2023-10-23 29658, 2023

      • bitmap
        if there is an easy way to add haproxy stats to telegraf, it might be useful too
      • 2023-10-23 29611, 2023

      • bitmap
        pgbouncer connects to haproxy, haproxy to pg
      • 2023-10-23 29643, 2023

      • zas
        bitmap: yes, I was thinking about it, we may need to make few changes to haproxy configs so telegraf can gather stats from them, I'll have a look at this this week
      • 2023-10-23 29652, 2023

      • zas
        bitmap: we need to have /stats endpoint accessible on haproxy containers on hendrix/jimmy, it runs on port 8182 according to config, but this port isn't exposed by containers it seems
      • 2023-10-23 29613, 2023

      • BrainzGit
        [bookbrainz-site] 14MonkeyDo merged pull request #1027 (03master…bb-data-ts-migration): TypeScript migration of the ORM https://github.com/metabrainz/bookbrainz-site/pul…
      • 2023-10-23 29658, 2023

      • bitmap
        zas: ok, I'll look into it
      • 2023-10-23 29622, 2023

      • bitmap
        should be able to restart these without downtime, let's see
      • 2023-10-23 29644, 2023

      • reosarevok
        bitmap: is MBS-13325 intentional do you remember? (because of grouping or whatnot)
      • 2023-10-23 29645, 2023

      • BrainzBot
        MBS-13325: Open performer removal edits are not shown in credits at bottom https://tickets.metabrainz.org/browse/MBS-13325
      • 2023-10-23 29658, 2023

      • bitmap
        zas: alright, it should be exposed on port 65439 now
      • 2023-10-23 29600, 2023

      • bitmap
        here's how I restarted the containers, will have to simplify this and incorporate it into syswiki https://gist.github.com/mwiencek/c3ae84396425d10a…
      • 2023-10-23 29610, 2023

      • reosarevok
        <BANG>
      • 2023-10-23 29639, 2023

      • reosarevok
        Monday rolling back again, like a tackling Wooloo
      • 2023-10-23 29618, 2023

      • reosarevok
        Today I have four mailed in reviews (well, two were IRCd in, but we'll count it):
      • 2023-10-23 29633, 2023

      • reosarevok
        ansh said:
      • 2023-10-23 29634, 2023

      • reosarevok
        '''
      • 2023-10-23 29637, 2023

      • reosarevok
        Hi everyone!
      • 2023-10-23 29641, 2023

      • reosarevok
        Last week, I worked on adding the new design changes aerozol made. I opened the PR for Fresh Release page LB#2603.
      • 2023-10-23 29642, 2023

      • BrainzBot
        LB-1172: Fresh releases improvements: https://github.com/metabrainz/listenbrainz-server…
      • 2023-10-23 29644, 2023

      • reosarevok
        I had a great meeting with monkey where we discussed some of the blockers for this PR.
      • 2023-10-23 29648, 2023

      • reosarevok
        Other than that I spent time working on getting spark to work with the new filters, and fixing some frontend tests.
      • 2023-10-23 29649, 2023

      • reosarevok
        Thats it for me
      • 2023-10-23 29650, 2023

      • reosarevok
        '''
      • 2023-10-23 29638, 2023

      • reosarevok
        yvanzo said:
      • 2023-10-23 29639, 2023

      • reosarevok
        '''
      • 2023-10-23 29647, 2023

      • reosarevok
        Last week I worked mostly on actionable tasks following the summit: writing notes, prepping weblate, fixing small issues, triaging tickets, and a bit of support…
      • 2023-10-23 29650, 2023

      • reosarevok
        '''
      • 2023-10-23 29617, 2023

      • reosarevok
        aerozol said:
      • 2023-10-23 29618, 2023

      • reosarevok
        '''
      • 2023-10-23 29623, 2023

      • reosarevok
        It’s a holiday/long weekend here, and Lucy-Belle’s birthday, so I almost forgot it was Monday! I didn’t though. Lucky lucky you.
      • 2023-10-23 29635, 2023

      • mayhem
        stitches? what did I miss?
      • 2023-10-23 29606, 2023

      • mayhem
        sounds like an unusal merge conflict
      • 2023-10-23 29610, 2023

      • reosarevok
        aerozol said:
      • 2023-10-23 29611, 2023

      • reosarevok
        '''
      • 2023-10-23 29613, 2023

      • reosarevok
        It’s a holiday/long weekend here, and Lucy-Belle’s birthday, so I almost forgot it was Monday! I didn’t though. Lucky lucky you.
      • 2023-10-23 29616, 2023

      • mayhem
        oy, not too bad I hope!
      • 2023-10-23 29617, 2023

      • reosarevok
        Last week I mainly worked on other client work, but I did a bit more work on the summit 23 blog post. I will post it Wednesday.
      • 2023-10-23 29628, 2023

      • reosarevok
        Anyone who wants to proof it, do it now: https://wordpress.com/post/blog.metabrainz.org/10…
      • 2023-10-23 29634, 2023

      • reosarevok
        yvanzo and reosarevok there are no notes for your topics of: Service alerts, Spam, Security audits
      • 2023-10-23 29637, 2023

      • reosarevok
        Add them to the word doc before Wednesday if they’re important. If they’re not important it’s fine to leave them out. The doc: https://docs.google.com/document/d/1bz32duq95jSfi…
      • 2023-10-23 29642, 2023

      • reosarevok
        Other than that I made some changes to the Fresh Releases redesign mockup, that Ansh has made great progress on, and noodled about with some design considerations with monkey
      • 2023-10-23 29645, 2023

      • reosarevok
        I also added YouTube videos/walkthroughs to a bunch of wiki pages
      • 2023-10-23 29646, 2023

      • reosarevok
        The person who is second (after me) most likely to forget it’s Monday can go next
      • 2023-10-23 29648, 2023

      • reosarevok
        '''
      • 2023-10-23 29603, 2023

      • monkey
        Mon-what?
      • 2023-10-23 29604, 2023

      • reosarevok
        I'm not sure how likely FishFish is to forget it's Monday but here's his review:
      • 2023-10-23 29606, 2023

      • reosarevok
        '''