#metabrainz

/

      • cuanim joined the channel
      • 2022-03-28 08756, 2022

      • cuanim has quit
      • 2022-03-28 08756, 2022

      • cuanim joined the channel
      • 2022-03-28 08704, 2022

      • Shubh joined the channel
      • 2022-03-28 08737, 2022

      • akshaaatt
        Hi reosarevok, bitmap ! Can we merge https://github.com/metabrainz/musicbrainz-server/… ?
      • 2022-03-28 08750, 2022

      • aerozol
        akshaaatt, check out this design system: https://design-system-alpha.digital.govt.nz/compo…
      • 2022-03-28 08758, 2022

      • akshaaatt
        Thanks for the share aerozol! This looks great.
      • 2022-03-28 08743, 2022

      • aerozol
        #goals
      • 2022-03-28 08737, 2022

      • aerozol
        They aim to meet web accessibility guidelines - doesn't look good for CatQuest's wishes for small font sizes :(
      • 2022-03-28 08747, 2022

      • aerozol
        But let's see
      • 2022-03-28 08745, 2022

      • kuno has quit
      • 2022-03-28 08745, 2022

      • lucifer has quit
      • 2022-03-28 08753, 2022

      • lucifer joined the channel
      • 2022-03-28 08754, 2022

      • kuno joined the channel
      • 2022-03-28 08722, 2022

      • d4rk-ph0enix joined the channel
      • 2022-03-28 08723, 2022

      • akshaaatt
        Haha yuss!
      • 2022-03-28 08725, 2022

      • reosarevok
        akshaaatt: I'll take a quick check for functionality and maybe bitmap can check the bootstrap one last time later today
      • 2022-03-28 08737, 2022

      • reosarevok
        yvanzo: anything for docker this release?
      • 2022-03-28 08708, 2022

      • akshaaatt
        Thanks a lot reosarevok!
      • 2022-03-28 08724, 2022

      • akshaaatt
        Hi monkey! Regarding https://github.com/metabrainz/listenbrainz-server… we might need a discussion. Please let me know when you're reviewing it
      • 2022-03-28 08740, 2022

      • Shubh has quit
      • 2022-03-28 08755, 2022

      • odnes joined the channel
      • 2022-03-28 08735, 2022

      • PrathameshG joined the channel
      • 2022-03-28 08731, 2022

      • cloverfield has quit
      • 2022-03-28 08719, 2022

      • DjSlash has quit
      • 2022-03-28 08703, 2022

      • DjSlash joined the channel
      • 2022-03-28 08747, 2022

      • lnl has quit
      • 2022-03-28 08747, 2022

      • kepstin has quit
      • 2022-03-28 08748, 2022

      • yellowhatpro has quit
      • 2022-03-28 08748, 2022

      • Ansh has quit
      • 2022-03-28 08749, 2022

      • elomatreb[m] has quit
      • 2022-03-28 08749, 2022

      • legoktm[m] has quit
      • 2022-03-28 08750, 2022

      • yyoung[m] has quit
      • 2022-03-28 08750, 2022

      • riksucks has quit
      • 2022-03-28 08754, 2022

      • intrnl_[m] has quit
      • 2022-03-28 08754, 2022

      • Lorenzo[m] has quit
      • 2022-03-28 08758, 2022

      • reg[m] has quit
      • 2022-03-28 08759, 2022

      • goldenshimmer has quit
      • 2022-03-28 08759, 2022

      • tandy1000 has quit
      • 2022-03-28 08745, 2022

      • elomatreb[m] joined the channel
      • 2022-03-28 08708, 2022

      • leonardo joined the channel
      • 2022-03-28 08751, 2022

      • kepstin joined the channel
      • 2022-03-28 08751, 2022

      • reg[m] joined the channel
      • 2022-03-28 08751, 2022

      • legoktm[m] joined the channel
      • 2022-03-28 08751, 2022

      • tandy1000 joined the channel
      • 2022-03-28 08751, 2022

      • riksucks joined the channel
      • 2022-03-28 08751, 2022

      • lnl joined the channel
      • 2022-03-28 08751, 2022

      • Ansh joined the channel
      • 2022-03-28 08751, 2022

      • intrnl_[m] joined the channel
      • 2022-03-28 08751, 2022

      • Lorenzo[m] joined the channel
      • 2022-03-28 08751, 2022

      • yyoung[m] joined the channel
      • 2022-03-28 08755, 2022

      • PrathameshG has quit
      • 2022-03-28 08703, 2022

      • goldenshimmer joined the channel
      • 2022-03-28 08705, 2022

      • yellowhatpro joined the channel
      • 2022-03-28 08749, 2022

      • CatQuest
        aerozol: it's not so much the giant fonts, but the giantfonts in combination with lots of whitespace
      • 2022-03-28 08704, 2022

      • CatQuest
        (had to zoom out like 3 times on that site btw)
      • 2022-03-28 08747, 2022

      • aerozol
        I hear you! But that's what the nz govt is recommending if you want your site to be accessible (e.g. readable/parsable by disabled people)
      • 2022-03-28 08702, 2022

      • aerozol
        note: I'm not saying MB should look like that, it's just interesting to note
      • 2022-03-28 08730, 2022

      • aerozol
        MB is much more content heavy so we wouldn't try look similar I don't think
      • 2022-03-28 08735, 2022

      • trolley has quit
      • 2022-03-28 08727, 2022

      • trolley joined the channel
      • 2022-03-28 08726, 2022

      • CatQuest
        oh don't get my wrong, I definitely thing we need to make things usable for disabled people
      • 2022-03-28 08733, 2022

      • CatQuest
        especially screen readers etc
      • 2022-03-28 08736, 2022

      • mayhem
        moooooin!
      • 2022-03-28 08737, 2022

      • nbin has quit
      • 2022-03-28 08707, 2022

      • nbin joined the channel
      • 2022-03-28 08739, 2022

      • aerozol
        moin moin!
      • 2022-03-28 08710, 2022

      • Shubh joined the channel
      • 2022-03-28 08739, 2022

      • reosarevok
        zas, atj: has something changed with how we keep connections open recently?
      • 2022-03-28 08744, 2022

      • reosarevok
        I just got reosarevok@hip:~$ Timeout, server hip.metabrainz.org not responding.
      • 2022-03-28 08748, 2022

      • reosarevok
        Earlier got the same from floyd
      • 2022-03-28 08756, 2022

      • reosarevok
        I didn't use to get this at all unless my net went down
      • 2022-03-28 08724, 2022

      • reosarevok
        (just wondering if it's somehow connected to why sir loses connection to rabbitmq)
      • 2022-03-28 08701, 2022

      • zas
        I don't think, is the problem always with hip?
      • 2022-03-28 08727, 2022

      • reosarevok
        No, earlier was floyd, as I said
      • 2022-03-28 08746, 2022

      • reosarevok
        I dunno, it might just be I haven't paid enough attention earlier :)
      • 2022-03-28 08707, 2022

      • reosarevok
        And it just caught my eye because of that pink <-> clash thing
      • 2022-03-28 08702, 2022

      • zas
        That's weird, I'll proceed to few network checks
      • 2022-03-28 08719, 2022

      • alastairp
        morning
      • 2022-03-28 08706, 2022

      • reosarevok
        yvanzo: I see there's something new about ts in docker, should this be part of the release docs?
      • 2022-03-28 08717, 2022

      • reosarevok
        I'll push the update commit for now
      • 2022-03-28 08706, 2022

      • reosarevok
        I also saved a basic draft
      • 2022-03-28 08731, 2022

      • zas
        reosarevok: this timeout is on your ssh connection?
      • 2022-03-28 08744, 2022

      • reosarevok
        Yes
      • 2022-03-28 08709, 2022

      • reosarevok
        So maybe it's just something my side or something, just wanted to make sure nothing weird happened :)
      • 2022-03-28 08753, 2022

      • zas
        reosarevok: was it an active session? I mean did the timeout appeared after a while without activity?
      • 2022-03-28 08702, 2022

      • reosarevok
        Yes
      • 2022-03-28 08711, 2022

      • reosarevok
        A while without activity, I mean
      • 2022-03-28 08721, 2022

      • yvanzo
        reosarevok: just a patch for MBS development, I will edit the release notes
      • 2022-03-28 08728, 2022

      • reosarevok
        Thanks!
      • 2022-03-28 08729, 2022

      • zas
        reosarevok: I see nothing abnormal in logs, do you have ServerAliveInterval set in your ssh config?
      • 2022-03-28 08725, 2022

      • reosarevok
        ... I do, 60. I assume that's it then? Maybe the generate ssh config script only added this recently or maybe I only noticed it recently but eh, sorry
      • 2022-03-28 08732, 2022

      • reosarevok
        Well, still a mystery re: sir and rmq then
      • 2022-03-28 08758, 2022

      • reosarevok
        yvanzo: dunno if you saw from telegram but it happened again yesterday
      • 2022-03-28 08754, 2022

      • zas
        reosarevok: set it to 0 (or remove it as it is the default)
      • 2022-03-28 08704, 2022

      • zas
        about rmq / sir that's another problem
      • 2022-03-28 08736, 2022

      • reosarevok
        yvanzo: is there a reason we default to 60 in https://github.com/metabrainz/docker-server-confi… ?
      • 2022-03-28 08704, 2022

      • yvanzo
        reosarevok: added a note to release notes
      • 2022-03-28 08754, 2022

      • yvanzo
        reosarevok: Not really, what is the issue with it?
      • 2022-03-28 08730, 2022

      • reosarevok
        That sir does seem to lose connection with rabbitmq, not notice it somehow, and keep trying to connect
      • 2022-03-28 08702, 2022

      • yvanzo
        How is it linked to this option?
      • 2022-03-28 08708, 2022

      • reosarevok
        Oooh
      • 2022-03-28 08719, 2022

      • reosarevok
        I thought you meant about the sir thing
      • 2022-03-28 08721, 2022

      • reosarevok
        Sorry
      • 2022-03-28 08751, 2022

      • reosarevok
        No problem as such re: 60, I just don't know if there's a benefit on making the user need to reconnect after a while rather than keeping them in
      • 2022-03-28 08753, 2022

      • yvanzo
        I will check sir later on.
      • 2022-03-28 08755, 2022

      • reosarevok
        So mostly just wondering why
      • 2022-03-28 08708, 2022

      • reosarevok
        The sir thing was just the same unexpectedly closed connection bit :)
      • 2022-03-28 08731, 2022

      • reosarevok
        Wondering since zas says default is 0, anyway, so I assumed there was a reason to deviate :)
      • 2022-03-28 08705, 2022

      • reosarevok
        Saved a blog draft, btw :)
      • 2022-03-28 08724, 2022

      • yvanzo
        I guess it is to keep scripts moving on if a server is not responsive.
      • 2022-03-28 08725, 2022

      • Ansh
        alastairp monkey I’ve updated my GSoC application. Please have a look at it. I’d love to get your feedback.
      • 2022-03-28 08747, 2022

      • Ansh
      • 2022-03-28 08717, 2022

      • yvanzo
        This option is supposed to help with low connectivity issues.
      • 2022-03-28 08709, 2022

      • reosarevok
        yvanzo: I guess the tag is still meant to be only 'Upgrade MusicBrainz Server.' then?
      • 2022-03-28 08717, 2022

      • reosarevok
        For docker
      • 2022-03-28 08745, 2022

      • agatzk has quit
      • 2022-03-28 08734, 2022

      • zas
        about sir/rmq issue, I checked logs of rabbitmq, but I don't see anything special when it happened last time
      • 2022-03-28 08703, 2022

      • agatzk joined the channel
      • 2022-03-28 08753, 2022

      • reosarevok
        zas: yeah, atj said he checked and rabbitmq said it was sir not connecting, sir said it was rabbitmq
      • 2022-03-28 08703, 2022

      • zas
        yvanzo: can you give a datetime for the last time it went to bananas, according to https://stats.metabrainz.org/d/T4MODrIiz/solr-clo… that's around 4:30 UTC on 27th. What do you have on sir side exactly?
      • 2022-03-28 08723, 2022

      • atj
        reosarevok: I thought you said that? :)
      • 2022-03-28 08745, 2022

      • atj
        I checked the system logs at the time it happened and didn't find anything relevant
      • 2022-03-28 08720, 2022

      • atj
        my thoughts was that it could be a network resource related issue
      • 2022-03-28 08728, 2022

      • reosarevok
        Huh. Iirc you told me there was something like that. Am I going nuts? Maybe!
      • 2022-03-28 08739, 2022

      • zas
      • 2022-03-28 08702, 2022

      • zas
        shows a bunch of peaks in network & disk I/O around the time it happened
      • 2022-03-28 08743, 2022

      • zas
        it is possible the machine was slowed down enough to lead to timeouts, but nothing certain at this point
      • 2022-03-28 08759, 2022

      • zas
        https://stats.metabrainz.org/d/000000048/hetzner-… looks more a consequence than a cause
      • 2022-03-28 08713, 2022

      • atj
        reosarevok: I think it was lucifer actually - https://chatlogs.metabrainz.org/libera/metabrainz…
      • 2022-03-28 08733, 2022

      • zas
      • 2022-03-28 08743, 2022

      • zas
        is the corresponding activity on pink
      • 2022-03-28 08759, 2022

      • lucifer
        hi! yes, i saw `client closed connection unexpectedly` in the rmq logs at that time. usually that happens (in my experience from similar issues in LB) when some error went unhandled on client side.
      • 2022-03-28 08700, 2022

      • zas
        why is clash having so high traffic on external interface
      • 2022-03-28 08759, 2022

      • lucifer
        zas, maybe micahel? it sends GBs worth of stats daily. but unsure if its external or internal interface.
      • 2022-03-28 08726, 2022

      • zas
        it was data out of clash, backups?
      • 2022-03-28 08748, 2022

      • atj
        do we have a graph that shows exactly when the issue started (from an application perspective), so we can see work out what is cause vs. consequence?
      • 2022-03-28 08748, 2022

      • lucifer
        oh no idea then. michael data should have been incoming.
      • 2022-03-28 08735, 2022

      • zas
        backups on 27th on clash ended at 3UTC, so that's not it
      • 2022-03-28 08703, 2022

      • reosarevok
        atj: ooooh, sorry then! sorry lucifer for mixing you up too :D
      • 2022-03-28 08726, 2022

      • zas
        so what is creating this traffic on both interfaces on clash between 4:25UTC and 5:27UTC?
      • 2022-03-28 08756, 2022

      • zas
        ah sorry that's IN on eth0
      • 2022-03-28 08738, 2022

      • lucifer
        eth0 is external and eth1 internal? or vice versa
      • 2022-03-28 08757, 2022

      • zas
        I think the problem lies here, huge network traffic prolly slowing down I/O, leading to timeouts
      • 2022-03-28 08713, 2022

      • zas
        eth0 is external
      • 2022-03-28 08729, 2022

      • atj
        zas: link me the graph you're looking at please
      • 2022-03-28 08751, 2022

      • zas
        but we have traffic on pink too, starting few mins later, and in the reverse direction
      • 2022-03-28 08752, 2022

      • atj
        because I'm looking at clash and I don't see it :(
      • 2022-03-28 08703, 2022

      • zas
        you don't see what?
      • 2022-03-28 08722, 2022

      • zas