#metabrainz

/

      • CatQuest
        btw re "my feed" and the like. personally i disliek the "my-ing" of terms on pages
      • 2021-10-01 27442, 2021

      • CatQuest
        I even somewhat the "your-ing" of terms
      • 2021-10-01 27452, 2021

      • CatQuest
        it's not "my feed" or "your feed" it's "the feed"
      • 2021-10-01 27407, 2021

      • ruaok
        CatQuest: I agree!
      • 2021-10-01 27432, 2021

      • akshat
        Cool CatQuest I agree as well
      • 2021-10-01 27440, 2021

      • Lotheric
        I would call it "The Q Continuum"
      • 2021-10-01 27458, 2021

      • Lotheric
        but I'm a nerd
      • 2021-10-01 27421, 2021

      • CatQuest
        :P
      • 2021-10-01 27405, 2021

      • Lotheric
        502 bad gateway on MB beta
      • 2021-10-01 27424, 2021

      • Lotheric
        "server being naughty!"
      • 2021-10-01 27440, 2021

      • Lotheric
        ruaok, zas
      • 2021-10-01 27445, 2021

      • ruaok
        beta, hm ok.
      • 2021-10-01 27454, 2021

      • ruaok
        yvanzo, bitmap, reosarevok ? ^^
      • 2021-10-01 27427, 2021

      • bitmap
        I'll have alook
      • 2021-10-01 27435, 2021

      • bitmap
        Lotheric: which page did you get that on?
      • 2021-10-01 27429, 2021

      • Lotheric
        on this one I think
      • 2021-10-01 27430, 2021

      • Lotheric
      • 2021-10-01 27428, 2021

      • bitmap
        oh I see, we hit max_client_conn in pgbouncer again
      • 2021-10-01 27450, 2021

      • bitmap
        perhaps time to raise that slightly
      • 2021-10-01 27434, 2021

      • Lotheric
      • 2021-10-01 27445, 2021

      • Lotheric
        this one according to my browser history
      • 2021-10-01 27441, 2021

      • bitmap
        thanks
      • 2021-10-01 27403, 2021

      • ruaok
        why not raise it a lot?
      • 2021-10-01 27448, 2021

      • bitmap
        ruaok: I guess I'm worried about overloading PG if something goes wrong, but we could try it
      • 2021-10-01 27401, 2021

      • bitmap
        how much is a lot? :) I was thinking 500 -> 600 initially
      • 2021-10-01 27423, 2021

      • ruaok
        well, is the hardware near breaking right now?
      • 2021-10-01 27435, 2021

      • ruaok
        which one is the primary right now? floyd?
      • 2021-10-01 27441, 2021

      • ruaok
        floyd's max load is at 4 per day.
      • 2021-10-01 27459, 2021

      • bitmap
        yes, floyd
      • 2021-10-01 27426, 2021

      • ruaok looks for the IO stats
      • 2021-10-01 27430, 2021

      • ruaok
        I cant find one, but friday afternoon, one of our busier times and read/write on floyd is in the hundreds of Ks.
      • 2021-10-01 27438, 2021

      • ruaok
        uhm, double it?
      • 2021-10-01 27441, 2021

      • ruaok
        1000?
      • 2021-10-01 27409, 2021

      • bitmap
        looking at the stats we can probably double it. it's only a theoretical maximum anyway
      • 2021-10-01 27414, 2021

      • bitmap
        let's try 1000 then
      • 2021-10-01 27438, 2021

      • ruaok
        if we find that floyd is struggling, lets back off. but floyd is bored right now.
      • 2021-10-01 27410, 2021

      • agatzk has quit
      • 2021-10-01 27456, 2021

      • agatzk joined the channel
      • 2021-10-01 27423, 2021

      • bitmap
        okay, it's set & applied to 1000 now
      • 2021-10-01 27442, 2021

      • ruaok
        great, lets watch the sparks fly. :(
      • 2021-10-01 27406, 2021

      • bitmap
        😰
      • 2021-10-01 27412, 2021

      • dgw has quit
      • 2021-10-01 27417, 2021

      • dgw joined the channel
      • 2021-10-01 27448, 2021

      • bitmap
        I'm still wondering where 500 client connections are coming from...right now we have around 70. no good stats for this though. I might have to write a script that dumps 'show clients;' every 10 seconds or so and collects stats on where the connections are coming from
      • 2021-10-01 27428, 2021

      • ruaok
        well, consider that LB uses the MB DB directly. So does CB and soon AB.
      • 2021-10-01 27404, 2021

      • ruaok
        and soon LB will add more "info about the music" type stuff that will also need more traffic.
      • 2021-10-01 27425, 2021

      • ruaok
        oh, and did you see that the riksucks is working on the "you data missing from MB" feature?
      • 2021-10-01 27431, 2021

      • ruaok
        *your
      • 2021-10-01 27402, 2021

      • ruaok
        we're trying to convert some LB users to add more data to MB.
      • 2021-10-01 27433, 2021

      • bitmap
        does LB connect to MB in bursts (for the stats it collects that you mentioned before)?
      • 2021-10-01 27441, 2021

      • bitmap
        oh yeah, I saw talk about that. that'll be sweet
      • 2021-10-01 27427, 2021

      • ruaok
        not that I can think off right now -- the burst type lookups are pointed at the sitemaps MB mirror.
      • 2021-10-01 27419, 2021

      • ruaok
        the MBID mapping does do direct MB lookups. not sure about its burstiness.
      • 2021-10-01 27427, 2021

      • bitmap
        hmm okay. I didn't think LB alone would make us jump from 70 -> 500 client connections
      • 2021-10-01 27442, 2021

      • ruaok
        that is unlikely, yes
      • 2021-10-01 27458, 2021

      • ruaok
        but zas also mentioned earlier that there was a lot more traffic.
      • 2021-10-01 27410, 2021

      • ruaok
        which is a different question altogether.
      • 2021-10-01 27421, 2021

      • ruaok
        have a look at this:
      • 2021-10-01 27421, 2021

      • ruaok
      • 2021-10-01 27402, 2021

      • ruaok
        the incoming listens graph is one of the jobs that could be bursty. you can see there are peaks when users do imports.
      • 2021-10-01 27425, 2021

      • ruaok
        when you see peaky DB activity, check this graph. if there is a hint of overlap, ping me.
      • 2021-10-01 27428, 2021

      • bitmap
        thanks. yeah, it doesn't really line up with the recent max_client_conn issues from what I can tell
      • 2021-10-01 27401, 2021

      • lucifer
        and we have ....drumroll.... sitewide recordings too now, https://beta-api.listenbrainz.org/1/stats/sitewid…
      • 2021-10-01 27401, 2021

      • riksucks
        Hey guys, I will be afk a bit for this weekend. I will be going to my uni for few days.
      • 2021-10-01 27401, 2021

      • riksucks
        Also lucifer: that looks pretty sweet!
      • 2021-10-01 27453, 2021

      • Guest24 joined the channel
      • 2021-10-01 27413, 2021

      • riksucks
        I hope they open up unis soon in India, I haven't had a single offline class there. And I am already a sophomore ;_;
      • 2021-10-01 27431, 2021

      • bitmap
        I have a script running every 15 seconds on floyd that'll log connections whenever there's > 500, we'll see if it helps
      • 2021-10-01 27455, 2021

      • Guest24 has quit
      • 2021-10-01 27443, 2021

      • ruaok
        good plan.
      • 2021-10-01 27401, 2021

      • ruaok
        for that sort of stuff we now have a method to submit stats to metrics.
      • 2021-10-01 27413, 2021

      • ruaok
        we added a function to our python base layer that writes datapoints to a redis queue.
      • 2021-10-01 27422, 2021

      • v6lur has quit
      • 2021-10-01 27452, 2021

      • ruaok
        we have a scrip that takes the redis queue and shoves it into grafana. the graphs I linked earlier, do the exact same thing.
      • 2021-10-01 27415, 2021

      • ruaok
        you can use this or implement a perl version of it: https://github.com/metabrainz/metric-writer/blob/…
      • 2021-10-01 27430, 2021

      • ruaok
        then you can submit metrics and make graphs -- this sounds like a good graph to make.
      • 2021-10-01 27431, 2021

      • bitmap
        neat, thanks! I'll look into it
      • 2021-10-01 27452, 2021

      • ruaok
        it helps a lot to look at graphs. I've really become a convert.
      • 2021-10-01 27447, 2021

      • Lotheric has quit
      • 2021-10-01 27432, 2021

      • Lotheric joined the channel
      • 2021-10-01 27414, 2021

      • MrClon has quit