#metabrainz

/

      • mruszczyk has quit
      • akshaaatt has quit
      • mruszczyk joined the channel
      • akshaaatt joined the channel
      • zas has quit
      • zas joined the channel
      • KassOtsimine has quit
      • revi has quit
      • revi joined the channel
      • KassOtsimine joined the channel
      • Sophist-UK has quit
      • Shubh joined the channel
      • gcrkrause1 has quit
      • gcrkrause1 joined the channel
      • yyoung[m]
        akshaaatt: One thing I noticed on login page: the "Create an account" link should be more distinguishable in a text context.
      • akshaaatt
        Agreed yyoung[m] ! Thanks for the feedback :)
      • Shubh has quit
      • Shubh joined the channel
      • Shubh80 joined the channel
      • Shubh has quit
      • Shubh80 has quit
      • Shubh joined the channel
      • Shubh has quit
      • Shubh joined the channel
      • Shubh has quit
      • Shubh joined the channel
      • Shubh has quit
      • Shubh joined the channel
      • Shubh has quit
      • Shubh81 joined the channel
      • Shubh81 has quit
      • Shubh joined the channel
      • Shubh has quit
      • Shubh joined the channel
      • zas
        dumpjson on aretha needs to be adjusted somehow, alerts are back (they were silenced), but the problem wasn't addressed.
      • it takes 10 seconds to login to aretha via ftp/ssh or https when they run, not really great for our own file server.
      • that's my last warning about this btw.
      • ruaok
        who are you speaking to, zas? if you need someone to do some, you need to address a person in particular.
      • *something
      • zas
        I don't know who developed this, I don't even know what they are supposed to do. All what I know is that, 2 months ago, I warned about those processes being a problem, and I was asked to silence alerts, which I did, but the problem is still there, and it doesn't seem anyone will ever fix it. So tell me, what I'm supposed to do?
      • ruaok
        You have a number of options that are better than making threats to no one in specific.
      • Some of them are:
      • 1) Enlist me to help with this, ideally a last resort.
      • 2) Speak to the devs who have worked on this before, when they are around, not when they aren't or are asleep
      • 3)Offer solutions to the problems: Make a new VM; make room on another machine; ask to see what they are doing to restrict resource use
      • 4) Bring it up during the meeting so that the whole team knows what is up.
      • So, please pick on and do that instead of passive aggressive anger in this channel.
      • *one
      • zas
        yes, but those are post-coffee options, and it seems I already tried all those in the past
      • ruaok
        You didn't do #4. #3 or #1.
      • zas
      • TOPIC: MetaBrainz Community and Development channel | MusicBrainz non-development: #musicbrainz | BookBrainz: #bookbrainz | Channel is logged; see https://musicbrainz.org/doc/IRC for details | Agenda: Reviews, Follow up on Summit Notes -> Tickets, theless/insolite (reosarevok), aretha load (zas)
      • ruaok
        yep, another communication without a target. If yvanzo had been your contact person before, why not address yvanzo?
      • zas
        I cannot target anyone, I don't know who did this stuff nor who deployed it
      • ruaok
        2 lines down from your comment yvanzo explained it to you.
      • you know WHAT it is and WHO knows the process. that is plenty to go on.
      • and previously I jumped in and offered solutions like getting a dedicated machine do run the dumps on. you never responded.
      • zas
        ok, so that's my responsibility. I guess I should take a coffee first. I'll find a solution.
      • ruaok
        that is how we work. and worst case, if you can't track who you talked to, then ask me again.
      • but really, I don't get it. its a musicbrainz task. that's its name. you know who is on the MB team.
      • yes, please coffee and solutions.
      • Sophist-UK joined the channel
      • zas
        yvanzo: I don't think https://github.com/metabrainz/docker-server-con... actually changes anything because the load issue comes from disk I/O saturation not cpu. This machine uses HDDs, with software RAID, so heavy I/O aren't exactly their strength. Additionally those processes write to a docker volume, and to internal /tmp, so I guess move operations are in fact copies
      • so, first I would ensure scripts dump & compress in the same directory and avoid cross-filesystem operations
      • ruaok
        one option is to make a VM just for the purpose of making dumps and compressing them. we can open a IP/port on the firewall for PG access. once the dumps on the VM are done, copy back to aretha.
      • and sod the VM, if is overloaded, fine. it will just take longer and that is its sole job.
      • zas
        second, the use of ionice for all intensive processes in play or use docker io options (docker run --help | egrep 'bps|IO')
      • ruaok: yes, offloading is the last resort, but imho the process isn't efficient, it seems to me it moves files across dirs/fs and this is something to address first, to reduce needed resources. According to what I saw, it writes to /tmp in the container, but files ends on another volume, it means extra I/O for sure.
      • Sophist-UK has quit
      • ruaok: about offloading, we can start a VM on demand for this task. On Aretha, it runs for 20 hours every week, so that's a very good candidate for that
      • CatQuest pets zas
      • reg[m]
        Coffee sounds like a good idea
      • Shubh has quit
      • Shubh joined the channel
      • Shubh has quit
      • Shubh joined the channel
      • Shubh has quit
      • Shubh joined the channel
      • Shubh has quit
      • Shubh joined the channel
      • Shubh has quit
      • Shubh joined the channel
      • Shubh has quit
      • Shubh joined the channel
      • Shubh has quit
      • Shubh joined the channel
      • Shubh has quit
      • Shubh joined the channel
      • Shubh has quit
      • yvanzo
        zas: Thanks for the pointers. I will check if copies can be made on the same filesystem and try using ionice. Already tried to use some docker io options: https://github.com/metabrainz/docker-server-con...
      • zas
        yvanzo: how long can we allow this dump to run? If reducing I/O load isn't enough we can spawn a VM for it, but I guess it depends on consul too, so we may need a virtual network for the VM to access consul/pg
      • also I noticed it writes to a temporary directory (in /tmp inside the container), is that used before compression? or are files written there then moved?
      • yvanzo
        Allowing these dumps to run for 12h on Wed and Sat is more than enough. I think it takes even less than 6h in general, would have to check again.
      • SothoTalKer_ joined the channel
      • SothoTalKer has quit
      • SothoTalKer joined the channel
      • SothoTalKer_ has quit
      • zas
        according to graphs, it seems to run longer than that, and I noticed barman & search dumps run during the same time window
      • I think we should ensure those don't run all together to start with, I wonder about barman too, it is pretty intensive on disk I/O too
      • CatQuest
        hm barman
      • tht was one of those tickets
      • BrainzBot
        MBH-565: Recovering prod databases after a disaster
      • CatQuest
        should I add points for BB, picard, mb? too?
      • yvanzo
        Neither MB (already tested by bitmap) nor Picard (not having a prod database), but probably for AB, BB, CB, LB.
      • CatQuest
        ab and lb are menioned, cb and bb not, i'l ladd thme as well
      • .. once this damn browser stops loking up -__-
      • yvanzo
        zas: temporary directory is used to build dumps and then compress those
      • using ionice is totally applicable to compression commands
      • v6lur joined the channel
      • zas
        are compressed files written to /tmp before they are moved to final destination?
      • yvanzo
        yes
      • zas
        I wonder if we could just use a temporary dir on the target fs instead
      • yvanzo
        we can do that
      • zas: the current /tmp directory isn’t a special volume either (just container's root fs I guess)
      • zas
        yes, so that's overlay2 fs: not sure how it works in details, there are performance tips: https://docs.docker.com/storage/storagedriver/o...
      • "Use volumes for write-heavy workloads"
      • yvanzo
        zas: the destination volume is stored on / partition of aretha.
      • So it's probably not a good idea to build/compress these dumps on aretha as it uses HDDs.
      • I can do these changes (ionice and using target volume directly for building/compressing dumps) and see if it gets any better on Wed.
      • Last we can try to serialize these dumps (which are still expected to be made on the same day.)
      • Clint_ is now known as Clint
      • dseomn has quit
      • dseomn joined the channel
      • v6lur has quit