#metabrainz

akshaaatt: One thing I noticed on login page: the "Create an account" link should be more distinguishable in a text context.

Agreed yyoung[m] ! Thanks for the feedback :)

dumpjson on aretha needs to be adjusted somehow, alerts are back (they were silenced), but the problem wasn't addressed.

it takes 10 seconds to login to aretha via ftp/ssh or https when they run, not really great for our own file server.

that's my last warning about this btw.

who are you speaking to, zas? if you need someone to do some, you need to address a person in particular.

*something

I don't know who developed this, I don't even know what they are supposed to do. All what I know is that, 2 months ago, I warned about those processes being a problem, and I was asked to silence alerts, which I did, but the problem is still there, and it doesn't seem anyone will ever fix it. So tell me, what I'm supposed to do?

You have a number of options that are better than making threats to no one in specific.

Some of them are:

1) Enlist me to help with this, ideally a last resort.

2) Speak to the devs who have worked on this before, when they are around, not when they aren't or are asleep

3)Offer solutions to the problems: Make a new VM; make room on another machine; ask to see what they are doing to restrict resource use

4) Bring it up during the meeting so that the whole team knows what is up.

So, please pick on and do that instead of passive aggressive anger in this channel.

*one

yes, but those are post-coffee options, and it seems I already tried all those in the past

You didn't do #4. #3 or #1.

yep, another communication without a target. If yvanzo had been your contact person before, why not address yvanzo?

I cannot target anyone, I don't know who did this stuff nor who deployed it

2 lines down from your comment yvanzo explained it to you.

you know WHAT it is and WHO knows the process. that is plenty to go on.

and previously I jumped in and offered solutions like getting a dedicated machine do run the dumps on. you never responded.

ok, so that's my responsibility. I guess I should take a coffee first. I'll find a solution.

that is how we work. and worst case, if you can't track who you talked to, then ask me again.

but really, I don't get it. its a musicbrainz task. that's its name. you know who is on the MB team.

yes, please coffee and solutions.

yvanzo: I don't think https://github.com/metabrainz/docker-server-con... actually changes anything because the load issue comes from disk I/O saturation not cpu. This machine uses HDDs, with software RAID, so heavy I/O aren't exactly their strength. Additionally those processes write to a docker volume, and to internal /tmp, so I guess move operations are in fact copies

so, first I would ensure scripts dump & compress in the same directory and avoid cross-filesystem operations

one option is to make a VM just for the purpose of making dumps and compressing them. we can open a IP/port on the firewall for PG access. once the dumps on the VM are done, copy back to aretha.

and sod the VM, if is overloaded, fine. it will just take longer and that is its sole job.

second, the use of ionice for all intensive processes in play or use docker io options (docker run --help | egrep 'bps|IO')

ruaok: yes, offloading is the last resort, but imho the process isn't efficient, it seems to me it moves files across dirs/fs and this is something to address first, to reduce needed resources. According to what I saw, it writes to /tmp in the container, but files ends on another volume, it means extra I/O for sure.

ruaok: about offloading, we can start a VM on demand for this task. On Aretha, it runs for 20 hours every week, so that's a very good candidate for that

Coffee sounds like a good idea

zas: Thanks for the pointers. I will check if copies can be made on the same filesystem and try using ionice. Already tried to use some docker io options: https://github.com/metabrainz/docker-server-con...

yvanzo: how long can we allow this dump to run? If reducing I/O load isn't enough we can spawn a VM for it, but I guess it depends on consul too, so we may need a virtual network for the VM to access consul/pg

also I noticed it writes to a temporary directory (in /tmp inside the container), is that used before compression? or are files written there then moved?

Allowing these dumps to run for 12h on Wed and Sat is more than enough. I think it takes even less than 6h in general, would have to check again.

according to graphs, it seems to run longer than that, and I noticed barman & search dumps run during the same time window

I think we should ensure those don't run all together to start with, I wonder about barman too, it is pretty intensive on disk I/O too

hm barman

tht was one of those tickets

eg https://tickets.metabrainz.org/browse/MBH-565

MBH-565: Recovering prod databases after a disaster

should I add points for BB, picard, mb? too?

Neither MB (already tested by bitmap) nor Picard (not having a prod database), but probably for AB, BB, CB, LB.

ab and lb are menioned, cb and bb not, i'l ladd thme as well

.. once this damn browser stops loking up -__-

zas: temporary directory is used to build dumps and then compress those

using ionice is totally applicable to compression commands

are compressed files written to /tmp before they are moved to final destination?

yes

I wonder if we could just use a temporary dir on the target fs instead

we can do that

zas: the current /tmp directory isn’t a special volume either (just container's root fs I guess)

yes, so that's overlay2 fs: not sure how it works in details, there are performance tips: https://docs.docker.com/storage/storagedriver/o...

"Use volumes for write-heavy workloads"

zas: the destination volume is stored on / partition of aretha.

So it's probably not a good idea to build/compress these dumps on aretha as it uses HDDs.

I can do these changes (ionice and using target volume directly for building/compressing dumps) and see if it gets any better on Wed.

Last we can try to serialize these dumps (which are still expected to be made on the same day.)