akshaaatt: One thing I noticed on login page: the "Create an account" link should be more distinguishable in a text context.
akshaaatt
Agreed yyoung[m] ! Thanks for the feedback :)
Shubh has quit
Shubh joined the channel
Shubh80 joined the channel
Shubh has quit
Shubh80 has quit
Shubh joined the channel
Shubh has quit
Shubh joined the channel
Shubh has quit
Shubh joined the channel
Shubh has quit
Shubh joined the channel
Shubh has quit
Shubh81 joined the channel
Shubh81 has quit
Shubh joined the channel
Shubh has quit
Shubh joined the channel
zas
dumpjson on aretha needs to be adjusted somehow, alerts are back (they were silenced), but the problem wasn't addressed.
it takes 10 seconds to login to aretha via ftp/ssh or https when they run, not really great for our own file server.
that's my last warning about this btw.
ruaok
who are you speaking to, zas? if you need someone to do some, you need to address a person in particular.
*something
zas
I don't know who developed this, I don't even know what they are supposed to do. All what I know is that, 2 months ago, I warned about those processes being a problem, and I was asked to silence alerts, which I did, but the problem is still there, and it doesn't seem anyone will ever fix it. So tell me, what I'm supposed to do?
ruaok
You have a number of options that are better than making threats to no one in specific.
Some of them are:
1) Enlist me to help with this, ideally a last resort.
2) Speak to the devs who have worked on this before, when they are around, not when they aren't or are asleep
3)Offer solutions to the problems: Make a new VM; make room on another machine; ask to see what they are doing to restrict resource use
4) Bring it up during the meeting so that the whole team knows what is up.
So, please pick on and do that instead of passive aggressive anger in this channel.
*one
zas
yes, but those are post-coffee options, and it seems I already tried all those in the past
TOPIC: MetaBrainz Community and Development channel | MusicBrainz non-development: #musicbrainz | BookBrainz: #bookbrainz | Channel is logged; see https://musicbrainz.org/doc/IRC for details | Agenda: Reviews, Follow up on Summit Notes -> Tickets, theless/insolite (reosarevok), aretha load (zas)
ruaok
yep, another communication without a target. If yvanzo had been your contact person before, why not address yvanzo?
zas
I cannot target anyone, I don't know who did this stuff nor who deployed it
ruaok
2 lines down from your comment yvanzo explained it to you.
you know WHAT it is and WHO knows the process. that is plenty to go on.
and previously I jumped in and offered solutions like getting a dedicated machine do run the dumps on. you never responded.
zas
ok, so that's my responsibility. I guess I should take a coffee first. I'll find a solution.
ruaok
that is how we work. and worst case, if you can't track who you talked to, then ask me again.
but really, I don't get it. its a musicbrainz task. that's its name. you know who is on the MB team.
yes, please coffee and solutions.
Sophist-UK joined the channel
zas
yvanzo: I don't think https://github.com/metabrainz/docker-server-con... actually changes anything because the load issue comes from disk I/O saturation not cpu. This machine uses HDDs, with software RAID, so heavy I/O aren't exactly their strength. Additionally those processes write to a docker volume, and to internal /tmp, so I guess move operations are in fact copies
so, first I would ensure scripts dump & compress in the same directory and avoid cross-filesystem operations
ruaok
one option is to make a VM just for the purpose of making dumps and compressing them. we can open a IP/port on the firewall for PG access. once the dumps on the VM are done, copy back to aretha.
and sod the VM, if is overloaded, fine. it will just take longer and that is its sole job.
zas
second, the use of ionice for all intensive processes in play or use docker io options (docker run --help | egrep 'bps|IO')
ruaok: yes, offloading is the last resort, but imho the process isn't efficient, it seems to me it moves files across dirs/fs and this is something to address first, to reduce needed resources. According to what I saw, it writes to /tmp in the container, but files ends on another volume, it means extra I/O for sure.
Sophist-UK has quit
ruaok: about offloading, we can start a VM on demand for this task. On Aretha, it runs for 20 hours every week, so that's a very good candidate for that
CatQuest pets zas
reg[m]
Coffee sounds like a good idea
Shubh has quit
Shubh joined the channel
Shubh has quit
Shubh joined the channel
Shubh has quit
Shubh joined the channel
Shubh has quit
Shubh joined the channel
Shubh has quit
Shubh joined the channel
Shubh has quit
Shubh joined the channel
Shubh has quit
Shubh joined the channel
Shubh has quit
Shubh joined the channel
Shubh has quit
yvanzo
zas: Thanks for the pointers. I will check if copies can be made on the same filesystem and try using ionice. Already tried to use some docker io options: https://github.com/metabrainz/docker-server-con...
zas
yvanzo: how long can we allow this dump to run? If reducing I/O load isn't enough we can spawn a VM for it, but I guess it depends on consul too, so we may need a virtual network for the VM to access consul/pg
also I noticed it writes to a temporary directory (in /tmp inside the container), is that used before compression? or are files written there then moved?
yvanzo
Allowing these dumps to run for 12h on Wed and Sat is more than enough. I think it takes even less than 6h in general, would have to check again.
SothoTalKer_ joined the channel
SothoTalKer has quit
SothoTalKer joined the channel
SothoTalKer_ has quit
zas
according to graphs, it seems to run longer than that, and I noticed barman & search dumps run during the same time window