#metabrainz

alastairp: which commands did you use?

ah you ran start_services.sh

so yes, it called start_registrator, which rm the running container and start a new one

this is how I've always done it. is there a better way that doesn't try and restart all services?

I just copy the run script (cp cage.sh cage_tmp.sh), remove unneeded lines to keep only the concerned service, and run it, then if all is ok, then copy changes to real scripts, remove tmp script, commti changes, push

but we could seriously improve this... ;)

right

I guess we could also source services.sh in the shell and just run the start_ method too

https://stats.metabrainz.org/d/gorXwgPMz/all-cp... to monitor cage's cpu activity

_lucifer: the `pytest -m "not database"` on the python2 tests is amazing. I'm glad it works!

just waiting for travis to run and I'll merge 44

yeah, worked like a charm :D

_lucifer: hmm, I'm just thinking about https://github.com/metabrainz/brainzutils-pytho...

in the context of the other stuff that we've discussed

I wonder if psycopg2 should be installed in requirements_dev, not requirements?

technically, it's only needed for the tests?

sqlalchemy uses it under the hood or no?

only if you connect to a postgres database

(yes, sure all of our downstream packages do, but...)

I agree, it's probably not too much of an issue to have it there, but I think in this context it makes sense to add it to requirements_dev instead

I'll do it

sure makes sense

travis passed!

[brainzutils-python] alastair merged pull request #44 (master…mbdb-test): Test musicbrainz db methods against a real musicbrainz sample database https://github.com/metabrainz/brainzutils-pytho...

[brainzutils-python] alastair merged pull request #45 (master…version-ranges): Use open-ended versions so that downstream projects can pin exact versions https://github.com/metabrainz/brainzutils-pytho...

_lucifer: 46 is all yours

after that it might be nice to double-check https://github.com/metabrainz/listenbrainz-serv... again :)

ignore alerts

alastairp: are you on the telegram group for nagios and for MB Alerts?

I am no

not

in fact, I've made it this far without even installing telegram

I was just made aware of that fact. plz to be installing it.

done. same number as always

ok, added you.

I like it when we wait long enough to deal with an "add autoselect to domain X" that domain X is no longer active

Saved us the time I guess

oh you

yvanzo, bitmap: do you think it's important to maintain consistency in this case? https://github.com/metabrainz/musicbrainz-serve...

We shouldn't do center-alignment on the release view because the display of rels and recording artists expect the main content to be on top

But I'm fine with top-aligning in edits unless you think it should stay like that

alastairp: 46 ready to merge

[listenbrainz-server] MonkeyDo opened pull request #1276 (master…playlist_collaborators): Allow playlist collaborators to access private playlist https://github.com/metabrainz/listenbrainz-serv...

More pairs of eyes welcome on this python ^

[brainzutils-python] alastair merged pull request #46 (master…bugfix2): Use correct number of variables in unpacking merged entities https://github.com/metabrainz/brainzutils-pytho...

thanks _lucifer! that's it for BU, right?

do you want to test LB and CB again with the latest master (check that it installs, check that stuff like CB database lookups work), and confirm with me?

then we can merge the LB change, I'll schedule a CB merge party for Friday

CPU usage on cage spiked again

alastairp: yes that's for BU. sure, i'll check again with LB and then later for CB as well

zas: ^^

let me turn jenkins workers down to 3

and with CPU spikes come MB service degradations. boo.

zas: do you have a machine at hetzerner dedicated to "processing" tasks that are not impacting services with sensitive response times?

ruaok: best choice atm would be paco

gateways-redis isn't used anymore (but I keep it for now in case keydb has issues)

and there's pg-williams (<-- why???)

lol.

yes, that used to be on.. williams. but the service was never renamed.

services shouldn't be named after machines

agreed.

but nvm

ok, looking at the weekly load graph, I see that paco would be pretty good.

there are two loads spikes that go to 2. and no time sensitive services.

alastairp: what is needed to move the service from cage to paco? just editing the nodes file and stopping/starting services or more than that?

> Formatting correction: i hate this

I feel you Mr_Monkey

I really do.

So counter-intuitive for me https://www.flake8rules.com/rules/E125.html

ruaok: we need to copy the jenkins-data volume too: /var/lib/docker/volumes/jenkins-data/_data

want me to do that?

yes please, I don't really have any time left today to look at this

ok.

what is the procedure for migrating a volume?

rsync :)

take containers down, create new volume, rsync?

note that ci.metabrainz.org is magic based on the service existing, so you'll have to shut down on cage before starting up on paco, then it should just magically work

at least that's what happened when I migrated from williams

k

I wonder why a cpu load on cage affects all mbs services

alastairp: https://github.com/metabrainz/docker-server-con...

ruaok: looks good

can you just check while you have this PR open

I think I may have forgotten to remove start_jenkins from williams

cage is cleaned up.

why do we host our own jenkins instance instead of using a third party service? wouldn't that save resources?

alastairp: removed that line from williams setup. approve the PR after that plz.

approved, thanks

_lucifer: yeah, it's a tradeoff. some of it comes from the fact that we've had it for a long time

keep in mind also all the trouble that we had with travis, just by moving it to another service isn't going to guarantee that all the problems go away

troubles like docker limits?

we also have a few musicbrainz-specific tasks that run on jenkins for which it is nice to have access to the rest of our servers

troubles like travis taking 30 minutes before it would start running tests, because they had load issues too

from consulserver logs https://www.irccloud.com/pastebin/c1LxkDuY/

yeah that makes sense

load shot up to 90 when I logged in after _lucifer said so

(_lucifer why did you know??)

but dropped almost immediately

from cage's consulagent https://www.irccloud.com/pastebin/f2FD4SSD/

zas had shared the link and it was open in a tab. i was closing tabs when it just spiked, alastairp :)

but there were also LB js tests running. oddly, `ps` hung while printing info about js processes :/

I hope jest doesn't spawn a million threads for whatever reason

Mr_Monkey: when my time frees up next month, I agree that we should sit down and try and improve python style guides. This should include instructions to make vscode do the right thing when you press return, consolidation of the tools that we have locally, in jenkins and squaking, and removal of stupid warnings that we don't want

[listenbrainz-server] MonkeyDo opened pull request #1277 (master…react-api-errors): Improve API error management on front-end https://github.com/metabrainz/listenbrainz-serv...

Yeah, that would be nice. I kept looking in the repo for a tool I can run to fix such issues

it's unfortunate that there's only 1 python tool that does it automatically (black), and I'm not a big fan of it

all of the other tools only warn, rather than actually fixing it :(

Ah, yes, we talked about opinionated formatters

Meh.

alastairp: and they should allow for others to use different editors than just VScode

alastairp: autopep8?

And even as an automatic github action: https://github.com/marketplace/actions/autopep8

jenkins starting on paco.

someone push jenkins to retest a project so we can see if it worked.

#1276, alastairp ?

there are already a million things (still?) in the queue

let me cancel them

> java.lang.Exception: Could not cancel run for id 12583

maybe we should've cleared the queue before moving?

so, it's definitely responding then :)

I think it's because I tried to delete something twice

seems positive so far.

ok, all queues and workers cleared. I'll trigger a job

yep, everything's fine

williams docker configs updated.

good idea

should I nuke the old volume on cage?

I still have tar archive of the volume, just in case.

yes, I guess so

in fact, lol. good timing. I forgot to set up backups on cage ;)

the volume is still on williams too, and I have a task for friday to remove it

I can do that too.

alastairp: tested LB with latest BU, works.

_lucifer: great, let me make a release then

oh, it'd be good to test CB too, with the db changes

yeah, that will need manual testing.

This looks very much to what is described at https://github.com/hashicorp/consul/issues/1007... https://www.irccloud.com/pastebin/idRySunN/

and full mb db. let me setup a dev instance on bono.

the jenkins container was stopped, but still alive on williams.

cage is cleaned up, williams is cleaned up.

thanks

np.

lets hope that things are more stable going forward.

there will still be docker images on cage, because we haven't merged 1256 yet

I removed everything that looked to be jenkinsy.