14:23 PM
FichteFoll has quit
14:23 PM
monkey
See the last 8 commits, repeat of previous ones.
14:24 PM
CatQuest
personally i want to SETTINGS ALL THE THINGS :D
14:24 PM
monkey
If you have the branch without those extra commits I pushed, you can force push
14:28 PM
CatQuest
mayhem: lmao <3
14:28 PM
oh god that list. i are at al labout exactly uh one song (soad)
14:29 PM
al lthe other songs are like "never heard it" or "have heard it on the radio or osmething an age ago and odn't care about it"
14:29 PM
FichteFoll joined the channel
14:29 PM
it's basically late 90's/early 00's radio pop
14:30 PM
play some older stuff liek beatles or pink plonk but it's always the one hit or whatever
14:30 PM
not to diss the wall, but why play that when yo ucna play "Money" ?
14:35 PM
mayhem
14:41 PM
monkey wonders if there should also be a way to manually set the canonical recording like we do for RG cover art for example
14:41 PM
in this case, it would be a good idea.
14:41 PM
14:41 PM
zas
atj: ping
14:41 PM
mayhem
the japanese version is the first one, so its not a bug per se.
14:42 PM
but in this case, the data should be against release groups anyway.
14:45 PM
atj
zas: hi
14:45 PM
zas
14:49 PM
14:52 PM
mayhem
14:52 PM
we do care about these things anymore??
14:55 PM
Sophist-UK has quit
14:55 PM
zas
well, not even sure on what basis they award, and I'm always suspicious about websites providing free binaries to windows users...
14:55 PM
14:55 PM
outsidecontext
14:56 PM
atj
why is a company that runs a download site registered in the british virgin islands
14:56 PM
mayhem
front for drugs.
14:56 PM
zas
bitmap: around?
14:57 PM
outsidecontext
I'd much rather recommend users to get the software from the actual website and not some fishy download site
14:57 PM
Sophist-UK joined the channel
14:57 PM
zas
definitively
14:57 PM
bitmap
zas: yep
14:57 PM
zas
and I would also recommend users to stop using Windows ;)
14:58 PM
atj
ok, let's start breaking things
14:58 PM
zas
we take down everything in 2 minutes
14:59 PM
mayhem
scary
14:59 PM
bitmap
let me know if I'm needed, I'll just watch IRC for now :)
14:59 PM
atj
bitmap: did you see that test. is broken?
14:59 PM
bitmap
no I did not, I'll have a look
14:59 PM
atj
can't connect to database error
15:00 PM
CatQuest
15:00 PM
becasue of ルパン三世
15:00 PM
trolley has quit
15:02 PM
zas
all services marked down on gateways
15:02 PM
atj
ready to go then?
15:02 PM
zas
atj: you can proceed
15:02 PM
atj
ok thanks
15:02 PM
CatQuest
ahh..
15:02 PM
trolley joined the channel
15:03 PM
monkey
And down they go 🫡
15:03 PM
atj
pour one out for them
15:03 PM
mayhem pretends to pour out mezcal
15:04 PM
mayhem
15:04 PM
is up! ;)
15:04 PM
atj
aretha, aphex, clash done (they already were but re-run to be sure)
15:04 PM
floyd done
15:04 PM
zas
we upgrade after?
15:05 PM
atj
do you want to upgrade now? i can't follow the guide without writing a new playbook
15:05 PM
zas
atj: we'll do after
15:05 PM
atj
ok
15:06 PM
gaga done
15:06 PM
hip done
15:07 PM
kiss done
15:07 PM
ludwig done
15:09 PM
minguys / paco / patton ✅
15:10 PM
mayhem
is maxguys done?
15:10 PM
atj
pink / quest / rakim ✅
15:13 PM
rex / rudi / serge / trille / zappa ✅
15:13 PM
all done
15:14 PM
zas
ok, let's check now
15:14 PM
atj
`consul members` output looks fine
15:14 PM
Sophist-UK has quit
15:14 PM
Sophist-UK joined the channel
15:15 PM
zas
15:15 PM
atj
a very long winded way of fixing the issue
15:15 PM
mayhem
YYYYYAAAAAAYYYY, thanks!
15:16 PM
zas
lb team, can you check if rabbitmq works now?
15:16 PM
(i mean with consul)
15:17 PM
mayhem
lucifer: you about?
15:17 PM
not sure how to do this, zas.
15:17 PM
lucifer
mayhem: hi yes.
15:17 PM
mayhem
^^
15:17 PM
check to see if rabbitmq is happy now, please.
15:20 PM
lucifer
zas: i get 503 on beta.lb
15:21 PM
15:21 PM
container logs look fine though so rmq seems to be working fine.
15:22 PM
zas
503s are normal from the web, services are all down on gateways
15:23 PM
atj: everything looks fine, can we proceed to the upgrade? (in order to avoid another downtime later)
15:24 PM
atj
ok, let me see how we can do this
15:24 PM
zas
bump the version, and run on 4 non-leaders servers first
15:24 PM
rex-server is leader
15:25 PM
the leader has to be done last
15:25 PM
that's for servers, then we can deploy new clients
15:27 PM
mayhem
15:28 PM
zas
bitmap: can you check caa-indexer?
15:29 PM
bitmap
ok
15:29 PM
lucifer
mayhem: to answer the earlier question, this is MLHD+ only
15:30 PM
mayhem
ok, that data suggested that, thanks.
15:30 PM
bitmap
zas: started the caa-indexer container (it was stopped earlier) and uploads are succeeding, so looks fine
15:31 PM
zas
everything seems to work
15:31 PM
lucifer
monkey: no worries, i'll do some cleanups on the base branch first.
15:31 PM
zas
atj: I think we can do the upgrade, then restore services, we'll have a bit of post-move cleanup to do (mainly old containers to remove)
15:31 PM
monkey
Thank you, and sorry for fucking it up
15:34 PM
atj
zas: gimme a few minutes, i need to orchestrate this properly
15:34 PM
lucifer
monkey: i don't think it was anything on your end, rather i forced pushed the base branch a few and that messed it up.
15:40 PM
atj
zas: starting on mingus
15:40 PM
zas
ok
15:42 PM
just noticed mingus' syslog is flooded by docker community container logs... weird
15:43 PM
but unrelated to current move
15:44 PM
mingus upgraded
15:46 PM
atj
yep
15:46 PM
will do all except rex now
15:46 PM
zas
ok
15:46 PM
one after the other
15:46 PM
atj
serial is set to 1
15:48 PM
i tried taking a snapshot as it said in the guide but it tries to write to /tmp which is readonly
15:48 PM
zas
/tmp is readonly?
15:48 PM
atj
i locked the systemd service down
15:49 PM
zas
15:51 PM
atj
15:52 PM
systemd sandboxing is very useful
15:52 PM
ok, all done except for rex
15:52 PM
zas: happy for me to proceed with rex?
15:52 PM
zas
yes
15:54 PM
atj
done
15:54 PM
zas
yup, confirmed
15:54 PM
atj
all servers showing 1.15.3
15:54 PM
zas
mingus is now leader
15:54 PM
atj
good for them
15:54 PM
zas
ok, now proceed to clients?
15:55 PM
atj
i think just restarting the service should be enough
15:55 PM
it clearly issues a leave on orderly shutdown
15:55 PM
zas
stats with aphex and let's see
15:55 PM
atj
ok
15:55 PM
zas
*start
15:56 PM
atj
done
15:57 PM
15:57 PM
shows it issued a leave
15:57 PM
CatQuest
I didn't know there would be a server maintenance today..
15:57 PM
I was lucky i did some work in the morning :D
15:58 PM
zas
atj: looks good, we can proceed for the rest
15:59 PM
15:59 PM
not sure about this error, but consul logs are full of errors that should be warnings...
16:00 PM
16:03 PM
aretha left and rejoin, but still on 1.15.2?
16:03 PM
atj
yeah, my bad I thought I could do it with a oneshot
16:04 PM
one second
16:06 PM
ok here we go, aretha first