14:23 PM
FichteFoll has quit
2023-06-06 15757, 2023
14:23 PM
monkey
See the last 8 commits, repeat of previous ones.
2023-06-06 15701, 2023
14:24 PM
CatQuest
personally i want to SETTINGS ALL THE THINGS :D
2023-06-06 15720, 2023
14:24 PM
monkey
If you have the branch without those extra commits I pushed, you can force push
2023-06-06 15715, 2023
14:28 PM
CatQuest
mayhem: lmao <3
2023-06-06 15743, 2023
14:28 PM
CatQuest
oh god that list. i are at al labout exactly uh one song (soad)
2023-06-06 15715, 2023
14:29 PM
CatQuest
al lthe other songs are like "never heard it" or "have heard it on the radio or osmething an age ago and odn't care about it"
2023-06-06 15715, 2023
14:29 PM
FichteFoll joined the channel
2023-06-06 15743, 2023
14:29 PM
CatQuest
it's basically late 90's/early 00's radio pop
2023-06-06 15715, 2023
14:30 PM
CatQuest
play some older stuff liek beatles or pink plonk but it's always the one hit or whatever
2023-06-06 15738, 2023
14:30 PM
CatQuest
not to diss the wall, but why play that when yo ucna play "Money" ?
2023-06-06 15749, 2023
14:35 PM
mayhem
2023-06-06 15704, 2023
14:41 PM
monkey wonders if there should also be a way to manually set the canonical recording like we do for RG cover art for example
2023-06-06 15719, 2023
14:41 PM
mayhem
in this case, it would be a good idea.
2023-06-06 15724, 2023
14:41 PM
mayhem
2023-06-06 15739, 2023
14:41 PM
zas
atj: ping
2023-06-06 15759, 2023
14:41 PM
mayhem
the japanese version is the first one, so its not a bug per se.
2023-06-06 15711, 2023
14:42 PM
mayhem
but in this case, the data should be against release groups anyway.
2023-06-06 15703, 2023
14:45 PM
atj
zas: hi
2023-06-06 15734, 2023
14:45 PM
zas
2023-06-06 15705, 2023
14:49 PM
zas
2023-06-06 15725, 2023
14:52 PM
mayhem
2023-06-06 15733, 2023
14:52 PM
mayhem
we do care about these things anymore??
2023-06-06 15705, 2023
14:55 PM
Sophist-UK has quit
2023-06-06 15714, 2023
14:55 PM
zas
well, not even sure on what basis they award, and I'm always suspicious about websites providing free binaries to windows users...
2023-06-06 15741, 2023
14:55 PM
zas
2023-06-06 15745, 2023
14:55 PM
outsidecontext
2023-06-06 15704, 2023
14:56 PM
atj
why is a company that runs a download site registered in the british virgin islands
2023-06-06 15717, 2023
14:56 PM
mayhem
front for drugs.
2023-06-06 15724, 2023
14:56 PM
zas
bitmap: around?
2023-06-06 15706, 2023
14:57 PM
outsidecontext
I'd much rather recommend users to get the software from the actual website and not some fishy download site
2023-06-06 15723, 2023
14:57 PM
Sophist-UK joined the channel
2023-06-06 15724, 2023
14:57 PM
zas
definitively
2023-06-06 15753, 2023
14:57 PM
bitmap
zas: yep
2023-06-06 15757, 2023
14:57 PM
zas
and I would also recommend users to stop using Windows ;)
2023-06-06 15748, 2023
14:58 PM
atj
ok, let's start breaking things
2023-06-06 15753, 2023
14:58 PM
zas
we take down everything in 2 minutes
2023-06-06 15701, 2023
14:59 PM
mayhem
scary
2023-06-06 15701, 2023
14:59 PM
bitmap
let me know if I'm needed, I'll just watch IRC for now :)
2023-06-06 15718, 2023
14:59 PM
atj
bitmap: did you see that test. is broken?
2023-06-06 15734, 2023
14:59 PM
bitmap
no I did not, I'll have a look
2023-06-06 15744, 2023
14:59 PM
atj
can't connect to database error
2023-06-06 15733, 2023
15:00 PM
CatQuest
2023-06-06 15752, 2023
15:00 PM
CatQuest
becasue of ルパン三世
2023-06-06 15756, 2023
15:00 PM
trolley has quit
2023-06-06 15700, 2023
15:02 PM
zas
all services marked down on gateways
2023-06-06 15716, 2023
15:02 PM
atj
ready to go then?
2023-06-06 15717, 2023
15:02 PM
zas
atj: you can proceed
2023-06-06 15721, 2023
15:02 PM
atj
ok thanks
2023-06-06 15735, 2023
15:02 PM
CatQuest
ahh..
2023-06-06 15740, 2023
15:02 PM
trolley joined the channel
2023-06-06 15714, 2023
15:03 PM
monkey
And down they go 🫡
2023-06-06 15726, 2023
15:03 PM
atj
pour one out for them
2023-06-06 15748, 2023
15:03 PM
mayhem pretends to pour out mezcal
2023-06-06 15706, 2023
15:04 PM
mayhem
2023-06-06 15709, 2023
15:04 PM
mayhem
is up! ;)
2023-06-06 15723, 2023
15:04 PM
atj
aretha, aphex, clash done (they already were but re-run to be sure)
2023-06-06 15733, 2023
15:04 PM
atj
floyd done
2023-06-06 15738, 2023
15:04 PM
zas
we upgrade after?
2023-06-06 15706, 2023
15:05 PM
atj
do you want to upgrade now? i can't follow the guide without writing a new playbook
2023-06-06 15741, 2023
15:05 PM
zas
atj: we'll do after
2023-06-06 15743, 2023
15:05 PM
atj
ok
2023-06-06 15715, 2023
15:06 PM
atj
gaga done
2023-06-06 15745, 2023
15:06 PM
atj
hip done
2023-06-06 15714, 2023
15:07 PM
atj
kiss done
2023-06-06 15746, 2023
15:07 PM
atj
ludwig done
2023-06-06 15704, 2023
15:09 PM
atj
minguys / paco / patton ✅
2023-06-06 15709, 2023
15:10 PM
mayhem
is maxguys done?
2023-06-06 15722, 2023
15:10 PM
atj
pink / quest / rakim ✅
2023-06-06 15735, 2023
15:13 PM
atj
rex / rudi / serge / trille / zappa ✅
2023-06-06 15742, 2023
15:13 PM
atj
all done
2023-06-06 15703, 2023
15:14 PM
zas
ok, let's check now
2023-06-06 15719, 2023
15:14 PM
atj
`consul members` output looks fine
2023-06-06 15728, 2023
15:14 PM
Sophist-UK has quit
2023-06-06 15748, 2023
15:14 PM
Sophist-UK joined the channel
2023-06-06 15706, 2023
15:15 PM
zas
2023-06-06 15738, 2023
15:15 PM
atj
a very long winded way of fixing the issue
2023-06-06 15754, 2023
15:15 PM
mayhem
YYYYYAAAAAAYYYY, thanks!
2023-06-06 15709, 2023
15:16 PM
zas
lb team, can you check if rabbitmq works now?
2023-06-06 15716, 2023
15:16 PM
zas
(i mean with consul)
2023-06-06 15724, 2023
15:17 PM
mayhem
lucifer: you about?
2023-06-06 15729, 2023
15:17 PM
mayhem
not sure how to do this, zas.
2023-06-06 15739, 2023
15:17 PM
lucifer
mayhem: hi yes.
2023-06-06 15747, 2023
15:17 PM
mayhem
^^
2023-06-06 15757, 2023
15:17 PM
mayhem
check to see if rabbitmq is happy now, please.
2023-06-06 15759, 2023
15:20 PM
lucifer
zas: i get 503 on beta.lb
2023-06-06 15709, 2023
15:21 PM
lucifer
2023-06-06 15743, 2023
15:21 PM
lucifer
container logs look fine though so rmq seems to be working fine.
2023-06-06 15746, 2023
15:22 PM
zas
503s are normal from the web, services are all down on gateways
2023-06-06 15754, 2023
15:23 PM
zas
atj: everything looks fine, can we proceed to the upgrade? (in order to avoid another downtime later)
2023-06-06 15712, 2023
15:24 PM
atj
ok, let me see how we can do this
2023-06-06 15728, 2023
15:24 PM
zas
bump the version, and run on 4 non-leaders servers first
2023-06-06 15755, 2023
15:24 PM
zas
rex-server is leader
2023-06-06 15725, 2023
15:25 PM
zas
the leader has to be done last
2023-06-06 15747, 2023
15:25 PM
zas
that's for servers, then we can deploy new clients
2023-06-06 15749, 2023
15:27 PM
mayhem
2023-06-06 15744, 2023
15:28 PM
zas
bitmap: can you check caa-indexer?
2023-06-06 15707, 2023
15:29 PM
bitmap
ok
2023-06-06 15752, 2023
15:29 PM
lucifer
mayhem: to answer the earlier question, this is MLHD+ only
2023-06-06 15724, 2023
15:30 PM
mayhem
ok, that data suggested that, thanks.
2023-06-06 15743, 2023
15:30 PM
bitmap
zas: started the caa-indexer container (it was stopped earlier) and uploads are succeeding, so looks fine
2023-06-06 15701, 2023
15:31 PM
zas
everything seems to work
2023-06-06 15708, 2023
15:31 PM
lucifer
monkey: no worries, i'll do some cleanups on the base branch first.
2023-06-06 15751, 2023
15:31 PM
zas
atj: I think we can do the upgrade, then restore services, we'll have a bit of post-move cleanup to do (mainly old containers to remove)
2023-06-06 15757, 2023
15:31 PM
monkey
Thank you, and sorry for fucking it up
2023-06-06 15715, 2023
15:34 PM
atj
zas: gimme a few minutes, i need to orchestrate this properly
2023-06-06 15733, 2023
15:34 PM
lucifer
monkey: i don't think it was anything on your end, rather i forced pushed the base branch a few and that messed it up.
2023-06-06 15717, 2023
15:40 PM
atj
zas: starting on mingus
2023-06-06 15728, 2023
15:40 PM
zas
ok
2023-06-06 15754, 2023
15:42 PM
zas
just noticed mingus' syslog is flooded by docker community container logs... weird
2023-06-06 15711, 2023
15:43 PM
zas
but unrelated to current move
2023-06-06 15755, 2023
15:44 PM
zas
mingus upgraded
2023-06-06 15702, 2023
15:46 PM
atj
yep
2023-06-06 15736, 2023
15:46 PM
atj
will do all except rex now
2023-06-06 15741, 2023
15:46 PM
zas
ok
2023-06-06 15753, 2023
15:46 PM
zas
one after the other
2023-06-06 15758, 2023
15:46 PM
atj
serial is set to 1
2023-06-06 15707, 2023
15:48 PM
atj
i tried taking a snapshot as it said in the guide but it tries to write to /tmp which is readonly
2023-06-06 15732, 2023
15:48 PM
zas
/tmp is readonly?
2023-06-06 15753, 2023
15:48 PM
atj
i locked the systemd service down
2023-06-06 15728, 2023
15:49 PM
zas
2023-06-06 15705, 2023
15:51 PM
atj
2023-06-06 15700, 2023
15:52 PM
atj
systemd sandboxing is very useful
2023-06-06 15706, 2023
15:52 PM
atj
ok, all done except for rex
2023-06-06 15723, 2023
15:52 PM
atj
zas: happy for me to proceed with rex?
2023-06-06 15727, 2023
15:52 PM
zas
yes
2023-06-06 15717, 2023
15:54 PM
atj
done
2023-06-06 15723, 2023
15:54 PM
zas
yup, confirmed
2023-06-06 15725, 2023
15:54 PM
atj
all servers showing 1.15.3
2023-06-06 15729, 2023
15:54 PM
zas
mingus is now leader
2023-06-06 15735, 2023
15:54 PM
atj
good for them
2023-06-06 15753, 2023
15:54 PM
zas
ok, now proceed to clients?
2023-06-06 15702, 2023
15:55 PM
atj
i think just restarting the service should be enough
2023-06-06 15717, 2023
15:55 PM
atj
it clearly issues a leave on orderly shutdown
2023-06-06 15740, 2023
15:55 PM
zas
stats with aphex and let's see
2023-06-06 15746, 2023
15:55 PM
atj
ok
2023-06-06 15746, 2023
15:55 PM
zas
*start
2023-06-06 15758, 2023
15:56 PM
atj
done
2023-06-06 15702, 2023
15:57 PM
atj
2023-06-06 15726, 2023
15:57 PM
atj
shows it issued a leave
2023-06-06 15758, 2023
15:57 PM
CatQuest
I didn't know there would be a server maintenance today..
2023-06-06 15758, 2023
15:57 PM
CatQuest
I was lucky i did some work in the morning :D
2023-06-06 15720, 2023
15:58 PM
zas
atj: looks good, we can proceed for the rest
2023-06-06 15708, 2023
15:59 PM
zas
2023-06-06 15750, 2023
15:59 PM
zas
not sure about this error, but consul logs are full of errors that should be warnings...
2023-06-06 15720, 2023
16:00 PM
zas
2023-06-06 15728, 2023
16:03 PM
zas
aretha left and rejoin, but still on 1.15.2?
2023-06-06 15753, 2023
16:03 PM
atj
yeah, my bad I thought I could do it with a oneshot
2023-06-06 15707, 2023
16:04 PM
atj
one second
2023-06-06 15738, 2023
16:06 PM
atj
ok here we go, aretha first
