not anymore? like we can use it for something else?
2023-08-01 21357, 2023
monkey
I tweeted for BB earlier, forgot to mention it
2023-08-01 21320, 2023
zas
rebooting zappa
2023-08-01 21331, 2023
mayhem
zas: quest has been yours for weeks now.
2023-08-01 21346, 2023
mayhem
I remember lucifer passing it to you. you don't want it?
2023-08-01 21318, 2023
zas
I do, but I thought it was still be used, we'll reuse it soon
2023-08-01 21340, 2023
mayhem
gah, how can we handle these pass better?
2023-08-01 21349, 2023
mayhem
wasted a month or two of rental fees.
2023-08-01 21346, 2023
zas
we need to have a good way to track uses, and activity
2023-08-01 21306, 2023
mayhem
5 minutes to downtime. are we ready?
2023-08-01 21327, 2023
mayhem
do we have a google doc to coordinate us?
2023-08-01 21317, 2023
bitmap
zas: is patton down?
2023-08-01 21329, 2023
zas
it shouldn't
2023-08-01 21301, 2023
zas
it pings, but I can't ssh to it
2023-08-01 21305, 2023
bitmap
hmm I couldn't either but now I can
2023-08-01 21310, 2023
bitmap
just got in
2023-08-01 21320, 2023
zas
me too, weird
2023-08-01 21348, 2023
bitmap
I got logged out though
2023-08-01 21355, 2023
bitmap
Connection reset by peer
2023-08-01 21313, 2023
zas
it looks like it has connectivity issues atm
2023-08-01 21346, 2023
zas
mayhem: no google doc for that, but basically we just set services down in docker-server-configs and then iterate through servers to reboot them
2023-08-01 21324, 2023
zas
bitmap: can you mark services as down so we can proceed? It's time
2023-08-01 21353, 2023
bitmap
yes
2023-08-01 21344, 2023
zas
patton issue seems unrelated to what we did til now, I'll look into this if it persists, it seems a transient network issue to me
2023-08-01 21349, 2023
bitmap
done
2023-08-01 21325, 2023
zas
aretha?
2023-08-01 21358, 2023
bitmap
aretha ready
2023-08-01 21317, 2023
bitmap
i'll work on shutting down pg primary
2023-08-01 21336, 2023
zas
postgres-aretha still running, is that ok?
2023-08-01 21323, 2023
zas
rebooting (redundant) rex atm
2023-08-01 21300, 2023
bitmap
yeah I stopped the pg process inside the container
2023-08-01 21309, 2023
zas
ok, rebooting aretha
2023-08-01 21329, 2023
BrainzGit has quit
2023-08-01 21346, 2023
monkey
I don't have to run away early anymor, so no need to do BB first
2023-08-01 21306, 2023
bitmap
do you have to reboot jimmy too?
2023-08-01 21359, 2023
zas
bitmap: yes, but we can skip it for now
2023-08-01 21325, 2023
bitmap
ok
2023-08-01 21326, 2023
zas
hmmm aretha is slow to come back, and rex failed for some reason
2023-08-01 21338, 2023
zas
aretha is likely checking drives
2023-08-01 21354, 2023
bitmap
I stopped pg process on pink & floyd
2023-08-01 21329, 2023
bitmap
(and jimmy)
2023-08-01 21339, 2023
zas
we can reboot pink first then
2023-08-01 21346, 2023
bitmap
sure
2023-08-01 21340, 2023
zas
res is pinging, but I can't ssh
2023-08-01 21347, 2023
BrainzGit joined the channel
2023-08-01 21350, 2023
zas
bitmap: can you try ssh to rex
2023-08-01 21352, 2023
zas
?
2023-08-01 21302, 2023
bitmap
it works for me
2023-08-01 21318, 2023
zas
weird
2023-08-01 21334, 2023
zas
I can ssh to it from rudi, but not from home, I'll debug that later
2023-08-01 21341, 2023
zas
aretha is back
2023-08-01 21308, 2023
zas
rebooting pink
2023-08-01 21356, 2023
bitmap
started pg and json dumps cron back up on aretha
2023-08-01 21305, 2023
zas
hmmm, rex isn't answering on IPv4 it seems
2023-08-01 21352, 2023
zas
pink is back
2023-08-01 21359, 2023
zas
can I do floyd?
2023-08-01 21340, 2023
bitmap
yes
2023-08-01 21305, 2023
zas
I reboot rex again, I think we had this issue already
2023-08-01 21307, 2023
bitmap
pg on pink is back (it errors trying to connect to floyd, but normal)
2023-08-01 21340, 2023
zas
rebooting floyd
2023-08-01 21303, 2023
zas
floyd is back
2023-08-01 21338, 2023
bitmap
pg is back on floyd though there were some errors archiving wal files to barman since the barman container didn't start back up automatically after aretha reboot for some reason
2023-08-01 21352, 2023
bitmap
it should retry later and work fine
2023-08-01 21339, 2023
zas
ok
2023-08-01 21347, 2023
zas
let's do clash now
2023-08-01 21301, 2023
soundandvision joined the channel
2023-08-01 21314, 2023
zas
monkey: ^^
2023-08-01 21330, 2023
bitmap
pink streaming replication from floyd is ok
2023-08-01 21354, 2023
bitmap
i'll disable service maintenance on all the pg services
2023-08-01 21315, 2023
zas
bitmap: when you tested connection to rex it was on IPv6 right?
2023-08-01 21331, 2023
zas
can you ping4 it?
2023-08-01 21341, 2023
zas
something is messed up with it
2023-08-01 21300, 2023
zas
We'll not reboot rudi until I sorted that
2023-08-01 21313, 2023
zas
I'll reboot flobot
2023-08-01 21338, 2023
bitmap
ipv4 ping to rex works for me
2023-08-01 21332, 2023
atj
rex ipv4 ping+ssh WFM
2023-08-01 21327, 2023
zas
ok, thanks
2023-08-01 21320, 2023
zas
remaining servers: clash gaga kiss patton serge trille wolf
2023-08-01 21326, 2023
zas
I can connect to rex now... (made no change)
2023-08-01 21355, 2023
zas
mayhem, lucifer: can we reboot kiss?
2023-08-01 21332, 2023
lucifer
zas: yes
2023-08-01 21336, 2023
zas
bitmap: ok to reboot patton? It has redis instances on it
2023-08-01 21345, 2023
zas
lucifer: ok, thanks, I will proceed
2023-08-01 21355, 2023
bitmap
yep, they shouldn't be in use rn
2023-08-01 21321, 2023
zas
ok, rebooting then
2023-08-01 21326, 2023
zas
bitmap: next serge & trille
2023-08-01 21350, 2023
bitmap
ok
2023-08-01 21342, 2023
zas
lucifer: can we reboot gaga after kiss?
2023-08-01 21347, 2023
bitmap
I already stopped the production cron service on trille, I don't think anything else MB-related needs attention on those
2023-08-01 21321, 2023
lucifer
zas: yes.
2023-08-01 21337, 2023
zas
kiss & patton are slow to come back
2023-08-01 21349, 2023
zas
kiss is back now
2023-08-01 21313, 2023
zas
rebooting gaga
2023-08-01 21326, 2023
zas
patton is back
2023-08-01 21306, 2023
zas
bitmap: I'll do trille now, please check serge
2023-08-01 21346, 2023
bitmap
serge looks ready to go
2023-08-01 21308, 2023
lucifer
zas: is gaga back?
2023-08-01 21323, 2023
zas
yes
2023-08-01 21329, 2023
zas
now it is, it was a bit long
2023-08-01 21349, 2023
zas
trille is back too
2023-08-01 21353, 2023
zas
doing serge now
2023-08-01 21324, 2023
zas
monkey: can we reboot clash?
2023-08-01 21331, 2023
monkey
Absolutely
2023-08-01 21349, 2023
zas
ok, rebooting clash
2023-08-01 21300, 2023
zas
after serge is back, we're done I think (mingus left, but that's discourse, I'll reboot it later)
2023-08-01 21328, 2023
zas
waiting for serge & clash
2023-08-01 21347, 2023
bitmap
you can reboot jimmy too, but it can be done after services are back
2023-08-01 21350, 2023
zas
serge is back
2023-08-01 21307, 2023
zas
I'll reboot jimmy right now
2023-08-01 21341, 2023
zas
clash is back
2023-08-01 21358, 2023
bitmap
is that everything? should I bring services back up?
2023-08-01 21326, 2023
zas
I think so
2023-08-01 21339, 2023
bitmap
ok, done
2023-08-01 21308, 2023
bitmap
right in time tbh
2023-08-01 21318, 2023
zas
ok, let's stay around for a while in case issues appear
2023-08-01 21306, 2023
zas
jimmy didn't come back yet, weird
2023-08-01 21347, 2023
bitmap
yea I was just checking it
2023-08-01 21357, 2023
zas
it's back now
2023-08-01 21357, 2023
bitmap
now it's back
2023-08-01 21330, 2023
reosarevok
back now it's
2023-08-01 21345, 2023
mayhem
baby's got back?
2023-08-01 21358, 2023
zas
:D
2023-08-01 21320, 2023
zas
we still have few alerts on, but they should be cleared within next minutes
2023-08-01 21331, 2023
zas
apparently everything is back to normal
2023-08-01 21354, 2023
reosarevok
Should I tweet?
2023-08-01 21314, 2023
zas
Yes, you can
2023-08-01 21336, 2023
reosarevok
Done
2023-08-01 21305, 2023
zas
Thanks, guys. Those reboot sessions are quite annoying, but at least the docker network issue should be solved now (plus some security kernel fixes applied)
2023-08-01 21321, 2023
monkey
BookBrainz isn't back up, looking into it now
2023-08-01 21358, 2023
zas
monkey: we need to work on alerts for it
2023-08-01 21313, 2023
soundandvision has quit
2023-08-01 21356, 2023
monkey
I see what the issue is, we've got an empty section in the configuration file, supposedly because the elasticsearch services weren't up fast enough and consul just omitted that section of the config
2023-08-01 21316, 2023
monkey
BookBrainz back online
2023-08-01 21323, 2023
monkey
I've created an issue in JIRA for that issue
2023-08-01 21310, 2023
vscode_
Hi riksucks: and Lucifer: I have two questions
2023-08-01 21337, 2023
vscode_
1. Apple Music API endpoints require storefront parameter while querying, should we query all artists/albums with each storefront or any one storefront is fine?
2023-08-01 21337, 2023
vscode_
2. I am currently working on it as a standalone script. How should the script be integrated into ListenBrainz?
2023-08-01 21330, 2023
Pratha-Fish
hey reosarevok sorry missed you ping yesterday :D
2023-08-01 21333, 2023
Pratha-Fish
rebasing rn
2023-08-01 21320, 2023
Pratha-Fish
also, I am about 2 decades late, but I finally got wifi. No more excuses :skul
Soo do you want the branch test_add_area to reflect all the previous changes from master, or do you want master to just have all the updates from test_add_area?
2023-08-01 21323, 2023
Pratha-Fish
If it's the first one, the test_add_area is already derived from the up-to-date master branch (I've also force pushed the changes to the PR)
