lucifer: I haven't tested with link with the same volume, I'm not sure the best way to do that
2022-03-09 06808, 2022
lucifer
that should only be a small change, i'll prepare for that now.
2022-03-09 06815, 2022
alastairp
thanks
2022-03-09 06821, 2022
alastairp
then we should start adding commands to the doc
2022-03-09 06829, 2022
lucifer
so let's do without link?
2022-03-09 06842, 2022
alastairp
I think that's the safer more understood operation
2022-03-09 06850, 2022
lucifer
👍
2022-03-09 06800, 2022
lucifer
yes i'll upload all the files somewhere.
2022-03-09 06802, 2022
alastairp
do you want to commit your dockerfile to an LB branch?
2022-03-09 06857, 2022
lucifer
i was going to do a new repo because may come useful in future too.
2022-03-09 06849, 2022
lucifer
thoughts on new repo in meb?
2022-03-09 06857, 2022
alastairp
no problem
2022-03-09 06811, 2022
lucifer
👍
2022-03-09 06844, 2022
alastairp
I was just thinking - you're right that we can bring the old version up immediately if we leave the indexes, but if we delete them then that's less data to copy too :)
2022-03-09 06821, 2022
lucifer
yeah true. let me see how big the indices are.
2022-03-09 06837, 2022
alastairp
if it's more than 50gb, I think it'd be worth it
2022-03-09 06844, 2022
alastairp
I'm going to double-check the final backup steps that we had (see how long a full no-listen no-spark backup takes)
lucifer: from the logs on the 1st. I don't understand why there was an hour between "Creating dump of public data..." and "Creating dump of timescale public data..."
2022-03-09 06815, 2022
alastairp
I guess that's the public postgres dump? what's in there?
2022-03-09 06817, 2022
lucifer
so makes sense to drop indices then
2022-03-09 06829, 2022
lucifer
alastairp: stats.
2022-03-09 06829, 2022
alastairp
(timescale public data is mappings)
2022-03-09 06853, 2022
alastairp
lucifer: ah, right. we don't need that, wrong database.
2022-03-09 06805, 2022
lucifer
right. only need timescale data
2022-03-09 06808, 2022
alastairp
maybe we just go for a pg_dump of the mappings again this time
2022-03-09 06823, 2022
lucifer
playlists too (private timescale dumps)
2022-03-09 06816, 2022
alastairp
yep
2022-03-09 06836, 2022
alastairp
lucifer: oh - we only need to recreate indexes which have a field which is of type `text`
it was about 10 mins when you made the user id ones when we did that migration, right?
2022-03-09 06834, 2022
lucifer
i think so yes, less than 30 mins for sure.
2022-03-09 06843, 2022
alastairp
oh neat, I didn't know about INCLUDE. Could be nice to investigate
2022-03-09 06802, 2022
alastairp
right, but that's just 1 index out of 3 or 4 which we need to do
2022-03-09 06808, 2022
alastairp
we can add a time estimate anyway
2022-03-09 06816, 2022
alastairp
(to the doc)
2022-03-09 06819, 2022
lucifer
yeah makes sense
2022-03-09 06831, 2022
alastairp
a dump backup to gaga is going to take about 40 minutes, do we want to save 10 minutes by dropping indexes first, or leave them there so we can bring it up in a hurry if needed (if we end up restoring from this backup we'd need to recreate indexes)
2022-03-09 06847, 2022
alastairp
for saving only 5-10 mins I'm tempted to leave them there?
2022-03-09 06835, 2022
alastairp
I started adding steps for the pg_ugprade to the doc, please sanity check
2022-03-09 06832, 2022
lucifer
alastair, i'd say let's drop indexes just before pg_upgrade. so take backup with indexes.
CatQuest: can you save that question for in about 3 hours?
2022-03-09 06843, 2022
alastairp
just finished lunch
2022-03-09 06848, 2022
alastairp
lucifer: what do you need from me?
2022-03-09 06803, 2022
skelly37 has quit
2022-03-09 06803, 2022
CatQuest
hey I'm going to do dishes and start dinner myself so sure? :D
2022-03-09 06815, 2022
alastairp
zas: atj: what maintenance do you need to do on gaga, approx how long do you think, and will you need to reboot?
2022-03-09 06822, 2022
lucifer
alastairp: can you review docker-sever-configs PR and sanity check that the volume is fine or we need to make change to the script?
2022-03-09 06829, 2022
atj
we're going to start managing the server using ansible, this is the first we've done it to an already configured system, so it could throw some issues up
2022-03-09 06836, 2022
skelly37 joined the channel
2022-03-09 06847, 2022
alastairp
will we do it at the beginning of downtime or the end?
2022-03-09 06850, 2022
atj
however I would say 60-90 minutes at most
2022-03-09 06854, 2022
alastairp
(if you don't need to reboot, we can try and make use of parallelism and I'll make the db backup at the same time)
2022-03-09 06801, 2022
CatQuest
alastairp: you don't have to tell me that you will talk about it in 3 hours, jsut talk about it in 3 hours :)
2022-03-09 06813, 2022
CatQuest
but apreciate it nevertheless
2022-03-09 06828, 2022
atj
a reboot would be good to ensure everything comes back as expected
2022-03-09 06841, 2022
atj
however I will defer to zas on that
2022-03-09 06807, 2022
alastairp
CatQuest: we're about to do a scary database upgrade, so I don't want to think about other things for now, but I'm sure I'll forget about you when we're done, so it'd be great if you could ask again later tonight if you want an answer for your question
2022-03-09 06810, 2022
alastairp
atj: ok, thanks
2022-03-09 06823, 2022
atj
I think it's probably best you do the upgrade first
2022-03-09 06823, 2022
alastairp
lucifer: looking at PR
2022-03-09 06831, 2022
CatQuest
sure thing! good luck with the database thing!
2022-03-09 06838, 2022
lucifer
alastairp: volume creation at the start. bring up new container near the end (after PG before before TS upgrade)
2022-03-09 06828, 2022
CatQuest
actualy
2022-03-09 06846, 2022
CatQuest
good luck everyone ✨
2022-03-09 06810, 2022
alastairp
lucifer: is the name field needed anywhere in consul?
2022-03-09 06834, 2022
lucifer
oh right, in LB config.
2022-03-09 06840, 2022
alastairp
{{if service "timescale-listenbrainz"}}
2022-03-09 06808, 2022
bitmap
reosarevok: so they are all pre-ngs or predate the control_for_whitespace constraints, and I guess it makes sense if the constraints were created with NOT VALID. but I wonder how nobody ran into this or reported it before?
2022-03-09 06811, 2022
alastairp
I don't have a problem just editing the volume create command and the name of the volume in the docker run command
reosarevok: surely some people are using RT_STANDALONE, even I used to import normal dumps with RT_STANDALONE before we had the sample dumps, so I'm confused how I never saw that
2022-03-09 06822, 2022
lucifer
alastairp, what about the postgres health check thingy? we had issues with it last time iirc.
2022-03-09 06830, 2022
alastairp
lucifer: oh, neat. although again - I'm not sure I like the idea of having the container name and service name different
2022-03-09 06815, 2022
lucifer
its different already though. listenbrainz_timescale vs timescale-listenbrainz. but yes fine to with modifying the script
2022-03-09 06840, 2022
alastairp
listenbrainz_timescale ? where's that from?
2022-03-09 06833, 2022
lucifer
listenbrainz-timescale sorry.
2022-03-09 06847, 2022
lucifer
so listenbrainz-timescale vs timescale-listenbrainz
2022-03-09 06849, 2022
alastairp
oh haha, I see what you mean
2022-03-09 06851, 2022
alastairp
good catch!
2022-03-09 06856, 2022
alastairp
ok, your fix is good then ;)
2022-03-09 06834, 2022
lucifer
👍
2022-03-09 06811, 2022
lucifer
just finishing dinner, will be back in ~10 mins for upgrade.
2022-03-09 06831, 2022
alastairp
ok!
2022-03-09 06840, 2022
lucifer
oh duh forgot dst again. we have an hour. will take my time then lol
2022-03-09 06853, 2022
alastairp
lucifer: no, we have 10 mins :)
2022-03-09 06837, 2022
lucifer
oh duh again.
2022-03-09 06855, 2022
alastairp
taking down test & beta containers
2022-03-09 06800, 2022
alastairp
lucifer: is the LB container for taking down listenstore ready?
