#metabrainz

lucifer: I haven't tested with link with the same volume, I'm not sure the best way to do that

that should only be a small change, i'll prepare for that now.

thanks

then we should start adding commands to the doc

so let's do without link?

I think that's the safer more understood operation

👍

yes i'll upload all the files somewhere.

do you want to commit your dockerfile to an LB branch?

i was going to do a new repo because may come useful in future too.

thoughts on new repo in meb?

no problem

👍

I was just thinking - you're right that we can bring the old version up immediately if we leave the indexes, but if we delete them then that's less data to copy too :)

yeah true. let me see how big the indices are.

if it's more than 50gb, I think it'd be worth it

I'm going to double-check the final backup steps that we had (see how long a full no-listen no-spark backup takes)

👍

~almost 200G 😱

lucifer: from the logs on the 1st. I don't understand why there was an hour between "Creating dump of public data..." and "Creating dump of timescale public data..."

I guess that's the public postgres dump? what's in there?

so makes sense to drop indices then

alastairp: stats.

(timescale public data is mappings)

lucifer: ah, right. we don't need that, wrong database.

right. only need timescale data

maybe we just go for a pg_dump of the mappings again this time

playlists too (private timescale dumps)

yep

lucifer: oh - we only need to recreate indexes which have a field which is of type `text`

so for example listened_at_user_id_ndx_listen and listen_listened_at_idx isn't needed

there's also 1 on user-name but that we can drop

yeah, did we not drop the username col yet?

*drop permanently

agreed

not yet

we should probably look into using INCLUDE in our indexes later

but yeah - listened_at_username and the msid one will need to be remade

what's that?

i had no idea our indices were so huge.

do you know how long these indexes take to make?

https://www.postgresql.org/docs/11/sql-createin... INCLUDE clause

it was about 10 mins when you made the user id ones when we did that migration, right?

i think so yes, less than 30 mins for sure.

oh neat, I didn't know about INCLUDE. Could be nice to investigate

right, but that's just 1 index out of 3 or 4 which we need to do

we can add a time estimate anyway

(to the doc)

yeah makes sense

a dump backup to gaga is going to take about 40 minutes, do we want to save 10 minutes by dropping indexes first, or leave them there so we can bring it up in a hurry if needed (if we end up restoring from this backup we'd need to recreate indexes)

for saving only 5-10 mins I'm tempted to leave them there?

I started adding steps for the pg_ugprade to the doc, please sanity check

alastair, i'd say let's drop indexes just before pg_upgrade. so take backup with indexes.

agreed, thanks

alastairp: https://github.com/metabrainz/timescale-upgrade

lucifer: recording_msid_ndx_listen doesn't appear to be in the create_indexes.sql, can you confirm?

lucifer: awesome, thanks

https://github.com/metabrainz/listenbrainz-serv... ?

yes, that one :/

maybe I'm in an old branch, mine doesn't look like that!

oh ok

when you ran initdb did you need to chown the data directory first?

yes i did that in Dockerfile

spent a good chunk of time fiddling with docker magic.

only works if you do chown before VOLUME command. otherwise gets overwritten 😞

ah yes, volumes

oh, I see you set these envs in the dockerfile. I was just going to ask that

~/win go #linguistics

please update my commands with your updated versions in the doc, then

yeah volume too gave a hard time. i mounted empty volume to container at pg 13 so it instead copied container stuff to volume.

Clint: I'm sorry dave, I can't allow you to do that

I was trying to come up with some linguistical joke

there's still time

but what does time really mean?

what do we want to name the new volume alastairp, need to update docker-server-configs script accordingly

is the meaning of words linguistics? I guess not

semantics is a subfield of linguistics

well, I disagree

(see what I did there?)

lucifer: maybe add -pg13 instead of -13?

listenbrainz-timescale-pg13-data

I can make a branch for docker-server-configs with these changes if you want

atj: hah, the compressed data is only 150gb!

in fact, in that case we don't really even need to ssh it to another machine, plenty of free space in gaga

linguinistics

alastairp: wow, that's even better than I expected

actually alastairp should recordings show links toreviews? eg https://beta.musicbrainz.org/recording/cfa03134... has one by monkey

alastairp: cool, i'll open a PR.

reosarevok: the constraint thing sounds concerning, I'll take a look before you fix them

[listenbrainz-server] 14amCap1712 opened pull request #1896 (03master…timescale-down): Remove timescale service from consul config for upgrade https://github.com/metabrainz/listenbrainz-serv...

CatQuest: can you save that question for in about 3 hours?

just finished lunch

lucifer: what do you need from me?

hey I'm going to do dishes and start dinner myself so sure? :D

zas: atj: what maintenance do you need to do on gaga, approx how long do you think, and will you need to reboot?

alastairp: can you review docker-sever-configs PR and sanity check that the volume is fine or we need to make change to the script?

we're going to start managing the server using ansible, this is the first we've done it to an already configured system, so it could throw some issues up

will we do it at the beginning of downtime or the end?

however I would say 60-90 minutes at most

(if you don't need to reboot, we can try and make use of parallelism and I'll make the db backup at the same time)

alastairp: you don't have to tell me that you will talk about it in 3 hours, jsut talk about it in 3 hours :)

but apreciate it nevertheless

a reboot would be good to ensure everything comes back as expected

however I will defer to zas on that

CatQuest: we're about to do a scary database upgrade, so I don't want to think about other things for now, but I'm sure I'll forget about you when we're done, so it'd be great if you could ask again later tonight if you want an answer for your question

atj: ok, thanks

I think it's probably best you do the upgrade first

lucifer: looking at PR

sure thing! good luck with the database thing!

alastairp: volume creation at the start. bring up new container near the end (after PG before before TS upgrade)

actualy

good luck everyone ✨

lucifer: is the name field needed anywhere in consul?

oh right, in LB config.

{{if service "timescale-listenbrainz"}}

reosarevok: so they are all pre-ngs or predate the control_for_whitespace constraints, and I guess it makes sense if the constraints were created with NOT VALID. but I wonder how nobody ran into this or reported it before?

I don't have a problem just editing the volume create command and the name of the volume in the docker run command

should be fine.

reosarevok: surely some people are using RT_STANDALONE, even I used to import normal dumps with RT_STANDALONE before we had the sample dumps, so I'm confused how I never saw that

alastairp, what about the postgres health check thingy? we had issues with it last time iirc.

lucifer: oh, neat. although again - I'm not sure I like the idea of having the container name and service name different

its different already though. listenbrainz_timescale vs timescale-listenbrainz. but yes fine to with modifying the script

listenbrainz_timescale ? where's that from?

listenbrainz-timescale sorry.

so listenbrainz-timescale vs timescale-listenbrainz

oh haha, I see what you mean

good catch!

👍

just finishing dinner, will be back in ~10 mins for upgrade.

ok!

oh duh forgot dst again. we have an hour. will take my time then lol

lucifer: no, we have 10 mins :)

oh duh again.

taking down test & beta containers

lucifer: is the LB container for taking down listenstore ready?

lucifer: alastairp running late, sorry.

mayhem: no prob

zas: atj: ping, just making sure that you're here

alastairp: not yet, does this look fine? https://github.com/metabrainz/listenbrainz-serv... will build then