#metabrainz

zas: Whenever you have a moment: https://community.metabrainz.org/t/discourse-do... :)

but otherwise also lot of differences in conf.

some of those are commented so don't matter but i guess best to copy over the .conf file from old to new cluster?

yeah this also does it https://github.com/metabrainz/docker-postgres-c...

lucifer: yeah, I'm doing this process as well, and it looks like I'm at about the same stage as you. let me look at this diff. I almost agree that we should just copy it, though we should also verify that there are no new options that arrived in 11 or 12

cool, I also see the update_extensions.sql message. this just has `ALTER EXTENSION "timescaledb" UPDATE;` in it, which is great - because that's what we need to do anyway, right?

yes

and I see the message about re-running vaccum/analyze, so that seems good

were you able to start up the new cluster?

not yet. trying to do that currently.

alastairp, my container has closed connection so i had to start another timescale one. yup works for me as well.

i ran alter extension command but got error that it already exists and which probably makes sense.

running vacuum, errors huh.

ERROR: could not resize shared memory segment "/PostgreSQL.175590062" to 67128672 bytes: No space left on device

well, that error is understandable :)

ah, but I see that bono has plenty of space

are you using the pre-generated vaccum script?

/dev/md2 4.0T 1.8T 2.0T 49% /

yeah. no not pre-generated script.

just vacuum in session while connected to db

it worked when i ran as postgres user.

Hi folks, in the last few days I experienced some issues related to scrobbling on LB. What I send to LB is simply not listed on my profile (I'm pretty sure it's not an error on my side)

I've checked the Bug Tracker and there is noting related to this issue (at least not in the last few weeks)

Is it a known problem or should I open a ticket?

Lorenzo[m]: oh hi!

Lorenzo[m]: we did in fact have another report from another person here, and today we're going to do an upgrade of our database to help us fix this issue, so fingers crossed this will fix your problem too

Oh nice, I'll try to scrobble some music tomorow and I'll check if everything is fixed

Thank you for your time folks, I really appreciate the project and your efforts

Lorenzo[m]: <3

lucifer: oh, I just had a thought. if we don't use --link, then pg_upgrade is going to copy all data files anyway. it's going to take longer (need to copy all files), but it's not going to touch the old ones, so maybe we can get by without doing a backup?

alternatively, we make a network backup to another server, we could start with rsync now, and then re-run it to catch up modified files once we take the cluster down

yes right.

i think MB did rsync last time.

right, but is that just copying the data directory to the replica for quicker startup?

800gb is going to take 2 hours to copy somewhere else over gigabit at least

oh ok, makes sense.

alastairp: i am unsure which is better. your call.

zas: do we have a server with 900gb free space that we can rsync to?

postgres data file compress really well btw

*data files

atj: ah, interesting, might try that

although it depends on the contents

atj: let me get you up to date - we're doing a pg 11 to 13 upgrade. because we run pg in a docker container, we have the data in a volume. however, if we mount 2 volumes (1 for 11-data and 1 for 13-data) we can't hardlink between them with pg_upgrade, because they're different logical disks :(

makes sense

alastairp: kiss has 1.07Tb free on /dev/md2

so now we're wondering about doing the copy version of the upgrade, but we'd like enough disk space (on gaga) to have 1) the v11 data, 2) a backup of it in case anything goes wrong, 3) the v13 data - but gaga doesnt have enough disk for this (db is about 770gb)

zas: yes, I was just looking through servers and found that, thanks

atj: so - if you have any postgres upgrade and/or docker/volume/hardlink experience, it'd be interesting to hear your thoughts

lucifer: did you try the upgrade with --link with both data directories in the same volume?

well, hardlinks definitely aren't going to work unless you're in the same volume

alastairp: no haven't done yet.

reosarevok: Not sure if you missed this: https://community.metabrainz.org/t/announcement...

atj: I had hoped that because these volumes were on the same partition on the host, it'd just work :)

Freso: iirc I transcluded but I guess I didn't answer?

alastairp: have you tried creating manual hardlinks?

it would work from the host, but I don't think the container can see that they're the same filesystem

lucifer: oh, one other thing - we should decide if we want to keep running on pg/timescale on debian, and if so decide which base image, how to install it, and how to start it (because the debian version splits the config/data directories, but our alpine data dir has the combined directory)

atj: should we disable IPv6 on shorewall for now?

atj: on the host or in the container? this is done by pg_upgrade (in the container), so I don't think I can do it myself on the host and have it work

zas: I think so yes :/

the role is nearly done, just working on it now

lucifer: actually, https://github.com/metabrainz/docker-postgres-c... is basically good for us, I think it'd be a good idea to base it off of this

alastairp: does the container have a shell? can you exec in and touch a file on one volume then try to hardlink it to the other volume?

eg. docker exec -ti <container> bash

worth a try but as expected

https://bbs.archlinux.org/viewtopic.php?id=241866 seems to indicate that overlay2 causes it, and there are workarounds like using devicemapper as the storage driver

but not something I want to get into now

my suggestion would be to create a compressed backup using tar and send it to another machine via ssh

best to have a backup regardless

any suggestions based on time tradeoff between just copying 700gb, or waiting for it to compress first and then copying it?

compression will be worth it IMO

bzip or gzip?

I'd recommend using lzip for better speed

or lzip!

needs installing

installed on gaga

what's the tar flag for lzip?

alastairp: also, use pv so you can see the speed of the backup

--lzip

tar | pv | lzip, to be able to measure against the real size of the data directory?

(or is that an extraction parameter)

tar --lzip -cf - /path |pv |ssh foo "cat > file.tar.lzip"

thanks!

I'd run the command from the root of the volume you are backing up

ah, so you directly stream the compressed file, rather than write to file then copy

just so the archive has relative paths - easier to unpack

exactly

but pv is going to be reporting the compressed size, right?

so we'll expect it to be smaller than the size of the data dir, but we don't know how much smaller

you can't know until you do it unfortunately

you could try compressing a few files to see

yeah, which is why I was thinking of tar -cf - /path | pv | lzip | ssh

that should work

you just won't know the transfer speed over ssh

let me try it on another machine I have

yeah, hopefully close to gige less overhead, but unclear

reosarevok: Has the old docker volume staticbrainz-data been replaced with musicbrainz-static-build-prod and musicbrainz-static-build-beta?

alastairp: you might want to try lzma vs. lzip, I always get confused between these various lz compression algos

atj: hmm, adding lzma in the middle makes it 100x _slower_

CPU limited?

lzma: [1.27MiB/s], no compression: [ 110MiB/s], gzip: [21.5MiB/s]

checking now

yes, 100% cpu

lzip?

other option is tar -> file, pbzip2, copy file, but there's no parallelism there

zstd (yes another algo) supports multiple threads

even with input from stdin?

this is what I'm wondering

I think these more advanced algos have large window sizes which requires more buffering in RAM

lzip -0 says that it's about as fast as gzip, and I'm seeing speeds about the same, but it's still CPU-bound rather than network-bound

not sure how much difference letting tar do the compression would make

2 minutes to copy a 2gb file, compared to 20sec using just rsync

does hetzner offer 10ge yet? :)

trying that now

no, that's just as slow

alastairp: on my system "tar --zstd -cf -"

1.46GiB 0:00:17 [83.3MiB/s]

lzma and lzip were very slow

yes, zstd is 3x faster than lzip or gzip immediately

no compression: [ 110MiB/s], pv|zstd: [ 123MiB/s]

so yes, zstd is giving a slight advantage

I just compressed a 4.7GB VDI to 1.9GB at pretty much line speed

looks like that's your best bet, whether the compression is worth it I don't know.

alastairp: the MB docker setup is definitely useful for us to base on but not sure what all we need to test and prepare for that move so probably best to not do it now?

oh yeah - I'm just doing a test with the pg dir now, it's hovering around 300MiB/sec with zstd (measuring with pv before going into zstd), so we're about 3x faster than just a regular copy over the same network link

thanks atj!

glad it worked :)

still 40 mins to do the backup

make sure to check the tar archive once it's completed - "tar --zstd -tf filename.tar.zstd"

actually, tar does recognise the file extension so you can do "tar -vtf <filename>.tar.zstd"

lucifer: yeah, that's a good point. so maybe we drop the indexes in current image, bring it up in the migrate image, do the upgrade, bring it up in the new timescale image, recreate indexes ?

lucifer: I had a look at the config file diff, there are some things I don't understand, but I think it'll be OK to copy directly in place

alastairp: are we going to do with --link or without? for without, we could keep the indexes and drop/recreate in the new image in case we need to bring up old cluster back up.

lucifer: I know you have a few items on the migration doc for preparing listenstore downtime, do you need time for that?