lucifer: yeah, I'm doing this process as well, and it looks like I'm at about the same stage as you. let me look at this diff. I almost agree that we should just copy it, though we should also verify that there are no new options that arrived in 11 or 12
2022-03-09 06834, 2022
alastairp
cool, I also see the update_extensions.sql message. this just has `ALTER EXTENSION "timescaledb" UPDATE;` in it, which is great - because that's what we need to do anyway, right?
2022-03-09 06851, 2022
lucifer
yes
2022-03-09 06809, 2022
alastairp
and I see the message about re-running vaccum/analyze, so that seems good
alastairp, my container has closed connection so i had to start another timescale one. yup works for me as well.
2022-03-09 06825, 2022
lucifer
i ran alter extension command but got error that it already exists and which probably makes sense.
2022-03-09 06855, 2022
lucifer
running vacuum, errors huh.
2022-03-09 06857, 2022
lucifer
ERROR: could not resize shared memory segment "/PostgreSQL.175590062" to 67128672 bytes: No space left on device
2022-03-09 06810, 2022
alastairp
well, that error is understandable :)
2022-03-09 06834, 2022
alastairp
ah, but I see that bono has plenty of space
2022-03-09 06842, 2022
alastairp
are you using the pre-generated vaccum script?
2022-03-09 06848, 2022
lucifer
/dev/md2 4.0T 1.8T 2.0T 49% /
2022-03-09 06803, 2022
lucifer
yeah. no not pre-generated script.
2022-03-09 06819, 2022
lucifer
just vacuum in session while connected to db
2022-03-09 06831, 2022
lucifer
it worked when i ran as postgres user.
2022-03-09 06811, 2022
Lorenzo[m] joined the channel
2022-03-09 06828, 2022
Lorenzo[m]
Hi folks, in the last few days I experienced some issues related to scrobbling on LB. What I send to LB is simply not listed on my profile (I'm pretty sure it's not an error on my side)
2022-03-09 06830, 2022
Lorenzo[m]
I've checked the Bug Tracker and there is noting related to this issue (at least not in the last few weeks)
2022-03-09 06839, 2022
Lorenzo[m]
Is it a known problem or should I open a ticket?
2022-03-09 06818, 2022
alastairp
Lorenzo[m]: oh hi!
2022-03-09 06830, 2022
alastairp
Lorenzo[m]: we did in fact have another report from another person here, and today we're going to do an upgrade of our database to help us fix this issue, so fingers crossed this will fix your problem too
2022-03-09 06851, 2022
Lorenzo[m]
Oh nice, I'll try to scrobble some music tomorow and I'll check if everything is fixed
2022-03-09 06836, 2022
Lorenzo[m]
Thank you for your time folks, I really appreciate the project and your efforts
2022-03-09 06854, 2022
atj
Lorenzo[m]: <3
2022-03-09 06854, 2022
alastairp
lucifer: oh, I just had a thought. if we don't use --link, then pg_upgrade is going to copy all data files anyway. it's going to take longer (need to copy all files), but it's not going to touch the old ones, so maybe we can get by without doing a backup?
2022-03-09 06824, 2022
alastairp
alternatively, we make a network backup to another server, we could start with rsync now, and then re-run it to catch up modified files once we take the cluster down
right, but is that just copying the data directory to the replica for quicker startup?
2022-03-09 06805, 2022
alastairp
800gb is going to take 2 hours to copy somewhere else over gigabit at least
2022-03-09 06811, 2022
lucifer
oh ok, makes sense.
2022-03-09 06859, 2022
lucifer
alastairp: i am unsure which is better. your call.
2022-03-09 06845, 2022
alastairp
zas: do we have a server with 900gb free space that we can rsync to?
2022-03-09 06837, 2022
v6lur joined the channel
2022-03-09 06849, 2022
atj
postgres data file compress really well btw
2022-03-09 06856, 2022
atj
*data files
2022-03-09 06820, 2022
alastairp
atj: ah, interesting, might try that
2022-03-09 06834, 2022
atj
although it depends on the contents
2022-03-09 06814, 2022
alastairp
atj: let me get you up to date - we're doing a pg 11 to 13 upgrade. because we run pg in a docker container, we have the data in a volume. however, if we mount 2 volumes (1 for 11-data and 1 for 13-data) we can't hardlink between them with pg_upgrade, because they're different logical disks :(
2022-03-09 06835, 2022
atj
makes sense
2022-03-09 06801, 2022
zas
alastairp: kiss has 1.07Tb free on /dev/md2
2022-03-09 06817, 2022
alastairp
so now we're wondering about doing the copy version of the upgrade, but we'd like enough disk space (on gaga) to have 1) the v11 data, 2) a backup of it in case anything goes wrong, 3) the v13 data - but gaga doesnt have enough disk for this (db is about 770gb)
2022-03-09 06831, 2022
alastairp
zas: yes, I was just looking through servers and found that, thanks
2022-03-09 06806, 2022
alastairp
atj: so - if you have any postgres upgrade and/or docker/volume/hardlink experience, it'd be interesting to hear your thoughts
2022-03-09 06847, 2022
alastairp
lucifer: did you try the upgrade with --link with both data directories in the same volume?
2022-03-09 06838, 2022
atj
well, hardlinks definitely aren't going to work unless you're in the same volume
atj: I had hoped that because these volumes were on the same partition on the host, it'd just work :)
2022-03-09 06814, 2022
reosarevok
Freso: iirc I transcluded but I guess I didn't answer?
2022-03-09 06854, 2022
atj
alastairp: have you tried creating manual hardlinks?
2022-03-09 06846, 2022
atj
it would work from the host, but I don't think the container can see that they're the same filesystem
2022-03-09 06852, 2022
alastairp
lucifer: oh, one other thing - we should decide if we want to keep running on pg/timescale on debian, and if so decide which base image, how to install it, and how to start it (because the debian version splits the config/data directories, but our alpine data dir has the combined directory)
2022-03-09 06835, 2022
zas
atj: should we disable IPv6 on shorewall for now?
2022-03-09 06837, 2022
alastairp
atj: on the host or in the container? this is done by pg_upgrade (in the container), so I don't think I can do it myself on the host and have it work
I'd run the command from the root of the volume you are backing up
2022-03-09 06846, 2022
alastairp
ah, so you directly stream the compressed file, rather than write to file then copy
2022-03-09 06849, 2022
atj
just so the archive has relative paths - easier to unpack
2022-03-09 06854, 2022
atj
exactly
2022-03-09 06809, 2022
alastairp
but pv is going to be reporting the compressed size, right?
2022-03-09 06828, 2022
alastairp
so we'll expect it to be smaller than the size of the data dir, but we don't know how much smaller
2022-03-09 06837, 2022
atj
you can't know until you do it unfortunately
2022-03-09 06845, 2022
atj
you could try compressing a few files to see
2022-03-09 06801, 2022
alastairp
yeah, which is why I was thinking of tar -cf - /path | pv | lzip | ssh
2022-03-09 06832, 2022
atj
that should work
2022-03-09 06841, 2022
atj
you just won't know the transfer speed over ssh
2022-03-09 06848, 2022
alastairp
let me try it on another machine I have
2022-03-09 06810, 2022
alastairp
yeah, hopefully close to gige less overhead, but unclear
2022-03-09 06815, 2022
yvanzo
reosarevok: Has the old docker volume staticbrainz-data been replaced with musicbrainz-static-build-prod and musicbrainz-static-build-beta?
2022-03-09 06831, 2022
atj
alastairp: you might want to try lzma vs. lzip, I always get confused between these various lz compression algos
2022-03-09 06826, 2022
v6lur has quit
2022-03-09 06818, 2022
v6lur joined the channel
2022-03-09 06856, 2022
alastairp
atj: hmm, adding lzma in the middle makes it 100x _slower_
2022-03-09 06835, 2022
atj
CPU limited?
2022-03-09 06852, 2022
alastairp
lzma: [1.27MiB/s], no compression: [ 110MiB/s], gzip: [21.5MiB/s]
2022-03-09 06856, 2022
alastairp
checking now
2022-03-09 06817, 2022
alastairp
yes, 100% cpu
2022-03-09 06806, 2022
atj
lzip?
2022-03-09 06808, 2022
alastairp
other option is tar -> file, pbzip2, copy file, but there's no parallelism there
2022-03-09 06831, 2022
atj
zstd (yes another algo) supports multiple threads
2022-03-09 06845, 2022
alastairp
even with input from stdin?
2022-03-09 06858, 2022
atj
this is what I'm wondering
2022-03-09 06824, 2022
atj
I think these more advanced algos have large window sizes which requires more buffering in RAM
2022-03-09 06811, 2022
alastairp
lzip -0 says that it's about as fast as gzip, and I'm seeing speeds about the same, but it's still CPU-bound rather than network-bound
2022-03-09 06824, 2022
atj
not sure how much difference letting tar do the compression would make
2022-03-09 06825, 2022
alastairp
2 minutes to copy a 2gb file, compared to 20sec using just rsync
2022-03-09 06806, 2022
alastairp
does hetzner offer 10ge yet? :)
2022-03-09 06812, 2022
alastairp
trying that now
2022-03-09 06837, 2022
alastairp
no, that's just as slow
2022-03-09 06839, 2022
atj
alastairp: on my system "tar --zstd -cf -"
2022-03-09 06842, 2022
atj
1.46GiB 0:00:17 [83.3MiB/s]
2022-03-09 06804, 2022
atj
lzma and lzip were very slow
2022-03-09 06826, 2022
alastairp
yes, zstd is 3x faster than lzip or gzip immediately
2022-03-09 06801, 2022
alastairp
no compression: [ 110MiB/s], pv|zstd: [ 123MiB/s]
2022-03-09 06810, 2022
alastairp
so yes, zstd is giving a slight advantage
2022-03-09 06824, 2022
atj
I just compressed a 4.7GB VDI to 1.9GB at pretty much line speed
2022-03-09 06818, 2022
atj
looks like that's your best bet, whether the compression is worth it I don't know.
2022-03-09 06855, 2022
lucifer
alastairp: the MB docker setup is definitely useful for us to base on but not sure what all we need to test and prepare for that move so probably best to not do it now?
2022-03-09 06840, 2022
alastairp
oh yeah - I'm just doing a test with the pg dir now, it's hovering around 300MiB/sec with zstd (measuring with pv before going into zstd), so we're about 3x faster than just a regular copy over the same network link
2022-03-09 06843, 2022
alastairp
thanks atj!
2022-03-09 06856, 2022
atj
glad it worked :)
2022-03-09 06805, 2022
alastairp
still 40 mins to do the backup
2022-03-09 06857, 2022
atj
make sure to check the tar archive once it's completed - "tar --zstd -tf filename.tar.zstd"
2022-03-09 06812, 2022
skelly37 has quit
2022-03-09 06823, 2022
skelly37 joined the channel
2022-03-09 06811, 2022
atj
actually, tar does recognise the file extension so you can do "tar -vtf <filename>.tar.zstd"
2022-03-09 06824, 2022
alastairp
lucifer: yeah, that's a good point. so maybe we drop the indexes in current image, bring it up in the migrate image, do the upgrade, bring it up in the new timescale image, recreate indexes ?
2022-03-09 06854, 2022
alastairp
lucifer: I had a look at the config file diff, there are some things I don't understand, but I think it'll be OK to copy directly in place
2022-03-09 06819, 2022
lucifer
alastairp: are we going to do with --link or without? for without, we could keep the indexes and drop/recreate in the new image in case we need to bring up old cluster back up.
2022-03-09 06836, 2022
alastairp
lucifer: I know you have a few items on the migration doc for preparing listenstore downtime, do you need time for that?
