bitmap: I just truncated postgres-floyd log file (it was 252Gb big), daemon config was changed but daemon never restarted. There's a way to do that at runtime somehow, but I'd prefer you are around if we try that....
2021-01-20 02015, 2021
diru1100 joined the channel
2021-01-20 02027, 2021
diru1100
Good afternoon!!
2021-01-20 02018, 2021
diru1100
Just completed FOSS exam, literally wrote everything related to MetaBrainz.
2021-01-20 02024, 2021
shivam-kapila
interesting subject
2021-01-20 02026, 2021
shivam-kapila
jenkins appears to be down. I hope its known
2021-01-20 02044, 2021
zas
bitmap, ruaok, alastairp: we still have disk space issues on williams
2021-01-20 02042, 2021
zas
shivam-kapila: it should be ok now, it was me, I had to reload docker daemon config on williams and jenkins issue is likely related
2021-01-20 02050, 2021
zas
can you confirm it works?
2021-01-20 02031, 2021
shivam-kapila
zas: yeah its working now. thanks
2021-01-20 02002, 2021
alastairp
zas: I freed up less than 10gb
2021-01-20 02058, 2021
alastairp
on Friday I will work on the automatic cleanup tasks that I mentioned last time, and I will move to cage, we should check if there are extra wal files again like bitmap found last week. I don't know where they are
2021-01-20 02038, 2021
d4rkie has quit
2021-01-20 02010, 2021
Etua joined the channel
2021-01-20 02047, 2021
zas
there are in /var/lib/docker/volumes/barman-backups/_data/floyd 427Gb
2021-01-20 02054, 2021
zas
bitmap: ^^
2021-01-20 02016, 2021
loujine has quit
2021-01-20 02057, 2021
Etua1 joined the channel
2021-01-20 02014, 2021
loujine joined the channel
2021-01-20 02037, 2021
Etua has quit
2021-01-20 02037, 2021
Etua1 is now known as Etua
2021-01-20 02052, 2021
d4rkie joined the channel
2021-01-20 02004, 2021
ruaok
moooin!
2021-01-20 02021, 2021
ruaok
zas: that file is on williams? that would explain the diskspace problems. :)
2021-01-20 02000, 2021
zas
ruaok: that's multiple files, that's barman collecting wals from postgres on floyd to back up main db, but for some reason they sometimes accumulate. Issue was identified a while ago, but we don't know yet what causes it. bitmap could prolly tell more about this.
2021-01-20 02033, 2021
zas
but that's not the sole disk space user on williams, jenkins is not cleaning all its mess
2021-01-20 02045, 2021
zas
(alastairp will take care of this soon)
2021-01-20 02059, 2021
zas
and ftp uses almost 700Gb
2021-01-20 02027, 2021
alastairp
but jenkins runs over by ~10gb every few weeks, with is small fry compared to stuck barman backups and ftp
2021-01-20 02037, 2021
zas
mainly because of listenbrainz dumps: 439gb atm
2021-01-20 02018, 2021
zas
alastairp: yup
2021-01-20 02001, 2021
alastairp
did I recall ruaok mentioning that we could use the storage box for ftp?
2021-01-20 02017, 2021
zas
yes, we could for ftp at least
2021-01-20 02012, 2021
zas
ah, listenbrainz is 539gb (not 439) atm
2021-01-20 02020, 2021
zas
mainly in /home/ftp/ftp/root/pub/musicbrainz/listenbrainz/fullexport
2021-01-20 02022, 2021
ruaok
an the FTP situation is only going to get worse as we publish more data sets.
2021-01-20 02052, 2021
ruaok
for hetzner cloud there are attachable storage devices that perform and work much better than storage boxes.
2021-01-20 02018, 2021
ruaok
I wonder if we can attach these to bare metal servers yet. that would be the ideal method for moving forward on this. let me look.
doesn't seem like those can be mounted into a FS and are used on a "sync" basis.
2021-01-20 02048, 2021
reosarevok
bitmap, yvanzo: what do you think of https://tickets.metabrainz.org/browse/MBS-11339 ? It seems sensible to me, but if we do should we still keep a redirect for those /artist/gid/credit/id links in case someone is using them externally for some reason? I doubt it, but :)
2021-01-20 02049, 2021
BrainzBot
MBS-11339: Move Edit Artist Credit from /artist to /artist-credit
2021-01-20 02051, 2021
ruaok
ok, storage boxes support HTTPS and FTP. what if we get rid of our FTP setup and simply expose things we put into the storage box? that would simplify dump creation for LB, for sure.
2021-01-20 02055, 2021
reosarevok
(the new pages aren't on beta yet, but they're merged)
2021-01-20 02034, 2021
alastairp
ruaok: is that public ftp or private? from that page it sounds like they have a transfer limit externally, but it's unlimited internally
2021-01-20 02053, 2021
alastairp
which means that it might make sense to proxy through an existing server, given that they have unlimited external traffic too?
2021-01-20 02024, 2021
ruaok
yes, the page is not very clear -- what does 10 concurrent connections me? internal or external?
2021-01-20 02003, 2021
ruaok
but yes, that makes sense overall. serve the FTP from williams via the storage box. and then any of our servers which need to publish dumps, can publish them to the storage box, not to williams.
2021-01-20 02001, 2021
Gazooo79494 has quit
2021-01-20 02049, 2021
Gazooo79494 joined the channel
2021-01-20 02011, 2021
yvanzo
reosarevok: it's your own ticket, of course it seems sensible to you :D
2021-01-20 02004, 2021
yvanzo
reosarevok: there is no redirect mechanism for artist credits and they do not have MBIDs either. I’m not sure how you would implement that. It probably requires a schema change.
2021-01-20 02052, 2021
yvanzo
reosarevok: maybe /artist/…/credit/… would be a better place than the currently merged /artist-credit/… so to clarify artist-credits depend on artist?
2021-01-20 02003, 2021
antlarr2 joined the channel
2021-01-20 02032, 2021
antlarr has quit
2021-01-20 02040, 2021
reosarevok
It doesn't though. They have their own IDs. We don't need a redirect mechanism, the original links already include the ac id
2021-01-20 02041, 2021
Sophist-UK
2021-01-20 02042, 2021
Sophist-UK
2021-01-20 02001, 2021
Sophist-UK
2021-01-20 02002, 2021
Sophist-UK
2021-01-20 02003, 2021
Sophist-UK
2021-01-20 02004, 2021
Sophist-UK
2021-01-20 02005, 2021
Sophist-UK
2021-01-20 02006, 2021
Sophist-UK
2021-01-20 02007, 2021
Sophist-UK
2021-01-20 02008, 2021
Sophist-UK
2021-01-20 02009, 2021
Sophist-UK
2021-01-20 02010, 2021
Sophist-UK
2021-01-20 02011, 2021
Sophist-UK
2021-01-20 02012, 2021
Sophist-UK
2021-01-20 02024, 2021
Sophist-UK
Oops - sorry.
2021-01-20 02004, 2021
yvanzo
reosarevok: I mean, having a top level URL like /artist-credit/… may let think that this is a permanent link like /artist/…; this is not the case because there is no redirect mechanism.
2021-01-20 02037, 2021
yvanzo
If only URLs such as /artist/…/credit/… are provided, it can at least be redirected to the related artist page.
2021-01-20 02019, 2021
Etua has quit
2021-01-20 02043, 2021
Sophist_UK joined the channel
2021-01-20 02011, 2021
Sophist-UK has quit
2021-01-20 02057, 2021
diru1100 has quit
2021-01-20 02044, 2021
sumedh joined the channel
2021-01-20 02037, 2021
loujine
zas: I get random timeouts when querying test.musicbrainz.org/ws/2/ (sometimes answers in < 1 sec, sometimes > 10 sec)
2021-01-20 02049, 2021
loujine
no problem on mb.o and beta.mb.o
2021-01-20 02009, 2021
loujine
just so that you know, it's not a real problem for me
2021-01-20 02016, 2021
sumedh has quit
2021-01-20 02023, 2021
sumedh joined the channel
2021-01-20 02038, 2021
sumedh has quit
2021-01-20 02026, 2021
sumedh joined the channel
2021-01-20 02013, 2021
ruaok
pristine___: should we merge that massive PR and work on a release?
2021-01-20 02055, 2021
pristine___
Yes!
2021-01-20 02055, 2021
ruaok
ok, I need one other PR merged before that. let me chase alastairp for this...
2021-01-20 02029, 2021
alastairp
the prod release of typesense?
2021-01-20 02042, 2021
ruaok
sentry cleanup.
2021-01-20 02048, 2021
alastairp
ah
2021-01-20 02002, 2021
ruaok
let me merger master in to fix that failing test.
2021-01-20 02021, 2021
alastairp
I made one comment about the format of the title of the inactive user messages
2021-01-20 02058, 2021
alastairp
along the lines of saying if the aim is to reduce the number of emails that we get, making the title consistent is going to prevent sentry from emailing the alert for each user
2021-01-20 02059, 2021
ruaok
yep, but I see no reason to deal with that, if it isn't clear that we need to keep that message.
2021-01-20 02019, 2021
alastairp
will changing from error to debug stop sentry from sending emails?
2021-01-20 02008, 2021
ruaok
one way to deal with that.
2021-01-20 02017, 2021
alastairp
heh, that certainly is a fix
2021-01-20 02025, 2021
ruaok
right. now all 4 tests failed. is our jenkins config borked?
2021-01-20 02023, 2021
pristine___
Does the Jenkins integration test fail because of that, I mean the PR which you want to merge beforehand
