#metabrainz

mayhem, monkey: hey! can you send me a personal track recommendation? testing feed changes.

lucifer: Sent !

thanks!

oh well yeah, its been broken forever

it only shows up in the sender's feed, but never in the receiver's feed.

Oh what ?!

Crazy

I've been sending stuff, always assumed it worked because it showed up on my side

Thanks for finding this

monkey: investigating more, it actually does work as long as recording msid is present.

did you recommend from a now playing listen?

lucifer: got sec?

yes

I might just take you up on your hoffer to help with the MeB backend for sign-ups.

I'm really sick of this project and I'll never finish it without more help. :(

Negative lucifer

This page could be improved -- I think the sign up page for commercial users is fine already

From a player page (https://listenbrainz.org/player/release/94e6d3a...)

monkey: oh yeah, that doesn't have msids either.

mayhem: i see, yes

on this page I would improve 2 things: 1) the how are you using our data question: Make the question better and take feedback up to 500 charts. 2) Allow a user to indicate what data they are using and then save that data to the DB.

I am uncertain if that should be a free text field that people are not going to fill out well, or if we should have all of our projects with tickboxes data-sets that are available. something like:

i think tickboxes would be better for us to get a list of all users of a project.

1. [ ] MusicBrainz project data, music brainz canonical data, etc.

2. [ ] ListenBrainz, listen data, similiar artists, similar tracks

. . .

yeah a list like that sounds good.

I'm leaning towards tickboxes myself, but I am trying to think of us trying to make sense of the data later on.

the list above is simple, but it doesn't give a lot of granularity.

multilevel tickbox?

that.

where a user can tick the specific dataset they plan to download.

also need to consider if a user suddenly starts using more kinds of data, will they take the pain to go back to that page to update.

they never will.

yeah :/

but I think this data is a good first order approximation of what people are doing.

does this project make sense to you?

would you like me to summarize it in a ticket?

sounds good to me.

ok

sure ticket would be good.

will add a table to store various types of data we have and an admin panel to add more.

and monkey and I have started a brainstorm of how to get organized for the next 1-2 months.

we'll add to that today/monday and then we'll circle back to you to set goals for that timeframe.

we've got a lot of loose ends that we should finish up and the ideas is for us get more focused than we are now.

awesome, makes sense.

[musicbrainz-docker] 14yvanzo opened pull request #245 (03master…mbvm-64-resume-dl): MBVM-64: Resume fetching partially downloaded dump https://github.com/metabrainz/musicbrainz-docke...

lucifer: https://tickets.metabrainz.org/browse/MEB-145

MEB-145: Improve non commercial sign-up page

👍

it would be good if you can focus on that one sooner than later. ideally we can push the new datasets page out next week.

sure doable by next week.

[musicbrainz-docker] 14yvanzo opened pull request #246 (03master…mbvm-89-info-dl): MBVM-89: Show info about signing up on first fetch https://github.com/metabrainz/musicbrainz-docke...

thanks!

atj, reosarevok: Please review the two above (interdependent) PRs to MB Docker when available ^

is there anything stopping us having a canonical version of createdb.sh that is symlinked or something?

same with fetch-dump.sh

atj: yes, these are for building two different images.

atj: see https://stackoverflow.com/questions/31881904/do...

yvanzo: "COPY ../shared-scripts/* /usr/local/bin" ?

yvanzo: FYI - https://gist.github.com/atj/2d3c476db6f2905603f...

no, as mentioned in the above, you cannot pick files out of the build context, even using .. or symlink.

Good catch.

The "update-perl.sh" is expected, not the other (minor) diffs.

ok, the other minor diffs are my fault, sorry

np, will fix these

fixed in master

atj: did you reboot sentry already?

zas: no, i asked who used sentry so i could coordinate downtime but nobody answered

zas: that means you can reboot it, no?

you asked. :)

yup ;)

zas: I'm slg

wauidhuwaqd

? ;)

that's a rather elaborate command to exit from an editor, atj.

zas: I'm slightly more concerned about the risks of replacing docker.io with docker-ce than the reboot

zas: sorry, frustration due to my poor typing

and I want to do the package change before rebooting

AFAICS it shouldn't cause any issues, but I can't find a definitive answer and blowing up an entire service by accident isn't really on my TODO list

well, we can shutdown the server, take a snapshot, reboot, deploy, reboot. At worse, we use the snapshot to restore things

well, can we do that then? :P

ok, I'll shutdown sentry and take a snapshot

great

<lucifer> "also need to consider if a user suddenly starts using more kinds of data, will they take the pain to go back to that page to update." If the user could be identified somehow at the download stage, then their download types coule be updated automatically? (Possible long-term solution?)

*could be updated*

atj: server powered off, snapshot in progress

yvanzo: the PR for MBVM-64 looks good, I tested it and it seems to work

MBVM-64: Resume fetching partially downloaded dumps after a stop-and-start https://tickets.metabrainz.org/browse/MBVM-64

I was hoping it might resume partial downloads though ;)

however I can see that isn't possible using wget without the risk of ending up with a corrupted file

atj: It does resume partial downloads.

yvanzo: I don't think it does, but happy to be wrong

you need to specify the "-c" argument to wget for that

-N does that too, based on time-stamping

no, -N just sends an "If-Modified-Since" header so the server can return 304

Ok, can we combine both?

there is no way for wget to know if the part of the file it has download was subsequently changed

atj: You're right, I thought it was resuming partial downloads but I tested with a too small file.

Just tested with a larger file, it doesn't resume partial downloads at all.

yvanzo: it's still better than just downloading the files everytime regardless

yes but it's not doing what it's claimed to do

yvanzo: check the wget man page for --continue: https://www.man7.org/linux/man-pages/man1/wget....

atj: snapshot done, server powered up, you can proceed

"if the file is bigger on the server because it's been changed, as opposed to just appended to, you'll end up with a garbled file. Wget has no way of verifying that the local file is really a valid prefix of the remote file."

atj: Ok, I avoided using -c because of that and because I thought that -N would do it, but I have (had) a patch for using -c instead.

yvanzo: if you're downloading unique (e.g. filename contains a timestamp) then it should be safe to use -c

*unique files

however if you have a static filename, but the contents change then it's not

atj: or (as in my previous not pushed patch) just remove previous files if the content of LATEST changed.

(the filename is not unique but the path is, as the timestamp is in the directory name)

yvanzo: the destination path?

if you download 2 files with the same name from different URLs, wget doesn't know

zas: docker.io installed - containers weren't restarted 🧐

cannot be from different URLs if LATEST is checked beforehand

yvanzo: ok, sounds good

thanks, will do that

can save using some GBs of bandwidth

i always found it quite annoying how if the startup process failed you had to re-download everything

zas: we need to restore from snapshot - https://gist.github.com/atj/657e0a6c57448d18d53...

helpfully, despite the message, pressing Ctrl+C did nothing

i didn't want to leave a landmine waiting to go off when someone ran `apt autopurge`

So, I shutdown and restore from snapshot?

yes, sorry I think docker is in an inconsistent state

your mom is in an inconsistent state

frequently

heh

zas: any chance I have access to the Hetzner control panel so I don't have to rely on you to do this?

you can also ask me to do this.

mayhem: any chance I have access to the Hetzner control panel so I don't have to rely on zas to do this?

atj: I'm on it to give you access

zas: thanks!

atj: you should receive an invitation by email

got it

rebuilding server from snapshot is quite slow