#metabrainz

is it musicbrainz derived data?

there is already a derived data dump for MB.

right, so perhaps separate folders for each type of dump might be better - that is, go with your suggestion of canonical_data

ok

alastairp: there's a module that seems to do it though -> https://www.nginx.com/resources/wiki/modules/fa...

OK, I'll put it in /pub/musicbrainz/canonical_data, for now I'll make a new key but I'll open an MBH ticket for streamlining this with atj's suggestion of a wrapper script

i can sort the wrapper script out

atj: although see also my other comment about the start_sshd_musicbrainz_fullexport function, it's possible that we can use a single key per project just by moving the locations of volumes

alastairp: https://github.com/metabrainz/listenbrainz-serv...

[listenbrainz-server] 14mayhem opened pull request #2347 (03master…various-mapping-fixes): Various mapping fixes https://github.com/metabrainz/listenbrainz-serv...

mayhem: I was looking at other uses of execute values in LB and they seem to also work with lists of tuples like you originally had, so not sure what's going on there

e.g. https://github.com/metabrainz/listenbrainz-serv...

I'm sure lucifer knows. :)

alastairp: if you'd like, gimme three or more artist mbids as seeds. I'll make ya a playlist.

alastairp, mayhem: hi! i am not sure what you are trying to fix there?

mayhem: oh, I don't think you showed me that exception

ok, that won't work i think. you need to use POST. [{"artist_mbid": uuid1, "algorithm": alg}, {"artist_mbid": uuid2, "algorithm": alg}]

I swear, I can be on a tear with programming, but when it comes time to pass args to psycopg2, its given I'm going to lose a couple of hours of work due to this fuckery.

looks like it's treating the artistid as a single string "mbid,mbid" rather than a list

hmm.

but it will always use the algorithm from the first item even if you pass different ones in items (improvements pending).

or judicious use of `artist_mbids = mbid.split(",")`

lucifer: that is a fine solution if you ask me.

lucifer: (hi!), are you around for a bit?

well, now that you're here, could you please make similar-artists correctly accept [artist_mbid] as an argument?

mayhem: basically, same as what you were doing for similar recordings patch: https://github.com/metabrainz/troi-recommendati...

alastairp: hi! yup

mayhem: you mean the comma split solution?

i dont really care how it works, I just want that endpoint to allow me to give multiple artist ids.

lucifer: LB#2340 is ready, including shell script. I'm just making a docker-server-configs PR now

Add option to select the database connection to dump canonical data from: https://github.com/metabrainz/listenbrainz-serv...

i see, well the post thing already works. so to confirm, you want it to work in get version as well?

no that isn't needed. let me check something.

I'm only using post in my troi code, but something wasn't working. let me look again.

Let me “troi” and work on it

heh

alastairp: PR lgtm, thanks! but can't be really with anything dumps related until tested in prod.

mayhem: if you can point me to the code, can take a look at it.

lucifer: yeah, of course. I've been running it locally and it seems to work fine for me. so maybe we can deploy and check

sounds good

mayhem: one more thing for your feedback on - given that we've decided to put this dump in a `canonical_data` directory

Currently I've named the dump this ("Metabrainz metadata"), should we change it to "musicbrainz canonical" perhaps?

lucifer: I get a 500 error if I pass in a comma seperated list of artist_mids or a list of mbids.

see the last 4 errors in sentry.

alastairp: that suggestion seems good.

mayhem: yes, it doesn't work that way currently.

I'd like it to work that way, please. :)

alastairp's comma suggestion should work then. should be fine as a temp fix.

Hi team. I'm Arjun, a CS undergrad with some experience in JavaScript, Java and SQL. I'm new to this community and I look forward to contribute to MetaBrainz as part of GSoC'23 and otherwise.

mayhem: try again now

try what?

the patch or get request

I wasn't making get requests.

ah ok, post will work too

is it expecting a JSON list or a comma separated string?

comma separated string

to support json list, will be problematic until dataset hoster is updated.

Hi ArjunM ! Please be a bit more specific about which project you would like to contribute to.

ah, its expecting artist_mbid not with []

let me add those

<p>No parameters supplied. Required: [artist_mbid]</p>

something is odd. :)

akshaaatt: should i clean up anything related to lookup and collections?

Yes jasje_ , please do so

[listenbrainz-server] 14alastair opened pull request #2348 (03master…canonical-data-dump-csv): Return artist mbids as comma separated instead of postgres array format https://github.com/metabrainz/listenbrainz-serv...

lucifer: another quick one which I noticed when using the data

can you just gimme what stuff "Keywords" do you remember to clean up

so i can just have an idea and compare with musicbrainz

what stuff to remove what not

alastairp: looks good

nice

[listenbrainz-server] 14alastair closed pull request #2348 (03master…canonical-data-dump-csv): Return artist mbids as comma separated instead of postgres array format https://github.com/metabrainz/listenbrainz-serv...

[listenbrainz-server] 14alastair reopened pull request #2348 (03master…canonical-data-dump-csv): Return artist mbids as comma separated instead of postgres array format https://github.com/metabrainz/listenbrainz-serv...

[listenbrainz-server] 14alastair merged pull request #2348 (03master…canonical-data-dump-csv): Return artist mbids as comma separated instead of postgres array format https://github.com/metabrainz/listenbrainz-serv...

ups

akshaaatt: ill should keep repositories and data classes intact tho right?

i should**

[listenbrainz-server] 14alastair merged pull request #2340 (03master…canonical-dump-connection): Add option to select the database connection to dump canonical data from https://github.com/metabrainz/listenbrainz-serv...

finally!

lucifer: I'll make a release with these changes if that sounds good

alastairp: yes sounds good

[listenbrainz-server] release 03v-2023-01-31.0 has been published by 14github-actions[bot]: https://github.com/metabrainz/listenbrainz-serv...

alastairp: I was really hoping to get the daily jams PR merged before the next LB release.

Remove those as well jasje_

Lookups can also be removed jasje_

ill keep the repos which i think could be used in future

most of the stuff ill just remove

ok, at the moment I'll just use this release for cron and testing the dumps, let me coordinate with lucifer and we can do a .1 release for the website

ah, ok.

bitmap: (or yvanzo, reosarevok?): is there a way if I can tell if sshd-musicbrainz-fullexport on aretha is currently receiving an rsync dump? `ps` shows nothing running in the container, so I suspect it's fine

lucifer: quick PR in docker-server-configs related to this too

good question

I need to restart it to install a new key

let me check

or I guess if we don't want to restart it I could copy the key contents in manually, but that doesn't let me verify that the PR that I opened works correctly

mayhem: please try again

still getting 400

can you share the code that is making the request

[artist_mbid] now?

apologies, akshaaatt. i'll be contributing to bookbrainz

Sounds great ArjunM ! You should definitely join the #bookbrainz channel and discuss things in detail there. The best way to start is by understanding the codebase and kicking things off with a PR or two :) There’s tons of things you could explore in the project and have a look at the bug tracker for issues to fix or discuss.

alastairp: don't have any better to suggest than 'ps aux' atm

thanks yvanzo, restarting the service now

will head right there, akshaaatt!

mayhem: i think there's a bug in ds hoster code.

i'll put a temp fix in the patch and labs api endpoint for now.

k.

DSH needs some love in any case.

ArjunM: monkey should be you PoC. But go through the docs first and try to exhaust all the available information before making conclusions. That being said, genuine doubts are welcome 24*7 !

Your*

yes, got it

lucifer: CB security alerts will need a lighthouse keeper for the next months, would you agree to replace alastairp?

sure yvanzo

🙏 thank you

alastairp: should I reassign the current CB security alerts too?

yes please

Done, thank you both!

mayhem: dataset fetcher element code in troi has some special casing for artist_mbid which is creating troubles

uh actually nvm

!!!;lfoifw9psacjlkwfe

por fin

yayyyyyyyy

what a drama

thanks lucifer and mayhem for dragging me across the line

alastairp: Should GH security alerts for AB repos be ignored by SEC from now on? (There is only SEC-778 still open.)

SEC-778: [acousticbrainz-server] CVE-2021-34141: numpy <= 1.21.6 https://tickets.metabrainz.org/browse/SEC-778

alastairp: if there's no rrsync process it should be fine

yvanzo: yes, I think that's reasonable

thanks bitmap

bitmap: related, not sure if you saw earlier discussion about the proliferation of ssh keys in that container - it seems that it's because we use `command` in a pubkey in authorized_keys to run `rrsync` which limits the connecting client to a particular directory

we're already up to 4 keys for ListenBrainz, not sure if you know the original motivation behind that, but it could be something to clean up and improve

mayhem: its the popular recordings element that is erroring! not the labs api one

d'oh