OK, I'll put it in /pub/musicbrainz/canonical_data, for now I'll make a new key but I'll open an MBH ticket for streamlining this with atj's suggestion of a wrapper script
2023-01-31 03121, 2023
mayhem
good plan
2023-01-31 03100, 2023
atj
i can sort the wrapper script out
2023-01-31 03154, 2023
alastairp
atj: although see also my other comment about the start_sshd_musicbrainz_fullexport function, it's possible that we can use a single key per project just by moving the locations of volumes
mayhem: I was looking at other uses of execute values in LB and they seem to also work with lists of tuples like you originally had, so not sure what's going on there
mayhem: oh, I don't think you showed me that exception
2023-01-31 03112, 2023
ArjunM joined the channel
2023-01-31 03120, 2023
lucifer
ok, that won't work i think. you need to use POST. [{"artist_mbid": uuid1, "algorithm": alg}, {"artist_mbid": uuid2, "algorithm": alg}]
2023-01-31 03120, 2023
mayhem
I swear, I can be on a tear with programming, but when it comes time to pass args to psycopg2, its given I'm going to lose a couple of hours of work due to this fuckery.
2023-01-31 03122, 2023
alastairp
looks like it's treating the artistid as a single string "mbid,mbid" rather than a list
2023-01-31 03114, 2023
mayhem
hmm.
2023-01-31 03125, 2023
lucifer
but it will always use the algorithm from the first item even if you pass different ones in items (improvements pending).
2023-01-31 03127, 2023
alastairp
or judicious use of `artist_mbids = mbid.split(",")`
2023-01-31 03149, 2023
mayhem
lucifer: that is a fine solution if you ask me.
2023-01-31 03109, 2023
alastairp
lucifer: (hi!), are you around for a bit?
2023-01-31 03127, 2023
mayhem
well, now that you're here, could you please make similar-artists correctly accept [artist_mbid] as an argument?
Currently I've named the dump this ("Metabrainz metadata"), should we change it to "musicbrainz canonical" perhaps?
2023-01-31 03159, 2023
mayhem
lucifer: I get a 500 error if I pass in a comma seperated list of artist_mids or a list of mbids.
2023-01-31 03106, 2023
mayhem
see the last 4 errors in sentry.
2023-01-31 03130, 2023
mayhem
alastairp: that suggestion seems good.
2023-01-31 03133, 2023
lucifer
mayhem: yes, it doesn't work that way currently.
2023-01-31 03111, 2023
mayhem
I'd like it to work that way, please. :)
2023-01-31 03106, 2023
lucifer
alastairp's comma suggestion should work then. should be fine as a temp fix.
2023-01-31 03114, 2023
ArjunM
Hi team. I'm Arjun, a CS undergrad with some experience in JavaScript, Java and SQL. I'm new to this community and I look forward to contribute to MetaBrainz as part of GSoC'23 and otherwise.
akshaaatt: should i clean up anything related to lookup and collections?
2023-01-31 03124, 2023
agatzk has quit
2023-01-31 03130, 2023
akshaaatt
Yes jasje_ , please do so
2023-01-31 03132, 2023
BrainzGit
[listenbrainz-server] 14alastair opened pull request #2348 (03master…canonical-data-dump-csv): Return artist mbids as comma separated instead of postgres array format https://github.com/metabrainz/listenbrainz-server…
2023-01-31 03149, 2023
alastairp
lucifer: another quick one which I noticed when using the data
2023-01-31 03105, 2023
jasje_
can you just gimme what stuff "Keywords" do you remember to clean up
2023-01-31 03119, 2023
jasje_
so i can just have an idea and compare with musicbrainz
2023-01-31 03133, 2023
jasje_
what stuff to remove what not
2023-01-31 03107, 2023
lucifer
alastairp: looks good
2023-01-31 03112, 2023
alastairp
nice
2023-01-31 03125, 2023
BrainzGit
[listenbrainz-server] 14alastair closed pull request #2348 (03master…canonical-data-dump-csv): Return artist mbids as comma separated instead of postgres array format https://github.com/metabrainz/listenbrainz-server…
2023-01-31 03130, 2023
BrainzGit
[listenbrainz-server] 14alastair reopened pull request #2348 (03master…canonical-data-dump-csv): Return artist mbids as comma separated instead of postgres array format https://github.com/metabrainz/listenbrainz-server…
2023-01-31 03135, 2023
BrainzGit
[listenbrainz-server] 14alastair merged pull request #2348 (03master…canonical-data-dump-csv): Return artist mbids as comma separated instead of postgres array format https://github.com/metabrainz/listenbrainz-server…
2023-01-31 03136, 2023
alastairp
ups
2023-01-31 03139, 2023
jasje_
akshaaatt: ill should keep repositories and data classes intact tho right?
2023-01-31 03117, 2023
agatzk joined the channel
2023-01-31 03127, 2023
jasje_
i should**
2023-01-31 03132, 2023
BrainzGit
[listenbrainz-server] 14alastair merged pull request #2340 (03master…canonical-dump-connection): Add option to select the database connection to dump canonical data from https://github.com/metabrainz/listenbrainz-server…
2023-01-31 03107, 2023
alastairp
finally!
2023-01-31 03141, 2023
alastairp
lucifer: I'll make a release with these changes if that sounds good
ill keep the repos which i think could be used in future
2023-01-31 03130, 2023
jasje_
most of the stuff ill just remove
2023-01-31 03137, 2023
alastairp
ok, at the moment I'll just use this release for cron and testing the dumps, let me coordinate with lucifer and we can do a .1 release for the website
2023-01-31 03147, 2023
mayhem
ah, ok.
2023-01-31 03126, 2023
alastairp
bitmap: (or yvanzo, reosarevok?): is there a way if I can tell if sshd-musicbrainz-fullexport on aretha is currently receiving an rsync dump? `ps` shows nothing running in the container, so I suspect it's fine
2023-01-31 03148, 2023
alastairp
lucifer: quick PR in docker-server-configs related to this too
2023-01-31 03122, 2023
yvanzo
good question
2023-01-31 03155, 2023
alastairp
I need to restart it to install a new key
2023-01-31 03112, 2023
yvanzo
let me check
2023-01-31 03138, 2023
alastairp
or I guess if we don't want to restart it I could copy the key contents in manually, but that doesn't let me verify that the PR that I opened works correctly
apologies, akshaaatt. i'll be contributing to bookbrainz
2023-01-31 03126, 2023
akshaaatt
Sounds great ArjunM ! You should definitely join the #bookbrainz channel and discuss things in detail there. The best way to start is by understanding the codebase and kicking things off with a PR or two :) There’s tons of things you could explore in the project and have a look at the bug tracker for issues to fix or discuss.
2023-01-31 03127, 2023
yvanzo
alastairp: don't have any better to suggest than 'ps aux' atm
2023-01-31 03100, 2023
alastairp
thanks yvanzo, restarting the service now
2023-01-31 03135, 2023
ArjunM
will head right there, akshaaatt!
2023-01-31 03137, 2023
lucifer
mayhem: i think there's a bug in ds hoster code.
2023-01-31 03151, 2023
lucifer
i'll put a temp fix in the patch and labs api endpoint for now.
2023-01-31 03106, 2023
mayhem
k.
2023-01-31 03113, 2023
mayhem
DSH needs some love in any case.
2023-01-31 03151, 2023
akshaaatt
ArjunM: monkey should be you PoC. But go through the docs first and try to exhaust all the available information before making conclusions. That being said, genuine doubts are welcome 24*7 !
2023-01-31 03113, 2023
akshaaatt
Your*
2023-01-31 03137, 2023
ArjunM
yes, got it
2023-01-31 03158, 2023
yvanzo
lucifer: CB security alerts will need a lighthouse keeper for the next months, would you agree to replace alastairp?
2023-01-31 03112, 2023
lucifer
sure yvanzo
2023-01-31 03120, 2023
alastairp
🙏 thank you
2023-01-31 03124, 2023
yvanzo
alastairp: should I reassign the current CB security alerts too?
2023-01-31 03132, 2023
alastairp
yes please
2023-01-31 03137, 2023
yvanzo
Done, thank you both!
2023-01-31 03100, 2023
lucifer
mayhem: dataset fetcher element code in troi has some special casing for artist_mbid which is creating troubles
alastairp: if there's no rrsync process it should be fine
2023-01-31 03133, 2023
alastairp
yvanzo: yes, I think that's reasonable
2023-01-31 03139, 2023
alastairp
thanks bitmap
2023-01-31 03109, 2023
alastairp
bitmap: related, not sure if you saw earlier discussion about the proliferation of ssh keys in that container - it seems that it's because we use `command` in a pubkey in authorized_keys to run `rrsync` which limits the connecting client to a particular directory
2023-01-31 03143, 2023
alastairp
we're already up to 4 keys for ListenBrainz, not sure if you know the original motivation behind that, but it could be something to clean up and improve
2023-01-31 03151, 2023
jasje_ has quit
2023-01-31 03135, 2023
lucifer
mayhem: its the popular recordings element that is erroring! not the labs api one
