https://listenbrainz.org/similar-users suggests that the two users listed have identical listening histories, but that doesn’t seem apparent from looking at their profiles, and also their MB profiles don’t seem to hint at them being the same person. Any chance someone with more direct access to the data can give them a look and give me some more insight in what is or might be going on?
2021-07-26 20738, 2021
lucifer
Freso: that page is currently bugged. i'll look into fixing it soon.
2021-07-26 20750, 2021
Freso
lucifer: 👍
2021-07-26 20725, 2021
Freso
lucifer: Is there a ticket I can watch? :)
2021-07-26 20758, 2021
lucifer
Freso: none that i know of. but i'll ping you once its fixed.
2021-07-26 20753, 2021
lucifer
ruaok: hi! as you can see from my later comments, it worked but showed another problem. to fix the issue completely we also need to offload the spark invocation to a different thread. i'll probably take the MBID mapping writer route for that.
2021-07-26 20701, 2021
lucifer
ruaok: alastairp: let me know when you have some time. need to discuss about artist credit id -> artist mbid in LB stats.
2021-07-26 20731, 2021
ruaok
I have a few mins. (On mobile)
2021-07-26 20735, 2021
ruaok
What is the q?
2021-07-26 20752, 2021
ruaok
I suspect that you need a mapping available in spark, yes?
2021-07-26 20725, 2021
lucifer
lb stats now return artist_credit_id and artist name, but frontend needs a artist mbid so we need to add a translation step somewhere
2021-07-26 20738, 2021
ruaok
That should be easy to create when you have an mb db connection
2021-07-26 20700, 2021
lucifer
yeah so doing on LB side is probably better.
2021-07-26 20722, 2021
ruaok
Look at the mb schema and perhaps get reosarevok to help you with this query, if it isn't obvious.
2021-07-26 20749, 2021
lucifer
when LB receives stats from spark, replace artist_credit_ids with artist_mbids, right?
2021-07-26 20757, 2021
ruaok
Then write the data to a pandas dataframe then parquet, then import.
2021-07-26 20708, 2021
lucifer
ah ok, understood
2021-07-26 20727, 2021
ruaok
A longer term solution night be to have them included in the mbid mapping.
2021-07-26 20751, 2021
ruaok
Export ac id and array[mbids]
2021-07-26 20703, 2021
ruaok
It would make the mapping dump a bit larger...
2021-07-26 20723, 2021
lucifer
makes sense
2021-07-26 20757, 2021
lucifer
we are going to need artist mbid almost always though, artist credit are not very useful to expose to users.
2021-07-26 20712, 2021
lucifer
*artisy credit ids are not
2021-07-26 20704, 2021
lucifer
thanks, i'll try the separate dump for now. we can look into the longer term solution later
2021-07-26 20759, 2021
peterhil has quit
2021-07-26 20722, 2021
lucifer
yvanzo: hi! i am investigating some rabbitmq issues in LB. i see sir is having those too according to rabbitmq-clash logs. mentioning in case its relevant.
annoy fails to compile on the prod build action, I just tried it locally from scratch and it worked fine. Perhaps something related to a cached layer that's incorrect?
I need a way to check the version of gcc in the build container, I'll check
2021-07-26 20732, 2021
alastairp
but I've just had a thought - It could be something weird like the the build farm hardware advertising that it has a particular avx extension
2021-07-26 20755, 2021
alastairp
which could cause problems if we try and build on one architecture and then deploy on another :(
2021-07-26 20715, 2021
lucifer
yeah :/, we'll need to configure according to our machine's hardware
2021-07-26 20739, 2021
alastairp
they have a fix for it which I think will help for now - because we're compiling on a lower version of gcc it'll skip the extension, but this is definitely something to check and fix when we upgrade gcc
OK, I just worked out what mtune=native does. it means "use all of the flags that are available on this specific build machine", which isn't what we want
in newer versions of gcc there are zen flags that we can use
2021-07-26 20727, 2021
lucifer
nice.
2021-07-26 20730, 2021
alastairp
oh amazing... I guess you don't know which machine you're going to run on!
2021-07-26 20709, 2021
alastairp
let me open another PR to upgrade the version of annoy. at least we'll use the right extensions when using gcc 5. We can address it again when we upgrade to a new baseimage
you can ignore the latest commits, i am trying to come up with a threading solution but haven't reached there yet.
2021-07-26 20752, 2021
lucifer
i have written down the issue i found in context of the request consumer but these issues exist elsewhere as well. in api, spark reader so on. i didn't check ts writer yet so don't know about that one.
2021-07-26 20737, 2021
peterhil has quit
2021-07-26 20705, 2021
peterhil joined the channel
2021-07-26 20719, 2021
yvanzo
hi lucifer: thanks, it's likely relevant, btw you would be very welcome to look at sir again after you are done with your current project.
2021-07-26 20722, 2021
lucifer
sure :D
2021-07-26 20739, 2021
yvanzo
yyoung: thanks, did not reply the other thread yet, will check your PR as well.
2021-07-26 20756, 2021
BrainzGit joined the channel
2021-07-26 20723, 2021
alastairp
lucifer: sure, I'll look at it tomororw when I'm on LB tasks
2021-07-26 20710, 2021
lucifer
awesome, thanks!
2021-07-26 20735, 2021
CatQuest
ok yvanzo I'm here now
2021-07-26 20744, 2021
BrainzGit
[acousticbrainz-server] 14alastair merged pull request #409 (03master…annoy-valid-extensions): Upgrade annoy, force a CPU architecture to avoid unavailable extensions https://github.com/metabrainz/acousticbrainz-serv…
Thanks a lot lucifer . I have written a medium article as well recently regarding the GSoC experience. Will share it when we have the meeting in an hour :D
2021-07-26 20747, 2021
alastairp
zas: I need to rsync 50gb of data between servers. should it be faster through public IP, or private network, or should they be the same? I'm seeing only 3-6MB/sec transfer, which seems low. Are you imposing any speed restrictions?
2021-07-26 20754, 2021
alastairp
(frank to clash)
2021-07-26 20755, 2021
zas
nope no speed restriction, it should be faster over private network
2021-07-26 20744, 2021
ruaok
Freso: slow week for me, just some bits and bobs, was on semi vacation am on vacation now.
Last week I mostly worked on some improvements for the ListenBrainz player. Some stability improvements and the addition of new features, namely native notifications and media control. Associated PRs are #1539 and #1561 which have more detailed descriptions.
2021-07-26 20726, 2021
Freso
You can test this all out on test.listenbrainz.org <http://test.listenbrainz.org/>, and feedback would be very appreciated if you find anything that looks broken or unintended (or if you’re happily surprised too).
