#metabrainz

Hi Freso, I will be in a location today which doesn’t have any communication means, so will have to skip today’s meeting. For my update, I’ve been on a vacation and had a look at yellowhatpro’s work on the android app! Thank you.

Looks like it's still going https://usercontent.irccloud-cdn.com/file/rPHMD...

alastairp: looks like there has been a _slight miscalculation_

MLHD has 600k files, not 60k. So the estimated total time is 50hours, not 5 hours ⚰️

However the good news is, the processing is going fine so far with 0.3s avg testing time per file

No track-MBIDs detected in recording-MBID so far

[critiquebrainz] 14anshg1214 opened pull request #446 (03master…CB_440): CB-440: Recording entity [unknown artist] https://github.com/metabrainz/critiquebrainz/pu...

akshaaatt: Noted. Thanks. :)

Pratha-Fish: and imagine if we were taking 3 seconds per file instead of 0.3!

alastairp: Also, looks like the process is constantly using > 90% CPU on wolf. I hope it doesn't disturb other user's work

Pratha-Fish: note that this is 90% of 1 CPU core

we have 12 CPU cores

_wow_

btw, one thing we could have done is start up 6-8 parallel workers for the same process

get it done in 7 hours

no worries, let's just leave it to do its thing

Damn that would've been nice

Pratha-Fish: any indication so far about track ids?

Pratha-Fish: are you also saving the zst files? in what location?

alastairp: Nothing as of today morning

alastairp: I am saving it all in snaek/MLHD/rec_track_checker/MLHD

I see it, grat

great

Also, all logs are being written in 1 level above the dir where MLHD is being written. So far so good

alastairp: 163k files checked so far. Nothing found. Do you need any other numbers while we're at it?

nothing yet

hmm

Pratha-Fish: from what I can see of the code, you're just loading it in, checking the first column against your db tables, and then writing the dataframe out again?

alastairp: yes that's right

however, I just randomly sampled a few of your compressed zst files and compared them against the gzip version of the same file, and the resulting uncompressed data is different

How exactly?

good question

I ran this:

diff <(zstdcat /home/snaek/MLHD/rec_track_checker/MLHD/0a/0a118981-15b5-46df-8666-080ca5a1af62.csv.zst) <(zcat /data/mlhd/0a/0a118981-15b5-46df-8666-080ca5a1af62.txt.gz)

which should have no output (indicating that the files are the same)

oh wait -

sorry, of course, we're using csv for zstd and tsv for txt

ah right that could be the reason

or did you use tabs in the end?

I think I ended up using tabs

yes, right. so I would expect these files to be identical

Hmm

Lemme load up a few files in python and cross check

I hope the difference is only limited to something trivial like row indices being written with the data

moin!

Pratha-Fish: at least the first 10 lines of the files are the same

oh, hmm. interesting

one sec

Pratha-Fish: right, so the original files have \r\n line terminators, and the ones that we generated have only \n

phew, that's less terrible than I thought

alastairp: Phew

pandas confirms that too

alastairp: does not having \r make a significant difference?

no, it just tends to appear more often on files created from windows

thankgod

in fact, I believe that the way that we are doing it is more correct

oh cool, there's a flag to `diff`:

diff --strip-trailing-cr <(zstdcat /home/snaek/MLHD/rec_track_checker/MLHD/0a/0a118981-15b5-46df-8666-080ca5a1af62.csv.zst) <(zcat /data/mlhd/0a/0a118981-15b5-46df-8666-080ca5a1af62.txt.gz)

that correctly outputs nothing

Nicee

Linux CLI is surprisingly powerful NGL. Makes me wanna switch back to arch

I just couldn't live with the constantly breaking system as a daily driver tbh

alastairp: I tests are passing on CB#445. I tries running them locally.

Remove script to update Bookbrainz Database: https://github.com/metabrainz/critiquebrainz/pu...

The*

Pratha-Fish: That’s what eventually pushed me to drop Windows for good. 🙃 IME Windows is as likely to break as Linux, but with Linux I at least have an idea of what’s going on and a fighting chance to fix it.

Is there any way to retest them on github before merging?

Freso: relatable haha but the opposite

The only reason why I am sticking with windows at this point is because of excellent software support, and force of habit

ansh: I can trigger it again

should be running again

yes it started running

alastairp: What should I do while the computation is running?

We could jump back on the artist conflation issue, or even start converting all pandas.isin() code to set queries

Pratha-Fish: I think that the next interesting step is going to be a comparison of our two data lookup methods

remember back at the beginning of the year when we were explaining that we might need to rewrite some lookup methods in spark or some other faster system?

right

so, given a recording mbid in the data file, we currently have 2 ways of looking up a canonical id: mbid -> canonical mbid table; or mbid -> text metadata -> mapper

and the previous experiment you did a few weeks back shows that some items give different results

what we're interested in doing is seeing why these results are different, and what we can do to make them the same

because ideally we could continue to use the canonical mbid table, because it's super fast (otherwise we need to look up all 27 billion rows in the mapper, which is slow)

the mapper method won't complete the computing this year tbh

so we need to decide if the mapper really is "better" (we don't know what the definition of better is here, we need to investigate the data and make a decision)

and if it _is_ better, we need to move on to the next steps of seeing if we can re-implement in something faster (spark? something else) in order to do the processing in a reasonable time

very interesting :D

So I'll take a look at the data first ig. Let's see if there's any patterns

Pratha-Fish: this dataset endpoint should be useful: https://labs.api.listenbrainz.org/explain-mbid-...

you give it a single artist and recording, and it'll return debugging about how it finds the item

Sounds good. I can try mapping the API to some data

moooin!

for any one who knows about Don Norman's "Design of Everyday things", but hasn't been able to read/finish it, these notes look quite cool" https://elvischidera.com/2022-06-24-design-ever...

mayhem: What a coincidence, I started reading that one today :))

[critiquebrainz] 14alastair merged pull request #445 (03master…remove_temp_script): Remove script to update Bookbrainz Database https://github.com/metabrainz/critiquebrainz/pu...

"Perceived affordances help people figure out what actions are possible without the need for labels or instructions."

this is the bullshit mentality that makes everything have icons and boxes now instead of CLEAR LABELS AND INSTRUCTIONS

I *LIKE* Labels and Instructions!!!!

aaaaaaaaaaaaaaaaa

.. but later on they say that a simple "you are offline" wouldsuffice as a notfication that connection is broken..

i'm confused

but in short: please label things with text, & write succinct instructions where needed." thanks

lucifer: https://github.com/metabrainz/listenbrainz-serv...

is now ready with PR feedback and missing test file added.

alastairp: The tests passed successfully after rebasing CB#441 last time. If there are any more changes required, pls let me know :)

CB-437: Add entity metadata to review get endpoints: https://github.com/metabrainz/critiquebrainz/pu...

mayhem: lgtm, thanks. it would be nice to add some tests with real mb data as well but currently we don't have MB db in LB tests so I'll open a ticket for it.

k

[listenbrainz-server] 14mayhem merged pull request #2065 (03master…add-upcoming-releases-backend): Add fresh releases backend https://github.com/metabrainz/listenbrainz-serv...

lucifer: is it time for us to chat about how to integrate the three separarate branches of fresh releases work we've got going on?

yes sure

the fetching of user specific data needs to be added to the endpoint I just added, that is one thing I see.

and now that I made space for the react work, chinmay can drop his work on top of the blank template that was just merged.

we have a couple of options there, either spark calls the api or lb fetches the data from db and sends it as a part of the rmq message

what else?

I was expecting for LB to fetch the data from couchdb/postgres.

and for the endpoint to return sidewide fresh releases unless a user name was given.

rest of the backend is almost done. most of the couchdb integration will be done when migrating stats to it. after that i'll finish the fresh releases pr.

ok, maybe we should just wait for that to be done before doing more stuff.

yes, that makes sense.

a few tests and dumps are pending on that front fwiw.

ok, ping me if you need anything. I'm going to see if I can classify tracks as high/low energy with the data we have at our disposal.... see if I can make another playlist for users.

will do. sounds great! :D

daily jams are making me pretty happy. looking quite nice now. I really need to make a point of listening to them each day to see how things shape up over time.

BP makes that pretty hard though. It plays a handful of tracks and then halts. :(

yeah spotify does not have any documentation on how to fix the issue and no one answered on forums either.

yeah, its fully meh.

I think I might try my hand at the spotify cache using couchdb as the document store.

hi lucifer, are you up?

riksucks: yes. sorry forgot to answer your question earlier. there are 2 things to consider here: 1) mulitple notifications on feed 2) allowing individual recommendee's to delete a personal notification they received without affecting others.

also maybe allow the recommender to unsend the recommendation to a particular person without unsending it to others?

for instance, Instagram allows you to send a post to multiple persons at a time but then you can unsend to a particular person later if you want.

true, I thought about the 2) one, and realised that in normal recommendation, only the recommender can delete it, and the recommendees can hide it from their timelines. So maybe we can implement a similar feature. Similarly for unsending for a particular person, we can try removing that specific ID from the array, and update it in the DB

yes that's possible. alternative option is to keep 1 row per user and instead group all the notifications by recording id.

mayhem, alastairp: thoughts on how to handle this: say a user sends a track recommendation to multiple people. should we create 1) 1 row per user or 2) 1 row with array containing all the users' ids.

2

in 1 its easier to handle deletes/hiding. but more work to manage notifications in feed. vice versa in 2.

alastairp: I've ran the explainer API on 333 rows of faulty data. Now how do we debug it?

also lucifer, I wanted to tell you another thing. I was reading up on how postgres handles JSONB, and what happens when we update certain keys or certain parts of that JSONB. Turns out, postgres always writes a new version of the whole row whenever we are updating. Do you think that would create overhead for lots of personal recommendation?