#metabrainz

musicbrainz

Yeah got it. Its running

(from https://github.com/metabrainz/musicbrainz-docke... )

Yeah, noticed that just as I asked

Its in progress now, should be done in 10ish minutes

yvanzo From the looks of it, its working now. I think when we run the command with --fetch, it is not creating the musicbrainz_db database, but when you run it without, it does.

I'll have the logs for you in a second

This is the logs with --fetch (https://pastebin.com/2tpDGN11) If you want the one for without as well I can get it for you. Its very very long though

Indeed, it is a recent regression, sorry for that, fixed now.

ishaanshah[m]: the PR is ready for review again.

i'm testing the import manually right now and everything seems stable

let's merge and deploy this to beta today

[listenbrainz-server] paramsingh merged pull request #786 (master…param/api-service-ts): Convert API service, last fm importer and tests to typescript https://github.com/metabrainz/listenbrainz-serv...

typescript running on the last.fm importer on beta.listenbrainz.org/import now, would appreciate if anyone here could test

iliekcomputers: Nice! 4 more files to go

nice indeed :D

i feel like we'll probably knock those out by monday too

Should I start working on follow users?

i think you should work on the spotify player while i look at follow user and the recent listens

i have more context on what exactly follow etc is supposed to do and a lot of it is not very obvious / bad UX I wrote

:D

Cool

also, i think it might be a good idea to just add tests while we're at it too

ran import on b.lb.o, no errors (also no imports since I switched to Spotify)

i overestimated how much work it would be to convert i suppose

Do I need a pro account to make spotify work?

don't suppose there's an option to only delete imported last.fm listense?

Zastai: None as of now

Zastai: thanks for testing!

Zastai: None till now

would not mind re-importing but don't necessarily want to discard my existing Spotify listens to do so

nvm I can reset the ate

date*

That might lead to duplicates though

ishaanshah[m]: about premium, i think so, yes.

Because the structure for lastfm listens was changed

Oh, I use a free account

ah

ishaanshah[m]: Its a chance though. Sometimes t works without premium. But rarely

Page 3 of 4750 and counting :)

nvm then. you do recent listens, i'll take follow and the player

ishaanshah[m]: I also decided on a filename convention in the last pull request

Yep saw that Pascal Case right

ReactComponent.js ReactComponent.test.js

yes

let's move to that as well

also

i think the RecentListens component is in a file called profile.jsx

it should ideally be in RecentListens.tsx

I guess this means my l.fm listens will now have mbids too, if I read the recent updates correctly

Yep, I will change that

Zastai: Yes, artist MBID and release MBIDs have been added

Zastai: we started storing the artist mbids but in a seperate field from normal mbids because of accuracy issues.

we haven't changed the frontend to show those yet, however

ah that's good to know

as in db only, or shown in api just not website?

The trackMBIDs are still to be fixed though

it should show up in the API

not the website

ok - means I may need to adjust my .NET bindings to handle em :)

the fields aren't documented rn, should be something like artist_mbid_lastfm in the additional info

iliekcomputers: I will update the frontend to link them while porting

Maybe add an astrix and tell about possible inacurracies

ishaanshah[m]: i would like to do it in a seperate pull request

I assume describing them as "MBID for the listened track's artist, as determined from a Last.fm import (may not be completely accurate)." is good?

Zastai: yes!

exactly

iliekcomputers: Yep that makes sense, I will try to finish the port by tonight and open a PR

which other MBIDs are planned to be added in a similar fashion (so I can already set up my bindings to recognise them)?

no plans as of now. last.fm only gives us artist, release and track and we've started importing those

always one, or could there be a artist_mbids_lastfm too?

always one, iirc

ok

we have a list of clients and bindings on listenbrainz.org, if you wanna open a PR adding your bindings there :D

incidentally does the db also contain a "reference user" with a static set of listens? would be nice to have for testing purposes.

right, I currently have them only linked on the MB wiki, not for LB yet

Zastai: not right now. I think solving this by creating a test.LB makes more sense, but we haven't had the capacity to do that.

true - but creating a SampleUser (or whatever) user in the db is probably very cheap compared to a test.lb.o setup; and can be easily dropped once test.lb.o exists

(import at page 920 of 4750 and ticking along)

(So you have around 2 lakh listens. Nice)

that is true. I'm not sure how it would work for stuff that needs authentication however. Technically, we could add logic for a particular user to take any auth token, i guess.

and a cron job resetting the user at regular intervals

i'll open a ticket, this is a nice idea.

LB-517: Create a SampleUser that can be used by developers to test API integrations

might want two users then - for unit testing it would be good to have an invariant user as well (so that you know exactly how many and which listens result from a query with a particular min_ts), plus that sandbox user (I set up a local lb server essentially just to be able to test submissions)

by the way is there a technical limitation preventing the use of both min_ts and max_ts at the same time? seems convenient to me to be able to ask for "what did I listen to in March 2010?" without having to keep reading until I hit a listen in April (or beyond)

i can't think of any right now.

yeah, nope, just seems like some old decision maybe

i'll open another ticket! :D

was about to ask if I should, but if you're doing it already, that's great :)

3000/4750 and counting

I seem to see that the additional info for imports always has 13 properties set, even if they're null. are those "guaranteed" properties then? seems wasteful to send '"release_group_mbid": null' over the line

(this is a general issue I have with MeB JSON responses - there seems to be no standard for how "not set" is handled; sometimes the property is omitted, sometimes it's present but null. feels like that's something that could be standardized)

also: is the import page capable of showing the timestamps involved? 3500/4750 is nice for tracking progress, but it does not tell me which timestamp it got to so far, so I can't tell whether a query gives me an old or new import

yes, that is possible.

I mean, sure, I can wait until the import is done

this is pretty valuable feedback Zastai, thanks! I'll open a ticket for both of those things

I am not sure about the additional info properties, that is something I'll have to look into

hehe, sorry for being a pain :)

no, not at all! better to have users asking for improvements than it not being used at all :D

a lot of these are really good first bugs, so they'll be easier to just give out to new contributors as well :D

just as an example, a query I was just testing in LinqPad returned 25 listens. json payload was 14975 bytes. after stripping properties that had value null or [], that becomes 11006. more than 25% reduction in bandwidth for what I assume to be very little processing effort

4400/4750 - nearly there

Zastai: For the release_group_mbid I think its because of the influx data storage structure. Some fields were explicitly extracted out for storage

I guess this problem will be solved in the Timescale DB migration the team is working on because we wont we explicitly pulling out the fields like this for storage. We will be storing the track_metadata as it comes in payload

Although the necessaey ones will stay the same

import done - 237434 listens

the "done" page overflows the "popup" though

and it indeed looks like I'm getting dupes - in the range of this query, ALL listens are dupes - they alternate between new ones (17 additional info fields) and old ones (13 additional info fields)

the new ones add dedup_tag=1,listening_from=lastfm and lastfm_{artist,release}_mbid

It seems that a page wasn't imported correctly

Maybe rate limit was hit

i'd assume the import either bypasses or honors the rate limiting

Can you share your you lastFM profile

Zastai

It does that for lb api but not for lastfm api

ahh their rate limit

I will take a look

my lb profile now says I have 457,926 listens - pretty sure I did not listen to over 200,000 tracks since dropping my scrobbling from the spotify client this year

is the dedup_tag something that should be there in API results? it sounds like an internal field

(but i guess if the importer uses the api, it needs it)

Zastai: Did you delete your listens before importing?

no

Just reset the timestamp for last.fm?

as I said above, I wanted to avoid losing my post-last-fm listens, so I just reset my import timestamp

seems a bit strange that it happily creates new listens with the same timestamp, artist, track and release

So there were gonna be duplicates. iliekcomputers Do we check for duplicates while inserting it inDB?

the recording MSID has changed though - but that could just be the import creating new messybrainz entries

(the import reset info does say reimporting should avoid dupes. looks like it failed in 80%+ of cases, so that wording should perhaps be "might avoid dupes" :) )

Zastai: As I checked my imports the uniqueness is checked. But as you reported the recording_msid being changed that might have caused duplicates

Seems a strange issue. No dupes formed for me

shivam-kapila: If you deleted your listens before and reimported then you wont get duplicates

As I said before, some fields were added recently which makes both the listens different

No I didnt delete

I use two lb accounts for the same purpose :p. Testing!!