Its in progress now, should be done in 10ish minutes
2020-04-11 10257, 2020
DjSlash has quit
2020-04-11 10218, 2020
DjSlash joined the channel
2020-04-11 10227, 2020
Darkloke has quit
2020-04-11 10209, 2020
Xellos
yvanzo From the looks of it, its working now. I think when we run the command with --fetch, it is not creating the musicbrainz_db database, but when you run it without, it does.
2020-04-11 10225, 2020
Xellos
I'll have the logs for you in a second
2020-04-11 10237, 2020
Xellos
This is the logs with --fetch (https://pastebin.com/2tpDGN11) If you want the one for without as well I can get it for you. Its very very long though
2020-04-11 10236, 2020
yvanzo
Indeed, it is a recent regression, sorry for that, fixed now.
2020-04-11 10246, 2020
D4RK-PH0ENiX has quit
2020-04-11 10200, 2020
D4RK-PH0ENiX joined the channel
2020-04-11 10241, 2020
iliekcomputers
ishaanshah[m]: the PR is ready for review again.
2020-04-11 10256, 2020
iliekcomputers
i'm testing the import manually right now and everything seems stable
2020-04-11 10205, 2020
iliekcomputers
let's merge and deploy this to beta today
2020-04-11 10237, 2020
Darkloke joined the channel
2020-04-11 10223, 2020
BestSteve has quit
2020-04-11 10259, 2020
BestSteve joined the channel
2020-04-11 10249, 2020
BrainzGit
[listenbrainz-server] paramsingh merged pull request #786 (master…param/api-service-ts): Convert API service, last fm importer and tests to typescript https://github.com/metabrainz/listenbrainz-server…
2020-04-11 10210, 2020
iliekcomputers
typescript running on the last.fm importer on beta.listenbrainz.org/import now, would appreciate if anyone here could test
2020-04-11 10202, 2020
ishaanshah[m]
iliekcomputers: Nice! 4 more files to go
2020-04-11 10215, 2020
iliekcomputers
nice indeed :D
2020-04-11 10239, 2020
iliekcomputers
i feel like we'll probably knock those out by monday too
2020-04-11 10258, 2020
iliekcomputers feels productive over 4 day weekend
2020-04-11 10253, 2020
ishaanshah[m]
Should I start working on follow users?
2020-04-11 10242, 2020
iliekcomputers
i think you should work on the spotify player while i look at follow user and the recent listens
2020-04-11 10211, 2020
iliekcomputers
i have more context on what exactly follow etc is supposed to do and a lot of it is not very obvious / bad UX I wrote
2020-04-11 10214, 2020
iliekcomputers
:D
2020-04-11 10229, 2020
ishaanshah[m]
Cool
2020-04-11 10235, 2020
iliekcomputers
also, i think it might be a good idea to just add tests while we're at it too
2020-04-11 10238, 2020
Zastai
ran import on b.lb.o, no errors (also no imports since I switched to Spotify)
2020-04-11 10250, 2020
iliekcomputers
i overestimated how much work it would be to convert i suppose
2020-04-11 10253, 2020
ishaanshah[m]
Do I need a pro account to make spotify work?
2020-04-11 10255, 2020
Zastai
don't suppose there's an option to only delete imported last.fm listense?
2020-04-11 10220, 2020
ishaanshah[m]
Zastai: None as of now
2020-04-11 10220, 2020
iliekcomputers
Zastai: thanks for testing!
2020-04-11 10230, 2020
shivam-kapila
Zastai: None till now
2020-04-11 10232, 2020
Zastai
would not mind re-importing but don't necessarily want to discard my existing Spotify listens to do so
2020-04-11 10251, 2020
Zastai
nvm I can reset the ate
2020-04-11 10254, 2020
Zastai
date*
2020-04-11 10225, 2020
ishaanshah[m]
That might lead to duplicates though
2020-04-11 10238, 2020
shivam-kapila
Importer working sleekly
2020-04-11 10240, 2020
iliekcomputers
ishaanshah[m]: about premium, i think so, yes.
2020-04-11 10246, 2020
ishaanshah[m]
Because the structure for lastfm listens was changed
2020-04-11 10205, 2020
ishaanshah[m]
Oh, I use a free account
2020-04-11 10211, 2020
iliekcomputers
ah
2020-04-11 10216, 2020
shivam-kapila
ishaanshah[m]: Its a chance though. Sometimes t works without premium. But rarely
2020-04-11 10232, 2020
Zastai
Page 3 of 4750 and counting :)
2020-04-11 10238, 2020
iliekcomputers
nvm then. you do recent listens, i'll take follow and the player
2020-04-11 10255, 2020
iliekcomputers
ishaanshah[m]: I also decided on a filename convention in the last pull request
2020-04-11 10215, 2020
ishaanshah[m]
Yep saw that Pascal Case right
2020-04-11 10216, 2020
iliekcomputers
ReactComponent.js ReactComponent.test.js
2020-04-11 10225, 2020
iliekcomputers
yes
2020-04-11 10235, 2020
iliekcomputers
let's move to that as well
2020-04-11 10237, 2020
iliekcomputers
also
2020-04-11 10251, 2020
iliekcomputers
i think the RecentListens component is in a file called profile.jsx
2020-04-11 10201, 2020
iliekcomputers
it should ideally be in RecentListens.tsx
2020-04-11 10222, 2020
Zastai
I guess this means my l.fm listens will now have mbids too, if I read the recent updates correctly
2020-04-11 10227, 2020
ishaanshah[m]
Yep, I will change that
2020-04-11 10203, 2020
ishaanshah[m]
Zastai: Yes, artist MBID and release MBIDs have been added
2020-04-11 10205, 2020
iliekcomputers
Zastai: we started storing the artist mbids but in a seperate field from normal mbids because of accuracy issues.
2020-04-11 10218, 2020
iliekcomputers
we haven't changed the frontend to show those yet, however
2020-04-11 10226, 2020
Zastai
ah that's good to know
2020-04-11 10238, 2020
Zastai
as in db only, or shown in api just not website?
2020-04-11 10246, 2020
ishaanshah[m]
The trackMBIDs are still to be fixed though
2020-04-11 10251, 2020
iliekcomputers
it should show up in the API
2020-04-11 10254, 2020
iliekcomputers
not the website
2020-04-11 10218, 2020
Zastai
ok - means I may need to adjust my .NET bindings to handle em :)
2020-04-11 10224, 2020
iliekcomputers
the fields aren't documented rn, should be something like artist_mbid_lastfm in the additional info
2020-04-11 10249, 2020
ishaanshah[m]
iliekcomputers: I will update the frontend to link them while porting
2020-04-11 10214, 2020
ishaanshah[m]
Maybe add an astrix and tell about possible inacurracies
2020-04-11 10230, 2020
iliekcomputers
ishaanshah[m]: i would like to do it in a seperate pull request
2020-04-11 10233, 2020
Zastai
I assume describing them as "MBID for the listened track's artist, as determined from a Last.fm import (may not be completely accurate)." is good?
2020-04-11 10249, 2020
iliekcomputers
Zastai: yes!
2020-04-11 10251, 2020
iliekcomputers
exactly
2020-04-11 10240, 2020
ishaanshah[m]
iliekcomputers: Yep that makes sense, I will try to finish the port by tonight and open a PR
2020-04-11 10253, 2020
Zastai
which other MBIDs are planned to be added in a similar fashion (so I can already set up my bindings to recognise them)?
2020-04-11 10225, 2020
iliekcomputers
no plans as of now. last.fm only gives us artist, release and track and we've started importing those
2020-04-11 10242, 2020
Zastai
always one, or could there be a artist_mbids_lastfm too?
2020-04-11 10202, 2020
iliekcomputers
always one, iirc
2020-04-11 10207, 2020
Zastai
ok
2020-04-11 10249, 2020
iliekcomputers
we have a list of clients and bindings on listenbrainz.org, if you wanna open a PR adding your bindings there :D
2020-04-11 10250, 2020
Zastai
incidentally does the db also contain a "reference user" with a static set of listens? would be nice to have for testing purposes.
right, I currently have them only linked on the MB wiki, not for LB yet
2020-04-11 10251, 2020
iliekcomputers
Zastai: not right now. I think solving this by creating a test.LB makes more sense, but we haven't had the capacity to do that.
2020-04-11 10206, 2020
Zastai
true - but creating a SampleUser (or whatever) user in the db is probably very cheap compared to a test.lb.o setup; and can be easily dropped once test.lb.o exists
2020-04-11 10258, 2020
Zastai
(import at page 920 of 4750 and ticking along)
2020-04-11 10208, 2020
shivam-kapila
(So you have around 2 lakh listens. Nice)
2020-04-11 10211, 2020
iliekcomputers
that is true. I'm not sure how it would work for stuff that needs authentication however. Technically, we could add logic for a particular user to take any auth token, i guess.
2020-04-11 10216, 2020
iliekcomputers
and a cron job resetting the user at regular intervals
LB-517: Create a SampleUser that can be used by developers to test API integrations
2020-04-11 10229, 2020
Zastai
might want two users then - for unit testing it would be good to have an invariant user as well (so that you know exactly how many and which listens result from a query with a particular min_ts), plus that sandbox user (I set up a local lb server essentially just to be able to test submissions)
2020-04-11 10240, 2020
Zastai
by the way is there a technical limitation preventing the use of both min_ts and max_ts at the same time? seems convenient to me to be able to ask for "what did I listen to in March 2010?" without having to keep reading until I hit a listen in April (or beyond)
2020-04-11 10246, 2020
iliekcomputers
i can't think of any right now.
2020-04-11 10232, 2020
iliekcomputers
yeah, nope, just seems like some old decision maybe
2020-04-11 10242, 2020
iliekcomputers
i'll open another ticket! :D
2020-04-11 10208, 2020
Zastai
was about to ask if I should, but if you're doing it already, that's great :)
2020-04-11 10241, 2020
Zastai
3000/4750 and counting
2020-04-11 10245, 2020
Zastai
I seem to see that the additional info for imports always has 13 properties set, even if they're null. are those "guaranteed" properties then? seems wasteful to send '"release_group_mbid": null' over the line
2020-04-11 10251, 2020
Zastai
(this is a general issue I have with MeB JSON responses - there seems to be no standard for how "not set" is handled; sometimes the property is omitted, sometimes it's present but null. feels like that's something that could be standardized)
2020-04-11 10253, 2020
jbs1 joined the channel
2020-04-11 10212, 2020
Zastai
also: is the import page capable of showing the timestamps involved? 3500/4750 is nice for tracking progress, but it does not tell me which timestamp it got to so far, so I can't tell whether a query gives me an old or new import
2020-04-11 10235, 2020
iliekcomputers
yes, that is possible.
2020-04-11 10237, 2020
Zastai
I mean, sure, I can wait until the import is done
2020-04-11 10259, 2020
iliekcomputers
this is pretty valuable feedback Zastai, thanks! I'll open a ticket for both of those things
2020-04-11 10216, 2020
iliekcomputers
I am not sure about the additional info properties, that is something I'll have to look into
2020-04-11 10225, 2020
Zastai
hehe, sorry for being a pain :)
2020-04-11 10208, 2020
iliekcomputers
no, not at all! better to have users asking for improvements than it not being used at all :D
2020-04-11 10220, 2020
iliekcomputers
a lot of these are really good first bugs, so they'll be easier to just give out to new contributors as well :D
2020-04-11 10217, 2020
Zastai
just as an example, a query I was just testing in LinqPad returned 25 listens. json payload was 14975 bytes. after stripping properties that had value null or [], that becomes 11006. more than 25% reduction in bandwidth for what I assume to be very little processing effort
2020-04-11 10216, 2020
Zastai
4400/4750 - nearly there
2020-04-11 10259, 2020
shivam-kapila
Zastai: For the release_group_mbid I think its because of the influx data storage structure. Some fields were explicitly extracted out for storage
I guess this problem will be solved in the Timescale DB migration the team is working on because we wont we explicitly pulling out the fields like this for storage. We will be storing the track_metadata as it comes in payload
and it indeed looks like I'm getting dupes - in the range of this query, ALL listens are dupes - they alternate between new ones (17 additional info fields) and old ones (13 additional info fields)
2020-04-11 10238, 2020
Zastai
the new ones add dedup_tag=1,listening_from=lastfm and lastfm_{artist,release}_mbid
2020-04-11 10248, 2020
ishaanshah[m]
It seems that a page wasn't imported correctly
2020-04-11 10213, 2020
ishaanshah[m]
Maybe rate limit was hit
2020-04-11 10250, 2020
Zastai
i'd assume the import either bypasses or honors the rate limiting
2020-04-11 10250, 2020
ishaanshah[m]
Can you share your you lastFM profile
2020-04-11 10257, 2020
Zastai
Zastai
2020-04-11 10215, 2020
ishaanshah[m]
It does that for lb api but not for lastfm api
2020-04-11 10227, 2020
Zastai
ahh their rate limit
2020-04-11 10232, 2020
ishaanshah[m]
I will take a look
2020-04-11 10225, 2020
Zastai
my lb profile now says I have 457,926 listens - pretty sure I did not listen to over 200,000 tracks since dropping my scrobbling from the spotify client this year
2020-04-11 10258, 2020
Zastai
is the dedup_tag something that should be there in API results? it sounds like an internal field
2020-04-11 10227, 2020
Zastai
(but i guess if the importer uses the api, it needs it)
2020-04-11 10227, 2020
shivam-kapila
Zastai: Did you delete your listens before importing?
2020-04-11 10232, 2020
Zastai
no
2020-04-11 10253, 2020
shivam-kapila
Just reset the timestamp for last.fm?
2020-04-11 10254, 2020
Zastai
as I said above, I wanted to avoid losing my post-last-fm listens, so I just reset my import timestamp
2020-04-11 10224, 2020
Zastai
seems a bit strange that it happily creates new listens with the same timestamp, artist, track and release
2020-04-11 10240, 2020
shivam-kapila
So there were gonna be duplicates. iliekcomputers Do we check for duplicates while inserting it inDB?
2020-04-11 10216, 2020
Zastai
the recording MSID has changed though - but that could just be the import creating new messybrainz entries
2020-04-11 10230, 2020
Zastai
(the import reset info does say reimporting should avoid dupes. looks like it failed in 80%+ of cases, so that wording should perhaps be "might avoid dupes" :) )
2020-04-11 10226, 2020
shivam-kapila
Zastai: As I checked my imports the uniqueness is checked. But as you reported the recording_msid being changed that might have caused duplicates
2020-04-11 10212, 2020
shivam-kapila
Seems a strange issue. No dupes formed for me
2020-04-11 10254, 2020
ishaanshah[m]
shivam-kapila: If you deleted your listens before and reimported then you wont get duplicates
2020-04-11 10228, 2020
ishaanshah[m]
As I said before, some fields were added recently which makes both the listens different
2020-04-11 10204, 2020
shivam-kapila
No I didnt delete
2020-04-11 10228, 2020
shivam-kapila
I use two lb accounts for the same purpose :p. Testing!!
