#metabrainz

hmm.. listenbrainz: Listen Count: 1,156,786 last.fm Scrobbles 1,157,888

what is getting eaten?

even if you take in account the stuff playing while I import (last thing is about 6 hours ago) it's still over 1000 songs

doing a reimport of the few pages afterthe fact

yields Listen Count: 1,156,866

so still over 1000 songs :/

how do i add the legal name of somebody using a alias

CatQuest: that is a bug and we need to look more closely. can you please file a bug?

dragonzeron: just set the "legal name" alias type when adding a new alias from the aliases tab

oh CatQuest, I just thought of one thing.

some of the data in last.fm is buggy and incomplete, for whatever reason.

and when it is incomplete, that data will get rejected by LB.

I could try and do an import onto my own machine and see where it complains.

some of the listens are getting detected as duplicates, it seems

oh, did you see some error messages?

even in my import, there are 9-10 listens less

no errors, just the duplicate count in influx-writer is sometimes >0

it would be good to dig into this further and verify what is going on.

yep

doing just that :)

<3

hi ferbncode. what are you up to?

alastairp: hi :), working on the PR, will update it for review by today

ok. any problems with mbdata?

no, no problems, the delay is just the result of my many stupid doubts I cleared yesterday :)

OK. remember that if you have questions you can ask them here too

listenbrainz_bigquery_1 exited with code 0

after leaving it running all night. not sure when it stopped

sure, will ask, thanks :)

alastairp: if you don't have write_to_bigquery on in config.py, it sleeps for a long time and then quits https://github.com/metabrainz/listenbrainz-serv...

that will be it

why?

(e.g., why not quit immediately?)

I think ruaok added a sleep because it helped debug some docker issue where credentials weren't getting loaded as a volume correctly

my goal was to prevent the container from restarting all the time if no big query credentials were present.

or having to edit the compose file.

this way it just quietly sits and shuts up.

I'm certainly not married to this method.

yeah, I've fought with docker restart policies before too :)

caught a bunch of listens we consider duplicates in my last.fm import

interesting. they look like valid duplicates, no?

how did you generate this? what import/reimport steps did you take?

I'd love some help getting me talk title/desc up to scracth

scratch, even.

ruaok: you namedrop me in an interview and don't tell me? :D

reosarevok: hey you! I dropped your name to get some street cred. Did you hear???

:-P

(just got a message from a guy I know "hey this post at the top of the subreddit I follow mentions you wtf" :D )

lol

ruaok: I did a simple last.fm import, just added a print to influx_writer wherever it thought a listen to be a duplicate

did you start with an empty DB?

yes

oh. well, that is a problem, then.

clearly there shouldn't be duplicates on a single import.

yeah, I dunno what the problem is, though

do we have an edge condition somewhere

?

>= vs > ?

which might causes listens at the end of a block to be duplicated/omitted?

I failed to import 2

out of 400 pages

could be, I haven't seen much of the scraper code in detail. Although, the data is a bit weird, I don't understand how a difference of 2 seconds between listens is possible. We couldn't be the ones doing that, because the last.fm api returns timestamps themselves, maybe the last.fm data has listens with differences less than 30 seconds.

I would have been more suspicious if I had failed to import 400 ;)

iliekcomputers: yeah, it would be worth doing a scrape of an API and checking the values of the timestamps

alastairp: it failed to import more than 1000 for CatQuest

alastairp: yup, that's what I was gonna do :)

CatQuest: how many pages do you have?

alastairp: I did a force push to the PR https://github.com/metabrainz/critiquebrainz/pu... (still WIP), it would be great to know if I am headed in the correct direction. :)

inb4 flame war: I've been using XFCE forever but I'm getting a bit tired of some of its issues. Out of the flavours Ubuntu does ship with, what's the one I should install when I reinstall it? :p

ferbncode: cool, I'll look this afternoon. thanks

honestly, gnome3 is way better than it was 6 years ago

I don't think anything annoys me on a day-to-day basis, but I installed quite a number of extensions to make it better

alastairp: great, thanks :). I used gnome an year ago, it was a heavy one for me, but then I switched to i3wm, superlight :P

weird feeling of the day: logging into quickbooks and having new companies appear there that I've never heard of or dealt with.

I <3 you Quesito!

so the weird data is from lastfm itself

Lol! Lots of new ones ruaok!

Quesito: :)

iliekcomputers: I kinda thought so. you and i had been scouring all aspects of data ingestion.

there is this one edge case that I think we might have missed in influx-writer, if there are duplicates in the same rabbitmq batch, it might not find out because the timestamps dict here doesn't contain that timestamp?

should we do a `timestamps[t] = result` here also? https://github.com/metabrainz/listenbrainz-serv...

interesting thought.

haha

!m Quesito

You're doing good work, Quesito!

(and thanks alastairp, maybe it is time to give gnome a chance again)

iliekcomputers: not sure I full understand your suggested solution, but yes we should cover that edge case.

ruaok: just adding every new timestamp in the rabbitmq to the dict which contains all the timestamps is what I suggested

that way, if there is another listen in the batch with the same timestamp it'll get detected and thrown out

*the rabbitmq batch

k, sounds good

:) thanks reosarevok!

is there a schema available for the json dumps?

the data contained in the JSON dumps is the same format as the data returned by our API:

[listenbrainz-server] paramsingh opened pull request #204: LB-180: Account for duplicates in same RabbitMQ batch for influx-writer (master...influx-writer/same-batch-dup) https://git.io/vQL0v

with the fmt=json flag, of course.