#metabrainz

ohh, the LB tweet was retweeted by someone with 2.6k followers https://twitter.com/iconoclaste

Moin!

moiin!

thats the dumb recommender running.

unzipping lb data now to try the load_data

ruaok: ferbncode: we still need to make the dataframes persistent

yeah, I noticed that the script trailed off a bit, but I wanted to start playing with it anyway.

the old recommendation training ran 10 minutes on the cluster. O_O

yike

yikes

yeah. and that was with 4 * 27GB of ram.

and the LB dump appears corrupt. I can reproduce mineo's problem.

iliekcomputers: the other LB dump is corrupt too. :(

we need to fix it so that it won't recommend tracks you've got in your listen history.

We subtract user tracks from all tracks already: https://github.com/metabrainz/listenbrainz-reco...

not sure why that isn't working

maybe it is recommending different versions of the same recording?

maybe you didn't listen to the track in 2017?

ah! great observation!

the latest data dump inside the LB container is not corrupt.

copying that to the master.

LordSputnik: The tests on bb-data indeed fail. There's been a schema change in last two years?

Here's a gist: relation "bookbrainz._editor_entity_visits" does not exist

column "link_phrase" of relation "relationship_type" does not exist

(and so does reverse_link_phrase)

Same with "editor_type"

Nopes, not editor_type, sorry.

Initially ran into some permission issues, thought that were causing it - but I added all permissions.

ruaok: did it stop with an error?

iliekcomputers: ferbncode: yep.

out of heap space trying to import a user with 28k listens.

I think we need to brainstorm how to refactor the import to not have to do much work on the client

first idea that comes to mind is

we modify the listenbrainz dumps code to add username to the listen json doc

after that create a new dump and load it directly using sc.textFile

doh. that's a great idea.

given that we're running into this problem, others will have the same problem.

let's do that.

alright, on it

bukwurm: https://gist.github.com/LordSputnik/059d95f91b4...

That's the data the client will send to the server for relationships (from the edition editor)

LordSputnik: Ok

LordSputnik: I relationships I have (got through the pg_dump and it's same with the schameSpy diagram on readthedocs) doesn't have linkphrase?

How do I update it?

Mineo: well spotted, the import runs faster with that correct. :)

bukwurm: let me check what's going on

https://github.com/bookbrainz/bookbrainz-sql/bl... is what's correct

iliekcomputers: we should still proceed with the LB dump change -- that will still make this faster.

but now it dies with an assertion errore:

LordSputnik: https://github.com/bookbrainz/bookbrainz-sql/co...

bukwurm: ok, so the relationship thing is fairly easy - that's part of the new relationship editor stuff and not yet applied on the server

So you'll need to run the migration scrip

LordSputnik: Ok

"up.sql" will upgrade you there

LordSputnik: Ok

LordSputnik: Also, one side question.

And you're right that there's no _editor_entity_visits on the server, that'll need looking into

bookbrainz_test is using same schema and sequences as bookbrainz db

This is causing "insert into "bookbrainz"."relationship_set" ("id") values ($1) returning "id" - duplicate key value violates unique constraint "relationship_set_pkey""

Any idea how to fix it? I am not well versed with postgres

But I get the notion that sequence of bookbrainz db is pointing to the next value to be inserted with few relations

And as the bookbrainz-data relations (tables) are still empty, this error comes up

bukwurm: is that during tests?

Yes!

That usually means that the tables haven't been truncated properly

That will only happen when we specify an ID when inserting data, and the ID already exists in the table

LordSputnik: Ok

Let me check

If you rerun the test when the relationship issue is fixed, the truncation stuff should work properly :)

Ok that's written in the err message itself :P

LordSputnik: Ok that great!

On it!

(we should also change the tests so that truncation doesn't break when something else breaks)

Ok

iliekcomputers: ferbncode: more data: https://gist.github.com/mayhem/6f714ab5d1a312ac...

LordSputnik: I'll look into it too. :D

[listenbrainz-server] paramsingh opened pull request #397: Add username to listen json docs in dumps (production...usernames-in-dumps) https://git.io/vxKNM

LordSputnik: Probably should update the schemaSpy diagram on the docs. That's a lifesaver for beginners!

bukwurm_: definitely, that applies to everything on the docs really!

bukwurm_: aha! So I think _editor_entity_visits used to exist. Then we restored the database from a dump, and it's not in the dumps (it's personal info), so we lost it

[listenbrainz-server] mayhem closed pull request #397: Add username to listen json docs in dumps (production...usernames-in-dumps) https://git.io/vxKNM

bukwurm_: I've re-added it now, but you'll want to reflect that in your own copy by using the last few lines of https://github.com/bookbrainz/bookbrainz-sql/bl...

LordSputnik: Ok

LordSputnik: Does BB-site has functionality to add explorer achievement?

Yup

I'm not sure why there was no error

Weird

LordSputnik: Another thing, bb-site uses unminified react

Is it by design?

LordSputnik: hey :)

Project bookbrainz-data-js build #753: passed in 3 min 26 sec: https://travis-ci.org/bookbrainz/bookbrainz-dat...

bukwurm: well, it should get minified along with everything else in the build process

I think that's merged now, maybe not updated on the server