#metabrainz

armalcolite, alastairp: ping

what's "postgres: checkpointer process"?

taking up over 10G of memory on spike

ruaok: pong!

ah, looks like it's WAL stuff

not sure if it's normal though

alastairp: should we mark these as "failed" at this point https://beta.acousticbrainz.org/datasets/list/r...

ruaok: When was it you'd be heading this way?

modwizcode: Who are you doing GSoC with?

alastairp: we also need to define service properly for dataset evaluation. currently it's running as your user

there are a ton of `python evaluate.py` processes. I have no idea what they are doing

Freso: 18th

armalcolite: still here?

yeah

ruaok: Okay. I plan on getting the mattress today or this week.

armalcolite: I'm really sorry that I didn't get around to finishing reviewing the current PRs.

Gentlecat: I just pulled the latest code for AB server and a lot of update files are added. Is there a way I can run them one by one in the order of their addition?

ruaok: np.

so we have some kind of function to update the databases?

*do

no, you have to run them manually

I'm headed out for the weekend -- back late sunday, realistically monday.

ah, ok.

those threw me and I lost track then.

ruaok: oh. it would be nice if we can devise a rough plan for the weekend

that is my goal.

i pushed a few tests yesterday

given how close you are to your overall goal, i would like to work on a performance improvement for the postgres setup.

that is good idea, its very important as well.

the theory is that we want to keep the most important data in RAM.

so, then the idea is to split the listen table in listen and listen_json.

put the json into the listen table and then time and username into listen table.

so all queryable bits are in the listens table, all the heavy data is in the listens_json table.

with a foreign key between them.

so just move the json column to listen_json table?

the acousticbrainz project does this -- so if you have any questions both Gentlecat and alastairp can help you while I am gone.

raw_data -> listen_json

sure.

and the other column will be a listen_id foreign key to listen.id

this should allow the fetching of listens to be faster.

does this make sense?

our idea is to reduce the date being fetched in each attempt so as to reduce load.

I would suggest that you make a gist with the proposed SQL changes and get alastairp or Gentlecat to give you feedback on them before you commit them to git

yeah, its making sense.

yes, pretty much that.

reduce memory footprint is a better way of putting it.

sure, i'll gather their help before making final commits

the idea is to keep all of the listen table in memory and then fetch data from listen_json as needed.

look at the lowlevel and lowlevel_json table.

yeah, it makes sense

of course, then the code in liststore needs to change to handle this new table format.

just a question, as of now we are not using any data from the "raw_data" (which is just a json)

so is the plan to use it in future?

while displaying statistics

we're using it to display the "your listens"

but yes, we're going to do it for statistics and such.

also, if you run out of things to do, please add more tests. test everything.

ruaok: i added a few tests

can u have a look to make sure they are good enough so that i can keep adding 'em

As ruaok said, if you have any specific questions this week, ask me. I'll try and answer them for you

thanks alastairp

armalcolite: ideally the tests would be a little more detailed.

alastairp: sure.

ruaok: detailed, as in?

insert multiple listens complete listens.

try to insert bad data and expect it to catch the error.

insert good data and expect it to accept the data.

oh, yeah. i will do some robust tests as well

also, for each page that we serve, we should at least be able to try and load each page.