#metabrainz

Great :)

We can work on that tomorrow if you want

That sounds good

Project bookbrainz-site build #3165: passed in 3 min 33 sec: https://travis-ci.org/bookbrainz/bookbrainz-sit...

Just wanted to convey to you, I wasnt much active for 2 days.

Also hi everyone :)

No issue. Design takes time, and you needed to investigate the possibilities for this tricky issue

sorry for not being on top of this (sucky energy in the past few days) but which is the tricky issue you're trying to solve?

Sidebar navigation works great on desktop, but where do we put that in mobile and small screens?

got a link to a recent design so I can see?

See https://www.figma.com/file/PSXpqDEQ7Melwj14hBLZ...

thx

sidebar in this context is the left sidebar that contains the player, right?

jmp_music: what computer are you using? mac or linux? how much memory do you have, and if you're on a mac, how much memory have you assigned to docker?

Yes. And more importantly, the dashboard secondary navigation

I live and have seen mini player 2 in other apps.

(the player is easier to solve, see the mini player a thte bottom of the mobile view)

history | playlists | stats <- secondary navigation?

ok, kewl. yeah, that looks good to me.

Yes. With sub-items

ok, I see the difficulty. the most obvious solution is the "collapsed top nav" which I also dislike from bootstrap's menu.

Yeah. I suggested combining the two navs in mobile into that collapsed bootstrap menu thing

alastairp: At home I use a mac. The memory that is assigned to docker is at size of 2GB.

I'll assign more to it

jmp_music: and how many cores?

4

right, we have a known memory leak in this process (https://github.com/MTG/gaia/issues/96

Might be very tricky though, considering the top navbar is rendered by flask

what happens is that if you do too many iterations, the thread will run out of memory and stop accepting new training iterations

what Job did you get up to?

x/1728?

Oh ok! Yeap, x/1728

how many did it complete before it stopped responding?

oh - now I remember

edit your project_danceability.yaml file, and set `clusterMode: True`

then run it again

this will start a separate process for each combination, and so the memory will be freed at the end of each. that should fix the issue of it hanging, you won't need to increase memory if you don't have it available

Oh ok!

because I didn't remember the on which process it stopped

at the previous run

What is your suggestion about the memory I should assign to docker?

4gb would be ok?

sure. I asked because if you set it higher, it will process more items before running out of memory ;) I was trying to see if we can set it high enough to be able to finish all combinations before the memory leak gets too big

however clusterMode: True means that we no longer have to consider this problem

it's not a problem if you can't remember where it stopped. make these changes and try again

yeap, I did the change to the yaml file, and I assigned more memory to the docker.

let's see if that works

you can see how complicated it is to get gaia working... this is why we want to replace it!

True that! While it is a great ML system, I understand these reasons.

with `clusterMode: True`, continues exceptions were shown up in my latest runs, but with the larger memory assignment, the process finished

and the related .html was created

I tested the run with `className: danceability`

not `testdanceability`, and it finished the process

perfect!

so it should also output the parameters for the combination that got the highest accuracy

can you show an example of the exceptions that you get with clusterMode: True? That doesn't sound good

here are the parameters gaia chose for the danceability classification: https://usercontent.irccloud-cdn.com/file/xwznE...

there it is!

yes if course. Wait a sec to run it

so, now that we have those parameters, the question is: can we put those parameters into sklearn, and get the same 92.8% accuracy?

i think it could be done

I 'll try it

here is the traceback when clusterMode is True

but the process does not stop running

oh right. I wonder if this is the same problem with the name

I recommend that we go back a step - delete your results directory, and edit the grountruth file and the profile file to change the className

on my version I just called it `testdanceability`

then run it again and see if it completes

as profile file you mean the project yaml that is created by the template?

basically, this is what you meant! I just run it and it worked

'=D :')

I'm going to eat something, and I'm back i a few minutes

alastairp: the battery holders are on my desk in the office.

ruaok: thanks

jmp_music: right, again I told you to do it like this because there are some complexities....

you could edit your script to call train_model(... cluster_mode=True)

and then delete the results/ directory, delete the project file, edit groundtruth.yaml to change className, and then run it again

alastairp: ok! Finally, now the training process finished correctly without errors and exceptions.

Loading all results...[1728/1728] (100% done)... number of results: 1728

and not 576/576 as did it before

excellent

I saw you merged also the PR. Thanks

does anyone have any optimized indexer.ini settings for Live Indexing? I noticed that my server can run a full reindex in about 2.5 hrs but live indexing takes almost 24 hours

yvanzo would be the person who could help you, JoshDi

thank you ruaok

and now is the time of day when yvanzo might appear too

I have been working with yvanzo via the bug report on git: https://github.com/metabrainz/musicbrainz-docke...

:) why I am posting these questions right about now

heh.

yvanzo can you share a doc or a brief description what all of the settings in the indexer.ini do? here is my latest config: https://github.com/metabrainz/musicbrainz-docke...

I have 128gb of ram and 24 cores / 48 threads

daaaaamn.

lol yea

reosarevok: https://community.metabrainz.org/t/482032/22

I used an intel QL1K with 4x 32gb Samsung DDR 4 ECC ram. I also have a 70TB RAID6 with cachecade in front, and a 1TB bcache as well

pretty proud of this machine. built the whole thing (without the 70TB) for about 1K

bigger than anything we have in production, lol

same with my actual job lol

these intel ES processors are great if you know how to modify bios files to get them to work

if anyone is interested, I can help you build one. this processor as an ES version is only about 200-300 on ebay

ruaok ah didnt realize until now - hi Rob! good speaking with you this morning

hehehehehehehe. :)

was I pulling a lot of data via my token?

how often can I set my cron to pull replicating packets. Was thinking about changing it to every 3 hours versus every 24 hours

naw. the only thing that got me interested was your data usage description "for replication" or somesuch.

as you see fit, it matters not to us.

understandable :)

Marcel looks like a spammer: https://tickets.metabrainz.org/secure/ViewProfi...

iliekcomputers: ping

I've written the mogrifier and it writes one large file with all the spark formatted listens.

the previous dump breaks them into users. is that still needed?

ruaok: the user grouping isn't necessary. The directory structure (year/month.listens) is important though

great.

that is how the dumps are structured now, I'll just follow that method then.

yvanzo: maybe we should still hotfix? :p https://community.metabrainz.org/t/edits-for-yo...

(linked by chaban above)

also, are we able to generate a dump without a spark dump yet?

I'm now blocked on that.

iliekcomputers: https://gist.github.com/mayhem/3a53846912a93c69...

like dat?

written to the filesystem ok? or do you want tar/tar:xz ?

Tar.xz

k

wait, really?

xz is dog slow and I anticipate running this script *right* before an import to spark.

just tar would make one file without the wait for compression/decompression.

We could work with just filesystem tbh

Would just need some changes on the importer, which I'm happy to take

For just dumping the full dump, need to remove this line: https://github.com/metabrainz/listenbrainz-serv...

I think that makes sense to me -- a little bit of easy programming in exchanger for speed.

Sgtm too

ishaanshah: I'll have to push our meeting by 30 minutes today.

given that you've been leading managing deployment, can you please trigger a dump for me?

that will be the last sanity check. if everything checks out we can do the first attempt at a real migration.

Is it possible to use one of the more recent ones?

We have one from this Sunday on ftp

they wont have inserted_timestamps, will they?