#metabrainz

thank you yvanzo

40Gb shared buffers deployed. now to wait.

zas: we're working on some patches that should remove the need for complex sorts from the majority of our queries

which should reduce disk writes

however it's true that they might come back as we get more load, so trying to give more resources would be good

machine is powerful already, db on SSD would be faster ofc, but since the db is big (and get bigger) it would cost a lot of more

and i have the feeling the server doesn't get that many queries atm, and is under performing

the data returned in a single query is large - 100s of kb to MB depending on what query

[bookbrainz-site] MonkeyDo opened pull request #283 (master…hide-deprecated-relationships): Hide deprecated relationships in relationship editor https://github.com/bookbrainz/bookbrainz-site/p...

Project bookbrainz-site build #2164: passed in 3 min 38 sec: https://travis-ci.org/bookbrainz/bookbrainz-sit...

alastairp: did you measure at which rate the db is growing ? i did a rough estimate of 50gb/month (based on disk space, so perhaps wrong)

no, but we could estimate it based on average data size and rate of growth of number of rows

can you do that ? if we need to throw more resources it will be useful. Perhaps time to think about scalability of this project

ruaok: last change looks rather positive, still need more stats to be sure though

Oh, good. I guess the caches are warm now.

btw, what's the traffic on eth0 atm ?

Akf atm. Will look when home again.

ftp download

Sigh. Everytime i collect a parcel, it's not a matter of picking it up, but some great mystery the staff needs to solve.

Mr_Monkey: still at office?

alastairp: Yep

Where do you want to meet?

Oh !

yvanzo: You around for a bit ?

yup!

I've been trying to figure out how to connect my redis container to work as a redis-server for my server container. Previously when i did it manually I used to run the command `redis-server` in a new terminal

But I cant do the same with compose... This is probably one of the issues Im facing before which my compose-file can be completed

I tried changing the values in `DBDefs.pm` so now my server pings to redis:6379 but it doesnt do the job yet

Here is the log https://www.irccloud.com/pastebin/wSuqEVzH/

yvanzo: any idea why labels don't seem to be editable anymore from the ticket in jira?

(like, components has click to edit but labels does not)

reosarevok: you might not have permissions, I will check

yvanzo: could you look into my query too please

Cyna: what is your query?

yvanzo: CatQuest needs them for INST

(the permissions)

I mean, I could use them too but that's less important

reosarevok: I added you to the jira-developers group

Above reosarevok question

around 3 to 4 messages

sigh

yvanzo: please make sure CatQuest can use them too, at least for the INST project :)

:)

zas: you were right, no improvement for frank with more shared_buffers.

I can try work_mem 256MB, but I dunno.

Cyna: but what is your setup?

reosarevok: oops, ok, checking that now.

its the basic musicbrainz server setup

This is my docker-compose with Dockerfle

ruaok: I think there is a small improvement, cpu load decreased a bit, but hard to be sure, traffic is varying a lot

Cyna: ok, so far so good, it is very close to https://github.com/metabrainz/musicbrainz-docke...

BTW arriving at Sants in few minutes

Well I couldnt successfully run musicbrainz-docker

Where do we meet?

This is my own docker compose in which I know what every command does and its the minimal setup required to setup a dev or test environment

Cyna: do you really need that networking stuff in compsoe?

Well to communicate between containers I do need to create a network, dont I >

*?

Yes but there is a default network already.

I missed adding networks to server container config

Cyna: exactly, that might be the bug :)

I've replaced and started composing it... Bounce back in 20 mins

(or you can remove network musicbrainz instead)

Damn... it takes atleast 10 mins to get all the containers set up

mainly because of cpamn

2019-05-09 17:35:42.622 GMT @ 2514 LOG: temporary file: path "base/pgsql_tmp/pgsql_tmp2514.2", size 63991560

64MB temp files. how the hell?

alastairp: https://gist.github.com/mayhem/301e3b6cfffeaf8a...

is that the hl extractor DDoSing us?

so, that query generates the huge temp files.

and there were quite a few of those queries -- we should increase the delay between queries so that we do this less often

CatQuest: I added you to INST developers, can you please check editing labels on INST issues?

it worked

Srta

*sorta

alastairp: iliekcomputers: we should deploy the rate limiter on AB ASAP.

after dinner zas and I will have to take a closer look at this traffic. someone is hitting us too hard.

I've finally created a first working build

Congrats I guess! :)

zas: frank is still fine. I'm turning on the hl calc again to see if the problem returns

of course, now it has a backlog it has to churn through first.

and the load comes back, right on queue after it runs out of items to examine.

iliekcomputers: alastairp: here is my assessment of what is currently wrong with AB: https://gist.github.com/mayhem/a3916f616d670e52...

I'll start working on fixes for both in the morning.

Hey there. Are there any 'getting started' docs on how to contribute to the listenbrainz server issues? Looking for the 101 -- how to launch the server, any guidance on the workflow for submitting changes etc.

Ahhh, just discovered this: https://blog.metabrainz.org/2018/10/11/listenbr...

Seems relevant!

So I am installing the dev environment on a Mac. I ran this line:

docker-compose -f docker/docker-compose.yml -p listenbrainz run --

The error message says ERROR: No such service: --

Whatever '--' means is probably something supported by linux but not macOS.

Any idea what the translation might be?

Hey punkscience! I don't work on listenbrainz but I just followed through the installation process, and I think it should work on mac - it did on mine, at least

The problem you're having there is that the command continues onto the next line in that blog post

So the full command is "docker-compose -f docker/docker-compose.yml -p listenbrainz run --rm web python3 manage.py init_db --create-db", for example

If you haven't found this already, I think a more comprehensive guide would be the listenbrainz-server docs