#metabrainz

Leo_Verto: what about?

Leo_Verto: ah yes, just saw that, will sort it out tomorrow

Anyone here know how to specify negative niceness via schedtool?

listenbrainz seems to be returning a lot of 502s

alastairp: ^ (no ruaok awake yet it seems)

I just restarted it. we’re still having some problems with it

what is "it" in this casE?

an app server?

yes

flask + uwsgi

we’re having problems where some workers seem to get stuck after about a day on high cpu, and uwsgi stops sending requests to them

workers live that long?

web workers. Yes, they’re supposed to be long running, and nginx passes requests to uwsgi, which is supposed to hand it off to a worker

alastairp: can you tell uwsgi to restart them after serving n requests?

I'm pretty sure that's what we do at my work.

like, I would not expect them to run "about a day"

(but I am "just the DBA" hehe)

russss: it has so many options, I'm sure it's a possibility

though we've never had this problem on acousticbrainz (definitely less submissions though)

might be worth doing that as a first possibility. It could be there's a resource leak somewhere

but finding those is hard

the other alternative is that occasionally something is looping for ever

yeah, those are our thoughts too. and we also know that finding them is hard

one thing I wanted to do was attach strace to a high cpu processes

just to see what it's doing

there's a high probability it'll show either nothing, a load of select()s on the epoll socket, or a load of futex_wait. None of which is helpful :)

but worth a try. (also ltrace)

ah, never seen ltrace

oh right, strace is system calls, ltrace is library calls. neat

still waiting on a good dtrace-alike for Linux which works out of the box. Apparently Systemtap may have userspace tracing support for python now, but I need to spend some time looking at it.

sysdig might be interesting to look at as well

sysdig is cool but it's basically just a new frontend to existing tools rather than a new datasource.

were there any problems with the caa-indexer in the last month? I had to perform some edits on https://musicbrainz.org/release/0df9fe6e-d374-4... to make it return the front type in the json file correctly

ruaok: :(

I can't change the status of this from in progress to open again

alastairp: done

seems like only assigned user can do that

silly

yep

I guess I'm not an admin or anything on AB

Gentlecat: will you be around some time?

yeah

I wanted to talk about next steps for ab

(and your outstanding bugs)

just to see where we are

right

and we should try and come up with a project for you too

I'm thinking of changing the file structure of the server again :(

I think it's actually a good thing to do occasionally

I want to have acousticbrainz related stuff in an acousticbrainz package

e.g. http://tickets.musicbrainz.org/browse/AB-67

we should be moving logging out of webserver if we're to use it in highlevel

but I don't want to call it `logging`. it should be called `acousticbrainz.logging`

it will also need some changes to work outside of flask apps

shouldn't need much

I just specify the handlers outside of webapp, and include them inside webapp

yeah, probably

then I can use the handlers in other packages