#metabrainz

mo’’in’

mo'in!

yvanzo: zas: shall we put some thought behind fixing trille today?

ruaok: is listenbrainaz using rabbitmq too?

yes

MB uses it to update search indexes, it should preferably not be stopped or search indexes won’t be up-to-date.

I don’t think that this is what take the most resources on trille, but we could move this queue to PostgreSQL.

LB is less sensitive. if a user cannot submit a listen, clients must re-try. so, restarts are ok.

something is a miss. load on trille keeps growing, traffic in rabbitmq hasn't grown.

IMHO, CB has probably a lot of margin for reducing its footprint on resources.

Does CB use RMQ or CB is on trille as well?

it is on trille as well

have we ascertained if the resource hog is CB or RMQ?

teletgraf is the top process on trille? that feels odd to me.

zas: Is it possible to monitor trille’s containers from grafana more closely, for example using cadvisor?

well, we already have reports for containers on trille

you read my mind, we need to a have a % usage per container graph....

zas: got link?

heh

ignore empty graphs, scroll down

guys, we already have 2 suspects: rabbitmq and critiquebrainz-redis

first one is known to eat cpu for nothing in certain cases

second one is actually having huge write to disk spikes

I just dont see rabbitmq as the culprit. but I see redis. those peaks in BlkIo are worrying.

yes^^

it writes far too much data

yesterday I reduced share of trille mbs to almost nothing, so only few queries goes to it, and even with that, we still have very slow queries (read: seconds instead of milliseconds)

on some ws queries (usually < 100ms) we can reach > 10s

that looks like we need to investigate wtf is happening here.

what did we do on 10/9, for instance?

with less empty graphs: https://stats.metabrainz.org/d/000000051/hetzne...

yvanzo: thank you. that helps.

so, rabbitmq is not the problem. agreed?

+1

that coincides with a CB release.

and presumably a CB container restart.

though the release on 10.26 didn't cause the same drop. perhaps redis was not restarted then?

zas,yvanzo has redis been restarted recently?

last time 2 months ago

this instance of redis doesn't run with --appendonly=yes, like most instances, so it doesn't use aof

any objections to restarting it to see what happens to the graph?

but imho beam.smp cannot be excluded yet

I could see a situation where CB is keeping a list in redis that keeps growing. and it is written over and over again.

a bug, for sure.

zas: I didn't. I'm trying to exclude on clear trouble maker to get another data point.

*one

_lucifer: ping

2273 process (beam.smp) is writing a lot

can you tell where the data goes, zas?

wait

I catched redis write ops

beam is deffo the highest disk user. didn't CB add more telegraf logging? could it be overdoing it?

it goes up to 50mb/s

yes, that is why I am focusing on redis.

while beam.smp doesn't go over 400kb/s

beam.smp is an issue too, but its less spikey.

yes, so unlikely to cause huge delays we see

I'm going to restart redis, ok?

It seems due to be CB usage of redis.

not sure restarting it will help, but you can try

it might drop the traffic back to 0 and then start growing again. but that would clearly indicate CB is doing something bad.

we will probably the same graph as from September.

ruaok: pong

At least, it should be lower CPU/Mem usage first.

peaks are 5-6 mins apart.

hi _lucifer !

can you please follow the scroll back for the last 20 minutes?

sure

we're seeing strange redis use coming from CB.

I'm curious if redis use in CB has recently changed. is there anything that gets processed every 5 minutes or so?

it is causing massive disk io spikes: https://stats.metabrainz.org/d/000000051/hetzne...

zas, yvanzo : as expected the disk io for redis has dropped to nothing.

beam.smp too. which might suggest that whatever the redis bug is it might be logging info to telegraf.

there were a couple of bug fixes regarding that, a key mismatch but nothing comes to mind that happens at a regular interval

ok, the regular interval might be a redis behaviour due to increased use.

all data served by CB is cached

what is weird is that we are seeing a lot of data being written to redis, but with very little read. that is fully upside-down of what it should be.

yeah right

can we like a sample of the latest read/writes?

so, the primary traffic for CB comes from MB hitting its API.

/ws/1/review/?limit=1&offset=0&release_group=ee9b6cad-ee58-3529-81ba-cc204769459c&sort=rating

*view a sample

is the endpoint that gets all the traffic. can you please review the entire code chain of this endpoint and review the redis use in great detail to see if we can find something where redis might be used incorrectly?

sure, i'll do that

_lucifer: that is what I am hoping to do next. what is being written and read. let me dig

wow.

an incredible number of new keys are being generated in redis.

my preliminary guess is that if there is no bug, then different MB entities are being queried (different release groups being viewed but the same page is not viewed frequently) but they are not viewed that often. hence, the writes are frequent but the reads are not.

6 keys a second are being created. that would be the problem.

there must be a problem with the page cache.

worst case, each page fetched from MB would cause 1 fetch and 1 write to redis.

but I would expect some cache hits, so there should be fewer writes than reads.

do you know where the cache keys are generated in CB, _lucifer?

yes a sec

ruaok: https://github.com/metabrainz/critiquebrainz/bl...

was caching in brainzutils changed recently?

no doesn't seem so, the last commit to brainzutils cache was 2 years agp

_lucifer: could you do me a quick favor? can you disable caching from that function and make a small PR?

then we can deploy that and observe.

sure, on it

because right now caching is creating load problems , not solving them.

thx

[critiquebrainz] amCap1712 opened pull request #337 (master…disable-cache): Disable release-group caching https://github.com/metabrainz/critiquebrainz/pu...

thx

1 test is failing but that is expected.

agreed.

let me see about deploying.

[critiquebrainz] mayhem merged pull request #337 (master…disable-cache): Disable release-group caching https://github.com/metabrainz/critiquebrainz/pu...

I think MB only requires the review text and review ratings but CB provides additional entity data which MB already has. The number of reviews is much less than the number of entities. So for most entities, we are just caching the entity data which MB is not useful to MB. I can add a MB mode to cache only the review text and ratings. I think that would reduce the cache writes to a large extent.

+1

oh, that is interesting. maybe make a separate endpoint for that?

hello. reading backlog

need help with CB release?

hopefully not. 🤞