#metabrainz

[bookbrainz-site] prabalsingh24 closed pull request #374 (master…Entity-Revision-Table): Entity revision table https://github.com/bookbrainz/bookbrainz-site/p...

[bookbrainz-site] prabalsingh24 reopened pull request #374 (master…Entity-Revision-Table): Entity revision table https://github.com/bookbrainz/bookbrainz-site/p...

great Freso

mo’’in’

updating beta.mb.o

https://musicbrainz.osuosl.org/pub/musicbrainz/... is serving a 404, this is what the homepage links to. for Mac, naturally (because dmg)

the other downloads page works, the correct link is https://musicbrainz.osuosl.org/pub/musicbrainz/...

VxJasonxV: oh well, we need to update the homepage, the file name changed slightly. thanks for reporting. please use the links on the download page

done!

zas: I pushed the download link fix to https://github.com/metabrainz/picard-website . can you update the website when you have time?

updating main mb.o

done

iliekcomputers: https://gist.github.com/mayhem/77818163cc3469f1...

I'm taking a closer look at why the queue is getting backed up.

iliekcomputers: https://gist.github.com/mayhem/ce90f2543f359385...

iliekcomputers: zas: this is super interesting.

cb delta is the amount of time that passes between callbacks from rabbitmq.

300-400ms. our code uses 5ms of that. the system is sitting idle for a good long chunk of time, doing fuck all.

zas: I could use your help, please.

our instance of rabbitmq was started *20* months ago.

which on its own is amazing, given hetzner fans. lol.

I dont know what the procedure is for restarting rabbitmq, but I think we may want to try that.

trille is neither IO bound nor CPU bound.

upgrading mb sitemaps, json-dump, cron

ruaok: what do you mean by callbacks specifically here?

The time it takes pika to call our handler function?

the time between calls to our callback from pika/rabbitmq

in theory it should be near instant, maybe 1ms. but it is ~300ms.

maybe a bug in pika?

released MBS, blog post is pending review at https://wordpress.com/block-editor/post/blog.me...

\O/

yvanzo: edited the blog post, does it look good?

zas: plz read the posts from the last few hours.

Ok

having strange latency issues with rabbitmq

You mean network-related latency?

not sure what kind of latency, but I dont think it is network related.

the way we read data from rabbitmq is via a callback.

and the callback sits idle for up to 300ms between calls -- that is the reason why the queue is backed up so much.

we're just not getting called to process data faster.

I noticed that rabbitmq has been running for *20* months.

do you know how/if we can safely restart rabbitmq?

Unsure

well, help me find out, please.

Let me check what is using it

BTW rebooting the whole server would be a good thing too

yes, for sure.

I think it is ok to restart it (rabbitmq), at worse apps using it will reveal how they suck by not handling this case

other docker containers should also be ok

yvanzo: can you check what's running on trille?

zas: agreed.

I suggest to do a restart of rabbitmq alone, to start with

ok, wanna do the honors?

>at worse apps using it will reveal how they suck by not handling this case

pretty bad worst case

its only the CAA using it, no?

does sir use it's own instance?

oh, right. that.

totally forot about that little detail.

we can stop sir & caa

i'm pretty sure we have good handling in LB.

iliekcomputers: yeah, we're fine.

zas: lets do that.

no need to stop LB. we have loads of retry and resiliency built in.

famous last words

(kidding)

ok, rabbitmq restart (caa & sir stopped)

let's see if it has any impact on lb

that was it!!

2020-02-18 11:43:53,974 INFO cb delta: 0.002 parse: 0.000 write: 0.002 stats: 0.000

2ms between calls now!

woot!

🎉

times are wildly fluctuating

back to 400ms now.

hmmm

I'd reboot the whole server (to upgrade kernel)

ok, good plan.

is this trille?

y

CB will have downtime then (if we wanna tweet)

yup, CB on it too

only one instance?

i think so, yes.

ruaok: tweet about it please

ok.

https://stats.metabrainz.org/d/000000048/hetzne... shows a lot of disk writes

but that's the same since 1 year

tweeted.

ok, proceeding, time to pray ;)

I hate having no ping to a server... make me anxious....

you n' me both.

trille's back :)

still waiting for rmq to come back

rabbitmq failed to restart

not sure why yet

ok started

verfiied.

fast now.

annnnd slow again.

can it be a issue with the data volume?

not sure.

can we tell if sir has been moving slow as well?

oh!

zas: checking

our writes are slow now.

btw, we still run rmq 3.6.5

https://www.rabbitmq.com/versions.html -> 3.7 out of support at end of March

ok, seems that the rabbitmq slowdown was fixed -- now the problem has moved to influxdb. :)

ok, make a ticket for that, zas.

should I start sir to see?

and maybe work with bitmap or yvanzo to upgrade it.

yvanzo: fine by me.

MBH-534: Upgrade rabbitmq to 3.8.x on trille

yvanzo: bitmap: can one of you two please work on that?

sir is back, not much to mention, its rate is limited to 100 msg in a batch.