-
D4RK-PH0ENiX has quit
2020-02-18 04928, 2020
-
BrainzGit
2020-02-18 04938, 2020
-
BrainzGit
2020-02-18 04939, 2020
-
D4RK-PH0ENiX joined the channel
2020-02-18 04959, 2020
-
shivam-kapila has quit
2020-02-18 04947, 2020
-
sampsyo has quit
2020-02-18 04940, 2020
-
supersandro20005 has quit
2020-02-18 04914, 2020
-
sampsyo joined the channel
2020-02-18 04955, 2020
-
sbvkrishna joined the channel
2020-02-18 04914, 2020
-
Gore has quit
2020-02-18 04943, 2020
-
Gore joined the channel
2020-02-18 04901, 2020
-
amCap1712
great Freso
2020-02-18 04902, 2020
-
rahul24 joined the channel
2020-02-18 04910, 2020
-
shivam-kapila joined the channel
2020-02-18 04920, 2020
-
darkstardev13 joined the channel
2020-02-18 04934, 2020
-
raven__ has quit
2020-02-18 04931, 2020
-
sbvkrishna has quit
2020-02-18 04939, 2020
-
rahul24 has quit
2020-02-18 04938, 2020
-
rahul24 joined the channel
2020-02-18 04920, 2020
-
Darkloke joined the channel
2020-02-18 04954, 2020
-
rahul24 has left the channel
2020-02-18 04959, 2020
-
shivam-kapila has quit
2020-02-18 04905, 2020
-
yvanzo
mo’’in’
2020-02-18 04955, 2020
-
yvanzo
updating beta.mb.o
2020-02-18 04958, 2020
-
VxJasonxV
2020-02-18 04954, 2020
-
VxJasonxV
2020-02-18 04957, 2020
-
outsidecontext
VxJasonxV: oh well, we need to update the homepage, the file name changed slightly. thanks for reporting. please use the links on the download page
2020-02-18 04905, 2020
-
VxJasonxV
done!
2020-02-18 04936, 2020
-
outsidecontext
2020-02-18 04938, 2020
-
yvanzo
updating main mb.o
2020-02-18 04948, 2020
-
CatQuest has quit
2020-02-18 04941, 2020
-
yvanzo
done
2020-02-18 04952, 2020
-
ruaok
2020-02-18 04906, 2020
-
ruaok
I'm taking a closer look at why the queue is getting backed up.
2020-02-18 04901, 2020
-
Gazooo has quit
2020-02-18 04900, 2020
-
ruaok
2020-02-18 04916, 2020
-
ruaok
iliekcomputers: zas: this is super interesting.
2020-02-18 04939, 2020
-
ruaok
cb delta is the amount of time that passes between callbacks from rabbitmq.
2020-02-18 04948, 2020
-
Gazooo joined the channel
2020-02-18 04917, 2020
-
ruaok
300-400ms. our code uses 5ms of that. the system is sitting idle for a good long chunk of time, doing fuck all.
2020-02-18 04942, 2020
-
ruaok
zas: I could use your help, please.
2020-02-18 04904, 2020
-
ruaok
our instance of rabbitmq was started *20* months ago.
2020-02-18 04926, 2020
-
ruaok
which on its own is amazing, given hetzner fans. lol.
2020-02-18 04947, 2020
-
ruaok
I dont know what the procedure is for restarting rabbitmq, but I think we may want to try that.
2020-02-18 04925, 2020
-
ruaok
trille is neither IO bound nor CPU bound.
2020-02-18 04936, 2020
-
yvanzo
upgrading mb sitemaps, json-dump, cron
2020-02-18 04951, 2020
-
iliekcomputers
ruaok: what do you mean by callbacks specifically here?
2020-02-18 04905, 2020
-
iliekcomputers
The time it takes pika to call our handler function?
2020-02-18 04958, 2020
-
MusicbrainzB0T joined the channel
2020-02-18 04901, 2020
-
MusicbrainzB0T1 has quit
2020-02-18 04934, 2020
-
ruaok
the time between calls to our callback from pika/rabbitmq
2020-02-18 04959, 2020
-
ruaok
in theory it should be near instant, maybe 1ms. but it is ~300ms.
2020-02-18 04915, 2020
-
ruaok
maybe a bug in pika?
2020-02-18 04935, 2020
-
ruaok upgrades to pika 1.1.0
2020-02-18 04941, 2020
-
shivam-kapila joined the channel
2020-02-18 04957, 2020
-
yvanzo
2020-02-18 04921, 2020
-
ruaok
\O/
2020-02-18 04954, 2020
-
reosarevok
yvanzo: edited the blog post, does it look good?
2020-02-18 04900, 2020
-
ruaok
zas: plz read the posts from the last few hours.
2020-02-18 04927, 2020
-
zas
Ok
2020-02-18 04931, 2020
-
ruaok
having strange latency issues with rabbitmq
2020-02-18 04931, 2020
-
zas
You mean network-related latency?
2020-02-18 04956, 2020
-
ruaok
not sure what kind of latency, but I dont think it is network related.
2020-02-18 04906, 2020
-
ruaok
the way we read data from rabbitmq is via a callback.
2020-02-18 04934, 2020
-
ruaok
and the callback sits idle for up to 300ms between calls -- that is the reason why the queue is backed up so much.
2020-02-18 04944, 2020
-
ruaok
we're just not getting called to process data faster.
2020-02-18 04902, 2020
-
ruaok
I noticed that rabbitmq has been running for *20* months.
2020-02-18 04913, 2020
-
ruaok
do you know how/if we can safely restart rabbitmq?
2020-02-18 04953, 2020
-
zas
Unsure
2020-02-18 04923, 2020
-
ruaok
well, help me find out, please.
2020-02-18 04924, 2020
-
zas
Let me check what is using it
2020-02-18 04909, 2020
-
zas
BTW rebooting the whole server would be a good thing too
2020-02-18 04903, 2020
-
ruaok
yes, for sure.
2020-02-18 04948, 2020
-
CatQuest joined the channel
2020-02-18 04949, 2020
-
CatQuest has quit
2020-02-18 04949, 2020
-
CatQuest joined the channel
2020-02-18 04937, 2020
-
zas
I think it is ok to restart it (rabbitmq), at worse apps using it will reveal how they suck by not handling this case
2020-02-18 04909, 2020
-
zas
other docker containers should also be ok
2020-02-18 04904, 2020
-
zas
yvanzo: can you check what's running on trille?
2020-02-18 04955, 2020
-
ruaok
zas: agreed.
2020-02-18 04944, 2020
-
zas
I suggest to do a restart of rabbitmq alone, to start with
2020-02-18 04945, 2020
-
ruaok
ok, wanna do the honors?
2020-02-18 04933, 2020
-
iliekcomputers
>at worse apps using it will reveal how they suck by not handling this case
2020-02-18 04937, 2020
-
iliekcomputers
pretty bad worst case
2020-02-18 04955, 2020
-
ruaok
its only the CAA using it, no?
2020-02-18 04912, 2020
-
iliekcomputers
does sir use it's own instance?
2020-02-18 04919, 2020
-
ruaok
oh, right. that.
2020-02-18 04927, 2020
-
ruaok
totally forot about that little detail.
2020-02-18 04932, 2020
-
zas
we can stop sir & caa
2020-02-18 04937, 2020
-
iliekcomputers
i'm pretty sure we have good handling in LB.
2020-02-18 04944, 2020
-
ruaok
iliekcomputers: yeah, we're fine.
2020-02-18 04950, 2020
-
ruaok
zas: lets do that.
2020-02-18 04907, 2020
-
ruaok
no need to stop LB. we have loads of retry and resiliency built in.
2020-02-18 04927, 2020
-
iliekcomputers
famous last words
2020-02-18 04931, 2020
-
iliekcomputers
(kidding)
2020-02-18 04916, 2020
-
zas
ok, rabbitmq restart (caa & sir stopped)
2020-02-18 04946, 2020
-
zas
let's see if it has any impact on lb
2020-02-18 04949, 2020
-
ruaok
that was it!!
2020-02-18 04901, 2020
-
ruaok
2020-02-18 11:43:53,974 INFO cb delta: 0.002 parse: 0.000 write: 0.002 stats: 0.000
2020-02-18 04907, 2020
-
ruaok
2ms between calls now!
2020-02-18 04908, 2020
-
ruaok
woot!
2020-02-18 04956, 2020
-
iliekcomputers
🎉
2020-02-18 04910, 2020
-
ruaok
times are wildly fluctuating
2020-02-18 04917, 2020
-
ruaok
back to 400ms now.
2020-02-18 04924, 2020
-
zas
hmmm
2020-02-18 04943, 2020
-
zas
I'd reboot the whole server (to upgrade kernel)
2020-02-18 04953, 2020
-
ruaok
ok, good plan.
2020-02-18 04911, 2020
-
iliekcomputers
is this trille?
2020-02-18 04915, 2020
-
ruaok
y
2020-02-18 04924, 2020
-
iliekcomputers
CB will have downtime then (if we wanna tweet)
2020-02-18 04947, 2020
-
zas
yup, CB on it too
2020-02-18 04959, 2020
-
zas
only one instance?
2020-02-18 04906, 2020
-
iliekcomputers
i think so, yes.
2020-02-18 04950, 2020
-
zas
ruaok: tweet about it please
2020-02-18 04954, 2020
-
ruaok
ok.
2020-02-18 04949, 2020
-
zas
2020-02-18 04942, 2020
-
zas
but that's the same since 1 year
2020-02-18 04947, 2020
-
ruaok
tweeted.
2020-02-18 04903, 2020
-
zas
ok, proceeding, time to pray ;)
2020-02-18 04909, 2020
-
ruaok fetches pasta
2020-02-18 04940, 2020
-
zas
I hate having no ping to a server... make me anxious....
2020-02-18 04953, 2020
-
ruaok
you n' me both.
2020-02-18 04946, 2020
-
zas
trille's back :)
2020-02-18 04917, 2020
-
yvanzo is back too
2020-02-18 04955, 2020
-
ruaok
still waiting for rmq to come back
2020-02-18 04914, 2020
-
zas
rabbitmq failed to restart
2020-02-18 04922, 2020
-
zas
not sure why yet
2020-02-18 04945, 2020
-
zas
2020-02-18 04912, 2020
-
zas
ok started
2020-02-18 04922, 2020
-
ruaok
verfiied.
2020-02-18 04929, 2020
-
ruaok
fast now.
2020-02-18 04911, 2020
-
ruaok
annnnd slow again.
2020-02-18 04908, 2020
-
zas
can it be a issue with the data volume?
2020-02-18 04925, 2020
-
ruaok
not sure.
2020-02-18 04942, 2020
-
ruaok
can we tell if sir has been moving slow as well?
2020-02-18 04945, 2020
-
ruaok
oh!
2020-02-18 04947, 2020
-
yvanzo
zas: checking
2020-02-18 04950, 2020
-
ruaok
our writes are slow now.
2020-02-18 04938, 2020
-
zas
btw, we still run rmq 3.6.5
2020-02-18 04956, 2020
-
zas
2020-02-18 04956, 2020
-
ruaok
ok, seems that the rabbitmq slowdown was fixed -- now the problem has moved to influxdb. :)
2020-02-18 04946, 2020
-
ruaok
ok, make a ticket for that, zas.
2020-02-18 04947, 2020
-
yvanzo
should I start sir to see?
2020-02-18 04959, 2020
-
D4RK-PH0ENiX has quit
2020-02-18 04959, 2020
-
ruaok
and maybe work with bitmap or yvanzo to upgrade it.
2020-02-18 04905, 2020
-
ruaok
yvanzo: fine by me.
2020-02-18 04924, 2020
-
zas
2020-02-18 04925, 2020
-
BrainzBot
MBH-534: Upgrade rabbitmq to 3.8.x on trille
2020-02-18 04949, 2020
-
ruaok
yvanzo: bitmap: can one of you two please work on that?
2020-02-18 04918, 2020
-
yvanzo
sir is back, not much to mention, its rate is limited to 100 msg in a batch.
