#metabrainz

[listenbrainz-android] 14kingbass01 closed pull request #176 (03main…fix157): Fix theme and colors for About and NewsBrainz Activities in Compose https://github.com/metabrainz/listenbrainz-andr...

[listenbrainz-android] 14kingbass01 reopened pull request #176 (03main…fix157): Fix theme and colors for About and NewsBrainz Activities in Compose https://github.com/metabrainz/listenbrainz-andr...

~/sb end

whoops

zas, atj, lucifer : https://labs.api.listenbrainz.org/ ssl is broken with a cert problem. we need to add monitoring to this site too!

a restart fixed it

the python requests module cannot call the labs.api endpoint for the reason that lucifer mentioned before where a local update fixes the problem. but how many other legit callers will get a failed call??

time to look at this tomorrow

mayhem: I had a look, there's a problem with redis (keydb)

we hit a limit, not sure why yet

odd. are older connections not being closed?

something like that

weird

eh?

not good

on rudi https://www.irccloud.com/pastebin/3DteTs29/

eeeek

I stopped the instance on rudi

4.7G...

there are supposed to be the same size

so the instance on rudi went to bananas (heavily)

that will please monkey 🍌

ok, replica resynced, things are back to normal

I have no idea about what happened though

yep, looks better! thanks!

perhaps we hit a bug in keydb, I'll check

it never happened before

or perhaps rudi still has a problem

aof file on rudi was to maxsize, so this replica was failing, causing connections left open on rex keydb (the other replica)

we need alerts on this, it took several hours to get critical

too late for me to have clear thoughts on this

I solved it by totally removing the buggy db, and just starting again the container on rudi so it resynced with rex

agreed, I"m off to bed.

gn