#metabrainz

I'm away soon, I'll check back tomorrow

akshat: meeting is an hour late today :). (EU DST switch)

Oh right lucifer! I'll have tea up and ready then :)

wait o it's not 19 but 20?

CatQuest, i think its the same time for you as you are in the EU. only changed for those outside it

Cool let's do it now then outsidecontext. The automation is done so far and for the rest of the stuff, we need to directly follow the Fastlane documentation

lucifer: we could, but that's a prod server, it is likely to explode if we do that

basically we don't know what broke....

lucifer: honestly i don't change times on my clocks becasue i'm dead against it :[

and why suddenly

it's 18 :50 for me now

yeah indeed

I didn't see any error at openresty level that could explain the problem, but...

musibrainz is using hard certs, and it doesn't work either

if there is anything i can do, help test, etc please holler

there are very few entries in nginx error logs and those that appear are ... benign.

outsidecontext: hit `fastlane run download_from_playstore`

lots of non-http traffic getting through.

*non-https

But have a play_config.json ready in your local setup and don't add it to git

yes...

so I suspect https to be broken, for both LE & hard certs, which leads us to ... DNS, openresty config didn't change (I reverted recent changes), nor openresty version

how many instances of openresty are supposed to be running? I see 8 instances in top, but that seems too few.

one per cpu

that's ok

can't be DDNS becasue that would affect http

a simple curl -vik https://musicbrainz.org hangs

that's the problem, why does it hang?

dns seems to be working but then it hangs

so that's the handshake

I also changed my /etc/hosts to mask musicbrainz.org directly to herb, still hangs. unlikely the floating IP.

how about we block all incoming traffic except from say bono and then restart openresty in debug?

:O i just got through to https://beta.musicbrainz

very slow

it kidna crashed my browser XD

curl -vik https://listenbrainz.org did work... but after a looong time

yes

so https works (kinda)

I couldn't load a page from beta

right, thats what i saw some time ago. it looks like responses are too slow. sometime too slow that it times out at other times it loads but late.

dns_resolution: 0.005, tcp_established: 31.402, ssl_handshake_done: 111.763, TTFB: 111.829

everything seems slow. not any one single thing.

fwiw, when it *does* load it loads fine, all css etc

and http loads fast

ok, how about this interpretation: TLS fails because we're too busy and it times out? we're too busy because of something else and we're chasing the wrong symptom?

ok, my idea would hold if something on the TLS path would cause something to slow down.

we're CPU bound, and calculating TLS connections backs up, causing listens to drop.

but what is causing the slowdown?

I have another theory, since everything is slow, we exhaust open socket or the like, and ssl fails due to that

but something is the root of this slowdown

yes, that could do it.

dmesg has messages with : [ 5026.378971] TCP: too many orphaned sockets

A cron job gone bad?

DDoS?

lucifer: DDOS would now allow our HTTP traffic to flow freely.

i was gonna ask

ah makes sense

rdswift: something kinda like that is my guess. we don't really have cron job on the gateways, but something is eating CPI.

ruaok: look at network traffic on herb

on stats.mb ?

yes

on it

which dashboard?

cpu is high, temp too, network traffic on eth0 & eth1 high, so I guess load is "normal", it doesn't explain why https fails that much especially for hard certs which don't depend on anything

nginx processes have high prio

nothing stands out as weird.

can it be a cascading effect? slowdown leading to https failures leading to slowdown?

ping from kiki to outside is working again. should we retry switching?

zas: it could be.

well, we can, but since we didn't find the reason behind this shit...

herb/kiki might just be saturated. and its a holiday in the EU.

we can rule out hardware failure since HTTP is working.

it might be fans :P

ah ignore me

one thing that speaks against a cascading failure is that HTTP is working.

if everything was overloaded, HTTP would suck too.

yes, but http doesn't required as much resources

but I agree, seems weird

and HTTPS is not *that* much more resource intensive that it should fail like this.

ok, any https specialist around?

Could the certs have changed but for some reason nginx didn't reload them?

kepstin: ping!

zas: https://stats.metabrainz.org/d/000000050/hetzne...

kiki's eth0 was just under 100mbit for a long time before things went bad.

that doesn't feel like a coincidence.

oh. CAA?

zas: what if we leave the CAA down or block its traffic for right now.

that is a decent chunk of requests, possible in HTTPS.

reminder: all this started on failover IP switching

yes

going back to lucifer's idea: lets block something that causes a lot of traffic. CAA seems a good candidate.

is all traffic of caa towards mbs in https?

if someone had loads of requests on caa spesfically

zas: look at this: https://stats.metabrainz.org/d/000000061/mbstat...

as drastic spike in the LB traffic just before this started.

hmmm

if something picked up huesound, that would give 25 requests for 1 LB request.

25 CAA requests for 1 LB request.

oooooooh

and that might be a problem.

CatQuest: good idea.

lucifer: wanna try something?

:)

sure. what?

try deploying lb-web-prod with using NON https for CAA URLs.

on it

thanks.

one tricky thing to note if you have an https endpoint is that people might be doing http/2 to it, and multiple requests can be in progress on a single http/2 endpoint that get reverse proxied to many individual http/1.1 requests

so pure connection count comparisons between the two might not be valid

ruaok: also bring the container on gaga down which hits CAA?

lucifer: ok, will do.

I doubt it will do much of anything. its very low traffic.

and it runs in batches, 10 mins after the hous.

the traffic doesn't look much higher than usual...

writer is down. PR approved.

zas: yes, it could be that we hit a threshold and that tipped us over.

eh. question, did we make a huesound.org site?

CatQuest: no, but I should make a redirect to the LB on.

one

yea http://huesound.org/ doesnt load

is there any specific reason for that to go through CAA at all? IIRC after the plex situation, MB switched to linking directly to IA instead of following the redirect

the logic for going direct is complex and not available to LB, ROpdebee

getting 502's on listenbrainz

i don't think the http thing is going to work. i took LB prod down while the image built but nothing seems to have chagned.

the CAA had a drastic spike in traffic before it all went bad.

lucifer: we may be trying the wrong thing.

zas thoughts on blocking the CAA from the gateways

?

makes sense. let's bring CAA down for a while

agreed.

zas?

yeah like that. everything is inaccessible anyways

if the CAA is problem we can get another server to run that.

ruaok: isn't it just `https://archive.org/download/mbid-${selectedRelease.release_mbid}/mbid-${selectedRelease.release_mbid}-${selectedRelease.caa_id}_thumb250.jpg`?

or am i missing something

yup, once we know the problem we can plan the mitigation but first we need to figure out the issue.

we can take it down in docker-server-configs

zas: want a PR?

but... it will not prevent incoming reqs

I'll do it manually

reduce the TTL on caa.org and point it elsewhere?

ROpdebee: could be. but we want a CAA cache ideally.

I dropped the TTL on caa.org, just in case

this is the only real clue that something went wrong, IMHO: https://stats.metabrainz.org/d/000000061/mbstat...

caa is now in maintenance mode, 503