ruaok now imagines zas as an american football player ready to tackle quee
2019-09-11 25459, 2019
zas
:D
2019-09-11 25446, 2019
zas
ruaok: I just cancelled queen and did various clean-up related to it (admin tools, nagios, dns, ...), plus I reinstalled a fresh linux on it (re-format)
2019-09-11 25456, 2019
zas
ruaok: so you can order floyd !
2019-09-11 25434, 2019
zas
now I'm going to prepare diner for my hungry monsters...
2019-09-11 25458, 2019
CallerNo6 joined the channel
2019-09-11 25458, 2019
CallerNo6 has quit
2019-09-11 25458, 2019
CallerNo6 joined the channel
2019-09-11 25432, 2019
ruaok
ok to do that in the morning?
2019-09-11 25409, 2019
yvanzo now imagines zas as a space pionneer feeding tamed aliens
what were all the previous 302s from? http to https redirects?
2019-09-11 25414, 2019
zas
as is, no idea, we have to look at logs to know
2019-09-11 25428, 2019
bitmap
looking at the nginx logs it looks like the majority of 302s are /search pages, which may explain the connection with search
2019-09-11 25441, 2019
bitmap
but I don't know how those would redirect other than http->https
2019-09-11 25446, 2019
bitmap
so I assume it's that
2019-09-11 25414, 2019
zas
but we didn't change anything on gateways or solr (afaik)
2019-09-11 25428, 2019
bitmap
right
2019-09-11 25439, 2019
zas
can it be a significative change in incoming traffic?
2019-09-11 25432, 2019
bitmap
I mean if the numer of /search requests just dropped significantly I guess that'd explain the drop in 302s and solr traffic
2019-09-11 25431, 2019
bitmap
but idk why that would happen
2019-09-11 25441, 2019
zas
that's the question ;)
2019-09-11 25401, 2019
zas
before 14:00 utc we had 5k 302s per 2 minutes
2019-09-11 25415, 2019
zas
after it dropped to ~600
2019-09-11 25426, 2019
zas
number of 200s decreased too
2019-09-11 25448, 2019
zas
we lost 3k 200s per 2 minutes
2019-09-11 25458, 2019
bitmap
there are definitely a lot less 302s in the logs for 13 vs 14 vs 15 utc
2019-09-11 25438, 2019
zas
yes, and a lot less 200s too (more or less the same drop)
2019-09-11 25409, 2019
bitmap
ah
2019-09-11 25421, 2019
bitmap
so it's just more noticeable for 302s since there are less
2019-09-11 25441, 2019
zas
but we had a peak of 503s
2019-09-11 25426, 2019
zas
starting around 14:08, ending around 14:34
2019-09-11 25448, 2019
bitmap
yeah, and it didn't recover after that
2019-09-11 25415, 2019
zas
have a look to those, 8k per 2 minutes difference (usual ~2k, during this period > 10k)
2019-09-11 25448, 2019
zas
not sure if it's a cause or consequence though
2019-09-11 25441, 2019
bitmap
I'm going through each website container and seeing if there's anything weird in the logs, then restarting them
2019-09-11 25429, 2019
zas
k
2019-09-11 25438, 2019
SothoTalKer has quit
2019-09-11 25410, 2019
Gazooo joined the channel
2019-09-11 25435, 2019
SothoTalKer joined the channel
2019-09-11 25434, 2019
zas
bitmap and I found what happened: a nasty distributed bot suddenly stopped hitting us, it was responsible of 2/3 of the traffic apparently ... that's a good and a bad news
2019-09-11 25459, 2019
zas
the bad one: we waste a lot of resources for nothing...
2019-09-11 25431, 2019
zas
the good one: legit users have now faster search responses
2019-09-11 25422, 2019
zas
conclusion: we need much stricter policy and better tools
2019-09-11 25446, 2019
zas
our investigation concerns only mb website, for this case. But it is very likely same shit hit our other fans....
