should check to make sure that nginx is sending an X-Forwarded_Proto or whatever header so you can check it
2014-09-01 24403, 2014
kepstin_
(it's non-standard, but most servers set it to the string https for https requests)
2014-09-01 24405, 2014
bitmap
yeah, I see a line for x-forwarded-proto on the load balancer
2014-09-01 24430, 2014
kepstin_
the other option would be to do the redirects in nginx rather than the application, but that won't work if you're doing it only for logged in users.
2014-09-01 24425, 2014
Gentlecat
why not?
2014-09-01 24446, 2014
kepstin_
hmm, well, I suppose it could if you're setting a cookie for logged in users
2014-09-01 24405, 2014
kepstin_
and set it up to always redirect the login page, of course
2014-09-01 24429, 2014
kepstin_
would be an if at the top level of the http server block containing a rewrite, and a separate rewrite, possibly inside a location block, to do the login page
2014-09-01 24434, 2014
kepstin_
shouldn't be hard, actually.
2014-09-01 24404, 2014
bitmap
hm, it looks like we already rewrite all urls to https in the nginx config?
2014-09-01 24430, 2014
ianmcorvidae
yeah, I assume what's happening here is
2014-09-01 24431, 2014
kepstin_
ah, so it's unconditionally https?
2014-09-01 24440, 2014
ianmcorvidae
a.) nginx is redirecting it
2014-09-01 24445, 2014
kepstin_
that should be fine, unless the app server is also redirecting it :)
yeah, when running with the nginx redirector, you'll want to disable that. One way would be to hook up the X-Forwarded-Proto header so it knows the url is already https, the other would be to just remove the decorator.
2014-09-01 24427, 2014
ianmcorvidae
you should ensure request.is_secure is appropriately set based on X-Forwarded-Proto and/or X-MB-https, both of which we set (the former to 'https', the latter to 'on')
the 'derive.php' are the things that generate thumbnails, of course
2014-09-01 24423, 2014
kepstin_
hmm. oh, do you have to be logged in for that?
2014-09-01 24425, 2014
ianmcorvidae
I guess for a non-admin that page doesn't show the "server readonly -- tasks waiting for harddrive fix" status, but still
2014-09-01 24428, 2014
ianmcorvidae
probably
2014-09-01 24455, 2014
KillDaBOB
hm. one set of images i just uploaded is showing up fine, but the previous few sets i uploaded aren’t. that’s just strange. but maybe it is just a backlog.
2014-09-01 24423, 2014
kepstin_
yeah, they have some sort of queue system to do processing of uploads, and it can get backlogged.
2014-09-01 24405, 2014
ianmcorvidae
theoretically our side of the queue can also get 503d and that can cause problems, which is annoying to check from here so I won't, but :P
2014-09-01 24413, 2014
KillDaBOB
heh
2014-09-01 24416, 2014
ianmcorvidae
(also it takes it a bit since it retries it a few times first)
2014-09-01 24417, 2014
KillDaBOB
fair enough
2014-09-01 24438, 2014
kepstin_
ah, that item history thing is pretty cool
2014-09-01 24443, 2014
kepstin_ found his archive login :)
2014-09-01 24402, 2014
ianmcorvidae
one thing to check is if the archive.org item page (the /details/ one) has an index.json
2014-09-01 24409, 2014
ianmcorvidae
that's what our queue is doing, is uploading that file
2014-09-01 24434, 2014
ianmcorvidae
which at least at some points in history has had to be there before the IA would derive the 250/500px thumbnails
2014-09-01 24428, 2014
KillDaBOB
yeah, that’s what the problem was once before when this was happening. there was a MB server restart and that process didn’t get restarted with it or just didn’t start up. something like that.
2014-09-01 24446, 2014
KillDaBOB
but the json file is present on the releases i’m having problems with, so it’s not that.
2014-09-01 24459, 2014
ianmcorvidae
yeah, probably just backlog at the IA
2014-09-01 24437, 2014
ianmcorvidae
or a broken hard drive needing replacement (which makes servers readonly)
2014-09-01 24413, 2014
kepstin
has nginx reverse-proxying itself in order to stack the a cache in front of a ratelimiter
2014-09-01 24454, 2014
ianmcorvidae
hehe :)
2014-09-01 24410, 2014
ianmcorvidae
arguably we should just use nginx's ratelimiting ourselves
2014-09-01 24436, 2014
ianmcorvidae
but our rules are rather complex so it's been nice having custom code for it
2014-09-01 24414, 2014
ianmcorvidae
(especially when you include the stats handling -- though probably that could be done by postprocessing a log)
2014-09-01 24442, 2014
kepstin
the nginx one is kind of interesting, the way I have it set up, it'll actually queue up and delay requests up to a point rather than rejecting them immediately.
2014-09-01 24408, 2014
ianmcorvidae
does that tie up nginx worker processes?
2014-09-01 24421, 2014
ianmcorvidae
I guess probably not, nginx is usually smart about that
2014-09-01 24439, 2014
kepstin
no, nginx is internally event-based
2014-09-01 24449, 2014
kepstin
the worker processes is just to allow it to scale to multiple cores
2014-09-01 24410, 2014
ianmcorvidae
right, yeah
2014-09-01 24439, 2014
ianmcorvidae
how far will it queue/delay rather than failing (and how does it fail? 503/429/something else?)
2014-09-01 24429, 2014
kepstin
the queue is configurable via the "burst" parameter, and failure is configurable, defaults to 503.
2014-09-01 24446, 2014
ianmcorvidae
cool
2014-09-01 24419, 2014
ianmcorvidae
I should try to do our ratelimiting/complexity thereof in nginx sometime, presumably
2014-09-01 24433, 2014
kepstin
the ratelimiting is key-based; so the trick is just setting up the key somehow.
2014-09-01 24435, 2014
ianmcorvidae
figuring out how to set the key would be the hard part, making it be something other than a chain of gross ifs
2014-09-01 24438, 2014
ianmcorvidae
yeah
2014-09-01 24409, 2014
kepstin
in mbjs, the key is $server_name, because the whole thing has to be in a single ratelimit key, so that was just a static string which was available.
2014-09-01 24445, 2014
kepstin
you'll note that i'm also correctly setting the user agent ;)
2014-09-01 24455, 2014
ianmcorvidae
hehe
2014-09-01 24408, 2014
kepstin
anyways, the goal of this is that the mbjs page itself doesn't actually need a ratelimiter, since cached content will be served with no limit, and nginx will automatically delay requests for uncached content.
2014-09-01 24409, 2014
ianmcorvidae
right now we have a several-step thing where it first modifies the key for some things like customers that get higher ratelimits, then it looks up keys by a regex to determine the ratelimit parameters (count and period the count applies to, mostly)
2014-09-01 24419, 2014
ianmcorvidae
and then it does this three times, one for IP, one for UA, one for global
2014-09-01 24429, 2014
ianmcorvidae
yeah
2014-09-01 24450, 2014
kepstin
yeah, you'd have to have multiple rate limit zones in nginx for the different settings
2014-09-01 24410, 2014
kepstin
I wonder if you can use the limit_req inside an if.
2014-09-01 24425, 2014
ianmcorvidae
heh
2014-09-01 24445, 2014
ianmcorvidae
at least you can limit_req and then proxy_pass it, like you are
2014-09-01 24445, 2014
kepstin
... no. it's http,server,location context only.
2014-09-01 24459, 2014
ianmcorvidae
so we'd still only need one final backend that passes to the real WS servers
2014-09-01 24423, 2014
kepstin
the nginx ratelimiter's design inherently smooths out bursts
2014-09-01 24429, 2014
kepstin
rather than passing through 10 requests immediately then rejecting others, depending on config it'll e.g. (with burst=10) either accept 10 but delay each one to e.g. 1 req/s on the output, or (burst=0) accept 1 and reject others until 1 second passes.
2014-09-01 24406, 2014
kepstin
so it's very different behaviour from what musicbrainz has now.
2014-09-01 24447, 2014
kepstin
... unless that's not what it does. It's not exactly super-clear
2014-09-01 24408, 2014
kepstin
I wonder if they have the actual algorithm used documented somewhere
2014-09-01 24424, 2014
kepstin
given that they talk about delay in the docs, I have to assume that it's doing queueing
KillDaBOB: sorry about that, the main database was locked up for some reason
2014-09-01 24425, 2014
KillDaBOB
heh.
2014-09-01 24454, 2014
KillDaBOB
hopefully it wasn’t down for long
2014-09-01 24426, 2014
luks
submissions were down for about one hour :(
2014-09-01 24420, 2014
KillDaBOB
that’s not too bad. i guess it was caught pretty early.
2014-09-01 24412, 2014
Gentlecat joined the channel
2014-09-01 24431, 2014
reosarevok joined the channel
2014-09-01 24431, 2014
jesus2099 joined the channel
2014-09-01 24430, 2014
jesus2099
do we kno why nikki’s simple Remove relationship edits are but « An error occured while loading this edit » ? they are just some normal AR being removed, nothing like new types or anything…
2014-09-01 24414, 2014
reosarevok
The way attributes and removals are handled changed
2014-09-01 24421, 2014
reosarevok
Will be fixed tonight anyway
2014-09-01 24440, 2014
chirlu`
It’s both because they now store the “ended” flag and because they may contain instrument credits.
2014-09-01 24406, 2014
jesus2099
oh great tonight ! i feared it would be long term, i can stop switching back and forth tonight :)
2014-09-01 24423, 2014
reosarevok
Normally this is at most two weeks, since it's just caused by a difference on how edits are treated in beta vs. prod
