#musicbrainz-devel

? there's an importer...

I know, I just dislike viewing a tracklist on itunes.apple.com where every other line is truncated

:)

Lotheric: make one! (and while you're at it, can you also make it stop redirecting to the desktop app? ;)

ianmcorvidae: I'm getting redirect loop at https://critiquebrainz.org/profile/edit for some reason

http://gentlecat.mbsandbox.org/profile/edit is fine

maybe something is wrong with old HTTPS redirects?

maybe flask doesn't know that the original request is https after it's proxied through

should check to make sure that nginx is sending an X-Forwarded_Proto or whatever header so you can check it

(it's non-standard, but most servers set it to the string https for https requests)

yeah, I see a line for x-forwarded-proto on the load balancer

the other option would be to do the redirects in nginx rather than the application, but that won't work if you're doing it only for logged in users.

why not?

hmm, well, I suppose it could if you're setting a cookie for logged in users

and set it up to always redirect the login page, of course

would be an if at the top level of the http server block containing a rewrite, and a separate rewrite, possibly inside a location block, to do the login page

shouldn't be hard, actually.

hm, it looks like we already rewrite all urls to https in the nginx config?

yeah, I assume what's happening here is

ah, so it's unconditionally https?

a.) nginx is redirecting it

that should be fine, unless the app server is also redirecting it :)

b.) critiquebrainz also redirects it

this is the view https://github.com/metabrainz/critiquebrainz/bl...

ssl_required at https://github.com/metabrainz/critiquebrainz/bl...

yeah, when running with the nginx redirector, you'll want to disable that. One way would be to hook up the X-Forwarded-Proto header so it knows the url is already https, the other would be to just remove the decorator.

you should ensure request.is_secure is appropriately set based on X-Forwarded-Proto and/or X-MB-https, both of which we set (the former to 'https', the latter to 'on')

http://stackoverflow.com/questions/24914563/tro... seems to be related

looks like enabling the proxyfix midleware should be sufficient to get X-Forwarded-Proto to work

I think the ProxyFix thing should be it, yeah

the plain-http one doesn't work either, btw, as far as I can tell

if it did it was presumably accidental/cached

alrighty, I'll try that

thanks guys

hm. looks like CAA stopped generating thumbnails for uploaded artwork. or maybe CAA just hates me. been going on for 20-30 minutes now.

(do keep in mind that running a server with the proxyfix middleware enabled which is on the open internet is a security issue)

the full size images are there and load just fine, but not the 250 and 500 sizes.

hmm. archive's s3 has been having a higher than usual rates of 503s recently

probably a partial failure of servers on their end

KillDaBOB: might just be backed-up, 20-30 minutes isn't much to be worried about

if it's not better tomorrow, then we start to worry.

you can check their status page yourself, also

not that one

though you can check that one too

oh, there's another status page?

mm.. nice red lines.

line*

anyway e.g. for http://musicbrainz.org/release/189d7863-a23c-49... it's https://archive.org/details/mbid-189d7863-a23c-... for the archive page, replace 'details' with 'history' for the status page for that item

the 'derive.php' are the things that generate thumbnails, of course

hmm. oh, do you have to be logged in for that?

I guess for a non-admin that page doesn't show the "server readonly -- tasks waiting for harddrive fix" status, but still

probably

hm. one set of images i just uploaded is showing up fine, but the previous few sets i uploaded aren’t. that’s just strange. but maybe it is just a backlog.

yeah, they have some sort of queue system to do processing of uploads, and it can get backlogged.

theoretically our side of the queue can also get 503d and that can cause problems, which is annoying to check from here so I won't, but :P

heh

(also it takes it a bit since it retries it a few times first)

fair enough

ah, that item history thing is pretty cool

one thing to check is if the archive.org item page (the /details/ one) has an index.json

that's what our queue is doing, is uploading that file

which at least at some points in history has had to be there before the IA would derive the 250/500px thumbnails

yeah, that’s what the problem was once before when this was happening. there was a MB server restart and that process didn’t get restarted with it or just didn’t start up. something like that.

but the json file is present on the releases i’m having problems with, so it’s not that.

yeah, probably just backlog at the IA

or a broken hard drive needing replacement (which makes servers readonly)

has nginx reverse-proxying itself in order to stack the a cache in front of a ratelimiter

hehe :)

arguably we should just use nginx's ratelimiting ourselves

but our rules are rather complex so it's been nice having custom code for it

(especially when you include the stats handling -- though probably that could be done by postprocessing a log)

the nginx one is kind of interesting, the way I have it set up, it'll actually queue up and delay requests up to a point rather than rejecting them immediately.

does that tie up nginx worker processes?

I guess probably not, nginx is usually smart about that

no, nginx is internally event-based

the worker processes is just to allow it to scale to multiple cores

right, yeah

how far will it queue/delay rather than failing (and how does it fail? 503/429/something else?)

the queue is configurable via the "burst" parameter, and failure is configurable, defaults to 503.

cool

I should try to do our ratelimiting/complexity thereof in nginx sometime, presumably

the ratelimiting is key-based; so the trick is just setting up the key somehow.

figuring out how to set the key would be the hard part, making it be something other than a chain of gross ifs

yeah

in mbjs, the key is $server_name, because the whole thing has to be in a single ratelimit key, so that was just a static string which was available.

you'll note that i'm also correctly setting the user agent ;)

hehe

anyways, the goal of this is that the mbjs page itself doesn't actually need a ratelimiter, since cached content will be served with no limit, and nginx will automatically delay requests for uncached content.

right now we have a several-step thing where it first modifies the key for some things like customers that get higher ratelimits, then it looks up keys by a regex to determine the ratelimit parameters (count and period the count applies to, mostly)

and then it does this three times, one for IP, one for UA, one for global

yeah

yeah, you'd have to have multiple rate limit zones in nginx for the different settings

I wonder if you can use the limit_req inside an if.

heh

at least you can limit_req and then proxy_pass it, like you are

... no. it's http,server,location context only.

so we'd still only need one final backend that passes to the real WS servers

the nginx ratelimiter's design inherently smooths out bursts

rather than passing through 10 requests immediately then rejecting others, depending on config it'll e.g. (with burst=10) either accept 10 but delay each one to e.g. 1 req/s on the output, or (burst=0) accept 1 and reject others until 1 second passes.

so it's very different behaviour from what musicbrainz has now.

... unless that's not what it does. It's not exactly super-clear

I wonder if they have the actual algorithm used documented somewhere

given that they talk about delay in the docs, I have to assume that it's doing queueing

seems to work like basically this picture https://en.wikipedia.org/wiki/File:Leaky_bucket...

oh, it's configurable. by default it queues, but you can disable that by adding the 'nodelay' parameter.

oh noes! the 502 sickness has spread to acoustid.org! E: 00:34:34 AcoustID: submission failed with error 'Error downloading http://api.acoustid.org:80/v2/submit - server replied: Bad Gateway'

MB servers are contagious.

KillDaBOB: sorry about that, the main database was locked up for some reason

heh.

hopefully it wasn’t down for long

submissions were down for about one hour :(

that’s not too bad. i guess it was caught pretty early.

do we kno why nikki’s simple Remove relationship edits are but « An error occured while loading this edit » ? they are just some normal AR being removed, nothing like new types or anything…

The way attributes and removals are handled changed

Will be fixed tonight anyway

It’s both because they now store the “ended” flag and because they may contain instrument credits.

oh great tonight ! i feared it would be long term, i can stop switching back and forth tonight :)

Normally this is at most two weeks, since it's just caused by a difference on how edits are treated in beta vs. prod

it’s been probably two weeks then :)

Almost, it was deployed on 2014-08-19.