#metabrainz

sitemaps have been failing 'cause of the meb api 504ing since july 24 https://gist.github.com/mwiencek/780c0df49f5d42...

not every day, just every few days

was never a problem before that though

updated the gist with the timestamp of every occurence, if that helps narrow down a problem

yeah, we've been seeing random 504s. they are not just limited to one site, which suggests that it is a gateway problem

ah

I'll add some code to retry to request there anyway

Gentlecat: does the sentry setup catch 504s for meb on sakura?

Still no mobile internet, mind

ruaok: of its in the app, it should

if*

ok, then it idn

isn't sakura's fault.

Morning. Just came in to see if you were interested in another 504 report, but it seems like you already know about them :)

For what it's worth, I had another at 05:16:02 downloading a replication packet.

yeah, we know.

at least we no know that it is a gateway problem and not a app server problem.

and I'd hate to stick zas on it, because we should be focusing on moving to NewHost, but zas remains on vacation.

so, shitty service will remain.

It's not so bad to be honest. I only notice it because occasionally I get a failure email from my slave server. It seems to recover itself the following hour.

Replication related 504s are due to Sakura, they happen when it swaps heavily. I'm working on openresty container atm

and why is sakura swapping?

75 uwsgi processes running.

gee, I wonder.

adhawkins: let me know if it happens again

zas: does graphana have a graph for examining memory usage for a given server?

Yes server overview

thx

yay, metabrainz.org 500s now.

sigh.

I've got an idea.

let's just shut everything down.

bring it back up when we're ready.

in a few months.

much less stress that way.

what did you decide on in the end for NewHost ruaok?

hetzner. their deal can't be beat.

yeah?

why is sakura such a freaking mess?

I blame zas, and when it isn't his fault it's surely Fresos

;)

yeah, hetzner doesn't charge us for VAT either, since we're a US company. there is 20% savings right there.

and we got clarification on the terms and things are better than we first thought.

the only real downside is that we need to rent space in a rack for future use.

but even that sunk cost is considerably less than going with OVH.

and Google Cloud is FOUR times as expensive as Hetzner.

no brainer then

yeah. and we know what we're getting with hetzner. we have an established relationship with them.

we're going to get everything setup on docker, then do a test deploy on google cloud.

once that works on some level, I'll go order a pile of machines of hetzner.

interesting, why docker?

we started using chef, but that was write once debug many.

docker has been considerably simpler for our needs.

well.. they aren't really comparable are they? chef automates setup whereas docker is a container

so the alternatives to docker would be lxc or kvm, whereas the alternatives to chef would be ansible or puppet

we've looked at ansible, puppet and chef. they all suck.

I don't disagree :)

and our configuration needs are simple enough that we can do it with docker.

don't get me wrong, I'm a docker advocate as part of the linuxserver.io docker group, but it's not as mature as say lxc so was just a bit surprised

I agree that it is an apples and oranges comparison, but the isolation aspects of docker make writing configurations for our setup much easier.

that is what zas got excited about. Also a long time MB contributor works for Docker. :)

we all started playing with it and really liked it.

ok, next question: why the fuck did bookbrainz not start.

what is the app server for bookbrainz? anyone know?

fair enough, it's a good platform, and my group is responsible for the most popular musicbrainz docker ;)

ah, I've been using musicbrainz-docker from jsturgis.

I can't even see that one on the docker hub? lol

Leftmost, LordSputnik: why does bookbrainz on sakura not start?

ahh, it's not on the hub just on github?

we've had so many start on reboot issues, I'm just going to leave bookbrainz.org down until someone decides to get more serious about it.

ruaok: Will do.

thanks!

doh, I'm actually interested in bookbrainz for a future possible project, lol

:)

LordSputnik: look, test.bookbrainz.org is down too

Gentlecat: any idea why metabrainz.org cannot connect to postres after a reboot?

the replication packets are not working.

let me read the backlog

I rebooted sakura and tons of shit broke, so I am slowly working my way through all the issues, hoping to fix them.

CB seems to be connecting fine, mbspotify too

meb for other pages too.

which is really weird.

but not for replication?

correct.