#metabrainz

earlier in the day EU would be best, is my guess.

that makes bitmap the main limiting factor.

when do you suppose you could be up tomorrow morning bitmap?

I could probably be up at 5AM CST if I go to sleep early

wow, really?

we also need all the replication packets in that volume

I don't want to over stress you. also since you need to get NewHost through the first EU night.

Gentlecat: is an extra step needed in the migration doc?

ok, let's keep to 6am then.

I'll try for 6AM anyway

perfect.

ruaok: I suppose

Gentlecat, zas: go time for starting the migration is 6am Chicago. That should be 13h CEST, if my brain can do math.

unless it can be started now

Gentlecat: I don't see why not. we've got time.

k

start it please. the step for syncing the last packets is the in doc.

yes, 13 https://www.wolframalpha.com/input/?i=6+AM+chic...

Freso: will you be around tomorrow 13h BCN time?

"bitmap the main limiting factor" ... not kind ;)

as far as time. sorry bitmap. :)

haha

ok, I think that covers my concerns.

keep working to tidy things up, but make sure to get to bed at a good hour to get good rest.

I wont be up that early, so start without me.

I'll try and rope Freso into doing all the tweeting/blogging/announcing things that you guys are doing.

any questions so far?

and in general: don't worry about downtime. worry about getting the job done in a sane and secure way.

zas: as far as new rate limits, I would set them at 300 as we are now.

and just observe. if you happen to get bored because not enough stuff is breaking, think about increasing it.

but methinks 300/sec should be cake on the new setup.

ruaok: well, yes... but we have few issues to solve ;) like the one bitmap and i are on ....

understood.

basically consul health checks overreacting on load, removing upstream servers (thinking they are stopped), where they are just under load

ruaok: That's the plan, yes.

with a nice cascade effect, ending in no service instead of slow service

Freso: can you please write a blog post and post immediately that sums up what we just said?

trying a tcp check instead of http

also include the link to

it is now world readable.

Freso: follow along in the switchover. when something is done, strike it out. then people can follow along.

ok, zas, bitmap and Gentlecat: Can I go to DWNI and tell them today that tomorrow will be our last day?

I don't know how final that will be but, I just want to make sure I am not jumping the gun and that we are all in agreement.

I'd like to hear some response from each of you.

my stuff is ready according to this sheet of paper next to me

that is a yes, proceed from you then?

most of the things are already in prod on new host

apart from CAA redirect and MeB

well, i would prefer tomorrow not being my last day..... i would say yes... we have to do it

those are being tested and seem to be working fine

a bit nervous but it's doable

heh. love those responses.

yeah, I am nervous too.

^^ same, i expect bad surprises but they aren't really surprises since i expect them ....

but even with whatever bumps may come, lets hope that this isn't going to be worse than what we have now.

zas: good attitude. :)

ok, motion carried. I'll head over to DWNI in a bit and start cleanup.

D:

be sure to save as much stuff you can, storing disks somewhere in case of ... we need some forgotten stuff in 6 months

which is very likely to happen according to Murphy's law

zas: yes, but what?

if it isn't in the gist, It isn't on my radar.

i think it is, wiley mostly, we have nothing on mbs servers, nor on gateways

I think I have a copy of it somewhere, but saving the old chatlogger (code + logs) from scooby would be good

pretty sure it was in wiley backups already

'twas on scooby.

recovering anything from scooby is going to be an adventure.

it has SCSI drives.

old chatlogger added to list.

zas: we still have a free IP on our blocks at DWNI, right?

I want to put rika there for people to be able to log in for 24 hours to fetch stuff.

🙌 https://opensource.googleblog.com/2016/11/annou...

gcilou: "403 Forbidden" -_-

hm

But, hey: "GCI 2016: MetaBrainz Foundation has been accepted as a mentoring organization!"

yeah!

So I guess it's on. :)

17 this year

zas: I guess we'll need a ws.beta.musicbrainz.org too

any easy way we can block outside connections to those ws.* domains too?

it's weird that there isn't an official sshd docker image

bitmap: the easy way is what i did with ws.mb.o ... just keep http & https fields empty

see current config

ah ok

Freso: nice

Requests per second: 43.49 [#/sec] (mean)

bitmap: ^^ with 40 c

much better

nice

don't forget to change to tcp checks on ws too

already done

this is something we'll need to sort out, but i think the main issue is the code, http checks need 429 when too many queries

that said, we *should* use 429...

eh, results are much better

100 concurrent connections, same rate, no failed query

and cpu% on bowie < 10% with same test

bitmap: \o/

woot

amazing how much those area_containment queries suck

slower to answer, but no failure

expected

and load on servers is much more normal

bitmap: our fancy new hardware will reveal tons of software bottlenecks... not sure it is good news for devs ;)

job security, I guess :P

sort of ;)

ok i'm off for diner, i will run similar tests on ws after

chirlu: you have those queries from logs, can you send them to me so i can build extensive stress tests ?

bitmap: you added zappa as website server too ?

just the beta site

ah ok, it uses the same container name

yeah

perhaps better to have it named after beta, and keep all configs separated

should be under beta.musicbrainz-website?

I don't see zappa's IP in that paste

it isn't

hence my question

ah

^ that should add a new upstream automatically right?

yup

cool

chrome doesn't even let me add cert exceptions anymore

I wonder if they changed that

ok hello ,not to butt in where I am adding to loads and stuff but: will tomorrow be a bad time to add instrments? ie, will mb be down in the morning cet?

i read something about 13 cest

(ie is this happening *tomorrow*) 8th november

CallerNo6: Funky Town!