#metabrainz

Zas: now that we have hetzner machines, please shut down the test machines at google.

k

ruaok: can you add my ssh key to bowie ? or is it possible to do it through Robot ?

ruaok: on new gateways, using Hetzner Failover IP is great, but it doesn't help with local IPs, we'll still need corosync for that, but instead of switching its own external IP, it will just run a script to command Robot to do the switch. Also it means both intances of openresty are running (so no need to switch from one to another using corosync). Do i

miss something ? I think you expected to get rid of corosync on new setup, but it looks to me we still need it (though with a simpler config).

I think your assessment on internal is is correct, we still need corosync. :-(

bowie root passwd in same file on ernie in tmp.

not sure how to install a different SSH key other than mine during an installation. :(

so, if you decide to re-install, you may need me to give you access again.

Leftmost: managed to replicate the issue with relationships, no relationship set is added for the target entity in cases where both entity data entries are using the dummy relationship set (1)

alastairp: I'm getting more SMART errors on spike.

let me know when you're happy with your backup situation and then I'll ask them to replace the drive.

ruaok: bowie is not installed yet, when i log in it is the rescue system, should i proceed ?

LordSputnik: fixed it, I'd missed returning a promise when fixing the other problem with the relationship editor

Err, Leftmost ^ :P

I am in bowie right now.

I'm trying to install MegaCLI as per https://wiki.hetzner.de/index.php/LSI_RAID_Cont...

but it is shipped in an rpm. :(

I've made a cpio out of it, but I have to run off now.

see /root on bowie. bowie login credentials are in /tmp on ernie.

ruaok: megacli is working on rescue for me

See also http://hwraid.le-vert.net/wiki/LSIMegaRAIDSAS

ok, so it looks like it was configured in two raid-1 volumes.

I wonder if the controller can do raid-1,0 or if we have to do that in software.

megacli -PDList -aALL

it will list physical drives

there are 4

yep.

question is, can the controller do 1,0?

Yes, it can: https://www.supermicro.com/manuals/other/LSI_21... (page 11)

zas: I'll let you focus on the raid controler. let me look at the failover IPs.

here is someething *very* interesting:

> A failover IP gives you the possibility to route the IP to any server in your Robot account. You can change the routing whenever you want. Traffic accounted for the failover IP will be assigned to the server for which you order the failover IP regardless what the current routing target of the failover IP is. The failover routing can be configured here in

your Robot webinterface or script based over a webservice.

so, check me on this... if we order the failover IP on the machines with a 30TB budget we might run into trouble later on.

but, we could order the IP for the PX91, and immediately move it to kiki/herb.

that way kiki/herb are using the 50TB budget, not the 30TB budget.

do you interpret that in the same way?

neat, really neat.

But we may, for whatever reason, be down at any given time, and it would be good to still have alternatives. Also, people may be physically closer to other mirrors than the German servers (even if not that much compared to US ones).

Unless we're paying for the newer EU mirror, in which case, sure, close it down. :)

ruaok: I have a backup from yesterday on babar

-> please go ahead with drive change

I'm away for the rest of the afternoon

the backup is just for disaster recovery, right?

Freso|laptop: I am not talking about reducing any services. Just moving them around.

alastairp: the backup is in case something goes horribly wrong with the drive swap.

👍

thanks

have you guys started moving AB to docker yet>

?

ok, I'll proceed then.

not yet. good idea

not urgent since the AB move is now towards the bottom of the priority list, but something to keep on the radar. all things hosted at the new server farm shall be dockerized.

zas: that looks good. is that your doing or was it that way?

do we need to do an install now?

ruaok: it'd be useful to have root access on sakura so that I can start and stop the bookbrainz service during updates - is that possible?

LordSputnik: not sure I am comfortable with that.

it was my doing, and it needs to be reinstalled, i'm on it ;)

what if we move BB to the google cloud instance. there you can have the keys to the kingdom.

zas: fabu. Ill leave you to it.

I

I'm talking with hetzner about bandwidth and failover IPs.

we will have half a petabyte of traffic from them before too long.

per month!

ruaok: on herb,kiki,zappa you ran https://github.com/metabrainz/syswiki/tree/mast... scripts right ?

zas: correct.

alastairp: it appears we need to take downtime with the disk swap, since it is software raid and the disks do not appear to the hot-swappable. I'll need to ask them exactly how that works first.

hang tight.

ruaok: localhost-setup should be modified, it has few incorrect bits: mail and ntp setups need changes at least

ruaok: check if you can log in bowie

bowie is asking for a key for robert@,

ruaok: OK, but I need to do an update at some point today to fix these issues with the relationship editor

can you please use the last key on https://github.com/mayhem.keys ?

I'll work with bitmap when he comes on if you're busy with zas at the moment

LordSputnik: no problem. bitmap, zas or I can restart.

this is fast enough any one of us can stop for a sec and do this.

ruaok: OK, well, I've just pulled the code, so restart when ready :)

ruaok: try now

btw, you should add the key to server-setup

[critiquebrainz] gentlecat opened pull request #55: Integration with Consul and consul-tempate (beta...consul) https://git.io/vi4jx

Zas: hah. No one at hetzner is watching the failover traffic to make sure it goes to the budgeted machine.

:C

(I think bb serve could be called like "byron" or "dickens" :D)

or "dahl"

or "adams"

:DD

CatQuest: all good choices, I don't know what we're doing with hosting though at the moment

I'm having some trouble getting a search server setup. It is giving an error durring mvn package here is a pastebin: http://paste.ubuntu.com/23154926/

LordSputnik: I can pretty much guarantee that servers at NewHost will not be used for only one project.

so, there being a single "bookbrainz server" is not likely going to be the case.

zas: given the conversations about the failover IPs, what should we do?

Gentlecat: MBS already does namespacing (by default, every key is prefixed with “MB:”, though that can be changed in DBDefs.pm).

OTOH, Redis does support multiple databases within one server, so it might make sense to separate projects that way.

ruaok: not sure yet, the initial plan was to failover on both gateways (EX41), is this an issue ?

chirlu: no, I was talking about consul's storage

:)

Ah, sorry, that wasn’t clear.

Gentlecat: dunno if you saw this link I posted a couple days ago https://github.com/Cimpress-MCP/git2consul

you define a config to watch a git repo like https://github.com/mwiencek/musicbrainz-server/...

and can store the config in a json file https://github.com/mwiencek/musicbrainz-server/...

and I can access those values under musicbrainz-server/config.dev.json/services/musicbrainz-server/DBDefs

I was thinking we'd keep these .json files for all our projects in the docker-server-configs repo

bitmap: i saw it, looks like a good idea

it seems to work well

makes sense

ruaok: can I put in servername ideas in the "not in use yet" section or should there be an "ideas" section? http://wiki.musicbrainz.org/Hosting/Server_names

also, since I needed to install consul-template and runit inside every container, I added the script for it to docker-helpers https://github.com/mwiencek/docker-helpers/comm...

so it can be used in other projects if needed

Great 😃