#metabrainz

(I am transiting denmark, lol)

darwin: where ya headed to in the end?

reosarevok_: hi!

hi reosarevok

Ok, this IRCCloud thing seems to work fine :p

:)

I'm running a separate browser in order to not have it impact my use of chrome.

ruaok: which behavior do we expect after a service failure ? let me explain

Cursing and grumbling!

heh

all is running on ernie, suddenly nginx crashed on ernie, so it failovers and move all services to bert

we can either: do nothing and wait admin noticed and eventually move all back to ernie

add a failover timeout, with a preferred location (ernie), so it will try to move all back to ernie after some time

should we have a "preferred location" ?

or we don't care at all, until all is running on one server (but then it can migrate from ernie to bert, and stay on bert, until next issue)

in case of failover we can clear failover history automatically or not

ie. ernie -> service failure -> migrate to bert -> 2 days passed, no admin reset the failover -> service failure -> all stopped (it can't migrate back to ernie, since it knows it has failed)

\ø/

preferred location: ideally not, methinks.

or ernie -> service failure -> migrate to bert -> 2 days passed, autoreset done after 6 hours -> service failure -> migrate to ernie -> if succeeded, ok, else all stopped

shall we start simple and then adjust as we go?

for me that means: do nothing, but notify admins.

admins can inspect and adjust as needed.

imho having a failover timeout of 24 hours ie. can help

and when that becomes painful, commit it to code to automate

it will try to migrate back if 24 hours have passed

so it is an intermediate solution

I'm ok with that if you prefer that.

i think it is good, still require admin to check the thing in case of failure, but is able to recover in case it was a temporary failure, followed by another failure

having a long delay prevents any flapping

ernie -> bert -> ernie -> bert ... loop wouldn't be nice ;)

so i will set a preferred location on ernie, so it will start services on it, and only migrate to bert in case of failures, then services will stay on bert, i'll adjust stickiness so it will only try to migrate back in case of failure and 24 hours passed

ruaok: like this https://www.dropbox.com/s/i4g7kj3ynkea1dk/Scree...

perfect!

on a slightly different topic, I'm noodling on the idea of re-working Huesound ( http://s.huesound.org ) to work with the CAA and the tomahawk applet.

then maybe put it on the homepage of the CAA as a showcase for the CA??

+A

ok, merged

ruaok: i have a setup exhibiting the correct behavior ;) : ip-dns tinydns dnscache on ernie, i stop dnscache, pacemaker migrates ip-dns tinydns dnscache to bert, after 2 minutes (for test, it will be 24 hours), i stop dnscache on bert, services are migrated back to ernie, dnscache restarted

ruaok: do you want to deploy it?

if i stop dnscache before the 2 minutes delay, the whole thing is shutdown (since it cannot work on ernie nor bert)

and if i don't stop dnscache, it stays on bert

i will add the rest now and do more tests

Gentlecat: I can try.

hmm. unsure how that should be done. there are two dirs.

bitmap: do you know the procedure for upgrading the CAA?

I think someone suggested you to use XMPP?

yes

that is the basic idea behind it.

I asked Arc Riley, an XMPP dude about this.

can you clarify a bit?

the first question I asked was if this was suited for this task.

he said yes.

so, XMPP can store message for offline users

and using XMPP allows users to also connect their MB XMPP accounts to their favorite IM client.

which is really really cool, IMHo.

People still use IM clients? :p

(but yeah, for the ones who do that sounds nice)

he suggested that we use this server: https://prosody.im/

he personally hacks on an apache module called Modxmpp.org

he thought that within 2 weeks there would be a production ready release of this module.

he further suggested that we should use Stanza.io for JS bits to make all of this easier.

so, I haven't really looked into any of these pieces.

your first task should be to look at all of these elements and see if they fit and meet our needs.

then, adjust the jira tickets as apporpriate to reflect the new technology direction

right

mail intro to arc done.

ruaok: can you explain me how the rate-limiter is working ? it is listening on UDP 10.1.1.245:2000, but does it need to run on the same machine of any other service ?

no, it runs on the gateways only

___<;3

when a request that comes in, the app that handles the request makes a call to the ratelimiter to see if the request is allowed.

if so, it proceeds, if not a 503 is returned.

it modifies iptables chains so it needs to run on the same machine as nginx, is there more constraints ?

s/is/are/

ie. dnscache/tinydns could run on ernie while the rest is on bert

it modifies iptables??

it doesn't ?

hmmm, ok i really have to look at the code ;)

not that I am familiar with.

it runs in the application space, not the network space.

Mm. Such shiny. https://www.openhub.net/orgs/MetaBrainz

ruaok: yes, i was thinking of blockpw ... my bad

Should I add project logos for stuff such as libmusicbrainz and libcoverart too?

phew. :)

Freso: I'd say yes.

ruaok | and using XMPP allows users to also connect their MB XMPP accounts to their favorite IM client.

^ was one of my arguments when I suggested it. :)

(As well as generally having tons and tons of software and libraries and what not that is XMPP compatible, meaning it would "easily" be possible to do some neat things with it.)

(So I'm glad Arc Riley managed to win you over, is what I'm trying to say. ;))

he merely confirmed that the things we wanted to do are actually designed into the XMPP and work well.

which was not easy to ascertain.

Freso: can you merge MetaBrainz and MetaBrainz Foundation there?

ruaok: so ratelimit-server could possibly run on any machine, since apps are querying it through network, and i see no dependency in the code, it only depends on 10.1.1.245 IP address, it doesn't store starts on disk, am i wrong thinking it it has no stat (and therefore do not rate limit anything) when restarted ?

reosarevok: I sent a request about that ~2 hours ago. ;)

Ok :)

zas: that sounds right

reosarevok: If you're talking about the two organisations. If you're talking about the MetaBrainz Website project and the MetaBrainz Foundation organisation, then no, those can't be merged.

Two orgs yep

Okay. Good. :)

Anyway, lib{coverart,musicbrainz} have had logos added too.

All dem shiny logos. *-*

Now looking at how to claim my stuff :p

i'll set pacemaker accordingly, colocating this IP address and this service, it will make things easier to not create useless dependencies between IPs and services, failing over will run smoother (it will migrate this IP/ratelimiter independently of ie. external IP addresses or nginx)

and it allows to run ratelmiter on bert, while we run nginx on ernie ;)

woah.

ok, coffee time

might make my trip in SLO much much more work, but this is for a good cause.

alastairp: http://tickets.musicbrainz.org/browse/AB-97 :)

what?

I just use incr-4

which is the smallest incremental dump ;)

alastairp: did you bring the package in or should I grab it from your house?

you should grab it from my house

ok, lets coordinate that over the weekend.

I guess the problem is that incremental dumps still aren't published :)

Gentlecat: sure they are. we just don't list their names. they're still there :)

but yeah

this is something else I want to do after the data migration

there's a huge list of things waiting for that. in fact, I /just/ finalised some data formatting stuff with dmitry, and once I've implemented that the migration will be done

cool