I'm running a separate browser in order to not have it impact my use of chrome.
2015-11-20 32442, 2015
kepstin-laptop joined the channel
2015-11-20 32436, 2015
zas
ruaok: which behavior do we expect after a service failure ? let me explain
2015-11-20 32400, 2015
reosarevok
Cursing and grumbling!
2015-11-20 32432, 2015
ruaok
heh
2015-11-20 32442, 2015
zas
all is running on ernie, suddenly nginx crashed on ernie, so it failovers and move all services to bert
2015-11-20 32407, 2015
zas
we can either: do nothing and wait admin noticed and eventually move all back to ernie
2015-11-20 32444, 2015
zas
add a failover timeout, with a preferred location (ernie), so it will try to move all back to ernie after some time
2015-11-20 32424, 2015
zas
should we have a "preferred location" ?
2015-11-20 32452, 2015
zas
or we don't care at all, until all is running on one server (but then it can migrate from ernie to bert, and stay on bert, until next issue)
2015-11-20 32437, 2015
zas
in case of failover we can clear failover history automatically or not
2015-11-20 32452, 2015
zas
ie. ernie -> service failure -> migrate to bert -> 2 days passed, no admin reset the failover -> service failure -> all stopped (it can't migrate back to ernie, since it knows it has failed)
2015-11-20 32407, 2015
ruaok submits the invoice for the JSON-LD contract
2015-11-20 32409, 2015
ruaok
\ø/
2015-11-20 32414, 2015
ruaok reads what zas wrote
2015-11-20 32436, 2015
ruaok
preferred location: ideally not, methinks.
2015-11-20 32400, 2015
zas
or ernie -> service failure -> migrate to bert -> 2 days passed, autoreset done after 6 hours -> service failure -> migrate to ernie -> if succeeded, ok, else all stopped
2015-11-20 32422, 2015
ruaok
shall we start simple and then adjust as we go?
2015-11-20 32440, 2015
ruaok
for me that means: do nothing, but notify admins.
2015-11-20 32452, 2015
ruaok
admins can inspect and adjust as needed.
2015-11-20 32405, 2015
zas
imho having a failover timeout of 24 hours ie. can help
2015-11-20 32410, 2015
ruaok
and when that becomes painful, commit it to code to automate
2015-11-20 32420, 2015
zas
it will try to migrate back if 24 hours have passed
2015-11-20 32429, 2015
Gentlecat has quit
2015-11-20 32447, 2015
zas
so it is an intermediate solution
2015-11-20 32402, 2015
ruaok
I'm ok with that if you prefer that.
2015-11-20 32451, 2015
zas
i think it is good, still require admin to check the thing in case of failure, but is able to recover in case it was a temporary failure, followed by another failure
so i will set a preferred location on ernie, so it will start services on it, and only migrate to bert in case of failures, then services will stay on bert, i'll adjust stickiness so it will only try to migrate back in case of failure and 24 hours passed
on a slightly different topic, I'm noodling on the idea of re-working Huesound ( http://s.huesound.org ) to work with the CAA and the tomahawk applet.
2015-11-20 32434, 2015
ruaok
then maybe put it on the homepage of the CAA as a showcase for the CA??
2015-11-20 32436, 2015
ruaok
+A
2015-11-20 32453, 2015
johtso joined the channel
2015-11-20 32412, 2015
Gentlecat
ok, merged
2015-11-20 32413, 2015
zas
ruaok: i have a setup exhibiting the correct behavior ;) : ip-dns tinydns dnscache on ernie, i stop dnscache, pacemaker migrates ip-dns tinydns dnscache to bert, after 2 minutes (for test, it will be 24 hours), i stop dnscache on bert, services are migrated back to ernie, dnscache restarted
2015-11-20 32430, 2015
Gentlecat
ruaok: do you want to deploy it?
2015-11-20 32401, 2015
zas
if i stop dnscache before the 2 minutes delay, the whole thing is shutdown (since it cannot work on ernie nor bert)
2015-11-20 32422, 2015
zas
and if i don't stop dnscache, it stays on bert
2015-11-20 32408, 2015
zas
i will add the rest now and do more tests
2015-11-20 32421, 2015
ruaok
Gentlecat: I can try.
2015-11-20 32425, 2015
ruaok
hmm. unsure how that should be done. there are two dirs.
2015-11-20 32440, 2015
ruaok
bitmap: do you know the procedure for upgrading the CAA?
2015-11-20 32437, 2015
Gentlecat
ruaok: you mentioned something about in-site communication a couple of weeks ago
2015-11-20 32421, 2015
Gentlecat
I think someone suggested you to use XMPP?
2015-11-20 32424, 2015
ruaok
yes
2015-11-20 32443, 2015
ruaok
that is the basic idea behind it.
2015-11-20 32456, 2015
ruaok
I asked Arc Riley, an XMPP dude about this.
2015-11-20 32458, 2015
Gentlecat
can you clarify a bit?
2015-11-20 32405, 2015
ruaok is doing just that
2015-11-20 32407, 2015
Gentlecat wants to start looking into that
2015-11-20 32419, 2015
ruaok
the first question I asked was if this was suited for this task.
2015-11-20 32421, 2015
ruaok
he said yes.
2015-11-20 32444, 2015
ruaok
so, XMPP can store message for offline users
2015-11-20 32405, 2015
ruaok
and using XMPP allows users to also connect their MB XMPP accounts to their favorite IM client.
he personally hacks on an apache module called Modxmpp.org
2015-11-20 32458, 2015
ruaok
he thought that within 2 weeks there would be a production ready release of this module.
2015-11-20 32428, 2015
ruaok
he further suggested that we should use Stanza.io for JS bits to make all of this easier.
2015-11-20 32446, 2015
ruaok
so, I haven't really looked into any of these pieces.
2015-11-20 32403, 2015
ruaok
your first task should be to look at all of these elements and see if they fit and meet our needs.
2015-11-20 32429, 2015
ruaok
then, adjust the jira tickets as apporpriate to reflect the new technology direction
2015-11-20 32452, 2015
Gentlecat
right
2015-11-20 32433, 2015
ruaok
mail intro to arc done.
2015-11-20 32457, 2015
zas
ruaok: can you explain me how the rate-limiter is working ? it is listening on UDP 10.1.1.245:2000, but does it need to run on the same machine of any other service ?
2015-11-20 32422, 2015
ruaok
no, it runs on the gateways only
2015-11-20 32435, 2015
Mogmi
___<;3
2015-11-20 32453, 2015
ruaok
when a request that comes in, the app that handles the request makes a call to the ratelimiter to see if the request is allowed.
2015-11-20 32414, 2015
ruaok
if so, it proceeds, if not a 503 is returned.
2015-11-20 32407, 2015
zas
it modifies iptables chains so it needs to run on the same machine as nginx, is there more constraints ?
2015-11-20 32417, 2015
zas
s/is/are/
2015-11-20 32424, 2015
UmkaDK_ joined the channel
2015-11-20 32437, 2015
zas
ie. dnscache/tinydns could run on ernie while the rest is on bert
2015-11-20 32438, 2015
ruaok
it modifies iptables??
2015-11-20 32415, 2015
zas
it doesn't ?
2015-11-20 32428, 2015
zas
hmmm, ok i really have to look at the code ;)
2015-11-20 32446, 2015
UmkaDK has quit
2015-11-20 32446, 2015
ruaok
not that I am familiar with.
2015-11-20 32400, 2015
ruaok
it runs in the application space, not the network space.
Should I add project logos for stuff such as libmusicbrainz and libcoverart too?
2015-11-20 32444, 2015
ruaok
phew. :)
2015-11-20 32458, 2015
ruaok
Freso: I'd say yes.
2015-11-20 32447, 2015
Freso
ruaok | and using XMPP allows users to also connect their MB XMPP accounts to their favorite IM client.
2015-11-20 32455, 2015
Freso
^ was one of my arguments when I suggested it. :)
2015-11-20 32421, 2015
ruaok nods
2015-11-20 32406, 2015
Freso
(As well as generally having tons and tons of software and libraries and what not that is XMPP compatible, meaning it would "easily" be possible to do some neat things with it.)
2015-11-20 32454, 2015
Freso
(So I'm glad Arc Riley managed to win you over, is what I'm trying to say. ;))
2015-11-20 32431, 2015
ruaok
he merely confirmed that the things we wanted to do are actually designed into the XMPP and work well.
2015-11-20 32440, 2015
ruaok
which was not easy to ascertain.
2015-11-20 32417, 2015
samphippen has quit
2015-11-20 32459, 2015
samphippen joined the channel
2015-11-20 32456, 2015
reosarevok
Freso: can you merge MetaBrainz and MetaBrainz Foundation there?
2015-11-20 32403, 2015
zas
ruaok: so ratelimit-server could possibly run on any machine, since apps are querying it through network, and i see no dependency in the code, it only depends on 10.1.1.245 IP address, it doesn't store starts on disk, am i wrong thinking it it has no stat (and therefore do not rate limit anything) when restarted ?
2015-11-20 32425, 2015
Freso
reosarevok: I sent a request about that ~2 hours ago. ;)
2015-11-20 32430, 2015
reosarevok
Ok :)
2015-11-20 32437, 2015
ruaok
zas: that sounds right
2015-11-20 32414, 2015
Freso
reosarevok: If you're talking about the two organisations. If you're talking about the MetaBrainz Website project and the MetaBrainz Foundation organisation, then no, those can't be merged.
2015-11-20 32425, 2015
reosarevok
Two orgs yep
2015-11-20 32432, 2015
Freso
Okay. Good. :)
2015-11-20 32424, 2015
Freso
Anyway, lib{coverart,musicbrainz} have had logos added too.
2015-11-20 32443, 2015
Freso
All dem shiny logos. *-*
2015-11-20 32447, 2015
reosarevok
Now looking at how to claim my stuff :p
2015-11-20 32414, 2015
zas
i'll set pacemaker accordingly, colocating this IP address and this service, it will make things easier to not create useless dependencies between IPs and services, failing over will run smoother (it will migrate this IP/ratelimiter independently of ie. external IP addresses or nginx)
2015-11-20 32420, 2015
samphippen has quit
2015-11-20 32429, 2015
zas
and it allows to run ratelmiter on bert, while we run nginx on ernie ;)
2015-11-20 32442, 2015
samphippen joined the channel
2015-11-20 32417, 2015
reosarevok has quit
2015-11-20 32417, 2015
reosarevok_ is now known as reosarevok
2015-11-20 32404, 2015
samphippen has quit
2015-11-20 32417, 2015
samphippen joined the channel
2015-11-20 32444, 2015
ruaok
woah.
2015-11-20 32400, 2015
zas
ok, coffee time
2015-11-20 32458, 2015
ruaok
might make my trip in SLO much much more work, but this is for a good cause.
alastairp: did you bring the package in or should I grab it from your house?
2015-11-20 32401, 2015
alastairp
you should grab it from my house
2015-11-20 32411, 2015
ruaok
ok, lets coordinate that over the weekend.
2015-11-20 32413, 2015
Gentlecat
I guess the problem is that incremental dumps still aren't published :)
2015-11-20 32430, 2015
alastairp
Gentlecat: sure they are. we just don't list their names. they're still there :)
2015-11-20 32432, 2015
alastairp
but yeah
2015-11-20 32454, 2015
alastairp
this is something else I want to do after the data migration
2015-11-20 32433, 2015
alastairp
there's a huge list of things waiting for that. in fact, I /just/ finalised some data formatting stuff with dmitry, and once I've implemented that the migration will be done
