-- BotBot disconnected, possible missing messages --
2015-11-20 32425, 2015
-- BotBot disconnected, possible missing messages --
2015-11-20 32449, 2015
BrainzBot joined the channel
2015-11-20 32440, 2015
qtjojo has quit
2015-11-20 32431, 2015
qtjojo joined the channel
2015-11-20 32401, 2015
Leftmost
LordSputnik, I don't think I have a link to the doc on updating rika. Could you send that my way when you're on next?
2015-11-20 32442, 2015
Leftmost
Also, I have an inkling of an idea on how to deal with revisions. Instead of creating them directly with a create function, we do it with a trigger on saving an Entity or EntityData instance.
2015-11-20 32435, 2015
qtjojo has quit
2015-11-20 32439, 2015
qtjojo2 joined the channel
2015-11-20 32413, 2015
diana_olhovik_ joined the channel
2015-11-20 32407, 2015
qtjojo2 has quit
2015-11-20 32429, 2015
JonnyJD_ joined the channel
2015-11-20 32438, 2015
ariscop has quit
2015-11-20 32402, 2015
reosarevok joined the channel
2015-11-20 32442, 2015
jesus2099 joined the channel
2015-11-20 32409, 2015
ariscop joined the channel
2015-11-20 32439, 2015
jesus2099 has quit
2015-11-20 32427, 2015
ruaok joined the channel
2015-11-20 32455, 2015
zas
Good morning, Rob
2015-11-20 32435, 2015
samphippen joined the channel
2015-11-20 32454, 2015
samphippen has quit
2015-11-20 32411, 2015
ariZon_a has quit
2015-11-20 32420, 2015
ariZon_a joined the channel
2015-11-20 32459, 2015
ruaok has quit
2015-11-20 32446, 2015
samphippen joined the channel
2015-11-20 32448, 2015
ruaok joined the channel
2015-11-20 32426, 2015
ruaok has quit
2015-11-20 32403, 2015
ruaok joined the channel
2015-11-20 32419, 2015
ruaok
awesome network is awesome.
2015-11-20 32450, 2015
ruaok has quit
2015-11-20 32439, 2015
ruaok joined the channel
2015-11-20 32449, 2015
ruaok sighs
2015-11-20 32414, 2015
reosarevok
Heh. Where are you this time?
2015-11-20 32427, 2015
ruaok
at the uni.
2015-11-20 32407, 2015
ruaok
and this is the worst network connection in weeks. :(
it concerns the modgroup command, which doesn't do what it should, not preserving the order in the group after you commit
2015-11-20 32457, 2015
Nyanko-sensei has quit
2015-11-20 32440, 2015
ruaok
gah network lag. :(
2015-11-20 32430, 2015
zas
if you have group G A B and modgroup G add C before B, it works (crm configure show: G A C B) but when after the commit it is set to G A B C ... and since order in the group is very important, it mess up the setup
2015-11-20 32420, 2015
zas
i didn't search yet if it is a known issue (perhaps fixed in later versions)
2015-11-20 32406, 2015
zas
in fact group is working like colocation + order, so they are redundant if used together (ie. initial setup you made)
2015-11-20 32441, 2015
zas
ruaok: to recover you need to reduce rate limiter value to 500 ie
2015-11-20 32446, 2015
zas
else it will not work :(
2015-11-20 32453, 2015
ruaok
500?
2015-11-20 32454, 2015
ruaok
wow.
2015-11-20 32420, 2015
zas
500, then let search servers recover, then move it back to original value
2015-11-20 32427, 2015
ruaok nods
2015-11-20 32429, 2015
zas
i wonder with they went mad again
2015-11-20 32430, 2015
ruaok
it used to be 1000
2015-11-20 32451, 2015
zas
you can try 1000, but last time i had to go even further down
2015-11-20 32409, 2015
ruaok
1000 didn't work, on 500 now
2015-11-20 32423, 2015
zas
this is damn annoying
2015-11-20 32437, 2015
ruaok nods
2015-11-20 32450, 2015
ruaok
still not recovering.
2015-11-20 32458, 2015
zas
try to stop one server, and let the other going up (it takes 7-8 minutes)
2015-11-20 32406, 2015
ruaok did just that
2015-11-20 32447, 2015
zas
this is incredible they can't recover by themselves... we need to find a way to slow start them, bitmap suggested to use the rate limiter for that, though i have no idea yet on how to set up something simple enough to be reliable
2015-11-20 32417, 2015
ruaok
I dont think slow starting them is all the hard.
2015-11-20 32429, 2015
zas
but another approach would to find WHY they suddenly crash, and fix the issue at the root
2015-11-20 32445, 2015
ruaok
ok, recovered now
2015-11-20 32417, 2015
ruaok
so, in order to cold start a search server we just need to write a script that executes searches on each of the indexes.
2015-11-20 32449, 2015
ruaok
pick a random dictionary word, do a search, repeat.
2015-11-20 32406, 2015
ruaok
do that for X times on each index and things should be safe enough to restart.
2015-11-20 32443, 2015
zas
we need to block requests until the slow start process is finished
2015-11-20 32404, 2015
ruaok
that part is harder. :(
2015-11-20 32407, 2015
zas
a simple iptables rule could do
2015-11-20 32429, 2015
zas
with a REJECT
2015-11-20 32418, 2015
ruaok
and then nginx would fail out that search server.
2015-11-20 32421, 2015
ruaok
nice idea. :)
2015-11-20 32456, 2015
zas
block, start server, do slow start queries, unblock
2015-11-20 32457, 2015
zas
ok both servers are back to business, until next crash
2015-11-20 32419, 2015
ruaok nods to both
2015-11-20 32455, 2015
zas
about gateway-chef, i'm working on the pacemaker config, and it is a damn hell, but i'm now knowing much more about it
2015-11-20 32418, 2015
zas
rule: it doesn't work as you think it works ;)
2015-11-20 32420, 2015
ruaok
there is a lot of know. :(
2015-11-20 32423, 2015
ruaok
oy
2015-11-20 32439, 2015
zas
plus it has bugs
2015-11-20 32450, 2015
zas
small ones, annoying ones
2015-11-20 32422, 2015
zas
i rewrote all cookbooks, so they play nicely now
2015-11-20 32439, 2015
ruaok
<3
2015-11-20 32441, 2015
ruaok
thank you!
2015-11-20 32413, 2015
zas
i can install both gateways on VMs easily, but the crm config isn't ready yet
2015-11-20 32429, 2015
zas
i'm thinking about not using groups at all
2015-11-20 32447, 2015
zas
dnscache / tinydns depends only on one internal IP
2015-11-20 32409, 2015
zas
mx/mail IPs are independent from the rest (well, i think...)
2015-11-20 32436, 2015
zas
nginx / rate limiter depends on both external IPs and few internal IPs
2015-11-20 32413, 2015
zas
tinydns has to start before dnscache (since it forwards requests to tinydns)
2015-11-20 32455, 2015
zas
we have 2 possibilities, either all resources can run anywhere by default (meaning they start as soon they are configured), or the reverse which implies to explicitely define where they can run, more complex but it ensures no resource is started until we say to
2015-11-20 32430, 2015
zas
also i found a nice thing
2015-11-20 32442, 2015
ruaok
I always wondered about all the IPs we use.
2015-11-20 32444, 2015
zas
ifconfig doesn't report configured interfaces
2015-11-20 32450, 2015
zas
ip command does
2015-11-20 32456, 2015
ruaok
would everything be simpler if we used 1 IP for everything?
2015-11-20 32416, 2015
zas
but there is a way to have them displayed, using iflabel in primitives
2015-11-20 32426, 2015
zas
one IP ?
2015-11-20 32446, 2015
ruaok
we currently have 5 or so IPs that can fail over.
2015-11-20 32400, 2015
ruaok
would it be easier with 1 and have all the services on 1?
2015-11-20 32435, 2015
zas
it will prolly easier, though i think i can make it work as is
2015-11-20 32413, 2015
Nyanko-sensei joined the channel
2015-11-20 32417, 2015
zas
basically if a service fails (or an IP) we migrate all to the other server anyway
2015-11-20 32427, 2015
ruaok
exactly. :)
2015-11-20 32446, 2015
zas
this is something that was not working, ie. if dnscache failed, all were stopped ...
2015-11-20 32441, 2015
zas
now the thing i didnt do yet, is deep checks: we currently test the services are running ... not they are working (nginx can run, and still not answering ie.)
2015-11-20 32413, 2015
zas
i prepared what is needed to do that, we can add an extracheck=program for each service
2015-11-20 32438, 2015
ruaok is starving
2015-11-20 32411, 2015
ruaok
feh, I need to buy is the irccloud package. but I can't think straight, so hungry.
2015-11-20 32413, 2015
ruaok
back in a bit.
2015-11-20 32414, 2015
zas
obviously each check program is different, depending on what we check (dns requests / web requests)
