ruaok: i set up daemontools service for jetty, check /etc/service/jetty-service, on dora & roobarb. It uses softlimit to take care of memory leaks and run under unpriviledged user, logs available too
2015-11-24 32819, 2015
zas
but i dont think it works as it should ... :(
2015-11-24 32822, 2015
zas
servers are receiving queries and answering but stats shows our ratio between 2xx and 5xx is still abd
2015-11-24 32832, 2015
outsidecontext joined the channel
2015-11-24 32842, 2015
zas
they exhibit the exact same behavior at startup, huge cpu consumption for 7-8 mins
2015-11-24 32822, 2015
zas
ok, they are running now. But imho they still have the (not) slow start issue, they couldn't catch up when started with full request rate.
2015-11-24 32843, 2015
zas
i had to reduce it, start servers, and increase it again
2015-11-24 32832, 2015
zas
so this is more related to the search server code than the java env. Though jetty memory usage is lower and cpu looks a bit better too
2015-11-24 32807, 2015
zas
ruaok: i modified search restart scripts on dora/roobarb to svc -t /etc/service/jetty-service instead of the wild killall -9 java, and restarted the indexer
2015-11-24 32832, 2015
zas
apart monitoring (nagios/graphana), the rest is ok.
2015-11-24 32854, 2015
zas
we'll see if they are stabler than tomcat
2015-11-24 32858, 2015
zas
i have to do a pause now, if you want to set nagios, perhaps just change ports to 8080 on jetty configs and ernie's nginx one.
2015-11-24 32818, 2015
ariscop joined the channel
2015-11-24 32849, 2015
zas
bitmap: Guess Case in artist's editor doesn't match english guidelines for "stouffi the stouves" it capitalizes it as "Stouffi The Stouves" instead of "Stouffi the Stouves" (https://musicbrainz.org/doc/Style/Language/English, 2a)
I love that they have a .it domain for a .es shop. :)
2015-11-24 32812, 2015
Freso
Grazie.
2015-11-24 32829, 2015
Freso
(It's a .com though, but sure.)
2015-11-24 32832, 2015
zas
ruaok_: about search servers, read the back log ;)
2015-11-24 32844, 2015
ruaok_
ah, yes.
2015-11-24 32805, 2015
ruaok_
zas: I did, but it wasn't quite clear enough.
2015-11-24 32815, 2015
ruaok_
running on jetty now, but still have a slow start problem, right?
2015-11-24 32828, 2015
zas
yes, it seems, i set up daemontools for it, restarted on both, and they failed to catch up, so i did the usual procedure, reducing rate limit, then stop one, restart the other, and set the rate limit up after a while, and it works
2015-11-24 32832, 2015
zas
also the search server code is emitting quite a lot errors, but i think it isn't jetty related
2015-11-24 32806, 2015
ruaok_
sigh.
2015-11-24 32813, 2015
zas
i did enough for today really need to do something else ;)
2015-11-24 32832, 2015
ruaok_
ok, I'll keep an eye on things for the next couple of hours.
2015-11-24 32842, 2015
ruaok_
lets touch base on this tomorrow.
2015-11-24 32853, 2015
zas
yes
2015-11-24 32854, 2015
zas
ah, and it seems Leftmost killed rika
2015-11-24 32828, 2015
zas
it answers to ping, but no port open, no ssh
2015-11-24 32836, 2015
zas
after the reboot
2015-11-24 32803, 2015
zas
do we have a kvm or smt like that on those machines ?
2015-11-24 32832, 2015
ruaok_
nothing of the like. :(
2015-11-24 32847, 2015
zas
any human ? ;)
2015-11-24 32854, 2015
Freso
ruaok_: Did we have other sponsors than Google and Spotify?
Might be interested to note when we made "add release" et al edits auto-edits.
2015-11-24 32810, 2015
Freso
*interesting
2015-11-24 32822, 2015
bitmap
oh, good question, I'm not sure
2015-11-24 32844, 2015
ruaok_
not hard, IIRC. we need to back-fill a few things...
2015-11-24 32806, 2015
Freso
Mostly I'm asking whether it's at the level of "sure, your ping here is enough, let me get right on it" or "please make a ticket so I can look into this later".
2015-11-24 32811, 2015
ruaok_
bitmap: do you remember me packing my own care package in alviso?
Freso: Leftmost: It might be good to do a followup blog post on rika saying it'll be down for a longer period than anticipated
2015-11-24 32848, 2015
Freso
LordSputnik: chirlu` made a comment on the blog post about it.
2015-11-24 32856, 2015
Freso
I think that's probably enough.
2015-11-24 32813, 2015
chirlu`
Yeah, I guess there are very few people who care.
2015-11-24 32818, 2015
Freso
Yeah.
2015-11-24 32820, 2015
chirlu`
And even fewer who arent here.
2015-11-24 32826, 2015
Freso
Yes. :p
2015-11-24 32844, 2015
Freso
And those who do and aren't here should be tech savvy enough to read the comments to the post.
2015-11-24 32819, 2015
Freso
Perhaps Leftmost could make an edit-addendum to the blog post noting that the unforeseen has happened and the downtime will be significantly longer than planned.
2015-11-24 32834, 2015
Freso
(Ie., so it's right in the blog post and not in a comment.)
2015-11-24 32839, 2015
LordSputnik
I think if chirlu` has commented it should probably be enough ;)
2015-11-24 32844, 2015
Freso
(Well, it'd also be in a comment. :))
2015-11-24 32803, 2015
LordSputnik
As you say, anyone with a sandbox who cares is likely to already know one way or the other
2015-11-24 32821, 2015
LordSputnik
Time to install BookBrainz on a Raspberry Pi for pre-release testing :D
