14:22 PM
ruaok
I had an import at the top of the file, when it should've been just prior to the init call
2015-10-22 29533, 2015
14:22 PM
ruaok
alastairp: you familia with berkshelf?
2015-10-22 29542, 2015
14:22 PM
alastairp
not at all
2015-10-22 29546, 2015
14:22 PM
samphippen has quit
2015-10-22 29547, 2015
14:22 PM
ruaok
I'm trying to get redis added to chef.
2015-10-22 29550, 2015
14:22 PM
ruaok
no luck yet.
2015-10-22 29557, 2015
14:22 PM
samphippen joined the channel
2015-10-22 29535, 2015
14:23 PM
xBytez joined the channel
2015-10-22 29535, 2015
14:23 PM
xBytez has quit
2015-10-22 29535, 2015
14:23 PM
xBytez joined the channel
2015-10-22 29548, 2015
14:23 PM
zas joined the channel
2015-10-22 29521, 2015
14:28 PM
Nyanko-sensei has quit
2015-10-22 29528, 2015
14:38 PM
ruaok
2015-10-22 29529, 2015
14:42 PM
Nyanko-sensei joined the channel
2015-10-22 29558, 2015
14:42 PM
Nyanko-sensei has quit
2015-10-22 29505, 2015
14:43 PM
Nyanko-sensei joined the channel
2015-10-22 29543, 2015
14:43 PM
Lotheric has quit
2015-10-22 29524, 2015
14:44 PM
Lotheric joined the channel
2015-10-22 29536, 2015
14:45 PM
github joined the channel
2015-10-22 29536, 2015
14:45 PM
github
2015-10-22 29536, 2015
14:45 PM
github
listenbrainz-server/rate-limiting 38b4ed3 Robert Kaye: Fix for a problem that GC is getting
2015-10-22 29536, 2015
14:45 PM
github has left the channel
2015-10-22 29503, 2015
14:46 PM
ruaok
alastairp: when exactly is beer time?
2015-10-22 29536, 2015
14:48 PM
github joined the channel
2015-10-22 29536, 2015
14:48 PM
github
2015-10-22 29536, 2015
14:48 PM
github
listenbrainz-server/rate-limiting 9d9d030 Robert Kaye: Fix two wrong log_raise calls
2015-10-22 29536, 2015
14:48 PM
github has left the channel
2015-10-22 29524, 2015
14:51 PM
samphippen has quit
2015-10-22 29530, 2015
14:54 PM
github joined the channel
2015-10-22 29530, 2015
14:54 PM
github
2015-10-22 29530, 2015
14:54 PM
github
listenbrainz-server/rate-limiting 080d41d Robert Kaye: Adding redis to requirements
2015-10-22 29530, 2015
14:54 PM
github has left the channel
2015-10-22 29540, 2015
14:55 PM
github joined the channel
2015-10-22 29540, 2015
14:55 PM
github
2015-10-22 29540, 2015
14:55 PM
github
listenbrainz-server/rate-limiting cd8baf6 Robert Kaye: Fix copypasta in set_rate_limits
2015-10-22 29540, 2015
14:55 PM
github has left the channel
2015-10-22 29552, 2015
14:55 PM
alastairp
ruaok: ~30 minutes?
2015-10-22 29511, 2015
14:56 PM
ruaok
alas, I won't be free until much later. :(
2015-10-22 29524, 2015
14:56 PM
alastairp
well, someone here did mention an hour
2015-10-22 29528, 2015
14:56 PM
alastairp
would that be better?
2015-10-22 29510, 2015
14:57 PM
ruaok
not free til 20:30. :(
2015-10-22 29521, 2015
14:57 PM
alastairp
ok. no problem
2015-10-22 29539, 2015
14:57 PM
alastairp
I think I'll just go and decompress. this hasn't been a good day
2015-10-22 29550, 2015
14:57 PM
github joined the channel
2015-10-22 29550, 2015
14:57 PM
github
2015-10-22 29550, 2015
14:57 PM
github
listenbrainz-server/rate-limiting 2c10d38 Robert Kaye: FUSS!
2015-10-22 29550, 2015
14:57 PM
github has left the channel
2015-10-22 29551, 2015
14:57 PM
alastairp gunzips
2015-10-22 29558, 2015
14:57 PM
ruaok
k. ping me if you're still at it come 20:30 :)
2015-10-22 29503, 2015
14:58 PM
ocharles___ joined the channel
2015-10-22 29524, 2015
14:58 PM
RJ2_ joined the channel
2015-10-22 29525, 2015
14:58 PM
Gentlecat
🍻🐝🐝🐝
2015-10-22 29528, 2015
14:58 PM
CallerNo6 joined the channel
2015-10-22 29554, 2015
14:58 PM
madmouser1_ joined the channel
2015-10-22 29533, 2015
14:59 PM
alastairp
Gentlecat: so, enjoying barcelona?? :)
2015-10-22 29536, 2015
14:59 PM
ruaok
this day has been a complete waste.
2015-10-22 29543, 2015
14:59 PM
alastairp
Gentlecat: what's the state of the stuff you were doing?
2015-10-22 29544, 2015
14:59 PM
ruaok
I should cancel my appt and come have beer
2015-10-22 29548, 2015
14:59 PM
alastairp
well, let me come an talk to you
2015-10-22 29505, 2015
15:00 PM
Gentlecat
alastairp: "vagrant up" is the state :)
2015-10-22 29524, 2015
15:00 PM
legoktm_ joined the channel
2015-10-22 29551, 2015
15:00 PM
sonnenstrahl joined the channel
2015-10-22 29559, 2015
15:01 PM
github joined the channel
2015-10-22 29559, 2015
15:01 PM
github
2015-10-22 29559, 2015
15:01 PM
github
listenbrainz-server/rate-limiting f7a460c Roman Tsukanov: Typos. :rage:
2015-10-22 29559, 2015
15:01 PM
github has left the channel
2015-10-22 29550, 2015
15:02 PM
ruaok
zas: are you working in jira?
2015-10-22 29554, 2015
15:02 PM
ruaok
*on
2015-10-22 29507, 2015
15:04 PM
Leo__Verto joined the channel
2015-10-22 29510, 2015
15:04 PM
zas
nope, i'm trying to sort out the chef mess, and figuring out how we can simplify and regroup our cookbooks
2015-10-22 29526, 2015
15:04 PM
ruaok
yep, I'm stuck in chef hell too
2015-10-22 29529, 2015
15:04 PM
zas
i'm testing gateway-chef on vm
2015-10-22 29509, 2015
15:05 PM
zas
and i noticed we already using daemontools recipe in chef-cookbooks, so it would perhaps be good to regroup chef-gateway and server-configs
2015-10-22 29520, 2015
15:06 PM
zas
after all, we have nodes configured in server-configs, we can add more, and merge chef-gateway, collectd-chef, (and the new graphite cookbook) in server-configs + chef-cookbooks
2015-10-22 29546, 2015
15:06 PM
madmouser1 has quit
2015-10-22 29546, 2015
15:06 PM
Leo_Verto has quit
2015-10-22 29546, 2015
15:06 PM
domme has quit
2015-10-22 29546, 2015
15:06 PM
ocharles__ has quit
2015-10-22 29546, 2015
15:06 PM
RJ2 has quit
2015-10-22 29547, 2015
15:06 PM
RustyB has quit
2015-10-22 29547, 2015
15:06 PM
legoktm has quit
2015-10-22 29547, 2015
15:06 PM
Leo__Verto is now known as Leo_Verto
2015-10-22 29507, 2015
15:07 PM
zas
server-configs is deploying depending on the hostname (short, ie. astro or ernie)
2015-10-22 29501, 2015
15:08 PM
Gentlecat
2015-10-22 29513, 2015
15:10 PM
ocharles___ is now known as ocharles__
2015-10-22 29529, 2015
15:10 PM
RJ2_ is now known as RJ2
2015-10-22 29535, 2015
15:10 PM
ruaok
RJ2: Got a sec for a Berkself question?
2015-10-22 29554, 2015
15:10 PM
ruaok
I need to install redis for LB.
2015-10-22 29502, 2015
15:11 PM
ruaok
so I added "cookbook 'redis', '~> 3.0.4'" to Berksfile
2015-10-22 29523, 2015
15:11 PM
ruaok
and ran "berks install" and then "kitchen converge"
2015-10-22 29534, 2015
15:11 PM
ruaok
still no redis. Is there a step I am missing?
2015-10-22 29546, 2015
15:12 PM
diana_olhovik has quit
2015-10-22 29555, 2015
15:37 PM
RustyB joined the channel
2015-10-22 29508, 2015
15:40 PM
ruaok has quit
2015-10-22 29526, 2015
15:53 PM
reosarevok has quit
2015-10-22 29539, 2015
15:55 PM
ruaok joined the channel
2015-10-22 29511, 2015
16:04 PM
New__ joined the channel
2015-10-22 29512, 2015
16:06 PM
Major_Lurker has quit
2015-10-22 29529, 2015
16:32 PM
blup
in norway it's beer and coffee :P
2015-10-22 29529, 2015
16:51 PM
The_Freso joined the channel
2015-10-22 29506, 2015
16:53 PM
Freso has quit
2015-10-22 29506, 2015
16:53 PM
The_Freso is now known as Freso
2015-10-22 29533, 2015
16:57 PM
UmkaDK has quit
2015-10-22 29518, 2015
17:05 PM
JonnyJD has quit
2015-10-22 29558, 2015
17:37 PM
legoktm_ has quit
2015-10-22 29558, 2015
17:37 PM
legoktm_ joined the channel
2015-10-22 29526, 2015
17:58 PM
Lotheric
"read timeout at /usr/share/perl5/Net/HTTP/Methods.pm line 257, line 3." searching for an artist
2015-10-22 29537, 2015
17:58 PM
Lotheric
on regular musicbrainz
2015-10-22 29541, 2015
17:58 PM
Lotheric
(not beta)
2015-10-22 29528, 2015
17:59 PM
Lotheric
same thing on beta
2015-10-22 29531, 2015
17:59 PM
Lotheric
just tried
2015-10-22 29531, 2015
18:02 PM
legoktm_ is now known as legoktm
2015-10-22 29550, 2015
18:02 PM
Lotheric
2015-10-22 29528, 2015
18:13 PM
diana_olhovik_ joined the channel
2015-10-22 29559, 2015
18:22 PM
diana_olhovik_ has quit
2015-10-22 29559, 2015
18:27 PM
zas
ruaok, search servers have hard times again, i killed tomcat once
2015-10-22 29509, 2015
18:52 PM
ruaok returns
2015-10-22 29518, 2015
18:53 PM
ruaok
yep,this is the same thing that happened yesterday.
2015-10-22 29536, 2015
18:53 PM
ruaok
I didn't do anything to fix it, so I have no idea what the problem could be.
2015-10-22 29524, 2015
18:56 PM
ruaok
zas:ping
2015-10-22 29547, 2015
18:56 PM
zas
i'm checking, no idea for now
2015-10-22 29558, 2015
18:56 PM
ruaok
there you are.
2015-10-22 29509, 2015
18:57 PM
zas
we have a lot of free memory on those servers, weird
2015-10-22 29509, 2015
18:57 PM
ruaok
lets compare notes, maybe we can sort this together.
2015-10-22 29529, 2015
18:57 PM
ruaok
top with show threads shows MASSIVE numbers of threads running all at the same time.
2015-10-22 29536, 2015
18:57 PM
ruaok
way more than normal.
2015-10-22 29515, 2015
18:58 PM
zas
is there a limit to this number somewhere ?
2015-10-22 29533, 2015
18:58 PM
ruaok
often limited by number of open file handles.
2015-10-22 29537, 2015
18:58 PM
ruaok
but have that cranked way up.
2015-10-22 29550, 2015
18:58 PM
ruaok
funny thing -- this happened about the same time as yesterday.
2015-10-22 29554, 2015
18:59 PM
ruaok
the las serious problem we had with the search servers ended up being our old gateways.
2015-10-22 29557, 2015
18:59 PM
zas
yes, a bit later, by one hour
2015-10-22 29507, 2015
19:00 PM
ruaok
they were dropping inbound packets like crazy.
2015-10-22 29526, 2015
19:00 PM
ruaok
so the rate limiter, was very slow to be reached.
2015-10-22 29542, 2015
19:00 PM
ruaok
but that meant that the load skyrocketed, but CPU was near 0%
2015-10-22 29553, 2015
19:00 PM
ruaok
in this case, we're CPU maxxed.
2015-10-22 29533, 2015
19:02 PM
zas
2015-10-22 29555, 2015
19:02 PM
zas
2015-10-22 29534, 2015
19:03 PM
ruaok
is that a symptom of the search servers misbehaving?
2015-10-22 29548, 2015
19:03 PM
ruaok
or is that something that causes the search server to misbehave.
2015-10-22 29501, 2015
19:04 PM
ruaok
I get a 404 on the second link.
2015-10-22 29502, 2015
19:04 PM
zas
i think it is because they misbehave
2015-10-22 29505, 2015
19:04 PM
zas
2015-10-22 29527, 2015
19:04 PM
zas
much more 5xx than 2xx
2015-10-22 29543, 2015
19:04 PM
zas
it started at 19:50
2015-10-22 29504, 2015
19:05 PM
zas
without noticeable increase in traffic
2015-10-22 29504, 2015
19:05 PM
ruaok
yesterday the problem lasted about an hour.
2015-10-22 29518, 2015
19:05 PM
ruaok
maybe it will go away in a minute?
2015-10-22 29525, 2015
19:05 PM
ruaok
wishful thinking, I know.
2015-10-22 29516, 2015
19:06 PM
zas
do we have a log of queries ?
