0:00 AM
Lotheric has quit
2023-06-11 16204, 2023
0:00 AM
atj has quit
2023-06-11 16204, 2023
0:00 AM
jasje has quit
2023-06-11 16204, 2023
0:00 AM
antlarr has quit
2023-06-11 16204, 2023
0:00 AM
rain0r has quit
2023-06-11 16213, 2023
0:00 AM
jasje joined the channel
2023-06-11 16218, 2023
0:00 AM
antlarr joined the channel
2023-06-11 16222, 2023
0:00 AM
rain0r joined the channel
2023-06-11 16238, 2023
0:00 AM
atj joined the channel
2023-06-11 16201, 2023
0:02 AM
Lotheric joined the channel
2023-06-11 16223, 2023
2:26 AM
Sciencentistguy4 joined the channel
2023-06-11 16241, 2023
2:28 AM
Sciencentistguy has quit
2023-06-11 16241, 2023
2:28 AM
Sciencentistguy4 is now known as Sciencentistguy
2023-06-11 16202, 2023
2:38 AM
ben62 joined the channel
2023-06-11 16202, 2023
3:00 AM
ben62 has quit
2023-06-11 16202, 2023
4:59 AM
BrainzGit
2023-06-11 16248, 2023
4:59 AM
BrainzGit
2023-06-11 16205, 2023
7:47 AM
Sophist-UK has quit
2023-06-11 16225, 2023
7:47 AM
Sophist-UK joined the channel
2023-06-11 16247, 2023
9:05 AM
santiagofn has quit
2023-06-11 16206, 2023
9:43 AM
smach joined the channel
2023-06-11 16214, 2023
9:44 AM
smach has quit
2023-06-11 16204, 2023
11:16 AM
tykling
~/sb end
2023-06-11 16211, 2023
11:16 AM
tykling
whoops
2023-06-11 16211, 2023
12:07 PM
rozlav joined the channel
2023-06-11 16252, 2023
14:26 PM
rozlav has quit
2023-06-11 16201, 2023
14:36 PM
rozlav joined the channel
2023-06-11 16201, 2023
17:50 PM
Maxr1998_ joined the channel
2023-06-11 16207, 2023
17:51 PM
Maxr1998 has quit
2023-06-11 16224, 2023
19:11 PM
q3lont joined the channel
2023-06-11 16259, 2023
19:22 PM
ZaphodBeeblebrox joined the channel
2023-06-11 16259, 2023
19:22 PM
ZaphodBeeblebrox has quit
2023-06-11 16259, 2023
19:22 PM
ZaphodBeeblebrox joined the channel
2023-06-11 16257, 2023
19:24 PM
CatQuest joined the channel
2023-06-11 16257, 2023
19:24 PM
CatQuest has quit
2023-06-11 16257, 2023
19:24 PM
CatQuest joined the channel
2023-06-11 16212, 2023
19:25 PM
CatQuest has quit
2023-06-11 16228, 2023
19:25 PM
CatQuest joined the channel
2023-06-11 16228, 2023
19:25 PM
CatQuest has quit
2023-06-11 16228, 2023
19:25 PM
CatQuest joined the channel
2023-06-11 16229, 2023
19:26 PM
CatQuest joined the channel
2023-06-11 16229, 2023
19:26 PM
CatQuest has quit
2023-06-11 16229, 2023
19:26 PM
CatQuest joined the channel
2023-06-11 16213, 2023
19:28 PM
CatQuest has quit
2023-06-11 16205, 2023
19:37 PM
CatQuest joined the channel
2023-06-11 16205, 2023
19:37 PM
CatQuest has quit
2023-06-11 16205, 2023
19:37 PM
CatQuest joined the channel
2023-06-11 16256, 2023
19:49 PM
q3lont has quit
2023-06-11 16214, 2023
21:10 PM
ben62 joined the channel
2023-06-11 16233, 2023
21:34 PM
mayhem
2023-06-11 16201, 2023
21:36 PM
mayhem
a restart fixed it
2023-06-11 16231, 2023
21:39 PM
mayhem
the python requests module cannot call the labs.api endpoint for the reason that lucifer mentioned before where a local update fixes the problem. but how many other legit callers will get a failed call??
2023-06-11 16246, 2023
21:39 PM
mayhem
time to look at this tomorrow
2023-06-11 16208, 2023
21:51 PM
ben62 has quit
2023-06-11 16250, 2023
21:53 PM
zas
mayhem: I had a look, there's a problem with redis (keydb)
2023-06-11 16207, 2023
21:58 PM
zas
we hit a limit, not sure why yet
2023-06-11 16212, 2023
21:58 PM
zas
2023-06-11 16236, 2023
21:58 PM
mayhem
odd. are older connections not being closed?
2023-06-11 16228, 2023
22:01 PM
zas
something like that
2023-06-11 16223, 2023
22:02 PM
zas
2023-06-11 16230, 2023
22:02 PM
zas
weird
2023-06-11 16250, 2023
22:02 PM
mayhem
eh?
2023-06-11 16226, 2023
22:05 PM
zas
2023-06-11 16232, 2023
22:05 PM
zas
not good
2023-06-11 16209, 2023
22:06 PM
zas
2023-06-11 16225, 2023
22:08 PM
mayhem
eeeek
2023-06-11 16254, 2023
22:12 PM
zas
I stopped the instance on rudi
2023-06-11 16252, 2023
22:15 PM
zas
2023-06-11 16208, 2023
22:16 PM
zas
2023-06-11 16215, 2023
22:16 PM
zas
4.7G...
2023-06-11 16227, 2023
22:16 PM
zas
there are supposed to be the same size
2023-06-11 16237, 2023
22:16 PM
zas
so the instance on rudi went to bananas (heavily)
2023-06-11 16212, 2023
22:21 PM
mayhem
that will please monkey 🍌
2023-06-11 16228, 2023
22:29 PM
zas
ok, replica resynced, things are back to normal
2023-06-11 16242, 2023
22:29 PM
zas
I have no idea about what happened though
2023-06-11 16200, 2023
22:30 PM
mayhem
yep, looks better! thanks!
2023-06-11 16212, 2023
22:30 PM
zas
perhaps we hit a bug in keydb, I'll check
2023-06-11 16246, 2023
22:30 PM
zas
it never happened before
2023-06-11 16258, 2023
22:30 PM
zas
or perhaps rudi still has a problem
2023-06-11 16213, 2023
22:33 PM
zas
aof file on rudi was to maxsize, so this replica was failing, causing connections left open on rex keydb (the other replica)
2023-06-11 16258, 2023
22:33 PM
zas
we need alerts on this, it took several hours to get critical
2023-06-11 16213, 2023
22:35 PM
zas
too late for me to have clear thoughts on this
2023-06-11 16204, 2023
22:36 PM
zas
I solved it by totally removing the buggy db, and just starting again the container on rudi so it resynced with rex
2023-06-11 16243, 2023
22:36 PM
mayhem
agreed, I"m off to bed.
2023-06-11 16250, 2023
22:36 PM
zas
gn
2023-06-11 16247, 2023
23:28 PM
nick5 joined the channel
2023-06-11 16241, 2023
23:29 PM
nick5 has quit
