3:48 AM
minimal has quit
4:01 AM
opticdelusion has quit
4:02 AM
opticdelusion joined the channel
5:56 AM
v6lur joined the channel
7:15 AM
Kladky joined the channel
8:34 AM
zas[m]
yvanzo: any idea what happens with SIR? It seems to handle messages correctly, but the index queue increased a lot, and the traffic from SIR seems unusually low
8:54 AM
d4rk-ph0enix has quit
8:54 AM
d4rk-ph0enix joined the channel
9:34 AM
d4rk-ph0enix has quit
9:34 AM
d4rk-ph0enix joined the channel
9:37 AM
jasje joined the channel
9:49 AM
Maxr1998 has quit
9:52 AM
Maxr1998 joined the channel
9:57 AM
zerodogg has quit
10:40 AM
zerodogg joined the channel
11:03 AM
yvanzo[m]
zas: No. I confirm that SIR “seems to handle messages correctly” (from its own logs), and that “the index queue [is still increasing] a lot” (from our dashboard).
11:03 AM
Some search queries are timing out (from Solr admin logging), but I can’t tell whether or not this is unusual and/or relevant.
11:08 AM
antlarr has quit
11:15 AM
antlarr joined the channel
11:47 AM
jasje has quit
11:49 AM
The initial surge of index messages seems to be due to hundreds of release countries being removed from 18 releases totalizing 678 tracks fully detailed with several relationships each:
11:49 AM
11:53 AM
kellnerd joined the channel
11:53 AM
antlarr has quit
11:53 AM
kellnerd has quit
12:06 PM
Reported the issue since this likely triggered way more index messages than it should.
12:07 PM
zas: Where do you check the traffic from SIR?
12:10 PM
antlarr joined the channel
12:13 PM
(I can see the alert, but I am looking for actual logs to compare with SIR logs.)
12:25 PM
zas
12:25 PM
LB HTTP post rates
12:25 PM
receive bytes per frontend, and few more dashboards
12:28 PM
yvanzo[m]
Thank you.
12:28 PM
There are some requests to /update in Solr logs without being able to compare with the numbers provided by SIR logs.
12:28 PM
zas
the number of TCP SYN too (it dropped abnormally low)
12:29 PM
yvanzo[m]
Restarting SIR now.
12:29 PM
zas
I tried a while ago, but it had no impact
12:29 PM
antlarr has quit
12:35 PM
yvanzo[m]
No change indeed.
12:35 PM
Also the number of documents is slightly decreasing since it started.
12:38 PM
However, it is expected that the number of documents to stagnate as it does if all of these index messages are indeed just updating release events.
12:38 PM
s/to/does/
12:44 PM
Restarting again with debug.
12:51 PM
Turned debug off again.
13:08 PM
zas: I confirm that it is working as expected are trustworthy as indexed entity IDs can be found in both SIR (debug) logs and Solr (update) logs.
13:08 PM
* as expected (and logs are trustworthy, * are trustworthy) as
13:09 PM
zas[m]
But then why do we have such big difference on graphs?
13:11 PM
akshaaatt[m] has quit
13:11 PM
zas[m] uploaded an image: (155KiB) < https://matrix.chatbrainz.org/_matrix/media/v3/download/matrix.org/IbqcTTtikYJlZEcVVCtTmNYn/image.png >
13:13 PM
Hmmm, actually we are back to normal somehow, after a high traffic period
13:13 PM
13:14 PM
zas[m] uploaded an image: (174KiB) < https://matrix.chatbrainz.org/_matrix/media/v3/download/matrix.org/xzQSvpPCUqszqEEHwRmLfDTZ/image.png >
13:15 PM
antlarr joined the channel
13:21 PM
minimal joined the channel
13:59 PM
zerodogg has quit
14:31 PM
mayhem[m] has quit
14:43 PM
zerodogg joined the channel
15:58 PM
monkey[m] has quit
18:15 PM
mukti has quit
18:29 PM
mukti joined the channel
19:44 PM
Jigen
:D not me, just web searching for "picard" expecting the regular wide web to know I mean MB picard
19:44 PM
(just "picard mb" worked!)
20:01 PM
uh... i have discovered that, if i press delete ex in the edit artist credit box, which delete the entire artist as expected since there was only one artist
20:01 PM
and then click ok or out or whatever. artist is impossible to fill in addrelease:
20:01 PM
20:02 PM
zas[m]
solr cloud is having issues
20:02 PM
Jigen
is that the reason for this bug?
20:03 PM
zas
dunno, maybe, did it just happen?
20:03 PM
Jigen
i mean..
20:03 PM
let me try it again
20:04 PM
zas[m]
yvanzo: I stopped sir-prod for now
20:05 PM
zas
solr cloud is very unstable right now
20:06 PM
Jigen
nope. ok here how to reproduce:*
20:06 PM
20:06 PM
select an artist "dire wolf" for the search. thne open the edit thing and red x the entire line. thne clcik done
20:06 PM
actually don't need ot select naything, justred x in the edit dialogue and thne done
20:06 PM
no way toreselect any artist
20:07 PM
zas
reducing rate limit on solr lb 4 in order to let the cluster stabilize
20:08 PM
Jigen
same thing in the tracklist tab too actually, if i do he exact same thing in a line thne the entire artsit line becomes unusable
20:09 PM
i don't really see how this is relevant to the search server being flakey tbh. :/
20:13 PM
Sophist-UK
Getting 500 and 504 status on ajax calls when editing a release.
20:13 PM
Jigen
huu
20:14 PM
search worksfine here
20:18 PM
zas[m]
Sophist-UK (IRC): yes, likely related to the search cluster issue, it is slowly recovering
20:18 PM
yvanzo: I think the failure is due to 2 factors: peak time + sir submitting a lot of entries
20:19 PM
Jigen
um.....
20:19 PM
what
20:19 PM
20:20 PM
was my entire account nuked???
20:21 PM
why didn't you tell me???
20:21 PM
I mean I can just reimport but
20:21 PM
Sending page 8 of 36907 to ListenBrainz
20:22 PM
😂
20:22 PM
please tell me next time :D
20:23 PM
also why did it happen? was it because of that weird user bug?
20:28 PM
Sophist-UK
@zas I managed to get around it by doing a non-indexed web search for the artist and pasting the MBID into the editor field.
20:29 PM
Jigen
ok wait all my loved tracks are also gone and also my old playlists and things.. guuyyysss
20:29 PM
>:|
20:29 PM
man
20:30 PM
20:30 PM
20:30 PM
BrainzBot
I will remember "man" for you Jigen.
20:32 PM
zas[m]
solr-4 & solr-8 are still on high load, solr-4 is reading disk at very high rate (1.7Gb/s), current search cluster rate limit is 160 req/s (normal = 260), sir is stopped
20:52 PM
v6lur has quit
21:14 PM
Jigen
21:14 PM
21:14 PM
BrainzBot
I will remember "srsly?" for you Jigen.
21:15 PM
Jigen
oos
21:15 PM
21:15 PM
BrainzBot
I'm sorry, I don't remember "srsly?=https://usercontent.irccloud-cdn.com/file/12B5Kakx/man.jpg", are you sure I should know about it?
21:15 PM
Jigen
!forget srsly?
21:15 PM
BrainzBot
What was "srsly?" all about?
21:15 PM
Jigen
21:15 PM
BrainzBot
I will remember "srsly?" for you Jigen.
21:16 PM
Jigen
thanks BrainzBot, atleast you respond to me
21:16 PM
😂
21:25 PM
LupinIII
!recall srsly?
21:25 PM
BrainzBot
21:30 PM
Jigen
21:30 PM
BrainzBot
I will remember "spit take" for you Jigen.
21:31 PM
Jigen
21:31 PM
BrainzBot
I will remember "haha" for you Jigen.
21:32 PM
Jigen
!remember ???=https://usercontent.irccloud-cdn.com/file/saPALvOR/confused%20lup.jpg
21:32 PM
BrainzBot
I will remember "???" for you Jigen.
21:32 PM
Jigen
21:32 PM
BrainzBot
I will remember "blink" for you Jigen.
21:32 PM
Jigen
21:32 PM
BrainzBot
I will remember "mood" for you Jigen.
21:32 PM
Kladky has quit
21:33 PM
Jigen
21:33 PM
BrainzBot
I will remember "YES" for you Jigen.
21:33 PM
Jigen
21:33 PM
BrainzBot
I will remember "oh no, that's bad" for you Jigen.
21:34 PM
Jigen
21:34 PM
BrainzBot
I will remember "oh no, bad" for you Jigen.
21:34 PM
Jigen
21:34 PM
BrainzBot
I will remember "slacking" for you Jigen.
21:34 PM
Jigen
21:34 PM
BrainzBot
I will remember "run!" for you Jigen.
21:37 PM
Jigen
21:37 PM
BrainzBot
I will remember "superman" for you Jigen.
21:37 PM
Jigen
21:37 PM
BrainzBot
I will remember "you disappoint me" for you Jigen.
21:37 PM
Jigen
21:37 PM
BrainzBot
I will remember "disappoint" for you Jigen.
21:38 PM
Jigen
21:38 PM
BrainzBot
I will remember "elegant" for you Jigen.
21:38 PM
Jigen
21:38 PM
BrainzBot
I will remember "hmph" for you Jigen.
21:39 PM
Jigen