13:20 PM
reosarevok
I did not!
13:20 PM
Here at least people with fire stoves can just burn that stuff, but there...
13:20 PM
ruaok
they came up with something creative. let me see if I can find it.
13:21 PM
13:23 PM
ruaok doesn't share the disgusting video with everyone else
13:28 PM
iliekcomputers
alastairp: I've made changes to gh:AB#233 and gh:AB#222. I think they should be ready for merge now, please take a look when you get the time :)
13:28 PM
hmm, seems like BrainzBot doesn't link AB PRs :?
13:28 PM
gh:LB#201
13:28 PM
BrainzBot
13:28 PM
iliekcomputers
gh:AB#222
13:29 PM
weird
13:30 PM
ruaok
gah, I haven't looked at AB in such a long time. :(
13:30 PM
samj1912
iliekcomputers: you need to add the alias
13:32 PM
BrainzBot: gh:AB=acousticbrainz-server
13:32 PM
BrainzBot
Successfully stored the repo acousticbrainz-server as AB for Github lookups
13:32 PM
samj1912
gh:AB#222
13:32 PM
BrainzBot
13:32 PM
iliekcomputers
ruaok: we've made some progress with AB in the past few days, going through the PR backlog :)
13:32 PM
samj1912: thanks!
13:32 PM
samj1912
o/
13:43 PM
drsaunders has quit
14:02 PM
drsaunders joined the channel
14:06 PM
naiveai joined the channel
14:12 PM
sampsyo has quit
14:14 PM
jesus2099 joined the channel
14:15 PM
sampsyo joined the channel
14:21 PM
14:21 PM
It is finally working (fingers crossed)
14:21 PM
and no its not ending up in the retry queue :P
14:22 PM
ruaok: ^
14:22 PM
and I also figured out more on how to solve our throughput problem
14:23 PM
ruaok
nice! how is that?
14:23 PM
samj1912
currently sir is processing 1 days worth of backlog in about 8 minutes
14:23 PM
anyways, I was going through old sir PRs, looking at mineo and gentlecat's comments
14:23 PM
apparently sir was made to be run on multiple nodes
14:23 PM
and it plays nicely with rmq
14:24 PM
14:24 PM
current rmq utilization is about 1%
14:24 PM
it can handle multiple consumers
14:24 PM
ruaok
awesome.
14:25 PM
samj1912
so if we have sir running on 2 or 3 nodes, it can easily work through anomalies
14:25 PM
ruaok
so in theory we can run one consumer normally and monitor it.
14:25 PM
samj1912
yup
14:25 PM
ruaok
should it fall behind, we start more consumers.
14:25 PM
samj1912
currently I put a limit on indexed entities
14:25 PM
if they go past a certain number they end up on a different queue
14:25 PM
this way we can have 1 sir instance just churning through large reindexes
14:26 PM
14:26 PM
processing 30k entities, it encountered 33 cases where number was greater than 100k (configurable)
14:26 PM
alastairp
iliekcomputers: gah, I'm in the middle of that
14:26 PM
OK, I'll merge your changes
14:26 PM
samj1912
so we can have one sir instance doing timely updates
14:26 PM
and one instance taking care of things that will be blocking the normal one
14:27 PM
does that sound like a good plan?
14:27 PM
iliekcomputers
sorry! they were small changes, so I thought I might as well...
14:27 PM
samj1912
14:27 PM
alastairp
yeah, no worries
14:27 PM
samj1912
14:27 PM
alastairp
I'm just making some other changes to 222 too, adding in "automatic submit" if the item is valid
14:27 PM
which is pretty cool
14:29 PM
samj1912
so I started sir at 19:44, out of all the entities only 73 ended up in the retry queue (which was already filled upto 57 before I started the latest experiment)
14:30 PM
jesus2099 has left the channel
14:30 PM
ruaok
samj1912: it sounds very reasonable.
14:31 PM
are the solr indexes being kept up to date now?
14:31 PM
iliekcomputers
alastairp: ππ½
14:31 PM
samj1912
and I am also logging everything with tracebacks so we can actually examine the queue for messages and fix bugs
14:31 PM
I figured out 2 more bugs right now
14:31 PM
ruaok: they can be now
14:32 PM
ruaok
ok, when you fix those two bugs, keep the indexes updating.
14:32 PM
samj1912
I will have to run a reindex first since I loaded the triggers 3 days back
14:32 PM
ruaok
watch for a day or so.
14:32 PM
samj1912
cool
14:32 PM
ruaok
then we should switch beta over to use the new search.
14:32 PM
and then put out a massive call for testing.
14:32 PM
samj1912
there are some bugs on config reloads that I need to iron out
14:33 PM
which I will do once I have indexing up and running perfectly
14:34 PM
ruaok
great.
14:34 PM
samj1912
also, I am still running at a modest config, so we can probably churn more out of sir
14:34 PM
ruaok
remember that I promised you a $500 bonus if this was up and running before the end of the year?
14:34 PM
UmkaDK joined the channel
14:34 PM
samj1912
yeah :P
14:34 PM
ruaok
I think you've already earned that.
14:34 PM
so, make sure to add that to your jan invoice.
14:34 PM
UmkaDK_ has quit
14:35 PM
samj1912
cool :D! thanks!
14:35 PM
ruaok
I'm sooooo happy that you're rocking this. we've needed a hero like you!
14:35 PM
:D
14:35 PM
still, I worry about you and iliekcomputers. how many hours of sleep do you normally get?
14:35 PM
samj1912
this feels so great after the last week or 2 of struggling like shit with it :P
14:35 PM
ruaok
its been a long slog, not just the last two weeks. :)
14:36 PM
samj1912
14:36 PM
ruaok
\ΓΈ/
14:36 PM
samj1912
14:36 PM
14:37 PM
alastairp
what does failed mean? temporary or permanent?
14:37 PM
samj1912
alastairp: I currently have those for messages which end up querying more than 100k rows
14:37 PM
I had a bug earlier in sql generation which is why it hit 33
14:37 PM
alastairp
oh right, so it's not exactly "was unable to index entity x"
14:37 PM
samj1912
but I solved it, and in the latest churning through 50k messages not 1 hit 100k rows
14:38 PM
yup
14:38 PM
alastairp
nice
14:38 PM
samj1912
retry is caused on exceptions
14:38 PM
50k messages caused 21 messages to end up on the retry queue
14:39 PM
all of them correspond to the same bug
14:39 PM
and it took 15 minutes to process 50k messages
14:41 PM
I will push the latest changes to my PR now, this was a lot of test, debug, retry, repeat
14:45 PM
naiveai
!m samj1912
14:45 PM
BrainzBot
You're doing good work, samj1912!
14:45 PM
naiveai
I have no idea what you're doing, but the charts look very fancy.
14:45 PM
:D
14:46 PM
reosarevok
zas: prod server issues?
14:48 PM
samj1912
nothing on alrets?
14:48 PM
let me check
14:48 PM
zas
nothing i'm aware of, which kind of issues are you experiencing ?
14:48 PM
ruaok
slow web pages.
14:49 PM
samj1912
might be because of sir, but I kept the queries very limited
14:50 PM
let me check
14:50 PM
ruaok
sir points to bowie or queen?
14:50 PM
zas
one backend is slow, on hip, i'll check
14:50 PM
samj1912
bowie, it was a disaster last time we had it point to queen, bitmap told me to use master
14:50 PM
sir runs on queen though
14:50 PM
we should move it elsewhere
14:51 PM
it takes a lot of cpu %
14:51 PM
Slurpee joined the channel
14:51 PM
Slurpee has quit
14:51 PM
Slurpee joined the channel
14:51 PM
naiveai
haruute[m]: what're you working on rn?
14:51 PM
samj1912
14:52 PM
ruaok
cpu on postgres master didn't seem to care about your efforts.
14:52 PM
samj1912
sir doesn't seem to have affect pg-master much?
14:52 PM
ruaok
I don't see the impact, do you?
14:52 PM
samj1912
nope
14:52 PM
not at all
14:52 PM
phew
14:53 PM
haruute[m]
naiveai: for now, i'am claiming anison generation url task
14:53 PM
samj1912
I was keeping my fingers crossed :X
14:53 PM
naiveai
haruute[m]: aight. will do smth else then
14:54 PM
samj1912
ruaok: once sir is up and running, next we will need to take a look at solr
14:54 PM
Slurpee has quit
14:54 PM
I did a stress test on solr a month back
14:55 PM
a single node could handle about 30 req/s without much problems
14:55 PM
dragonzeron joined the channel
14:55 PM
our current search servers are hit 100reqs/s over 2 nodes
14:56 PM
ruaok gets a little excited.
14:56 PM
ruaok
when we get this deployed this will be a project 10 years in the making, finally done.
14:56 PM
samj1912
but we will have to probably end up using solr cloud
14:56 PM
yeah :D
14:57 PM
ruaok
why do you say that?
14:57 PM
samj1912
because solr shat itself as I started bombarding it with 45 req/s (while I was doing a reindex of recordings alongside)
14:58 PM
and I was only querying the recording core