13:20 PM
reosarevok
I did not!
2017-12-20 35424, 2017
13:20 PM
reosarevok
Here at least people with fire stoves can just burn that stuff, but there...
2017-12-20 35438, 2017
13:20 PM
ruaok
they came up with something creative. let me see if I can find it.
2017-12-20 35446, 2017
13:21 PM
ruaok
2017-12-20 35452, 2017
13:23 PM
ruaok doesn't share the disgusting video with everyone else
2017-12-20 35413, 2017
13:28 PM
iliekcomputers
alastairp: I've made changes to gh:AB#233 and gh:AB#222. I think they should be ready for merge now, please take a look when you get the time :)
2017-12-20 35435, 2017
13:28 PM
iliekcomputers
hmm, seems like BrainzBot doesn't link AB PRs :?
2017-12-20 35440, 2017
13:28 PM
iliekcomputers
gh:LB#201
2017-12-20 35441, 2017
13:28 PM
BrainzBot
2017-12-20 35447, 2017
13:28 PM
iliekcomputers
gh:AB#222
2017-12-20 35409, 2017
13:29 PM
iliekcomputers
weird
2017-12-20 35419, 2017
13:30 PM
ruaok
gah, I haven't looked at AB in such a long time. :(
2017-12-20 35448, 2017
13:30 PM
samj1912
iliekcomputers: you need to add the alias
2017-12-20 35406, 2017
13:32 PM
samj1912
BrainzBot: gh:AB=acousticbrainz-server
2017-12-20 35406, 2017
13:32 PM
BrainzBot
Successfully stored the repo acousticbrainz-server as AB for Github lookups
2017-12-20 35414, 2017
13:32 PM
samj1912
gh:AB#222
2017-12-20 35415, 2017
13:32 PM
BrainzBot
2017-12-20 35418, 2017
13:32 PM
iliekcomputers
ruaok: we've made some progress with AB in the past few days, going through the PR backlog :)
2017-12-20 35421, 2017
13:32 PM
iliekcomputers
samj1912: thanks!
2017-12-20 35440, 2017
13:32 PM
samj1912
o/
2017-12-20 35401, 2017
13:43 PM
drsaunders has quit
2017-12-20 35433, 2017
14:02 PM
drsaunders joined the channel
2017-12-20 35446, 2017
14:06 PM
naiveai joined the channel
2017-12-20 35457, 2017
14:12 PM
sampsyo has quit
2017-12-20 35400, 2017
14:14 PM
jesus2099 joined the channel
2017-12-20 35430, 2017
14:15 PM
sampsyo joined the channel
2017-12-20 35431, 2017
14:21 PM
samj1912
2017-12-20 35439, 2017
14:21 PM
samj1912
It is finally working (fingers crossed)
2017-12-20 35448, 2017
14:21 PM
samj1912
and no its not ending up in the retry queue :P
2017-12-20 35413, 2017
14:22 PM
samj1912
ruaok: ^
2017-12-20 35425, 2017
14:22 PM
samj1912
and I also figured out more on how to solve our throughput problem
2017-12-20 35402, 2017
14:23 PM
ruaok
nice! how is that?
2017-12-20 35415, 2017
14:23 PM
samj1912
currently sir is processing 1 days worth of backlog in about 8 minutes
2017-12-20 35432, 2017
14:23 PM
samj1912
anyways, I was going through old sir PRs, looking at mineo and gentlecat's comments
2017-12-20 35443, 2017
14:23 PM
samj1912
apparently sir was made to be run on multiple nodes
2017-12-20 35457, 2017
14:23 PM
samj1912
and it plays nicely with rmq
2017-12-20 35418, 2017
14:24 PM
samj1912
2017-12-20 35418, 2017
14:24 PM
samj1912
current rmq utilization is about 1%
2017-12-20 35438, 2017
14:24 PM
samj1912
it can handle multiple consumers
2017-12-20 35456, 2017
14:24 PM
ruaok
awesome.
2017-12-20 35405, 2017
14:25 PM
samj1912
so if we have sir running on 2 or 3 nodes, it can easily work through anomalies
2017-12-20 35408, 2017
14:25 PM
ruaok
so in theory we can run one consumer normally and monitor it.
2017-12-20 35412, 2017
14:25 PM
samj1912
yup
2017-12-20 35415, 2017
14:25 PM
ruaok
should it fall behind, we start more consumers.
2017-12-20 35428, 2017
14:25 PM
samj1912
currently I put a limit on indexed entities
2017-12-20 35439, 2017
14:25 PM
samj1912
if they go past a certain number they end up on a different queue
2017-12-20 35451, 2017
14:25 PM
samj1912
this way we can have 1 sir instance just churning through large reindexes
2017-12-20 35410, 2017
14:26 PM
samj1912
2017-12-20 35428, 2017
14:26 PM
samj1912
processing 30k entities, it encountered 33 cases where number was greater than 100k (configurable)
2017-12-20 35430, 2017
14:26 PM
alastairp
iliekcomputers: gah, I'm in the middle of that
2017-12-20 35434, 2017
14:26 PM
alastairp
OK, I'll merge your changes
2017-12-20 35443, 2017
14:26 PM
samj1912
so we can have one sir instance doing timely updates
2017-12-20 35457, 2017
14:26 PM
samj1912
and one instance taking care of things that will be blocking the normal one
2017-12-20 35407, 2017
14:27 PM
samj1912
does that sound like a good plan?
2017-12-20 35427, 2017
14:27 PM
iliekcomputers
sorry! they were small changes, so I thought I might as well...
2017-12-20 35429, 2017
14:27 PM
samj1912
2017-12-20 35437, 2017
14:27 PM
alastairp
yeah, no worries
2017-12-20 35443, 2017
14:27 PM
samj1912
2017-12-20 35455, 2017
14:27 PM
alastairp
I'm just making some other changes to 222 too, adding in "automatic submit" if the item is valid
2017-12-20 35459, 2017
14:27 PM
alastairp
which is pretty cool
2017-12-20 35408, 2017
14:29 PM
samj1912
so I started sir at 19:44, out of all the entities only 73 ended up in the retry queue (which was already filled upto 57 before I started the latest experiment)
2017-12-20 35404, 2017
14:30 PM
jesus2099 has left the channel
2017-12-20 35458, 2017
14:30 PM
ruaok
samj1912: it sounds very reasonable.
2017-12-20 35426, 2017
14:31 PM
ruaok
are the solr indexes being kept up to date now?
2017-12-20 35435, 2017
14:31 PM
iliekcomputers
alastairp: 👍🏽
2017-12-20 35436, 2017
14:31 PM
samj1912
and I am also logging everything with tracebacks so we can actually examine the queue for messages and fix bugs
2017-12-20 35444, 2017
14:31 PM
samj1912
I figured out 2 more bugs right now
2017-12-20 35457, 2017
14:31 PM
samj1912
ruaok: they can be now
2017-12-20 35424, 2017
14:32 PM
ruaok
ok, when you fix those two bugs, keep the indexes updating.
2017-12-20 35425, 2017
14:32 PM
samj1912
I will have to run a reindex first since I loaded the triggers 3 days back
2017-12-20 35430, 2017
14:32 PM
ruaok
watch for a day or so.
2017-12-20 35432, 2017
14:32 PM
samj1912
cool
2017-12-20 35442, 2017
14:32 PM
ruaok
then we should switch beta over to use the new search.
2017-12-20 35450, 2017
14:32 PM
ruaok
and then put out a massive call for testing.
2017-12-20 35454, 2017
14:32 PM
samj1912
there are some bugs on config reloads that I need to iron out
2017-12-20 35415, 2017
14:33 PM
samj1912
which I will do once I have indexing up and running perfectly
2017-12-20 35413, 2017
14:34 PM
ruaok
great.
2017-12-20 35426, 2017
14:34 PM
samj1912
also, I am still running at a modest config, so we can probably churn more out of sir
2017-12-20 35431, 2017
14:34 PM
ruaok
remember that I promised you a $500 bonus if this was up and running before the end of the year?
2017-12-20 35435, 2017
14:34 PM
UmkaDK joined the channel
2017-12-20 35442, 2017
14:34 PM
samj1912
yeah :P
2017-12-20 35445, 2017
14:34 PM
ruaok
I think you've already earned that.
2017-12-20 35453, 2017
14:34 PM
ruaok
so, make sure to add that to your jan invoice.
2017-12-20 35456, 2017
14:34 PM
UmkaDK_ has quit
2017-12-20 35404, 2017
14:35 PM
samj1912
cool :D! thanks!
2017-12-20 35408, 2017
14:35 PM
ruaok
I'm sooooo happy that you're rocking this. we've needed a hero like you!
2017-12-20 35413, 2017
14:35 PM
ruaok
:D
2017-12-20 35434, 2017
14:35 PM
ruaok
still, I worry about you and iliekcomputers. how many hours of sleep do you normally get?
2017-12-20 35435, 2017
14:35 PM
samj1912
this feels so great after the last week or 2 of struggling like shit with it :P
2017-12-20 35455, 2017
14:35 PM
ruaok
its been a long slog, not just the last two weeks. :)
2017-12-20 35422, 2017
14:36 PM
samj1912
2017-12-20 35433, 2017
14:36 PM
ruaok
\ø/
2017-12-20 35437, 2017
14:36 PM
samj1912
2017-12-20 35444, 2017
14:36 PM
samj1912
2017-12-20 35401, 2017
14:37 PM
alastairp
what does failed mean? temporary or permanent?
2017-12-20 35424, 2017
14:37 PM
samj1912
alastairp: I currently have those for messages which end up querying more than 100k rows
2017-12-20 35440, 2017
14:37 PM
samj1912
I had a bug earlier in sql generation which is why it hit 33
2017-12-20 35452, 2017
14:37 PM
alastairp
oh right, so it's not exactly "was unable to index entity x"
2017-12-20 35455, 2017
14:37 PM
samj1912
but I solved it, and in the latest churning through 50k messages not 1 hit 100k rows
2017-12-20 35401, 2017
14:38 PM
samj1912
yup
2017-12-20 35406, 2017
14:38 PM
alastairp
nice
2017-12-20 35420, 2017
14:38 PM
samj1912
retry is caused on exceptions
2017-12-20 35451, 2017
14:38 PM
samj1912
50k messages caused 21 messages to end up on the retry queue
2017-12-20 35403, 2017
14:39 PM
samj1912
all of them correspond to the same bug
2017-12-20 35429, 2017
14:39 PM
samj1912
and it took 15 minutes to process 50k messages
2017-12-20 35413, 2017
14:41 PM
samj1912
I will push the latest changes to my PR now, this was a lot of test, debug, retry, repeat
2017-12-20 35415, 2017
14:45 PM
naiveai
!m samj1912
2017-12-20 35415, 2017
14:45 PM
BrainzBot
You're doing good work, samj1912!
2017-12-20 35428, 2017
14:45 PM
naiveai
I have no idea what you're doing, but the charts look very fancy.
2017-12-20 35433, 2017
14:45 PM
naiveai
:D
2017-12-20 35446, 2017
14:46 PM
reosarevok
zas: prod server issues?
2017-12-20 35409, 2017
14:48 PM
samj1912
nothing on alrets?
2017-12-20 35413, 2017
14:48 PM
samj1912
let me check
2017-12-20 35429, 2017
14:48 PM
zas
nothing i'm aware of, which kind of issues are you experiencing ?
2017-12-20 35447, 2017
14:48 PM
ruaok
slow web pages.
2017-12-20 35432, 2017
14:49 PM
samj1912
might be because of sir, but I kept the queries very limited
2017-12-20 35400, 2017
14:50 PM
samj1912
let me check
2017-12-20 35401, 2017
14:50 PM
ruaok
sir points to bowie or queen?
2017-12-20 35413, 2017
14:50 PM
zas
one backend is slow, on hip, i'll check
2017-12-20 35424, 2017
14:50 PM
samj1912
bowie, it was a disaster last time we had it point to queen, bitmap told me to use master
2017-12-20 35452, 2017
14:50 PM
samj1912
sir runs on queen though
2017-12-20 35457, 2017
14:50 PM
samj1912
we should move it elsewhere
2017-12-20 35405, 2017
14:51 PM
samj1912
it takes a lot of cpu %
2017-12-20 35422, 2017
14:51 PM
Slurpee joined the channel
2017-12-20 35422, 2017
14:51 PM
Slurpee has quit
2017-12-20 35422, 2017
14:51 PM
Slurpee joined the channel
2017-12-20 35428, 2017
14:51 PM
naiveai
haruute[m]: what're you working on rn?
2017-12-20 35450, 2017
14:51 PM
samj1912
2017-12-20 35400, 2017
14:52 PM
ruaok
cpu on postgres master didn't seem to care about your efforts.
2017-12-20 35418, 2017
14:52 PM
samj1912
sir doesn't seem to have affect pg-master much?
2017-12-20 35429, 2017
14:52 PM
ruaok
I don't see the impact, do you?
2017-12-20 35438, 2017
14:52 PM
samj1912
nope
2017-12-20 35440, 2017
14:52 PM
samj1912
not at all
2017-12-20 35456, 2017
14:52 PM
samj1912
phew
2017-12-20 35411, 2017
14:53 PM
haruute[m]
naiveai: for now, i'am claiming anison generation url task
2017-12-20 35417, 2017
14:53 PM
samj1912
I was keeping my fingers crossed :X
2017-12-20 35426, 2017
14:53 PM
naiveai
haruute[m]: aight. will do smth else then
2017-12-20 35439, 2017
14:54 PM
samj1912
ruaok: once sir is up and running, next we will need to take a look at solr
2017-12-20 35446, 2017
14:54 PM
Slurpee has quit
2017-12-20 35450, 2017
14:54 PM
samj1912
I did a stress test on solr a month back
2017-12-20 35428, 2017
14:55 PM
samj1912
a single node could handle about 30 req/s without much problems
2017-12-20 35437, 2017
14:55 PM
dragonzeron joined the channel
2017-12-20 35443, 2017
14:55 PM
samj1912
our current search servers are hit 100reqs/s over 2 nodes
2017-12-20 35409, 2017
14:56 PM
ruaok gets a little excited.
2017-12-20 35438, 2017
14:56 PM
ruaok
when we get this deployed this will be a project 10 years in the making, finally done.
2017-12-20 35444, 2017
14:56 PM
samj1912
but we will have to probably end up using solr cloud
2017-12-20 35448, 2017
14:56 PM
samj1912
yeah :D
2017-12-20 35402, 2017
14:57 PM
ruaok
why do you say that?
2017-12-20 35436, 2017
14:57 PM
samj1912
because solr shat itself as I started bombarding it with 45 req/s (while I was doing a reindex of recordings alongside)
2017-12-20 35405, 2017
14:58 PM
samj1912
and I was only querying the recording core
