Hi yvanzo.... sorry I got disconnect from the chat and didn't see your message. I did some digging and I think the amqp trigger retries after a failure, so I think it's just a warning, but I had some other questions about the indexer because it does seem to be taking a long time for the live indexing. I followed the thread on github, I seem to be
2021-02-25 05606, 2021
roger_that
seeing a lot of "Index limit exceeded. Entity: recording, Total Rows: 5038988" but is it ok to bump that number up in my indexer.ini? What happens after the index limit is exceeded? It seems like activity in the indexer stops for about 10-15 minutes then processes more messages until it hits the index limit again. Just wanted to figure out what was
2021-02-25 05607, 2021
roger_that
going on so that maybe I could debug it. Sorry for the long message! Thanks!
2021-02-25 05622, 2021
roger_that has quit
2021-02-25 05621, 2021
Lotheric has quit
2021-02-25 05622, 2021
adk0971 has quit
2021-02-25 05625, 2021
iliekcomputers
good morning!
2021-02-25 05633, 2021
MajorLurker joined the channel
2021-02-25 05636, 2021
MajorLurker has quit
2021-02-25 05643, 2021
Lotheric joined the channel
2021-02-25 05648, 2021
roger_that joined the channel
2021-02-25 05655, 2021
dseomn_ joined the channel
2021-02-25 05626, 2021
ZaphodBeeblebrox joined the channel
2021-02-25 05626, 2021
ZaphodBeeblebrox has quit
2021-02-25 05626, 2021
ZaphodBeeblebrox joined the channel
2021-02-25 05642, 2021
navap1 joined the channel
2021-02-25 05640, 2021
spuniun- joined the channel
2021-02-25 05621, 2021
Zhele_ joined the channel
2021-02-25 05635, 2021
ijc_ joined the channel
2021-02-25 05607, 2021
kloeri_ joined the channel
2021-02-25 05625, 2021
yvanzo
roger_that: It limits the number of rows queried from PostgreSQL at once. If exceeded, it aborts processing the current messages and requeues them as failed.
2021-02-25 05609, 2021
spuniun has quit
2021-02-25 05609, 2021
dseomn has quit
2021-02-25 05609, 2021
CatQuest has quit
2021-02-25 05609, 2021
Zhele has quit
2021-02-25 05609, 2021
ijc has quit
2021-02-25 05609, 2021
kloeri has quit
2021-02-25 05609, 2021
navap has quit
2021-02-25 05620, 2021
dseomn_ is now known as dseomn
2021-02-25 05616, 2021
yvanzo
It is ok to bump that number up as long as it fits allocated resources.
yvano: So if those messages fail, my search indexer can never be up to date?
2021-02-25 05655, 2021
roger_that20
I have 8 cpus, 32gb ram... is it not enough to run the live indexing?
2021-02-25 05635, 2021
roger_that20
And also... why do the failed messages cause such a big delay in processing the next messages? (Although the search server seems to be processing stuff - there's just nothing in the indexer logs for about 10-15 minutes after the failed query)
roger_that20: exactly, these queries require a lot of RAM and take a lot of time to be processed by PostgreSQL, so better reject them to allow for other queries to go through.
2021-02-25 05646, 2021
yvanzo
It does not use the full power of your CPUs/RAM because each component (indexer, solr, postgres) are limited for safety.
2021-02-25 05626, 2021
roger_that20
I see
2021-02-25 05634, 2021
yvanzo
The tricky part is to correctly allocate a balanced amount of resources to each component so it works smoothly.
2021-02-25 05655, 2021
roger_that20
What happens after they fail? Like I'm trying to understand what happens between the failed query and the next batch of messages that get processed
2021-02-25 05657, 2021
yvanzo
Rejected messages are not lost, they are in a failed queue that can be processed later on.
2021-02-25 05604, 2021
roger_that20
Oh ok
2021-02-25 05605, 2021
kloeri_ is now known as kloeri
2021-02-25 05622, 2021
roger_that20
And if I restart the containers.... the messages are saved?
2021-02-25 05631, 2021
roger_that20
Like if I wanted to change the indexer_limit
2021-02-25 05640, 2021
yvanzo
Yes, they are in the *_mqdata volume
2021-02-25 05650, 2021
adk0971 joined the channel
2021-02-25 05624, 2021
roger_that20
I noticed though, in that github thread... that JoshDi set his index limit to 1500000
2021-02-25 05632, 2021
roger_that20
So i wonder how those messages were getting processed
MBS-10004: inconsistency between JSON-LD and documentation
2021-02-25 05641, 2021
reosarevok
They kinda have a point that JSON-LD URIs probably should not be https?
2021-02-25 05644, 2021
yvanzo
makes sense
2021-02-25 05627, 2021
reosarevok
bitmap: do you remember if the https there was intentional?
2021-02-25 05637, 2021
yvanzo
This is the same reason we use http for VIAF URLs.
2021-02-25 05632, 2021
yvanzo
roger_that20: The only difference between 1.x and 2.x is about indexing recording’s first release date (in 2.x only).
2021-02-25 05625, 2021
yvanzo
1.x is a bit behind 2.x for a few other points but that it will be catching up with them.
2021-02-25 05645, 2021
reosarevok wonders about https://tickets.metabrainz.org/browse/MBS-9997 - seems that solved "itself", which usually means we fixed something in the meantime :p
ruaok: iliekcomputers: i was looking in sentry logs and saw that there are over 11m spotify errors related to same user over the past few months. any ideas?
2021-02-25 05630, 2021
ruaok
yeah, it been on my list to do. there are some simple things, some more complicated.
2021-02-25 05603, 2021
_lucifer
i see that spotipy hides the actual error message.
2021-02-25 05605, 2021
ruaok
I was trying to do weekly sentry cleanups for while, but got distracted by other things.
2021-02-25 05613, 2021
_lucifer
i can help :)
2021-02-25 05655, 2021
ruaok
the 400 errors are harder to clean up -- I think some users may have deleted their spotify account, but not turned off LB listen saving.
2021-02-25 05639, 2021
ruaok
I think once we get a 400 error enough times, we need to disable that account. we used to have some logic like that, but it ended up disabled 2/3 of the accounts. ;(
2021-02-25 05650, 2021
ruaok
please do help!3
2021-02-25 05616, 2021
_lucifer
makes sense
2021-02-25 05650, 2021
_lucifer
the 400 message might actually have some useful info.
2021-02-25 05636, 2021
_lucifer
i'll try to reproduce this error locally and see if we can come up with a better solution
2021-02-25 05628, 2021
ruaok
kewl.
2021-02-25 05624, 2021
ruaok
_lucifer: alastairp I'm running a "not nice" query on bono. if it starts smoking and burning, you know whom to blame.
