sir just logs several "Error encountered while processing messages: Post to Solr failed. Requeueing all pending messages for retry." around that time
zas
aerozol[m]: you can post about us having problems with search cluster, for now it works, but at 20% of capacity, and it may starts again to fail anytime, as the exact source of the issue isn't found
bitmap: I think that was caused by the misbehavior of the cluster, sir got errors from it
and errors started with crazy disk I/O on solr 5 then 4 (it never happened on 6,7,8...)
bitmap
ah weird
aerozol[m]
“We are aware of problems with our #MusicBrainz search cluster. Your search experience may vary, until we find and fix the source of the issue -a”
bitmap
I'm not sure why it just put 9000 messages back onto the retry queue either
aerozol[m]
^ look ok?
zas
basically a cascading effect: 4 & 5 got very slow, failed to health check, traffic sent to 6-8, slowdown, health check failures, etc....
sir errors I just posted are very weird, it was processing messages normally, and nothing noticeable happened. I was tail -f the docker log.
aerozol[m]
Posted to socials re. search clusters
bitmap
yeah I saw the KeyboardInterrupt error too, dunno what that's about but I didn't touch it
if the queues can't catch up overnight I can just clear them (dump them to a file) so that new updates can be processed at least
thanks aerozol[m], that looked ok to me
I fear these 10,000 items in the retry queue are gonna be added back to the index queue all at once later
zas
or... does SIR container use consul-template? I mean this Keyboard Interrupt wasn't sent by a keyboard for sure, but maybe some incorrectly sent signal (SIGINT)
bitmap: what's the impact of clearing those?
bitmap
well, some entries will be outdated until those updates are processed, but they can be pushed back onto the queue later IIUC
zas
I guess you can clear them then
60r/s
I'm going by small steps and observe metrics, for now everything seems normal, and crazy disk I/O didn't re-appear
bitmap
maybe I can slowly shovel messages from the retry queue into the index queue since it seems to be processing things now
zas
yes
bitmap
sir does indeed use consul-template
zas
We had a failure on bookbrainz, I think both are related
zerodogg has quit
90r/s
the index queue should be ingested in ~20 mins
bitmap
i moved messages from the retry queue back onto the index queue, seems to be dropping now
aerozol[m]
kellnerd: I have someone asking about deleting a specific date/time range of LB listens, using elbisaur. I remember you walked me through that at some point, but I can’t find the chat… can you remember? I was even thinking about making it into a post or wiki page, at the time, oops
kellnerd: Found it!! Too many chat programs :D
zas
solr cloud rate limit bumped to 140r/s, index queue emptied
bitmap
thanks zas, fingers crossed
zas
solr 4 & 5 read more than others..
thejoker8814 joined the channel
thejoker88143 has quit
but now solr 5 dropped below the others... weird
180 r/s
solr 5 now reads 10x more than others, why?
25-40MB/s for all but solr5 is around 650MB/s now (and it keeps increasing)
Ok, I'll keep the rate limit on 180 for now, I fear increasing it further will crash the whole cluster again
Fortunately the new cluster is not too far now (thanks to atj)
bitmap
thanks for resolving this zas
I can keep an eye on the queues at least
zas
well, not like I really found a solution, we have yet to understand the problem... which is: why one node over 5 identical nodes is reading so heavily from disk, exhausting its resources and eventually causing a cascading effect leading to a full cluster crash...
I restarted the cluster, in different start order, and whatever I did, that's this node that has issues (first)
those are VMs, identical setup, RR traffic
this difference in behavior is very weird. So solr is acting differently on this node, not sure why, and logs are useless to me(far too much stuff is logged imho)
solr5 is now reading ~850MB/s while others are still around ~30-50
aerozol[m], monkey[m]: Hi! I have a task assigned to me "Add entity search to LB pages.". Do we have any mockups on how the page would look like? Or where should we place the search bar?
akshaaatt
Hi pranav! I think you should start working on your GSoC project and pick up BP revamp later :)
pranav[m] joined the channel
pranav[m]
Alright akshaaatt (IRC) so I will be making prs to dev only right?
Coz one commit of revamp has been pushed to dev
akshaaatt
Yus
pranav[m]
So we can’t merge it to main without completing that up first
akshaaatt
Yus
Let’s do a release with gsoc stuff only
pranav[m]
Cool, that works.. super excited
akshaaatt
We can occasionally push to beta and get feedback
pranav[m]
Yeah, sure
akshaaatt
But a prod release only comes when everything is done
lusciouslover has quit
lusciouslover joined the channel
aerozol[m] has quit
aerozol[m] joined the channel
aerozol[m]
pranav: did you have a look at the mockups? All good? Also if you need the playlist and created for you pages - e.g. if you will have time to create all the web pages in-app.
No rush to look at it all, just let me know if you have enough from me for now
pranav[m]
Yes I looked at it in brief
But didn’t go over them in detail
If u could give me 1-2 hrs ill be happy to report
<akshaaatt> "But a prod release only comes..." <- Yup, agreed
aerozol[m]
ansh: monkey: For search, the ‘current’ way seems to be to let people type anything into a simple search bar, and then display all the results in a dropdown or in a separate page (dedicated page is probably more appropriate for the level of detail we may want to eventually support)
aerozol[m] uploaded an image: (52KiB) < https://matrix.moviebrainz.org/_matrix/media/v3/download/matrix.org/bFqzDnPZgmeadFsAcqclsVcV/image.png >
e.g. Discogs
aerozol[m] uploaded an image: (93KiB) < https://matrix.moviebrainz.org/_matrix/media/v3/download/matrix.org/PcamBUOtZNJYjpTJkCLlaUhi/image.png >
pranav: Yup, no rush from me, just make sure to leave me a couple of days to get round to making changes or more mockups, in case I’m busy.
atj
searching all entities would be nice, but it isn't really feasible with the customer search setup
we'd need to create a new core in Solr
aerozol[m]
atj: do you mean a search result with all results aggregated atj? I’m thinking we just display one result per tab, so we only run one search for whatever result tab we default to
Aggregate results would be a great stretch goal, but MB has struggled to get one for so long, I thought it might be hard… that said, search is important enough that we might want to go for it?
atj
yeah I guess so, but Discogs shows you search counts in the tabs
aerozol[m]
Oops, I think my terminology is off. “unified search”?
atj
and defaults to "all"
search is a rare example where Discogs does something really well!
aerozol[m]
I think that’s the ideal behaviour. But if we can’t do it then we can’t do it
If Discogs had disambiguations then their search would crush ours 100%
atj
it could be done, just would involve a reasonable amount of effort
now that Solr is deployable from Ansible, it will make it pretty trivial to set up test environments etc.
monkey: ansh: It was on my mind so I made an example mockup of how I imagine the entity search looking. Partly also because this seems the easiest way to do it. I’ve tried not to cram extras in (no unified search, no result numbers on tabs), but couldn’t resist having some fun example columns in the results (number of releases, number of listens):
can't really interact with it unless i get an account. sigh
but i can see it atleast
nowi jsut needto un-learn like 15 yearsofpressing the | for ' instead
hm...
Ansh[m] joined the channel
Ansh[m]
aerozol: This looks pretty good. Since we don't have universal search right now, as you mentioned, we can show one entity per category, and add a link to the search page.
How about also showing the release cards for the albums or songs?
MonkeyPython
i think that, much like picard made tagging more *release* fokused (as opposed to track-fokused)
lb ____really___ needs to be mademore release fokused
aerozol[m]
ansh: That’s a really good point, using release cards. Then you can ‘play’ the search results. I’ll update the mockup
I don’t know what you mean re. add a link to the search page though
Ansh[m]
What I mean is that we'll use the searchbar on the left sidebar to search. Then, we can show a dropdown with a few search results.
There, we'll add a link "View all results" to this search page.
aerozol[m]
I thought about that too, but tbh if TikTok can get away with just taking you to a results page, then we can get away with it too. imo if a dynamic drop down adds a ton of work (esp querying all the different entities at once, based off what atj has said) leave it for last. If you are keen though, I think it is nice to have
Kladky joined the channel
Ansh[m]
Understood. I'll work on the search page first, and then will try figuring out the best way to implement the search without adding a lot of performance overhead
zerodogg has quit
aerozol[m]
ansh: I’ve updated my drafts. But maybe wait and see what monkey thinks. There might be something clever we haven’t thought of
aerozol[m] uploaded an image: (161KiB) < https://matrix.moviebrainz.org/_matrix/media/v3/download/matrix.org/CJylWMgEuvVGguQnvsUpSrIn/Search%20_%20Albums.png >
aerozol[m] uploaded an image: (60KiB) < https://matrix.moviebrainz.org/_matrix/media/v3/download/matrix.org/GqwpmMhzJtAVlQVLrZHhDovI/Search%20_%20Playlists.png >
Ansh[m]
Sure! Thanks
rgggn joined the channel
zerodogg joined the channel
lucifer
rimskii: hi! let me know when you'll be around today or tomorrow for a brief discussion on getting started on the GSoC project?