sir just logs several "Error encountered while processing messages: Post to Solr failed. Requeueing all pending messages for retry." around that time
2024-05-27 14832, 2024
zas
aerozol[m]: you can post about us having problems with search cluster, for now it works, but at 20% of capacity, and it may starts again to fail anytime, as the exact source of the issue isn't found
2024-05-27 14817, 2024
zas
bitmap: I think that was caused by the misbehavior of the cluster, sir got errors from it
2024-05-27 14821, 2024
zas
and errors started with crazy disk I/O on solr 5 then 4 (it never happened on 6,7,8...)
2024-05-27 14801, 2024
bitmap
ah weird
2024-05-27 14815, 2024
aerozol[m]
“We are aware of problems with our #MusicBrainz search cluster. Your search experience may vary, until we find and fix the source of the issue -a”
2024-05-27 14815, 2024
bitmap
I'm not sure why it just put 9000 messages back onto the retry queue either
2024-05-27 14819, 2024
aerozol[m]
^ look ok?
2024-05-27 14821, 2024
zas
basically a cascading effect: 4 & 5 got very slow, failed to health check, traffic sent to 6-8, slowdown, health check failures, etc....
2024-05-27 14838, 2024
zas
sir errors I just posted are very weird, it was processing messages normally, and nothing noticeable happened. I was tail -f the docker log.
2024-05-27 14818, 2024
aerozol[m]
Posted to socials re. search clusters
2024-05-27 14852, 2024
bitmap
yeah I saw the KeyboardInterrupt error too, dunno what that's about but I didn't touch it
2024-05-27 14818, 2024
bitmap
if the queues can't catch up overnight I can just clear them (dump them to a file) so that new updates can be processed at least
2024-05-27 14828, 2024
bitmap
thanks aerozol[m], that looked ok to me
2024-05-27 14846, 2024
bitmap
I fear these 10,000 items in the retry queue are gonna be added back to the index queue all at once later
2024-05-27 14859, 2024
zas
or... does SIR container use consul-template? I mean this Keyboard Interrupt wasn't sent by a keyboard for sure, but maybe some incorrectly sent signal (SIGINT)
2024-05-27 14822, 2024
zas
bitmap: what's the impact of clearing those?
2024-05-27 14818, 2024
bitmap
well, some entries will be outdated until those updates are processed, but they can be pushed back onto the queue later IIUC
2024-05-27 14829, 2024
zas
I guess you can clear them then
2024-05-27 14808, 2024
zas
60r/s
2024-05-27 14809, 2024
zas
I'm going by small steps and observe metrics, for now everything seems normal, and crazy disk I/O didn't re-appear
2024-05-27 14813, 2024
bitmap
maybe I can slowly shovel messages from the retry queue into the index queue since it seems to be processing things now
2024-05-27 14852, 2024
zas
yes
2024-05-27 14816, 2024
bitmap
sir does indeed use consul-template
2024-05-27 14853, 2024
zas
We had a failure on bookbrainz, I think both are related
2024-05-27 14852, 2024
zerodogg has quit
2024-05-27 14844, 2024
zas
90r/s
2024-05-27 14838, 2024
zas
the index queue should be ingested in ~20 mins
2024-05-27 14858, 2024
bitmap
i moved messages from the retry queue back onto the index queue, seems to be dropping now
2024-05-27 14810, 2024
aerozol[m]
kellnerd: I have someone asking about deleting a specific date/time range of LB listens, using elbisaur. I remember you walked me through that at some point, but I can’t find the chat… can you remember? I was even thinking about making it into a post or wiki page, at the time, oops
2024-05-27 14800, 2024
aerozol[m]
kellnerd: Found it!! Too many chat programs :D
2024-05-27 14825, 2024
zas
solr cloud rate limit bumped to 140r/s, index queue emptied
2024-05-27 14821, 2024
bitmap
thanks zas, fingers crossed
2024-05-27 14840, 2024
zas
solr 4 & 5 read more than others..
2024-05-27 14859, 2024
thejoker8814 joined the channel
2024-05-27 14805, 2024
thejoker88143 has quit
2024-05-27 14816, 2024
zas
but now solr 5 dropped below the others... weird
2024-05-27 14804, 2024
zas
180 r/s
2024-05-27 14813, 2024
zas
solr 5 now reads 10x more than others, why?
2024-05-27 14804, 2024
zas
25-40MB/s for all but solr5 is around 650MB/s now (and it keeps increasing)
Ok, I'll keep the rate limit on 180 for now, I fear increasing it further will crash the whole cluster again
2024-05-27 14836, 2024
zas
Fortunately the new cluster is not too far now (thanks to atj)
2024-05-27 14832, 2024
bitmap
thanks for resolving this zas
2024-05-27 14804, 2024
bitmap
I can keep an eye on the queues at least
2024-05-27 14819, 2024
zas
well, not like I really found a solution, we have yet to understand the problem... which is: why one node over 5 identical nodes is reading so heavily from disk, exhausting its resources and eventually causing a cascading effect leading to a full cluster crash...
2024-05-27 14852, 2024
zas
I restarted the cluster, in different start order, and whatever I did, that's this node that has issues (first)
2024-05-27 14819, 2024
zas
those are VMs, identical setup, RR traffic
2024-05-27 14836, 2024
zas
this difference in behavior is very weird. So solr is acting differently on this node, not sure why, and logs are useless to me(far too much stuff is logged imho)
2024-05-27 14837, 2024
zas
solr5 is now reading ~850MB/s while others are still around ~30-50
aerozol[m], monkey[m]: Hi! I have a task assigned to me "Add entity search to LB pages.". Do we have any mockups on how the page would look like? Or where should we place the search bar?
2024-05-27 14812, 2024
akshaaatt
Hi pranav! I think you should start working on your GSoC project and pick up BP revamp later :)
2024-05-27 14801, 2024
pranav[m] joined the channel
2024-05-27 14801, 2024
pranav[m]
Alright akshaaatt (IRC) so I will be making prs to dev only right?
2024-05-27 14810, 2024
pranav[m]
Coz one commit of revamp has been pushed to dev
2024-05-27 14816, 2024
akshaaatt
Yus
2024-05-27 14820, 2024
pranav[m]
So we can’t merge it to main without completing that up first
2024-05-27 14826, 2024
akshaaatt
Yus
2024-05-27 14839, 2024
akshaaatt
Let’s do a release with gsoc stuff only
2024-05-27 14854, 2024
pranav[m]
Cool, that works.. super excited
2024-05-27 14855, 2024
akshaaatt
We can occasionally push to beta and get feedback
2024-05-27 14805, 2024
pranav[m]
Yeah, sure
2024-05-27 14817, 2024
akshaaatt
But a prod release only comes when everything is done
2024-05-27 14812, 2024
lusciouslover has quit
2024-05-27 14854, 2024
lusciouslover joined the channel
2024-05-27 14830, 2024
aerozol[m] has quit
2024-05-27 14807, 2024
aerozol[m] joined the channel
2024-05-27 14807, 2024
aerozol[m]
pranav: did you have a look at the mockups? All good? Also if you need the playlist and created for you pages - e.g. if you will have time to create all the web pages in-app.
2024-05-27 14851, 2024
aerozol[m]
No rush to look at it all, just let me know if you have enough from me for now
2024-05-27 14855, 2024
pranav[m]
Yes I looked at it in brief
2024-05-27 14802, 2024
pranav[m]
But didn’t go over them in detail
2024-05-27 14813, 2024
pranav[m]
If u could give me 1-2 hrs ill be happy to report
2024-05-27 14825, 2024
pranav[m]
<akshaaatt> "But a prod release only comes..." <- Yup, agreed
2024-05-27 14824, 2024
aerozol[m]
ansh: monkey: For search, the ‘current’ way seems to be to let people type anything into a simple search bar, and then display all the results in a dropdown or in a separate page (dedicated page is probably more appropriate for the level of detail we may want to eventually support)
2024-05-27 14837, 2024
aerozol[m] uploaded an image: (52KiB) < https://matrix.moviebrainz.org/_matrix/media/v3/download/matrix.org/bFqzDnPZgmeadFsAcqclsVcV/image.png >
2024-05-27 14840, 2024
aerozol[m]
e.g. Discogs
2024-05-27 14852, 2024
aerozol[m] uploaded an image: (93KiB) < https://matrix.moviebrainz.org/_matrix/media/v3/download/matrix.org/PcamBUOtZNJYjpTJkCLlaUhi/image.png >
pranav: Yup, no rush from me, just make sure to leave me a couple of days to get round to making changes or more mockups, in case I’m busy.
2024-05-27 14823, 2024
atj
searching all entities would be nice, but it isn't really feasible with the customer search setup
2024-05-27 14854, 2024
atj
we'd need to create a new core in Solr
2024-05-27 14802, 2024
aerozol[m]
atj: do you mean a search result with all results aggregated atj? I’m thinking we just display one result per tab, so we only run one search for whatever result tab we default to
2024-05-27 14859, 2024
aerozol[m]
Aggregate results would be a great stretch goal, but MB has struggled to get one for so long, I thought it might be hard… that said, search is important enough that we might want to go for it?
2024-05-27 14828, 2024
atj
yeah I guess so, but Discogs shows you search counts in the tabs
2024-05-27 14840, 2024
aerozol[m]
Oops, I think my terminology is off. “unified search”?
2024-05-27 14802, 2024
atj
and defaults to "all"
2024-05-27 14845, 2024
atj
search is a rare example where Discogs does something really well!
2024-05-27 14848, 2024
aerozol[m]
I think that’s the ideal behaviour. But if we can’t do it then we can’t do it
2024-05-27 14832, 2024
aerozol[m]
If Discogs had disambiguations then their search would crush ours 100%
2024-05-27 14824, 2024
atj
it could be done, just would involve a reasonable amount of effort
2024-05-27 14849, 2024
atj
now that Solr is deployable from Ansible, it will make it pretty trivial to set up test environments etc.
monkey: ansh: It was on my mind so I made an example mockup of how I imagine the entity search looking. Partly also because this seems the easiest way to do it. I’ve tried not to cram extras in (no unified search, no result numbers on tabs), but couldn’t resist having some fun example columns in the results (number of releases, number of listens):
can't really interact with it unless i get an account. sigh
2024-05-27 14815, 2024
MonkeyPython
but i can see it atleast
2024-05-27 14838, 2024
MonkeyPython
nowi jsut needto un-learn like 15 yearsofpressing the | for ' instead
2024-05-27 14855, 2024
MonkeyPython
hm...
2024-05-27 14838, 2024
Ansh[m] joined the channel
2024-05-27 14838, 2024
Ansh[m]
aerozol: This looks pretty good. Since we don't have universal search right now, as you mentioned, we can show one entity per category, and add a link to the search page.
2024-05-27 14837, 2024
Ansh[m]
How about also showing the release cards for the albums or songs?
2024-05-27 14852, 2024
MonkeyPython
i think that, much like picard made tagging more *release* fokused (as opposed to track-fokused)
2024-05-27 14852, 2024
MonkeyPython
lb ____really___ needs to be mademore release fokused
2024-05-27 14827, 2024
aerozol[m]
ansh: That’s a really good point, using release cards. Then you can ‘play’ the search results. I’ll update the mockup
2024-05-27 14803, 2024
aerozol[m]
I don’t know what you mean re. add a link to the search page though
2024-05-27 14856, 2024
Ansh[m]
What I mean is that we'll use the searchbar on the left sidebar to search. Then, we can show a dropdown with a few search results.
2024-05-27 14816, 2024
Ansh[m]
There, we'll add a link "View all results" to this search page.
2024-05-27 14828, 2024
aerozol[m]
I thought about that too, but tbh if TikTok can get away with just taking you to a results page, then we can get away with it too. imo if a dynamic drop down adds a ton of work (esp querying all the different entities at once, based off what atj has said) leave it for last. If you are keen though, I think it is nice to have
2024-05-27 14855, 2024
Kladky joined the channel
2024-05-27 14838, 2024
Ansh[m]
Understood. I'll work on the search page first, and then will try figuring out the best way to implement the search without adding a lot of performance overhead
2024-05-27 14801, 2024
zerodogg has quit
2024-05-27 14822, 2024
aerozol[m]
ansh: I’ve updated my drafts. But maybe wait and see what monkey thinks. There might be something clever we haven’t thought of
2024-05-27 14834, 2024
aerozol[m] uploaded an image: (161KiB) < https://matrix.moviebrainz.org/_matrix/media/v3/download/matrix.org/CJylWMgEuvVGguQnvsUpSrIn/Search%20_%20Albums.png >
2024-05-27 14845, 2024
aerozol[m] uploaded an image: (60KiB) < https://matrix.moviebrainz.org/_matrix/media/v3/download/matrix.org/GqwpmMhzJtAVlQVLrZHhDovI/Search%20_%20Playlists.png >
2024-05-27 14801, 2024
Ansh[m]
Sure! Thanks
2024-05-27 14811, 2024
rgggn joined the channel
2024-05-27 14846, 2024
zerodogg joined the channel
2024-05-27 14813, 2024
lucifer
rimskii: hi! let me know when you'll be around today or tomorrow for a brief discussion on getting started on the GSoC project?
