#metabrainz

wtf

yvanzo: it reports a KeyboardInterrupt, how? https://www.irccloud.com/pastebin/ewDHrwpZ/

?? https://www.irccloud.com/pastebin/UwO3uecp/

Terminating... but continuing

cluster looks stable, sir running, increasing rate limit to 40/s

zas: ahh! Not a relaxing night for you. Should I make any posts to socials or is it likely to be sorted in the next hour or two?

the retry queue started growing around 17:30 for some reason https://stats.metabrainz.org/d/000000078/sir-an...

sir just logs several "Error encountered while processing messages: Post to Solr failed. Requeueing all pending messages for retry." around that time

aerozol[m]: you can post about us having problems with search cluster, for now it works, but at 20% of capacity, and it may starts again to fail anytime, as the exact source of the issue isn't found

bitmap: I think that was caused by the misbehavior of the cluster, sir got errors from it

and errors started with crazy disk I/O on solr 5 then 4 (it never happened on 6,7,8...)

ah weird

“We are aware of problems with our #MusicBrainz search cluster. Your search experience may vary, until we find and fix the source of the issue -a”

I'm not sure why it just put 9000 messages back onto the retry queue either

^ look ok?

basically a cascading effect: 4 & 5 got very slow, failed to health check, traffic sent to 6-8, slowdown, health check failures, etc....

sir errors I just posted are very weird, it was processing messages normally, and nothing noticeable happened. I was tail -f the docker log.

Posted to socials re. search clusters

yeah I saw the KeyboardInterrupt error too, dunno what that's about but I didn't touch it

if the queues can't catch up overnight I can just clear them (dump them to a file) so that new updates can be processed at least

thanks aerozol[m], that looked ok to me

I fear these 10,000 items in the retry queue are gonna be added back to the index queue all at once later

or... does SIR container use consul-template? I mean this Keyboard Interrupt wasn't sent by a keyboard for sure, but maybe some incorrectly sent signal (SIGINT)

bitmap: what's the impact of clearing those?

well, some entries will be outdated until those updates are processed, but they can be pushed back onto the queue later IIUC

I guess you can clear them then

60r/s

I'm going by small steps and observe metrics, for now everything seems normal, and crazy disk I/O didn't re-appear

maybe I can slowly shovel messages from the retry queue into the index queue since it seems to be processing things now

yes

sir does indeed use consul-template

We had a failure on bookbrainz, I think both are related

90r/s

the index queue should be ingested in ~20 mins

i moved messages from the retry queue back onto the index queue, seems to be dropping now

kellnerd: I have someone asking about deleting a specific date/time range of LB listens, using elbisaur. I remember you walked me through that at some point, but I can’t find the chat… can you remember? I was even thinking about making it into a post or wiki page, at the time, oops

kellnerd: Found it!! Too many chat programs :D

solr cloud rate limit bumped to 140r/s, index queue emptied

thanks zas, fingers crossed

solr 4 & 5 read more than others..

but now solr 5 dropped below the others... weird

180 r/s

solr 5 now reads 10x more than others, why?

25-40MB/s for all but solr5 is around 650MB/s now (and it keeps increasing)

It makes no sense.

Ok, I'll keep the rate limit on 180 for now, I fear increasing it further will crash the whole cluster again

Fortunately the new cluster is not too far now (thanks to atj)

thanks for resolving this zas

I can keep an eye on the queues at least

well, not like I really found a solution, we have yet to understand the problem... which is: why one node over 5 identical nodes is reading so heavily from disk, exhausting its resources and eventually causing a cascading effect leading to a full cluster crash...

I restarted the cluster, in different start order, and whatever I did, that's this node that has issues (first)

those are VMs, identical setup, RR traffic

this difference in behavior is very weird. So solr is acting differently on this node, not sure why, and logs are useless to me(far too much stuff is logged imho)

solr5 is now reading ~850MB/s while others are still around ~30-50

I'm off to bed

Thanks zas! Great work

kellnerd: I have saved your instructions for deleting a LB listen time range here, hope that is okay: https://wiki.musicbrainz.org/listenbrainz/elbisaur

Looks like Plex beat us to using MusicBrainz data to make ‘linktree’ style pages! Great minds think alike. I like the simple design they’ve used: https://x.com/volumetrique/status/1794673609002...

Monkey ^

aerozol[m], monkey[m]: Hi! I have a task assigned to me "Add entity search to LB pages.". Do we have any mockups on how the page would look like? Or where should we place the search bar?

Hi pranav! I think you should start working on your GSoC project and pick up BP revamp later :)

Alright akshaaatt (IRC) so I will be making prs to dev only right?

Coz one commit of revamp has been pushed to dev

Yus

So we can’t merge it to main without completing that up first

Yus

Let’s do a release with gsoc stuff only

Cool, that works.. super excited

We can occasionally push to beta and get feedback

Yeah, sure

But a prod release only comes when everything is done

pranav: did you have a look at the mockups? All good? Also if you need the playlist and created for you pages - e.g. if you will have time to create all the web pages in-app.

No rush to look at it all, just let me know if you have enough from me for now

Yes I looked at it in brief

But didn’t go over them in detail

If u could give me 1-2 hrs ill be happy to report

<akshaaatt> "But a prod release only comes..." <- Yup, agreed

ansh: monkey: For search, the ‘current’ way seems to be to let people type anything into a simple search bar, and then display all the results in a dropdown or in a separate page (dedicated page is probably more appropriate for the level of detail we may want to eventually support)

e.g. Discogs

TikTok

If that basic premise sounds okay to you both (ansh and monkey) I can work on appropriate mockups of our search results page https://listenbrainz.org/search/?search_term=test

pranav: Yup, no rush from me, just make sure to leave me a couple of days to get round to making changes or more mockups, in case I’m busy.

searching all entities would be nice, but it isn't really feasible with the customer search setup

we'd need to create a new core in Solr

atj: do you mean a search result with all results aggregated atj? I’m thinking we just display one result per tab, so we only run one search for whatever result tab we default to

Aggregate results would be a great stretch goal, but MB has struggled to get one for so long, I thought it might be hard… that said, search is important enough that we might want to go for it?

yeah I guess so, but Discogs shows you search counts in the tabs

Oops, I think my terminology is off. “unified search”?

and defaults to "all"

search is a rare example where Discogs does something really well!

I think that’s the ideal behaviour. But if we can’t do it then we can’t do it

If Discogs had disambiguations then their search would crush ours 100%

it could be done, just would involve a reasonable amount of effort

now that Solr is deployable from Ansible, it will make it pretty trivial to set up test environments etc.

which reduces the cost of experimentation

Different topic: My LB feed isn’t loading - https://listenbrainz.org/feed/

Global works

!recall oh no.

monkey: ansh: It was on my mind so I made an example mockup of how I imagine the entity search looking. Partly also because this seems the easiest way to do it. I’ve tried not to cram extras in (no unified search, no result numbers on tabs), but couldn’t resist having some fun example columns in the results (number of releases, number of listens):

actually it's WORKING

!recall ohno

i can see FIGMA now

figma balls. idk

MonkeyPython: :P

can't really interact with it unless i get an account. sigh

but i can see it atleast

nowi jsut needto un-learn like 15 yearsofpressing the | for ' instead

hm...

aerozol: This looks pretty good. Since we don't have universal search right now, as you mentioned, we can show one entity per category, and add a link to the search page.

How about also showing the release cards for the albums or songs?

i think that, much like picard made tagging more *release* fokused (as opposed to track-fokused)

lb ____really___ needs to be mademore release fokused

ansh: That’s a really good point, using release cards. Then you can ‘play’ the search results. I’ll update the mockup

I don’t know what you mean re. add a link to the search page though

What I mean is that we'll use the searchbar on the left sidebar to search. Then, we can show a dropdown with a few search results.

There, we'll add a link "View all results" to this search page.

I thought about that too, but tbh if TikTok can get away with just taking you to a results page, then we can get away with it too. imo if a dynamic drop down adds a ton of work (esp querying all the different entities at once, based off what atj has said) leave it for last. If you are keen though, I think it is nice to have

Understood. I'll work on the search page first, and then will try figuring out the best way to implement the search without adding a lot of performance overhead

ansh: I’ve updated my drafts. But maybe wait and see what monkey thinks. There might be something clever we haven’t thought of

Sure! Thanks

rimskii: hi! let me know when you'll be around today or tomorrow for a brief discussion on getting started on the GSoC project?