#metabrainz

[listenbrainz-server] 14granth23 opened pull request #3253 (03master…LB-1776--Artist-origin-map-tidy-up): LB-1776: Artist origin map tidy up https://github.com/metabrainz/listenbrainz-serv...

Hi, is there any method to fetch the total number of artists grouped by a country?

* Hi, is there any function defined to fetch the total number of artists grouped by a country?

holycow23[m]: I personally don't know of one, but I suspect that if it does exist in the spark world, lucifer would know.

<lucifer[m]> "monkey: sure i can create a..." <- Great, some help would be very helpful, thank you :)

I got the SSE endpoint working and sending updates. The part I need help with is somehow consuming incoming listening-now listens for a specific person, and how to set that up as a generator function for the SSE endpoint.

Let me upload the code I have for now...

<BrainzGit> "[listenbrainz-server] granth23..." <- ansh: could you please review this PR

Here we are: https://github.com/metabrainz/listenbrainz-serv...

Currently sleeps 5 seconds before checking the playing-now again, for testing only.

I struggled trying to figure out how to consume the RMQ channel, didn't get it to work. I figured this would be a good time to pass the baton and learn from your implementation

holycow23: its not a function but yes that data is available in a table you can join to. what's your use case?

monkey: i saw your latest commit in the widgets branch but i have a feeling it might not work, because as long as the sse request is running uwsgi won't be able to take up another request for processing. so 10-20 widgets rendering would block the server.

also one more thing, sse has a limit of 6 connections per site per browser.

Oh...

Well, that's unfortunate, then. Do you think I should go back to a timed polling system on the client?

can you do it without real time updates for now?

or can you use websockets?

Yeah, I suppose we're back to WS in that case. Poop 💩

lucifer[m]: this could be potentially solved by migrating to async flask but that is a big task.

monkey[m]: does WS has a downside for your use case?

I'll have to see what the support is like with HTMX

But overall, I suspect heavier library, less god interaction with HTMX

Well, never mind that last one: https://v1.htmx.org/extensions/web-sockets/

htmx seems to support websockets but not socket.io

we are using socket.io which is a wrapper around websockets and provides some additional capabilities on top of it. but yeah try using it and if you face issues let me know and i can try to help fix it.

Cool, I'll have a look. Although I think I'll have the same questions: How does one connect to the WS feed for a specific user?

that would be same as the listens page.

or the metadata viewer page: https://github.com/metabrainz/listenbrainz-serv...

on the connection start, you emit a message with the username to add a user's feed to your connection. then you monitor it using the playing now events.

the WS feeds already exist so you only need frontend side to connect properly. i am not sure if we can connect properly without the socket.io extension but try it and let me know. otherwise i'll to find a work around.

lucifer: I think this is a different mechanism than what I am trying to achieve. I'm trying to do all the templating and rendering on the server, and send rendered HTML fragments to the client.

So I'd like to avoid the type of mechanism we have for the metadataviewer, where we render everything on the client.

monkey: yes i understand that but do you want constant updates or just show the playing now listen at the time of server rendering?

What I'm looking for is how to consume incoming listens for the user on the server-side, then I'll try to set up the fragment update mechanism using WS + HTMX>

SO not connecting to the same WS feed I suppose

The initial rendering of playing now listen is all working fine already, that's sorted.

I do want constant updates when a new listening-now comes in, if we can

right so at that point the code to consume the playing now listens would run on the client side no?

But I could try to have the client do regular polling if that's easier than sending updates from the server

since the initial render on the server side has already completed.

lucifer[m]: No, it would be consumed on the server, rendered using jinja templates, and the rendered HTML will be sent to the client

(i.e. no the initial playing-now WS message)

not*

i see and how would you send this rendered html to the client?

So currently that's done with SSE, but with all the new info I suppose we'll

ditch that.

ah okay got it.

so instead of the json listen payload, you want a html payload created from the json

So I guess setting up a new WS connection (at that point, doesn't need to be socket.io maybe?) and use HTMX to connect to an update endpoint which will send the HTML fragment

Yes

that shouldn't be hard, we can create a new WS room/endpoint

can you just test one thing, somehow log the json payload in htmx and see if its ws library works fine with our websockets setup.

if that works, i can setup a new endpoint for you to test. otherwise we'll first have to make the connection work.

Well, I still need help with setting up the json payload consumer

One sec

sure

OK, understood better now :)

I'll try to set up the HTMX code to hit our existing WS/socketi.io endpoint, see what happens

okay i don't it will work. htmx has custom event names that need to be defined on our server.

i'll try to set it up later today and try to make it work.

<lucifer[m]> "holycow23: its not a function..." <- I was working on a [ticket](https://tickets.metabrainz.org/projects/LB/issues/LB-1777?filter=allopenissues&orderby=created+DESC%2C+priority+DESC%2C+updated+DESC) so wanted the count in that

LB-1777: Artist origin map "play something from this country"

holycow23: i see, can you tell what's your plan is to implement it? i don't think you'll need spark for it.

also mayhem might want to work this through LB radio so it might be better to add country support to LB radio if missing and then you only need to create a prompt.

lucifer: I found one way that does work, it might just be enough...... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/...>)

monkey: polling the playing now endpoint is pretty efficient since it only queries redis.

lucifer[m]: I didn't intend to use spark, but if needed will have to make an API/function to retrieve the count for a country, which I will call everytime the stats are loaded

lucifer[m]: Country is already supported by radio and thus the LB-Radio side and prompt work is already done

lucifer[m]: I think this would probably be hitting the widget endpoint that renders the HTML fragment instead, but maybe it could be improved in the same way I used the socket.io event, i.e. hit the plyaing-now endpoint, parse the response, compare if it is the same trans_name, otherwise trigger an event to reload the fragment...

there is country support in LB Radio. play random recordings from a country.

monkey: yes that sounds good to me

i think that is simplest for now.

We're assuming it is better than loading the JS for socket.io and using a long-lived WS connection, then?

OK

Thank for the assist! No more need to consume the RMQ message on the server then, simpler to set up.

holycow23: i see, how do you intend to use the total artists count? anyway if you want to use it here then we'll have to create a LB metadata or labs api query for that.

Shame about SSE though, that seemed like a really nice solution

it is a nice solution yes, but i don't think it is meant for this use case.

Apparently HTTP/2 removes the issue with 6 concurrent connections (defaults to 100)

oh i didn't know that, that's great.

holycow23[m]: The mockup prepared by aerozol has a total artist count from the country

i see. so that will need a new labs api query.

Yes

you can take a look at for some examples on how to create that: https://github.com/metabrainz/listenbrainz-serv...

and this sql query as a reference to generate the artist count to be served by that api.

actually, you can just add this as a normal LB api endpoint.

no need to make a labs api one.

also, add caching for like an hour so on the api.

so forget the labs api link, take reference from the sql query above. to connect to MB db, you will need to use the mb_conn (one example here: https://github.com/metabrainz/listenbrainz-serv...)

if you face any issues, let me know and i'll help out

aerozol, bitmap: MBS-13984 seems like a reasonable complaint to me - not sure what would be the best way to deal with this, do you have a preference from the suggested options?

MBS-13984: Reduce space used on Release pages by associated Recordings' streaming/download links https://tickets.metabrainz.org/browse/MBS-13984

"not show them at all", "show them collapsed" both sound good to me, potentially. I don't think it's the end of the world the way it is now, but still gets a bit silly :)

<lucifer[m]> "so forget the labs api link..." <- cool will look into it

[musicbrainz-server] 14reosarevok opened pull request #3518 (03master…MBS-13977): MBS-13977: Ignore country codes in yesasia.com links https://github.com/metabrainz/musicbrainz-serve...

<holycow23[m]> "cool will look into it" <- lucifer: I don't have the data on local so, would I have to download the dump?

holycow23: yes you need to setup a local MB db (https://github.com/metabrainz/musicbrainz-docker/), you can use the test or full dump.

Okay so I need MB as well and then the dump

* the dump?

yes

okay

if you cannot setup the dump, i can create the endpoint for you

I will give it a try

How do I use the dump?

And I assume I need to go ahead with admin/configure with alt-db-only-mirror

s//`/, s//` to just use the db not the website/

<lucifer[m]> "if you cannot setup the dump..." <- And I assume I need to go ahead with `admin/configure with alt-db-only-mirror` to just use the db not the website

Yes

okay

<reosarevok[m]> ""not show them at all", "show..." <- my thought would be to hide them entirely (I've never even seen these before since I use the inline mode, and they aren't shown inline for some reason?) but maybe wait for more feedback on the ticket

I think we hide a bunch of things online we do show on the bottom

Inline even :D

hmm, well we do have an isIrrelevantLinkType util but it's called from both places. only the inline code skips urls though. there might be other differences I'm missing

Hmm, ok

Let's see what aerozol thinks (any other opinions welcome!)

Kinda want some advice on this. Critiquebrainz uses oauth for user api requests, meaning I need to securely store the app tokens. However, how can I do that in an user script? Should I just give up and let it in plain text?