#metabrainz

[musicbrainz-server] 14mwiencek opened pull request #3337 (03master…mbs-13703): MBS-13703: Recognize tokens issued by the MeB OAuth Provider https://github.com/metabrainz/musicbrainz-serve...

lucifer: hi!

I'm working on the export functionality for Apple Music and have a question. For Spotify, our code currently retrieves the spotify_id from the Recording element, adds those tracks to the playlist, and then uses Spotify's lookup to find any remaining unplayable tracks. We don't store apple_music_ids for tracks.

here is the code https://github.com/metabrainz/troi-recommendati...

Should I consider adding apple_music_id to the Recording element and the model in LB, or should we rely solely on the Labs API lookup to search for tracks? The issue with the Labs API is that it sometimes returns multiple track IDs, and we can't always be certain which one is correct

[@kubrimskii:matrix.org](https://matrix.to/#/@kubrimskii:matrix.org) We should start storing those IDs

Ues

*yes

okay

So I should update the troi and [this model ](https://github.com/metabrainz/listenbrainz-server/blob/2031a0ccc1b8cb717431b774e31774f7d99c9d8b/data/model/listen.py#L39C1-L39C30) in LB, right?

with enough tweaks the nmslib mega index finished building in 2 hours with a max string length of... 20. but before I could test it crashed. lol.

rimskii: you only need to update troi, that model is not useful.

I'll finish debugging with a partial dataset and then put up a model for us to test with, before building the lager one for performance testing.

lucifer: do you have an updated file with fuzzy lookup test cases?

mayhem: nope

ok, I'll cobble one together.

lucifer: ping

Pong

hey. I was having lunch with monkey talking about the fuzzy matching stuff.

and now that I am very keenly aware of the fact that string lengths are rather quite important to the performance of a fuzzy index, I want to re-examine typesense.

because we might just limit the size of strings to X chars and then post filter them by fetching the rows from the DB.

and if the strings we store in typesense are much shorter, the whole thing might become much more performant.

and we could do an artist index and recording index to further make the strings shorter.

thoughts?

db fetch would likely be still slower but we can have two typesense indexes.

One for shorter match and one for falling back

yes, we will need to check the trade-offs.

but this path might give us the scalabilty we need without a lot of custom code.

artist and recording name matching separately makes sense yes.

but we also need to check match rates.

and we might be able to further partition the recording space. given that most typo correction algs assume that the first 2-3 characters are correct, then we can build smaller indexes, that might perform better.

I don't think we need to go that far.

At least for first try

possibly not... yet. but having a path for the future is also good.

yep. like minds...

I would prefer to avoid the assumption that the first two characters are right.

its one I would be ok with.

I think it would affect match rates.

Assuming typos are uniformly spread.

do we have a single test case in our tests that has a mistake in the first 2 characters? I doubt it.

Probably no

and if you check autocorrect on a mobile, if you get one of the first two characters wrong, you'll never get the right suggestion.

Yeah true that.

I think I will take a break from the nmslib approach for a bit. I would much prefer a solution based on solr or typesense.

Yes sounds good to me.

Let's try to make it work with typesense

And we can look into solr later if things don't work out

I'll try tweaking the typesense index and see how it goes.

ok, sounds good.

Sounds good thanks!

Introducing: Metrics for the mail service!

Jade: that's awesome, what stats are you collecting currently?

Currently, I've just set up counters for emails_requested, emails_sent and healthchecks

Not sure which metrics would be the most useful

that's already very useful, I could see us adding some metrics for specific types of internal errors too

Yep!

I'm gonna grab something to eat, be back in a bit

https://github.com/JadedBlueEyes/mb-mail-servic... is looking very nice, cool that you were able to submit some more fixes upstream 🙂

And it got merged!

Really quicky, too!

> TODO: Find a way to avoid wrapping inline links and find a better way of postfix URLs - another PR?

do the new templates use inline links anywhere? and what sort of improvements were you thinking of for the postfix URLs?

* > TODO: Find a way to avoid wrapping inline links and find a better way of postfix URLs - another PR?

do the new templates use inline links anywhere? and what sort of improvements were you thinking of for the postfix URLs?

The text emails currently look like this

With borders enabled:... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/...>)

right, it's a little bit awkward

re postfix URLs, I was thinking markdown-style inline links?

But I couldn't find an easy way of disabling wrapping for that, so the kinks got broken

Ideally, I'd like to be able to choose to render some of the borders, but you can see in text.rs - that's basically all the control I've got without patching the library more

It's just one global switch for borders or not

<Jade[m]> "Hello Jade,..." <- > <@jadedblueeyes:matrix.org> ```... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/...>)

But yeah, awkward

use custom css in templates?

you need to test it out on multiple devices and browsers but if there's a common structure and finite templates then its doable.

we do that for Year in Music emails.

I'd leave the borders off if it's all-or-nothing, yeah. but it looks pretty good overall, and most people will be viewing the HTML emails anyway, I imagine

bitmap[m]: Yep!

lucifer[m]: CSS is disabled by default, but when enabled it's only able to hide `display: none` elements

ah okay

Jade: looks like you were able to get Weblate syncing fine?

Nope

I can manually pull translations, but weblate auto-locks the project when it errors out

I think [@outsidecontext:matrix.org](https://matrix.to/#/@outsidecontext:matrix.org) was going to look at it?

do you know how credentials are specified?

I can unlock it (and even disable "Lock on error") but that doesn't really solve the error

bitmap[m]: I can't find the settings in my interface, and I've looked 😅

have you set up a webhook on github?

bitmap[m]: Yep, and Weblate is happy with it

lucifer: a typesense based artist index takes 15ms - 25ms to search. 1-2ms using nmslib.

hrm. we also have some configuration to allow weblate-metabrainz to push (and force push) to the translations branch, not sure if that's relevant

I've tried a few different setups - the one I've left it on is the default GitHub PR workflow

I'd tried one that should have pushed directly to main as well

But that didn't work either

It might be helpful to see the setup of one of the other projects?

mayhem: can you also test artist name and recording name lookup simultaneously?

Jade[m]: I'm not even sure where to look TBH, I think yvanzo and outsidecontext will have to help 🙏

our internal syswiki lists the credentials we use, but not much about the setup at the moment

Weblate is a bit confusing tbh

But this should be the only issue, given I *can* pull manually

So, summary since last Thursday:... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/...>)

very productive week! with a working Docker image, we can look into setting up your project on metabrainz servers soon and point test.musicbrainz.org to it eventually

will you be looking into patching musicbrainz-server to communicate with the mb-mail-service soon?

I guess to start with we won't be reading subscriptions from the mail service

Yeah, definitely

I really don't know any perl though, so my code is likely to be shocking

bitmap[m]: Yep. I haven't done any work on that yet

And the subscriptions template isn't fully functional yet

haha. MB's Perl isn't so bad, and I can help guide you through it

Jade[m]: When I start on that it'll be my turn to crib from [@yellowhatpro:matrix.org](https://matrix.to/#/@yellowhatpro:matrix.org) 😁

bitmap[m]: Thank you 😊

on wolf we have an MB docker mirror running -- if it would be helpful to you, we could give you ssh access there to test your mail service setup (unless you already have a full working setup locally)

bitmap[m]: I have a Dev setup with the testing database as described in the musicbrainz-docker repo

Which should be good enough for local devel

Not sure how accurately that represents production though

<Jade[m]> "When I start on that it'll be my..." <- for the amount of times I have taken reference from your configs, I highly doubt you will need my help 🤣🤣

But even if it's 1 percent of the case it's helpful then I am really heppyy

cool, much easier to use that then

yellowhatpro[m]: > <@yellowhatpro:matrix.org> for the amount of times I have taken reference from your configs, I highly doubt you will need my help 🤣🤣

> But even if it's 1 percent of the case it's helpful then I am really heppyy

I technically haven't written a single project which interacts with a SQL database! I'll at least be looking at your sqlx code 😀

ohh nicee lesgooo 🚀 😁😁

thanks to rustynova it smells less now, but yeah worth a watch ig

If you have any questions about anything or want any help, don't hesitate to ask me haha

I'm a follower of the Not Rocket Science Rule as much as possible, so my CI setup is comprehensive 😆

trust me i won't , it has only helped me develop instincts, and since jan this year, I feel a lot more confident coding in rust now

<Jade[m]> "I really don't know any perl..." <- one small hint for lib/MusicBrainz/Server/Email.pm: `$self->c->lwp` should give you access to an [LWP instance](https://metacpan.org/pod/LWP) that can be used to send requests

bitmap[m]: Thanks!

I've also been working on my writing a bit more