#metabrainz

atj: haproxy is good at load balancing, we use it too for different things, but, we use openresty for historical reasons, our gateways do much more than load balancing. That said, we are hitting the limits of this approach, hence current discussions

ruaok: generally KISS is the best option, so if you can get away with IP RR then that's ideal. However writing code to reliably detect application state and invoke IP failovers via a third party API sounds less than trivial to me.

I'm sure zas will be able to manage it though :)

Don't get us wrong.. we're lazy. We want kiss. We just havent figured out how to get there.

I think we may need to move that wiki page somewhere where others can read it too, so we can get input from folks like atj

ruaok: IME sometimes KISS isn't possible, and by trying to implement it you end up with something more fragile and complex than a better but slightly more involved solution

the thing is that KISS doesn't work well along uber complex problems we are tacking with... like handling a lot of traffic, on lowest resources possible (hardware & humans), using only open source & free software, while providing high performance & high reliability

"uber complex" seems a little extreme, but I guess I'm missing something

are infrastructure costs a significant part of your budget?

nope, if you are considering infrastructure in terms of servers & bandwidth

zas: ruaok: they are using this https://github.com/nginx-proxy/docker-gen/blob/... . don't think it is easy at all to modify this without modifying bunch of other stuff. will try to find a workaround.

lucifer: this template doesn't match the output you posted

you want to insert something in location / {} ?

right, i had posted a subsection of the output only.

yes right i want to insert in location / {}

zas: yeah that's what I meant. It's good that you're not resource constrained in that area, but obviously you need a horizontally scalable solution.

atj: yes, definitively. We still battle against SPoFs, and a long history

zas: to be honest, having lots of additional logic at the gateway/load balancer level is less than ideal. Not sure how feasible it would be to try and reduce/remove that.

atj: I totally agree, this is why we want to redesign that (among plenty of other reasons)

Unfortunately I guess that will require changes at the application level :)

well, that's part of the problem, every move we do always involves a ton of sub problems ;) but we are used to that, it just takes time

lucifer: got it?

zas: yes, thanks! trying it now

lucifer: Is this comment still valid? Do we still have issue submitting MBID and MSID together for pinning a recording?

monkey: yes.

OK, thanks :)

I know there's been some discussion on the subject the past few days, just checking if this should still be commented out

right, we know how to fix this now. just need to implement it.

thanks spotify for putting Frank Sinatra - Stardust on my Release Radar

a real great recently released song

lucifer , ruaok : I created a working version of missing mb data page locally and I tried to use it for a day to see how useful it is. I realised that many are false positives, and that the names of the tracks or the albums are stored as a slightly different name. Should there also be some sort of button to report false positives? Maybe a modal with a textinput box in it?

riksucks: not sure what you mean by false positives. can you show an example?

sure

gimme a sex

*sex

*sec

oops

👍

lmao

[musicbrainz-server] 14reosarevok opened pull request #2327 (03master…MBS-12055): MBS-12055: Allow admins to see private editor-editor subscriptions https://github.com/metabrainz/musicbrainz-serve...

so this is what's showing up on my missing mb data page

then I click the plus and it leads me to the missing mb release add form

which is this

but after putting some detail manually mb identifies duplicate

so now, the end user will know that, oh okay so no need to add, but the missing mb data page will still have that entry

and may clog the page, and hence hinder UX

you feel what I am saying lucifer? I feel like that we need to remove these false positives from LB once the user realises that yeah the release exists

i understand the issue. thinking about what's the best way to handle this.

also monkey, akshat you can comment on the UI, its responsive, but if you would like to make me do some changes you can offer feedback

if data exists and it didn't get mapped then that's an issue on the mapping writer i guess.

yes

we can remove this reported data somehow but the next week will regenerate the data and add it back

alternatively, we create a separate table to store the reported data and delete it from the missing data every week after a new set has been generated

maybe ruaok or alastairp have better suggestions?

hmm true, but someone has to check it regularly right. But I think tasks like these can't be automated, at least all of them

hi, can someone give me a higher-level overview of this?

it seems that LB doesn't have a mapping? but it actually is in MB

what lucifer said sounds good, and I came up with basically the same thing before I read the comment!

We have a missing mb data table in LB which contains some listen data that the user submitted but could not be mapped. It is generated by spark weekly.

the intent is to add a page for each user and so that they can go to MB and and that the data.

now the issue is some of the data is already present in MB but there still are listens in the missing MB data table which could not be mapped to it.

1) allow manual mappings, use this all throughout LB at the same level as automatic mapping, 2) if a manual mapping is different to an automatic mapping, prefer it, 3) if at some later stage an automatic mapping is added that is the same, don't delete the manual one (in case the auto mapping changes and becomes wrong in the future), 4) possibly in the future use manual mapping as feedback for the automapper

so the missing data page should have "I know this is in MB, the id is x" and "add this to MB"

makes sense

mhm yes this is better, also I wanted to ask, we can have that autcompleting search bar thingy in LB right

because manually pasting the ID is not very rewarding, I can maybe use that search thingy that MB uses that will show results, and you can click it, and voila, manual mapping done

is that implementable? or will I run into some cors hell (I am not familiar with MB codebase)

the autocomplete search bar is done using typesense. ruaok can tell if its usable for this use case.

not a cors issue but whether typesense have the needed data and indexes for this.

I see, I will add a manual entry of id and the search bar thingy in a nice modal ig. Does that sound right?

you mean the playlist add track search completion, right? that does not use MB website actually.

yes, like when I was playing around with MB, I was typing the name of song and it was autocompleting for me

I really liked that

MB probably has some endpoints I guess that we can use for this typesense thing

typesense is a part of LB setup actually. it uses the data from MB database but is a part of LB.

ohhh

ruaok can create some endpoints if needed. but there might be other things to consider.

yes

no worries, nothing needed urgently :)

Typesense is pretty slow. We need to be careful how we use it.

indeed.

Adding data to MB? That might be ok. Good reason.

thoughts on manual mapping idea as alastairp suggested above

no worries. till then I have lot of other work to do, I havent written tests and also, I can till then add the feature of manually putting the ID thing. typesense is more of UX sugar that encourages more and more users to add data to mb, which is why I asked

(we can discuss later if you are not available currently)

yes, let's discuss when everyone is free

also lucifer, idk much of how the mapping and stuff is done, I will go over the codebase and then pester you to explain me later on lol. I hope that's okay. right now most of the spark stuff is wizardry to me, so please be patient with me :D

riksucks, sure no worries.

[musicbrainz-server] 14reosarevok opened pull request #2328 (03beta…MBS-12067): MBS-12067: Actually use link_type_filtered on SELECT columns https://github.com/metabrainz/musicbrainz-serve...

bitmap ^ whoops, trivial fix but

hi reosarevok

riksucks: thanks for your work, it looks good to me. But yeah I guess lucifer and others have provided you with a better insight into how to deal with this. Excited for your work btw!

are you avail in 1-2 hours time t owork on adding instruments?

alastairp: do you know how to make docker obey /etc/hosts ?

not sure what you mean

i have added `127.0.0.1 coverartarchive.org` to my `/etc/hosts`. my browser sees it. but the acme proxy docker container resolves coverartarchive.org to the actual url running in Hetzner.

is this on your local setup, or something that you're deploying?

local

what thing is trying to connect to coverartarchive.org?

you're running acme proxy locally?

acme proxy container to verify certs

yes

(in short, no I don't really know how to do this)

remember also that in a container 127.0.01 isn't what you think it is

https://docs.docker.com/engine/reference/comman... did you see this?

oh that looks it. let me try it.

i looked up. looks like extra_hosts is compose equivalent of add host. using that didn't work :/

KassOtsimine: yeah, should be around in a couple hours :)

(earlier too but I might be having lunch)

akshat: thanks :D

alastairp: 🤦 i think LE servers do the dns verification stuff not the container so not appearing as working for me.

from inside container, it pings to my mapping.

i'll try to try this out with similarity.ab on bono instead then.

it should have the necessary dns records in plcae

bitmap, ruaok: I just finished the deployment of serviceregistrator, we shouldn't have any registrator running anywhere anymore.

very good

alastairp: also, did you get a chance to look at those LB PRs?

now that gitzconsul & serviceregistror are deployed, we can think about deploying new instances of consul (on different ports) along gitzconsul + serviceregistrator feeding the new cluster (we can easily run instances of those using different ports)

then we can switch containers to use new consul (switching port used by consul-template in those containers)

still a long way

I'll work on that next week

basically we'll run things in parallel for a while, but both consul clusters should be in sync

lucifer: I'm working on it now!

ah cool, thanks!

lucifer: what are you trying to achieve with the host stuff?

alastairp: i wanted to test the referer redirect thing locally before doing it on purple.

if the referer belongs to one of those radios in the list, then serve a static json response pointing to the cover art ruaok uploaded yesterday otherwise proxy pass to actual caa backend.

i have the nginx proxy config here https://github.com/amCap1712/nginx-playground/t... which i thikn should work.

but when i bring up the containers to test, it gives me a no dns record found error (if i use lucifer.caa) or an invalid response error (if i use caa)

phw: Hi ! I'm looking at PR #1607 again with alastairp, and we're wondering if from the Rhythmbox LB plugin you have access to a version number (and if so, is that plugin version or Rhythmbox version?)

monkey: I guess both would be possible. I personally don't really see the point, but if wanted version numbers could be added