atj: haproxy is good at load balancing, we use it too for different things, but, we use openresty for historical reasons, our gateways do much more than load balancing. That said, we are hitting the limits of this approach, hence current discussions
2021-11-04 30813, 2021
atj
ruaok: generally KISS is the best option, so if you can get away with IP RR then that's ideal. However writing code to reliably detect application state and invoke IP failovers via a third party API sounds less than trivial to me.
2021-11-04 30842, 2021
atj
I'm sure zas will be able to manage it though :)
2021-11-04 30840, 2021
ruaok
Don't get us wrong.. we're lazy. We want kiss. We just havent figured out how to get there.
2021-11-04 30854, 2021
aerozol has quit
2021-11-04 30837, 2021
ruaok
I think we may need to move that wiki page somewhere where others can read it too, so we can get input from folks like atj
2021-11-04 30836, 2021
atj
ruaok: IME sometimes KISS isn't possible, and by trying to implement it you end up with something more fragile and complex than a better but slightly more involved solution
2021-11-04 30859, 2021
zas
the thing is that KISS doesn't work well along uber complex problems we are tacking with... like handling a lot of traffic, on lowest resources possible (hardware & humans), using only open source & free software, while providing high performance & high reliability
2021-11-04 30825, 2021
atj
"uber complex" seems a little extreme, but I guess I'm missing something
2021-11-04 30837, 2021
atj
are infrastructure costs a significant part of your budget?
2021-11-04 30830, 2021
zas
nope, if you are considering infrastructure in terms of servers & bandwidth
2021-11-04 30857, 2021
lucifer
zas: ruaok: they are using this https://github.com/nginx-proxy/docker-gen/blob/ma… . don't think it is easy at all to modify this without modifying bunch of other stuff. will try to find a workaround.
2021-11-04 30811, 2021
zas
lucifer: this template doesn't match the output you posted
2021-11-04 30828, 2021
zas
you want to insert something in location / {} ?
2021-11-04 30842, 2021
lucifer
right, i had posted a subsection of the output only.
zas: yeah that's what I meant. It's good that you're not resource constrained in that area, but obviously you need a horizontally scalable solution.
2021-11-04 30847, 2021
zas
atj: yes, definitively. We still battle against SPoFs, and a long history
2021-11-04 30814, 2021
atj
zas: to be honest, having lots of additional logic at the gateway/load balancer level is less than ideal. Not sure how feasible it would be to try and reduce/remove that.
2021-11-04 30858, 2021
zas
atj: I totally agree, this is why we want to redesign that (among plenty of other reasons)
2021-11-04 30842, 2021
atj
Unfortunately I guess that will require changes at the application level :)
2021-11-04 30839, 2021
zas
well, that's part of the problem, every move we do always involves a ton of sub problems ;) but we are used to that, it just takes time
2021-11-04 30821, 2021
zas
lucifer: got it?
2021-11-04 30837, 2021
lucifer
zas: yes, thanks! trying it now
2021-11-04 30812, 2021
monkey
lucifer: Is this comment still valid? Do we still have issue submitting MBID and MSID together for pinning a recording?
I know there's been some discussion on the subject the past few days, just checking if this should still be commented out
2021-11-04 30846, 2021
lucifer
right, we know how to fix this now. just need to implement it.
2021-11-04 30823, 2021
monkey is excited to see MBIDs used instead of MSIDs everywhere on the front-end
2021-11-04 30824, 2021
alastairp
thanks spotify for putting Frank Sinatra - Stardust on my Release Radar
2021-11-04 30830, 2021
alastairp
a real great recently released song
2021-11-04 30809, 2021
riksucks
lucifer , ruaok : I created a working version of missing mb data page locally and I tried to use it for a day to see how useful it is. I realised that many are false positives, and that the names of the tracks or the albums are stored as a slightly different name. Should there also be some sort of button to report false positives? Maybe a modal with a textinput box in it?
2021-11-04 30858, 2021
lucifer
riksucks: not sure what you mean by false positives. can you show an example?
so now, the end user will know that, oh okay so no need to add, but the missing mb data page will still have that entry
2021-11-04 30809, 2021
riksucks
and may clog the page, and hence hinder UX
2021-11-04 30859, 2021
riksucks
you feel what I am saying lucifer? I feel like that we need to remove these false positives from LB once the user realises that yeah the release exists
2021-11-04 30847, 2021
lucifer
i understand the issue. thinking about what's the best way to handle this.
2021-11-04 30818, 2021
riksucks
also monkey, akshat you can comment on the UI, its responsive, but if you would like to make me do some changes you can offer feedback
2021-11-04 30821, 2021
lucifer
if data exists and it didn't get mapped then that's an issue on the mapping writer i guess.
2021-11-04 30834, 2021
riksucks
yes
2021-11-04 30857, 2021
lucifer
we can remove this reported data somehow but the next week will regenerate the data and add it back
2021-11-04 30852, 2021
lucifer
alternatively, we create a separate table to store the reported data and delete it from the missing data every week after a new set has been generated
2021-11-04 30819, 2021
lucifer
maybe ruaok or alastairp have better suggestions?
2021-11-04 30842, 2021
riksucks
hmm true, but someone has to check it regularly right. But I think tasks like these can't be automated, at least all of them
2021-11-04 30811, 2021
alastairp
hi, can someone give me a higher-level overview of this?
2021-11-04 30832, 2021
alastairp
it seems that LB doesn't have a mapping? but it actually is in MB
2021-11-04 30853, 2021
alastairp
what lucifer said sounds good, and I came up with basically the same thing before I read the comment!
2021-11-04 30813, 2021
lucifer
We have a missing mb data table in LB which contains some listen data that the user submitted but could not be mapped. It is generated by spark weekly.
2021-11-04 30857, 2021
lucifer
the intent is to add a page for each user and so that they can go to MB and and that the data.
2021-11-04 30851, 2021
lucifer
now the issue is some of the data is already present in MB but there still are listens in the missing MB data table which could not be mapped to it.
2021-11-04 30802, 2021
alastairp
1) allow manual mappings, use this all throughout LB at the same level as automatic mapping, 2) if a manual mapping is different to an automatic mapping, prefer it, 3) if at some later stage an automatic mapping is added that is the same, don't delete the manual one (in case the auto mapping changes and becomes wrong in the future), 4) possibly in the future use manual mapping as feedback for the automapper
2021-11-04 30856, 2021
alastairp
so the missing data page should have "I know this is in MB, the id is x" and "add this to MB"
2021-11-04 30816, 2021
lucifer
makes sense
2021-11-04 30810, 2021
riksucks
mhm yes this is better, also I wanted to ask, we can have that autcompleting search bar thingy in LB right
2021-11-04 30848, 2021
riksucks
because manually pasting the ID is not very rewarding, I can maybe use that search thingy that MB uses that will show results, and you can click it, and voila, manual mapping done
2021-11-04 30819, 2021
riksucks
is that implementable? or will I run into some cors hell (I am not familiar with MB codebase)
2021-11-04 30821, 2021
lucifer
the autocomplete search bar is done using typesense. ruaok can tell if its usable for this use case.
2021-11-04 30859, 2021
lucifer
not a cors issue but whether typesense have the needed data and indexes for this.
2021-11-04 30821, 2021
riksucks
I see, I will add a manual entry of id and the search bar thingy in a nice modal ig. Does that sound right?
2021-11-04 30827, 2021
lucifer
you mean the playlist add track search completion, right? that does not use MB website actually.
2021-11-04 30859, 2021
riksucks
yes, like when I was playing around with MB, I was typing the name of song and it was autocompleting for me
2021-11-04 30809, 2021
riksucks
I really liked that
2021-11-04 30840, 2021
riksucks
MB probably has some endpoints I guess that we can use for this typesense thing
2021-11-04 30812, 2021
lucifer
typesense is a part of LB setup actually. it uses the data from MB database but is a part of LB.
2021-11-04 30836, 2021
riksucks
ohhh
2021-11-04 30839, 2021
lucifer
ruaok can create some endpoints if needed. but there might be other things to consider.
2021-11-04 30849, 2021
riksucks
yes
2021-11-04 30820, 2021
ruaok is not really here
2021-11-04 30842, 2021
lucifer
no worries, nothing needed urgently :)
2021-11-04 30825, 2021
ruaok
Typesense is pretty slow. We need to be careful how we use it.
2021-11-04 30835, 2021
lucifer
indeed.
2021-11-04 30840, 2021
ruaok
Adding data to MB? That might be ok. Good reason.
2021-11-04 30812, 2021
lucifer
thoughts on manual mapping idea as alastairp suggested above
2021-11-04 30814, 2021
riksucks
no worries. till then I have lot of other work to do, I havent written tests and also, I can till then add the feature of manually putting the ID thing. typesense is more of UX sugar that encourages more and more users to add data to mb, which is why I asked
2021-11-04 30836, 2021
lucifer
(we can discuss later if you are not available currently)
2021-11-04 30806, 2021
riksucks
yes, let's discuss when everyone is free
2021-11-04 30831, 2021
riksucks
also lucifer, idk much of how the mapping and stuff is done, I will go over the codebase and then pester you to explain me later on lol. I hope that's okay. right now most of the spark stuff is wizardry to me, so please be patient with me :D
Guess no point hotfixing beta, we can release on next beta update
2021-11-04 30855, 2021
CatQuest
hi reosarevok
2021-11-04 30803, 2021
akshat
riksucks: thanks for your work, it looks good to me. But yeah I guess lucifer and others have provided you with a better insight into how to deal with this. Excited for your work btw!
2021-11-04 30819, 2021
CatQuest
are you avail in 1-2 hours time t owork on adding instruments?
2021-11-04 30848, 2021
MRiddickW joined the channel
2021-11-04 30841, 2021
lucifer
alastairp: do you know how to make docker obey /etc/hosts ?
2021-11-04 30812, 2021
alastairp
not sure what you mean
2021-11-04 30831, 2021
lucifer
i have added `127.0.0.1 coverartarchive.org` to my `/etc/hosts`. my browser sees it. but the acme proxy docker container resolves coverartarchive.org to the actual url running in Hetzner.
2021-11-04 30842, 2021
alastairp
is this on your local setup, or something that you're deploying?
2021-11-04 30847, 2021
lucifer
local
2021-11-04 30859, 2021
alastairp
what thing is trying to connect to coverartarchive.org?
2021-11-04 30811, 2021
alastairp
you're running acme proxy locally?
2021-11-04 30813, 2021
lucifer
acme proxy container to verify certs
2021-11-04 30814, 2021
lucifer
yes
2021-11-04 30826, 2021
alastairp
(in short, no I don't really know how to do this)
2021-11-04 30854, 2021
alastairp
remember also that in a container 127.0.01 isn't what you think it is
i'll try to try this out with similarity.ab on bono instead then.
2021-11-04 30847, 2021
lucifer
it should have the necessary dns records in plcae
2021-11-04 30823, 2021
zas
bitmap, ruaok: I just finished the deployment of serviceregistrator, we shouldn't have any registrator running anywhere anymore.
2021-11-04 30826, 2021
ruaok
very good
2021-11-04 30816, 2021
lucifer
alastairp: also, did you get a chance to look at those LB PRs?
2021-11-04 30813, 2021
zas
now that gitzconsul & serviceregistror are deployed, we can think about deploying new instances of consul (on different ports) along gitzconsul + serviceregistrator feeding the new cluster (we can easily run instances of those using different ports)
2021-11-04 30851, 2021
zas
then we can switch containers to use new consul (switching port used by consul-template in those containers)
2021-11-04 30802, 2021
zas
still a long way
2021-11-04 30824, 2021
zas
I'll work on that next week
2021-11-04 30808, 2021
zas
basically we'll run things in parallel for a while, but both consul clusters should be in sync
2021-11-04 30847, 2021
alastairp
lucifer: I'm working on it now!
2021-11-04 30801, 2021
lucifer
ah cool, thanks!
2021-11-04 30810, 2021
alastairp
lucifer: what are you trying to achieve with the host stuff?
2021-11-04 30814, 2021
lucifer
alastairp: i wanted to test the referer redirect thing locally before doing it on purple.
2021-11-04 30855, 2021
lucifer
if the referer belongs to one of those radios in the list, then serve a static json response pointing to the cover art ruaok uploaded yesterday otherwise proxy pass to actual caa backend.
but when i bring up the containers to test, it gives me a no dns record found error (if i use lucifer.caa) or an invalid response error (if i use caa)
2021-11-04 30850, 2021
MRiddickW has quit
2021-11-04 30837, 2021
monkey
phw: Hi ! I'm looking at PR #1607 again with alastairp, and we're wondering if from the Rhythmbox LB plugin you have access to a version number (and if so, is that plugin version or Rhythmbox version?)
