#metabrainz

[musicbrainz-android] 14akshaaatt merged pull request #149 (03master…dependabot/gradle/androidx.test.espresso-espresso-core-3.5.0): Bump espresso-core from 3.4.0 to 3.5.0 https://github.com/metabrainz/musicbrainz-andro...

[musicbrainz-android] 14akshaaatt merged pull request #151 (03master…dependabot/gradle/androidx.test-runner-1.5.0): Bump runner from 1.4.0 to 1.5.0 https://github.com/metabrainz/musicbrainz-andro...

[musicbrainz-android] 14akshaaatt closed pull request #147 (03master…dependabot/gradle/org.jetbrains.kotlin-kotlin-gradle-plugin-1.7.21): Bump kotlin-gradle-plugin from 1.7.10 to 1.7.21 https://github.com/metabrainz/musicbrainz-andro...

alastairp: hi, sorry for disappearing. Just getting super busy with college assignment submissions nowadays 🥲

hi Pratha-Fish, no woriees. I knew that you had school stuff. I hope the assignments are going well

fyi, I'm travelling to Canada starting tomorrow for 10 days, so maybe it'd be better if we talked again after your exams

let me know what you think

alastairp: Thankyou, the assignments are going well, except for the part where the DBMS professor asked us to submit code in a DVD ☠️

haha, amazing

I don't even have any dvd players or writers in my house any more

DVD? That's wild.

I am pretty sure there aint any DVD players in the college or the nearby cyber cafe's either

monkey: I mean... How hard is it to understand git for a CS professor? I was surprised they didn't know about it already

CS != software development

Well, I mean, git and online storgae aside, why not a pendrive‽

pish posh. too modern. punched cards here.

hand written code

that must compile first try

Lol. Jacquard loom templates or nothing.

stone tablets with assembler only.

monkey: they asked for DVDs because they're cheap to archive. They can't just ask every student to submit code in a pendrive and just keep 100s of pendrives

I see.

Anyway, after massive protests by students, the professor finally decided to modernize his ways and asked us to take 100s of pages of printouts for the code and submit it physically.... multiply that by 100s of students that all have the same code submissin

I mean, they probably could? These days you get really small pendrives that might take less space than the same amount of DVDs, but it would probably be harder to store neatly

your professor isn't Elon Musk, by any chance?

Wow. "modernize"

alastairp: My college is devoid of any sight of intelligence. So no, he can't be Elon Musk.

oh he can be very much elon then

Just take my NLP professor for example... I asked her what stop word meant in NLP, and she replied it meant "full stops, commas, dashes, adn any other punctuation"

Stop word removal is like the ABC of NLP... 💀

re: stupidity

as part of cdstub inserting code "lookup_count => int(rand(10)) # FIXME - at least comment why we do this. -- aCiD2" 12 year old comment

I cannot for the life of me even start to imagine why we do this :D

did you just see that election mess, lucifer ?

yup!

seems business as usual for US in recent times /s

then this is no time to move to anchorage. :)

oh phew. and lol.

I mean, to be fair, if you need to move to the US, you might as well go somewhere where a grizzly bear might make it quick

(testing troi playlists which are generated based on timezone and anchorage had midnight in the last hour)

This might be of interest to lucifer: https://github.com/man-group/dtale "Flask back-end and a React front-end to bring you an easy way to view & analyze Pandas data structures"

eh? https://usercontent.irccloud-cdn.com/file/VRKV0...

monkey: that looks nice indeed!

mayhem: did you leave the comment on the wrong PR by chance? #79 stores the id in external url, #80 just uses it.

the comment wasn't specific to a PR, it was a general comment on the solution.

ah cool

[troi-recommendation-playground] 14amCap1712 merged pull request #79 (03main…store-spotify-uri): Improve support for additional playlist metadata https://github.com/metabrainz/troi-recommendati...

zas: let me know when you deploy the keydb changes

atj: let's try now

I deployed keydb, but openresty is still configured to use kiki/herb instances, so it shouldn't have any impact, we need to ensure everything works as expected before anything

well, the haproxy / keepalived stuff could have an impact :)

maybe deploy keepalived first, as that's not currently installed?

yes

you do it?

ok, give me a second

running on rudi now

keepalived config verification failed - "SECURITY VIOLATION - scripts are being executed but script_security not enabled." :)

it's nice to have the validate option catch these at the time the config is installed

another config error now, bare with me 🙄

:) errors are somehow expected

OK, installed "Keepalived_vrrp[1084696]: (KEYDB_VIP) Entering MASTER STATE"

the check script is returning 2 though, so I need to fix that

the IP is assigned to enp41s0

OK, all good, going to deploy on rex now

all working, just shutdown keepalived on rudi and the IP migrated

we can also try stopping one haproxy

alright, I'll stop it on rex?

yes, let's see

looks good https://www.irccloud.com/pastebin/86SWyCKc/

and picard.musicbrainz.org still works ;)

how frequent are the health checks on the Hetzner LB?

restarting haproxy

every 5 seconds, that's not great, but it shouldn't happen often

I think we can reduce the interval a bit, but stopping haproxy shouldn't happen but in case of serious failure or upgrade (reloading config should be transparent)

bit annoying how haproxy is flooding the journald logs

Yes, we need to solve that before sending even more traffic to it, I usually disable traffic logs for it for performance, but at this stage that's useful

btw, the minimum for health checks on load balancer is 3 seconds it seems

I set 3s interval, 3s timeout, 3 retries for all services (3 ports atm)

ok, I think that's fine to be honest, as you say should only occur when something real bad™ happens

so, the more risky part is updating the HAproxy configuration, do you want to disable one gw in the LB or shall we just deploy it?

if we deploy only on one gw and haproxy fails, load balancer will handle the traffic redirection, and haproxy shouldn't reload a broken config

so let's take the risk (btw, disabling a gw from load balancer isn't that easy it seems, no on/off button)

I'll look into this, perhaps that's easier using the API

weird, seems like an obvious piece of functionality

running on rex now

haproxy[3450290]: backend gw_keydb has no server available!

you can add/remove targets, but not mark one as faulty

weird, it is in the file, can it be related to "from" ?

I can't connect to keydb locally

10.2.3.60 -> 10.2.2 *

Server gw_keydb/keydb-rex is DOWN, reason: Layer4 timeout, info: "

keydb instances aren't on 10.2.3 but 10.2.2

wait, I think I know why

why is network_ips.keydb.address set to "10.2.3.60"?

try again

nope

am I missing something obvious

the docker container is listening on *:13062

firewall?

checking

I think keydb is never actually starting

each side is trying to do a MASTER <-> REPLICA sync

I can see traffic flowing between the containers in tcpdump

[troi-recommendation-playground] 14amCap1712 merged pull request #80 (03main…update-existing-playlist): Add support for updating existing spotify playlists https://github.com/metabrainz/troi-recommendati...

hmmm, it seems it's a network issue

[troi-recommendation-playground] release 03v-2022-11-10 has been published by 14amCap1712: https://github.com/metabrainz/troi-recommendati...

1:24:S 10 Nov 2022 14:21:35.886 # Timeout connecting to the MASTER...

[listenbrainz-server] 14amCap1712 opened pull request #2242 (03master…delete-old-playlists): Update existing daily jams playlist on spotify if available https://github.com/metabrainz/listenbrainz-serv...

I'm confused I can't connect to it from the same server, either on localhost or any of the internal IPs