#metabrainz

ruaok: i looked at the code looks good and the explanation is also nice.

are there pros and cons to systemd and forking in python?

*what are

less code for systemd, but then you need systemd -- not that that is going to go away

daemon is the traditional way to go.

if your daemon has some initialization steps that you want other dependent services to wait for, the standard way to handle that is to fork after init is complete (works with trad. init and systemd)

systemd has some proprietary service ready notification stuff for that which is easier to use tho :/

thinking again, we just want this script to run in the background https://github.com/metabrainz/listenbrainz-serv...

request consumer itself is run by spark, so i don't think we should need to change it

ruaok, the issue with using python code i see now is that we execute spark which executes request consumer.

so even if daemonize the request consumer, it won't affect how we run the script

(this is another thing about our setup which irks me sometimes but that's for another day)

hmmm. that is right. we spawn new processes.

that makes me lean towards the systemd solution

https://docs.python.org/3/library/heapq.html#pr... nice

ruaok: LB-891 LB-892

LB-891: Check that spotify importer is keeping up with user listens https://tickets.metabrainz.org/browse/LB-891

LB-892: Move spotify now-playing checks out of listen importer https://tickets.metabrainz.org/browse/LB-892

+!

I'll update these with the results of our discussion

ruaok: yeah currently, running the start script in background seems good enough.

in the longer run, we should look into daemonizing the request consumer and eliminate the the spark-submit script if possible.

daemonizing request consumer is only useful if we can get `spark-submit` out of the equation.

alastairp: thoughts on adding this to LB sentry project? https://github.com/metabrainz/syswiki/blob/mast...

this is so that you can click the "turn this into a ticket" button?

yup

I like it, I use it all the time with github integration in freesound

let's do it then :D

looks like the steps have changed. i'll try the sentry's documentation instead.

lucifer: just FYI, zas is still working on the server, since its a dist upgrade, he is going to take his time to get things working right.

lets just plan on moving to kiss tomorrow morning.

cool, makes sense.

unless of course a miracle happens and everything gets done before we're too tired to move.

have you resubmitted the Youtube auth? if not let's not do that till beta and test are also up

yea, waiting for everything to be stable again.

+1

yvanzo: hi! i am trying to setup JIRA integration for sentry. and it looks like https://github.com/metabrainz/syswiki/blob/mast... are no longer applicable. we probably need to do something like https://docs.sentry.io/product/integrations/jir.... could you please create a key pair and add it to tickets.meb ?

ruaok: alastairp: can we fix some date to work ahead on LB-899 soon, we also probably need to involve zas and bitmap.

LB-899: Create a plan for recovering production database in case something goes wrong https://tickets.metabrainz.org/browse/LB-899

how about we bug bitmap after the meeting on monday?

and discuss what it would take?

sounds like a good plan!

monkey: alastairp : this is the article I was talking about earlier: https://medium.com/cuepoint/like-it-is-bob-dyla...

Fanks

Hey! I spent a night working on the big BrainzBot upgrade (py2 -> py3, django 1.8 -> 3.1, go 1.2 -> 1.10) and it's running pretty stable so far. I'll set up a staging env next so people can try breaking it. :P

hi Leo_Verto! long time no see.

Hi Leo_Verto !

Hey ruaok! Yeah, it's been a while.

oh hey, I was just thinking about botbot django upgrades last week!

\o monkey

hi

hey alastair, yeah that was a pretty fun upgrade experience

I dropped the old Django migrations but the tables shouldn't have changed too much to make migration using a db dump impossible.

[listenbrainz-server] 14alastair opened pull request #1488 (03master…doc-cleanups): LB-552, LB-554, LB-507: Improve/remove out of date docs https://github.com/metabrainz/listenbrainz-serv...

What kind of artifacts would be ideal for this? Docker images? I've got a compose file mostly done, can also write a helm chart if you're doing k8s these days.

alastairp, in context of LB-552, we always need to format on first run.

LB-552: Should not need to format hadoop on first run https://tickets.metabrainz.org/browse/LB-552

its not a configuration error, https://hadoop.apache.org/docs/r3.1.1/hadoop-pr...

its mentioned under Operating the Hadoop cluster towards the end of the page

lucifer: ah, I may have confused this with a bug that we had when stopping and restarting containers

we can probably just remove that ticket from the list on the PR, then

the other two are valid

yup, right.

[listenbrainz-server] 14alastair merged pull request #1488 (03master…doc-cleanups): LB-554, LB-507: Improve/remove out of date docs https://github.com/metabrainz/listenbrainz-serv...

I had resolved a few earlier in the week, their status is now Review Submitted. those can be closed as well. https://tickets.metabrainz.org/browse/LB-876?jq...

LB-876: Listens for a user do not get removed if user is deleted via admin interface

lucifer: yeah, I saw all of those, that's what incentivised me to go through and close some myself

thanks

oh! nice :D

feel free to close them if you want? no reason to go first to review submitted if they've been merged ages ago

Kiss is nearly ready. Shall we move cron back?

give me 40 minutes to get home? unless you want to do it yourself

Perfect. Let's do that

sure, i was earlier under the impression that i needed someone else to close because i had marked it as fixed but i figured out that's not right later.

kiss is ready (apart few bits I'll fix tomorrow, mainly related to telegraf).

!m zas

You're doing good work, zas!

ruaok: there's no docker-server-configs yet though, that's needed for moving cron back?

yes, we'll need to check it out.

do we have a deploy key for kiss yet, zas?

nope

Leo_Verto, I don't know if Freso mentioned this to you but there is a gap in the logs that could be filled. The dumps from my local logging are at https://pastebin.com/a9GL9FUj and https://pastebin.com/AZkVZuzN if you want to import them.\

docker-server-configs exist now, lucifer

oh nice! only consul container left now before we can bring up cron i guess ;)

comment out all the services but the three non LB containers and run start_services

on it

looking good

did that, but 2 containers are restarting, did something go wrong?

huh

==> Failed to parse config: Failed to parse Bind address, "0" is not a valid IP address or socket

consul agent logs

hmmm. not good.

> dnsmasq: bad command line options: bad address

zas is off having dinner with the family. this is really his territory.

so maybe we can try later once he returns, otherwise tomorrow.

yup, makes sense.

I'll look into this after dinner

rdswift: ah perfect! for some reason I can't access those pastes, assuming they're not private pastebin may have flagged them as spam.

looks like the PRIVATE_IP is not getting set correctly, PRIVATE_IP is being set by https://github.com/metabrainz/docker-server-con...

no idea what it is or does

I'm here. anything I can help with or are we postponing?

the containers are not starting up right.

so, if we can fix those, then maybe we can proceed.

ah, yes.

many networking things have changed.

thats our probelm

yup, right

`ip -4 route get $PRIVATE_SUBNET | head -1 | colrm 1 34 | colrm 10`

works, but is hacky.

🤯

lol

I dont speak awk, sadly.

$NF is number of fields

so it's like saying "the last column"

ah, but there is more data now.

yeah

"uid 0" needs chopping off.

funny that in the non-"get" format, the route is the last column

ip -4 route |grep $PRIVATE_SUBNET |awk '{print $NF;exit}'

:(

oh! kiss is 20.04 but other nodes are 16.04. so that's why it changed?

yes

`ip -4 route get $PRIVATE_SUBNET|awk '{print $6;exit}'`

works.

works on boingo too.

right, and even better that it works in both casees

let me try it

registrator is uip.

dnsmasq: failed to create listening socket for port 53: Address in use

Leo_Verto, pastebin insists on saving them as private. I've moved them to a gist on github at https://gist.github.com/rdswift/56e8ff71514841a...

we can still bring up cron with dnsmasq though or not?

*without

gimme a sec to try to fix dnsmasq.

its something odd inside the container. there is nothing listening on port 53.

WARNING: Published ports are discarded when using host network mode

it is using host mode, but there is nothing on port 53.

Leo_Verto, note that there are a couple of lines of overlap at each end to help confirm the location of the missing log lines.

ruaok: it's possible that systemd's dns server is on udp:53 on kiss

"dnsmasq requires NET_ADMIN capabilities to run correctly. Start it with something like docker run -p 53:53/tcp -p 53:53/udp --cap-add=NET_ADMIN 4km3/dnsmasq:2.75."

possible yes. how does one check that?

there don't see to be any updates to this container and we pass NET_ADMIN.

`sudo systemctl status systemd-resolved` shows that it's definitely running, but it's uncear if it's listening on any ports that are important to us