#metabrainz

/

      • lucifer
        ruaok: i looked at the code looks good and the explanation is also nice.
      • 2021-05-27 14729, 2021

      • lucifer
        are there pros and cons to systemd and forking in python?
      • 2021-05-27 14755, 2021

      • lucifer
        *what are
      • 2021-05-27 14718, 2021

      • ruaok
        less code for systemd, but then you need systemd -- not that that is going to go away
      • 2021-05-27 14732, 2021

      • ruaok
        daemon is the traditional way to go.
      • 2021-05-27 14734, 2021

      • kepstin
        if your daemon has some initialization steps that you want other dependent services to wait for, the standard way to handle that is to fork after init is complete (works with trad. init and systemd)
      • 2021-05-27 14757, 2021

      • kepstin
        systemd has some proprietary service ready notification stuff for that which is easier to use tho :/
      • 2021-05-27 14727, 2021

      • lucifer
        thinking again, we just want this script to run in the background https://github.com/metabrainz/listenbrainz-server…
      • 2021-05-27 14714, 2021

      • lucifer
        request consumer itself is run by spark, so i don't think we should need to change it
      • 2021-05-27 14727, 2021

      • lucifer
        ruaok, the issue with using python code i see now is that we execute spark which executes request consumer.
      • 2021-05-27 14754, 2021

      • lucifer
        so even if daemonize the request consumer, it won't affect how we run the script
      • 2021-05-27 14721, 2021

      • lucifer
        (this is another thing about our setup which irks me sometimes but that's for another day)
      • 2021-05-27 14754, 2021

      • ruaok
        hmmm. that is right. we spawn new processes.
      • 2021-05-27 14718, 2021

      • ruaok
        that makes me lean towards the systemd solution
      • 2021-05-27 14745, 2021

      • alastairp
      • 2021-05-27 14703, 2021

      • alastairp
        ruaok: LB-891 LB-892
      • 2021-05-27 14704, 2021

      • BrainzBot
        LB-891: Check that spotify importer is keeping up with user listens https://tickets.metabrainz.org/browse/LB-891
      • 2021-05-27 14704, 2021

      • BrainzBot
        LB-892: Move spotify now-playing checks out of listen importer https://tickets.metabrainz.org/browse/LB-892
      • 2021-05-27 14720, 2021

      • ruaok
        +!
      • 2021-05-27 14724, 2021

      • alastairp
        I'll update these with the results of our discussion
      • 2021-05-27 14731, 2021

      • lucifer
        ruaok: yeah currently, running the start script in background seems good enough.
      • 2021-05-27 14744, 2021

      • lucifer
        in the longer run, we should look into daemonizing the request consumer and eliminate the the spark-submit script if possible.
      • 2021-05-27 14705, 2021

      • lucifer
        daemonizing request consumer is only useful if we can get `spark-submit` out of the equation.
      • 2021-05-27 14717, 2021

      • lucifer
        alastairp: thoughts on adding this to LB sentry project? https://github.com/metabrainz/syswiki/blob/master…
      • 2021-05-27 14720, 2021

      • alastairp
        this is so that you can click the "turn this into a ticket" button?
      • 2021-05-27 14725, 2021

      • lucifer
        yup
      • 2021-05-27 14731, 2021

      • alastairp
        I like it, I use it all the time with github integration in freesound
      • 2021-05-27 14743, 2021

      • lucifer
        let's do it then :D
      • 2021-05-27 14746, 2021

      • lucifer
        looks like the steps have changed. i'll try the sentry's documentation instead.
      • 2021-05-27 14712, 2021

      • ruaok
        lucifer: just FYI, zas is still working on the server, since its a dist upgrade, he is going to take his time to get things working right.
      • 2021-05-27 14727, 2021

      • ruaok
        lets just plan on moving to kiss tomorrow morning.
      • 2021-05-27 14742, 2021

      • lucifer
        cool, makes sense.
      • 2021-05-27 14702, 2021

      • ruaok
        unless of course a miracle happens and everything gets done before we're too tired to move.
      • 2021-05-27 14716, 2021

      • lucifer
        have you resubmitted the Youtube auth? if not let's not do that till beta and test are also up
      • 2021-05-27 14733, 2021

      • ruaok
        yea, waiting for everything to be stable again.
      • 2021-05-27 14741, 2021

      • lucifer
        +1
      • 2021-05-27 14720, 2021

      • lucifer
        yvanzo: hi! i am trying to setup JIRA integration for sentry. and it looks like https://github.com/metabrainz/syswiki/blob/master… are no longer applicable. we probably need to do something like https://docs.sentry.io/product/integrations/jira/…. could you please create a key pair and add it to tickets.meb ?
      • 2021-05-27 14731, 2021

      • lucifer
        ruaok: alastairp: can we fix some date to work ahead on LB-899 soon, we also probably need to involve zas and bitmap.
      • 2021-05-27 14731, 2021

      • BrainzBot
        LB-899: Create a plan for recovering production database in case something goes wrong https://tickets.metabrainz.org/browse/LB-899
      • 2021-05-27 14728, 2021

      • Leo__Verto has quit
      • 2021-05-27 14732, 2021

      • Leo_Verto joined the channel
      • 2021-05-27 14749, 2021

      • ruaok
        how about we bug bitmap after the meeting on monday?
      • 2021-05-27 14755, 2021

      • ruaok
        and discuss what it would take?
      • 2021-05-27 14710, 2021

      • lucifer
        sounds like a good plan!
      • 2021-05-27 14709, 2021

      • ruaok
        monkey: alastairp : this is the article I was talking about earlier: https://medium.com/cuepoint/like-it-is-bob-dylan-…
      • 2021-05-27 14747, 2021

      • monkey
        Fanks
      • 2021-05-27 14742, 2021

      • Leo_Verto
        Hey! I spent a night working on the big BrainzBot upgrade (py2 -> py3, django 1.8 -> 3.1, go 1.2 -> 1.10) and it's running pretty stable so far. I'll set up a staging env next so people can try breaking it. :P
      • 2021-05-27 14707, 2021

      • ruaok
        hi Leo_Verto! long time no see.
      • 2021-05-27 14757, 2021

      • monkey
        Hi Leo_Verto !
      • 2021-05-27 14758, 2021

      • Leo_Verto
        Hey ruaok! Yeah, it's been a while.
      • 2021-05-27 14704, 2021

      • alastairp
        oh hey, I was just thinking about botbot django upgrades last week!
      • 2021-05-27 14704, 2021

      • Leo_Verto
        \o monkey
      • 2021-05-27 14708, 2021

      • alastairp
        hi
      • 2021-05-27 14734, 2021

      • Leo_Verto
        hey alastair, yeah that was a pretty fun upgrade experience
      • 2021-05-27 14736, 2021

      • Leo_Verto
        I dropped the old Django migrations but the tables shouldn't have changed too much to make migration using a db dump impossible.
      • 2021-05-27 14725, 2021

      • BrainzGit
        [listenbrainz-server] 14alastair opened pull request #1488 (03master…doc-cleanups): LB-552, LB-554, LB-507: Improve/remove out of date docs https://github.com/metabrainz/listenbrainz-server…
      • 2021-05-27 14721, 2021

      • Leo_Verto
        What kind of artifacts would be ideal for this? Docker images? I've got a compose file mostly done, can also write a helm chart if you're doing k8s these days.
      • 2021-05-27 14756, 2021

      • lucifer
        alastairp, in context of LB-552, we always need to format on first run.
      • 2021-05-27 14756, 2021

      • BrainzBot
        LB-552: Should not need to format hadoop on first run https://tickets.metabrainz.org/browse/LB-552
      • 2021-05-27 14719, 2021

      • lucifer
      • 2021-05-27 14744, 2021

      • lucifer
        its mentioned under Operating the Hadoop cluster towards the end of the page
      • 2021-05-27 14713, 2021

      • alastairp
        lucifer: ah, I may have confused this with a bug that we had when stopping and restarting containers
      • 2021-05-27 14726, 2021

      • alastairp
        we can probably just remove that ticket from the list on the PR, then
      • 2021-05-27 14729, 2021

      • alastairp
        the other two are valid
      • 2021-05-27 14734, 2021

      • lucifer
        yup, right.
      • 2021-05-27 14711, 2021

      • BrainzGit
        [listenbrainz-server] 14alastair merged pull request #1488 (03master…doc-cleanups): LB-554, LB-507: Improve/remove out of date docs https://github.com/metabrainz/listenbrainz-server…
      • 2021-05-27 14714, 2021

      • alastairp closes more jira tickets
      • 2021-05-27 14744, 2021

      • lucifer
        I had resolved a few earlier in the week, their status is now Review Submitted. those can be closed as well. https://tickets.metabrainz.org/browse/LB-876?jql=…
      • 2021-05-27 14745, 2021

      • BrainzBot
        LB-876: Listens for a user do not get removed if user is deleted via admin interface
      • 2021-05-27 14753, 2021

      • alastairp
        lucifer: yeah, I saw all of those, that's what incentivised me to go through and close some myself
      • 2021-05-27 14754, 2021

      • alastairp
        thanks
      • 2021-05-27 14715, 2021

      • lucifer
        oh! nice :D
      • 2021-05-27 14731, 2021

      • alastairp
        feel free to close them if you want? no reason to go first to review submitted if they've been merged ages ago
      • 2021-05-27 14728, 2021

      • ruaok
        Kiss is nearly ready. Shall we move cron back?
      • 2021-05-27 14703, 2021

      • alastairp
        give me 40 minutes to get home? unless you want to do it yourself
      • 2021-05-27 14727, 2021

      • ruaok
        Perfect. Let's do that
      • 2021-05-27 14728, 2021

      • lucifer
        sure, i was earlier under the impression that i needed someone else to close because i had marked it as fixed but i figured out that's not right later.
      • 2021-05-27 14726, 2021

      • lucifer logs into the new machine \o/
      • 2021-05-27 14700, 2021

      • zas
        kiss is ready (apart few bits I'll fix tomorrow, mainly related to telegraf).
      • 2021-05-27 14725, 2021

      • lucifer
        !m zas
      • 2021-05-27 14725, 2021

      • BrainzBot
        You're doing good work, zas!
      • 2021-05-27 14733, 2021

      • lucifer
        ruaok: there's no docker-server-configs yet though, that's needed for moving cron back?
      • 2021-05-27 14736, 2021

      • ruaok
        yes, we'll need to check it out.
      • 2021-05-27 14747, 2021

      • ruaok
        do we have a deploy key for kiss yet, zas?
      • 2021-05-27 14756, 2021

      • zas
        nope
      • 2021-05-27 14702, 2021

      • rdswift
        Leo_Verto, I don't know if Freso mentioned this to you but there is a gap in the logs that could be filled. The dumps from my local logging are at https://pastebin.com/a9GL9FUj and https://pastebin.com/AZkVZuzN if you want to import them.\
      • 2021-05-27 14743, 2021

      • ruaok
        docker-server-configs exist now, lucifer
      • 2021-05-27 14701, 2021

      • jwf joined the channel
      • 2021-05-27 14744, 2021

      • lucifer
        oh nice! only consul container left now before we can bring up cron i guess ;)
      • 2021-05-27 14739, 2021

      • ruaok
        comment out all the services but the three non LB containers and run start_services
      • 2021-05-27 14755, 2021

      • lucifer
        on it
      • 2021-05-27 14751, 2021

      • ruaok
        looking good
      • 2021-05-27 14758, 2021

      • lucifer
        did that, but 2 containers are restarting, did something go wrong?
      • 2021-05-27 14717, 2021

      • ruaok
        huh
      • 2021-05-27 14730, 2021

      • lucifer
        ==> Failed to parse config: Failed to parse Bind address, "0" is not a valid IP address or socket
      • 2021-05-27 14737, 2021

      • lucifer
        consul agent logs
      • 2021-05-27 14741, 2021

      • ruaok
        hmmm. not good.
      • 2021-05-27 14704, 2021

      • ruaok
        > dnsmasq: bad command line options: bad address
      • 2021-05-27 14712, 2021

      • ruaok
        zas is off having dinner with the family. this is really his territory.
      • 2021-05-27 14722, 2021

      • ruaok
        so maybe we can try later once he returns, otherwise tomorrow.
      • 2021-05-27 14709, 2021

      • lucifer
        yup, makes sense.
      • 2021-05-27 14737, 2021

      • zas
        I'll look into this after dinner
      • 2021-05-27 14741, 2021

      • Leo_Verto
        rdswift: ah perfect! for some reason I can't access those pastes, assuming they're not private pastebin may have flagged them as spam.
      • 2021-05-27 14750, 2021

      • lucifer
        looks like the PRIVATE_IP is not getting set correctly, PRIVATE_IP is being set by https://github.com/metabrainz/docker-server-confi…
      • 2021-05-27 14736, 2021

      • lucifer
        no idea what it is or does
      • 2021-05-27 14717, 2021

      • alastairp
        I'm here. anything I can help with or are we postponing?
      • 2021-05-27 14739, 2021

      • ruaok
        the containers are not starting up right.
      • 2021-05-27 14716, 2021

      • ruaok
        so, if we can fix those, then maybe we can proceed.
      • 2021-05-27 14727, 2021

      • ruaok
        ah, yes.
      • 2021-05-27 14735, 2021

      • ruaok
        many networking things have changed.
      • 2021-05-27 14723, 2021

      • ruaok
      • 2021-05-27 14737, 2021

      • ruaok
        thats our probelm
      • 2021-05-27 14758, 2021

      • lucifer
        yup, right
      • 2021-05-27 14712, 2021

      • ruaok
        `ip -4 route get $PRIVATE_SUBNET | head -1 | colrm 1 34 | colrm 10`
      • 2021-05-27 14716, 2021

      • ruaok
        works, but is hacky.
      • 2021-05-27 14734, 2021

      • lucifer
        🤯
      • 2021-05-27 14757, 2021

      • ruaok
        lol
      • 2021-05-27 14721, 2021

      • ruaok
        I dont speak awk, sadly.
      • 2021-05-27 14748, 2021

      • alastairp
        $NF is number of fields
      • 2021-05-27 14755, 2021

      • alastairp
        so it's like saying "the last column"
      • 2021-05-27 14716, 2021

      • ruaok
        ah, but there is more data now.
      • 2021-05-27 14720, 2021

      • alastairp
        yeah
      • 2021-05-27 14727, 2021

      • ruaok
        "uid 0" needs chopping off.
      • 2021-05-27 14734, 2021

      • alastairp
      • 2021-05-27 14744, 2021

      • alastairp
        funny that in the non-"get" format, the route is the last column
      • 2021-05-27 14731, 2021

      • alastairp
        ip -4 route |grep $PRIVATE_SUBNET |awk '{print $NF;exit}'
      • 2021-05-27 14732, 2021

      • alastairp
        :(
      • 2021-05-27 14756, 2021

      • lucifer
        oh! kiss is 20.04 but other nodes are 16.04. so that's why it changed?
      • 2021-05-27 14741, 2021

      • ruaok
        yes
      • 2021-05-27 14708, 2021

      • ruaok
        `ip -4 route get $PRIVATE_SUBNET|awk '{print $6;exit}'`
      • 2021-05-27 14709, 2021

      • ruaok
        works.
      • 2021-05-27 14739, 2021

      • ruaok
        works on boingo too.
      • 2021-05-27 14743, 2021

      • lucifer
        right, and even better that it works in both casees
      • 2021-05-27 14708, 2021

      • ruaok
        let me try it
      • 2021-05-27 14712, 2021

      • ruaok
        registrator is uip.
      • 2021-05-27 14733, 2021

      • ruaok
        dnsmasq: failed to create listening socket for port 53: Address in use
      • 2021-05-27 14749, 2021

      • rdswift
        Leo_Verto, pastebin insists on saving them as private. I've moved them to a gist on github at https://gist.github.com/rdswift/56e8ff71514841a72…
      • 2021-05-27 14728, 2021

      • lucifer
        we can still bring up cron with dnsmasq though or not?
      • 2021-05-27 14735, 2021

      • lucifer
        *without
      • 2021-05-27 14743, 2021

      • ruaok
        gimme a sec to try to fix dnsmasq.
      • 2021-05-27 14727, 2021

      • ruaok
        its something odd inside the container. there is nothing listening on port 53.
      • 2021-05-27 14705, 2021

      • ruaok
        WARNING: Published ports are discarded when using host network mode
      • 2021-05-27 14730, 2021

      • ruaok
        it is using host mode, but there is nothing on port 53.
      • 2021-05-27 14718, 2021

      • rdswift
        Leo_Verto, note that there are a couple of lines of overlap at each end to help confirm the location of the missing log lines.
      • 2021-05-27 14757, 2021

      • alastairp
        ruaok: it's possible that systemd's dns server is on udp:53 on kiss
      • 2021-05-27 14709, 2021

      • ruaok
        "dnsmasq requires NET_ADMIN capabilities to run correctly. Start it with something like docker run -p 53:53/tcp -p 53:53/udp --cap-add=NET_ADMIN 4km3/dnsmasq:2.75."
      • 2021-05-27 14732, 2021

      • ruaok
        possible yes. how does one check that?
      • 2021-05-27 14710, 2021

      • ruaok
        there don't see to be any updates to this container and we pass NET_ADMIN.
      • 2021-05-27 14724, 2021

      • alastairp
        `sudo systemctl status systemd-resolved` shows that it's definitely running, but it's uncear if it's listening on any ports that are important to us
      • 2021-05-27 14709, 2021

      • lucifer