#metabrainz

/

      • jasondk_
        I received my GSoC acceptance, thank you metabrainz team! I'm really excited to work on listenbrainz this summer (and hopefully beyond), thanks for the great opportunity :)
      • RetroPunk has quit
      • RetroPunk joined the channel
      • thomasross has quit
      • agatzk joined the channel
      • yyoung
        I have received my GSoc acceptance email, thanks to Metabrainz team. It is my honor to be part of this community, and I'm glad to work with my mentor yvanzo, hope we'll have a good time coding this summer :)
      • D4RK-PH0ENiX has quit
      • _lucifer
        akshaaatt[m], we can continue most of the talks and work in the channel. but feel free to PM me anytime you need. regarding a call, let's see what others decide and we can do accordingly and yes I am from India as well.
      • !m bitmap, yvanzo and reosarevok
      • BrainzBot
        You're doing good work, bitmap, yvanzo and reosarevok!
      • _lucifer
        congrats jasondk_ and yyoung! :D
      • ruaok, just checked the logs on leader, full import succeeded.
      • D4RK-PH0ENiX joined the channel
      • akshaaatt[m]
        <_lucifer "akshaaatt, we can continue most "> Sounds great! :)
      • Kevin12 joined the channel
      • Kevin12 has quit
      • akashgp09 joined the channel
      • mglubb joined the channel
      • sumedh joined the channel
      • akashgp09 has quit
      • ruaok
        moooin!
      • !m _lucifer
      • BrainzBot
        You're doing good work, _lucifer!
      • ruaok
        have you checked to see if any of the stats ran? if not, we should request a full run. I plan to do nothing with the cluster today, so lets run everything.
      • _lucifer
        yes some of the requests had run when i last checked.
      • sure lets run all.
      • mglubb has quit
      • ruaok
        _lucifer: ohhh sorry. I didn't realize that leader was actually use in the spark configuration. that was never the intent; had I known I wouldn't have changed it on you. sorry!
      • (it was always meant to be a shortcut to michael, nothing more)
      • _lucifer
        ruaok, ah no its not used in spark configuration. you are talking about docker-server-configs PR right?
      • ruaok
        yes. either one wasn't intended to be using it, but I never communicated that to you, sorry.
      • _lucifer
        no worries :). we actually don't use that in the spark setup. we have an entry in /etc/hosts that points the internal IP of michael to leader host. https://github.com/metabrainz/syswiki/pull/39/f...
      • the same things keep popping up in the setup at different places so its a bit confusing. that's why i am putting together a document with general notes and observations on the setup.
      • ruaok
        very good
      • shivam-kapila
        Congrats GSoC students.
      • shivam-kapila goes to see who all are accepted
      • sumedh has quit
      • Congrats mentor _lucifer :p
      • _lucifer
        :)
      • shivam-kapila
        Ah the propsals sound really good. Exciting things coming. All the best fellas
      • _lucifer
      • there's no entry for similar users and recommendations. ( for similar user only dataframes are being requested not the actual users)
      • ruaok
      • I moved them. they felt more "dumpy" than staty
      • sumedh joined the channel
      • I think we should compact both of these and move them into one crontab -- having two is harder to manage.
      • _lucifer
        oh! makes sense.
      • i'll add a new command to request all kinds of stats at once.
      • ruaok
        makes sense.
      • thanks
      • ruaok goes to look after diskspace on lemmy
      • _lucifer
        all stats, similar users, recommendations requested.
      • ruaok
        sweet.
      • _lucifer
        are we going to upgrade lemmy today?
      • ruaok
        54% disk space used.
      • we upgrade when hetzner says we upgrade. there is no ETA, no insight, just beer lacking bavarians being grumpy.
      • _lucifer
        :(
      • ruaok
        does anyone of the gsoc mentors feel like writing a "welcome gsoc students" email?
      • I think tomorrow might need to be PR merge day...
      • _lucifer
        alastairp, i think that command issue might be coming from weird interactions between the default docker builder and buildkit. I just noticed that docker's default builder fills the `ContainerConfig` section seen in docker inspect but BuildKit doesn't.
      • ruaok
        questionable bug ticket name of the day: "Further dump hardening"
      • _lucifer
        while we haven't started using BuildKit in actions, I have locally started using it so it might be possible that one of my pushed images was built using BuildKit but actions' is using default docker builder so the error happened.
      • lol 🤣
      • ruaok
        _lucifer: the dumps logs have gone stale since may 11.
      • _lucifer
        how so?
      • ruaok
        `2021-05-12 13:00:05,335 INFO Created channel=1`
      • is the last entry.
      • nor am I getting mails about dumps being out of date.
      • if you get a minute to have a look, please do. if not, I'll pick it up this afternoon.
      • but I gotta go now and make my way towards getting a vaccine...
      • shivam-kapila
        Niceee
      • _lucifer
        noice :D
      • i'll look into dumps issue.
      • zas
        yvanzo, bitmap : we just got an alert for high number of WALs
      • _lucifer
        ruaok, we have had two incremental dumps since 12th, one on 13 and one on 16, it looks like both were manually triggered. was that so?
      • ruaok
        that sounds about right.
      • well, not the 16th, no.
      • _lucifer
        my theory was that dump-crontab is not being run due to some reason. but if crontab did run on 16th then that's wrong.
      • ruaok
        well, we've been starting/restarting so many containers, it is hard to know.
      • Mr_Monkey
        ruaok: I'm taking a stab at writing a welcome email. Will send you a draft
      • ruaok
        Mr_Monkey: thank you!
      • _lucifer
        ruaok: i am not familiar with how cron works but a lot of this coming up the fact each crontab is user specific so adding entries from different users to same crontab might be causing issues.
      • alastairp
        morning
      • _lucifer: we don't (or at least we shouldn't) install things into user-specific crontabs. we use global files
      • _lucifer
        ah ok!
      • also, i was looking for logs from refresh aggregates job to see if that ran but it seems that job doesn't log anything.
      • can we check somehow if that job ran?
      • alastairp
        in cron?
      • _lucifer
      • yes
      • alastairp
        there are a few ways - one is that cron should output to syslog when it starts something, but that'll go away when we delete a container
      • in the case of these items, we should definitely add >> /logs/something.log to all jobs
      • so that we have a record of when things run
      • ruaok
        that was the whole idea, but our logs are empty now. :(
      • _lucifer
        this job runs everyday so it should be in todays logs right?
      • ruaok
        it should
      • alastairp
        _lucifer: yeah
      • _lucifer
        its isn't
      • alastairp
        you could docker exec and then check /var/log/syslog
      • _lucifer
        so yeah dumps crontab is not running
      • alastairp, yes right the issue we are trying to debug is that the dumps log file has not been updated since May 12.
      • ruaok
        the dump log volume is correctly attached to the cron container.
      • alastairp
        oh yeah - one sec
      • that crontab file is invalid
      • 0 5 * * * /usr/local/bin/python /code/listenbrainz/manage.py refresh_continuous_aggregates
      • needs a username in column 6
      • ruaok
        d'oh
      • alastairp
        _lucifer's comment about a cron file validator comes back :)
      • ruaok
        oh. I see. the two cron tabs are managed differently.
      • stats does not specify a user.
      • alastairp
        mmm, it doesn't?
      • _lucifer
        :)
      • ruaok
        this is why I want to go to one crontab, not two.
      • alastairp
        it should - I thought that I changed them all to be consistent
      • ruaok
        too much confusion having them in two.
      • alastairp
        ruaok: `listenbrainz_stats_cron` is the username
      • ruaok
        yeah ok. validator.
      • ruaok sighs
      • ruaok has been too stressed last week
      • making waay too many mistakes.
      • thanks freso, token revoked.
      • zas
        I'll proceed to herb/kiki upgrades/reboot
      • alastairp
        _lucifer: let's open a PR for 1) adding this username to the dumps crontab, 2) merging all stats commands into a single item (which we can then move into a single crontab with dumps)
      • _lucifer
        sure, on it.
      • alastairp
        I think you understand the spark manage stuff better than me
      • ruaok
        thanks alastairp, _lucifer
      • alastairp
        do you want to make a new command? I'll open PR with the username
      • ruaok
        maybe I'll just take the rest of the day off and... chill a bit.
      • alastairp
        especially after a jab, I'd recommend it
      • _lucifer
        alastairp, i was thinking to add a new command and then use context.invoke to invoke all the other ones.
      • ruaok
        well, ping me if ya really need me. otherwise, I'll be fucking off. :)
      • alastairp
        _lucifer: sounds fine
      • ruaok: see you tomorrow, then
      • _lucifer
        have a nice day, ruaok :)
      • alastairp
        fingers crossed lemmy doesn't fall over, lol
      • _lucifer
        lol
      • alastairp
        ruaok: should we run continuous agg update manually?
      • ruaok
        sure, do it in tmux. takes a while.
      • alastairp
        will do.
      • _lucifer
        there's also a logging error in the command to refresh aggregates. will fix in the PR.
      • alastairp
        ok
      • zas
        switching to herb
      • Freso
        yvanzo: Are you going to make a PR of https://github.com/yvanzo/mbdata/tree/schema-ch... ? :)
      • zas
        rebooting kiki
      • atj
        did you figure out what the "|1" was about?
      • _lucifer
        it seems that |1 denoted the build arg's in the dockerfile, but no idea how that ended up in the CMD
      • unable to reproduce the issue currently.
      • alastairp
        yeah, it seems like this is something that docker does to keep track of how it built an image, but somehow it jumped from a RUN to a CMD, which is weird
      • _lucifer: I'm running refresh_continuous_aggregates in a shell in cron now.