#metabrainz

I received my GSoC acceptance, thank you metabrainz team! I'm really excited to work on listenbrainz this summer (and hopefully beyond), thanks for the great opportunity :)

akshaaatt[m], we can continue most of the talks and work in the channel. but feel free to PM me anytime you need. regarding a call, let's see what others decide and we can do accordingly and yes I am from India as well.

!m bitmap, yvanzo and reosarevok

You're doing good work, bitmap, yvanzo and reosarevok!

congrats jasondk_ and yyoung! :D

ruaok, just checked the logs on leader, full import succeeded.

<_lucifer "akshaaatt, we can continue most "> Sounds great! :)

moooin!

!m _lucifer

You're doing good work, _lucifer!

have you checked to see if any of the stats ran? if not, we should request a full run. I plan to do nothing with the cluster today, so lets run everything.

yes some of the requests had run when i last checked.

sure lets run all.

_lucifer: ohhh sorry. I didn't realize that leader was actually use in the spark configuration. that was never the intent; had I known I wouldn't have changed it on you. sorry!

(it was always meant to be a shortcut to michael, nothing more)

ruaok, ah no its not used in spark configuration. you are talking about docker-server-configs PR right?

yes. either one wasn't intended to be using it, but I never communicated that to you, sorry.

no worries :). we actually don't use that in the spark setup. we have an entry in /etc/hosts that points the internal IP of michael to leader host. https://github.com/metabrainz/syswiki/pull/39/f...

the same things keep popping up in the setup at different places so its a bit confusing. that's why i am putting together a document with general notes and observations on the setup.

very good

Congrats GSoC students.

Congrats mentor _lucifer :p

:)

Ah the propsals sound really good. Exciting things coming. All the best fellas

ruaok: https://github.com/metabrainz/listenbrainz-serv... is missing a few entries.

there's no entry for similar users and recommendations. ( for similar user only dataframes are being requested not the actual users)

I moved them. they felt more "dumpy" than staty

I think we should compact both of these and move them into one crontab -- having two is harder to manage.

oh! makes sense.

i'll add a new command to request all kinds of stats at once.

makes sense.

thanks

all stats, similar users, recommendations requested.

sweet.

are we going to upgrade lemmy today?

54% disk space used.

we upgrade when hetzner says we upgrade. there is no ETA, no insight, just beer lacking bavarians being grumpy.

:(

does anyone of the gsoc mentors feel like writing a "welcome gsoc students" email?

I think tomorrow might need to be PR merge day...

alastairp, i think that command issue might be coming from weird interactions between the default docker builder and buildkit. I just noticed that docker's default builder fills the `ContainerConfig` section seen in docker inspect but BuildKit doesn't.

questionable bug ticket name of the day: "Further dump hardening"

while we haven't started using BuildKit in actions, I have locally started using it so it might be possible that one of my pushed images was built using BuildKit but actions' is using default docker builder so the error happened.

lol 🤣

_lucifer: the dumps logs have gone stale since may 11.

how so?

`2021-05-12 13:00:05,335 INFO Created channel=1`

is the last entry.

nor am I getting mails about dumps being out of date.

if you get a minute to have a look, please do. if not, I'll pick it up this afternoon.

but I gotta go now and make my way towards getting a vaccine...

Niceee

noice :D

i'll look into dumps issue.

yvanzo, bitmap : we just got an alert for high number of WALs

ruaok, we have had two incremental dumps since 12th, one on 13 and one on 16, it looks like both were manually triggered. was that so?

that sounds about right.

well, not the 16th, no.

my theory was that dump-crontab is not being run due to some reason. but if crontab did run on 16th then that's wrong.

well, we've been starting/restarting so many containers, it is hard to know.

ruaok: I'm taking a stab at writing a welcome email. Will send you a draft

Mr_Monkey: thank you!

ruaok: i am not familiar with how cron works but a lot of this coming up the fact each crontab is user specific so adding entries from different users to same crontab might be causing issues.

morning

_lucifer: we don't (or at least we shouldn't) install things into user-specific crontabs. we use global files

ah ok!

also, i was looking for logs from refresh aggregates job to see if that ran but it seems that job doesn't log anything.

can we check somehow if that job ran?

in cron?

yes

there are a few ways - one is that cron should output to syslog when it starts something, but that'll go away when we delete a container

in the case of these items, we should definitely add >> /logs/something.log to all jobs

so that we have a record of when things run

that was the whole idea, but our logs are empty now. :(

this job runs everyday so it should be in todays logs right?

it should

_lucifer: yeah

its isn't

you could docker exec and then check /var/log/syslog

so yeah dumps crontab is not running

alastairp, yes right the issue we are trying to debug is that the dumps log file has not been updated since May 12.

the dump log volume is correctly attached to the cron container.

oh yeah - one sec

that crontab file is invalid

0 5 * * * /usr/local/bin/python /code/listenbrainz/manage.py refresh_continuous_aggregates

needs a username in column 6

d'oh

_lucifer's comment about a cron file validator comes back :)

oh. I see. the two cron tabs are managed differently.

stats does not specify a user.

mmm, it doesn't?

:)

this is why I want to go to one crontab, not two.

it should - I thought that I changed them all to be consistent

too much confusion having them in two.

ruaok: `listenbrainz_stats_cron` is the username

yeah ok. validator.

making waay too many mistakes.

thanks freso, token revoked.

I'll proceed to herb/kiki upgrades/reboot

_lucifer: let's open a PR for 1) adding this username to the dumps crontab, 2) merging all stats commands into a single item (which we can then move into a single crontab with dumps)

sure, on it.

I think you understand the spark manage stuff better than me

thanks alastairp, _lucifer

do you want to make a new command? I'll open PR with the username

maybe I'll just take the rest of the day off and... chill a bit.

especially after a jab, I'd recommend it

alastairp, i was thinking to add a new command and then use context.invoke to invoke all the other ones.

well, ping me if ya really need me. otherwise, I'll be fucking off. :)

_lucifer: sounds fine

ruaok: see you tomorrow, then

have a nice day, ruaok :)

fingers crossed lemmy doesn't fall over, lol

lol

ruaok: should we run continuous agg update manually?

sure, do it in tmux. takes a while.

will do.

there's also a logging error in the command to refresh aggregates. will fix in the PR.

ok

switching to herb

yvanzo: Are you going to make a PR of https://github.com/yvanzo/mbdata/tree/schema-ch... ? :)

rebooting kiki

did you figure out what the "|1" was about?

it seems that |1 denoted the build arg's in the dockerfile, but no idea how that ended up in the CMD

unable to reproduce the issue currently.

yeah, it seems like this is something that docker does to keep track of how it built an image, but somehow it jumped from a RUN to a CMD, which is weird

_lucifer: I'm running refresh_continuous_aggregates in a shell in cron now.