15:07 PM
bitmap
ok
15:08 PM
sigh, this update_mb_dbdefs_for_hosts.sh script is always broken
15:08 PM
or I don't understand how it works
15:09 PM
akshaaatt
mayhem do let me know if you are able to submit listens. Keep in mind that we do need to improve the submitted data from the app though.
15:11 PM
mayhem
I'm not.
15:11 PM
the last two track changes have not been recorded.
15:11 PM
akshaaatt
LB seems to be down
15:11 PM
That could be an issue?
15:11 PM
mayhem
🤦
15:11 PM
akshaaatt
Lol
15:12 PM
mayhem
one shall not play with these toys during downtime, yes.
15:12 PM
sorry akshaaatt .
15:12 PM
akshaaatt
It’s fine 😂😂
15:12 PM
bitmap
yvanzo: update_mb_dbdefs_for_hosts.sh is completely broken for me, it just reports "No matching containers on aphex." for every machine
15:12 PM
akshaaatt
We should maintain a local fb though.
15:13 PM
Thanks for testing mayhem. We got a new ticket because of it
15:13 PM
Db*
15:13 PM
yvanzo
bitmap: should I try it
15:13 PM
bitmap
yvanzo: yeah
15:13 PM
mayhem
lucifer: looking at the pop data now. what is the difference between those two queries?
15:13 PM
bitmap
yvanzo: I'm running it without any arguments because the args in the google doc didn't work
15:13 PM
mayhem
akshaaatt: I'll be doogfooding the LB app from now on. if it breaks, you'll hear from me, I promise. :)
15:13 PM
akshaaatt
Awesome! ⚡️⚡️
15:14 PM
bitmap
yvanzo: you have to edit the file to remove "sleep 30" first
15:15 PM
jasje
phew i was worried wtf did i do wrong glad the server is down (i mean not really)
15:15 PM
mayhem
15:15 PM
lucifer
mayhem: first one is popularity by total listen count, second by unique listener count.
15:15 PM
mayhem
should we reach out and ask about LB support?
15:15 PM
yvanzo
bitmap: it's skipped aphex and aretha as already up-to-date but it's now working for others, using arguments from the doc
15:16 PM
monkey
I saw that one, thought the same thing
15:16 PM
jasje
lucifer: understood!
15:16 PM
bitmap
yvanzo: OK, I guess it's just borked on macOS
15:16 PM
mayhem
lucifer: hard to evaluate like this, but it seems ok.
15:16 PM
bitmap
I merged the gsed PR already
15:16 PM
so must be something else
15:16 PM
mayhem
would be good to put up on DSH
15:16 PM
lucifer
mayhem: there is artist, release, recording data available as well
15:16 PM
monkey
But it's limited too, same as other ones I've seen:
15:16 PM
>automatically detect currently playing music from Apple Music or Spotify
15:16 PM
yvanzo
bitmap: it stopped on patton, now restarting
15:17 PM
mayhem
packed with radiohead, and RHCP, the data is likely ok, lucifer
15:17 PM
lucifer
i didn't put it on dsh because not sure what params to accept.
15:17 PM
mayhem
what params?
15:17 PM
jasje
mayhem: to record any app in the listening apps list, just play one or two songs.
15:17 PM
mayhem
monkey: better than nada.
15:17 PM
yvanzo
bitmap: did you update your local working copy of server conf?
15:17 PM
jasje
force shutting isn’t necessary
15:17 PM
lucifer
top recordings is a mbid with two listen counts.
15:17 PM
bitmap
yvanzo: yeah
15:18 PM
I'm on 6132bf00b231308de41be93da2364c7e5dd9e690
15:18 PM
mayhem
jasje: I think the problem is with LB being down. but likely it works now. I'll check once LB is up.
15:18 PM
jasje
doesn’t matter if LB is up or no
15:18 PM
yvanzo
bitmap: Sometimes it's also stopping with "Host key verification failed. fatal: Could not read from remote repository."
15:18 PM
lucifer
15:18 PM
jasje
listening apps will show regardless
15:18 PM
lucifer
all of this is available.
15:19 PM
yvanzo
bitmap: done
15:19 PM
bitmap
thanks
15:19 PM
mayhem
jasje: yes, I can see the LB app knowing the current playing now track.
15:19 PM
akshaaatt
jasje those are showing for him.
15:19 PM
yvanzo
bitmap: was json-dump not included at purpose?
15:19 PM
akshaaatt
The issue is only with the submissions currently
15:20 PM
jasje
oh then its okay. looks like i read an older convo
15:20 PM
mayhem
lucifer: why dont we have make one DSH query and then have an extra param that tells which table to use. then list the available tables in the desc for the query.
15:20 PM
bitmap
services should be coming back now
15:20 PM
mayhem
(and please add the available tables for recording and artists similarity while you're at it)
15:21 PM
oh, LB must be back. listen arrived. :)
15:21 PM
bitmap
yvanzo: that looks like an oversight
15:21 PM
yvanzo
okay, updating it now
15:21 PM
bitmap
apparently if you run the script without any arguments it should update all containers on all hosts?
15:21 PM
akshaaatt high fives mayhem
15:21 PM
mayhem
jasje: akshaaatt: all sorted, thank you!
15:21 PM
yvanzo
bitmap: yes
15:22 PM
done
15:22 PM
jasje
yay!
15:22 PM
bitmap
yvanzo: ty
15:22 PM
I'll restore standby service on pink now
15:24 PM
yvanzo: can you pull docker-server-configs on pink? now that's not working lol
15:24 PM
yvanzo
ok
15:24 PM
bitmap
I tried ssh agent forwarding
15:27 PM
yvanzo: I got it working with GIT_SSH_COMMAND='ssh -i ~/.ssh/docker_server_configs_key' git pull origin master
15:27 PM
yvanzo
👍
15:28 PM
I'll try to fix the config still
15:28 PM
lucifer
mayhem: i am not sure that will work.
15:29 PM
15:29 PM
how do you want to query this table for instance
15:30 PM
mayhem
ah, yes. and each table is going to need different params.
15:30 PM
if you can load these datasets on wolf, I can add the DSH and test.
15:30 PM
bitmap
yvanzo: I guess we have a separate deploy key on pink but not on floyd
15:31 PM
lucifer
sure i'll copy those to wolf.
15:31 PM
yvanzo
I checked deploy keys in the repo settings
15:31 PM
mayhem
thx
15:32 PM
yvanzo
bitmap: it seems that all websites are back already, making the announcement
15:33 PM
bitmap
yvanzo: yup
15:33 PM
ty
15:33 PM
monkey
Smooth! Nicely done team
15:36 PM
mayhem
monkey: neptunes creators have been asked to add LB support, monkey
15:36 PM
bitmap
only took an extra 5 mins this time, not terrible
15:36 PM
monkey
Thx !
15:36 PM
mayhem
👏
15:36 PM
bitmap
mayhem: monkey: lucifer: you can restart any cron jobs that were stopped
15:37 PM
mayhem leaves it to lucifer
15:37 PM
yvanzo
monkey: BB is back
15:37 PM
monkey
Yep, I checked as well, thanks :)
15:38 PM
bitmap
pink standby service is back, barman is running smoothly
15:38 PM
yvanzo
(I tweeted about it for MeB and MB only)
15:38 PM
bitmap
I restarted cron in json dump, search indexes dump, and production cron containers
15:38 PM
lucifer
done
15:38 PM
(cron jobs)
15:42 PM
rdswift has quit
15:42 PM
bitmap
thanks all
15:43 PM
working on improving the pg cluster so that switchovers require less (possibly zero) downtime, still in the testing phase though
15:46 PM
rdswift joined the channel
15:47 PM
yvanzo
!m bitmap
15:47 PM
BrainzBot
You're doing good work, bitmap!
15:48 PM
mayhem
monkey: rescrobbler scrobbles everything, even if it is playing on mobile.
15:48 PM
but only, of course, if the desktop machine is not sleeping.
15:48 PM
yvanzo
bitmap: Found the problem with pink: the key was named docker_server_configs_key instead of the default id_rsa.
15:48 PM
mayhem
which leads to .. dups. sigh.
15:48 PM
monkey
Yep, that's what I was trying to point out
15:49 PM
mayhem
I was trying to ask you exactly this question. lol
15:49 PM
monkey
If you have spotify open on desktop with rescrobbler and also on mobile with LB app, you'll get two listens.
15:49 PM
mayhem
indeed.
15:49 PM
monkey
There's no real way to figure out which one is 'correct"
15:49 PM
mayhem
and the listens are not off by some seconds, but by minutes.
15:50 PM
monkey
That's the issue indeed where deduplication is concerned
15:50 PM
mayhem
15:50 PM
yvanzo
bitmap: all other repositories have IdendityFile specified in ~/.ssh/config but docker-server-configs
15:50 PM
monkey
That is one of the reasons why I wouldn't suggest deprecating the spotify importer
15:50 PM
mayhem
I see that the desktop scrobbler and LB app have quite different timestamps as to when the track started playing.
15:50 PM
yvanzo
zas: is ~root/.ssh/config managed using ansible?
15:51 PM
mayhem
monkey: understood, but I think there is a discrepancy somewhere and when resolved, might fix this issue.
15:51 PM
jasje
mayhem: we record the initial play button hit timestamp
15:51 PM
zas
yvanzo: nope
15:51 PM
mayhem
huh. it seems that rescrobbler does this too.
15:51 PM
ok, I'll dig into this tomorrow, but outta time for today.
15:52 PM
akshaaatt
Right
15:52 PM
monkey
Fix how mayhem? Because each scrobbling app might have a different mechanism to decide what the timestamp is
15:52 PM
yvanzo
zas: is it fine to keep using id_rsa identity file for cloning docker-server-config repo on hosts?
15:52 PM
bitmap
yvanzo: I think that may have been me
15:53 PM
mayhem
monkey: right now the duplicates seem to be off by the duration of the track. something is not right.
15:53 PM
if both were working right, the differences in timestamps should be small.
15:53 PM
bitmap
at some point I wanted to separate the keys used for sshing between servers and the deploy keys for our private repos
15:53 PM
mayhem
seconds, not minutes.
15:53 PM
monkey
My point is that "working right" is vague :) Looks like rescrobbler chooses the end of the track as the listened_at timestamp maybe?
15:53 PM
zas
yvanzo: well, it would be better to use deploy keys I guess
15:53 PM
yvanzo
bitmap: which is great, and we should probably do for docker-server-configs repo too.
15:54 PM
zas
15:54 PM
yvanzo
zas: it's a deploy key, just the default one
15:54 PM
jasje
mayhem: judging by the track_duration /2 and timestamp diff