_lucifer: I think the Spotify player is broken for everyone on LB right now, so if I manage to fix it by the end of the day here I'd love to see it in the release. That being said, we can also do another release in the coming days, I don't want to hold you back
2021-04-14 10415, 2021
Mr_Monkey
Let me have a quick look at the open PRs
2021-04-14 10412, 2021
_lucifer
Mr_Monkey: oh! what's the error with the spotify player? wondering if its related to the spotify reader fixes we deployed some time ago.
2021-04-14 10434, 2021
Mr_Monkey
I honestly can't tell, but it could just as well be front-end only.
2021-04-14 10424, 2021
Mr_Monkey
In any case, trying to play a track gives 404 errors (searching for a track works fine), and after a bunch of retires I get a 429 rate limit error.
2021-04-14 10425, 2021
_lucifer
okay, let me see if i can get the sentry frontend PR up to speed. that should probably aid in debugging the issue.
That's all the front-end PRs I see that are ready at the moment.
2021-04-14 10418, 2021
BrainzGit
[listenbrainz-server] alastair opened pull request #1389 (master…listen-null-validation): When writing a batch of listens, skip any which have null characters https://github.com/metabrainz/listenbrainz-server…
2021-04-14 10421, 2021
alastairp
ruaok: ^
2021-04-14 10411, 2021
ruaok
alastairp: do we have a ticket to remind ourselves to remove this code when the front end checking has been improved?
2021-04-14 10443, 2021
alastairp
I'll add one for the checking, but I'm not sure we should remove this... in case something manages to get into the queue in the future
2021-04-14 10420, 2021
_lucifer
regarding that, is it possible to avoid checking every listen. because we wanted to remove the extra serialising/deserialising steps and this feels like adding one.
2021-04-14 10447, 2021
_lucifer
but i have not profiled anything so i might be wrong about the performance anyways.
2021-04-14 10415, 2021
ruaok
it is adding an extra step. but this is a temporary measure so we don't lose any of the queued listens.
2021-04-14 10422, 2021
ruaok
there are 200k stuck right now.
2021-04-14 10442, 2021
Mr_Monkey
ruaok: Anything I need to do on the new be-gone branch? I'm getting this:
2021-04-14 10443, 2021
Mr_Monkey
ImportError: cannot import name 'SECONDS_IN_TIME_RANGE' from 'listenbrainz.listenstore.timescale_listenstore'
2021-04-14 10445, 2021
_lucifer
oh ok, makes sense.
2021-04-14 10408, 2021
ruaok
Mr_Monkey: the branch is not fully baked yet.
2021-04-14 10418, 2021
ruaok
where is that coming from?
2021-04-14 10426, 2021
ruaok
that needs to be nuked too. api.py ?
2021-04-14 10430, 2021
alastairp
_lucifer: at the moment it's checking a bunch of values, but there's no extra serialisation/deserialisation step here
2021-04-14 10438, 2021
Mr_Monkey
listenbrainz/listenstore/timescale_listenstore.py
2021-04-14 10445, 2021
alastairp
but we may be able to find a better place for this check
2021-04-14 10448, 2021
alastairp
I'll open a ticket
2021-04-14 10452, 2021
Mr_Monkey
Err, yes listenbrainz/webserver/views/api.py
2021-04-14 10410, 2021
alastairp
ruaok: unit tests passed on CI
2021-04-14 10415, 2021
alastairp
should we merge and release?
2021-04-14 10435, 2021
ruaok
just finished reading the PR. looks good.
2021-04-14 10442, 2021
ruaok
shall I or do you want to do it?
2021-04-14 10458, 2021
alastairp
just done it
2021-04-14 10459, 2021
BrainzGit
[listenbrainz-server] alastair merged pull request #1389 (master…listen-null-validation): When writing a batch of listens, skip any which have null characters https://github.com/metabrainz/listenbrainz-server…
2021-04-14 10410, 2021
_lucifer
i can do it. as i have to release the db changes as well.
2021-04-14 10428, 2021
ruaok
is that something you're doing right this sec, _lucifer ?
2021-04-14 10436, 2021
ruaok
I really want to get the timescale writer going again
2021-04-14 10402, 2021
_lucifer
yup, just fixing the indent comment and going to do the release.
2021-04-14 10421, 2021
ruaok
ok, ping me when you start the release so I can keep an eye on things.
ah yes. the drop would be a near vertical line if it was just a single insert or two.
2021-04-14 10445, 2021
alastairp
which is the unique queue? I see beta_unique but don't know what the prod one is
2021-04-14 10400, 2021
ruaok
just unique
2021-04-14 10425, 2021
_lucifer
why did this happen suddenly though? i mean why did the listens began to pass through messybrainz without erring out?
2021-04-14 10447, 2021
alastairp
_lucifer: because there would have been a listen that managed to match a hash in messybrainz
2021-04-14 10455, 2021
alastairp
so therefore it didn't do an insert
2021-04-14 10409, 2021
alastairp
but it still had a null value in additioanl_data
2021-04-14 10413, 2021
alastairp
which was passed through into the queue
2021-04-14 10422, 2021
_lucifer
oh makes sense.
2021-04-14 10451, 2021
ruaok
nearly time for some noms, no, alastairp, Mr_Monkey ?
2021-04-14 10456, 2021
_lucifer
indeed, very lucky that messybrainz acted as a filter.
2021-04-14 10406, 2021
Mr_Monkey
Afirm
2021-04-14 10406, 2021
alastairp
and so this is why we should have done an explicit check in the body of the listen
2021-04-14 10413, 2021
alastairp
instead of relying on the database error
2021-04-14 10426, 2021
alastairp
I'm opening a ticket for that now
2021-04-14 10447, 2021
_lucifer
makes sense.
2021-04-14 10453, 2021
_lucifer
safe to upgrade cron, ruaok ?
2021-04-14 10459, 2021
ruaok
yes
2021-04-14 10420, 2021
alastairp
ruaok: do you know that just by intuition (you know more or less when the tasks run?)
2021-04-14 10451, 2021
ruaok
we purposely moved the schedule so that automated tasks run during EU night and the cluster is more or less ilde EU day.
2021-04-14 10455, 2021
alastairp
it'd be nice to have a single command that we can run to either 1) check if any processes are actively running in the cron container, or 2) show a calendar of upcoming events so that we can look at them
2021-04-14 10406, 2021
alastairp
yeah, I remember you saying that
2021-04-14 10457, 2021
alastairp
a quick hacky way of doing it could be to list processes inside the container and look for a python process
2021-04-14 10406, 2021
ruaok
oh wait, the PR I was waiting for wasn't even reviewed yet. boo.
that is the code with the updated listen fetch query that uses the listen_count cont agg.
2021-04-14 10411, 2021
alastairp
postgres' recommendation has always been that when upgrading between major versions, a dump and reload is always best (in which case it's going to make a new index anyway)
2021-04-14 10426, 2021
alastairp
in some cases you can get away with running a new version on the old database, which it seems like they may have done here
2021-04-14 10452, 2021
ruaok
(still untested)
2021-04-14 10457, 2021
_lucifer
yeah, seems that way.
2021-04-14 10415, 2021
alastairp
ruaok: right, I see. neat
2021-04-14 10440, 2021
zas
_lucifer: I installed docker on michael, the installation uses ufw-docker (see https://github.com/chaifeng/ufw-docker) you'll need to use ufw-docker command to allow ports
