lucifer: my listens are not submitting through spotify, it shows that im listening to them but none of them were submitted. Listens from lastfm submit fine.
Hm, I've had two 503 "Cannot submit listens to queue, please try again later." errors about 15 minutes ago, rest of my listens seem to be accepted by the server
I understand the openAI and Microsoft deal and how it relates to us. Bing uses MB data. Future AI are likely going to be trained on the data that is in bing, ergo MB. Imagine chatgpt fully speaking mbids.
mayhem goes to look at telegram
Input validation is a real pita.
alastairp
yeah. looks like it's the length of the recording name which is causing a problem on a messybrainz index
stopping timescale writer; I'm going to get a message off the queue manually and see if it's the bad one
lucifer
alastairp: mayhem: fixed. queue should clear in ~5 mins.
alastairp
lucifer: oh, thanks :)
lucifer: how did you do it? (should I restart ts writer?)
lucifer
alastairp: had removed the bad listen from queue. and it autodelivered remaining messages. queue fell from 12k to 7k so far.
yup, restarting sound good
alastairp
lucifer: how did you remove it?
lucifer
rmq admin dashboard
alastairp
admin interface -> get message -> reject requeue false?
lucifer
first requeue to verify then, automatic ack to remove.
alastairp
got it. I'm writing this down, because I've only done it once before when we were in a similar situateion
lucifer: that listen is back, ts writer logs is showing the same error
lucifer
hmm, let me check. i am still logged in
queue is going down though. maybe those are old logs?
jasje has quit
alastairp
I set --tail 100, so I didn't expect so. but let me look at the queue too
lucifer
(stop/restart doesn't clear container logs)
in normal functioning, it logs nothing so possible old logs.
alastairp
nope, it's definitely crashing again. the timestamp for "Timescale Writer started" is increasing
lets wait for the queue to clear or stop going down i guess
alastairp
sure
queue is empty, so I'm unsure about this message now
lucifer
alastairp: there was another bad listen, removed that as well.
alastairp
ah,. that's why it's empty :)
lucifer
can confirm there were 2 different listens because listened_at varies.
alastairp
lucifer: so I guess I see 2 possible fixes here: 1 is to limit the length of string columns that have an index. 2 is to try and catch this kind of exception and mark incoming listens as failed somehow?
lucifer
i am not sure why it kept on crashing on the first one but skipped the second one and processed all other listens fine.
alastairp: yeah, plan was to send such errorneous listens to a reject queue.
alastairp
right
lucifer
the issue i see with that we want to do batched inserts, so we would to reject entire batch,
alastairp
yep. one option is to try a batch, if it fails fall back to one at a time, then reject the specific one causing the issue
