#metabrainz

that is exactly what I figured out yesterday.

let me go read the follow code just to make sure

yeah, that's what i was gonna say

and the vlc listening works really well honestly

<3 livegrep

it looks like you created a separate exchange too.

"playing_now_exchange": "playing_now",

"playing_now_queue": "playing_now",

i have no remembrance at all of how the follow thing worked

docker-server-configs/consul/LB

oh

so

the playing_now queue is only for listens that people sent in as 'now_playing'

yeah, I just remembered that.

k, i'm gonna get food and then look into emails for the stats cron job

k

the observability mailer may be spammy for a while.

oh no.

amCap1712: I'm finished testing for now. tell me tomorrow/osme other tiem if yo uhave spesific test cases yo uwant me to test

iliekcomputers: self.create_and_bind_exchange_and_queue(channel, current_app.config['UNIQUE_EXCHANGE'], current_app.config['FOLLOW_LIST_QUEUE'])

this is the crux of it. makes sense.

yep

ZaphodBeeblebrox: pausing/resuming, stopping midway, repeat, playlist. thats all. if all of that is tested i don't have anything more in mind

oh, another correction

1. start shadowing the UNIQUE queue to insert all listens (starting timestamp x)

iliekcomputers: ruaok: do you think there could be some other edge cases which might require to tested separately?

2. directly import from influx all listens inserted to influx before timestamp x

I think i did all these. i guess palylist. but i'm tired and stuff i need ot eat too

ZaphodBeeblebrox: tommorrow lets test on windows then

yeah sure. thanks for the help

i'll try to move the windows machine . alright!

amCap1712: the answer is yes. but, what are we talking about?

hey no problem. this is awesome!!

ruaok: edge cases to test the plugin for. like one might be to stop the plugin before it is half finished

two different things beeing listened too at the same time (diff machines, diff programs, etc)

and should not be submitted

i cna test this tomorrow! windows and mac both play!

amCap1712: if the track played 30 seconds or more, it is considered a play.

so. if playing , say 40 seocnds thne double clicknig again on it

i cna test that

ruaok: even if it less than half the duration?

30 seconds is just the standard in the industry.

which seems BS to me, but...

that just a constant i'll have to modify.

then for now, keep it simple and set the constant to 30.

but i think it gives the idea of what i am trying to convey by different test cases

i have songs that are >30 secs whixh i wish would listenz

yes, but I can't really think of any off the top of my head.

alastairp is good at these things.

(ok so I made up that listening eg "scrobbling" is called "listenz" .. i uges it's dumb)

i'll post on the discourse for asking others to test.

https://beta.musicbrainz.org/recording/734bfc87... <-- shorther thna 30 seconds

once we are satisfied with the testing, i can send the patch upstream for merging

yes

iliekcomputers: for when you get back: https://gist.github.com/mayhem/fb944b7bde300643...

i tend towards the simple one

considering it's writing stuff at 20k/s right now?

a full import is 4-5 hours?

I have no idea if writing time series data in a willy nilly fashion will yield an optimal layout on disk.

not sure if timescale is smart enough to re lay things out.

hmm, will have to look into that, i guess.

but

the rate I am getting is about 880k inserts per minute.

listens will always come in willy nilly fashion afterwards anyways

6 hours.

true dat, but some at least 300M we might have a more performant setup.

let me read their best practices guide and see if I can glean anything.

the scale we're scaling for, 300M would be small potatoes

also true.

that's my intuition, maybe reality is different

aight, let me put it this way... I really want to know all the caveats this time.

as you said, switching in 3 years would be much harder.

yeah, that makes sense.

and I agree with zas' take on scaling.

partition users by server.

so when one node gets full, we add more and migrate users over.

see TIP under insert.

yup, designing for sharding is a good idea

also warning under UPDATE.

that is the "import form last.fm" will become a problem case.

that is updates though

oh wait it says inserts, sorry

basically messing with anything other than the current chunk, is going to cost you.

one way or another. read should continue to be fine though.

iliekcomputers: we've seen this before, no?

in AB, i think

> js = js.replace("\u0000", "")

that's json v jsonb maybe

doesn't seem to be doing the trick though.

I am using jsonb -- did you look at the schema in the pasted code?

yeah, just did

4 columns.

with inserted_at it will be 5.

and with PG JSONB support the data still remains queryable. and indexable.

[listenbrainz-server] paramsingh opened pull request #740 (master…param/email-when-new-stats-start-getting-written): Send a mail when new user stats start getting written into the db from spark https://github.com/metabrainz/listenbrainz-serv...

how philosophical... TIME_TO_CONSIDER_STATS_AS_OLD

😂

ugh

didn't push the actual email

i've deployed this to spark-reader-beta anyways, hopefully we'll see some email tomorrow

ok, my script now gets past those 0x0000 errors.

hopefully a full db in place in the morning.

10000 additions since the last LB release

🤷🏽

10000 new listens?

oh changed code?

yeah

most of it is spark tbh

exactly, that.

still, we need to get back to regular releases.

[listenbrainz-server] paramsingh merged pull request #740 (master…param/email-when-new-stats-start-getting-written): Send a mail when new user stats start getting written into the db from spark https://github.com/metabrainz/listenbrainz-serv...

[listenbrainz-server] paramsingh opened pull request #741 (master…param/fix-push-scripts): Fix spark image push scripts https://github.com/metabrainz/listenbrainz-serv...