19:57 PM
ruaok
that is exactly what I figured out yesterday.
2020-02-20 05142, 2020
19:57 PM
ruaok
let me go read the follow code just to make sure
2020-02-20 05152, 2020
19:57 PM
iliekcomputers
yeah, that's what i was gonna say
2020-02-20 05157, 2020
19:57 PM
ZaphodBeeblebrox
and the vlc listening works really well honestly
2020-02-20 05131, 2020
19:58 PM
iliekcomputers
<3 livegrep
2020-02-20 05135, 2020
19:58 PM
Raghav has quit
2020-02-20 05152, 2020
19:58 PM
iliekcomputers
2020-02-20 05143, 2020
19:59 PM
ruaok
it looks like you created a separate exchange too.
2020-02-20 05148, 2020
19:59 PM
ruaok
"playing_now_exchange": "playing_now",
2020-02-20 05148, 2020
19:59 PM
ruaok
"playing_now_queue": "playing_now",
2020-02-20 05152, 2020
19:59 PM
iliekcomputers
i have no remembrance at all of how the follow thing worked
2020-02-20 05158, 2020
19:59 PM
ruaok
docker-server-configs/consul/LB
2020-02-20 05125, 2020
20:00 PM
iliekcomputers
oh
2020-02-20 05127, 2020
20:00 PM
iliekcomputers
so
2020-02-20 05148, 2020
20:00 PM
iliekcomputers
the playing_now queue is only for listens that people sent in as 'now_playing'
2020-02-20 05155, 2020
20:00 PM
iliekcomputers
2020-02-20 05107, 2020
20:01 PM
iliekcomputers
2020-02-20 05110, 2020
20:01 PM
ruaok
yeah, I just remembered that.
2020-02-20 05104, 2020
20:02 PM
iliekcomputers
k, i'm gonna get food and then look into emails for the stats cron job
2020-02-20 05111, 2020
20:02 PM
ruaok
k
2020-02-20 05121, 2020
20:02 PM
iliekcomputers
the observability mailer may be spammy for a while.
2020-02-20 05139, 2020
20:02 PM
ZaphodBeeblebrox
oh no.
2020-02-20 05154, 2020
20:02 PM
Raghav joined the channel
2020-02-20 05105, 2020
20:03 PM
ZaphodBeeblebrox
amCap1712: I'm finished testing for now. tell me tomorrow/osme other tiem if yo uhave spesific test cases yo uwant me to test
2020-02-20 05152, 2020
20:03 PM
ruaok
iliekcomputers: self.create_and_bind_exchange_and_queue(channel, current_app.config['UNIQUE_EXCHANGE'], current_app.config['FOLLOW_LIST_QUEUE'])
2020-02-20 05100, 2020
20:04 PM
ruaok
this is the crux of it. makes sense.
2020-02-20 05125, 2020
20:04 PM
iliekcomputers
yep
2020-02-20 05114, 2020
20:05 PM
amCap1712
ZaphodBeeblebrox: pausing/resuming, stopping midway, repeat, playlist. thats all. if all of that is tested i don't have anything more in mind
2020-02-20 05141, 2020
20:05 PM
ruaok
oh, another correction
2020-02-20 05148, 2020
20:05 PM
ruaok
1. start shadowing the UNIQUE queue to insert all listens (starting timestamp x)
2020-02-20 05151, 2020
20:05 PM
amCap1712
iliekcomputers: ruaok: do you think there could be some other edge cases which might require to tested separately?
2020-02-20 05102, 2020
20:06 PM
ruaok
2. directly import from influx all listens inserted to influx before timestamp x
2020-02-20 05103, 2020
20:06 PM
ZaphodBeeblebrox
I think i did all these. i guess palylist. but i'm tired and stuff i need ot eat too
2020-02-20 05109, 2020
20:06 PM
amCap1712
ZaphodBeeblebrox: tommorrow lets test on windows then
2020-02-20 05122, 2020
20:06 PM
amCap1712
yeah sure. thanks for the help
2020-02-20 05122, 2020
20:06 PM
ZaphodBeeblebrox
i'll try to move the windows machine . alright!
2020-02-20 05126, 2020
20:06 PM
ruaok
amCap1712: the answer is yes. but, what are we talking about?
2020-02-20 05137, 2020
20:06 PM
ZaphodBeeblebrox
hey no problem. this is awesome!!
2020-02-20 05120, 2020
20:07 PM
amCap1712
ruaok: edge cases to test the plugin for. like one might be to stop the plugin before it is half finished
2020-02-20 05122, 2020
20:07 PM
ZaphodBeeblebrox
two different things beeing listened too at the same time (diff machines, diff programs, etc)
2020-02-20 05128, 2020
20:07 PM
amCap1712
and should not be submitted
2020-02-20 05132, 2020
20:07 PM
ZaphodBeeblebrox
i cna test this tomorrow! windows and mac both play!
2020-02-20 05104, 2020
20:08 PM
ruaok
amCap1712: if the track played 30 seconds or more, it is considered a play.
2020-02-20 05106, 2020
20:08 PM
ZaphodBeeblebrox
both should listenz. after all I 'm listening ot both :D
2020-02-20 05128, 2020
20:08 PM
ZaphodBeeblebrox
so. if playing , say 40 seocnds thne double clicknig again on it
2020-02-20 05129, 2020
20:08 PM
ZaphodBeeblebrox
i cna test that
2020-02-20 05132, 2020
20:08 PM
amCap1712
ruaok: even if it less than half the duration?
2020-02-20 05152, 2020
20:08 PM
ruaok
30 seconds is just the standard in the industry.
2020-02-20 05109, 2020
20:09 PM
ruaok
which seems BS to me, but...
2020-02-20 05159, 2020
20:09 PM
amCap1712
that just a constant i'll have to modify.
2020-02-20 05118, 2020
20:10 PM
ruaok
then for now, keep it simple and set the constant to 30.
2020-02-20 05119, 2020
20:10 PM
amCap1712
but i think it gives the idea of what i am trying to convey by different test cases
2020-02-20 05140, 2020
20:10 PM
ZaphodBeeblebrox
i have songs that are >30 secs whixh i wish would listenz
2020-02-20 05141, 2020
20:10 PM
ruaok
yes, but I can't really think of any off the top of my head.
2020-02-20 05147, 2020
20:10 PM
ruaok
alastairp is good at these things.
2020-02-20 05109, 2020
20:11 PM
ZaphodBeeblebrox
(ok so I made up that listening eg "scrobbling" is called "listenz" .. i uges it's dumb)
2020-02-20 05136, 2020
20:11 PM
amCap1712
i'll post on the discourse for asking others to test.
2020-02-20 05138, 2020
20:11 PM
ZaphodBeeblebrox
2020-02-20 05113, 2020
20:12 PM
amCap1712
once we are satisfied with the testing, i can send the patch upstream for merging
2020-02-20 05118, 2020
20:12 PM
ZaphodBeeblebrox
yes
2020-02-20 05137, 2020
20:16 PM
ruaok
2020-02-20 05138, 2020
20:22 PM
iliekcomputers
i tend towards the simple one
2020-02-20 05153, 2020
20:22 PM
iliekcomputers
considering it's writing stuff at 20k/s right now?
2020-02-20 05130, 2020
20:23 PM
iliekcomputers
a full import is 4-5 hours?
2020-02-20 05141, 2020
20:23 PM
ruaok
I have no idea if writing time series data in a willy nilly fashion will yield an optimal layout on disk.
2020-02-20 05153, 2020
20:23 PM
ruaok
not sure if timescale is smart enough to re lay things out.
2020-02-20 05112, 2020
20:24 PM
iliekcomputers
hmm, will have to look into that, i guess.
2020-02-20 05139, 2020
20:24 PM
iliekcomputers
but
2020-02-20 05108, 2020
20:25 PM
ruaok
the rate I am getting is about 880k inserts per minute.
2020-02-20 05128, 2020
20:25 PM
iliekcomputers
listens will always come in willy nilly fashion afterwards anyways
2020-02-20 05135, 2020
20:25 PM
ruaok
6 hours.
2020-02-20 05103, 2020
20:26 PM
ruaok
true dat, but some at least 300M we might have a more performant setup.
2020-02-20 05116, 2020
20:26 PM
ruaok
let me read their best practices guide and see if I can glean anything.
2020-02-20 05146, 2020
20:26 PM
iliekcomputers shrugs
2020-02-20 05118, 2020
20:27 PM
iliekcomputers
the scale we're scaling for, 300M would be small potatoes
2020-02-20 05135, 2020
20:27 PM
ruaok
also true.
2020-02-20 05141, 2020
20:27 PM
iliekcomputers
that's my intuition, maybe reality is different
2020-02-20 05148, 2020
20:27 PM
ruaok
aight, let me put it this way... I really want to know all the caveats this time.
2020-02-20 05100, 2020
20:28 PM
ruaok
as you said, switching in 3 years would be much harder.
2020-02-20 05109, 2020
20:28 PM
iliekcomputers
yeah, that makes sense.
2020-02-20 05121, 2020
20:28 PM
ruaok
and I agree with zas' take on scaling.
2020-02-20 05131, 2020
20:28 PM
ruaok
partition users by server.
2020-02-20 05106, 2020
20:29 PM
ruaok
so when one node gets full, we add more and migrate users over.
2020-02-20 05139, 2020
20:29 PM
ruaok
2020-02-20 05152, 2020
20:29 PM
ruaok
see TIP under insert.
2020-02-20 05158, 2020
20:29 PM
iliekcomputers
yup, designing for sharding is a good idea
2020-02-20 05149, 2020
20:30 PM
ruaok
also warning under UPDATE.
2020-02-20 05100, 2020
20:31 PM
ruaok
that is the "import form last.fm" will become a problem case.
2020-02-20 05138, 2020
20:31 PM
iliekcomputers
that is updates though
2020-02-20 05150, 2020
20:31 PM
iliekcomputers
oh wait it says inserts, sorry
2020-02-20 05123, 2020
20:32 PM
ruaok
basically messing with anything other than the current chunk, is going to cost you.
2020-02-20 05135, 2020
20:32 PM
ruaok
one way or another. read should continue to be fine though.
2020-02-20 05128, 2020
20:33 PM
ruaok
iliekcomputers: we've seen this before, no?
2020-02-20 05128, 2020
20:33 PM
ruaok
2020-02-20 05157, 2020
20:33 PM
iliekcomputers
in AB, i think
2020-02-20 05104, 2020
20:34 PM
Raghav has quit
2020-02-20 05122, 2020
20:34 PM
ruaok
> js = js.replace("\u0000", "")
2020-02-20 05123, 2020
20:34 PM
iliekcomputers
that's json v jsonb maybe
2020-02-20 05130, 2020
20:34 PM
ruaok
doesn't seem to be doing the trick though.
2020-02-20 05146, 2020
20:34 PM
ruaok
I am using jsonb -- did you look at the schema in the pasted code?
2020-02-20 05103, 2020
20:35 PM
iliekcomputers
yeah, just did
2020-02-20 05105, 2020
20:35 PM
ruaok
4 columns.
2020-02-20 05113, 2020
20:35 PM
ruaok
with inserted_at it will be 5.
2020-02-20 05139, 2020
20:35 PM
ruaok
and with PG JSONB support the data still remains queryable. and indexable.
2020-02-20 05135, 2020
20:36 PM
adhawkins joined the channel
2020-02-20 05153, 2020
20:47 PM
Gore has quit
2020-02-20 05133, 2020
20:48 PM
prabal has quit
2020-02-20 05158, 2020
21:00 PM
Gore joined the channel
2020-02-20 05132, 2020
21:05 PM
Etua joined the channel
2020-02-20 05159, 2020
21:06 PM
Raghav joined the channel
2020-02-20 05123, 2020
21:35 PM
BrainzGit
2020-02-20 05125, 2020
21:37 PM
ruaok
how philosophical... TIME_TO_CONSIDER_STATS_AS_OLD
2020-02-20 05158, 2020
21:38 PM
Raghav has quit
2020-02-20 05102, 2020
21:40 PM
iliekcomputers
😂
2020-02-20 05111, 2020
21:40 PM
iliekcomputers
ugh
2020-02-20 05118, 2020
21:40 PM
iliekcomputers
didn't push the actual email
2020-02-20 05139, 2020
21:44 PM
iliekcomputers
i've deployed this to spark-reader-beta anyways, hopefully we'll see some email tomorrow
2020-02-20 05107, 2020
21:46 PM
ruaok
ok, my script now gets past those 0x0000 errors.
2020-02-20 05119, 2020
21:46 PM
ruaok
hopefully a full db in place in the morning.
2020-02-20 05133, 2020
21:50 PM
Etua has quit
2020-02-20 05159, 2020
21:59 PM
iliekcomputers
10000 additions since the last LB release
2020-02-20 05102, 2020
22:00 PM
iliekcomputers
🤷🏽
2020-02-20 05120, 2020
22:00 PM
ruaok
10000 new listens?
2020-02-20 05150, 2020
22:00 PM
ruaok
oh changed code?
2020-02-20 05127, 2020
22:01 PM
iliekcomputers
yeah
2020-02-20 05128, 2020
22:01 PM
iliekcomputers
2020-02-20 05144, 2020
22:01 PM
iliekcomputers
most of it is spark tbh
2020-02-20 05150, 2020
22:01 PM
ruaok
exactly, that.
2020-02-20 05102, 2020
22:02 PM
ruaok
still, we need to get back to regular releases.
2020-02-20 05114, 2020
22:13 PM
BrainzGit
2020-02-20 05104, 2020
22:21 PM
BrainzGit
2020-02-20 05112, 2020
22:21 PM
kieto joined the channel
2020-02-20 05144, 2020
22:45 PM
prabal joined the channel
2020-02-20 05140, 2020
22:59 PM
ephem3r0l joined the channel
2020-02-20 05129, 2020
23:00 PM
supersandro2000 has quit
2020-02-20 05147, 2020
23:00 PM
supersandro2000 joined the channel
2020-02-20 05156, 2020
23:11 PM
ephem3r0l is now known as ephemer0l
2020-02-20 05106, 2020
23:45 PM
kieto has quit
