#metabrainz

we should run the tests in travis inside a docker container imo

idk what sdk is, but I do have a scrobbler for audioscrobble app thing.. I always thought it could be modified or reverse engineer or osmething

so that the tests we run locally are exactly the same as the tests run in travis

can you share the website where or something for that scrobbler ZaphodBeeblebrox

ah. it *was* old iScrobbler community on last.fm - but these are dead now

let me think

hm!

https://sourceforge.net/projects/iscrobbler/ ?

jessz, that's evne older thna the version *i* have

also i had a talk with the vlc devs, they are ready to include the plugin in vlc

cool!

well some stuff does work

i played now i'm pasuing (and playing osmething else in itunes)

i'll import it later and see what does LB do with that

> also i had a talk with the vlc devs, they are ready to include the plugin in vlc

awww yisss!

amCap1712: i'd like to lurk on wherever you chat with the vlc people

is there an irc chan?

#videolan

iliekcomputers: 🤣

amCap1712: ! https://github.com/cglong/iScrobbler

> 940f18f Add emoji

🤖

emoji are broken on mb so it's kidna bug :D

wait. damn wtf why does it say "itunes 7+ arg

[listenbrainz-server] paramsingh merged pull request #734 (master…param/cron-job-requesting-user-stats): Add cron job requesting stats from the spark cluster https://github.com/metabrainz/listenbrainz-serv...

amCap1712: idk if you cna even work wih https://sourceforge.net/p/iscrobbler/code/ or what

ZaphodBeeblebrox: i'll check and let you know whats possible

cooool

it's my dream to completely leave last.fm

this test has been really flaky for me.

hmm https://github.com/cglong/iScrobbler/commits/ma... says that 10.6 is min OS so anything ther should work. it's over 10 year old. olololol

do you know what's happening / what the best way to fix this problem is?

I've tried newer itunes. but they're all shit. and this one atleast jsut plays whatever iwant and stuff

hmmm 🤔 one idea is to also get QuodLibet inot this

a plugin for QuodLibet woudl be acebeams

(then I just need otfigure out how to shuffle a playlist on album not song and i'm readyto switch :D)

[listenbrainz-server] paramsingh merged pull request #736 (master…param/mail-on-dump-cron-job): Send a notification email when new data dumps are created https://github.com/metabrainz/listenbrainz-serv...

iliekcomputers: timescale is doing 20k inserts/s loading from the dump

how much did influx do?

how are you importing?

I forget what it was doing.

I just wrote a script that loads the data into a PG table -- I just added one extra call to create_hypertable(), otherwise it is bog-standard stuff.

woah

that's pretty nice

ZaphodBeeblebrox: it seems to be written in objective-c which apple abandoned a few years ago. it may work but needs a major overhaul

yeah, and all of PG's functionality is available to us.

can i look at the code?

k, we should get emails when dumps are created now.

amCap1712: :o

its not even in git yet. :)

next incremental dump is scheduled monday I think

i've roped in ferbncode to look into using protobuf instead of json in our queues

ferbncode: 👋🏽

nice!

>"INSERT INTO listen VALUES %s"

👏🏽 👏🏽

using timescale is ridiculously easy if you are used to PG.

I think I am going to re-write this to load from influx.

our dumps don't seem to include inserted_at timestamps.

which I really want to migrate

>our dumps don't seem to include inserted_at timestamps

add to trello?

sounds like a good first bug

jira if anything.

though I am not 100% sure on that. did we backfill those?

I looked at very early data.

no, there is no way to do that.

plus influx is a pita anyways

yeah.

we started tracking it a long time ago tho

if this approach looks promising, I will write the migration tool directly from influx.

will give us a better chance at being consistent.

because I can start writing new listens to it and THEN do the import of everything in influx.

the idea is to start shadowing the queue at timestamp x, and retroactively load everything before timestamp x, right?

when that process is done, we have everything.

yeah makes sense.

I wasn't even going to be so picky about X.

the system is designed to kick out dups, so let it kick out the dups at the tail end.

y?

yeah i guess that makes sense.

wait

but the dups are in the queue

dup logic*

are you gonna take from influx and put it in the queue?

first, and this is what I am going write right now, is to write a timescale_writer, much like the influx_writer.

iliekcomputers: what is odd about it? I have no idea tbh

then add a new queue off the incoming exchange and have it start writing the new listens.

pristine__: the test is very flaky, it fails randomly and then passes on rebuild

and it's only that test

at some point we will need to swap over and have the timescale writer write to the unique rmq, but that would be one of the last things to do.

ZaphodBeeblebrox: completed the testing or still on?

iliekcomputers: does that makes sense wrt to rmq? add a new queue off the incoming exchange, and then two consumers can consume the data at their own pace, effectively duplicating the data.

yes. i've done that with the follow feature

amCap1712: ag i'm palying something i'll test soem more

(just need food)

it follows the unique queue, iirc

great.

yeah sure let me know how it goes ZaphodBeeblebrox

ruaok: i'm with you until here.

how are you gonna backfill from influx to timescale

1. setup timescale

2. write code to fetch EVERY listen

iliekcomputers: yeah, I get that but I am not sure why it happens. Will have a look

3. insert into timescale.

lol this. same Arrepentimientos song goes to two differnt recordings??

1.a setup live writing from the incoming queue

ah

ah one is the borked itunes track-thing

s/itunes/last.fm/

oh dear. it

does apparently scrobble to last.fm even if i turned that damn thing of

and now they're duplicated :D

ruaok: 1. full import directly from influx 2. start shadowing the queue

this is what you mean basically right

no, the other way around.

start shadowing, the cross load

1. start shadowing the queue.

2. full import directly from influx

that would lead to duplicates

unless you write the dedup logic into the import process

postgres won't allow duplicates

👏🏽

hmm removed every thing I can of audioshattler, let's se if now it doesnt

now, that might make the migration quite slow, not sure.

yeah

oh!

migrations without constraints is what i learned from the AB migration

right. heh. rabbitmq.

why not just do the timestamp logic?

1. start shadowing the queue to insert all listens (starting timestamp x)

timestamp logic for de-duping? yes, the primary key will be (timestamp, username)

also testing restarting the same song

oic

2. directly import from influx all listens inserted to influx before timestamp x

yeah, given the no constraints bit, that might actually be easier.

I *really* hope this lives up to its promises.

if it does, it is the perfect solution for us.

can't wait to migrate over to the new thing 3 years later

(kidding)

well, I've only ever migrated to PG once.

best decision evar.

amCap1712: I fixed audioscrobbler bork!

great

ruaok: so about shadowing a queue. it's more shadowing an exchange which pushes to two queues iirc