deploying consul via ansible would be an improvement, it does have to run on docker, an instance (client and/or server) needs to run on nodes anyway. What I suggest is to try an Ansible deployment of a 3-nodes cluster, using 10.10.10.x network, and few clients deployments (consul agent)
2023-03-30 08910, 2023
atj
zas: ok we can look at doing a test deployment. i had a brief scan of the readme and there's a few things that concern me, e.g. "it does not currently concern itself (all that much) with performing ongoing maintenance of a cluster" and the OS support matrix only listing Ubuntu 16.04
yup, dunno (yet) how much those points are impacting us
2023-03-30 08918, 2023
atj
i'm going to try and merge the borgmatic PR today and migrate the settings from the borg-backup repo
2023-03-30 08928, 2023
zas
that said, the idea of managing this part of the infrastructure (consul-stuff) with Ansible makes sense to me
2023-03-30 08938, 2023
zas
ok great, keep me updated
2023-03-30 08953, 2023
zas
in practice, it means we'll run 2 consul clusters in parallel for a while, new install shouldn't interfere with old docker-based one
2023-03-30 08909, 2023
zas
new containers will be encouraged to use the new cluster, while we deprecate the old one, slowly moving from 10.2.2.x to 10.10.10.x which will offer more freedom (not being limited to be physically in same place)
2023-03-30 08938, 2023
zas
currently, AFAIK, consul network gets changes only from 2 sources: gitzconsul & serviceregistrator. The first one basically converts a part of docker-servers-config files to consul storage, the second registers running containers (mainly for openresty to autoconfigure http forwarding on gateways)
2023-03-30 08915, 2023
zas
both can be executed along old instances but pointing at new consul cluster instead
2023-03-30 08948, 2023
zas
then app containers can choose to use one or the other, until we totally remove the old cluster
2023-03-30 08940, 2023
zas
it should be noted they might be changes at docker-template level too (so likely rebuilding containers)
2023-03-30 08929, 2023
iconoclasthero joined the channel
2023-03-30 08937, 2023
zas
but the target is to make lucifer happy at this point.
mayhem: In my proposal should I add code snippets or directly link the closed PR
2023-03-30 08920, 2023
mayhem
link is fine
2023-03-30 08956, 2023
jivte
and a image could of the feature could be added too :)
2023-03-30 08919, 2023
jivte
or :|
2023-03-30 08924, 2023
mayhem
as you wish, really.
2023-03-30 08957, 2023
jivte
Okk thanks for the help :)
2023-03-30 08907, 2023
mayhem
np
2023-03-30 08945, 2023
lucifer
mayhem: looking into the IA stuff, there's an audio collection that seems to encompass all audio files of the archive.
2023-03-30 08923, 2023
lucifer
however, 1) not all of it is music. 2) the search is slow and does not seem to be recursive.
2023-03-30 08958, 2023
mayhem
yea, slow is pretty characteristic of the IA services. sadly.
2023-03-30 08915, 2023
mayhem
but I suppose we could just crawl the collection and build a content resolver from it.
2023-03-30 08921, 2023
lucifer
i think a content resolver like local cache will have to be the way
2023-03-30 08930, 2023
lucifer
lol. yup
2023-03-30 08932, 2023
mayhem nods
2023-03-30 08936, 2023
jivte has quit
2023-03-30 08924, 2023
lucifer
another thing, in favor of a local content resovler is that. most of the music on LB won't be from IA so not very sensible to make slow queries to IA for each track.
2023-03-30 08912, 2023
mayhem
yep.
2023-03-30 08954, 2023
jivte joined the channel
2023-03-30 08935, 2023
jivte
monkey: About that add album feature could you help in improving some design mockups :)
2023-03-30 08921, 2023
monkey
Hi jivte! I don't have time available until the 11th I'm afraid
monkey and I realized that we have another project that we'd need your help on.
2023-03-30 08918, 2023
lucifer
sure, what is it?
2023-03-30 08924, 2023
mayhem
you know how the CF results have a last listened timestamp? I need those for all tracks for all users.
2023-03-30 08941, 2023
lucifer
we have that already stored in spark.
2023-03-30 08952, 2023
mayhem
not just CF tracks -- so in artist radio I need to know if the user has listened to a track recently.
2023-03-30 08953, 2023
Khagan has quit
2023-03-30 08958, 2023
lucifer
where and how often updated do you need it?
2023-03-30 08907, 2023
mayhem
for *all* tracks and *all* users ?
2023-03-30 08910, 2023
lucifer
yup
2023-03-30 08917, 2023
lucifer
CF joins with that data to add it to recs.
2023-03-30 08924, 2023
mayhem
ha great!
2023-03-30 08945, 2023
mayhem
well, so troi needs it.
2023-03-30 08958, 2023
mayhem
which means that when we move this feature over to spark, that will be easy.
2023-03-30 08916, 2023
lucifer
yup indeed
2023-03-30 08919, 2023
mayhem
monkey also needs it and he would like to show play count of a given track and when the user last listened to it.
2023-03-30 08930, 2023
mayhem
I suppose adding playcounts isn't too hard in this case, right?
2023-03-30 08956, 2023
lucifer
yup should be easy.
2023-03-30 08906, 2023
lucifer
so need to store in couchdb i think.
2023-03-30 08907, 2023
mayhem
it seems like it is another case of needing to take data from spark and move it to PG then API it.
2023-03-30 08910, 2023
lucifer
updated weekly?
2023-03-30 08922, 2023
mayhem
not good enough.
2023-03-30 08926, 2023
lucifer
yeah i guess PG would be better in this case.
2023-03-30 08946, 2023
mayhem
really, i am realizing, we need data set hoster for HDFS
2023-03-30 08948, 2023
lucifer
so that we can query both ways by user and by artist.
2023-03-30 08908, 2023
mayhem
I really dislike taking data from spark moving it to PG just to serve it.
2023-03-30 08919, 2023
lucifer
that would mean exposing spark cluster to web though.
2023-03-30 08930, 2023
mayhem
is there any way that we can make APIs on top of spark? seems no, because of a mistmatch in approaches.
2023-03-30 08938, 2023
mayhem
-t
2023-03-30 08948, 2023
lucifer
i think we can but i don't think it would be fast enough.
2023-03-30 08951, 2023
mayhem
lets ignore that problem for a second.
2023-03-30 08910, 2023
lucifer
there are other techs built on top of spark for such querying i think
2023-03-30 08915, 2023
mayhem
yeah. seem unlikely to work well.
2023-03-30 08926, 2023
mayhem
oh? that might be worth exploring.
2023-03-30 08949, 2023
lucifer
yeah many even support realtime querying and stuff.
2023-03-30 08911, 2023
lucifer
we could get listens in realtime through rmq to spark
2023-03-30 08943, 2023
mayhem
I think this is really worth exploring.
2023-03-30 08940, 2023
lucifer
sounds good
2023-03-30 08934, 2023
mayhem
I mean, if we could leave the stats data in HDFS and not have to move it around, that would be great.
2023-03-30 08949, 2023
mayhem
well, true of all of the datasets we shovel back and forth.
2023-03-30 08941, 2023
lucifer
indeed indeed
2023-03-30 08916, 2023
arsh
Hi mayhem: I hope you had a chance to look at the document I sent. Do you any thoughts or ideas that you like that I could build on. Thanks
2023-03-30 08936, 2023
mayhem
yes, I do.
2023-03-30 08942, 2023
mayhem
let me finish an email and get back to you.
2023-03-30 08956, 2023
arsh
Sure
2023-03-30 08915, 2023
Shelly joined the channel
2023-03-30 08957, 2023
mayhem
hey arsh!
2023-03-30 08909, 2023
arsh
hello
2023-03-30 08929, 2023
mayhem
so, the first bit of feedback is that we dont have access to artist images.
2023-03-30 08946, 2023
mayhem
we used to but then some asshole sued us and ruined the party. very long story.
2023-03-30 08954, 2023
Shelly
I am currently on master branch and running the listenbrainz-server on local but there's an sql error "psycopg2.errors.UndefinedColumn: column "external_user_id" does not exist
2023-03-30 08955, 2023
Shelly
listenbrainz-web-1 | LINE 6: , external_user_id
2023-03-30 08955, 2023
Shelly
"
2023-03-30 08900, 2023
arsh
oh i see
2023-03-30 08917, 2023
mayhem
so, your designs need to use no images.
2023-03-30 08943, 2023
mayhem
and I looked at your mock-ups and idea 3 jumps out at me.
2023-03-30 08904, 2023
Shelly
Can someone please look into it beacuse when i git pull from master my current branch broke.=L
