#metabrainz

!m outsidecontext

You're doing good work, outsidecontext!

Mr_Monkey: So I was looking into that issue of relationship editor in which the search suggestions are not shown properly. But the problem is I cant see the console log statements since it is rendered on the server side. Now for that I thought to use the debug-watch-server script (hoping it would help in debugging this issue), but it doesn't seem to start elasticsearch on its own. So I am stuck here for a while now. Any help would be

appreciated.

It might be a noob issue

Just tell me if I am going in the right direction?

Moin!

mo’’in’

ruaok: pristine__: ^

[listenbrainz-server] paramsingh merged pull request #606 (master…param/stats-consumer-setup): Set up spark_reader config and uwsgi stuff https://github.com/metabrainz/listenbrainz-serv...

moooin!

moin :)

iliekcomputers: the biggest task it the first one, right?

yes!

let's break that into more pieces so it becomes more clear.

right.

on the lemmy side we have a nebulous idea that we want to have "user stats" calculated, right? or do we have more?

specifically, where do the actual queries live?

i'm not sure if the query on the lemmy side should be "get stats for this user" or "get this stat for this user"

or "get this stat for all users" or "get all stats for all users"

the queries were much faster when we just calculated one thing for all users without a where clause.

ok, there are two questions to hand then.

first, do we ever want to request individual stats?

I can't see a use case for this right now. but I suspect that we will in the future.

i think so, yes.

users might want their stats to be more up-to-date than the ones we have in the db.

not sure it makes sense to build that feature right this second, but to design for it being added later.

i think keeping it in mind for the future makes sense.

ok, let's do that.

so, I think we ought to implement: "get all stats, all users" and "get stats for user X"

so, that for every stat update cycle, we first all the former.

and then for every newly (re)arrived user, we can send the "get stats for user X" message.

makes sense to me.

already has a lot of these pieces, all jammed together into one file.

I see the following tasks:

1. break queries out into a separate file, possible one query per file

2. add support for single user queries

3. write a command listener that listens for requests and then runs jobs.

4. add/improve configuration bits for knowing where lemmy is and how to connect to MBs rabbitmq

what else?

5. define a language for requests to the user stats that is passed across rabbitmq from lemmy to leader.

yes. other than some implementation specific bits and bobs we might hit, this list seems comprehensive enough.

for the first thing in the doc

yep

I've added 1-4.

how should the commands look that we send across?

JSON? simple strings?

JSON

{ "stat" : "", "user": "" }

i think having a seperate file in lemmy listing each command and it's schema might be helpful in the future.

calc all stats for all users.

not in lemmy, in the LB repository

{ "stat" : "artists", "user" : "rob" }

{ "stat" : "", "user" : "rob" }

too simple?

thinking.

lol

i'm not sure if defining a "language" per se makes sense here.

i'd prefer to just have a list of queries with names in a file in LB

with a corresponding function in labs

and good docs on how exactly to add new queries.

or would that lead to too many combinations

I don't see a use case right now for just getting artists for a user. Ideally we'd just get everything from spark and query postgres as needed.

as far as I can see, we will always need to update both the lemmy and the spark side of things in order to add a stat, right?

yes

hmm. I wonder if we're conflating two topics here: how to define and name queries and how to request them.

requesting them really comes down to two cases we wish to support right now: all, or specific users.

I still think we should be using JSON for this, but perhaps add a request type to it.

here's what I was thinking

{ "request" : "calculate", "stats" : "all", "users" : "all" }

we have a query named "all_stats_all_users" and it doesn't need any params

so we send {"query": "all_stats_all_users"} to rmq

good thinking, we should preserve params. 🤣

:P

then we have a query "all_stats_for_user" and it needs a user

so we send {"query": "all_stats_for_user", params: {"user_name": "blah"}}

ok, I see where you're going and I think I like it.

here each query just seems like a function call to me and we don't have to do any specific logic.

I kinda dislike how both our approaches have a special case for "all".

ah, but when there is a 1:1 binding from request types to functions, then it comes together nicely.

yes!

ok, let's do that.

let's talk about naming stats.

I think each stat should have a clear and unambiguous name.

artist_stats -> shitty.

I agree.

top_artists_per_user -> good.

I'd also like to have a description for each stat anyways.

does that get stored at lemmy or spark?

I'd say lemmy.

hmm. not sure.

it is user facing metadata. that belong to lemmy, IMHO.

spark should be the dark magic voodoo stuff.

hmm, i don't think i have an opinion either way. if we enforce a 1-1 mapping, lemmy works for me.

ok, bake it so.

how do I indent that line?

I want to make a sub-list.

tab works for me?

ah. the obvious thing. lol.

listing all the stats is a good idea!

I just wanted to capture some examples on how the requests should be formatted.

but, lets make a list of stats too.

chhavi arrives in BCN tomorrow. I hope I can rope her into some graphics work for us. :)

that'd be amazing.

LB already has react so it'll hopefully be easier to add graphs this time around.

😅

yes.

ok, I think the spark consumer is well enough defined for my taste now. what do you think?

i agree.

ok, onward.

and yes, for staring the container on the spark side of things we should define and run a new servicem similar to how the master service is defined.

do we need a different service?

I don't recall the specifics right this second.

might just be a singleton container and not a service. I'll have to dig a little more.

but for now I am comfortable saying that we'll start the service according to how services are started now, extending things as needed.

and not bringing in consul.

hmm. sounds good.

do we have any sensitive data in the configs right now anyways?

ok, I see things breaking down into two larger groups: write the consumer and all the admin BS around running and connecting both ends of the consumer.

not on leader, we don't.

yes, totally agree

with the breaking into two groups.

so, wanna take care of whipping the consumer into shape while I take care of the other tasks?

definitely what I was hoping for :P

this sync was really helpful!

right. what else needs discussing before we dive in?

yes, I feel like I understand the task to hand now.

I don't think I have anything right now.