#metabrainz

/

      • SothoTalKer joined the channel
      • 2019-11-09 31322, 2019

      • D4RK-PH0ENiX has quit
      • 2019-11-09 31317, 2019

      • D4RK-PH0ENiX joined the channel
      • 2019-11-09 31330, 2019

      • rdswift
        !m outsidecontext
      • 2019-11-09 31330, 2019

      • BrainzBot
        You're doing good work, outsidecontext!
      • 2019-11-09 31334, 2019

      • abhishekpanwar joined the channel
      • 2019-11-09 31327, 2019

      • chaban has quit
      • 2019-11-09 31337, 2019

      • chaban joined the channel
      • 2019-11-09 31322, 2019

      • abhishekpanwar has quit
      • 2019-11-09 31325, 2019

      • abhishekpanwar joined the channel
      • 2019-11-09 31357, 2019

      • abhishekpanwar has quit
      • 2019-11-09 31349, 2019

      • abhishekpanwar joined the channel
      • 2019-11-09 31301, 2019

      • abhishekpanwar
        Mr_Monkey: So I was looking into that issue of relationship editor in which the search suggestions are not shown properly. But the problem is I cant see the console log statements since it is rendered on the server side. Now for that I thought to use the debug-watch-server script (hoping it would help in debugging this issue), but it doesn't seem to start elasticsearch on its own. So I am stuck here for a while now. Any help would be
      • 2019-11-09 31302, 2019

      • abhishekpanwar
        appreciated.
      • 2019-11-09 31331, 2019

      • abhishekpanwar
        It might be a noob issue
      • 2019-11-09 31358, 2019

      • abhishekpanwar
        Just tell me if I am going in the right direction?
      • 2019-11-09 31326, 2019

      • abhishekpanwar has quit
      • 2019-11-09 31312, 2019

      • abhishekpanwar joined the channel
      • 2019-11-09 31338, 2019

      • abhishekpanwar has quit
      • 2019-11-09 31351, 2019

      • iliekcomputers
        Moin!
      • 2019-11-09 31346, 2019

      • yvanzo
        mo’’in’
      • 2019-11-09 31302, 2019

      • iliekcomputers
      • 2019-11-09 31337, 2019

      • iliekcomputers
        ruaok: pristine__: ^
      • 2019-11-09 31348, 2019

      • BrainzGit
        [listenbrainz-server] paramsingh merged pull request #606 (master…param/stats-consumer-setup): Set up spark_reader config and uwsgi stuff https://github.com/metabrainz/listenbrainz-server…
      • 2019-11-09 31301, 2019

      • Gazooo has quit
      • 2019-11-09 31305, 2019

      • ruaok
        moooin!
      • 2019-11-09 31327, 2019

      • iliekcomputers
        moin :)
      • 2019-11-09 31349, 2019

      • Gazooo joined the channel
      • 2019-11-09 31302, 2019

      • ruaok
        iliekcomputers: the biggest task it the first one, right?
      • 2019-11-09 31325, 2019

      • iliekcomputers
        yes!
      • 2019-11-09 31347, 2019

      • ruaok
        let's break that into more pieces so it becomes more clear.
      • 2019-11-09 31306, 2019

      • iliekcomputers
        right.
      • 2019-11-09 31308, 2019

      • ruaok
        on the lemmy side we have a nebulous idea that we want to have "user stats" calculated, right? or do we have more?
      • 2019-11-09 31327, 2019

      • ruaok
        specifically, where do the actual queries live?
      • 2019-11-09 31351, 2019

      • iliekcomputers
      • 2019-11-09 31321, 2019

      • iliekcomputers
        i'm not sure if the query on the lemmy side should be "get stats for this user" or "get this stat for this user"
      • 2019-11-09 31338, 2019

      • iliekcomputers
        or "get this stat for all users" or "get all stats for all users"
      • 2019-11-09 31321, 2019

      • iliekcomputers
        the queries were much faster when we just calculated one thing for all users without a where clause.
      • 2019-11-09 31301, 2019

      • ruaok
        ok, there are two questions to hand then.
      • 2019-11-09 31313, 2019

      • ruaok
        first, do we ever want to request individual stats?
      • 2019-11-09 31339, 2019

      • ruaok
        I can't see a use case for this right now. but I suspect that we will in the future.
      • 2019-11-09 31339, 2019

      • iliekcomputers
        i think so, yes.
      • 2019-11-09 31359, 2019

      • iliekcomputers
        users might want their stats to be more up-to-date than the ones we have in the db.
      • 2019-11-09 31301, 2019

      • ruaok
        not sure it makes sense to build that feature right this second, but to design for it being added later.
      • 2019-11-09 31324, 2019

      • iliekcomputers
        i think keeping it in mind for the future makes sense.
      • 2019-11-09 31341, 2019

      • ruaok
        ok, let's do that.
      • 2019-11-09 31314, 2019

      • ruaok
        so, I think we ought to implement: "get all stats, all users" and "get stats for user X"
      • 2019-11-09 31332, 2019

      • ruaok
        so, that for every stat update cycle, we first all the former.
      • 2019-11-09 31357, 2019

      • ruaok
        and then for every newly (re)arrived user, we can send the "get stats for user X" message.
      • 2019-11-09 31311, 2019

      • iliekcomputers
        makes sense to me.
      • 2019-11-09 31346, 2019

      • ruaok
      • 2019-11-09 31307, 2019

      • ruaok
        already has a lot of these pieces, all jammed together into one file.
      • 2019-11-09 31341, 2019

      • ruaok
        I see the following tasks:
      • 2019-11-09 31357, 2019

      • ruaok
        1. break queries out into a separate file, possible one query per file
      • 2019-11-09 31309, 2019

      • ruaok
        2. add support for single user queries
      • 2019-11-09 31338, 2019

      • ruaok
        3. write a command listener that listens for requests and then runs jobs.
      • 2019-11-09 31309, 2019

      • ruaok
        4. add/improve configuration bits for knowing where lemmy is and how to connect to MBs rabbitmq
      • 2019-11-09 31313, 2019

      • ruaok
        what else?
      • 2019-11-09 31350, 2019

      • ruaok
        5. define a language for requests to the user stats that is passed across rabbitmq from lemmy to leader.
      • 2019-11-09 31311, 2019

      • iliekcomputers
        yes. other than some implementation specific bits and bobs we might hit, this list seems comprehensive enough.
      • 2019-11-09 31325, 2019

      • iliekcomputers
        for the first thing in the doc
      • 2019-11-09 31332, 2019

      • ruaok
        yep
      • 2019-11-09 31305, 2019

      • ruaok
        I've added 1-4.
      • 2019-11-09 31316, 2019

      • ruaok
        how should the commands look that we send across?
      • 2019-11-09 31325, 2019

      • ruaok
        JSON? simple strings?
      • 2019-11-09 31332, 2019

      • iliekcomputers
        JSON
      • 2019-11-09 31354, 2019

      • ruaok
        { "stat" : "", "user": "" }
      • 2019-11-09 31301, 2019

      • iliekcomputers
        i think having a seperate file in lemmy listing each command and it's schema might be helpful in the future.
      • 2019-11-09 31301, 2019

      • ruaok
        calc all stats for all users.
      • 2019-11-09 31314, 2019

      • iliekcomputers
        not in lemmy, in the LB repository
      • 2019-11-09 31319, 2019

      • ruaok nods
      • 2019-11-09 31340, 2019

      • ruaok
        { "stat" : "artists", "user" : "rob" }
      • 2019-11-09 31302, 2019

      • ruaok
        { "stat" : "", "user" : "rob" }
      • 2019-11-09 31314, 2019

      • ruaok
        too simple?
      • 2019-11-09 31342, 2019

      • iliekcomputers
        thinking.
      • 2019-11-09 31300, 2019

      • ruaok imagines a spinning wheel on iliekcomputers' forhead
      • 2019-11-09 31318, 2019

      • iliekcomputers
        lol
      • 2019-11-09 31327, 2019

      • iliekcomputers
        i'm not sure if defining a "language" per se makes sense here.
      • 2019-11-09 31348, 2019

      • iliekcomputers
        i'd prefer to just have a list of queries with names in a file in LB
      • 2019-11-09 31355, 2019

      • iliekcomputers
        with a corresponding function in labs
      • 2019-11-09 31306, 2019

      • iliekcomputers
        and good docs on how exactly to add new queries.
      • 2019-11-09 31335, 2019

      • iliekcomputers
        or would that lead to too many combinations
      • 2019-11-09 31343, 2019

      • iliekcomputers
        I don't see a use case right now for just getting artists for a user. Ideally we'd just get everything from spark and query postgres as needed.
      • 2019-11-09 31319, 2019

      • ruaok
        as far as I can see, we will always need to update both the lemmy and the spark side of things in order to add a stat, right?
      • 2019-11-09 31328, 2019

      • iliekcomputers
        yes
      • 2019-11-09 31357, 2019

      • ruaok
        hmm. I wonder if we're conflating two topics here: how to define and name queries and how to request them.
      • 2019-11-09 31352, 2019

      • ruaok
        requesting them really comes down to two cases we wish to support right now: all, or specific users.
      • 2019-11-09 31330, 2019

      • ruaok
        I still think we should be using JSON for this, but perhaps add a request type to it.
      • 2019-11-09 31317, 2019

      • iliekcomputers
        here's what I was thinking
      • 2019-11-09 31320, 2019

      • ruaok
        { "request" : "calculate", "stats" : "all", "users" : "all" }
      • 2019-11-09 31344, 2019

      • iliekcomputers
        we have a query named "all_stats_all_users" and it doesn't need any params
      • 2019-11-09 31311, 2019

      • iliekcomputers
        so we send {"query": "all_stats_all_users"} to rmq
      • 2019-11-09 31312, 2019

      • ruaok
        good thinking, we should preserve params. 🤣
      • 2019-11-09 31318, 2019

      • iliekcomputers
        :P
      • 2019-11-09 31338, 2019

      • iliekcomputers
        then we have a query "all_stats_for_user" and it needs a user
      • 2019-11-09 31359, 2019

      • iliekcomputers
        so we send {"query": "all_stats_for_user", params: {"user_name": "blah"}}
      • 2019-11-09 31328, 2019

      • ruaok
        ok, I see where you're going and I think I like it.
      • 2019-11-09 31347, 2019

      • iliekcomputers
        here each query just seems like a function call to me and we don't have to do any specific logic.
      • 2019-11-09 31353, 2019

      • ruaok
        I kinda dislike how both our approaches have a special case for "all".
      • 2019-11-09 31348, 2019

      • ruaok
        ah, but when there is a 1:1 binding from request types to functions, then it comes together nicely.
      • 2019-11-09 31355, 2019

      • iliekcomputers
        yes!
      • 2019-11-09 31322, 2019

      • ruaok
        ok, let's do that.
      • 2019-11-09 31327, 2019

      • ruaok
        let's talk about naming stats.
      • 2019-11-09 31346, 2019

      • ruaok
        I think each stat should have a clear and unambiguous name.
      • 2019-11-09 31352, 2019

      • ruaok
        artist_stats -> shitty.
      • 2019-11-09 31302, 2019

      • iliekcomputers
        I agree.
      • 2019-11-09 31308, 2019

      • ruaok
        top_artists_per_user -> good.
      • 2019-11-09 31314, 2019

      • iliekcomputers
        I'd also like to have a description for each stat anyways.
      • 2019-11-09 31336, 2019

      • ruaok
        does that get stored at lemmy or spark?
      • 2019-11-09 31339, 2019

      • ruaok
        I'd say lemmy.
      • 2019-11-09 31358, 2019

      • iliekcomputers
        hmm. not sure.
      • 2019-11-09 31317, 2019

      • ruaok
        it is user facing metadata. that belong to lemmy, IMHO.
      • 2019-11-09 31327, 2019

      • ruaok
        spark should be the dark magic voodoo stuff.
      • 2019-11-09 31352, 2019

      • iliekcomputers
        hmm, i don't think i have an opinion either way. if we enforce a 1-1 mapping, lemmy works for me.
      • 2019-11-09 31307, 2019

      • ruaok
        ok, bake it so.
      • 2019-11-09 31351, 2019

      • iliekcomputers
        +1
      • 2019-11-09 31334, 2019

      • ruaok
        how do I indent that line?
      • 2019-11-09 31343, 2019

      • ruaok
        I want to make a sub-list.
      • 2019-11-09 31314, 2019

      • iliekcomputers
        tab works for me?
      • 2019-11-09 31327, 2019

      • ruaok
        ah. the obvious thing. lol.
      • 2019-11-09 31341, 2019

      • iliekcomputers
        listing all the stats is a good idea!
      • 2019-11-09 31319, 2019

      • ruaok
        I just wanted to capture some examples on how the requests should be formatted.
      • 2019-11-09 31338, 2019

      • ruaok
        but, lets make a list of stats too.
      • 2019-11-09 31332, 2019

      • ruaok
        chhavi arrives in BCN tomorrow. I hope I can rope her into some graphics work for us. :)
      • 2019-11-09 31351, 2019

      • iliekcomputers
        that'd be amazing.
      • 2019-11-09 31326, 2019

      • iliekcomputers
        LB already has react so it'll hopefully be easier to add graphs this time around.
      • 2019-11-09 31335, 2019

      • iliekcomputers
        😅
      • 2019-11-09 31349, 2019

      • ruaok
        yes.
      • 2019-11-09 31312, 2019

      • D4RK-PH0ENiX has quit
      • 2019-11-09 31313, 2019

      • ruaok
        ok, I think the spark consumer is well enough defined for my taste now. what do you think?
      • 2019-11-09 31331, 2019

      • iliekcomputers
        i agree.
      • 2019-11-09 31336, 2019

      • ruaok
        ok, onward.
      • 2019-11-09 31352, 2019

      • ruaok
        and yes, for staring the container on the spark side of things we should define and run a new servicem similar to how the master service is defined.
      • 2019-11-09 31305, 2019

      • iliekcomputers
        do we need a different service?
      • 2019-11-09 31328, 2019

      • ruaok
        I don't recall the specifics right this second.
      • 2019-11-09 31348, 2019

      • ruaok
        might just be a singleton container and not a service. I'll have to dig a little more.
      • 2019-11-09 31310, 2019

      • ruaok
        but for now I am comfortable saying that we'll start the service according to how services are started now, extending things as needed.
      • 2019-11-09 31314, 2019

      • ruaok
        and not bringing in consul.
      • 2019-11-09 31337, 2019

      • iliekcomputers
        hmm. sounds good.
      • 2019-11-09 31353, 2019

      • iliekcomputers
        do we have any sensitive data in the configs right now anyways?
      • 2019-11-09 31305, 2019

      • ruaok
        ok, I see things breaking down into two larger groups: write the consumer and all the admin BS around running and connecting both ends of the consumer.
      • 2019-11-09 31318, 2019

      • ruaok
        not on leader, we don't.
      • 2019-11-09 31319, 2019

      • iliekcomputers
        yes, totally agree
      • 2019-11-09 31328, 2019

      • iliekcomputers
        with the breaking into two groups.
      • 2019-11-09 31356, 2019

      • ruaok
        so, wanna take care of whipping the consumer into shape while I take care of the other tasks?
      • 2019-11-09 31312, 2019

      • iliekcomputers
        definitely what I was hoping for :P
      • 2019-11-09 31331, 2019

      • iliekcomputers
        this sync was really helpful!
      • 2019-11-09 31332, 2019

      • ruaok
        right. what else needs discussing before we dive in?
      • 2019-11-09 31344, 2019

      • ruaok
        yes, I feel like I understand the task to hand now.
      • 2019-11-09 31353, 2019

      • iliekcomputers
        I don't think I have anything right now.