#metabrainz

/

      • LordSputnik
        Leftmost: who isn't?

        • 2016-01-13 01342, 2016

      • Leftmost
        Freso likes to act like he doesn't.

        • 2016-01-13 01355, 2016

      • ruaok
        Leftmost: people make it easy to stalk you. :)

        • 2016-01-13 01303, 2016

      • JESUS2099 joined the channel

        • 2016-01-13 01338, 2016

      • CallerNo6 has quit

        • 2016-01-13 01348, 2016

      • JESUS2099
        Freso chirlu` at my company, websites in HTTPS are sluggish like hell… the proxy acts like a man in the middle and redirects 2 or 3 times and invalidates all cache… then a page in HTTPS is likely to completely fail because of the borked certificate also

        • 2016-01-13 01314, 2016

      • ruaok
        remind me to never fly AF again.

        • 2016-01-13 01322, 2016

      • ruaok
        oh wait, I already don't. :)

        • 2016-01-13 01339, 2016

      • stanislas
        LordSputnik: Should I make get requests like this : get('/creator/{bbid}?user_id=3') ?

        • 2016-01-13 01341, 2016

      • LordSputnik
        stanislas: yup

        • 2016-01-13 01333, 2016

      • LordSputnik
        Freso: reosarevok: mind if I put myself as mentor on the CB reviews tasks to help clear the backlog?

        • 2016-01-13 01343, 2016

      • reosarevok
        Feel free if you want

        • 2016-01-13 01357, 2016

      • LordSputnik
        reosarevok: I've got some time, my desktop is doing a chkdsk on a 1TB drive :P

        • 2016-01-13 01305, 2016

      • JESUS2099
        ruaok: grrr ;)

        • 2016-01-13 01316, 2016

      • ruaok
        sorry. had to. :)

        • 2016-01-13 01323, 2016

      • JESUS2099
        it’s not AF the problem, the problem is that someone was convinced to buy something called Zscaler, which is an awful POS :/

        • 2016-01-13 01340, 2016

      • stanislas
        LordSputnik: Should I make my own tests (i have done some already) or there is a way to test it some other way ?

        • 2016-01-13 01354, 2016

      • LordSputnik
        stanislas: no need to make tests, but I'd be very happy if you did :)

        • 2016-01-13 01355, 2016

      • Leo_Verto
        \o LordSputnik

        • 2016-01-13 01313, 2016

      • LordSputnik
        If I were writing it, I'd try to test it if I could easily

        • 2016-01-13 01318, 2016

      • LordSputnik
        Leo_Verto: hey :)

        • 2016-01-13 01339, 2016

      • stanislas
        LordSputnik: Have you noticed that tests are becoming a serious part of bookbrainz-ws :) ?

        • 2016-01-13 01328, 2016

      • stanislas
        LordSputnik: I will add tests about display_alias to my actual tests after commiting this - this way it would make more sense.

        • 2016-01-13 01301, 2016

      • LordSputnik
        stanislas: I thought to myself the other day - there's probably more lines of tests than there are of code :P

        • 2016-01-13 01334, 2016

      • LordSputnik
        but who's testing the tests? :D haha

        • 2016-01-13 01329, 2016

      • ruaok
        search is dead. on it.

        • 2016-01-13 01326, 2016

      • stanislas
        LordSputnik: Yeah, I also thought of this. But then I've asked myself - what is wrong with me ? :)

        • 2016-01-13 01341, 2016

      • stanislas
        (about testing the tests)

        • 2016-01-13 01329, 2016

      • opatel99
        JESUS2099: What do I do about Facebook's react class naming since it is very dynamic?

        • 2016-01-13 01345, 2016

      • bitmap
        zas: which machine were you thinking of using for website requests?

        • 2016-01-13 01311, 2016

      • ruaok
        search fixed.

        • 2016-01-13 01335, 2016

      • ruaok
        bitmap: pingu now has and extra 8GB ram. total 24GB.

        • 2016-01-13 01351, 2016

      • ruaok
        its total number of clients it can handle should be increased accordingly.

        • 2016-01-13 01317, 2016

      • bitmap
        hm, I thought zas had already put it in rotation. it's commented on ernie

        • 2016-01-13 01359, 2016

      • ruaok
        that was from post RAID fix.

        • 2016-01-13 01319, 2016

      • ruaok
        I believe earlier he did some updates, but was waiting for you to put it into rotation

        • 2016-01-13 01325, 2016

      • stanislas
        LordSputnik: What do you think https://github.com/bookbrainz/bookbrainz-ws/compa… ?

        • 2016-01-13 01325, 2016

      • bitmap
        nm, it's uncommented on bert and appears to be handling requests

        • 2016-01-13 01334, 2016

      • LordSputnik
        stanislas: hmm, why did you use -1 as a default value?

        • 2016-01-13 01356, 2016

      • stanislas
        because I didn't know if I can use 0

        • 2016-01-13 01303, 2016

      • stanislas
        And ... why not ?

        • 2016-01-13 01324, 2016

      • bitmap
        so it's currently at weight=20 on bert, same as asterix and astro, and has been handling requests for about 1.5 hours

        • 2016-01-13 01341, 2016

      • LordSputnik
        stanislas: Well, generally it's best to avoid "magic numbers" if the language has something to represent "nothing" - in this case, Python has None, so I'd prefer that

        • 2016-01-13 01304, 2016

      • zas
        bitmap: yes

        • 2016-01-13 01306, 2016

      • LordSputnik
        And then directly check if the user_id is None, and avoid making the query if it is :)

        • 2016-01-13 01323, 2016

      • zas
        it is in the same class of astro/asterix now

        • 2016-01-13 01340, 2016

      • zas
        i put it in rotation to ensure it is working as it should, and it looks it is

        • 2016-01-13 01349, 2016

      • stanislas
        LordSputnik: ok, I will change it to None, and there is bug in get_entity_language_ids

        • 2016-01-13 01304, 2016

      • zas
        i think we could slow machines for websites, either lolo or dagon

        • 2016-01-13 01322, 2016

      • bitmap
        agreed

        • 2016-01-13 01351, 2016

      • LordSputnik
        stanislas: another thing is I think get_display_alias should return None if it can't find one, then the rest of the code should deal with that specially, rather than constructing a placeholder untitled alias

        • 2016-01-13 01310, 2016

      • zas
        insert "use" ^ ..

        • 2016-01-13 01344, 2016

      • LordSputnik
        stanislas: another way you could optimise it is by changing your find_alias_if function to run the function in the query rather than in Python, after the results have been fetched

        • 2016-01-13 01346, 2016

      • stanislas
        LordSputnik: But 'If there are no aliases, use "(untitled)" or "(unnamed)" ' :)

        • 2016-01-13 01341, 2016

      • LordSputnik
        Hmm, you can still do that without constructing an Alias ;)

        • 2016-01-13 01304, 2016

      • LordSputnik
        Just deal with it in the JSON function rather than the get_display_alias function

        • 2016-01-13 01305, 2016

      • stanislas
        LordSputnik: Ok, so I will return None

        • 2016-01-13 01308, 2016

      • zas
        how do we proceed ? i propose to change ernie/bert nginx config to route requests; matching ^/ws/[12]/ and using two upstream sets

        • 2016-01-13 01338, 2016

      • zas
        so we keep exact same setup on all machines, but just exclude/include requests from one set or the other

        • 2016-01-13 01357, 2016

      • LordSputnik
        stanislas: That way, if someone wants to use get_display_alias somewhere else, they can tell if they have a valid Alias or nothing - at the moment, there's no way to tell if the stored alias is actually "(untitled)"

        • 2016-01-13 01303, 2016

      • bitmap
        zas: I was thinking the same thing

        • 2016-01-13 01343, 2016

      • bitmap
        if you can do that, I can remove the musicbrainz-ws services

        • 2016-01-13 01347, 2016

      • stanislas
        LordSputnik: So I should just make {'display_alias':None} in return JSON ?

        • 2016-01-13 01356, 2016

      • stanislas
        * returned JSON

        • 2016-01-13 01312, 2016

      • ruaok
        whatever happened to the split where some instances do ws traffic and some do web page traffic? do we still do that?

        • 2016-01-13 01358, 2016

      • opatel99
        Freso: JESUS2099 ping regarding reactjs being ehhh

        • 2016-01-13 01301, 2016

      • bitmap
        I'm not sure we ever did that, at least while I was here

        • 2016-01-13 01345, 2016

      • bitmap
        what we have now is separate unix sockets on each instance, one for ws traffic and another for website traffic

        • 2016-01-13 01344, 2016

      • LordSputnik
        stanislas: yeah I think that would be better actually

        • 2016-01-13 01356, 2016

      • ruaok
        yes, that.

        • 2016-01-13 01327, 2016

      • ruaok
        remind me what the goal for that split was. I thought it was to ensure that web traffic always got better loading speed.

        • 2016-01-13 01333, 2016

      • stanislas
        LordSputnik: How do you imagine doing it outside python ? What do you mean by that ?

        • 2016-01-13 01354, 2016

      • stanislas
        LordSputnik: I am not doing any SQL queries here.

        • 2016-01-13 01304, 2016

      • stanislas
        (on aliases

        • 2016-01-13 01329, 2016

      • bitmap
        ruaok: the goal there was to make sure the ws traffic couldn't use up all available socket connections, and leave none for the website

        • 2016-01-13 01332, 2016

      • LordSputnik
        stanislas: entity_data.aliases is technically doing a query

        • 2016-01-13 01359, 2016

      • LordSputnik
        Ignore what I said though, it'd be adding a load of complexity for not much benerfit - premature optimisation ;)

        • 2016-01-13 01321, 2016

      • ruaok
        DB sockets or overall sockets?

        • 2016-01-13 01333, 2016

      • zas
        bitmap: was it ? i thought it was to use a server with less load to serve website only requests

        • 2016-01-13 01355, 2016

      • zas
        we currently have no exhaustion of sockets

        • 2016-01-13 01333, 2016

      • bitmap
        ruaok: whatever the connection limit for for unix sockets is

        • 2016-01-13 01304, 2016

      • ruaok
        odd. something there doesn't make sense to me.

        • 2016-01-13 01320, 2016

      • ruaok
        but we are not favoring web traffic, so we should start doing that.

        • 2016-01-13 01342, 2016

      • zas
        i propose to test it: https://github.com/metabrainz/nginx/blob/master/c…

        • 2016-01-13 01344, 2016

      • ruaok
        have you two agreed on a plan yet? if so, I'd love to hear it.

        • 2016-01-13 01348, 2016

      • bitmap
        zas: that's the current goal

        • 2016-01-13 01354, 2016

      • ruaok reads

        • 2016-01-13 01329, 2016

      • zas
        i propose to create a new upstream named mbwebsite or smt

        • 2016-01-13 01305, 2016

      • zas
        and use ngs upstream only for locations matching ^/ws/[12]/

        • 2016-01-13 01336, 2016

      • ruaok
        which machines belong to which of the two pools?

        • 2016-01-13 01358, 2016

      • zas
        website=dagon with lolo as spare

        • 2016-01-13 01328, 2016

      • zas
        ngs=asterix astro pingu lolo (when not used as website)

        • 2016-01-13 01353, 2016

      • zas
        though the nginx config may need a bit of cleanup

        • 2016-01-13 01308, 2016

      • zas
        https://github.com/metabrainz/nginx/blob/master/c… ??

        • 2016-01-13 01335, 2016

      • ruaok
        smells like JSB

        • 2016-01-13 01304, 2016

      • zas
        JSB?

        • 2016-01-13 01311, 2016

      • zas
        ;)

        • 2016-01-13 01315, 2016

      • ruaok
        johann sebastian bach

        • 2016-01-13 01317, 2016

      • ruaok
        and mozard.

        • 2016-01-13 01320, 2016

      • ruaok
        *t

        • 2016-01-13 01323, 2016

      • zas
        Yes i got it ;)

        • 2016-01-13 01354, 2016

      • zas
        Nice to have him in the team :))

        • 2016-01-13 01301, 2016

      • zas
        basically since the load comes mostly from ws requests, website requests (including static) will be served by a much-less loaded machine

        • 2016-01-13 01313, 2016

      • bitmap
        sorry, I was slightly misremembering things: https://github.com/metabrainz/chef-cookbooks/pull…

        • 2016-01-13 01328, 2016

      • bitmap
        the goal was to prevent worker process exhaustion, not socket connections

        • 2016-01-13 01309, 2016

      • ruaok
        ok, that makes more sense

        • 2016-01-13 01312, 2016

      • bitmap
        so there are always a number of worker process that can't be hogged by the ws

        • 2016-01-13 01345, 2016

      • zas
        but those workers are still on the same machine than ws workers

        • 2016-01-13 01356, 2016

      • zas
        so we could set 100% workers on machines dedicated to a role

        • 2016-01-13 01322, 2016

      • zas
        if we split on load balancer instead

        • 2016-01-13 01333, 2016

      • bitmap
        right, once the machines are partitioned we can revert that PR

        • 2016-01-13 01317, 2016

      • bitmap
        it might make sense on pino, still, but beta doesn't get a whole lot of traffic anyway

        • 2016-01-13 01352, 2016

      • zas
        i would not revert it, just change the ratio where it applies

        • 2016-01-13 01347, 2016

      • zas
        but instead of configuring from cookbooks we can use the envdir trick

        • 2016-01-13 01358, 2016

      • bitmap
        hm? if the traffic is partitioned by machine, then only one of the services will be used

        • 2016-01-13 01312, 2016

      • zas
        wait, what do you call services in this case ?

        • 2016-01-13 01356, 2016

      • bitmap
        starman, I guess. we run two different starman services (musicbrainz-server, musicbrainz-ws) which each have their own worker procs

        • 2016-01-13 01321, 2016

      • bitmap
        then the traffic is partitioned in nginx on the app server, to separate sockets: https://github.com/metabrainz/chef-cookbooks/blob…

        • 2016-01-13 01352, 2016

      • zas
        Ok, this is what i understood. So on each server atm, we have N workers for musicbrainz-server and M workers for musicbrainz-ws

        • 2016-01-13 01332, 2016

      • zas
        And this is defined by cookbook attributes

        • 2016-01-13 01301, 2016

      • bitmap nods

        • 2016-01-13 01306, 2016

      • zas
        If we have a machine doing only ws, we only need musicbrainz-ws ?

        • 2016-01-13 01300, 2016

      • zas
        N=0 M=max

        • 2016-01-13 01339, 2016

      • bitmap
        yeah, the musicbrainz-server one wouldn't get used, based on the current config

        • 2016-01-13 01346, 2016

      • stanislas
        LordSputnik: Sumbitted the task for review, I will add tests for it in my the master.

        • 2016-01-13 01349, 2016

      • bitmap
        to me it makes sense to have only one service, if the machines are only serving one type of traffic anyway

        • 2016-01-13 01354, 2016

      • LordSputnik
        stanislas: OK, I'll review it now

        • 2016-01-13 01300, 2016

      • zas
        I think we could change daemontools run scripts a bit, so nprocs value is read from env dir instead (and attributes used to set this default value)

        • 2016-01-13 01304, 2016

      • stanislas
        LordSputnik: One more thing

        • 2016-01-13 01323, 2016

      • stanislas
        LordSputnik: How can I abort if user_id is not valid ?

        • 2016-01-13 01347, 2016

      • zas
        then run script could check if the value is 0, and just sleep, rechecking for changed value on regular basis.

        • 2016-01-13 01323, 2016

      • stanislas
        just abort(404) ?

        • 2016-01-13 01326, 2016

      • LordSputnik
        stanislas: should just be able to call abort(<error_code>)

        • 2016-01-13 01328, 2016

      • zas
        It would be more dynamic, and would not require to re-run cookbooks for temporary changes

        • 2016-01-13 01336, 2016

      • bitmap
        ok, and then we could keep the current split on pino. that makes sense

        • 2016-01-13 01350, 2016

      • LordSputnik
        stanislas: Hmm, although I think it would be best to just ignore the user_id in that case

        • 2016-01-13 01300, 2016

      • LordSputnik
        stanislas: don't forget to change the default user_id from -1 to None

        • 2016-01-13 01324, 2016

      • zas
        Ok that's late for me, i can't think much more, but i'll have a look at changes needed tomorrow morning.

        • 2016-01-13 01322, 2016

      • JonnyJD has quit

        • 2016-01-13 01332, 2016

      • ruaok
        dwni said they are available for the baron downgrade soon.

        • 2016-01-13 01336, 2016

      • ruaok
        no early bed for me

        • 2016-01-13 01351, 2016

      • zas
        If a service is totally disabled, no request should arrive to the server in question

        • 2016-01-13 01305, 2016

      • zas
        i mean no request it isn't able to serve

        • 2016-01-13 01327, 2016

      • stanislas
        LordSputnik: Should be ok now

        • 2016-01-13 01355, 2016

      • zas
        ruaok: i'm not going to bed yet either, but i will not modify current nginx confs, it is more likely to break ;))

        • 2016-01-13 01339, 2016

      • ruaok
        wise

        • 2016-01-13 01352, 2016

      • zas
        and to be frank there's a Bowie concert on Arte starting now, i would like to see it (it was the last from Ziggy period, in 1973)

        • 2016-01-13 01303, 2016

      • ruaok
        even wiser.

        • 2016-01-13 01315, 2016

      • JESUS2099
        zag: enjoy it, i have that dvd, it’s great. :)

        • 2016-01-13 01332, 2016

      • JESUS2099
        opatel99: sorry i don’t know neither facebook nor react