#metabrainz

/

      • chirlu`
        gcilou: CatCat’s cat also likes to write something on IRC from time to time.
      • 2015-12-27 36159, 2015

      • ruaok
        zas: so much backscroll, so many distractions. do you have a link to a proposed fix?
      • 2015-12-27 36108, 2015

      • chirlu`
      • 2015-12-27 36114, 2015

      • gcilou
        chirlu`: She was trying to nap and bathe on top of my computer. She just fell off my desk, oh well
      • 2015-12-27 36155, 2015

      • gcilou
        lol looks like cat language to me chirlu`
      • 2015-12-27 36109, 2015

      • zas
        ruaok: no fix (i expected a java dev to provide one...), but here is the explanation of the bug
      • 2015-12-27 36114, 2015

      • zas
      • 2015-12-27 36158, 2015

      • zas
        This is one of the string returned by search server when rate limiter warns for over limit
      • 2015-12-27 36126, 2015

      • zas
      • 2015-12-27 36144, 2015

      • reosarevok
        ruaok: no idea what playing biff means, heh. But it's holiday season, I imagine they won't be shocked if it is slow now
      • 2015-12-27 36150, 2015

      • zas
        is where place holders are replaced by actual values
      • 2015-12-27 36100, 2015

      • reosarevok
        So if you're busy, I imagine they can wait :)
      • 2015-12-27 36111, 2015

      • zas
      • 2015-12-27 36131, 2015

      • zas
        is where answer from rate limiter is splitted by SPACE
      • 2015-12-27 36135, 2015

      • reosarevok lets zas talk his thing now
      • 2015-12-27 36103, 2015

      • zas
        response is stored in a byte buffer at https://bitbucket.org/metabrainz/search-server/sr…
      • 2015-12-27 36105, 2015

      • reosarevok
        ... oh ffs. "Phishing attack ahead Attackers on ia601303.us.archive.org might try to trick you to steal your information (for example, passwords, messages, or credit cards)."
      • 2015-12-27 36132, 2015

      • zas
        which is incorrectly converted to a String at https://bitbucket.org/metabrainz/search-server/sr…
      • 2015-12-27 36119, 2015

      • zas
        that's the bug: string ends with nul chars (from the byte buffer), which are never trimmed
      • 2015-12-27 36100, 2015

      • zas is debugging java code .... something is seriously wrong in our organization ;)
      • 2015-12-27 36147, 2015

      • zas
        this has the following effect: nginx proxy detects invalid header in response from search server (of course ... nul bytes in it)
      • 2015-12-27 36120, 2015

      • zas
        and therefore incorrectly interpret this as a misbehavior of the search server, eventually marking it as failed
      • 2015-12-27 36123, 2015

      • CatQuest joined the channel
      • 2015-12-27 36153, 2015

      • zas
        then 10 seconds later, it is marked again as healthy, etc ...
      • 2015-12-27 36127, 2015

      • zas
        ruaok: got it ?
      • 2015-12-27 36135, 2015

      • ruaok was reading docs and nows returns to backscrolls
      • 2015-12-27 36127, 2015

      • stanislas
        LordSputnik, Leftmost: When posting a creator with language_id, which doesn't link to any language created, there is an exception, and not a 400
      • 2015-12-27 36127, 2015

      • ruaok
        I get the problem, for sure.
      • 2015-12-27 36140, 2015

      • stanislas
        LordSputnik, Leftmost: It shouldn't work that way, should it ?
      • 2015-12-27 36159, 2015

      • opatel99
        LordSputnik: Is the alias function supposed to work on the book titles or the author names? Both?
      • 2015-12-27 36128, 2015

      • ruaok
        zas: so line 124 where it constructs the response is incorrect since it takes data from a string, when it should be copying bytes from the byte buffer.
      • 2015-12-27 36139, 2015

      • kinshukk has left the channel
      • 2015-12-27 36126, 2015

      • chirlu`
        Not line 122, where it turns the whole buffer into a string?
      • 2015-12-27 36142, 2015

      • zas
        ruaok: line 124 removes the id (an integer), then uses the rest as response
      • 2015-12-27 36155, 2015

      • zas
        yes, line 122 is the culprit
      • 2015-12-27 36107, 2015

      • zas
        it converts a byte buffer in a String
      • 2015-12-27 36112, 2015

      • ruaok
        understood.
      • 2015-12-27 36117, 2015

      • zas
      • 2015-12-27 36134, 2015

      • ruaok
        I'm looking at the startswith() function.
      • 2015-12-27 36144, 2015

      • ruaok
        doing that on a byte array would be a pain.
      • 2015-12-27 36155, 2015

      • zas
        the startswith() works imho
      • 2015-12-27 36109, 2015

      • ruaok
        I'm inclined to leave 122 and then fix 124 by appending from the bytearray, not the string.
      • 2015-12-27 36113, 2015

      • zas
        we just have to remove trailing nul chars
      • 2015-12-27 36153, 2015

      • ruaok
        or copy the exact number of characters.
      • 2015-12-27 36111, 2015

      • zas
        yes
      • 2015-12-27 36146, 2015

      • blueflute19
        my task is writing a review, when submitting should I copy paste it onto the submission or place a link to the review?
      • 2015-12-27 36147, 2015

      • zas
        ruaok: there are 3 unmerged PRs in search server project, did you have a look at them ?
      • 2015-12-27 36107, 2015

      • blueflute19
        sorry, this is my first task
      • 2015-12-27 36119, 2015

      • ruaok
        briefly. none of them looked promising, so I kept working on other approaches instead.
      • 2015-12-27 36153, 2015

      • zas
        it seems to me they are mostly cleanups
      • 2015-12-27 36104, 2015

      • chirlu`
        There is a matching String constructor: String(byte[] bytes, int offset, int length)
      • 2015-12-27 36121, 2015

      • chirlu`
      • 2015-12-27 36118, 2015

      • chirlu`
        So changing the line to “String result = new String(dpReceive.getData(), dpReceive.getOffset(), dpReceive.getLength());” might do it.
      • 2015-12-27 36157, 2015

      • ruaok
        so. much. useless. code.
      • 2015-12-27 36149, 2015

      • ruaok
        not sure about the getOffset()
      • 2015-12-27 36159, 2015

      • ruaok
        shouldn't it always be 0?
      • 2015-12-27 36138, 2015

      • Gentlecat
        blueflute19: you mean GCI website? just put a link there
      • 2015-12-27 36153, 2015

      • blueflute19
        Gentlecat: yup I do, thanks!
      • 2015-12-27 36105, 2015

      • chirlu`
        ruaok: Maybe, but I don’t see there is any guarantee on that.
      • 2015-12-27 36108, 2015

      • reosarevok
        blueflute19: the review should be on the CritiqueBrainz site, and then you should link to the review on the GCI page, and to wherever in your social media you've shared it (if I'm remembering the task correctly)
      • 2015-12-27 36105, 2015

      • ruaok
        I guess I am just confused about what that really means.
      • 2015-12-27 36112, 2015

      • blueflute19
        reosarevok: is there anyway to unsubmit a task? I forgot to link to the social media
      • 2015-12-27 36120, 2015

      • chirlu`
        From receive() documentation: “The length field of the datagram packet object contains the length of the received message. If the message is longer than the packet's length, the message is truncated.”
      • 2015-12-27 36120, 2015

      • opatel99
        reosarevok: Does alias sorting apply to books too?
      • 2015-12-27 36122, 2015

      • reosarevok
        Just comment again
      • 2015-12-27 36126, 2015

      • reosarevok
        blueflute19 ^
      • 2015-12-27 36131, 2015

      • chirlu`
        Now I’m confused.
      • 2015-12-27 36153, 2015

      • ruaok
        since the transmission will be complete by the time the code executes, I'm not sure what offset would or should be.
      • 2015-12-27 36154, 2015

      • ruaok
        heh.
      • 2015-12-27 36100, 2015

      • blueflute19
        reosarevok: got it, thanks haha
      • 2015-12-27 36103, 2015

      • chirlu`
        Does that mean that the length field may be more than the actual buffer length?
      • 2015-12-27 36153, 2015

      • reosarevok
        opatel99: I have no idea what this is about :) In MusicBrainz we have some degree of sorting for song alias sort-names and the like ("The" moves to the end and stuff like that) so I would expect something like that in BB but I don't really know
      • 2015-12-27 36141, 2015

      • reosarevok
        (basically our artist sort names are specially convoluted but we do sort every kind of alias - in English it rarely changes much but for example in Japanese apparently it changes script for sorting)
      • 2015-12-27 36130, 2015

      • opatel99
        I think I got the names down, but titles of entities like Book Titles is a different story.
      • 2015-12-27 36116, 2015

      • opatel99
      • 2015-12-27 36119, 2015

      • reosarevok
      • 2015-12-27 36110, 2015

      • ruaok
        chirlu`: the way I read that, no, the buffer len should never be longer.
      • 2015-12-27 36105, 2015

      • chirlu`
        Would make sense, but I think the wording is ambiguous.
      • 2015-12-27 36143, 2015

      • ruaok
        for sure.
      • 2015-12-27 36159, 2015

      • chirlu`
        In any case, DatagramSocket.receive() doesn’t promise it will start at offset 0. It probably will, but.
      • 2015-12-27 36102, 2015

      • ruaok
        well, I am compiling with your suggested line, even though the getoffset confuses/worries me
      • 2015-12-27 36120, 2015

      • ruaok
        chances are it will always be offset 0, since our message is so short.
      • 2015-12-27 36112, 2015

      • blueflute19 has quit
      • 2015-12-27 36118, 2015

      • opatel99
        Jane Austen -> Austen, Jane
      • 2015-12-27 36118, 2015

      • opatel99
        J. R. R. Tolkien -> Tolkien, J. R. R.
      • 2015-12-27 36118, 2015

      • opatel99
        Dr. Phil McGraw -> McGraw, Dr. Phil
      • 2015-12-27 36118, 2015

      • opatel99
        The Hunger Games -> Hunger Games, The
      • 2015-12-27 36122, 2015

      • opatel99
        Oops.
      • 2015-12-27 36152, 2015

      • opatel99
        Are those pretty much all the general cases I have to consider?
      • 2015-12-27 36153, 2015

      • ruaok
        zas: change builds fine. not sure if there are tests for the ratelimiter.
      • 2015-12-27 36159, 2015

      • ruaok
        but, I can't see how this could make things worse.
      • 2015-12-27 36154, 2015

      • ruaok
        shall we deploy this?
      • 2015-12-27 36108, 2015

      • ruaok
        and do a wireshark verification to ensure the fix is correct?
      • 2015-12-27 36102, 2015

      • zas
        yes
      • 2015-12-27 36115, 2015

      • zas
        we can simply check logs
      • 2015-12-27 36119, 2015

      • baykelper joined the channel
      • 2015-12-27 36129, 2015

      • zas
        number of errors should be reduced heavily
      • 2015-12-27 36113, 2015

      • zas
        /var/log/nginx/search-public-error.log on ernie/bert
      • 2015-12-27 36146, 2015

      • ruaok
        ok, is updated and pushed. bake it so.
      • 2015-12-27 36122, 2015

      • ruaok
        #search crash.
      • 2015-12-27 36124, 2015

      • ruaok
        I'm on it. :)
      • 2015-12-27 36128, 2015

      • zas
        k
      • 2015-12-27 36129, 2015

      • zas
        did you deploy the change already ?
      • 2015-12-27 36144, 2015

      • ruaok
        no.
      • 2015-12-27 36128, 2015

      • ruaok
        I don't recall how to deploy, wanna do it or give me a tip where to start?
      • 2015-12-27 36141, 2015

      • zas
        you commit it ?
      • 2015-12-27 36107, 2015

      • ruaok
        yep
      • 2015-12-27 36110, 2015

      • zas
        i'll do it
      • 2015-12-27 36115, 2015

      • ruaok
        4.10.4 branch
      • 2015-12-27 36111, 2015

      • zas
        rebuildiing on dora, using ./update.sh
      • 2015-12-27 36114, 2015

      • zas
        ok; done on dora, proceeding on roobarb
      • 2015-12-27 36143, 2015

      • ruaok
        deployed as well?
      • 2015-12-27 36148, 2015

      • ruaok
        or just compiled?
      • 2015-12-27 36110, 2015

      • reosarevok
        Are we hoping for this to be an actual full fix, a stability improvement, or?
      • 2015-12-27 36112, 2015

      • zas
        roobarb restarting atm
      • 2015-12-27 36104, 2015

      • zas
        upstream sent invalid header while reading response header from upstream
      • 2015-12-27 36111, 2015

      • zas
        ...
      • 2015-12-27 36128, 2015

      • zas
        ruaok: are you sure about the fix ?
      • 2015-12-27 36143, 2015

      • ruaok
        doesn't look good does it. no, we're not.
      • 2015-12-27 36102, 2015

      • ruaok
        can you please capture a wireshark output again?
      • 2015-12-27 36159, 2015

      • Nyanko-sensei has quit
      • 2015-12-27 36100, 2015

      • D4RK-PH0ENiX has quit
      • 2015-12-27 36105, 2015

      • zas
        done
      • 2015-12-27 36127, 2015

      • zas
        /home/zas/cap_10.1.1.22_8081.tcpdump2 on ernie
      • 2015-12-27 36133, 2015

      • zas
        500 packets
      • 2015-12-27 36156, 2015

      • opatel99
        Leftmost: LordSputnik Ping?
      • 2015-12-27 36152, 2015

      • ruaok
        confirmed, not fixed.
      • 2015-12-27 36115, 2015

      • ruaok tries it another way
      • 2015-12-27 36143, 2015

      • zas
        what about String result = new String(dpReceive.getData()).substring(0, dpReceive.getLength());
      • 2015-12-27 36143, 2015

      • chirlu`
        zas: Can you catch a response from the limiter service and see whether the NULs are present in its response already?
      • 2015-12-27 36138, 2015

      • chirlu` wonders where the code for the limiter is.
      • 2015-12-27 36113, 2015

      • zas
      • 2015-12-27 36122, 2015

      • zas
        and no it doesnt include nul chars
      • 2015-12-27 36145, 2015

      • chirlu`
        Ah, private repo apparently.
      • 2015-12-27 36140, 2015

      • zas
        i wonder why, i see no reason for it to be private (ruaok ??)
      • 2015-12-27 36156, 2015

      • ruaok
        none, let's open it.
      • 2015-12-27 36117, 2015

      • ruaok
      • 2015-12-27 36130, 2015

      • ruaok
        that is my approach. since it doesn't involve going to a string first.
      • 2015-12-27 36137, 2015

      • zas
        chirlu`: it should be public now
      • 2015-12-27 36126, 2015

      • mndhck has quit
      • 2015-12-27 36112, 2015

      • zas
        ruaok: it looks ok to me
      • 2015-12-27 36136, 2015

      • zas
        you can even directly return instead of using intermediate rlr var
      • 2015-12-27 36112, 2015

      • opatel99
        How does one bypass the Magic Number test for indexOf > -1?
      • 2015-12-27 36115, 2015

      • ruaok
        doesn't compile yet, but mexican food just fell from the heavens
      • 2015-12-27 36140, 2015

      • ariscop joined the channel
      • 2015-12-27 36142, 2015

      • chirlu`
        Hm. This also uses new String(...): https://stackoverflow.com/questions/8229064/how-t…
      • 2015-12-27 36121, 2015

      • ruaok
        man, mexican hot sauce is better than coffee for waking you up!
      • 2015-12-27 36110, 2015

      • chirlu`
        Well, it won’t compile because RateLimiterResponse expects a String.
      • 2015-12-27 36136, 2015

      • ruaok
        and the copyOrRange() was not qualified.
      • 2015-12-27 36158, 2015

      • chirlu`
        But I don’t see how this is doing it differently than the current code, except that it ignores the offset.