#metabrainz

/

      • ruaok
        3 failures? how will you live with that samj1912?

        • 2018-06-14 16526, 2018

      • ruaok
        :D

        • 2018-06-14 16557, 2018

      • samj1912
        peacefully :P

        • 2018-06-14 16529, 2018

      • dragonzeron has quit

        • 2018-06-14 16542, 2018

      • ruaok
        good plan. be buddha.

        • 2018-06-14 16556, 2018

      • ruaok
        wow, a peak of 60 result cache hits?

        • 2018-06-14 16501, 2018

      • ruaok
        thats a lot higher than I expected.

        • 2018-06-14 16511, 2018

      • ruaok
        that's gotta help a lot.

        • 2018-06-14 16542, 2018

      • samj1912
        zas: https://stats.metabrainz.org/d/T4MODrIiz/solr-clo…

        • 2018-06-14 16548, 2018

      • ruaok
        zas, samj1912: one test I would be interested in doing is to see if we can get a smaller cloud instance to handle the traffic.

        • 2018-06-14 16551, 2018

      • samj1912
        how come it stopped after 12:30?

        • 2018-06-14 16510, 2018

      • ruaok
        because three nearly idle instances that are required to run solr cloud seems less than ideal.

        • 2018-06-14 16506, 2018

      • samj1912
        hmm, looks like the replayer stopped?

        • 2018-06-14 16554, 2018

      • ruaok
        the disk usage stat seems broken

        • 2018-06-14 16514, 2018

      • ruaok
        and during peak the one node could't keep up -- surprising it only got 3 errors.

        • 2018-06-14 16516, 2018

      • ruaok
        !m samj1912

        • 2018-06-14 16516, 2018

      • BrainzBot
        You're doing good work, samj1912!

        • 2018-06-14 16545, 2018

      • samj1912
        ruaok: its not?

        • 2018-06-14 16500, 2018

      • samj1912
        its the disk space usage?

        • 2018-06-14 16517, 2018

      • ruaok
        oh, perhaps so.

        • 2018-06-14 16520, 2018

      • samj1912
        but yeah, the replayer stopped at 12:30

        • 2018-06-14 16522, 2018

      • ruaok
        I see disk io.

        • 2018-06-14 16526, 2018

      • samj1912
        weird

        • 2018-06-14 16537, 2018

      • ruaok
        no traceback on the replay?

        • 2018-06-14 16545, 2018

      • samj1912
        nope

        • 2018-06-14 16553, 2018

      • ruaok
        log rotation.

        • 2018-06-14 16555, 2018

      • samj1912
        oh maybe during the log rotation

        • 2018-06-14 16556, 2018

      • samj1912
        yeah

        • 2018-06-14 16559, 2018

      • ruaok
        input files rotated from under it

        • 2018-06-14 16500, 2018

      • samj1912
        was gonna say the same :P

        • 2018-06-14 16501, 2018

      • UmkaDK joined the channel

        • 2018-06-14 16506, 2018

      • ruaok
        catch the exception and restart.

        • 2018-06-14 16513, 2018

      • samj1912
        no exception though

        • 2018-06-14 16513, 2018

      • zas
        i'm off for lunch, haproxy machines are untuned yet, but should work

        • 2018-06-14 16532, 2018

      • samj1912
        I made a symbolic link to the access log

        • 2018-06-14 16538, 2018

      • samj1912
        probably that's why

        • 2018-06-14 16540, 2018

      • ruaok
        I'm deeply impressed that that it nearly handled the whole load.

        • 2018-06-14 16551, 2018

      • ruaok
        yeah, that would do it too.

        • 2018-06-14 16508, 2018

      • ruaok
        a restart *should* work once the new log appears

        • 2018-06-14 16526, 2018

      • yvanzo
        ruaok: pong -- will submit eval this evening

        • 2018-06-14 16542, 2018

      • ruaok
        ok, thanks!

        • 2018-06-14 16511, 2018

      • samj1912
        ruaok: peak avg. hit rate was 81r/s

        • 2018-06-14 16543, 2018

      • samj1912
        so the actual peak reqs/s might have been higher

        • 2018-06-14 16543, 2018

      • ruaok
        ok, keep the replay going and then also turn beta to the new solr cluster once the haproxy stuff is working.

        • 2018-06-14 16500, 2018

      • samj1912
        okay, what do you wanna do about the indexing?

        • 2018-06-14 16522, 2018

      • ruaok
        have you tried indexing on queen since the memory leak fix?

        • 2018-06-14 16526, 2018

      • samj1912
        nope

        • 2018-06-14 16532, 2018

      • UmkaDK_ has quit

        • 2018-06-14 16557, 2018

      • samj1912
        let me get all the nodes up and ready

        • 2018-06-14 16558, 2018

      • ruaok
        try it and watch it carefully. at the first sign on the apocalypse, stop it.

        • 2018-06-14 16502, 2018

      • samj1912
        yup

        • 2018-06-14 16503, 2018

      • ruaok
        of

        • 2018-06-14 16516, 2018

      • ruaok
        it would be ideal to get it updating from production.

        • 2018-06-14 16525, 2018

      • zas
        https://github.com/metabrainz/mb-solr-cloud/blob/…

        • 2018-06-14 16542, 2018

      • samj1912
        zas: how do you update the solr cloud repos on the machine?

        • 2018-06-14 16549, 2018

      • samj1912
        it always gives me an error on pull

        • 2018-06-14 16553, 2018

      • samj1912
        public key denied

        • 2018-06-14 16516, 2018

      • Lotheric has quit

        • 2018-06-14 16501, 2018

      • zas
        do you connect with ssh -A ?

        • 2018-06-14 16550, 2018

      • samj1912
        ah no

        • 2018-06-14 16518, 2018

      • KassOtsimine has quit

        • 2018-06-14 16542, 2018

      • KassOtsimine joined the channel

        • 2018-06-14 16542, 2018

      • KassOtsimine has quit

        • 2018-06-14 16542, 2018

      • KassOtsimine joined the channel

        • 2018-06-14 16531, 2018

      • Lotheric joined the channel

        • 2018-06-14 16515, 2018

      • m0n0g0n joined the channel

        • 2018-06-14 16556, 2018

      • D4RK-PH0ENiX has quit

        • 2018-06-14 16531, 2018

      • Lotheric has quit

        • 2018-06-14 16533, 2018

      • drsaunders joined the channel

        • 2018-06-14 16529, 2018

      • pingupingu joined the channel

        • 2018-06-14 16529, 2018

      • pingupingu has quit

        • 2018-06-14 16529, 2018

      • pingupingu joined the channel

        • 2018-06-14 16546, 2018

      • Leo__Verto has quit

        • 2018-06-14 16529, 2018

      • zas
        samj1912: ping

        • 2018-06-14 16500, 2018

      • samj1912
        zas: pong

        • 2018-06-14 16521, 2018

      • zas
        did you enable solr2 & 3 already?

        • 2018-06-14 16526, 2018

      • samj1912
        yup

        • 2018-06-14 16541, 2018

      • samj1912
        setting solr cloud up for beta

        • 2018-06-14 16551, 2018

      • zas
        can you do a bunch of requests to the floating IP 195.201.47.148 ?

        • 2018-06-14 16505, 2018

      • Monkey has quit

        • 2018-06-14 16520, 2018

      • zas
        i'm setting up telegraf for haproxy, i need few numbers

        • 2018-06-14 16532, 2018

      • samj1912
        requests like normal solr ones?

        • 2018-06-14 16542, 2018

      • zas
        yes

        • 2018-06-14 16546, 2018

      • samj1912
        okay

        • 2018-06-14 16508, 2018

      • samj1912
        zas: fyi I have deleted all the indexes

        • 2018-06-14 16513, 2018

      • samj1912
        so it wont show any results

        • 2018-06-14 16519, 2018

      • samj1912
        but it will respond

        • 2018-06-14 16541, 2018

      • samj1912
        zas: which port?

        • 2018-06-14 16548, 2018

      • zas
        80

        • 2018-06-14 16550, 2018

      • zas
        http

        • 2018-06-14 16508, 2018

      • Monkey joined the channel

        • 2018-06-14 16515, 2018

      • Monkey has quit

        • 2018-06-14 16523, 2018

      • samj1912
        zas: done

        • 2018-06-14 16546, 2018

      • samj1912
        around 20k reqs

        • 2018-06-14 16550, 2018

      • Monkey joined the channel

        • 2018-06-14 16501, 2018

      • samj1912
        let me know if you need more

        • 2018-06-14 16522, 2018

      • zas
        how long did it take on your side ?

        • 2018-06-14 16546, 2018

      • samj1912
        18.5 seconds?

        • 2018-06-14 16552, 2018

      • samj1912
        but its not accurate since there is no data

        • 2018-06-14 16505, 2018

      • Monkey has quit

        • 2018-06-14 16514, 2018

      • zas
        ok, it seems to work, but we'll need to improve that with much more data

        • 2018-06-14 16521, 2018

      • samj1912
        doing it right now

        • 2018-06-14 16530, 2018

      • zas
        at least haproxy is reporting stats

        • 2018-06-14 16535, 2018

      • samj1912
        cool

        • 2018-06-14 16537, 2018

      • zas
        and failover seems to work

        • 2018-06-14 16542, 2018

      • samj1912
        where can I see them?

        • 2018-06-14 16558, 2018

      • zas
        https://stats.metabrainz.org/d/T4MODrIiz/solr-clo…

        • 2018-06-14 16528, 2018

      • zas
        but that's just a proof it works, i don't know yet about useful metriics

        • 2018-06-14 16558, 2018

      • zas
        https://cbonte.github.io/haproxy-dconv/1.7/manage…

        • 2018-06-14 16509, 2018

      • samj1912
        cool

        • 2018-06-14 16515, 2018

      • samj1912
        I'll put some data in there

        • 2018-06-14 16540, 2018

      • zas
        btw, i took the smallest instances for load balancer, i'm curious on how it performs

        • 2018-06-14 16547, 2018

      • samj1912
        okay

        • 2018-06-14 16553, 2018

      • zas
        1 vcpu, 2gb ram

        • 2018-06-14 16557, 2018

      • samj1912
        zas: where rmq currently?

        • 2018-06-14 16511, 2018

      • zas
        trille i think

        • 2018-06-14 16555, 2018

      • samj1912
        cool, thanks

        • 2018-06-14 16543, 2018

      • samj1912
        zas: dont start sir on queen

        • 2018-06-14 16530, 2018

      • zas
        k

        • 2018-06-14 16508, 2018

      • rsh7 joined the channel

        • 2018-06-14 16546, 2018

      • samj1912
        zas: I am reindexing off of queen

        • 2018-06-14 16505, 2018

      • samj1912
        load might get high, but ram should be stable

        • 2018-06-14 16501, 2018

      • samj1912
        recordings should finish in 4 hrs acc. to my calculations

        • 2018-06-14 16525, 2018

      • sampsyo has quit

        • 2018-06-14 16540, 2018

      • Lotheric joined the channel

        • 2018-06-14 16526, 2018

      • D4RK-PH0ENiX joined the channel

        • 2018-06-14 16553, 2018

      • D4RK-PH0ENiX has quit

        • 2018-06-14 16517, 2018

      • samj1912
        actually, we might be looking at a record 3hr for recording reindex

        • 2018-06-14 16527, 2018

      • Monkey joined the channel

        • 2018-06-14 16506, 2018

      • D4RK-PH0ENiX joined the channel

        • 2018-06-14 16503, 2018

      • m0n0g0n has quit

        • 2018-06-14 16547, 2018

      • sampsyo joined the channel

        • 2018-06-14 16517, 2018

      • Leo__Verto joined the channel

        • 2018-06-14 16546, 2018

      • MrNIce joined the channel

        • 2018-06-14 16501, 2018

      • MrNIce
        Hello there!

        • 2018-06-14 16500, 2018

      • samj1912
        hey MrNIce

        • 2018-06-14 16541, 2018

      • MrNIce
        Hi Samj1912, can you help me to find out a bad setting or a bug in Picard?

        • 2018-06-14 16519, 2018

      • samj1912
        MrNIce: sure, are you looking to contribute in general?

        • 2018-06-14 16529, 2018

      • samj1912
        or do you have something in mind?

        • 2018-06-14 16508, 2018

      • MrNIce
        I have an issue using Picard and would like to fix it

        • 2018-06-14 16518, 2018

      • samj1912
        cool, what's the bug?

        • 2018-06-14 16557, 2018

      • MrNIce
        MB Picard version 1.4.2 Opensuse Leap 15.0 I have folders sorted by CD with music files inside. I drag 1 folder to middle window (I don't know the name of the window) and I click "Cluster" then I select the title of this CD and click "Lookup in Browser". Some time I get the page in my browser, it's OK but sometime I get in my browser a page with text only "Index of file:///home/username" and the list of my home folder files includ

        • 2018-06-14 16519, 2018

      • MrNIce
        Address bar has: file:///home/username/ When I say sometime, I mean with I CD folder it is always OK but with another it is never OK What does that mean? I did an upgrade from Leap 42.3 to 15.0 recently. I never had that with the previous version, I don't know what MB version was in 42.3.

        • 2018-06-14 16525, 2018

      • samj1912
        MrNIce: we have stopped dev. for 1.4.x

        • 2018-06-14 16534, 2018

      • samj1912
        You should switch to 2.0 beta

        • 2018-06-14 16556, 2018

      • MrNIce
        As said, I run Linux Opensuse leap 15.0, in the repo I have 1.4.2 version. I'll have to install manualy...

        • 2018-06-14 16510, 2018

      • MrNIce
        2.0 is a beta version, when stable will be released?

        • 2018-06-14 16552, 2018

      • samj1912
        MrNIce: all it should take is `pip install https://github.com/metabrainz/picard/archive/mast…

        • 2018-06-14 16503, 2018

      • ruaok
        yay, serge is back

        • 2018-06-14 16531, 2018

      • samj1912
        MrNIce: the beta is pretty stable, we are looking at getting a 2.0 final version out sometime by the end of this month