#metabrainz

/

      • samj1912
        It also houses the largest git repo

        • 2018-06-04 15534, 2018

      • kartikeyaSh
        nice! MS used github so much that it bought github

        • 2018-06-04 15540, 2018

      • samj1912
        Also nice is Nat Friedman, he was the CEO of Xamarin and has been involved in FOSS since quite a while

        • 2018-06-04 15502, 2018

      • hibiscuskazeneko joined the channel

        • 2018-06-04 15514, 2018

      • hibiscuskazeneko
        Hello? Anyone on?

        • 2018-06-04 15532, 2018

      • Clint
        no

        • 2018-06-04 15520, 2018

      • hibiscuskazeneko
        Damn. MB's giving me a bit of trouble.

        • 2018-06-04 15551, 2018

      • hibiscuskazeneko
        I was getting 504s in the midst of attempting to submit a URL for a release

        • 2018-06-04 15511, 2018

      • hibiscuskazeneko
        It looks to have subsided now, but I wouldn't be surprised if it happens again.

        • 2018-06-04 15528, 2018

      • kartikeyaSh
        ruaok: ping

        • 2018-06-04 15511, 2018

      • yvanzo
        zas: number of req/s to the website exploded since 14:00

        • 2018-06-04 15555, 2018

      • yvanzo
        70 vs 20

        • 2018-06-04 15525, 2018

      • D4RK-PH0ENiX has quit

        • 2018-06-04 15555, 2018

      • D4RK-PH0ENiX joined the channel

        • 2018-06-04 15536, 2018

      • hibiscuskazeneko has quit

        • 2018-06-04 15512, 2018

      • hibiscuskazeneko joined the channel

        • 2018-06-04 15519, 2018

      • ruaok
        kartikeyaSh: pong

        • 2018-06-04 15544, 2018

      • Dr-Flay joined the channel

        • 2018-06-04 15549, 2018

      • ruaok
        zas, yvanzo, iliekcomputers, samj1912, reosarevok, Freso: Everyone has now been paid and I expect most of the money to arrive by tomorrow, save for iliekcomputers who should have is almost now. when you do receive the money, please ping me with when exactly it arrives and if you were charged any fees.

        • 2018-06-04 15505, 2018

      • ruaok
        in theory, no one should be charged fees.

        • 2018-06-04 15510, 2018

      • samj1912
        bitmap: ping

        • 2018-06-04 15511, 2018

      • kartikeyaSh
        https://www.irccloud.com/pastebin/NADOYBQP/

        • 2018-06-04 15516, 2018

      • reosarevok
        ruaok: got it, no fees

        • 2018-06-04 15516, 2018

      • ruaok
        today, already? I just sent it 7 hours ago. nice.

        • 2018-06-04 15554, 2018

      • bitmap
        samj1912: pong

        • 2018-06-04 15505, 2018

      • kartikeyaSh
        ruaok: the listens posted above are different according to MSB. but these are same it's just the sequence of the "artist_mbids" that changes in the recording jsons

        • 2018-06-04 15516, 2018

      • samj1912
        bitmap: did you see the above discussion between me and zas?

        • 2018-06-04 15534, 2018

      • Leo_Verto
        zas: do you have a contingency plan for when stats.meb.org ends up on the frontpage of hackernews? because gitlab didn't :P https://news.ycombinator.com/item?id=17223116

        • 2018-06-04 15538, 2018

      • bitmap
        samj1912: not yet

        • 2018-06-04 15552, 2018

      • Leo_Verto
        at least they had two seperate prometheus instances and only the public one crashed

        • 2018-06-04 15500, 2018

      • kartikeyaSh
        so shouldn't we sort them before inserting into the database, to avoid such cases?

        • 2018-06-04 15535, 2018

      • samj1912
        bitmap: https://chatlogs.metabrainz.org/brainzbot/metabra…

        • 2018-06-04 15553, 2018

      • ruaok
        kartikeyaSh: doh. nice catch. yes, we should sort them.

        • 2018-06-04 15514, 2018

      • ruaok
        and these problems exist in the DB right now? if that is the case we need to make a ticket for fixing that.

        • 2018-06-04 15533, 2018

      • kartikeyaSh
        yes these exist in the DB

        • 2018-06-04 15557, 2018

      • kartikeyaSh
        we sort only json keys not the lists

        • 2018-06-04 15559, 2018

      • DunKno[m]
        erm... guys, "Server is being naughty" when accessing listenbrainz.org

        • 2018-06-04 15541, 2018

      • DunKno[m]
        are you colocated on gitlab's servers? :P

        • 2018-06-04 15532, 2018

      • kartikeyaSh
        I'll open a ticket for sorting!

        • 2018-06-04 15549, 2018

      • ruaok
        DunKno[m]: shit, thanks for letting me know. no idea why I didn't get an alert. oops.

        • 2018-06-04 15551, 2018

      • ruaok logs in

        • 2018-06-04 15554, 2018

      • bitmap
        samj1912: you want to get rid of _store and have mbs fill in the data from the database?

        • 2018-06-04 15554, 2018

      • samj1912
        if possible

        • 2018-06-04 15504, 2018

      • samj1912
        it will really make things easier

        • 2018-06-04 15512, 2018

      • samj1912
        and probably more scalable

        • 2018-06-04 15506, 2018

      • hibiscuskazeneko has quit

        • 2018-06-04 15514, 2018

      • samj1912
        rmq trip over?

        • 2018-06-04 15518, 2018

      • samj1912
        sir is also down'

        • 2018-06-04 15521, 2018

      • samj1912
        so is CAA

        • 2018-06-04 15525, 2018

      • samj1912
        *caa-indexer

        • 2018-06-04 15531, 2018

      • Leo_Verto
        ruaok: looks like LB stopped working at nearly exactly 14:00 UTC, maybe a scheduled task like something bigquery caused that?

        • 2018-06-04 15532, 2018

      • samj1912
        serge is down?

        • 2018-06-04 15536, 2018

      • Leo__Verto joined the channel

        • 2018-06-04 15549, 2018

      • samj1912
        yup serge is down :P

        • 2018-06-04 15551, 2018

      • samj1912
        zas: ^

        • 2018-06-04 15553, 2018

      • ruaok
        that long ago?

        • 2018-06-04 15555, 2018

      • ruaok
        fuck!

        • 2018-06-04 15506, 2018

      • samj1912
        yup

        • 2018-06-04 15514, 2018

      • samj1912
        telegram has notifs from around that time

        • 2018-06-04 15519, 2018

      • ruaok
        must be serge problem then.

        • 2018-06-04 15546, 2018

      • samj1912
        rmq stops = shit goes crazy :P

        • 2018-06-04 15508, 2018

      • zas
        i'm back, just now, you broke everything while i was away i see

        • 2018-06-04 15513, 2018

      • Leo__Verto
        haha

        • 2018-06-04 15534, 2018

      • ruaok
        zas: just ordered a reboot of serge.

        • 2018-06-04 15548, 2018

      • ruaok thinks we should've never named a server after anything french

        • 2018-06-04 15556, 2018

      • samj1912
        XD

        • 2018-06-04 15500, 2018

      • zas
        yvanzo: that's googlebot, i reduced its rate in google search console, let's see if it calms down

        • 2018-06-04 15515, 2018

      • ruaok
        I guess we found another single point of failure. :)

        • 2018-06-04 15534, 2018

      • zas
        well, this one was known already ;)

        • 2018-06-04 15546, 2018

      • ruaok
        ah. ok.

        • 2018-06-04 15549, 2018

      • ruaok
        90 seconds, nothing.

        • 2018-06-04 15554, 2018

      • zas
        hmmm

        • 2018-06-04 15556, 2018

      • ruaok
        will step up the reboot mechanism.

        • 2018-06-04 15531, 2018

      • ruaok
        power button reset requested.

        • 2018-06-04 15536, 2018

      • ruaok
        waiting 2 minutes.

        • 2018-06-04 15520, 2018

      • zas
        it stopped to answer at 16:00, no sign of anomaly before (temp ok, mem ok, load ok, etc..)

        • 2018-06-04 15536, 2018

      • ruaok
        serge back.

        • 2018-06-04 15538, 2018

      • ruaok
        LB back,.

        • 2018-06-04 15545, 2018

      • bitmap
        samj1912: is there an easy way to see everything that's needed/get's used from _store?

        • 2018-06-04 15502, 2018

      • bitmap
        gets*

        • 2018-06-04 15516, 2018

      • samj1912
        bitmap: as in?

        • 2018-06-04 15543, 2018

      • zas
        ruaok: i see nothign in logs, it seems it was an hardware failure, it just stopped to work

        • 2018-06-04 15542, 2018

      • outsidecontext has quit

        • 2018-06-04 15518, 2018

      • zas
        Leo_Verto: that's good for gitlab, they get a free load testing ;)

        • 2018-06-04 15515, 2018

      • bitmap
        samj1912: solr converts _store to java objects, then back to xml in some other format, right? but is all of the data in _store actually used (eventually displayed)?

        • 2018-06-04 15517, 2018

      • zas
        ruaok, bitmap, yvanzo, samj1912: i found nothing about the cause of serge crash. All indicators were totally normal just before the event, and nothing stands out from logs either. I suspect a cpu freeze.

        • 2018-06-04 15537, 2018

      • samj1912
        weird

        • 2018-06-04 15543, 2018

      • samj1912
        bitmap: it uses all the fields

        • 2018-06-04 15502, 2018

      • samj1912
        so basically both java and python(sir) use xml bindings of mmd-schema

        • 2018-06-04 15513, 2018

      • samj1912
        sir uses mb-rngpy and java uses jaxb bindings

        • 2018-06-04 15536, 2018

      • samj1912
        sir basically fetches and puts everything that needs to be displayed in the _store field

        • 2018-06-04 15506, 2018

      • samj1912
        solrwriter (which uses the jaxb bindings) then reads the _store field and displays it in xml/json accordingly

        • 2018-06-04 15514, 2018

      • samj1912
        for xml it is uses as in

        • 2018-06-04 15521, 2018

      • samj1912
        for json some adapters are needed

        • 2018-06-04 15530, 2018

      • zas
        About M$ & github, an hint is given in their propaganda at https://blogs.microsoft.com/blog/2018/06/04/micro… : "Finally, we will bring Microsoft’s developer tools and services to new audiences."

        • 2018-06-04 15521, 2018

      • bitmap
        samj1912: okay

        • 2018-06-04 15531, 2018

      • bitmap
        is _store not what solr is indexing?

        • 2018-06-04 15537, 2018

      • samj1912
        bitmap: nope

        • 2018-06-04 15542, 2018

      • samj1912
        it indexes other fields

        • 2018-06-04 15549, 2018

      • samj1912
        _store is just used for display

        • 2018-06-04 15508, 2018

      • samj1912
        it varies for each entity

        • 2018-06-04 15511, 2018

      • samj1912
        you can see https://github.com/metabrainz/mbsssss

        • 2018-06-04 15521, 2018

      • samj1912
        just go to any entity and look into schema.xml

        • 2018-06-04 15539, 2018

      • samj1912
        sir fetches those fields and just posts their values as a set/single string

        • 2018-06-04 15542, 2018

      • samj1912
        no schema

        • 2018-06-04 15551, 2018

      • samj1912
        as in no mb schema

        • 2018-06-04 15514, 2018

      • samj1912
        ruaok: got fed up with the duplicacy in mbsssss - https://github.com/metabrainz/mbsssss

        • 2018-06-04 15531, 2018

      • samj1912
        makes it more easy to manage and update

        • 2018-06-04 15542, 2018

      • bitmap
        okay got it

        • 2018-06-04 15522, 2018

      • samj1912
        if we remove store, I guess we can reduce the indexes by over 70-80%

        • 2018-06-04 15527, 2018

      • zas
        samj1912, bitmap: could _store return a key, and answers fetched from redis or the like ? Since, apparently we don't need to store this data in SOLR.

        • 2018-06-04 15549, 2018

      • samj1912
        zas: we already return the mbid/db-id(for cases there is no mbid)

        • 2018-06-04 15555, 2018

      • bitmap
        it's technically possible for mbs to fill in the response but I don't really trust it to do that

        • 2018-06-04 15558, 2018

      • samj1912
        you can easily fetch it from that

        • 2018-06-04 15511, 2018

      • zas
        oh, even better then

        • 2018-06-04 15544, 2018

      • samj1912
        bitmap: why so?

        • 2018-06-04 15504, 2018

      • ruaok
        samj1912: as long as we don't hit the DB for additional info after the search its fine.

        • 2018-06-04 15525, 2018

      • bitmap
        samj1912: not sure if it'll be able to keep up with the traffic

        • 2018-06-04 15547, 2018

      • samj1912
        hmm

        • 2018-06-04 15538, 2018

      • bitmap
        right now mbs doesn't really have to do anything for searches, it just has nginx wait for the response and goes on to do something else

        • 2018-06-04 15524, 2018

      • samj1912
        ruaok: how bout subbing solr for a fast kv store

        • 2018-06-04 15529, 2018

      • bitmap
        if it has to decode every document, fetch and add a bunch of data, and re-encode it, that's going to be very expensive

        • 2018-06-04 15535, 2018

      • samj1912
        sir posts the _store to solr

        • 2018-06-04 15505, 2018

      • samj1912
        instead we can have it post it to a kv store and fetch from there?

        • 2018-06-04 15540, 2018

      • bitmap
        did you mention that writing the responses is the bottleneck?

        • 2018-06-04 15512, 2018

      • samj1912
        yeah

        • 2018-06-04 15558, 2018

      • bitmap
        not sure I follow how using a separate kv store solves that

        • 2018-06-04 15515, 2018

      • samj1912
        I am not sure if its solr that's bad at it

        • 2018-06-04 15530, 2018

      • samj1912
        solr wasn't really designed to store and display large fields

        • 2018-06-04 15514, 2018

      • CatQuest
        [13:54] <reosarevok> I mean, I did grow up with M$ and all, but right now I don't feel they're worse than other big company in the IT world honestly.

        • 2018-06-04 15514, 2018

      • CatQuest
        this for me too

        • 2018-06-04 15514, 2018

      • CatQuest
        ms doesn't seem like they did 10 years ago not at all

        • 2018-06-04 15514, 2018

      • CatQuest
        (not nearly the "good guys") but stil

        • 2018-06-04 15514, 2018

      • dragonzeron
        has the meeting started yet

        • 2018-06-04 15542, 2018

      • CatQuest
        my say is to watch it with argus eyes, and the minute it starts to change to the worse; move our shit

        • 2018-06-04 15508, 2018

      • bitmap
        samj1912: but more specifically, the bottleneck is marshalling/unmarshalling _store to get the response I guess?

        • 2018-06-04 15517, 2018

      • samj1912
        bitmap: I dont think so

        • 2018-06-04 15522, 2018

      • samj1912
        I tested without that as well

        • 2018-06-04 15531, 2018

      • samj1912
        the entire thing was causing a perf. hit of 10%

        • 2018-06-04 15534, 2018

      • CatQuest
        but not to do anything just yet.. that coudl be premature

        • 2018-06-04 15556, 2018

      • CatQuest
        i agree that failbook is a worse comp thna MS

        • 2018-06-04 15528, 2018

      • bitmap
        hmm, okay

        • 2018-06-04 15502, 2018

      • samj1912
        I tried with the normal xml/json output and with mbjson/mbxml

        • 2018-06-04 15541, 2018

      • dragonzeron
        does the meeting start in 15 minutes

        • 2018-06-04 15555, 2018

      • dragonzeron
        I just woke up so Idk what the time is

        • 2018-06-04 15533, 2018

      • samj1912
        bitmap: http://195.201.149.141:8983/solr/release/select?q… vs http://195.201.149.141:8983/solr/release/select?q…

        • 2018-06-04 15546, 2018

      • zas
        CatQuest: if A isn't worse than B doesn't make A or B good. Don't be too confident with "big ones", they usually destroy smaller ones, that's why they became big. Yesteday i had a github account, today i have a M$ account... and i didn't even buy a M$ mice since 20 years.. fuck that.

        • 2018-06-04 15506, 2018

      • bitmap
        dragonzeron: it starts in ~11 minutes

        • 2018-06-04 15520, 2018

      • dragonzeron
        ok thank you

        • 2018-06-04 15550, 2018

      • samj1912
        bitmap: I think solr just shits itself while fetching docs which have large stored fields