#metabrainz

/

      • D4RK-PH0ENiX has quit
      • D4RK-PH0ENiX joined the channel
      • D4RK-PH0ENiX has quit
      • QuoraUK has quit
      • D4RK-PH0ENiX joined the channel
      • D4RK-PH0ENiX has quit
      • D4RK-PH0ENiX joined the channel
      • JesseW joined the channel
      • zas
        Whoo The Shadoks on Google.fr ... http://www.google.com/doodles/48th-anniversary-...
      • "If there is no solution, it is because there is no problem." -- The Shadoks ;)
      • kyan joined the channel
      • demonimin has quit
      • demonimin joined the channel
      • demonimin has quit
      • demonimin joined the channel
      • demonimin has quit
      • demonimin joined the channel
      • demonimin has quit
      • demonimin joined the channel
      • kyan has quit
      • kyan joined the channel
      • JesseW has quit
      • The_Catman joined the channel
      • CatmanIX has quit
      • JesseW joined the channel
      • The_Catman has quit
      • The_Catman joined the channel
      • bitmap
        zas: nothing broken with jenkins afaik, stuff is still building
      • yeeeargh joined the channel
      • yeeeargh has quit
      • kwikadi has quit
      • kyan has quit
      • kanha has quit
      • zag joined the channel
      • zag2 has quit
      • abhishek joined the channel
      • abhishek is now known as kanha
      • downtime is now known as e
      • CallerNo6 has quit
      • JesseW has quit
      • The_Catman has quit
      • The_Catman joined the channel
      • Leftmost
        LordSputnik, BB-186 is up for review. Comments on anything I've done there are welcome.
      • reosarevok
        bitmap: "not that i know about (which doesn’t necessarily mean there aren’t any). is it still happening?"
      • QuoraUK joined the channel
      • Slurpee has quit
      • mihaitish has quit
      • diana_olhovyk_ joined the channel
      • QuoraUK has quit
      • mihaitish joined the channel
      • zag
      • wondering about ALBUMARTISTS support?
      • Freso
        zas bitmap rahulr, ^
      • MajorLurker joined the channel
      • zag
        thx
      • zas
      • zag
        wonderful! i did just test the latest nightly though and couldnt see the tag
      • zas
        It isn't available as tag, but it is available for scripting, so one can add it to tag. See also http://tickets.musicbrainz.org/browse/PICARD-700
      • ruaok
        drsaunde and reosarevok: I gave an indirect shout out to you two: https://medium.com/subwoofr-insights/aim-music-...
      • > I had the chance to speak with Robert afterwards, and he told me that some volunteers have contributed as many as 1 million edits to the database — an absolutely staggering insight into the level of commitment this community provides.
      • well said.
      • Freso
        :)
      • reosarevok
        :D
      • That's... a lot of edits, isn't it?
      • Crazy people.
      • ruaok
        I did also mention that we're all a bit crazy. OCD at least and that got lots of laughs too. :)
      • zag
        cool zas, i will add $setmulti(albumartists,%_albumartists%) to the guide then. shame its not default though :(
      • zas
        "It’s basically an open source IMDb for Music." <-- is that a compliment ? ;)
      • zag: not sure why it isn't the default, but i guess the main reason is that ALBUMARTISTS is/was not standard enough or not very well supported by formats. You should add a comment in PICARD-700 about it.
      • reosarevok
        zas: well IMDb would be pretty good if it was open source and open data :p
      • zag
        thx will do
      • diana_olhovyk_ has quit
      • ruaok
        zas said: > i'm thinking of means to have better control on the web service access
      • and yeah, I kinda agree.
      • the main limiting factor has been capacity for a number of years.
      • but with a move to a new facility we should be able to handle much more traffic.
      • zas
        first, already discussed, is to use a fail2ban-like mechanism based on rate limited logs
      • zag
        certainly look at a dedicated server for the WS
      • ruaok
        I'd like to consider using a more intelligent rate limiting system too.
      • zas
        i'm not fond of this solution, because it relies on our currrent rate limiter which may not scale well
      • zag
        and api keys really do help monitoring
      • zas
        another approach, would be to use a separate IP for web service, and use iptables features to rate limit
      • ruaok
        so, for LB, I implemented the Twitter style dynamic rate limting.
      • it works on the concept of a window.
      • on the first request you get a reponse for "window size" and the number of requests granted per window.
      • the caller can use the requests as fast they want, or space them out.
      • and we can dynamically adjust the rate limits as the load on the overall system changes.
      • it was pretty easy to implement -- the hardest thing was getting the client JS to do the right thing.
      • zas
        how such mechanism does scale ?
      • ruaok
        and you do not need a central rate limit service.
      • each client can do it with redis as a backend.
      • and we're already talking about HA redis, so this should scale well.
      • zas
        looks like great
      • if we can get rid of the current rate limiter, i'll be quite happy, this one would not scale well
      • ruaok
        and it is something we can implement now. clients can start using it when they are ready.
      • agreed.
      • now, API keys is another question. I suppose we should do that too.
      • another thing we can build into the MeB site, I suppose.
      • get an API key, use the new system.
      • without API key, you get the old 1s limit, which we can throttle in favor of users with keys.
      • previously the team pushed back on the idea of doing this before ws/3
      • but ws/3 is far more mythical than unicorns, so I want to do something sooner.
      • zas
        about that, i think we should use wsN.mb.o instead of mb.o/ws/N, and use a different IP for each ws (different from website one)
      • ruaok
        I extended it to support two rate limits: per IP and per key. If you give a key you can get a higher rate limit than if you use per IP
      • I agree.
      • zas
        it would make it easier to firewall
      • ruaok
        that what was mm.musicbrainz.org was for, but ian and nikki just made fun of me for that.
      • I am inclined to write two tickets for this feature: one for MEB to implement API keys and one for MBS to implement this on wsN.mb.o
      • and we deploy it NewHost.
      • zas
        bitmap said having redirects from mb.o/ws/N to wsN.mb.o may cause issues with certain clients, but i answered they are supposed to handle HTTP redirects and conform to standards (... Picard is awful ...)
      • Still this is something to take in account
      • D4RK-PH0ENiX has quit
      • ruaok
        agreed. I think we should use human redirects rather than HTTP redirects.
      • QuoraUK joined the channel
      • meaning that authors need to use the new API endpoint with API key.
      • zas
        ruaok: how long would it take to have LB/Twitter-like rate limiting system ready for mb.o WS ?
      • ruaok
        My guess, if bitmap, Gentlecat and you all work in tandem for a week, we could get it done.
      • the API key signup is the most work, but Gentlecat can copy and paste some bits from the live feed signup bits.
      • if not outright re-use the same tables and such.
      • then we'd need to do testing of course. not sure if would make sense to deploy before NewHost.
      • zas
        Agreed 100%
      • Perhaps something to do just after the move
      • ruaok
        and yes, when you're here we need to make the mother of all spreadsheets analyzing OVH vs hetzner.
      • zas
        Yup.
      • ruaok
        I think we need to reshuffle our goals for the short term.
      • namely we need need stop doing anything that isn't on the critical path towards a new hosting setup.
      • zas
        The most urgent is to have more bandwidth
      • ruaok
        we can get bandwidth easy.
      • but more bandwidth means more smoking servers or am I wrong on that?
      • zas
        Well, i mean more bandwidth together to more servers
      • else it serves no purpose ;)
      • ruaok
        so, before we meet in BCN to make the spreadsheet, what tasks do you need to do in order to make the move easier?
      • zas
        That is moving to NewHost asap
      • ruaok
        exactly.
      • the app servers are easily deployable with fabric and all that, right?
      • postgres is alright, but replication needs to be setup.
      • gateways are nearly there, right?
      • zas
        gateways/web servers/redis/memcache are easy (well, minus Murphy's law)
      • ruaok
        CAA, CB, wiki, jira can move later.
      • zas
        ci too
      • ruaok
        wiki better sooner because of wikidocs. but legoktm can probably do that by hand.
      • zas
        stats can be installed fast enough at NewHost
      • ruaok
        let me make a new section in the migration doc that splits these into right away vs later.
      • zas
        discourse can be moved quite easily too
      • ruaok
        but not needed, right?
      • zas
        right
      • I wonder about IP/DNS changes though
      • We share a lot of things on the same IPs