#metabrainz

/

      • Freso
        zas: Got an e-mail there. Thanks! :)
      • 2016-03-17 07747, 2016

      • Freso
        !m zas
      • 2016-03-17 07747, 2016

      • BrainzBot
        You're doing good work, zas!
      • 2016-03-17 07709, 2016

      • D4RK-PH0ENiX has quit
      • 2016-03-17 07754, 2016

      • opatel99 has quit
      • 2016-03-17 07726, 2016

      • vidhan has quit
      • 2016-03-17 07746, 2016

      • adhawkins has quit
      • 2016-03-17 07721, 2016

      • adhawkins joined the channel
      • 2016-03-17 07729, 2016

      • D4RK-PH0ENiX joined the channel
      • 2016-03-17 07738, 2016

      • JesseW joined the channel
      • 2016-03-17 07744, 2016

      • kyan joined the channel
      • 2016-03-17 07733, 2016

      • Jormangeud has quit
      • 2016-03-17 07726, 2016

      • Jormangeud joined the channel
      • 2016-03-17 07747, 2016

      • Jormangeud has quit
      • 2016-03-17 07732, 2016

      • Jormangeud joined the channel
      • 2016-03-17 07721, 2016

      • Lotheric_ joined the channel
      • 2016-03-17 07754, 2016

      • JesseW has quit
      • 2016-03-17 07734, 2016

      • Lotheric has quit
      • 2016-03-17 07740, 2016

      • CallerNo6 has quit
      • 2016-03-17 07741, 2016

      • RJ2 has quit
      • 2016-03-17 07712, 2016

      • CallerNo6 joined the channel
      • 2016-03-17 07712, 2016

      • RJ2 joined the channel
      • 2016-03-17 07700, 2016

      • JesseW joined the channel
      • 2016-03-17 07735, 2016

      • MBJenkins
        Project musicbrainz-server_schema-change-2016-q2 build #100: SUCCESS in 18 min: http://ci.musicbrainz.org/job/musicbrainz-server_…
      • 2016-03-17 07713, 2016

      • Gentlecat
        bitmap: should I run `./admin/GenerateSQLScripts.pl admin/sql/caa` as well?
      • 2016-03-17 07740, 2016

      • Gentlecat
        it's updating other files as well, not only dropindexes
      • 2016-03-17 07731, 2016

      • Gentlecat
        not sure if I should include them
      • 2016-03-17 07755, 2016

      • bitmap
        yeah, sorry, run it for both dirs
      • 2016-03-17 07710, 2016

      • bitmap
        you can trust whatever changes it makes too
      • 2016-03-17 07730, 2016

      • bitmap
        (I hope)
      • 2016-03-17 07737, 2016

      • Gentlecat
        haha 👌
      • 2016-03-17 07737, 2016

      • Jormangeud has quit
      • 2016-03-17 07756, 2016

      • Jormangeud joined the channel
      • 2016-03-17 07740, 2016

      • Jormangeud has quit
      • 2016-03-17 07702, 2016

      • Jormangeud joined the channel
      • 2016-03-17 07721, 2016

      • kartikgupta0909 joined the channel
      • 2016-03-17 07726, 2016

      • justharshal joined the channel
      • 2016-03-17 07756, 2016

      • typhoe- joined the channel
      • 2016-03-17 07706, 2016

      • typhoe has quit
      • 2016-03-17 07706, 2016

      • typhoe- is now known as typhoe
      • 2016-03-17 07747, 2016

      • vidhan joined the channel
      • 2016-03-17 07754, 2016

      • Nyanko-sensei joined the channel
      • 2016-03-17 07758, 2016

      • ariscop has quit
      • 2016-03-17 07759, 2016

      • D4RK-PH0ENiX has quit
      • 2016-03-17 07730, 2016

      • Jormangeud has quit
      • 2016-03-17 07726, 2016

      • diana_olhovyk joined the channel
      • 2016-03-17 07737, 2016

      • Jormangeud joined the channel
      • 2016-03-17 07706, 2016

      • Jormangeud has quit
      • 2016-03-17 07743, 2016

      • Jormangeud joined the channel
      • 2016-03-17 07730, 2016

      • Mineo joined the channel
      • 2016-03-17 07703, 2016

      • ariscop joined the channel
      • 2016-03-17 07700, 2016

      • Gentlecat
        bitmap, zas: is there a way to see how many requests per second musicbrainz frontend gets? can't quite find the right graph at http://stats.musicbrainz.org/
      • 2016-03-17 07714, 2016

      • JonnyJD_ has quit
      • 2016-03-17 07731, 2016

      • yeeeargh joined the channel
      • 2016-03-17 07711, 2016

      • JesseW has quit
      • 2016-03-17 07735, 2016

      • Nyanko-sensei has quit
      • 2016-03-17 07702, 2016

      • D4RK-PH0ENiX joined the channel
      • 2016-03-17 07754, 2016

      • Mineo has quit
      • 2016-03-17 07755, 2016

      • justharshal has quit
      • 2016-03-17 07726, 2016

      • vidhan has quit
      • 2016-03-17 07742, 2016

      • chirlu
      • 2016-03-17 07712, 2016

      • chirlu
        No idea why the total number is minus 15 million at the moment, but the numbers for the individual frontends seem plausible.
      • 2016-03-17 07713, 2016

      • Jormangeud has quit
      • 2016-03-17 07706, 2016

      • Jormangeud joined the channel
      • 2016-03-17 07748, 2016

      • Jormangeud has quit
      • 2016-03-17 07717, 2016

      • Jormangeud joined the channel
      • 2016-03-17 07723, 2016

      • zas
      • 2016-03-17 07733, 2016

      • zas
        total value
      • 2016-03-17 07713, 2016

      • zas
        http://stats.musicbrainz.org/dashboard/db/web-ser… is correct too, though influx based graphs need more work, still few issues with them
      • 2016-03-17 07709, 2016

      • zas
        basically around 500 per second on all servers, and up to ~600 during peak hours
      • 2016-03-17 07721, 2016

      • zas
        HTTP 200 response are around 200-250 per second, the rest is errors (mainly rate limiter)
      • 2016-03-17 07704, 2016

      • armalcolite joined the channel
      • 2016-03-17 07719, 2016

      • UmkaDK_ joined the channel
      • 2016-03-17 07728, 2016

      • UmkaDK has quit
      • 2016-03-17 07759, 2016

      • zag
        MusicBrainz webservice is having real troubles recently
      • 2016-03-17 07724, 2016

      • zag
        getting mutiple errors on picard, xbmc and my site imports
      • 2016-03-17 07742, 2016

      • zag
        You guys should really move to a API key system and manage traffic better imo
      • 2016-03-17 07717, 2016

      • zag
        getting about a 15% fail rate in my tests over last week
      • 2016-03-17 07751, 2016

      • Freso
        CatQuest: It was reosarevok ho used the acronym though :)
      • 2016-03-17 07704, 2016

      • Freso
        zag: Did you just volunteer to write that? :D
      • 2016-03-17 07702, 2016

      • ruaok
        hey, zag: thanks for that vote of no confidence. we appreciate non constructive criticism.
      • 2016-03-17 07715, 2016

      • ruaok
        do you you not think that we're trying hard to make the service as good as it can be?
      • 2016-03-17 07742, 2016

      • ruaok
        -you
      • 2016-03-17 07710, 2016

      • ruaok
        alastairp: part printed. coming to mob or should I bring it tomorrow?
      • 2016-03-17 07732, 2016

      • ruaok
        I love how people offer us SEO packages that will allow us to dominate the google search rankings.
      • 2016-03-17 07741, 2016

      • ruaok
        clearly they have never actually done a search on "musicbrainz"
      • 2016-03-17 07757, 2016

      • ruaok
        most web sites could only wish to be as integrated as MB is with Google.
      • 2016-03-17 07728, 2016

      • vidhan joined the channel
      • 2016-03-17 07744, 2016

      • zag
        I actually run a very similar web service myself so can offer help
      • 2016-03-17 07705, 2016

      • zag
        json is best
      • 2016-03-17 07711, 2016

      • zag
        api keys are essential
      • 2016-03-17 07727, 2016

      • zag
        commercial apps should pay
      • 2016-03-17 07715, 2016

      • zag
        large apps should work with the api to improve it (headphones, picard, kodi/xbmc)
      • 2016-03-17 07745, 2016

      • Freso
        zag: JSON is already available.
      • 2016-03-17 07758, 2016

      • Freso
        zag: Commercial apps should already pay.
      • 2016-03-17 07704, 2016

      • zag
        but monitoring is the key, looking at demand, and server resrouces
      • 2016-03-17 07737, 2016

      • Freso
        One of the main Picard developers is the (employed) MusicBrainz system administrator.
      • 2016-03-17 07740, 2016

      • zag
        I think most people use the xml at the moment, as last time I checked the json was not officialyl stable
      • 2016-03-17 07754, 2016

      • zag
        freso picard has many errors for me
      • 2016-03-17 07711, 2016

      • zag
        more recently, hence the discusssion now
      • 2016-03-17 07712, 2016

      • ruaok
        and we're a non profit, so we offer use of our API to the public.
      • 2016-03-17 07713, 2016

      • Freso
        I'm just pointed out that several of the things you mention are already there.
      • 2016-03-17 07718, 2016

      • Freso
        *pointing
      • 2016-03-17 07733, 2016

      • ruaok
        what errors are you getting, zag?
      • 2016-03-17 07737, 2016

      • Freso
        Which means that you likely have not done your homework.
      • 2016-03-17 07750, 2016

      • ruaok
        have you filed tickets for the errors you're getting?
      • 2016-03-17 07754, 2016

      • zag
        I know very well, I am hoping to help
      • 2016-03-17 07730, 2016

      • zag
        I assumed the web service just gives errors due to rate limiter
      • 2016-03-17 07748, 2016

      • zag
        but a rate limiter is not a solution really, thats just hiding the problem (imo)
      • 2016-03-17 07717, 2016

      • ruaok
        we have so many idiotic users (see headphones) that pummel us with stupid requests.
      • 2016-03-17 07723, 2016

      • zag
        I'd be interested in statistics of api requests, if you have any
      • 2016-03-17 07730, 2016

      • ruaok
        and I am not interested int giving half of our traffic to mass pirates.
      • 2016-03-17 07731, 2016

      • zag
        I serve 12 million a night from my api
      • 2016-03-17 07743, 2016

      • zag
        5-10% cpu on a xeon SSD web server
      • 2016-03-17 07744, 2016

      • ruaok
        20 million requests a day.
      • 2016-03-17 07758, 2016

      • zag
        ok yes thats a lot indeed thanks
      • 2016-03-17 07714, 2016

      • ruaok
        and most of those require 1 or more database requests.
      • 2016-03-17 07738, 2016

      • ruaok
        complicated database requests at that.
      • 2016-03-17 07700, 2016

      • ruaok
        really, all the ideas you're throwing around are things that have been considered.
      • 2016-03-17 07714, 2016

      • zag
        I used a mysql analyzer on mine to improve the indexing and db requests, found some slow queiries that were killing the server
      • 2016-03-17 07731, 2016

      • zag
        doing it live, is useful at peak times as bigger problems are magnified
      • 2016-03-17 07702, 2016

      • ruaok
        which site do you run?
      • 2016-03-17 07719, 2016

      • zag
        TheAudioDB, and formerly TheTVDB and TheMovieDB
      • 2016-03-17 07734, 2016

      • zag
        handles 100's of millions of requests a day between them
      • 2016-03-17 07759, 2016

      • zag
        Have a lot of experience of this, and I see you guys coudl benefit from that
      • 2016-03-17 07714, 2016

      • Freso
        And has nowhere near the complexity that MB does...
      • 2016-03-17 07729, 2016

      • zag
        Yes very true, but api's dont always have to be complex ;)
      • 2016-03-17 07742, 2016

      • zag
        for example XBMC/Kodi just needs to lookup the MBID
      • 2016-03-17 07744, 2016

      • Freso
        No, but the SQL queries that WS calls do do.
      • 2016-03-17 07700, 2016

      • zag
        and thats probably responsible for 25% of your api hits
      • 2016-03-17 07701, 2016

      • Freso
        zag: With no ?inc's?
      • 2016-03-17 07711, 2016

      • ruaok
        ah perfect.
      • 2016-03-17 07730, 2016

      • ruaok
        you can handle all the useless requests from headphones and xmbc that will never really contribute to us anyway.
      • 2016-03-17 07746, 2016

      • ruaok
        we'll handle the requests for people who are more serious about their metadata.
      • 2016-03-17 07747, 2016

      • zas
      • 2016-03-17 07738, 2016

      • zag
        for things like headphones, if you had an API key, just revoke it if they use too much and give others priority
      • 2016-03-17 07702, 2016

      • zag
        force them to use private mirrors
      • 2016-03-17 07709, 2016

      • zas
        ruaok: about mail, not much issues it seems, or we didn't notice them yet (no report?)
      • 2016-03-17 07718, 2016

      • ruaok
        which is what we do anyway, minus having to police api keys.
      • 2016-03-17 07728, 2016

      • zag
        Sorry if im coming over demanding, i know how this is from the admin side hehe
      • 2016-03-17 07734, 2016

      • ruaok
        zas: i haven't seen any. maybe because mail is broken? :) :)
      • 2016-03-17 07747, 2016

      • zas
        ruaok: yes, maybe ;)
      • 2016-03-17 07704, 2016

      • Freso
        zag: FWIW, VLC runs their own MB mirror which they have VLC query against instead of pinging the main mb.o.
      • 2016-03-17 07738, 2016

      • zag
        yeh used to use it myself, but i the address must have changed
      • 2016-03-17 07755, 2016

      • ruaok
        and who knows how many millions of requests they serve per day.
      • 2016-03-17 07705, 2016

      • LordSputnik
        zag: Is any of the code for your webservices open source? A lot of what you said is what I've had in mind for the BookBrainz web service
      • 2016-03-17 07708, 2016

      • Freso
        zag: IMO, any large scale project should consider doing the same. Both to lessen the load on MB's servers, but also so that they're able to monitor their users' usage better.
      • 2016-03-17 07716, 2016

      • zag
        another thing would be to have a nicely updated mirror server all the time, I know we ran a MB mirror but gave up as it was so out of date each release
      • 2016-03-17 07728, 2016

      • Freso
        zag: And they stated that the mirror moved on Twitter.
      • 2016-03-17 07753, 2016

      • ruaok
        zag: we have about 400 mirrors around the world that are up to date.