#metabrainz

zas: Got an e-mail there. Thanks! :)

!m zas

You're doing good work, zas!

Project musicbrainz-server_schema-change-2016-q2 build #100: SUCCESS in 18 min: http://ci.musicbrainz.org/job/musicbrainz-serve...

bitmap: should I run `./admin/GenerateSQLScripts.pl admin/sql/caa` as well?

it's updating other files as well, not only dropindexes

not sure if I should include them

yeah, sorry, run it for both dirs

you can trust whatever changes it makes too

(I hope)

haha 👌

bitmap, zas: is there a way to see how many requests per second musicbrainz frontend gets? can't quite find the right graph at http://stats.musicbrainz.org/

Gentlecat: http://stats.musicbrainz.org/dashboard/db/web-s...

No idea why the total number is minus 15 million at the moment, but the numbers for the individual frontends seem plausible.

total value

http://stats.musicbrainz.org/dashboard/db/web-s... is correct too, though influx based graphs need more work, still few issues with them

basically around 500 per second on all servers, and up to ~600 during peak hours

HTTP 200 response are around 200-250 per second, the rest is errors (mainly rate limiter)

MusicBrainz webservice is having real troubles recently

getting mutiple errors on picard, xbmc and my site imports

You guys should really move to a API key system and manage traffic better imo

getting about a 15% fail rate in my tests over last week

CatQuest: It was reosarevok ho used the acronym though :)

zag: Did you just volunteer to write that? :D

hey, zag: thanks for that vote of no confidence. we appreciate non constructive criticism.

do you you not think that we're trying hard to make the service as good as it can be?

-you

alastairp: part printed. coming to mob or should I bring it tomorrow?

I love how people offer us SEO packages that will allow us to dominate the google search rankings.

clearly they have never actually done a search on "musicbrainz"

most web sites could only wish to be as integrated as MB is with Google.

I actually run a very similar web service myself so can offer help

json is best

api keys are essential

commercial apps should pay

large apps should work with the api to improve it (headphones, picard, kodi/xbmc)

zag: JSON is already available.

zag: Commercial apps should already pay.

but monitoring is the key, looking at demand, and server resrouces

One of the main Picard developers is the (employed) MusicBrainz system administrator.

I think most people use the xml at the moment, as last time I checked the json was not officialyl stable

freso picard has many errors for me

more recently, hence the discusssion now

and we're a non profit, so we offer use of our API to the public.

I'm just pointed out that several of the things you mention are already there.

*pointing

what errors are you getting, zag?

Which means that you likely have not done your homework.

have you filed tickets for the errors you're getting?

I know very well, I am hoping to help

I assumed the web service just gives errors due to rate limiter

but a rate limiter is not a solution really, thats just hiding the problem (imo)

we have so many idiotic users (see headphones) that pummel us with stupid requests.

I'd be interested in statistics of api requests, if you have any

and I am not interested int giving half of our traffic to mass pirates.

I serve 12 million a night from my api

5-10% cpu on a xeon SSD web server

20 million requests a day.

ok yes thats a lot indeed thanks

and most of those require 1 or more database requests.

complicated database requests at that.

really, all the ideas you're throwing around are things that have been considered.

I used a mysql analyzer on mine to improve the indexing and db requests, found some slow queiries that were killing the server

doing it live, is useful at peak times as bigger problems are magnified

which site do you run?

TheAudioDB, and formerly TheTVDB and TheMovieDB

handles 100's of millions of requests a day between them

Have a lot of experience of this, and I see you guys coudl benefit from that

And has nowhere near the complexity that MB does...

Yes very true, but api's dont always have to be complex ;)

for example XBMC/Kodi just needs to lookup the MBID

No, but the SQL queries that WS calls do do.

and thats probably responsible for 25% of your api hits

zag: With no ?inc's?

ah perfect.

you can handle all the useless requests from headphones and xmbc that will never really contribute to us anyway.

we'll handle the requests for people who are more serious about their metadata.

zag: http://stats.musicbrainz.org/dashboard/db/music...

for things like headphones, if you had an API key, just revoke it if they use too much and give others priority

force them to use private mirrors

ruaok: about mail, not much issues it seems, or we didn't notice them yet (no report?)

Sorry if im coming over demanding, i know how this is from the admin side hehe

zas: i haven't seen any. maybe because mail is broken? :) :)

ruaok: yes, maybe ;)

zag: FWIW, VLC runs their own MB mirror which they have VLC query against instead of pinging the main mb.o.

yeh used to use it myself, but i the address must have changed

and who knows how many millions of requests they serve per day.

zag: Is any of the code for your webservices open source? A lot of what you said is what I've had in mind for the BookBrainz web service

zag: IMO, any large scale project should consider doing the same. Both to lessen the load on MB's servers, but also so that they're able to monitor their users' usage better.

another thing would be to have a nicely updated mirror server all the time, I know we ran a MB mirror but gave up as it was so out of date each release

zag: And they stated that the mirror moved on Twitter.

zag: we have about 400 mirrors around the world that are up to date.