it says "For "python-musicbrainz/0.7.3": we allow through (on average) 50 requests per second, and decline the rest (though recently this has not been hit).For "anonymous" user-agents (see below): we allow through (on average) 50 requests per second, and decline (http 503) the rest.For other user-agents: allow through. "
2016-05-05 12656, 2016
bitmap
yeah, I don't think that's true for the new rate limiting that's in place since yesterday
2016-05-05 12659, 2016
pingupingu
so you mean for other user-agents: allow through. is actually 1req/sec
2016-05-05 12656, 2016
bitmap
I'll have to ask zas how the new stuff works. since headphones uses up the majority of our traffic it seems to be less fair to the rest
2016-05-05 12642, 2016
pingupingu
bitmap, I tried to download the mbdump, mbedit and mb-something else and load it in my load it on my postgresql instance and then I ran out of space at 25gb
2016-05-05 12605, 2016
pingupingu
how much disc space do I need for that db dump?
2016-05-05 12625, 2016
bitmap
you probably don't need the edit dump, which is the vast majority of that space
2016-05-05 12647, 2016
bitmap
exclude that and you should be fine, I think
2016-05-05 12625, 2016
JesseW
from my reading of the rate limit page, it's 50 req/sec for *all the clients with that user-agent*, divided up among IP addresses with 1 req/sec for each one (so effectively, up to 50 different IP addresses using the same user-agent will be allowed through simultanenously, but no more). And for other user-agents, there's no maximimum limit, but still a limit of 1 req/sec from the IP address.
2016-05-05 12628, 2016
bitmap
zas: I think the rate at which people receive 503s has changed because the 'global' limit is now applied before the user-agent limits, whereas previously it was applied after?
2016-05-05 12601, 2016
pingupingu
righto.. i just checked i actually used the mb-editor instead of mb-edit (which is 3.7gb)
2016-05-05 12613, 2016
pingupingu
those ones I am trying to import mbdump-derived.tar.bz2 mbdump-editor.tar.bz2 mbdump.tar.bz2 (2.xgb)
2016-05-05 12624, 2016
pingupingu
but I ran out of disk about 20gb
2016-05-05 12652, 2016
bitmap
try doubling the size of the disk
2016-05-05 12611, 2016
pingupingu
so what is required for the mimimal dump for me to start using the python api and get some results?
2016-05-05 12624, 2016
CallerNo6
JesseW, yeah, that was my understanding too
2016-05-05 12627, 2016
bitmap
those should be enough
2016-05-05 12615, 2016
pingupingu
hmm righto.. I will have to request more disk space
2016-05-05 12646, 2016
pingupingu
righto.. then I should not be bound by the rate requests 1/sec etc when I am running a local musicbrainz instance
2016-05-05 12606, 2016
bitmap
right
2016-05-05 12612, 2016
pingupingu
good
2016-05-05 12613, 2016
pingupingu
bitmap, ya seem to know quit a bit about mb, what do you do?
2016-05-05 12630, 2016
bitmap
I'm one of the devs :)
2016-05-05 12603, 2016
pingupingu
ahh great :)
2016-05-05 12619, 2016
JesseW wonders about the possibilities of leveraging one of the distributed content systems that are popular theses days to distribute the load of serving musicbrainz content
2016-05-05 12630, 2016
JesseW
maybe IPFS, or DAT, or one of those.
2016-05-05 12604, 2016
alastairp
"7:19 AM <pingupingu> it says "For "python-musicbrainz/0.7.3": we allow through (on average) 50 requests per second, "
2016-05-05 12607, 2016
regagain joined the channel
2016-05-05 12618, 2016
alastairp
note that python-musicbrainz is not the same as python-musicbrainzngs
2016-05-05 12630, 2016
alastairp
that text is wayyyyy out of date and should be deleted
2016-05-05 12608, 2016
mihaitish joined the channel
2016-05-05 12612, 2016
pingupingu has quit
2016-05-05 12628, 2016
xps2 has quit
2016-05-05 12619, 2016
xps2 joined the channel
2016-05-05 12634, 2016
diana_olhovyk joined the channel
2016-05-05 12603, 2016
JesseW has quit
2016-05-05 12636, 2016
yeeeargh joined the channel
2016-05-05 12628, 2016
Freso would certainly hope that bitmap knows "quit a bit about mb" :p
2016-05-05 12650, 2016
Freso
alastairp: WP:BB :)
2016-05-05 12611, 2016
mihaitish has quit
2016-05-05 12604, 2016
JonnyJD joined the channel
2016-05-05 12601, 2016
ariscop has quit
2016-05-05 12634, 2016
kartikgupta0909 joined the channel
2016-05-05 12640, 2016
kartikgupta0909
alastairp: We can start whenever you ready
2016-05-05 12637, 2016
zas
bitmap: you're right now global rate limit is applied before UA or IP ones, and it is less fair to small users. Though we don't have much choice.
I'll see how to improve things in next days. On the positive side, MB website has never been that responsive since months, and such drastic restriction on WS will encourage big users to use their own db.
2016-05-05 12652, 2016
zas
And search servers didn't crash since the change was made.
2016-05-05 12633, 2016
ariscop joined the channel
2016-05-05 12603, 2016
alastairp
kartikgupta0909: hey, I'm here
2016-05-05 12609, 2016
alastairp
let me find your document
2016-05-05 12653, 2016
alastairp
OK
2016-05-05 12652, 2016
kartikgupta0909
lets start
2016-05-05 12656, 2016
alastairp
I don't think that we need to change much in the proposal. The only thing that I see currently is that we could possibly target the latest ubuntu instead - 16.04 instead of 14.04
2016-05-05 12605, 2016
kartikgupta0909
yes
2016-05-05 12614, 2016
alastairp
zas: can you explain to me a little about chef - does it prefer to "own" the entire machine?
2016-05-05 12614, 2016
kartikgupta0909
also authentication
2016-05-05 12628, 2016
alastairp
we're looking at an automated way to install a bunch of packages and do configuration
2016-05-05 12639, 2016
alastairp
except people will probably run this on an existing server
2016-05-05 12656, 2016
alastairp
kartikgupta0909: right. Gentlecat started a patch for API keys for users
2016-05-05 12608, 2016
kartikgupta0909
oh
2016-05-05 12608, 2016
alastairp
I think this is a good start for us, we may not need anything else
yes, will do that. Also I have no idea what chef is. I was thinking of writing installation scripts similar to the ones already included in acousticbrainz server
2016-05-05 12604, 2016
kartikgupta0909
yes thats what i was talking about
2016-05-05 12611, 2016
alastairp
chef is a management tool for setting up servers
2016-05-05 12624, 2016
alastairp
it's a bit more structured than just writing a script
2016-05-05 12634, 2016
alastairp
and has features to easily deploy to many machines at a time
