#metabrainz

hello folks

I tried requesting json reposnses, but I get 503 response where as the same request with xml gives me the result: http://musicbrainz.org/ws/2/artist/05cbaf37-6dc...

change the &fmt=xml to &fmt=json and I get 503, any ideas why?

503 means the web service is over capacity, you'll just have to try again

hey there again bitmap

howdy?

I thought it was essentially the same service but just doing the xml to json translation

it is :) the &fmt= isn't having any effect

you probably just got unlucky, I only get 503s for the XML for example

isnt having any effect?

do you have a sample query for the json results?

it's not affecting the rate at which you get 503s at all

you already linked one, just change fmt=xml to fmt=json

refresh until the server isn't busy

bugger i quit accidentally

now I get 503 on xml too. are the servers usually this busy?

lately yes, but zas changed the rate limiting yesterday and that's having an affect too

ha I managed to get to beta.musicbrainz.org with both jsons and xml responses

I just wanted to take a look at the responses

it seems to be less fair if you're only making a few requests now. hmph

so the 50 responses per second per app query is changed now?

I didnt expect to gt a 503 immediately on the first json request

afaik our rate limit has always been 1 req/s, not sure where 50 req/s per app comes from

but it seems we don't have the capacity to honor 1 req/s for everyone right now

ha I just read that 50req/sec let me get that url for you

well, we have separate limits per-ip and per-user-agent, so that could be it

yeah

ah, right then

it says "For "python-musicbrainz/0.7.3": we allow through (on average) 50 requests per second, and decline the rest (though recently this has not been hit).For "anonymous" user-agents (see below): we allow through (on average) 50 requests per second, and decline (http 503) the rest.For other user-agents: allow through. "

yeah, I don't think that's true for the new rate limiting that's in place since yesterday

so you mean for other user-agents: allow through. is actually 1req/sec

I'll have to ask zas how the new stuff works. since headphones uses up the majority of our traffic it seems to be less fair to the rest

bitmap, I tried to download the mbdump, mbedit and mb-something else and load it in my load it on my postgresql instance and then I ran out of space at 25gb

how much disc space do I need for that db dump?

you probably don't need the edit dump, which is the vast majority of that space

exclude that and you should be fine, I think

from my reading of the rate limit page, it's 50 req/sec for *all the clients with that user-agent*, divided up among IP addresses with 1 req/sec for each one (so effectively, up to 50 different IP addresses using the same user-agent will be allowed through simultanenously, but no more). And for other user-agents, there's no maximimum limit, but still a limit of 1 req/sec from the IP address.

zas: I think the rate at which people receive 503s has changed because the 'global' limit is now applied before the user-agent limits, whereas previously it was applied after?

righto.. i just checked i actually used the mb-editor instead of mb-edit (which is 3.7gb)

those ones I am trying to import mbdump-derived.tar.bz2 mbdump-editor.tar.bz2 mbdump.tar.bz2 (2.xgb)

but I ran out of disk about 20gb

try doubling the size of the disk

so what is required for the mimimal dump for me to start using the python api and get some results?

JesseW, yeah, that was my understanding too

those should be enough

hmm righto.. I will have to request more disk space

righto.. then I should not be bound by the rate requests 1/sec etc when I am running a local musicbrainz instance

right

good

bitmap, ya seem to know quit a bit about mb, what do you do?

I'm one of the devs :)

ahh great :)

maybe IPFS, or DAT, or one of those.

"7:19 AM <pingupingu> it says "For "python-musicbrainz/0.7.3": we allow through (on average) 50 requests per second, "

note that python-musicbrainz is not the same as python-musicbrainzngs

that text is wayyyyy out of date and should be deleted

alastairp: WP:BB :)

alastairp: We can start whenever you ready

bitmap: you're right now global rate limit is applied before UA or IP ones, and it is less fair to small users. Though we don't have much choice.

Project acousticbrainz-server build #52: SUCCESS in 4 min 44 sec: https://ci.metabrainz.org/job/acousticbrainz-se...

I'll see how to improve things in next days. On the positive side, MB website has never been that responsive since months, and such drastic restriction on WS will encourage big users to use their own db.

And search servers didn't crash since the change was made.

kartikgupta0909: hey, I'm here

let me find your document

OK

lets start

I don't think that we need to change much in the proposal. The only thing that I see currently is that we could possibly target the latest ubuntu instead - 16.04 instead of 14.04

yes

zas: can you explain to me a little about chef - does it prefer to "own" the entire machine?

also authentication

we're looking at an automated way to install a bunch of packages and do configuration

except people will probably run this on an existing server

kartikgupta0909: right. Gentlecat started a patch for API keys for users

oh

I think this is a good start for us, we may not need anything else

for acousticbrainz users?

right

alastairp: chef is more about ensuring things are in a well defined state

zas: our goal is to install some dependencies, and some source code

it sounds like something that chef is well suited for, except I wonder if it's a very large stick

and we just need a shell script

Chef can do it of course

kartikgupta0909: I propose that you develop on a Vagrant system, this means that it's easy for us to test the entire setup process

your first milestones are to set up the software deployment

sure, will do that. Even my acousticbrainz server runs on vagrant

yes

perfect, so you can probably start by copying the acousticbrainz vagrant configuration

You may start to write a shell script, then convert it to chef recipes. Chef isn't easy to learn.

hmm, right

Chef + vagrant works welk

kartikgupta0909: OK, based on zas' feedback, let's stay with shell scripts for now

https://github.com/metabrainz/acousticbrainz-se... is the main entry point for the AB install scripts

yes, will do that. Also I have no idea what chef is. I was thinking of writing installation scripts similar to the ones already included in acousticbrainz server

yes thats what i was talking about

chef is a management tool for setting up servers

it's a bit more structured than just writing a script

and has features to easily deploy to many machines at a time

oh okay, I will look into that and try to learn it, For now I will start working on the scripts.

however, don't worry too much about this for now. If we have time we could look at changing to it

as zas says, it's a lot to learn

regarding installation of dependencies, I don't want to install gaia from source

also you were saying about maintaining our own repo for 16.04 which will contain all the dependencies

yes

right, exactly

although I dont know how those repository work. But will learn it before the start of the coding period.

We have tools to build debian/ubuntu packages for essentia: https://github.com/MTG/essentia/tree/master/debian

do we need essentia for this?

there are also tools for gaia, but they are more out of date: https://github.com/MTG/gaia/tree/master/debian

i doubt that

no, we only need gaia

yes

I suggest you start looking at debian package building. there are lots of tutorals about how to do it: https://wiki.debian.org/HowToPackageForDebian

other things are yaml, protobuf and maybe Screen

yes I will follow the tutorials over this weekend itself.