> I had the chance to speak with Robert afterwards, and he told me that some volunteers have contributed as many as 1 million edits to the database — an absolutely staggering insight into the level of commitment this community provides.
well said.
Freso
:)
reosarevok
:D
That's... a lot of edits, isn't it?
Crazy people.
ruaok
I did also mention that we're all a bit crazy. OCD at least and that got lots of laughs too. :)
zag
cool zas, i will add $setmulti(albumartists,%_albumartists%) to the guide then. shame its not default though :(
zas
"It’s basically an open source IMDb for Music." <-- is that a compliment ? ;)
zag: not sure why it isn't the default, but i guess the main reason is that ALBUMARTISTS is/was not standard enough or not very well supported by formats. You should add a comment in PICARD-700 about it.
reosarevok
zas: well IMDb would be pretty good if it was open source and open data :p
zag
thx will do
diana_olhovyk_ has quit
ruaok
zas said: > i'm thinking of means to have better control on the web service access
and yeah, I kinda agree.
the main limiting factor has been capacity for a number of years.
but with a move to a new facility we should be able to handle much more traffic.
zas
first, already discussed, is to use a fail2ban-like mechanism based on rate limited logs
zag
certainly look at a dedicated server for the WS
ruaok
I'd like to consider using a more intelligent rate limiting system too.
zas
i'm not fond of this solution, because it relies on our currrent rate limiter which may not scale well
zag
and api keys really do help monitoring
zas
another approach, would be to use a separate IP for web service, and use iptables features to rate limit
ruaok
so, for LB, I implemented the Twitter style dynamic rate limting.
it works on the concept of a window.
on the first request you get a reponse for "window size" and the number of requests granted per window.
the caller can use the requests as fast they want, or space them out.
and we can dynamically adjust the rate limits as the load on the overall system changes.
it was pretty easy to implement -- the hardest thing was getting the client JS to do the right thing.
zas
how such mechanism does scale ?
ruaok
and you do not need a central rate limit service.
each client can do it with redis as a backend.
and we're already talking about HA redis, so this should scale well.
zas
looks like great
if we can get rid of the current rate limiter, i'll be quite happy, this one would not scale well
ruaok
and it is something we can implement now. clients can start using it when they are ready.
agreed.
now, API keys is another question. I suppose we should do that too.
another thing we can build into the MeB site, I suppose.
get an API key, use the new system.
without API key, you get the old 1s limit, which we can throttle in favor of users with keys.
previously the team pushed back on the idea of doing this before ws/3
but ws/3 is far more mythical than unicorns, so I want to do something sooner.
about that, i think we should use wsN.mb.o instead of mb.o/ws/N, and use a different IP for each ws (different from website one)
ruaok
I extended it to support two rate limits: per IP and per key. If you give a key you can get a higher rate limit than if you use per IP
I agree.
zas
it would make it easier to firewall
ruaok
that what was mm.musicbrainz.org was for, but ian and nikki just made fun of me for that.
I am inclined to write two tickets for this feature: one for MEB to implement API keys and one for MBS to implement this on wsN.mb.o
and we deploy it NewHost.
zas
bitmap said having redirects from mb.o/ws/N to wsN.mb.o may cause issues with certain clients, but i answered they are supposed to handle HTTP redirects and conform to standards (... Picard is awful ...)
Still this is something to take in account
D4RK-PH0ENiX has quit
ruaok
agreed. I think we should use human redirects rather than HTTP redirects.
QuoraUK joined the channel
meaning that authors need to use the new API endpoint with API key.
zas
ruaok: how long would it take to have LB/Twitter-like rate limiting system ready for mb.o WS ?
ruaok
My guess, if bitmap, Gentlecat and you all work in tandem for a week, we could get it done.
the API key signup is the most work, but Gentlecat can copy and paste some bits from the live feed signup bits.
if not outright re-use the same tables and such.
then we'd need to do testing of course. not sure if would make sense to deploy before NewHost.
zas
Agreed 100%
Perhaps something to do just after the move
ruaok
and yes, when you're here we need to make the mother of all spreadsheets analyzing OVH vs hetzner.
zas
Yup.
ruaok
I think we need to reshuffle our goals for the short term.
namely we need need stop doing anything that isn't on the critical path towards a new hosting setup.
zas
The most urgent is to have more bandwidth
ruaok
we can get bandwidth easy.
but more bandwidth means more smoking servers or am I wrong on that?
zas
Well, i mean more bandwidth together to more servers
else it serves no purpose ;)
ruaok
so, before we meet in BCN to make the spreadsheet, what tasks do you need to do in order to make the move easier?
zas
That is moving to NewHost asap
ruaok
exactly.
the app servers are easily deployable with fabric and all that, right?
postgres is alright, but replication needs to be setup.
gateways are nearly there, right?
zas
gateways/web servers/redis/memcache are easy (well, minus Murphy's law)
ruaok
CAA, CB, wiki, jira can move later.
zas
ci too
ruaok
wiki better sooner because of wikidocs. but legoktm can probably do that by hand.
zas
stats can be installed fast enough at NewHost
ruaok
let me make a new section in the migration doc that splits these into right away vs later.