#metabrainz

ruaok: well, i'm not sure about the exact issue, i think docker itself is leaking stuff, the unnamed volume isn't really the issue

that's related to aufs, i cleaned up dangling containers/images/volumes, and rebooted to clear temp stuff, but that was only 3% of disk space. Then i stopped docker, moved /var/lib/docker, and restarted it, the difference is huge

only did it on prince for now

i think named volumes for /home/search/ dirs may reduce useless data transfer and possibly loads (since it will not start from zero on container restart)

Ether_Man: we mostly take them from Wikidata, https://github.com/metabrainz/musicbrainz-serve... is the relevant code in case it's useful

(I guess you figured the first half out, but just in case the second half helps :) )

reosarevok, any reason you're not simply using the commons-api? Seems to be a lot of waste to go through wikidata

We're picking the image set as the artist image in Wikidata

(which is the link we have for the artist)

Ether_Man: We try to link as much as possible to Wikidata, which is also what the Wikimedia Foundation prefers to use as their unique identifiers.

If we have a manually-set image for the artist, then we use that instead, of course, but otherwise Wikidata is the obvious place to check :)

So Wikidata is the entry point to Wikipedia, Commons, Quotes, …

Right. Which gives you the html for the image on commons. Isn't it more efficient to ask the commons api for the image url directly?

Can you ask directly for the image linked as main image to a specific wikidata ID?

If so, it'd be, I guess

https://commons.wikimedia.org/w/api.php?action=... as an example

Gives you both the url to the html page, and the image directly, as well as the url to the html for that specific revision

Yes, but you're asking for "Image:AlYankovicByKristineSlipson.jpg"

We don't have that, we just have "this is then entity at Wikidata ID Q24343543" or whatever

*the entity

ruaok: proceeding on boingo now

Right. But the data you get from there, is the url to the html descriptor page of the image.

k

Sure, and then we call Commons with it: https://github.com/metabrainz/musicbrainz-serve... etc

Right, but you're essentially fetching that full html page and then just discarding everything but the image. Rather than asking the API for just the image.

get_commons_image does exactly the API call you said https://github.com/metabrainz/musicbrainz-serve...

(unless I'm missing something)

And before anyone gets annoyed. I'm not meaning this as critique or anything, I'm genuinely interested which is more efficient here

And get_wikidata_properties also makes an API call: https://github.com/metabrainz/musicbrainz-serve...

oh sorry then :)

It does mean we need to make two calls, of course :)

I read it as it was reading the descriptor page :)

But at the same time, it saves us worrying about whether the images move or change or whatever, and if someone adds a new better image to the Wikidata page, we get it automatically, which is nice

Well yea, but one minimal call, and then the image, rather than a whole html page, at least feels like it should be more efficient :)

(before we started basing everything on Wikidata, we'd have to fix a few hundred Wikipedia links a month for example because they kept removing them or redirecting them :D )

But that's also, I guess, why they're not automatically added to the ws results

New blog post: https://blog.musicbrainz.org/2017/05/01/communi... 🙌

(because we'd have to call Wikidata/Commons every to give the URL, which the user might not even need, so it's better if the user does it)

Yeah. We provide the data we have, which includes a lot of additional IDs and links that data users can use to query for more/other data on other services.

If only Plex Agents actually had a system to STORE all those links, they'd be really handy indeed for me :)

Do you sell CDs ?

no. we dont sell anything.

Someone pinched my CRIMCD86 ... I have an empty Case !!

..right

I was expecting a leadup to a joke honestly.

you got it

"Do you dig graves?"

"Yea, they're alright"

"I think they're wonderful"

(and if people get that refrence they are officially Cool™)

hahah.... new jazz group

what part? :P

the digging part bro.... ;)

it's from https://en.wikipedia.org/wiki/The_Young_Ones_(T... actually

[picard] samj1912 closed pull request #713: Remove unneeded unicode prefix (master...remove_unicode) https://git.io/v9WcW

!m Freso

You're doing good work, Freso!

:D

nice blog post :D

also

!m everyone :D

You're doing good work, everyone :D!

:D

Freso: are you going to tweet the community thing with MeB?

[picard] mineo opened pull request #714: Reject functions with required keyword-only arguments (master...reject-kwonly) https://git.io/v9lcH

Ether_Man: so you're developing a MB plex plugin?

Freso: great blog post!

Leo_Verto[m], Sort of. I'm mainly updating and reworking my old one

Sounds pretty useful, even though I use Emby :P

are you setting a custom user agent?

Well it can't get more useless than the last.fm agent that comes with plex at least...

heh

Yes. 'MusicbrainzPlexAgent/1.0.0.0 ( [redacted]@gmail.com )' :)

ah, then you shouldn't run into any problems with rate limiting

well....

I mean, if it's not using a generic UA and there's a way to contact the dev, issues can be resolved more diplomatically

I'm more worried about being banned for doing something really dumb against the API really. But I also know that there's a lot of plex users that are seriously unhappy with the last.fm agent... Doesn't help that they released a paid agent, and broke the entire thing in the process so UI doesn't actually reflect the metadata for things like artist name and album titles... Unless ofc, you're using the paid version...

Leo_Verto[m]: i am a good citizen, too and my script gets a 503 sometimes :p

usually in the evening and night european time.

and i make 1 request every 5 seconds :p

hmm, does the API really throw 503s when rate-limiting?

Yes

According to the doc at least https://musicbrainz.org/doc/XML_Web_Service/Rat... "and decline (http 503) the rest."

possibly relevant: https://tickets.metabrainz.org/browse/MBS-5827

MBS-5827: for /ws/3, use HTTP 429 rather than HTTP 503

yeah, that seems like the cleaner solution

Leo_Verto[m]: yes it does. we used 503 before 429 existed.

ah

and SothoTalKer's 503's might be caused by other things, right?

Leo_Verto[m]: that's actually a fun thing. the first reply is a 503. when you request again too fast, you get an { "Error": "Server is busy" } or something :p

Leo_Verto[m]: no its not. my code is great and has no flaws. :p

hmm, unfortunately stats.meb.org doesn't seem to record 503s but couldn't those also be caused by genuine server-overload?

that's the case, indeed :x

when the usa is sleeping, i almost never get an error :-)

Leo_Verto[m]: https://stats.metabrainz.org/dashboard/db/mbsta... shows 503s

around 7k per minute

wow

yeah, I can see why those aren't on the other graphs :P

and i am responsible for a few :x

are 403s banned clients?

yes

403 are reserved for SothoTalKer. :)

*growl*

that said, 503s can be either individual rate limit or global one (the code is the same)

I didn't realize MB was blocking more than three times as many requests as it let through

one can check headers returned by rate limiter to have further info about the reason of the rate limit

is the list of the worst offenders public?

We block also at IP level (those don't appear of course, mbstats is based on actual web logs)

i can imagine why. All those userscript do heavily make use of the webservice :x

so most of those 403s are probably not offending clients but certain software/apps used by tons of clients then, right?

how would i request a key? do i need to sign up with MB to get a key?

prolly, it doesn't exist yet, but that's planned

you could still have the old keyless api for those people who want to run their own search servers?

are the people causing more than a hundred thousand of failing requests using custom UAs or just generic ones?

how do you even not notice your app being blocked

users are supposed to comply to simple requirements concerning UA and rate, if you are blocked you get 403s not 503s

i should print out my error messages XD

503s are when rate limited (and headers can tell when to retry, and if it is the global rate limit, in which case your rate may be correct, but servers are overloaded)

FYI we serve ~333 200 responses per second on the ws alone, that's ~30M per day, 10501488000 per year...

zas: did you see an increase in requests within the last month?

[picard] mineo opened pull request #715: patch_version: Use sys.platform as the default platform name (master...patch-version-default) https://git.io/v9l8M

Leo_Verto[m]: regarding the survey. what's the difference between rarely and sometimes? :p

SothoTalKer: we had an increase in requests since 4/18

that explains it :-)

SothoTalKer: as a non-native speaker, I'd say rarely is "I've used it once or twice" and sometimes is more along the lines of "I use it every once in a while"

do you feel I should clarify/change that in the survey?

I would not mind clarifying it. One question is marked as mandatory. is this intended?

ruaok, Freso, one of you two interested?: https://twitter.com/Tpt93/status/85902034563385...

I'd do that. :)

SothoTalKer: I was planning to set most of them as mandatory (except for the last section), I just didn't do that for now to make looking through the survey easier

arg now even twitter does this annoying "cropping your avatar into a circle with css/js" bullshit I have no idea why they are doing