ruaok: well, i'm not sure about the exact issue, i think docker itself is leaking stuff, the unnamed volume isn't really the issue
2017-05-01 12158, 2017
zas
that's related to aufs, i cleaned up dangling containers/images/volumes, and rebooted to clear temp stuff, but that was only 3% of disk space. Then i stopped docker, moved /var/lib/docker, and restarted it, the difference is huge
2017-05-01 12116, 2017
zas
only did it on prince for now
2017-05-01 12132, 2017
zas
i think named volumes for /home/search/ dirs may reduce useless data transfer and possibly loads (since it will not start from zero on container restart)
Right, but you're essentially fetching that full html page and then just discarding everything but the image. Rather than asking the API for just the image.
It does mean we need to make two calls, of course :)
2017-05-01 12141, 2017
Ether_Man
I read it as it was reading the descriptor page :)
2017-05-01 12123, 2017
reosarevok
But at the same time, it saves us worrying about whether the images move or change or whatever, and if someone adds a new better image to the Wikidata page, we get it automatically, which is nice
2017-05-01 12136, 2017
Ether_Man
Well yea, but one minimal call, and then the image, rather than a whole html page, at least feels like it should be more efficient :)
2017-05-01 12155, 2017
reosarevok
(before we started basing everything on Wikidata, we'd have to fix a few hundred Wikipedia links a month for example because they kept removing them or redirecting them :D )
2017-05-01 12125, 2017
reosarevok
But that's also, I guess, why they're not automatically added to the ws results
(because we'd have to call Wikidata/Commons every to give the URL, which the user might not even need, so it's better if the user does it)
2017-05-01 12117, 2017
Freso
Yeah. We provide the data we have, which includes a lot of additional IDs and links that data users can use to query for more/other data on other services.
2017-05-01 12122, 2017
Ether_Man
If only Plex Agents actually had a system to STORE all those links, they'd be really handy indeed for me :)
2017-05-01 12116, 2017
agentsim has quit
2017-05-01 12143, 2017
Norwich_ joined the channel
2017-05-01 12106, 2017
Norwich_
Do you sell CDs ?
2017-05-01 12100, 2017
ruaok
no. we dont sell anything.
2017-05-01 12159, 2017
Norwich_
Someone pinched my CRIMCD86 ... I have an empty Case !!
2017-05-01 12132, 2017
Norwich_ has quit
2017-05-01 12157, 2017
ZaphodBeeblebrox
..right
2017-05-01 12113, 2017
ZaphodBeeblebrox
I was expecting a leadup to a joke honestly.
2017-05-01 12151, 2017
MajorLurker
you got it
2017-05-01 12125, 2017
ZaphodBeeblebrox
"Do you dig graves?"
2017-05-01 12125, 2017
ZaphodBeeblebrox
"Yea, they're alright"
2017-05-01 12125, 2017
ZaphodBeeblebrox
"I think they're wonderful"
2017-05-01 12150, 2017
ZaphodBeeblebrox
(and if people get that refrence they are officially Cool™)
ah, then you shouldn't run into any problems with rate limiting
2017-05-01 12143, 2017
SothoTalKer
well....
2017-05-01 12134, 2017
Leo_Verto[m]
I mean, if it's not using a generic UA and there's a way to contact the dev, issues can be resolved more diplomatically
2017-05-01 12112, 2017
Ether_Man
I'm more worried about being banned for doing something really dumb against the API really. But I also know that there's a lot of plex users that are seriously unhappy with the last.fm agent... Doesn't help that they released a paid agent, and broke the entire thing in the process so UI doesn't actually reflect the metadata for things like artist name and album titles... Unless ofc, you're using the paid version...
2017-05-01 12110, 2017
SothoTalKer
Leo_Verto[m]: i am a good citizen, too and my script gets a 503 sometimes :p
2017-05-01 12132, 2017
SothoTalKer
usually in the evening and night european time.
2017-05-01 12150, 2017
SothoTalKer
and i make 1 request every 5 seconds :p
2017-05-01 12142, 2017
Leo_Verto[m]
hmm, does the API really throw 503s when rate-limiting?
MBS-5827: for /ws/3, use HTTP 429 rather than HTTP 503
2017-05-01 12152, 2017
Leo_Verto[m]
yeah, that seems like the cleaner solution
2017-05-01 12157, 2017
ruaok
Leo_Verto[m]: yes it does. we used 503 before 429 existed.
2017-05-01 12104, 2017
Leo_Verto[m]
ah
2017-05-01 12130, 2017
Leo_Verto[m]
and SothoTalKer's 503's might be caused by other things, right?
2017-05-01 12110, 2017
SothoTalKer
Leo_Verto[m]: that's actually a fun thing. the first reply is a 503. when you request again too fast, you get an { "Error": "Server is busy" } or something :p
2017-05-01 12133, 2017
SothoTalKer
Leo_Verto[m]: no its not. my code is great and has no flaws. :p
2017-05-01 12149, 2017
Leo_Verto[m]
hmm, unfortunately stats.meb.org doesn't seem to record 503s but couldn't those also be caused by genuine server-overload?
2017-05-01 12124, 2017
SothoTalKer
that's the case, indeed :x
2017-05-01 12146, 2017
SothoTalKer
when the usa is sleeping, i almost never get an error :-)
yeah, I can see why those aren't on the other graphs :P
2017-05-01 12145, 2017
SothoTalKer
and i am responsible for a few :x
2017-05-01 12145, 2017
Leo_Verto[m]
are 403s banned clients?
2017-05-01 12151, 2017
zas
yes
2017-05-01 12102, 2017
ruaok
403 are reserved for SothoTalKer. :)
2017-05-01 12126, 2017
SothoTalKer
*growl*
2017-05-01 12135, 2017
zas
that said, 503s can be either individual rate limit or global one (the code is the same)
2017-05-01 12150, 2017
Leo_Verto[m]
I didn't realize MB was blocking more than three times as many requests as it let through
2017-05-01 12159, 2017
zas
one can check headers returned by rate limiter to have further info about the reason of the rate limit
2017-05-01 12108, 2017
Leo_Verto[m]
is the list of the worst offenders public?
2017-05-01 12142, 2017
zas
We block also at IP level (those don't appear of course, mbstats is based on actual web logs)
2017-05-01 12105, 2017
SothoTalKer
i can imagine why. All those userscript do heavily make use of the webservice :x
2017-05-01 12141, 2017
Leo_Verto[m]
so most of those 403s are probably not offending clients but certain software/apps used by tons of clients then, right?
2017-05-01 12138, 2017
zas
yes, this is why we'll move to api key based ws in the future, to have better control
2017-05-01 12102, 2017
SothoTalKer
how would i request a key? do i need to sign up with MB to get a key?
2017-05-01 12149, 2017
zas
prolly, it doesn't exist yet, but that's planned
2017-05-01 12156, 2017
SothoTalKer
you could still have the old keyless api for those people who want to run their own search servers?
2017-05-01 12125, 2017
Leo_Verto[m]
are the people causing more than a hundred thousand of failing requests using custom UAs or just generic ones?
2017-05-01 12134, 2017
Leo_Verto[m]
how do you even not notice your app being blocked
2017-05-01 12118, 2017
khan joined the channel
2017-05-01 12147, 2017
zas
users are supposed to comply to simple requirements concerning UA and rate, if you are blocked you get 403s not 503s
2017-05-01 12108, 2017
SothoTalKer
i should print out my error messages XD
2017-05-01 12119, 2017
zas
503s are when rate limited (and headers can tell when to retry, and if it is the global rate limit, in which case your rate may be correct, but servers are overloaded)
2017-05-01 12139, 2017
zas
FYI we serve ~333 200 responses per second on the ws alone, that's ~30M per day, 10501488000 per year...
2017-05-01 12100, 2017
SothoTalKer
zas: did you see an increase in requests within the last month?
2017-05-01 12123, 2017
hibiscuskazeneko has quit
2017-05-01 12101, 2017
github joined the channel
2017-05-01 12101, 2017
github
[picard] mineo opened pull request #715: patch_version: Use sys.platform as the default platform name (master...patch-version-default) https://git.io/v9l8M
2017-05-01 12101, 2017
github has left the channel
2017-05-01 12106, 2017
SothoTalKer
Leo_Verto[m]: regarding the survey. what's the difference between rarely and sometimes? :p
2017-05-01 12115, 2017
Slurpee joined the channel
2017-05-01 12116, 2017
Slurpee has quit
2017-05-01 12116, 2017
Slurpee joined the channel
2017-05-01 12105, 2017
zas
SothoTalKer: we had an increase in requests since 4/18
2017-05-01 12100, 2017
SothoTalKer
that explains it :-)
2017-05-01 12120, 2017
Leo_Verto[m]
SothoTalKer: as a non-native speaker, I'd say rarely is "I've used it once or twice" and sometimes is more along the lines of "I use it every once in a while"
2017-05-01 12132, 2017
Leo_Verto[m]
do you feel I should clarify/change that in the survey?
2017-05-01 12127, 2017
SothoTalKer
I would not mind clarifying it. One question is marked as mandatory. is this intended?
SothoTalKer: I was planning to set most of them as mandatory (except for the last section), I just didn't do that for now to make looking through the survey easier
2017-05-01 12106, 2017
ZaphodBeeblebrox
arg now even twitter does this annoying "cropping your avatar into a circle with css/js" bullshit I have no idea why they are doing
