and the tag upvoting stuff isn't done, because I've been working on the schema upgrade script testing itself. so the primary things there to be tested are jessew's collections stuff, medium titles longer than 255 characters, and release/RG/recording aliases
2015-05-08 12825, 2015
napper joined the channel
2015-05-08 12806, 2015
kepstin
hmm, looks like ruoak's not around atm
2015-05-08 12816, 2015
kepstin has been really busy the last few days :/
2015-05-08 12859, 2015
ianmcorvidae
I can relay, the thing he emailed you about is via me anyway
2015-05-08 12800, 2015
kepstin
sounds like you think some of the issues might be related to the rate limiting, possibly the incoming direction?
2015-05-08 12806, 2015
ianmcorvidae
yes
2015-05-08 12841, 2015
ianmcorvidae
I'm not sure why there's ingress filtering at all, since that amounts to dropping inbound packets by way of redirecting them (since tc doesn't have support for it in the first place since it's not a terribly reasonable idea...)
2015-05-08 12851, 2015
ianmcorvidae
but on top of that I'm not sure why it's set to 1/4th of our actual limit
2015-05-08 12800, 2015
kepstin
well, it's redirecting them into a traffic shaper
2015-05-08 12819, 2015
kepstin
and i put the limit at that because that's what rob said to use
2015-05-08 12823, 2015
ianmcorvidae
when it's maxing out suddenly we have packet loss (because we told it to drop packets) and everything internally starts getting broken responses because it drops
2015-05-08 12835, 2015
ianmcorvidae
right. so the limit part is just rob being unaware, which I guess is nothing new
2015-05-08 12829, 2015
kepstin
if you don't want the incoming limit at all, it's just a matter of commenting out a chunk of the ingress section of the script and re-running it.
2015-05-08 12840, 2015
ianmcorvidae
I guess I just wonder what the logic is for the ingress filtering
and rerun, and it would clear out the ingress stuff
2015-05-08 12824, 2015
ianmcorvidae
the way the redirect works is putting it into a fake interface and then restricting what comes out of that interface, IIRC, since there's no thing to do except reduce output rate, right?
2015-05-08 12831, 2015
kepstin
yeah
2015-05-08 12847, 2015
ianmcorvidae
the related issue is that the server has been, in these same situations, reporting too many TCP orphans
2015-05-08 12815, 2015
kepstin
has the incoming data rate on the server been hitting 5mbit at those times?
especially right at the start of the month, but also the small spike yesterday is part of that
2015-05-08 12832, 2015
kepstin
hmm. do you know if the data center's limiting it to your nominal ingress rate outside of your stuff?
2015-05-08 12833, 2015
ianmcorvidae
and chunks of today, where it's been near that, same sorts of errors -- incomplete responses from wikipedia and the blog, ping warnings from nagios, etc.
2015-05-08 12842, 2015
ianmcorvidae
they don't limit it, they just charge us if we go over
2015-05-08 12859, 2015
ianmcorvidae
but the way it's calculated is just the max of ingress/egress, 95th-percentile
2015-05-08 12803, 2015
kepstin
that's pretty weird, the link muse be highly oversaturated to do that.
2015-05-08 12842, 2015
kepstin
hmm. so yeah, I'd probably do it first by trying to set the incoming limit at the same as outgoing then, see if that helps
2015-05-08 12820, 2015
ianmcorvidae
alright.
2015-05-08 12824, 2015
kepstin
which basically means just changing everywhere it says '5000kbit' in the script to '20000kbit' and rerunning it.
2015-05-08 12835, 2015
ianmcorvidae
so, other question, why pfifo_fast instead of staying with sfq like the old setup?
2015-05-08 12847, 2015
kepstin
why fq_codel you mean?
2015-05-08 12814, 2015
ianmcorvidae
oh, misread
2015-05-08 12816, 2015
kepstin
in theory, better behaviour under heavy load conditions by limiting queue lengths.
2015-05-08 12819, 2015
ianmcorvidae
I thought the one on the server itself said pfifo
2015-05-08 12824, 2015
ianmcorvidae checks that again
2015-05-08 12839, 2015
kepstin
pfifo_fast is the kernel default, effectively nothing running
2015-05-08 12846, 2015
ianmcorvidae
yeah
2015-05-08 12850, 2015
kepstin
it's a simple tail-drop queue
2015-05-08 12858, 2015
ianmcorvidae
but not in a hierarchy :)
2015-05-08 12820, 2015
ianmcorvidae
anyway, seems reasonable
2015-05-08 12830, 2015
ianmcorvidae
the first result being the bufferbloat people speaks well of the algorithm
2015-05-08 12817, 2015
kepstin
it's not really designed for big infra routers, but 20mbit should be well within the range it can handle properly
2015-05-08 12826, 2015
ianmcorvidae
yeah, certainly
2015-05-08 12820, 2015
kepstin
but the idea is that it's a 'fair queue', so no particular stream is preferred over another when selecting which packets to drop - it tries to keep all streams with about the same b/w
2015-05-08 12838, 2015
kepstin
when you might actually want it to be somewhat unfair in your favour
2015-05-08 12846, 2015
kepstin
hmm.
2015-05-08 12827, 2015
kepstin
well, try giving it the full 20mbit first then.
2015-05-08 12831, 2015
ianmcorvidae
well, we're not even hitting the outbound
2015-05-08 12842, 2015
ianmcorvidae
and the outbound is a lot higher than the inbound, always has been
2015-05-08 12849, 2015
ianmcorvidae
just typically more unbalanced than lately :)
2015-05-08 12855, 2015
kepstin
if you're charged by max of in/out when it goes over 20mbit, there's no reason to limit incoming to less.
2015-05-08 12830, 2015
ianmcorvidae
yeah
2015-05-08 12824, 2015
kepstin
but yeah, the script's designed so that re-running it will do a fairly clean update. It basically removes all the qdiscs and the traffic redirect for incoming, and then puts them back with the new settings.
2015-05-08 12852, 2015
ianmcorvidae
yeah
2015-05-08 12827, 2015
kepstin had a fun time at work today prodding the poorly documented linux ipsec tools into connecting a couple of subnets via a vpn tunnel :/
2015-05-08 12846, 2015
kepstin
(it had to be ipsec, because I don't control the other end...)
2015-05-08 12856, 2015
ianmcorvidae
heh
2015-05-08 12859, 2015
ianmcorvidae
sounds like "fun"
2015-05-08 12819, 2015
ianmcorvidae
pretty sure our (openvpn) vpn doesn't even work properly any more, I've just been using ssh tunners instead
2015-05-08 12823, 2015
kepstin
in the end, it turned out that I had it working halfway through the day before I took it apart again - my problem was that I had the mtu set wrong.
2015-05-08 12841, 2015
kepstin
if you can ever ping a machine but ssh hangs, that's usually the issue :/
2015-05-08 12833, 2015
kepstin
(also, apparently amazon vpc machines have a default mtu of 9001, which is hilarious.)
2015-05-08 12822, 2015
ianmcorvidae
(schema mbsandbox now has AC credits patch as well, for those watching at home)
2015-05-08 12844, 2015
napper joined the channel
2015-05-08 12857, 2015
napper joined the channel
2015-05-08 12859, 2015
napper joined the channel
2015-05-08 12844, 2015
napper joined the channel
2015-05-08 12813, 2015
Junior joined the channel
2015-05-08 12829, 2015
bitmap
o_O so the pgtap failures are related to LC_CTYPE/LC_COLLATE settings...if I harcode them to 'en_US.UTF-8' in InitDb.pl, and rerun create_test_db, the tests pass. but I can't think of anything that changed in this regard
2015-05-08 12804, 2015
diana_olhovik joined the channel
2015-05-08 12834, 2015
bitmap
dropping the test database, switching to beta code, and running create_test_db does not help, so something outside must've changed
2015-05-08 12848, 2015
reosarevok joined the channel
2015-05-08 12817, 2015
ianmcorvidae|alt joined the channel
2015-05-08 12847, 2015
Junior_ joined the channel
2015-05-08 12827, 2015
napper joined the channel
2015-05-08 12805, 2015
ruaok joined the channel
2015-05-08 12800, 2015
ruaok
waking up, reading gentlecat PR. \ø/
2015-05-08 12806, 2015
ruaok
:)
2015-05-08 12813, 2015
Gentlecat
just like old times
2015-05-08 12826, 2015
ruaok
I was *about* to say *exactly* that. :)
2015-05-08 12817, 2015
napper joined the channel
2015-05-08 12831, 2015
ijabz2 joined the channel
2015-05-08 12821, 2015
ijabz2
Just have to express my frustration, uk is soooooo fucked
2015-05-08 12817, 2015
ariscop joined the channel
2015-05-08 12849, 2015
ruaok
ijabz2: I hear that. it is exactly this BS that made me leave the US.
2015-05-08 12858, 2015
ruaok
seems like the UK is trying to copy the US as fast as it can.
2015-05-08 12807, 2015
ruaok
and not any of the best parts. only the worst. :(
2015-05-08 12823, 2015
ruaok
seems a lot of my friends are in the same boat as you. asking where next?
2015-05-08 12807, 2015
ijabz2
Yep, the tories ran a classic scaremongering campaign (esp Scotland) that has worked
2015-05-08 12811, 2015
reosarevok shrugs. The Scottish people I know weren't particularly scared, just tired of Labour being Tories v.2 where Scotland is concerned
2015-05-08 12840, 2015
reosarevok
As one friend of a friend said, "Even if Scotland returned all 59 seats as labour, based on exit poll, it would not influence the outcome. But, it would appear, we've all gone yellow. Lovely, bright, fuck you yellow."
2015-05-08 12810, 2015
ianmcorvidae wonders if ijabz meant fearmongering within england, *about* scotland, more than fearmongering in scotland itself?
2015-05-08 12801, 2015
ianmcorvidae
(that's what I've seen happening, at least, in my obviously not very comprehensive view on the thing)
2015-05-08 12807, 2015
ijabz2
ianmcorvidae yep
2015-05-08 12858, 2015
ijabz2
people voted ukip/tory instead of labour because of Scotland thing
2015-05-08 12828, 2015
ijabz2
and libdems voters couldnt bring themselves to vote libdems again - handling seats to the tories
2015-05-08 12803, 2015
reosarevok
The way your voting works is insane, also
2015-05-08 12811, 2015
ianmcorvidae
speaking of being like the US :P
2015-05-08 12820, 2015
reosarevok
UKIP was the party with a biggest growth in voting, yet they actually lost a seat?
2015-05-08 12824, 2015
reosarevok
(not that I'm complaining, but)
2015-05-08 12833, 2015
ijabz2
well yeah the voting system is insane
2015-05-08 12801, 2015
zarlino joined the channel
2015-05-08 12845, 2015
adhawkins
Morning all. Updated my Debian last night from Wheezy to Jessie. As a result I had to blow away my local perl directory.
2015-05-08 12848, 2015
zarlino
hi all, I’m using the XML ws. I can get the “composer” relation for a work. Is there a way to distinguish classical composers from pop song authors?
2015-05-08 12801, 2015
adhawkins
Just trying the server this mrorning, and I'm getting 'plugin error - JSON.Escape: plugin not found'
2015-05-08 12809, 2015
adhawkins
Is this a perl module I'm missing?
2015-05-08 12850, 2015
ruaok knows neither answer
2015-05-08 12852, 2015
reosarevok
zarlino: nope
2015-05-08 12834, 2015
zarlino
reosarevok: too bad. This makes the data much less useful
2015-05-08 12845, 2015
adhawkins
Template::Plugin::JSON::Escape looks possible.
2015-05-08 12847, 2015
adhawkins
I'll install that.
2015-05-08 12850, 2015
ianmcorvidae
adhawkins: yeah, it's that module
2015-05-08 12801, 2015
reosarevok shrugs. You can maybe compare track artists and composers, since classical style mandates composer as track artist
2015-05-08 12818, 2015
ianmcorvidae
zarlino: if you can distinguish classical from non-classical you're ahead of us as far as the weirder edge-cases :) that's sort of the core problem
2015-05-08 12834, 2015
adhawkins
Seems to have done the trick.
2015-05-08 12800, 2015
adhawkins
I had a number of modules that I had to install manually. Not sure why. I also had to downgrade a postgres module because it was failing to import.
2015-05-08 12806, 2015
adhawkins
Seems to be up and running again now.
2015-05-08 12811, 2015
reosarevok
Not even that much of an edge case, I mean, Gershwin? :)
2015-05-08 12811, 2015
Lotheric joined the channel
2015-05-08 12813, 2015
adhawkins
Also, rather bizarrely, I'm getting occasional 'no space left on device' messages.
2015-05-08 12822, 2015
adhawkins
All disks have loads of free space and inodes too...
2015-05-08 12841, 2015
ianmcorvidae
are you using a tmpdir setup that makes a ramdisk?
2015-05-08 12801, 2015
adhawkins
Yes, memory exhausted?
2015-05-08 12811, 2015
ianmcorvidae
or some other limit of that sort
2015-05-08 12813, 2015
ianmcorvidae
but yeah
2015-05-08 12827, 2015
adhawkins
According to df and free I have plenty...
2015-05-08 12836, 2015
ianmcorvidae
definitely at least for database imports you'll want an alternate temporary directory, possibly also for replication (which ends up using that as well)
2015-05-08 12856, 2015
zarlino
reosarevok: so if the track artist is the same as the composer that would be classical? What about solo artists (say, lou reed)?
2015-05-08 12858, 2015
ianmcorvidae
where precisely are you seeing the no space left on device errors?
2015-05-08 12808, 2015
reosarevok
zarlino: not saying it's always the case :)
2015-05-08 12811, 2015
adhawkins
When I type 'reboot'!
2015-05-08 12821, 2015
ianmcorvidae
weird
2015-05-08 12826, 2015
adhawkins
Very :)
2015-05-08 12839, 2015
reosarevok
But basically, there's no clear way of distinguishing the two. You can check for a "classical" tag, I guess?
2015-05-08 12843, 2015
zarlino
reosarevok: of course, I’m just trying to understand what you said. I’m not a MB expert.
