#musicbrainz-devel

All done, catching up on replication packets since the dump now.

alright folks, schema change UI testing sandbox is up: http://schema.mbsandbox.org/

right now it's somewhat limited

most notably, perhaps, https://bitbucket.org/metabrainz/musicbrainz-se... is not merged due to my comment there

and the tag upvoting stuff isn't done, because I've been working on the schema upgrade script testing itself. so the primary things there to be tested are jessew's collections stuff, medium titles longer than 255 characters, and release/RG/recording aliases

hmm, looks like ruoak's not around atm

I can relay, the thing he emailed you about is via me anyway

sounds like you think some of the issues might be related to the rate limiting, possibly the incoming direction?

yes

I'm not sure why there's ingress filtering at all, since that amounts to dropping inbound packets by way of redirecting them (since tc doesn't have support for it in the first place since it's not a terribly reasonable idea...)

but on top of that I'm not sure why it's set to 1/4th of our actual limit

well, it's redirecting them into a traffic shaper

and i put the limit at that because that's what rob said to use

when it's maxing out suddenly we have packet loss (because we told it to drop packets) and everything internally starts getting broken responses because it drops

right. so the limit part is just rob being unaware, which I guess is nothing new

if you don't want the incoming limit at all, it's just a matter of commenting out a chunk of the ingress section of the script and re-running it.

I guess I just wonder what the logic is for the ingress filtering

hmm. on https://gist.github.com/kepstin/901b5a7e523e496... you'd comment out lines 58-59, 62-63, and 68-end

and rerun, and it would clear out the ingress stuff

the way the redirect works is putting it into a fake interface and then restricting what comes out of that interface, IIRC, since there's no thing to do except reduce output rate, right?

yeah

the related issue is that the server has been, in these same situations, reporting too many TCP orphans

has the incoming data rate on the server been hitting 5mbit at those times?

yes

see the light-blue on http://stats.musicbrainz.org/mrtg/95th-percenti...

especially right at the start of the month, but also the small spike yesterday is part of that

hmm. do you know if the data center's limiting it to your nominal ingress rate outside of your stuff?

and chunks of today, where it's been near that, same sorts of errors -- incomplete responses from wikipedia and the blog, ping warnings from nagios, etc.

they don't limit it, they just charge us if we go over

but the way it's calculated is just the max of ingress/egress, 95th-percentile

that's pretty weird, the link muse be highly oversaturated to do that.

hmm. so yeah, I'd probably do it first by trying to set the incoming limit at the same as outgoing then, see if that helps

alright.

so, other question, why pfifo_fast instead of staying with sfq like the old setup?

why fq_codel you mean?

oh, misread

in theory, better behaviour under heavy load conditions by limiting queue lengths.

I thought the one on the server itself said pfifo

pfifo_fast is the kernel default, effectively nothing running

yeah

it's a simple tail-drop queue

but not in a hierarchy :)

anyway, seems reasonable

the first result being the bufferbloat people speaks well of the algorithm

it's not really designed for big infra routers, but 20mbit should be well within the range it can handle properly

yeah, certainly

but the idea is that it's a 'fair queue', so no particular stream is preferred over another when selecting which packets to drop - it tries to keep all streams with about the same b/w

when you might actually want it to be somewhat unfair in your favour

hmm.

well, try giving it the full 20mbit first then.

well, we're not even hitting the outbound

and the outbound is a lot higher than the inbound, always has been

just typically more unbalanced than lately :)

if you're charged by max of in/out when it goes over 20mbit, there's no reason to limit incoming to less.

yeah

but yeah, the script's designed so that re-running it will do a fairly clean update. It basically removes all the qdiscs and the traffic redirect for incoming, and then puts them back with the new settings.

yeah

(it had to be ipsec, because I don't control the other end...)

heh

sounds like "fun"

pretty sure our (openvpn) vpn doesn't even work properly any more, I've just been using ssh tunners instead

in the end, it turned out that I had it working halfway through the day before I took it apart again - my problem was that I had the mtu set wrong.

if you can ever ping a machine but ssh hangs, that's usually the issue :/

(also, apparently amazon vpc machines have a default mtu of 9001, which is hilarious.)

(schema mbsandbox now has AC credits patch as well, for those watching at home)

o_O so the pgtap failures are related to LC_CTYPE/LC_COLLATE settings...if I harcode them to 'en_US.UTF-8' in InitDb.pl, and rerun create_test_db, the tests pass. but I can't think of anything that changed in this regard

dropping the test database, switching to beta code, and running create_test_db does not help, so something outside must've changed

waking up, reading gentlecat PR. \ø/

:)

just like old times

I was *about* to say *exactly* that. :)

Just have to express my frustration, uk is soooooo fucked

ijabz2: I hear that. it is exactly this BS that made me leave the US.

seems like the UK is trying to copy the US as fast as it can.

and not any of the best parts. only the worst. :(

seems a lot of my friends are in the same boat as you. asking where next?

Yep, the tories ran a classic scaremongering campaign (esp Scotland) that has worked

As one friend of a friend said, "Even if Scotland returned all 59 seats as labour, based on exit poll, it would not influence the outcome. But, it would appear, we've all gone yellow. Lovely, bright, fuck you yellow."

(that's what I've seen happening, at least, in my obviously not very comprehensive view on the thing)

ianmcorvidae yep

people voted ukip/tory instead of labour because of Scotland thing

and libdems voters couldnt bring themselves to vote libdems again - handling seats to the tories

The way your voting works is insane, also

speaking of being like the US :P

UKIP was the party with a biggest growth in voting, yet they actually lost a seat?

(not that I'm complaining, but)

well yeah the voting system is insane

Morning all. Updated my Debian last night from Wheezy to Jessie. As a result I had to blow away my local perl directory.

hi all, I’m using the XML ws. I can get the “composer” relation for a work. Is there a way to distinguish classical composers from pop song authors?

Just trying the server this mrorning, and I'm getting 'plugin error - JSON.Escape: plugin not found'

Is this a perl module I'm missing?

zarlino: nope

reosarevok: too bad. This makes the data much less useful

Template::Plugin::JSON::Escape looks possible.

I'll install that.

adhawkins: yeah, it's that module

zarlino: if you can distinguish classical from non-classical you're ahead of us as far as the weirder edge-cases :) that's sort of the core problem

Seems to have done the trick.

I had a number of modules that I had to install manually. Not sure why. I also had to downgrade a postgres module because it was failing to import.

Seems to be up and running again now.

Not even that much of an edge case, I mean, Gershwin? :)

Also, rather bizarrely, I'm getting occasional 'no space left on device' messages.

All disks have loads of free space and inodes too...

are you using a tmpdir setup that makes a ramdisk?

Yes, memory exhausted?

or some other limit of that sort

but yeah

According to df and free I have plenty...

definitely at least for database imports you'll want an alternate temporary directory, possibly also for replication (which ends up using that as well)

reosarevok: so if the track artist is the same as the composer that would be classical? What about solo artists (say, lou reed)?

where precisely are you seeing the no space left on device errors?

zarlino: not saying it's always the case :)

When I type 'reboot'!

weird

Very :)

But basically, there's no clear way of distinguishing the two. You can check for a "classical" tag, I guess?

reosarevok: of course, I’m just trying to understand what you said. I’m not a MB expert.

hm, /tmp itself isn't a tmpfs so harder to say

Yep, just tried another reboot and it did it again.

But yeah, basically nobody has come with a way of saying "what is classical" that isn't "you'll know when you see it"

And nowadays even classical composers have non-classical albums too sometimes :)

So the difference is kinda artificial

reosarevok: yes, you’re right.