you think that's bad? once it's done I then have to do a chmod
2014-10-28 30151, 2014
ianmcorvidae
heh, fun. recursive, I'm sure
2014-10-28 30105, 2014
ianmcorvidae
(since I imagine it's constant time if you're not recursing)
2014-10-28 30156, 2014
ruaok
ianmcorvidae: that gist looks sane to me.
2014-10-28 30107, 2014
ianmcorvidae
alright
2014-10-28 30108, 2014
ruaok
and I can take the seach related bits, no biggie.
2014-10-28 30124, 2014
ianmcorvidae
yeah, I figured everything not explicitly labeled as yours we'd split up
2014-10-28 30125, 2014
ianmcorvidae
with more to me since I won't be sitting in a data center :)
2014-10-28 30152, 2014
chirlu`
You might consider warming the DB cache by running some queries such as SELECT * FROM artist on the big tables (or EXPLAIN ANALYZE SELECT ..., to discard the output immediately).
2014-10-28 30106, 2014
chirlu`
Makes the downtime a bit longer, but reduces 502s afterwards (hopefully).
2014-10-28 30144, 2014
ianmcorvidae
100% 503s versus a few more 502s than usual, the former seems like it's blocking more legitimate usage :)
2014-10-28 30148, 2014
chirlu`
Because sequentially reading the whole table is faster than random access to the bits required for live pages.
2014-10-28 30118, 2014
ianmcorvidae
might do that after things are back up though
2014-10-28 30154, 2014
chirlu`
Technically true, but psychologically people may be more annoyed by 502s after “everything is supposed to be back up” than by a few minutes more downtime.
2014-10-28 30144, 2014
chirlu`
The DB may also need a configuration change to actually use more RAM for caching.
2014-10-28 30155, 2014
ianmcorvidae
ah, config is a good point
2014-10-28 30100, 2014
ianmcorvidae checks the current config
2014-10-28 30109, 2014
ianmcorvidae
current shared_buffers is 10GB, I guess doubling that would make sense
2014-10-28 30138, 2014
ruaok muses that there are no seats in the data center
2014-10-28 30140, 2014
ruaok
:)
2014-10-28 30123, 2014
ianmcorvidae wonders if we'll need to increase shmmax/shmall too. probably, I guess
2014-10-28 30133, 2014
ruaok
yes
2014-10-28 30117, 2014
ruaok
looks good to me.
2014-10-28 30100, 2014
ruaok
ok, time to move on. again. off to JS again.
2014-10-28 30102, 2014
ruaok
ttfn
2014-10-28 30119, 2014
ruaok
not javascript. san jose. not sure which is worse.
2014-10-28 30131, 2014
SultS joined the channel
2014-10-28 30110, 2014
nikki joined the channel
2014-10-28 30119, 2014
ohrstrom joined the channel
2014-10-28 30150, 2014
nikki
I suppose it might be a silly question since it's not exactly easy to notice, but is anyone else seeing any missing mb emails? there was a period of about 6 hours or something a few days ago where jira failed to send any emails (they didn't even appear in mb-bugs) and now someone has said that they didn't receive some edit note emails
2014-10-28 30106, 2014
Freso
ianmcorvidae: ^
2014-10-28 30126, 2014
kepstin-laptop is getting jira emails
2014-10-28 30100, 2014
kepstin-laptop
and subscription emails
2014-10-28 30146, 2014
nikki
but are you receiving *all* of them? :P
2014-10-28 30116, 2014
kepstin-laptop
hard to tell :)
2014-10-28 30124, 2014
nikki
exactly :/
2014-10-28 30145, 2014
kepstin-laptop
my jira STYLE filter automatic email isn't coming every day, but I dunno if that's just because it didn't send it every day (no changes?) or what.
I wonder what my hair would do if I cut it short :P I haven't had it short since before puberty and my hair wasn't curly then
2014-10-28 30159, 2014
ruaok pokes at the search servers
2014-10-28 30103, 2014
ruaok
wtf is going on?
2014-10-28 30114, 2014
ianmcorvidae
roobarb is being shit, apparently :P
2014-10-28 30128, 2014
ianmcorvidae reverts the weight change and pulls it out to restart jetty
2014-10-28 30104, 2014
ruaok
ianmcorvidae: do you know where I need to change my pw to acces nagios.mb.org
2014-10-28 30105, 2014
ruaok
?
2014-10-28 30134, 2014
ianmcorvidae
I'm not sure where it can be changed, but I can show you where it's set, once I find it :)
2014-10-28 30142, 2014
ruaok
no need to change it, really. I just wanted to see what else needs attention.
2014-10-28 30149, 2014
ruaok
the rsync stuff apparently does.
2014-10-28 30107, 2014
ianmcorvidae
there's one of them that has always failed to work
2014-10-28 30111, 2014
ianmcorvidae
which should be disabled
2014-10-28 30124, 2014
ianmcorvidae
not sure about others, the replication ones are at least *better* lately but not perfect
2014-10-28 30111, 2014
ruaok
at least the search servers are looking fine now...
2014-10-28 30119, 2014
ianmcorvidae
yeah, I kicked roobarb's jetty
2014-10-28 30120, 2014
ruaok
voodoo.
2014-10-28 30123, 2014
ruaok
thx
2014-10-28 30133, 2014
ianmcorvidae
and restored them to equal weight
2014-10-28 30156, 2014
Gentlecat joined the channel
2014-10-28 30101, 2014
ruaok
hi Gentlecat
2014-10-28 30101, 2014
_Dave_
So I think I have a MB slave server running ... are there unit tests for the entire system I can run?
2014-10-28 30106, 2014
ruaok
enjoying SF?
2014-10-28 30111, 2014
Gentlecat
not yet
2014-10-28 30130, 2014
Gentlecat
just arrived there
2014-10-28 30142, 2014
ruaok
heh, I just arrived in San Jose.
2014-10-28 30119, 2014
_Dave_
Alternatively, I can program my mediafile importer to do two lookups, one to my slave server and one to your master ones and compare? O.o
2014-10-28 30146, 2014
ruaok
_Dave_: we don’t have anything to test a full live running setup.
2014-10-28 30154, 2014
_Dave_
ah ok :)
2014-10-28 30101, 2014
ruaok
my super quick check is to do a search, then to load any random result.
2014-10-28 30105, 2014
_Dave_
I did that
2014-10-28 30110, 2014
ruaok
that tests the web servers, DB and search servers.
2014-10-28 30122, 2014
ruaok
if that is ok, the rest has a decent chance of being ok as well
2014-10-28 30122, 2014
_Dave_
there's a number of complex systems hooked together here
2014-10-28 30133, 2014
ruaok
you should see the real site. :)
2014-10-28 30143, 2014
ruaok
rube goldberg comes to mind. :)
2014-10-28 30149, 2014
_Dave_
lol
2014-10-28 30107, 2014
_Dave_
totally familiar with that, I've been an SA since 1980...seen enough of em
2014-10-28 30100, 2014
ruaok
ianmcorvidae: so, looking at the nagios list, I can’t really tell which of the tests are still complaining. all thes ones that seem related to rsync have been failing for… nearly 2 years. :)
2014-10-28 30159, 2014
Freso *cough*submitting admitting to be an experienced SA*cough* >_>
2014-10-28 30103, 2014
Freso
*someone
2014-10-28 30124, 2014
Freso
Submitting... I need to do less ABz stuff, I think.
2014-10-28 30119, 2014
ruaok
ianmcorvidae: well, I see that all the replication packets are up to date.
2014-10-28 30125, 2014
ruaok
I’ll just leave it be for now.
2014-10-28 30128, 2014
ianmcorvidae
ruaok: well, at present nothing is actively in a warning or critical state, so that seems correct
2014-10-28 30137, 2014
ruaok
ok. kewl.
2014-10-28 30141, 2014
ruaok stands down
2014-10-28 30154, 2014
ruaok
> rob: Insert new memory
2014-10-28 30109, 2014
ruaok
++ > rob: remove old memory
2014-10-28 30110, 2014
ruaok
:)
2014-10-28 30138, 2014
ianmcorvidae
I could just change it to "change memory" :P I'd assumed it was just adding in more sticks to empty slots, I guess
2014-10-28 30154, 2014
ruaok
no, all slots are filled with 2GB sticks.
2014-10-28 30105, 2014
ruaok
so, I need to yank those and replace them with 4GB sticks.
2014-10-28 30112, 2014
ruaok
and yes, I’m just being pedantic. :)
2014-10-28 30137, 2014
Freso
ruaok: Well, your website sucks too.
2014-10-28 30118, 2014
ruaok knows that
2014-10-28 30100, 2014
Freso
(Actually, it is not too bad once NoScript is told your site isn't likely to burn down my apartment.)
2014-10-28 30115, 2014
Freso
(But it is a bit sucky to require JS. :p)
2014-10-28 30144, 2014
chirlu`
nikki: I noticed sometimes (but not recently) that JIRA emails were delayed, i.e. I could see updates in the activity stream, but the email notifications came hours later.
2014-10-28 30156, 2014
CatQuest will note that for a cheap coin he will do any editing someone will tell him to do. so if the bbc wans I can add this information or any such information.
