you think that's bad? once it's done I then have to do a chmod
ianmcorvidae
heh, fun. recursive, I'm sure
(since I imagine it's constant time if you're not recursing)
ruaok
ianmcorvidae: that gist looks sane to me.
ianmcorvidae
alright
ruaok
and I can take the seach related bits, no biggie.
ianmcorvidae
yeah, I figured everything not explicitly labeled as yours we'd split up
with more to me since I won't be sitting in a data center :)
chirlu`
You might consider warming the DB cache by running some queries such as SELECT * FROM artist on the big tables (or EXPLAIN ANALYZE SELECT ..., to discard the output immediately).
Makes the downtime a bit longer, but reduces 502s afterwards (hopefully).
ianmcorvidae
100% 503s versus a few more 502s than usual, the former seems like it's blocking more legitimate usage :)
chirlu`
Because sequentially reading the whole table is faster than random access to the bits required for live pages.
ianmcorvidae
might do that after things are back up though
chirlu`
Technically true, but psychologically people may be more annoyed by 502s after “everything is supposed to be back up” than by a few minutes more downtime.
The DB may also need a configuration change to actually use more RAM for caching.
ianmcorvidae
ah, config is a good point
ianmcorvidae checks the current config
current shared_buffers is 10GB, I guess doubling that would make sense
ruaok muses that there are no seats in the data center
ruaok
:)
ianmcorvidae wonders if we'll need to increase shmmax/shmall too. probably, I guess
yes
looks good to me.
ok, time to move on. again. off to JS again.
ttfn
not javascript. san jose. not sure which is worse.
SultS joined the channel
nikki joined the channel
ohrstrom joined the channel
nikki
I suppose it might be a silly question since it's not exactly easy to notice, but is anyone else seeing any missing mb emails? there was a period of about 6 hours or something a few days ago where jira failed to send any emails (they didn't even appear in mb-bugs) and now someone has said that they didn't receive some edit note emails
Freso
ianmcorvidae: ^
kepstin-laptop is getting jira emails
kepstin-laptop
and subscription emails
nikki
but are you receiving *all* of them? :P
kepstin-laptop
hard to tell :)
nikki
exactly :/
kepstin-laptop
my jira STYLE filter automatic email isn't coming every day, but I dunno if that's just because it didn't send it every day (no changes?) or what.
I wonder what my hair would do if I cut it short :P I haven't had it short since before puberty and my hair wasn't curly then
ruaok pokes at the search servers
ruaok
wtf is going on?
ianmcorvidae
roobarb is being shit, apparently :P
ianmcorvidae reverts the weight change and pulls it out to restart jetty
ruaok
ianmcorvidae: do you know where I need to change my pw to acces nagios.mb.org
?
ianmcorvidae
I'm not sure where it can be changed, but I can show you where it's set, once I find it :)
ruaok
no need to change it, really. I just wanted to see what else needs attention.
the rsync stuff apparently does.
ianmcorvidae
there's one of them that has always failed to work
which should be disabled
not sure about others, the replication ones are at least *better* lately but not perfect
ruaok
at least the search servers are looking fine now...
ianmcorvidae
yeah, I kicked roobarb's jetty
ruaok
voodoo.
thx
ianmcorvidae
and restored them to equal weight
Gentlecat joined the channel
ruaok
hi Gentlecat
_Dave_
So I think I have a MB slave server running ... are there unit tests for the entire system I can run?
ruaok
enjoying SF?
Gentlecat
not yet
just arrived there
ruaok
heh, I just arrived in San Jose.
_Dave_
Alternatively, I can program my mediafile importer to do two lookups, one to my slave server and one to your master ones and compare? O.o
ruaok
_Dave_: we don’t have anything to test a full live running setup.
_Dave_
ah ok :)
ruaok
my super quick check is to do a search, then to load any random result.
_Dave_
I did that
ruaok
that tests the web servers, DB and search servers.
if that is ok, the rest has a decent chance of being ok as well
_Dave_
there's a number of complex systems hooked together here
ruaok
you should see the real site. :)
rube goldberg comes to mind. :)
_Dave_
lol
totally familiar with that, I've been an SA since 1980...seen enough of em
ruaok
ianmcorvidae: so, looking at the nagios list, I can’t really tell which of the tests are still complaining. all thes ones that seem related to rsync have been failing for… nearly 2 years. :)
Freso *cough*submitting admitting to be an experienced SA*cough* >_>
Freso
*someone
Submitting... I need to do less ABz stuff, I think.
ruaok
ianmcorvidae: well, I see that all the replication packets are up to date.
I’ll just leave it be for now.
ianmcorvidae
ruaok: well, at present nothing is actively in a warning or critical state, so that seems correct
ruaok
ok. kewl.
ruaok stands down
> rob: Insert new memory
++ > rob: remove old memory
:)
ianmcorvidae
I could just change it to "change memory" :P I'd assumed it was just adding in more sticks to empty slots, I guess
ruaok
no, all slots are filled with 2GB sticks.
so, I need to yank those and replace them with 4GB sticks.
and yes, I’m just being pedantic. :)
Freso
ruaok: Well, your website sucks too.
ruaok knows that
(Actually, it is not too bad once NoScript is told your site isn't likely to burn down my apartment.)
(But it is a bit sucky to require JS. :p)
chirlu`
nikki: I noticed sometimes (but not recently) that JIRA emails were delayed, i.e. I could see updates in the activity stream, but the email notifications came hours later.
CatQuest will note that for a cheap coin he will do any editing someone will tell him to do. so if the bbc wans I can add this information or any such information.