16:38 PM
ocharles
but your ticket said replicated
16:38 PM
anyway, it uncovered a big problem
16:39 PM
which was that running the upgrade on slaves would create some foreign keys and stuff too
16:39 PM
so i had to rip it all apart
16:39 PM
warp
ocharles: printf "Foo\nBar\n" | sed 's/^/echo /' | sh
16:39 PM
ruaok
its said "replicated slave"
16:39 PM
ocharles
warp: oh, heh
16:39 PM
ruaok
not replication triggers.
16:39 PM
ocharles: yep, that was the problem I wanted you to fix.
16:39 PM
ocharles
ah, right
16:39 PM
ruaok
was there something else changed?
16:39 PM
ocharles
i guess i confused myself, but ended up fixing the right problem :)
16:40 PM
ruaok
yes. :)
16:40 PM
ok, I'm pushing seach indexes right now.
16:40 PM
ocharles
we did a full master import with this on hobbes earlier
16:40 PM
ruaok
next I will do a slave test to see if it works.
16:40 PM
adn upgrade.sh ran as RT_MASTER?
16:40 PM
ocharles
no, only tested as RT_STANDALONE
16:40 PM
ruaok
ok, I will do a slave and if I have time a master.
16:41 PM
first some fudz
16:41 PM
ocharles
but RT_MASTER just adds replication triggers to all that, so I would hope that's in the same area of testing
16:42 PM
ruaok
RT_MASTER does other things too. those code paths should be retested too
16:43 PM
luks
do you have an estimate on how long does admin/sql/updates/20120314-add-tracknumber.sql take to run?
16:43 PM
ruaok
less than 5 minutes.
16:43 PM
luks
cool, thanks
16:43 PM
ruaok
depends on the speed of your disks.
16:44 PM
was about 3 minutes on my SSD mac
16:44 PM
warp
ocharles, nikki: alright. all my stuff is done again. pushed to schema-change-2012-05-15 .
16:45 PM
ocharles
luks: it's just over 5 on hobbes
16:49 PM
ruaok
and hobbes only has 7.2k drives
16:50 PM
5.4k only!
17:00 PM
ocharles
it's with great pleasure I announce the latest in our line of Cover Art Archive projects
17:00 PM
Ladies and gentlemen, introducing...
17:00 PM
djcaa!
17:00 PM
17:00 PM
ruaok
lol. great naming.
17:01 PM
have you seen the latest bug on CAA?
17:01 PM
ocharles
i don't know, which bug are you thinking of?
17:01 PM
ruaok
CAA-22 IRC
17:01 PM
ocharles
yes, that's why I wrote this
17:01 PM
at least for the cleanup part
17:01 PM
ruaok
ok
17:01 PM
do we know why it happens?
17:01 PM
ocharles
still need to fix the trigger to not create events that don't need any work
17:01 PM
yep
17:02 PM
ruaok
k
17:02 PM
ocharles
but before I run this, it'd be nice to know if the IA have back ups :)
17:03 PM
I'm running find-empty piped into s3cmd ls for now
17:03 PM
hawke_
”
17:04 PM
ocharles
that'd be the one :)
17:04 PM
ruaok
ocharles, warp: is the schema-change branch updated with all the changes we believe we need for DB changes?
17:05 PM
I want to run upgrade.sh in my sandboxes
17:06 PM
question poised to the archive.
17:06 PM
ocharles
Let me double check
17:06 PM
warp: all your stuff pushed?
17:08 PM
ruaok
warp: prod
17:09 PM
ocharles
schema-changes-2012-05-15 pushed, a850ea3b8cac610
17:09 PM
ruaok
I'm ready to go with that branch.
17:10 PM
warp???
17:10 PM
ocharles
pinging him won't help :)
17:10 PM
ruaok
can you send him a txt please?
17:10 PM
ocharles
+31 is NL right?
17:11 PM
ruaok
yeah
17:11 PM
ocharles
sent
17:11 PM
ruaok
thx
17:11 PM
warp
yes
17:11 PM
ocharles
:)
17:12 PM
warp
you can also ring me on skype
17:12 PM
ruaok
ok
17:12 PM
is all your stuff pushed?
17:12 PM
warp
yes.
17:13 PM
only a few small fixes comitted today, but they're all merged into the schema-change-2012-05-15 branch and pushed.
17:13 PM
ruaok
ok, master Vm created.
17:13 PM
now duplicating a slave copy
17:16 PM
ocharles
ruaok: can you ask the IA what they want from a cleanup?
17:16 PM
17:16 PM
hmm, that doesn't help you actually
17:16 PM
basically, I can remove all files from a bucket but can't remove the bucket. is that enough?
17:20 PM
ruaok
if the buckets should be removed, just make a list of buckets to nuke.
17:20 PM
we can then pass that over and have sam do it.
17:20 PM
slave test fired off
17:22 PM
reosarevok_ joined the channel
17:22 PM
17:22 PM
which steps do I need to apply to totoro's config?
17:23 PM
ocharles
"Setting Up Archiving On The Master"
17:23 PM
We use nfs here, but the rsync method looks like it should be good
17:24 PM
ruaok
well, but we're not actually going to deploy it just yet.
17:25 PM
ocharles
right, but I don't know if you can turn it on without a restart
17:25 PM
one moment
17:25 PM
ruaok
fine with a restart.
17:25 PM
since we have down time anyway.
17:25 PM
its just that the slave isn't there yet
17:25 PM
ocharles
i mean when we decide to turn this on
17:26 PM
I think we might need to start shipping logs to the slave, and then just delete them on the slave
17:26 PM
ruaok
ah
17:26 PM
the slave isn't setup at all.
17:26 PM
just a blank machine with a blank install.
17:26 PM
ocharles
ok
17:26 PM
turns out you can change archive_command without a restart
17:26 PM
ruaok
can you have a null archive_command?
17:27 PM
hawke_ joined the channel
17:27 PM
ocharles
'Setting archive_command to a command that does nothing but return true, e.g. /bin/true (REM on Windows), effectively disables archiving, but also breaks the chain of WAL files needed for archive recovery, so it should only be used in unusual circumstances.'
17:27 PM
ruaok
so, no?
17:27 PM
ocharles
so yes :)
17:27 PM
ruaok
my feeling is that we should just skip it for now
17:28 PM
ocharles
wal_level = hot_standby; archive_mode = on; archive_command = /bin/true
17:28 PM
ianmcorvidae
breaking the write-ahead log seems like a bad idea
17:29 PM
ocharles
breaks the chain
17:29 PM
that's not a problem if there's no chain to break
17:29 PM
ruaok
tbh, I think I would be more comfortable with not doing this now.
17:29 PM
doing a single DB restart means 30 seconds of downtime. I can deal with that in the future.
17:31 PM
ocharles
it's literally what I pasted above. a restart is not just 30 seconds of downtime, it will clear all your indexes too
17:31 PM
but it's your call
17:31 PM
ruaok
clear indexes?
17:31 PM
ocharles
i can appreciate limiting the amount of work we do
17:31 PM
from ram
17:31 PM
ruaok
yeah, thats not a big deal.
17:32 PM
lets focus on the giant pile of stuff we have to get right.
17:32 PM
and not make it bigger.
17:32 PM
slave test passed.
17:33 PM
search indexes are in place.
17:35 PM
ocharles
I'm going to run to the shop to pick up something for dinner
17:35 PM
won't be more than a few minutes
17:35 PM
ruaok
k
17:35 PM
have all the tests passed?
17:36 PM
ocharles
yea
17:36 PM
shall I merge to master and get jenkins to see what it thinks?
17:36 PM
ruaok
plz
17:36 PM
ocharles
329 files changed, 14685 insertions(+), 1788 deletions(-)
17:36 PM
ruaok
and I'll do another slave test after that.
17:36 PM
warp
yay, merging to master.
17:37 PM
ocharles
is hobbes busy?
17:37 PM
17:37 PM
ruaok
I'm not using hobbes
17:39 PM
ocharles
ok, build kicked off
17:40 PM
ruaok duplices a vm
17:48 PM
the_metalgamer joined the channel
17:48 PM
MBJenkins
17:49 PM
ruaok
crap
17:53 PM
warp
meh
17:53 PM
ocharles
right, times to be fixings
17:55 PM
odd, the merge seems to have reverted a change somewhere...
17:55 PM
which is a bit unsettling
17:57 PM
ok, i think that's me just reading things wrong
18:01 PM
ruaok
release time is here!
18:01 PM
:-(
18:01 PM
MiX-MaN joined the channel
18:02 PM
warp looks into the failing seleniums.
18:03 PM
ocharles
isn't that due to the other failures?+
18:04 PM
warp
quite possibly.