16:38 PM
ocharles
but your ticket said replicated
2012-05-15 13646, 2012
16:38 PM
ocharles
anyway, it uncovered a big problem
2012-05-15 13600, 2012
16:39 PM
ocharles
which was that running the upgrade on slaves would create some foreign keys and stuff too
2012-05-15 13605, 2012
16:39 PM
ocharles
so i had to rip it all apart
2012-05-15 13616, 2012
16:39 PM
warp
ocharles: printf "Foo\nBar\n" | sed 's/^/echo /' | sh
2012-05-15 13625, 2012
16:39 PM
ruaok
its said "replicated slave"
2012-05-15 13627, 2012
16:39 PM
ocharles
warp: oh, heh
2012-05-15 13628, 2012
16:39 PM
ruaok
not replication triggers.
2012-05-15 13645, 2012
16:39 PM
ruaok
ocharles: yep, that was the problem I wanted you to fix.
2012-05-15 13647, 2012
16:39 PM
ocharles
ah, right
2012-05-15 13653, 2012
16:39 PM
ruaok
was there something else changed?
2012-05-15 13655, 2012
16:39 PM
ocharles
i guess i confused myself, but ended up fixing the right problem :)
2012-05-15 13600, 2012
16:40 PM
ruaok
yes. :)
2012-05-15 13611, 2012
16:40 PM
ruaok
ok, I'm pushing seach indexes right now.
2012-05-15 13614, 2012
16:40 PM
ocharles
we did a full master import with this on hobbes earlier
2012-05-15 13620, 2012
16:40 PM
ruaok
next I will do a slave test to see if it works.
2012-05-15 13633, 2012
16:40 PM
ruaok
adn upgrade.sh ran as RT_MASTER?
2012-05-15 13640, 2012
16:40 PM
ocharles
no, only tested as RT_STANDALONE
2012-05-15 13659, 2012
16:40 PM
ruaok
ok, I will do a slave and if I have time a master.
2012-05-15 13606, 2012
16:41 PM
ruaok
first some fudz
2012-05-15 13618, 2012
16:41 PM
ocharles
but RT_MASTER just adds replication triggers to all that, so I would hope that's in the same area of testing
2012-05-15 13639, 2012
16:42 PM
ruaok
RT_MASTER does other things too. those code paths should be retested too
2012-05-15 13632, 2012
16:43 PM
luks
do you have an estimate on how long does admin/sql/updates/20120314-add-tracknumber.sql take to run?
2012-05-15 13644, 2012
16:43 PM
ruaok
less than 5 minutes.
2012-05-15 13648, 2012
16:43 PM
luks
cool, thanks
2012-05-15 13651, 2012
16:43 PM
ruaok
depends on the speed of your disks.
2012-05-15 13600, 2012
16:44 PM
ruaok
was about 3 minutes on my SSD mac
2012-05-15 13645, 2012
16:44 PM
warp
ocharles, nikki: alright. all my stuff is done again. pushed to schema-change-2012-05-15 .
2012-05-15 13614, 2012
16:45 PM
ocharles
luks: it's just over 5 on hobbes
2012-05-15 13657, 2012
16:49 PM
ruaok
and hobbes only has 7.2k drives
2012-05-15 13611, 2012
16:50 PM
ruaok
5.4k only!
2012-05-15 13609, 2012
17:00 PM
ocharles
it's with great pleasure I announce the latest in our line of Cover Art Archive projects
2012-05-15 13615, 2012
17:00 PM
ocharles
Ladies and gentlemen, introducing...
2012-05-15 13616, 2012
17:00 PM
ocharles
djcaa!
2012-05-15 13619, 2012
17:00 PM
ocharles
2012-05-15 13649, 2012
17:00 PM
ruaok
lol. great naming.
2012-05-15 13608, 2012
17:01 PM
ruaok
have you seen the latest bug on CAA?
2012-05-15 13618, 2012
17:01 PM
ocharles
i don't know, which bug are you thinking of?
2012-05-15 13625, 2012
17:01 PM
ruaok
CAA-22 IRC
2012-05-15 13630, 2012
17:01 PM
ocharles
yes, that's why I wrote this
2012-05-15 13634, 2012
17:01 PM
ocharles
at least for the cleanup part
2012-05-15 13637, 2012
17:01 PM
ruaok
ok
2012-05-15 13645, 2012
17:01 PM
ruaok
do we know why it happens?
2012-05-15 13649, 2012
17:01 PM
ocharles
still need to fix the trigger to not create events that don't need any work
2012-05-15 13651, 2012
17:01 PM
ocharles
yep
2012-05-15 13600, 2012
17:02 PM
ruaok
k
2012-05-15 13656, 2012
17:02 PM
ocharles
but before I run this, it'd be nice to know if the IA have back ups :)
2012-05-15 13628, 2012
17:03 PM
ocharles
I'm running find-empty piped into s3cmd ls for now
2012-05-15 13646, 2012
17:03 PM
hawke_
”
2012-05-15 13651, 2012
17:04 PM
ocharles
that'd be the one :)
2012-05-15 13659, 2012
17:04 PM
ruaok
ocharles, warp: is the schema-change branch updated with all the changes we believe we need for DB changes?
2012-05-15 13607, 2012
17:05 PM
ruaok
I want to run upgrade.sh in my sandboxes
2012-05-15 13608, 2012
17:06 PM
ruaok
question poised to the archive.
2012-05-15 13621, 2012
17:06 PM
ocharles
Let me double check
2012-05-15 13625, 2012
17:06 PM
ocharles
warp: all your stuff pushed?
2012-05-15 13612, 2012
17:08 PM
ruaok
warp: prod
2012-05-15 13618, 2012
17:09 PM
ocharles
schema-changes-2012-05-15 pushed, a850ea3b8cac610
2012-05-15 13658, 2012
17:09 PM
ruaok
I'm ready to go with that branch.
2012-05-15 13600, 2012
17:10 PM
ruaok
warp???
2012-05-15 13605, 2012
17:10 PM
ocharles
pinging him won't help :)
2012-05-15 13639, 2012
17:10 PM
ruaok
can you send him a txt please?
2012-05-15 13659, 2012
17:10 PM
ocharles
+31 is NL right?
2012-05-15 13602, 2012
17:11 PM
ruaok
yeah
2012-05-15 13618, 2012
17:11 PM
ocharles
sent
2012-05-15 13623, 2012
17:11 PM
ruaok
thx
2012-05-15 13646, 2012
17:11 PM
warp
yes
2012-05-15 13653, 2012
17:11 PM
ocharles
:)
2012-05-15 13613, 2012
17:12 PM
warp
you can also ring me on skype
2012-05-15 13619, 2012
17:12 PM
ruaok
ok
2012-05-15 13623, 2012
17:12 PM
ruaok
is all your stuff pushed?
2012-05-15 13629, 2012
17:12 PM
warp
yes.
2012-05-15 13609, 2012
17:13 PM
warp
only a few small fixes comitted today, but they're all merged into the schema-change-2012-05-15 branch and pushed.
2012-05-15 13618, 2012
17:13 PM
ruaok
ok, master Vm created.
2012-05-15 13622, 2012
17:13 PM
ruaok
now duplicating a slave copy
2012-05-15 13605, 2012
17:16 PM
ocharles
ruaok: can you ask the IA what they want from a cleanup?
2012-05-15 13611, 2012
17:16 PM
ocharles
2012-05-15 13634, 2012
17:16 PM
ocharles
hmm, that doesn't help you actually
2012-05-15 13652, 2012
17:16 PM
ocharles
basically, I can remove all files from a bucket but can't remove the bucket. is that enough?
2012-05-15 13621, 2012
17:20 PM
ruaok
if the buckets should be removed, just make a list of buckets to nuke.
2012-05-15 13630, 2012
17:20 PM
ruaok
we can then pass that over and have sam do it.
2012-05-15 13650, 2012
17:20 PM
ruaok
slave test fired off
2012-05-15 13622, 2012
17:22 PM
reosarevok_ joined the channel
2012-05-15 13643, 2012
17:22 PM
ruaok
2012-05-15 13658, 2012
17:22 PM
ruaok
which steps do I need to apply to totoro's config?
2012-05-15 13612, 2012
17:23 PM
ocharles
"Setting Up Archiving On The Master"
2012-05-15 13633, 2012
17:23 PM
ocharles
We use nfs here, but the rsync method looks like it should be good
2012-05-15 13642, 2012
17:24 PM
ruaok
well, but we're not actually going to deploy it just yet.
2012-05-15 13617, 2012
17:25 PM
ocharles
right, but I don't know if you can turn it on without a restart
2012-05-15 13619, 2012
17:25 PM
ocharles
one moment
2012-05-15 13629, 2012
17:25 PM
ruaok
fine with a restart.
2012-05-15 13636, 2012
17:25 PM
ruaok
since we have down time anyway.
2012-05-15 13644, 2012
17:25 PM
ruaok
its just that the slave isn't there yet
2012-05-15 13645, 2012
17:25 PM
ocharles
i mean when we decide to turn this on
2012-05-15 13600, 2012
17:26 PM
ocharles
I think we might need to start shipping logs to the slave, and then just delete them on the slave
2012-05-15 13601, 2012
17:26 PM
ruaok
ah
2012-05-15 13615, 2012
17:26 PM
ruaok
the slave isn't setup at all.
2012-05-15 13620, 2012
17:26 PM
ruaok
just a blank machine with a blank install.
2012-05-15 13622, 2012
17:26 PM
ocharles
ok
2012-05-15 13628, 2012
17:26 PM
ocharles
turns out you can change archive_command without a restart
2012-05-15 13641, 2012
17:26 PM
ruaok
can you have a null archive_command?
2012-05-15 13628, 2012
17:27 PM
hawke_ joined the channel
2012-05-15 13632, 2012
17:27 PM
ocharles
'Setting archive_command to a command that does nothing but return true, e.g. /bin/true (REM on Windows), effectively disables archiving, but also breaks the chain of WAL files needed for archive recovery, so it should only be used in unusual circumstances.'
2012-05-15 13651, 2012
17:27 PM
ruaok
so, no?
2012-05-15 13657, 2012
17:27 PM
ocharles
so yes :)
2012-05-15 13659, 2012
17:27 PM
ruaok
my feeling is that we should just skip it for now
2012-05-15 13628, 2012
17:28 PM
ocharles
wal_level = hot_standby; archive_mode = on; archive_command = /bin/true
2012-05-15 13639, 2012
17:28 PM
ianmcorvidae
breaking the write-ahead log seems like a bad idea
2012-05-15 13600, 2012
17:29 PM
ocharles
breaks the chain
2012-05-15 13604, 2012
17:29 PM
ocharles
that's not a problem if there's no chain to break
2012-05-15 13630, 2012
17:29 PM
ruaok
tbh, I think I would be more comfortable with not doing this now.
2012-05-15 13645, 2012
17:29 PM
ruaok
doing a single DB restart means 30 seconds of downtime. I can deal with that in the future.
2012-05-15 13631, 2012
17:31 PM
ocharles
it's literally what I pasted above. a restart is not just 30 seconds of downtime, it will clear all your indexes too
2012-05-15 13641, 2012
17:31 PM
ocharles
but it's your call
2012-05-15 13646, 2012
17:31 PM
ruaok
clear indexes?
2012-05-15 13648, 2012
17:31 PM
ocharles
i can appreciate limiting the amount of work we do
2012-05-15 13650, 2012
17:31 PM
ocharles
from ram
2012-05-15 13657, 2012
17:31 PM
ruaok
yeah, thats not a big deal.
2012-05-15 13609, 2012
17:32 PM
ruaok
lets focus on the giant pile of stuff we have to get right.
2012-05-15 13612, 2012
17:32 PM
ruaok
and not make it bigger.
2012-05-15 13623, 2012
17:32 PM
ruaok
slave test passed.
2012-05-15 13616, 2012
17:33 PM
ruaok
search indexes are in place.
2012-05-15 13622, 2012
17:35 PM
ocharles
I'm going to run to the shop to pick up something for dinner
2012-05-15 13634, 2012
17:35 PM
ocharles
won't be more than a few minutes
2012-05-15 13640, 2012
17:35 PM
ruaok
k
2012-05-15 13656, 2012
17:35 PM
ruaok
have all the tests passed?
2012-05-15 13612, 2012
17:36 PM
ocharles
yea
2012-05-15 13627, 2012
17:36 PM
ocharles
shall I merge to master and get jenkins to see what it thinks?
2012-05-15 13633, 2012
17:36 PM
ruaok
plz
2012-05-15 13644, 2012
17:36 PM
ocharles
329 files changed, 14685 insertions(+), 1788 deletions(-)
2012-05-15 13646, 2012
17:36 PM
ruaok
and I'll do another slave test after that.
2012-05-15 13657, 2012
17:36 PM
warp
yay, merging to master.
2012-05-15 13611, 2012
17:37 PM
ocharles
is hobbes busy?
2012-05-15 13622, 2012
17:37 PM
ocharles
2012-05-15 13623, 2012
17:37 PM
ruaok
I'm not using hobbes
2012-05-15 13634, 2012
17:39 PM
ocharles
ok, build kicked off
2012-05-15 13649, 2012
17:40 PM
ruaok duplices a vm
2012-05-15 13625, 2012
17:48 PM
the_metalgamer joined the channel
2012-05-15 13654, 2012
17:48 PM
MBJenkins
2012-05-15 13614, 2012
17:49 PM
ruaok
crap
2012-05-15 13641, 2012
17:53 PM
warp
meh
2012-05-15 13651, 2012
17:53 PM
ocharles
right, times to be fixings
2012-05-15 13633, 2012
17:55 PM
ocharles
odd, the merge seems to have reverted a change somewhere...
2012-05-15 13650, 2012
17:55 PM
ocharles
which is a bit unsettling
2012-05-15 13656, 2012
17:57 PM
ocharles
ok, i think that's me just reading things wrong
2012-05-15 13611, 2012
18:01 PM
ruaok
release time is here!
2012-05-15 13613, 2012
18:01 PM
ruaok
:-(
2012-05-15 13623, 2012
18:01 PM
MiX-MaN joined the channel
2012-05-15 13601, 2012
18:02 PM
warp looks into the failing seleniums.
2012-05-15 13611, 2012
18:03 PM
ocharles
isn't that due to the other failures?+
2012-05-15 13616, 2012
18:04 PM
warp
quite possibly.
