what is this? ruaok gets on and everything breaks?
2005-12-28 36232, 2005
ruaok
ya, sorry. should be better in a bit
2005-12-28 36215, 2005
Trongersoll
heh, it seems to be working now, i was getting "can't find page"s
2005-12-28 36210, 2005
ruaok
I activated a swap file on catbus. that will slow things down overall, but stop the DB from crashing.
2005-12-28 36221, 2005
ruaok
catbus isn't very happy atm. load 18. :-(
2005-12-28 36230, 2005
nikki_
poor catbus :(
2005-12-28 36244, 2005
Trongersoll
maybe it needs some catnip. ;)
2005-12-28 36210, 2005
ruaok
dave and I are going to play with it later. maybe just needs a good ball of yarn.
2005-12-28 36243, 2005
Trongersoll
dave? hmmm don't let him talk to HAL!
2005-12-28 36239, 2005
chocomo
hi Trongersoll
2005-12-28 36234, 2005
djce joined the channel
2005-12-28 36237, 2005
ruaok
dave!
2005-12-28 36242, 2005
ruaok
how are things?
2005-12-28 36200, 2005
ruaok
quick update: we dropped a replication packet and I added a swap file to catbus
2005-12-28 36218, 2005
Trongersoll
hi mo
2005-12-28 36259, 2005
Trongersoll has left the channel
2005-12-28 36248, 2005
ruaok
ok, I'm running off to the office.
2005-12-28 36249, 2005
ruaok
brb
2005-12-28 36250, 2005
ruaok has quit
2005-12-28 36219, 2005
djce
doh. /me starts paying attention.
2005-12-28 36245, 2005
chocomo
lol djce
2005-12-28 36203, 2005
djce
I seem to have a habit of joining a room and then forgetting I did so, only reading the chat 10 minutes later...
2005-12-28 36213, 2005
chocomo
heh
2005-12-28 36245, 2005
ruaok joined the channel
2005-12-28 36254, 2005
ruaok returns
2005-12-28 36259, 2005
ruaok
nikki_: you there?
2005-12-28 36216, 2005
ruaok
nikki_: skipping the missing packet seems to work ok:
2005-12-28 36219, 2005
ruaok
update replication_control set current_replication_sequence = 6537;
2005-12-28 36242, 2005
djce
I wish I understood how to work out where all that memory goes.
2005-12-28 36253, 2005
ruaok
you and me both.
2005-12-28 36203, 2005
ruaok
and I wish PG knew to stay within the boundaries.
2005-12-28 36213, 2005
ruaok
did you see the swapfile I activated?
2005-12-28 36223, 2005
ruaok
the missing replication packet made me very nervous.
2005-12-28 36237, 2005
djce
How big is that swap?
2005-12-28 36243, 2005
ruaok
4GB
2005-12-28 36256, 2005
ruaok
we had the disk space...
2005-12-28 36207, 2005
ruaok
that will make some things slower, but at least pg shouldn't crash anymore.
2005-12-28 36214, 2005
ruaok
in theory, anyways.
2005-12-28 36227, 2005
djce
re. the missing packet... sadly because the DB has been crashing so frequently, all bets are off really.
2005-12-28 36252, 2005
ruaok
I skipped the packet and so far .nl is running along happily.
2005-12-28 36257, 2005
chocomo
in theory there is no difference betwene theory and pracsis, but in pracsis there is
2005-12-28 36203, 2005
ruaok
is there a relatively easy way to create an empty packet?
2005-12-28 36217, 2005
djce
I'm also kind of reluctant to upgrade to pg 8.x - I'd like to understand the memory issues first.
2005-12-28 36219, 2005
ruaok
if so, I'd like to create that and then toss that out there.
2005-12-28 36239, 2005
ruaok
how PG deals with mem in general?
2005-12-28 36255, 2005
ruaok
Russ mentioned that there was some index bloat in pg 7.4
2005-12-28 36257, 2005
djce
Yes, and also general Linux / UNIX memory things.
2005-12-28 36224, 2005
ruaok
so, my guess is that the indexes kept growing and all the funkyness we are seeing are related to PG running out of ram.
2005-12-28 36240, 2005
ruaok
I guess it hasn't been tested much without swap space.
2005-12-28 36258, 2005
djce
hmmmm.
2005-12-28 36202, 2005
ruaok
the one thing I am really worried about is the pysical memory going bad.
2005-12-28 36206, 2005
djce wonders which indexes are biggest
2005-12-28 36215, 2005
djce
you mean dodgy ram?
2005-12-28 36222, 2005
ruaok
but if we take the site down and we can build the new pg, then it should be ok.
2005-12-28 36232, 2005
ruaok
or plain bad ram.
2005-12-28 36240, 2005
ruaok
I didn't inspect the ram when we got the machine.
2005-12-28 36253, 2005
ruaok
one modbot report had an invalid free() call
2005-12-28 36257, 2005
ruaok
that scared me a bit.
2005-12-28 36214, 2005
ruaok
replication in nl caught up.
2005-12-28 36222, 2005
ruaok
at least that is good.
2005-12-28 36239, 2005
tru
if you don't have enough memory linux OOM will show up and kill of things.
2005-12-28 36256, 2005
tru
or poorly written applications will die because malloc() returns null.
2005-12-28 36201, 2005
ruaok nods
2005-12-28 36221, 2005
ruaok
catbus is now 22MB into swap. not bad.
2005-12-28 36230, 2005
tru
phew. moving sucks.
2005-12-28 36237, 2005
ruaok nods
2005-12-28 36255, 2005
tru
ruaok: is advanced relations reflected in the musicbrainz api yet? is it planned?
2005-12-28 36210, 2005
ruaok
djce: in theory, if nothing happens for 24 hours then it was all swap related.
2005-12-28 36225, 2005
ruaok
tru: libmb 2.1.2 supports retrieving them
2005-12-28 36225, 2005
djce
Probably true, given the frequency of recent crashes.
2005-12-28 36242, 2005
ruaok
but I am not keen on waiting that long.
2005-12-28 36247, 2005
tru
ruaok: nice! then I should start play around with that.
2005-12-28 36234, 2005
djce wonders what proportion of the DB is moderation_closed, byte for byte
2005-12-28 36247, 2005
ruaok
another valid thought.
2005-12-28 36258, 2005
ruaok
perhaps its time to prune the table a bit>
2005-12-28 36259, 2005
ruaok
?
2005-12-28 36219, 2005
djce
Ha!
2005-12-28 36221, 2005
djce
:-)
2005-12-28 36236, 2005
chocomo
would it be much trouble to store old moderation data on somewhere else thoguh?
2005-12-28 36246, 2005
ruaok
currently yes.
2005-12-28 36249, 2005
chocomo
hmm
2005-12-28 36253, 2005
ruaok
maybe we can host it on your machine?
2005-12-28 36257, 2005
chocomo
XD
2005-12-28 36259, 2005
ruaok
wipe ME and instal linux?
2005-12-28 36203, 2005
chocomo
how big is it?
2005-12-28 36241, 2005
ruaok
BIG
2005-12-28 36244, 2005
chocomo
:/
2005-12-28 36206, 2005
djce
moderation_closed + indexes == 21.7% of the database.
2005-12-28 36214, 2005
djce
"track" is 13.9%
2005-12-28 36220, 2005
djce
"trackwords" is 13.2%
2005-12-28 36233, 2005
djce
"trmjoin" is 8.8%
2005-12-28 36239, 2005
djce
"trm" is 7.5%
2005-12-28 36247, 2005
chocomo
cant prune ADD_TRM then?
2005-12-28 36248, 2005
ruaok
fairly well spread out overall.
2005-12-28 36221, 2005
chocomo
that is a moderation that realy inst necessary information. and it usualy ibg
2005-12-28 36227, 2005
chocomo
or so I have understood
2005-12-28 36241, 2005
djce
chocomo: some of them are big (the older ones).
2005-12-28 36250, 2005
djce
But the point is, there are a LOT of them.
2005-12-28 36213, 2005
chocomo
hhmhmm
2005-12-28 36215, 2005
djce counts them
2005-12-28 36246, 2005
chocomo bets they stand for more than 50% of all moderation_closed
2005-12-28 36257, 2005
chocomo
unless since they're auto it isnt in the same thing
2005-12-28 36206, 2005
djce
we shall see soon... I have no idea if I think you're right or not.
2005-12-28 36226, 2005
djce
"auto" still gets stored in the DB.
2005-12-28 36232, 2005
chocomo
I know
2005-12-28 36242, 2005
djce
so "auto" makes no difference here.
2005-12-28 36215, 2005
chocomo
but I thoghu since they where auto their 'closedness' wheren't stored in the same place as mods that needed voting
2005-12-28 36240, 2005
chocomo
nikki do yo uhave that unicode hart again?
2005-12-28 36247, 2005
chocomo
nm found it
2005-12-28 36246, 2005
yllona joined the channel
2005-12-28 36225, 2005
ruaok waves at yllona
2005-12-28 36231, 2005
ruaok
those tamales were GREAT!
2005-12-28 36232, 2005
djce
34.7% of "moderation_closed" rows are MOD_ADD_TRM
2005-12-28 36235, 2005
yllona
mornin' ru
2005-12-28 36252, 2005
chocomo
thats quite a lot, no?
2005-12-28 36255, 2005
djce
Next biggest is "edit track name" at 24.9%
2005-12-28 36258, 2005
chocomo
hi ylla
2005-12-28 36206, 2005
yllona
ru: yeah, i'm about to have one for breakfast, then freeze the rest
2005-12-28 36206, 2005
djce
chocomo: Without a doubt.
2005-12-28 36213, 2005
yllona
hello mo
2005-12-28 36229, 2005
chocomo
^^
2005-12-28 36245, 2005
Blippe joined the channel
2005-12-28 36246, 2005
chocomo
djce: would nuking MOD_ADD_TRM help at all?
2005-12-28 36235, 2005
djce
The row size of the MOD_ADD_TRMS rows is also disproportionately large.
2005-12-28 36253, 2005
ruaok
understandably so.
2005-12-28 36200, 2005
ruaok
I think we could stand to trim that.
2005-12-28 36216, 2005
ruaok
like toss all ADD_TRM more than a year old...
2005-12-28 36229, 2005
wolfsong
ruaok: there seems to be a lot of DB entries that can't be tagged because the Tagger button is missing when you do a lookup from Picard
2005-12-28 36208, 2005
chocomo
I can't see what usefullnes it could od either. seeing as the add trm mods don't 'do' anything, versus 'change trackname' can be helpful i ndeterminign what happened with a track
2005-12-28 36220, 2005
ruaok
wolfsong: is that a probelm for just you or are others reporting the same problem?
2005-12-28 36231, 2005
chocomo
I don't get that problem
2005-12-28 36209, 2005
wolfsong
i did ask someone to check the first time i saw it before i reported a bug and they saw the same thing
2005-12-28 36212, 2005
ruaok
wolfsong, if that happens, all you need to do is add a tport=8000 (or whatever your port number is) to the URL and it should show up.
2005-12-28 36236, 2005
ruaok
try that, please.
2005-12-28 36238, 2005
chocomo
ohh nice thing i'll remember that
2005-12-28 36223, 2005
djce
I estimate that MOD_ADD_TRM rows account for 60% of the bytes used in moderation_closed
2005-12-28 36253, 2005
ruaok
wow. good argument for nuking some of those.
2005-12-28 36223, 2005
chocomo
what did I say? over 50% :p
2005-12-28 36201, 2005
djce
ruaok: did you have any thoughts on the matter of not replicating trm* ?
2005-12-28 36223, 2005
wolfsong
ruaok: that didn't work
2005-12-28 36240, 2005
ruaok
djce: gut instinct tells me that I want to continune doing that.
2005-12-28 36213, 2005
ruaok
I think if we add a submit pass-through to the main server from the mirrors, the mirror servers can be used for tagging.
