11:47 AM
ruaok
FWIW, I turned off the recording index generation.
2015-03-12 07152, 2015
11:47 AM
ruaok
I may need to go install mbslave on cartman. :(
2015-03-12 07107, 2015
11:51 AM
ruaok
I wonder if the flaky carl kept our traffic artificially low.
2015-03-12 07116, 2015
11:51 AM
ruaok
and now it works, everything else is under more stress.
2015-03-12 07132, 2015
11:51 AM
ianmcorvidae
possible, I guess. not sure how we'd test the hypothesis
2015-03-12 07127, 2015
11:52 AM
ruaok
first step: get log analysis running again.
2015-03-12 07124, 2015
12:07 PM
ruaok
well, proof positive its not the search indexeing that is causing problem.
2015-03-12 07129, 2015
12:07 PM
ruaok
totoro is pissed off again.
2015-03-12 07148, 2015
12:07 PM
ujjwal joined the channel
2015-03-12 07115, 2015
12:24 PM
leyyin joined the channel
2015-03-12 07127, 2015
12:31 PM
ruaok
ianmcorvidae: still up?
2015-03-12 07118, 2015
12:59 PM
ruaok
right then. we may have a bit of a problem.
2015-03-12 07130, 2015
12:59 PM
ruaok
Our DB server has 48GB of ram. Our postgres config:
2015-03-12 07131, 2015
12:59 PM
ruaok
shared_buffers = 8192MB
2015-03-12 07140, 2015
12:59 PM
ruaok
anyone who is not alastairp see a problem?
2015-03-12 07150, 2015
12:59 PM
alastairp
ooh, me, me
2015-03-12 07156, 2015
12:59 PM
Freso
Just swap it!
2015-03-12 07158, 2015
12:59 PM
ruaok smacks alastairp
2015-03-12 07158, 2015
12:59 PM
alastairp
i know sir
2015-03-12 07116, 2015
13:00 PM
Freso
Oh, 48.
2015-03-12 07135, 2015
13:00 PM
Freso read as 8 first :|
2015-03-12 07150, 2015
13:01 PM
ruaok
ok, configuration updated.
2015-03-12 07111, 2015
13:02 PM
ruaok
does pg 9.1 need a full restart for this to take?
2015-03-12 07122, 2015
13:08 PM
ruaok
our kernel.shmmax allows 27474836480 bytes.
2015-03-12 07103, 2015
13:09 PM
ruaok
the docs don't state that we need a restart.
2015-03-12 07107, 2015
13:09 PM
ruaok
I'll try hupping.
2015-03-12 07106, 2015
13:32 PM
ruaok
ok, time for some lunch.
2015-03-12 07128, 2015
13:32 PM
leyyin joined the channel
2015-03-12 07115, 2015
13:33 PM
chirlu` joined the channel
2015-03-12 07125, 2015
13:34 PM
ohrstrom joined the channel
2015-03-12 07118, 2015
13:52 PM
ruaok
oh looky, load 5 now. :)
2015-03-12 07143, 2015
13:54 PM
leyyin joined the channel
2015-03-12 07106, 2015
14:00 PM
chirlu`
2015-03-12 07128, 2015
14:00 PM
chirlu`
(I’m not sure how large the whole DB currently is.)
2015-03-12 07110, 2015
14:01 PM
ruaok
I went with the docs that suggested 1/4 - 1/3 ram.
2015-03-12 07117, 2015
14:01 PM
ruaok
so, I went with 14GB.
2015-03-12 07121, 2015
14:01 PM
ruaok
of 48GB
2015-03-12 07135, 2015
14:01 PM
chirlu`
It also seems effective_cache_size should be set to something like 40 GB or so.
2015-03-12 07129, 2015
14:04 PM
ruaok
currently 20GB
2015-03-12 07138, 2015
14:04 PM
ruaok
does that setting need a full restart or reload ok?
2015-03-12 07148, 2015
14:05 PM
chirlu`
2015-03-12 07121, 2015
14:06 PM
ruaok
ok, coming right up.
2015-03-12 07151, 2015
14:06 PM
ruaok
server signaled
2015-03-12 07101, 2015
14:07 PM
chirlu`
“… if you have more than 1 GB …” :)
2015-03-12 07107, 2015
14:07 PM
ruaok
heh
2015-03-12 07137, 2015
14:07 PM
ruaok
index builder restarted.
2015-03-12 07105, 2015
14:09 PM
ruaok
that seems to have dropped the io load.
2015-03-12 07115, 2015
14:10 PM
ruaok
and now the load is spiking.
2015-03-12 07119, 2015
14:10 PM
ruaok
surely because of the search indexes.
2015-03-12 07147, 2015
14:21 PM
dufferzafar joined the channel
2015-03-12 07131, 2015
14:24 PM
ruaok
fuck. back into swap.
2015-03-12 07128, 2015
14:25 PM
chirlu`
Hm.
2015-03-12 07102, 2015
14:26 PM
ruaok
search indexer killed
2015-03-12 07102, 2015
14:27 PM
chirlu`
Has anything changed between yesterday and today?
2015-03-12 07108, 2015
14:27 PM
ruaok
and the load is drastically dropping.
2015-03-12 07120, 2015
14:27 PM
ruaok
not that I am aware of.
2015-03-12 07117, 2015
14:29 PM
chirlu`
According to the graph, there were some swap-outs, slowly, over long periods, but never before a quick increase in swap usage.
2015-03-12 07109, 2015
14:42 PM
ohrstrom joined the channel
2015-03-12 07105, 2015
14:54 PM
ohrstrom joined the channel
2015-03-12 07115, 2015
15:05 PM
ujjwal joined the channel
2015-03-12 07107, 2015
15:06 PM
kepstin-laptop joined the channel
2015-03-12 07144, 2015
15:10 PM
ruaok
ok, wasn't seach indexes.
2015-03-12 07152, 2015
15:10 PM
ruaok
load spiked again
2015-03-12 07109, 2015
15:13 PM
JesseW joined the channel
2015-03-12 07122, 2015
15:21 PM
chirlu`
Swapping again.
2015-03-12 07105, 2015
15:23 PM
ruaok
yep.
2015-03-12 07123, 2015
15:23 PM
chirlu`
The only DB-intensive thing that has changed (that I see) is sitemaps, but that should run only during UTC night.
2015-03-12 07147, 2015
15:23 PM
ruaok
and it would presumably only be one process. not the whole DB server.
2015-03-12 07120, 2015
15:25 PM
ruaok
the last server update was 3 days ago.
2015-03-12 07129, 2015
15:25 PM
ruaok
that would not explain why it started happeing some 12 hours ago.
2015-03-12 07141, 2015
15:25 PM
ruaok
we've ruled out the search indexer.
2015-03-12 07158, 2015
15:25 PM
ruaok
I believe that we have enough indexes.
2015-03-12 07123, 2015
15:26 PM
ruaok
2015-03-12 07152, 2015
15:26 PM
ruaok
just to rule things out, I've build indexes for the redirect tables and edit_url (edit).
2015-03-12 07154, 2015
15:26 PM
nikki
sitemaps stuff got included in a hotfix
2015-03-12 07101, 2015
15:27 PM
ruaok
no significant changes.
2015-03-12 07109, 2015
15:27 PM
chirlu`
A new beta was rolled out around that time.
2015-03-12 07119, 2015
15:27 PM
chirlu`
But I see no heavy changes, as I said.
2015-03-12 07129, 2015
15:27 PM
ruaok
should I turn off beta for a bit, just to check?
2015-03-12 07112, 2015
15:28 PM
ruaok
and now the load magically dropped again.
2015-03-12 07114, 2015
15:28 PM
ruaok
back to 8
2015-03-12 07153, 2015
15:28 PM
D4RK-PH0ENiX joined the channel
2015-03-12 07116, 2015
15:29 PM
nikki
2015-03-12 07131, 2015
15:36 PM
chirlu`
Regarding the indexes, these should already have had some.
2015-03-12 07139, 2015
15:36 PM
ruaok
they didn't.
2015-03-12 07148, 2015
15:37 PM
chirlu`
But for edit_url, there is CREATE INDEX edit_url_idx ON edit_url (url) in CreateIndexes.sql
2015-03-12 07107, 2015
15:38 PM
chirlu`
and (edit, url) is the PK, so that has an index automatically.#
2015-03-12 07110, 2015
15:38 PM
chirlu`
-#
2015-03-12 07138, 2015
15:38 PM
chirlu`
The redirect tables, it’s possible that there is/was none for the other direction.
2015-03-12 07147, 2015
15:38 PM
ruaok
ah yes.
2015-03-12 07153, 2015
15:38 PM
ruaok
the edit_url had two indexes.
2015-03-12 07102, 2015
15:39 PM
ruaok
I added one soley on edit, but that also made no diff.
2015-03-12 07147, 2015
15:39 PM
chirlu`
(edit, url) can be used for lookups on (edit), too.
2015-03-12 07122, 2015
15:41 PM
chirlu`
Though there seems to be/have been no index for the *_redirects on new_id, indeed.
2015-03-12 07138, 2015
15:41 PM
chirlu`
Was forgotten when load_gid_redirects was added, apparently.
2015-03-12 07104, 2015
15:43 PM
chirlu`
But this went live in v-2015-01-12 already.
2015-03-12 07157, 2015
15:50 PM
ohrstrom joined the channel
2015-03-12 07154, 2015
16:14 PM
ruaok_ joined the channel
2015-03-12 07136, 2015
16:24 PM
ruaok
ok, here is the next load spike.
2015-03-12 07131, 2015
16:27 PM
ijabz2 joined the channel
2015-03-12 07115, 2015
17:10 PM
utsav_ joined the channel
2015-03-12 07133, 2015
17:10 PM
utsav_
Hi !
2015-03-12 07138, 2015
17:10 PM
utsav_
I am a computer science undergrad from India looking to participate in GSoC 2015. I wish to know more about AcousticBrainz Project. Any info would be of great help
2015-03-12 07102, 2015
17:11 PM
alastairp
what do you want to know about?
2015-03-12 07141, 2015
17:11 PM
utsav_
I wish to know about the AcousticBrainz project and what are the organizations expectations with regards to the project ideas about it ..
2015-03-12 07131, 2015
17:16 PM
alastairp
2015-03-12 07151, 2015
17:17 PM
alastairp
it would be great if you could let us know some things that you think might be useful, based on your understanding of the project
2015-03-12 07135, 2015
17:37 PM
Leo_Verto
how do people find this channel but not the "about" page of AB?
2015-03-12 07119, 2015
17:38 PM
alastairp
to be fair, the proposal stub on the ideas page was very brief, and didn’t explain our specific problem very well
2015-03-12 07135, 2015
17:38 PM
alastairp
I’ve edited it a bit to add some more information
2015-03-12 07126, 2015
17:42 PM
ruaok joined the channel
2015-03-12 07135, 2015
17:42 PM
lokeshw24 joined the channel
2015-03-12 07158, 2015
17:57 PM
D4RK-PH0_ joined the channel
2015-03-12 07154, 2015
18:00 PM
kepstin-laptop joined the channel
2015-03-12 07141, 2015
18:13 PM
Gentlecat joined the channel
2015-03-12 07123, 2015
18:25 PM
rvedotrc joined the channel
2015-03-12 07142, 2015
18:25 PM
yeeeargh joined the channel
2015-03-12 07151, 2015
18:26 PM
ruaok joined the channel
2015-03-12 07130, 2015
18:29 PM
Gentlecat
ruaok: there are several pull requests for CB that need to be reviewed
2015-03-12 07154, 2015
18:29 PM
ruaok
I know. more server drama keeping me busy. sorry. :(
2015-03-12 07115, 2015
18:33 PM
Powlow joined the channel
2015-03-12 07147, 2015
18:43 PM
aron_kexp joined the channel
2015-03-12 07128, 2015
18:46 PM
lokeshw24 joined the channel
2015-03-12 07143, 2015
19:10 PM
ijabz2 joined the channel
2015-03-12 07143, 2015
19:13 PM
lokeshw24 has left the channel
2015-03-12 07115, 2015
19:20 PM
diana_olhovik joined the channel
2015-03-12 07146, 2015
19:33 PM
ijabz2 joined the channel
2015-03-12 07154, 2015
19:58 PM
MBJenkins
2015-03-12 07155, 2015
19:58 PM
MBJenkins
* priyankbhatt1234: Added title attriubute in anchor tag stating View on AcousticBrainz
2015-03-12 07105, 2015
20:01 PM
ruaok
the letter of the law has been satisfied, yes. the spirit remains 404
2015-03-12 07107, 2015
20:05 PM
ruaok
2015-03-12 07121, 2015
20:05 PM
ruaok
looks more like what I normally do to LEDs
2015-03-12 07134, 2015
20:09 PM
ruaok
2015-03-12 07151, 2015
20:09 PM
ruaok
anyone else worried/amazed/puzzled by this graph?
2015-03-12 07106, 2015
20:10 PM
ruaok
only totoro has spikes. everything else is purring and happy. (for once)
2015-03-12 07158, 2015
20:28 PM
chirlu` joined the channel
2015-03-12 07116, 2015
20:30 PM
chirlu`
2015-03-12 07135, 2015
20:30 PM
chirlu`
Now, this might be caused by people having trouble reaching
mb.org , hence searching less.
2015-03-12 07158, 2015
20:30 PM
chirlu`
But search-public also has some lows.
2015-03-12 07158, 2015
20:32 PM
ruaok
yeah, this looks like a knock on effect to me.
2015-03-12 07108, 2015
20:33 PM
ruaok
the overall shape of the curve remains the same.
2015-03-12 07144, 2015
20:33 PM
ruaok
at least the graph was pretty for 4 days, no?
2015-03-12 07145, 2015
20:34 PM
chirlu`
:)
2015-03-12 07141, 2015
20:35 PM
chirlu`
Sounds like a tomb inscription … “it worked well for four days”.
2015-03-12 07130, 2015
20:37 PM
ruaok
sounds better than what I expect to have on my tombstone: “Dumbass”
2015-03-12 07140, 2015
20:41 PM
Freso
I like Pratchett's post-mortem tweets.
2015-03-12 07125, 2015
20:42 PM
ruaok runs off to celebrate the opening of some cafe or something
2015-03-12 07104, 2015
20:43 PM
chirlu` has left the channel
2015-03-12 07122, 2015
20:44 PM
ruaok
huh, look the search server is wedged.
2015-03-12 07143, 2015
20:45 PM
ruaok
interesting that.
