11:12 AM
ruaok
so, why is that number above 40?
2015-04-24 11427, 2015
11:12 AM
chirlu`
Oh, it should be 40 … I though 50.
2015-04-24 11430, 2015
11:12 AM
chirlu`
+t
2015-04-24 11447, 2015
11:12 AM
ruaok
I tweaked that a little while ago.
2015-04-24 11418, 2015
11:13 AM
chirlu`
At least there is only one pool.
2015-04-24 11427, 2015
11:13 AM
ruaok
Ah!
2015-04-24 11438, 2015
11:13 AM
ruaok
sv and cl_active are now 41
2015-04-24 11454, 2015
11:13 AM
ruaok
I wonder if they had to die of old age before they were not replenished.
2015-04-24 11431, 2015
11:14 AM
chirlu`
SHOW DATABASES; should show what pgbouncer thinks the pool size is supposed to be.
2015-04-24 11449, 2015
11:14 AM
ruaok
2015-04-24 11453, 2015
11:14 AM
ruaok
sorry for the wall of text.
2015-04-24 11414, 2015
11:15 AM
ruaok
but that is me running the show pools command repeatedly. it appears that it is still working with 50 pool size.
2015-04-24 11435, 2015
11:15 AM
ruaok
name | host | port | database | force_user | pool_size | reserve_pool
2015-04-24 11437, 2015
11:15 AM
ruaok
musicbrainz_db_20110516 | 127.0.0.1 | 5432 | musicbrainz_db_20110516 | | 90 | 10
2015-04-24 11439, 2015
11:15 AM
ruaok
huh, that would do it. :)
2015-04-24 11459, 2015
11:15 AM
ruaok
I wonder if pool_size requires a restart of pgbouncer.
2015-04-24 11415, 2015
11:16 AM
chirlu`
SHOW CONFIG; should tell us that.
2015-04-24 11436, 2015
11:16 AM
chirlu`
There is a “changeable” column.
2015-04-24 11446, 2015
11:16 AM
ruaok
default_pool_size | 90 | yes
2015-04-24 11449, 2015
11:16 AM
ruaok
odd.
2015-04-24 11415, 2015
11:17 AM
chirlu`
Well, at least it somewhat explains why there are more than 50 connections …
2015-04-24 11421, 2015
11:17 AM
chirlu`
(Or 40, now.)
2015-04-24 11422, 2015
11:17 AM
ruaok
ah!
2015-04-24 11434, 2015
11:17 AM
ruaok
and the other config settings have not taken at all.
2015-04-24 11438, 2015
11:17 AM
ruaok
server lifetime is far too great.
2015-04-24 11458, 2015
11:17 AM
ruaok
so, that speaks for restarting pgbouncer.
2015-04-24 11410, 2015
11:18 AM
chirlu`
Try RELOAD;
2015-04-24 11413, 2015
11:18 AM
ruaok
which I trust requires some downtime.
2015-04-24 11425, 2015
11:18 AM
ruaok
no change.
2015-04-24 11440, 2015
11:18 AM
ruaok
hmmm.
2015-04-24 11401, 2015
11:19 AM
chirlu`
Is it looking in an unexpected place for a configuation file, perhaps?
2015-04-24 11414, 2015
11:19 AM
ruaok
ding!
2015-04-24 11433, 2015
11:19 AM
ruaok
I believe the chef config deploys to the wrong location.
2015-04-24 11448, 2015
11:19 AM
chirlu`
You could also try “SET default_pool_size = 40;” directly.
2015-04-24 11412, 2015
11:20 AM
ruaok
well, lets go with 50, since that is based in a good thought.
2015-04-24 11448, 2015
11:21 AM
ruaok
much better.
2015-04-24 11402, 2015
11:22 AM
ruaok
cl_waiting occasionally has a 2 or 3 in it.
2015-04-24 11422, 2015
11:23 AM
chirlu`
That sounds plausible.
2015-04-24 11435, 2015
11:23 AM
ruaok
hmmm
2015-04-24 11442, 2015
11:23 AM
ruaok
pgbouncer log still show: ogin failed: FATAL: remaining connection slots are reserved for non-replication superuser connections
2015-04-24 11402, 2015
11:25 AM
chirlu`
Could you SHOW DATABASES; again?
2015-04-24 11426, 2015
11:25 AM
ruaok
hasn't changed.
2015-04-24 11431, 2015
11:25 AM
ruaok
odd.
2015-04-24 11438, 2015
11:25 AM
ruaok
oh. do I need to reload again?
2015-04-24 11442, 2015
11:25 AM
chirlu`
So pool_size=90, reserve_pool=10?
2015-04-24 11458, 2015
11:25 AM
chirlu`
Possibly, depends on what you changed where. :)
2015-04-24 11427, 2015
11:26 AM
ruaok
oh. reload, reloads from file and just nuked all of my changes
2015-04-24 11452, 2015
11:26 AM
ruaok
let me get the actual file updated and then do a reload. hang on.
2015-04-24 11443, 2015
11:29 AM
ruaok
ok, now the right settings are loaded.
2015-04-24 11405, 2015
11:31 AM
ruaok
cl_waiting now gets more connections piled up.
2015-04-24 11410, 2015
11:31 AM
ruaok
no more pgbouncer login messages.
2015-04-24 11437, 2015
11:31 AM
ruaok
wow. nearly 10GB ram free.
2015-04-24 11449, 2015
11:31 AM
ruaok
next time we can probably to go to 75 max connections.
2015-04-24 11455, 2015
11:32 AM
ruaok
I smell a properly tuned db server. :) :) :)
2015-04-24 11452, 2015
11:34 AM
chirlu`
2015-04-24 11415, 2015
11:35 AM
ruaok
I'm going to commit these change and make sure they get deployed to the right place.
2015-04-24 11425, 2015
11:35 AM
ruaok
and I'll also commit a change for max_conn to be 75
2015-04-24 11447, 2015
11:41 AM
ruaok
pgbouncer log is quiet. \ø/
2015-04-24 11455, 2015
11:41 AM
ruaok
now to see if the search indexers are happier.
2015-04-24 11425, 2015
11:43 AM
reosarevok joined the channel
2015-04-24 11407, 2015
11:47 AM
ManiacTwister joined the channel
2015-04-24 11439, 2015
11:48 AM
UmkaDK_ joined the channel
2015-04-24 11441, 2015
11:50 AM
ruaok
woo recording indexes are building.
2015-04-24 11446, 2015
11:54 AM
chirlu`
:-)
2015-04-24 11409, 2015
12:12 PM
ruaok
things are chugging along nicely now. :)
2015-04-24 11449, 2015
12:17 PM
Junior_ joined the channel
2015-04-24 11449, 2015
12:44 PM
Nyanko-sensei joined the channel
2015-04-24 11419, 2015
13:13 PM
Leo_Verto
:D
2015-04-24 11457, 2015
13:47 PM
kepstin-laptop joined the channel
2015-04-24 11454, 2015
13:56 PM
ocharles_
ruaok: this sounds interesting! have the changes been committed anywhere?
2015-04-24 11412, 2015
13:57 PM
ocharles_
i imagine I wrote that initial pgbouncer config, but as we didn't know how to really configure it I probably mostly plucked numbers out of the air
2015-04-24 11417, 2015
13:57 PM
ocharles_
curious what optimal settings look like
2015-04-24 11430, 2015
13:58 PM
ruaok
then I suspect that you're the one who got the install dir wrong too. :)
2015-04-24 11445, 2015
13:58 PM
ruaok
which basically meant that our changes weren't actually being deployed.
2015-04-24 11420, 2015
14:00 PM
ocharles_
git blame would be the appropriately named tool for that :)
2015-04-24 11438, 2015
14:00 PM
ocharles_
typo or just flat out wrong?
2015-04-24 11442, 2015
14:00 PM
ruaok
2015-04-24 11447, 2015
14:00 PM
ruaok
flat out wrong. :)
2015-04-24 11450, 2015
14:00 PM
ocharles_
heh
2015-04-24 11402, 2015
14:01 PM
ruaok
confusing default install dir with our /usr/local install dir.
2015-04-24 11410, 2015
14:01 PM
ocharles_
hmm, I thought it was configured to look there
2015-04-24 11414, 2015
14:01 PM
ocharles_
evidently not
2015-04-24 11428, 2015
14:01 PM
ruaok
anyways, what chirlu` and I have figured out in the grand scheme of things...
2015-04-24 11430, 2015
14:01 PM
ocharles_
but i mean... it must have been using the auth_file or it wouldn't have worked?
2015-04-24 11436, 2015
14:01 PM
ocharles_ shrugs
2015-04-24 11452, 2015
14:01 PM
ruaok
max_connections to PG should be 2 * number of cores, plus some headroom for other users.
2015-04-24 11401, 2015
14:02 PM
ruaok
search indexes, etc.
2015-04-24 11418, 2015
14:02 PM
ruaok
pgbouncer pool_size should be set to cores * 2.
2015-04-24 11432, 2015
14:02 PM
ruaok
and reserve_pool is 0 for us.
2015-04-24 11439, 2015
14:02 PM
ocharles_
2015-04-24 11448, 2015
14:02 PM
ruaok
this way pgbouncer will never make more than 50 connections to postgres.
2015-04-24 11456, 2015
14:02 PM
ocharles_
yea, I had heard about that max_connections rule before
2015-04-24 11409, 2015
14:03 PM
ocharles_
hadn't heard about how to tune pgbouncer though - that's good knowledge to have
2015-04-24 11437, 2015
14:03 PM
ocharles_
what is reserve_pool 0 doing?
2015-04-24 11403, 2015
14:04 PM
ruaok
reserve pool is to keep some spare connections around in case that things get busy from a quiet period.
2015-04-24 11418, 2015
14:04 PM
ruaok
since we have no such thing as a quiet period, we set it to zero.
2015-04-24 11417, 2015
14:06 PM
ocharles_
ah, i see
2015-04-24 11420, 2015
14:06 PM
ruaok
2015-04-24 11444, 2015
14:06 PM
ruaok
feck. no something isn't right there.
2015-04-24 11449, 2015
14:06 PM
chirlu`
We are not sure yet what max_client_conn should be (i.e. the maximum of inbound connections to pgbouncer).
2015-04-24 11449, 2015
14:06 PM
ocharles_
I think I am - the commit sha is the same as your HEAD
2015-04-24 11417, 2015
14:07 PM
ruaok
my dyslexia makes this subproject commit stuff a real pain. :(
2015-04-24 11405, 2015
14:08 PM
ocharles_
so having too many connections sucks up all ram, is that the main symptom of getting that wrong?
2015-04-24 11417, 2015
14:08 PM
reosarevok joined the channel
2015-04-24 11427, 2015
14:08 PM
ruaok
it can suck up all the ram.
2015-04-24 11442, 2015
14:08 PM
ruaok
esp if pgbouncer can open too many connections.
2015-04-24 11426, 2015
14:09 PM
ruaok
so, now that we have reigned in pgbouncer, we've got ram to spare.
2015-04-24 11433, 2015
14:10 PM
chirlu`
Though it will be automatically used for caching, so it’s not actually “too much”. :-)
2015-04-24 11443, 2015
14:10 PM
ruaok
of course. :)
2015-04-24 11400, 2015
14:11 PM
ruaok
but I am now also going to leave the config as is and just observe.
2015-04-24 11420, 2015
14:11 PM
ruaok
having a few GB should make us more ready to deal with HUGE queries.
2015-04-24 11446, 2015
14:14 PM
ruaok
2015-04-24 11456, 2015
14:16 PM
ruaok
apparently the pgbouncer setup is deep voodoo.
2015-04-24 11405, 2015
14:17 PM
ruaok
we've had a lot of people talk about this on our blog post.
2015-04-24 11446, 2015
14:21 PM
ocharles_
i can believe that
2015-04-24 11413, 2015
14:40 PM
chirlu` has left the channel
2015-04-24 11411, 2015
15:03 PM
ruaok
alastairp: the printer here on piso 3 is out of ink. Who do I bug to get that replaced?
2015-04-24 11421, 2015
15:03 PM
alastairp
cristina or alba
2015-04-24 11428, 2015
15:03 PM
alastairp
but it's 5pm on friday
2015-04-24 11430, 2015
15:03 PM
alastairp
so good luck with that
2015-04-24 11440, 2015
15:03 PM
ruaok
crap.
2015-04-24 11445, 2015
15:03 PM
alastairp
let me ask
2015-04-24 11448, 2015
15:03 PM
ruaok
I'll just go to a print shop tomorrow.
2015-04-24 11453, 2015
15:03 PM
ruaok
not urgent, really.
2015-04-24 11455, 2015
15:03 PM
alastairp
have you tried tapping the drum?
2015-04-24 11415, 2015
15:05 PM
ruaok
I haven't.
2015-04-24 11445, 2015
15:09 PM
ijabz2 joined the channel
2015-04-24 11424, 2015
15:11 PM
JesseW joined the channel
2015-04-24 11430, 2015
15:16 PM
Enverex
Should the "Recent Additions" work on a Mirror? I've noticed everything seems to be fine in general but nothing is listed there.
2015-04-24 11410, 2015
15:23 PM
Enverex
Ah, it requires the edit data which isn't replicated to slaves. So yes, master only. although I've seen a slave with it so not sure how that worked.
2015-04-24 11439, 2015
15:24 PM
Enverex
In the VM instructions, there's a mention of using "bin/reindex". This doesn't exist when running Musicbrainz from source. Is it the equivilent of running an Analyze on the PGSQL tables?
2015-04-24 11457, 2015
15:25 PM
ruaok
no.
2015-04-24 11411, 2015
15:26 PM
ruaok
that builds the lucene search indexes.
2015-04-24 11430, 2015
15:35 PM
ruaok
ffs. I think carl is about to freak out again.
2015-04-24 11437, 2015
15:36 PM
sv2241
Fri Apr 24 06:18:14 2015 : This replication packet matches schema sequence #21, but the database is currently at #20
2015-04-24 11441, 2015
15:36 PM
sv2241
Fri Apr 24 06:18:14 2015 : You must upgrade your database in order to apply this replication packet
2015-04-24 11452, 2015
15:36 PM
sv2241
what exactly should I do here?
2015-04-24 11411, 2015
15:37 PM
ruaok
2015-04-24 11416, 2015
15:37 PM
sv2241
danke
2015-04-24 11438, 2015
15:37 PM
ruaok
kein thema
2015-04-24 11414, 2015
15:39 PM
sv2241
these steps apply to the vm images as well? I'm running the mirror off musicbrainz-server-vmware-2014-05-14.ova image
2015-04-24 11454, 2015
15:39 PM
ruaok
yep
2015-04-24 11439, 2015
15:51 PM
ruaok
rvedotrc: ping
2015-04-24 11413, 2015
16:05 PM
CatQuest joined the channel
2015-04-24 11438, 2015
16:16 PM
yeeeargh joined the channel
2015-04-24 11406, 2015
16:31 PM
zas joined the channel
2015-04-24 11453, 2015
16:35 PM
zas
tons of 302s and MB is very slow, it was a bit better this morning, but it started again recently
2015-04-24 11403, 2015
16:36 PM
zas
what is going on ?
2015-04-24 11439, 2015
16:37 PM
ruaok joined the channel
2015-04-24 11454, 2015
16:37 PM
zas
Hey ruaok
