You're doing good work, you guys! (even if you're not done yet)!
2018-04-30 12023, 2018
kaliko has quit
2018-04-30 12035, 2018
bitmap
it receives sighup when templates changes (which causes it to reload the config)
2018-04-30 12044, 2018
bitmap
after templates are rendered, I mean
2018-04-30 12057, 2018
zas
but then why a manual restart worked ?
2018-04-30 12023, 2018
zas
nvm, i restarted consul template (not pgbouncer directly)
2018-04-30 12026, 2018
bitmap
I suppose restarting it with sv would also restart the consul-template process
2018-04-30 12024, 2018
zas
bitmap: i propose to increase the log verbosity for this consul template process so we can have more infos about what's going on
2018-04-30 12053, 2018
zas
also we need to find a way to reproduce it, it happened after a full reboot (but not after a container restart)
2018-04-30 12002, 2018
i7c has quit
2018-04-30 12043, 2018
bitmap
sometimes consul-template will get triggered when you push *anything* to docker-server-configs, and write a blank config for pgbouncer
2018-04-30 12009, 2018
bitmap
possibly every time, I think we were thinking it was git2consul
2018-04-30 12042, 2018
zas
that's not the first time it bites us
2018-04-30 12038, 2018
i7c joined the channel
2018-04-30 12007, 2018
bitmap
the "No such database: musicbrainz_db" error comes from pgbouncer, which implied that database wasn't in its config, i.e. it was empty. so I'm guessing it was the same thing this time
2018-04-30 12059, 2018
zas
is there any other template using the same key ?
2018-04-30 12020, 2018
bitmap
the keys under docker-server-configs/services/postgres-master.json? nope
2018-04-30 12049, 2018
zas
what's happening when pgbouncer is started without a valid config ?
2018-04-30 12026, 2018
zas
i mean it was apparently running and happy, but obviously was not working
2018-04-30 12030, 2018
bitmap
it starts fine but doesn't allow any connections to dbs not in its config
basically there are conditions where a valid key/val is removed then written again, and it may explain things, but when they are written again consul template should "fix" the config with new values, but it doesnt
2018-04-30 12011, 2018
zas
and we didn't restart git2consul and/or consul to fix the issue, but just consul-template
2018-04-30 12021, 2018
bitmap
no new release since that pr was merged
2018-04-30 12049, 2018
bitmap
true, I would expect it to update teh template again after the write
2018-04-30 12049, 2018
zas
yes, but perhaps the behavior of git2consul and the quick replacement doesn't trigger an update
2018-04-30 12001, 2018
zas
we are trying to guess too much, we need more logs
2018-04-30 12020, 2018
bitmap
right
2018-04-30 12024, 2018
surtin joined the channel
2018-04-30 12051, 2018
zas
also consul-template 0.18.5 is used in pg master container, perhaps worth a upgrade to 0.19.4
2018-04-30 12056, 2018
ruaok
is service "postgres-master" points to bowie, does service "postgres-slave" point to queen and would be suitable for building indexes?
2018-04-30 12001, 2018
ruaok
if...
2018-04-30 12019, 2018
zas
back to the initial issue: response times are still bad, and that's due to bowie's load which didn't change after reboot
2018-04-30 12014, 2018
ruaok
bowie is overloaded. that is all there is to it.
2018-04-30 12027, 2018
bitmap
ruaok: yes, it should be, though if queen traffic increases the pg there may generate a lot of "conflict with recovery" errors
2018-04-30 12030, 2018
zas
but there's no reason for it to be overloaded
2018-04-30 12031, 2018
ruaok
we can keep guessing. we can keep hemming. we can keep hawing.
2018-04-30 12042, 2018
ruaok
we need to reduce the load on bowie or nothing gets better.
2018-04-30 12055, 2018
zas
the traffic is the same as usual (and we had much more in past months, without having this load)
2018-04-30 12058, 2018
ruaok
zas: yes, there is. we're handling more search traffic than ever before.
let me know when I can point the updated search stuff to queen, bitmap
2018-04-30 12025, 2018
bitmap
ok, a few mins while I update things
2018-04-30 12001, 2018
Slurpee has quit
2018-04-30 12055, 2018
zas
bitmap, yvanzo: i also noticed a thing with slowdowns: searching for locations is failing a lot, revealing sql requests are done each time, isn't this cached locally ? i mean if i look for a country named Spain when adding an artist, i expect it to be cached for a while (countries/towns are unlikely to change often, and i wasn't using direct search)
2018-04-30 12000, 2018
bitmap
the area containments should be cached, yes. I'll check why those are being queried so often once I update queen
2018-04-30 12050, 2018
bitmap
(I mean, we do actually cache them, but something's fishy)
