but wait, then freso can't one the meeting with "Today isNorways National holiday *and* the International Day Against Homophobia and Transphobia" :(
2021-05-16 13654, 2021
CatQuest
boo!
2021-05-16 13641, 2021
sumedh has quit
2021-05-16 13626, 2021
sumedh joined the channel
2021-05-16 13641, 2021
v6lur has quit
2021-05-16 13652, 2021
sumedh has quit
2021-05-16 13621, 2021
shivam-kapila
Lol
2021-05-16 13617, 2021
ruaok
moooin!
2021-05-16 13629, 2021
ruaok
> looks like the error did not get logged. probably that's why sentry didn't see it so no report.
2021-05-16 13643, 2021
ruaok
_lucifer: any idea why that is? maybe my logger isn't setup right?
2021-05-16 13621, 2021
sumedh joined the channel
2021-05-16 13637, 2021
_lucifer
ruaok, could be. but then it did log `INFO:root:metric writer starting!` correctly. do you know where consul template error logs go and how to view them? i know we run it consul with `syslog` so it should be in syslog. however, i couldn't find it there yesterday or maybe i was looking at the wrong place?
but I verified that data is being written to redis. but sometimes values don't make it to finflux, and i suspect somethging in the metric writer is borking.
2021-05-16 13624, 2021
_lucifer
ruaok, right. that's what i saw.
2021-05-16 13647, 2021
ruaok
hmm, indeed very weird. the first log message appears in the log, so logging does appear to be correct.
2021-05-16 13651, 2021
_lucifer
consul should be logging something i believe when the process restarts. it does when we run without syslog directly in container.
> r = requests.post("http://%s:%d/write" % (config.INFLUX_SERVER, config.INFLUX_PORT), params=params, data=lines)
2021-05-16 13632, 2021
ruaok
hmm. requests still interprets data as KV data, not as text.
2021-05-16 13642, 2021
_lucifer
ruaok, maybe lines.encode() works?
2021-05-16 13602, 2021
ruaok
trying that now.
2021-05-16 13657, 2021
ruaok
huh.
2021-05-16 13609, 2021
ruaok
list += item seems to work as list.append(item)
2021-05-16 13602, 2021
shivam-kapila
Wut. My whole python had been a lie
2021-05-16 13650, 2021
shivam-kapila
Doesnt work for ints
2021-05-16 13657, 2021
shivam-kapila
Only for strings
2021-05-16 13609, 2021
ruaok
`line = r.lpop(REDIS_METRICS_KEY)` apparently sometimes fetches data, removes it from redis and returns None. how that hell a, I doing THAT wrong?
2021-05-16 13649, 2021
_lucifer
ruaok, so the data exists but redis returns None?
2021-05-16 13630, 2021
ruaok
seems so, but right now I am fighting docker since it no longer updating the container.
2021-05-16 13644, 2021
_lucifer
what's the issue?
2021-05-16 13601, 2021
ruaok
williams doesn't pull the updated images I push.
2021-05-16 13647, 2021
_lucifer
Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit ?
2021-05-16 13635, 2021
ruaok
where do I see that error?
2021-05-16 13644, 2021
_lucifer
when doing docker pull on williams
2021-05-16 13611, 2021
ruaok
right. clearly time for a nap.
2021-05-16 13620, 2021
yvanzo joined the channel
2021-05-16 13627, 2021
ruaok
and maybe time for us to have our own docker image repo. this isn't workable anymore.
2021-05-16 13637, 2021
_lucifer
i was under the impression docker was logged in on prod machines.
2021-05-16 13653, 2021
ruaok
not necessarily.
2021-05-16 13655, 2021
_lucifer
we can use github container registry maybe?
2021-05-16 13659, 2021
ruaok
I'll just build from williams then.
2021-05-16 13610, 2021
_lucifer
or let's login on williams?
2021-05-16 13609, 2021
ruaok
we really need to make our start_services.sh more quiet. too much noise to see real error messages
2021-05-16 13642, 2021
_lucifer
yeah. for the time being you could try `. scripts/services.sh` and then just run the start service line for your particular service.
2021-05-16 13601, 2021
_lucifer
that's how i and alastairp do it usually.
2021-05-16 13613, 2021
ruaok
oh clever, I should get in the habit of doing that.
2021-05-16 13647, 2021
ruaok
`. scripts/services.sh start_metric_writer "prod"` like that?
2021-05-16 13652, 2021
ruaok
(doesn't output anything)
2021-05-16 13650, 2021
_lucifer
`sudo su; cd ~/docker-server-configs; . scripts/services.sh; start_metric_writer "prod";`
2021-05-16 13603, 2021
ruaok
ah, separate line.
2021-05-16 13609, 2021
_lucifer
something like this
2021-05-16 13610, 2021
_lucifer
yes
2021-05-16 13638, 2021
_lucifer
scripts/services.sh has the functions, we source them and then call the required one
