but wait, then freso can't one the meeting with "Today isNorways National holiday *and* the International Day Against Homophobia and Transphobia" :(
boo!
sumedh has quit
sumedh joined the channel
v6lur has quit
sumedh has quit
shivam-kapila
Lol
ruaok
moooin!
> looks like the error did not get logged. probably that's why sentry didn't see it so no report.
_lucifer: any idea why that is? maybe my logger isn't setup right?
sumedh joined the channel
_lucifer
ruaok, could be. but then it did log `INFO:root:metric writer starting!` correctly. do you know where consul template error logs go and how to view them? i know we run it consul with `syslog` so it should be in syslog. however, i couldn't find it there yesterday or maybe i was looking at the wrong place?
but I verified that data is being written to redis. but sometimes values don't make it to finflux, and i suspect somethging in the metric writer is borking.
_lucifer
ruaok, right. that's what i saw.
ruaok
hmm, indeed very weird. the first log message appears in the log, so logging does appear to be correct.
_lucifer
consul should be logging something i believe when the process restarts. it does when we run without syslog directly in container.
> r = requests.post("http://%s:%d/write" % (config.INFLUX_SERVER, config.INFLUX_PORT), params=params, data=lines)
hmm. requests still interprets data as KV data, not as text.
_lucifer
ruaok, maybe lines.encode() works?
ruaok
trying that now.
huh.
list += item seems to work as list.append(item)
shivam-kapila
Wut. My whole python had been a lie
Doesnt work for ints
Only for strings
ruaok
`line = r.lpop(REDIS_METRICS_KEY)` apparently sometimes fetches data, removes it from redis and returns None. how that hell a, I doing THAT wrong?
_lucifer
ruaok, so the data exists but redis returns None?
ruaok
seems so, but right now I am fighting docker since it no longer updating the container.
_lucifer
what's the issue?
ruaok
williams doesn't pull the updated images I push.
_lucifer
Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit ?
ruaok
where do I see that error?
_lucifer
when doing docker pull on williams
ruaok
right. clearly time for a nap.
yvanzo joined the channel
and maybe time for us to have our own docker image repo. this isn't workable anymore.
_lucifer
i was under the impression docker was logged in on prod machines.
ruaok
not necessarily.
_lucifer
we can use github container registry maybe?
ruaok
I'll just build from williams then.
_lucifer
or let's login on williams?
ruaok
we really need to make our start_services.sh more quiet. too much noise to see real error messages
_lucifer
yeah. for the time being you could try `. scripts/services.sh` and then just run the start service line for your particular service.
that's how i and alastairp do it usually.
ruaok
oh clever, I should get in the habit of doing that.
`. scripts/services.sh start_metric_writer "prod"` like that?
(doesn't output anything)
_lucifer
`sudo su; cd ~/docker-server-configs; . scripts/services.sh; start_metric_writer "prod";`
ruaok
ah, separate line.
_lucifer
something like this
yes
scripts/services.sh has the functions, we source them and then call the required one