#metabrainz

lol

but wait, then freso can't one the meeting with "Today isNorways National holiday *and* the International Day Against Homophobia and Transphobia" :(

boo!

Lol

moooin!

> looks like the error did not get logged. probably that's why sentry didn't see it so no report.

_lucifer: any idea why that is? maybe my logger isn't setup right?

ruaok, could be. but then it did log `INFO:root:metric writer starting!` correctly. do you know where consul template error logs go and how to view them? i know we run it consul with `syslog` so it should be in syslog. however, i couldn't find it there yesterday or maybe i was looking at the wrong place?

_lucifer: ^^ nuthing in syslog :(

but I verified that data is being written to redis. but sometimes values don't make it to finflux, and i suspect somethging in the metric writer is borking.

ruaok, right. that's what i saw.

hmm, indeed very weird. the first log message appears in the log, so logging does appear to be correct.

consul should be logging something i believe when the process restarts. it does when we run without syslog directly in container.

maybe zas or bitmap know where to look?

and I've got the most loose check for errors from influx, but it appears it never gets tripped: https://github.com/metabrainz/metric-writer/blo...

hmmm. maybe write_points doesn't throw exceptions?

yeah. also there was one another line in the log which did not make sense

* os: process already finished

yeah, no clue on that either.

not sure why this ^ got printed.

i don't know about write_points either. will need to check docs.

write _points returns bool. not useful, really.

the docs are shitty, I'm reading the code.

:(

I can try checking for false and attempt a re-try.

afterall the influx server is on another continent, so errors more likely.

true.

these are the current logs. any idea about those cron jobs?

oh, I need to installl cron/down.

because we need no cron in this container.

ah ok.

the v1.x version of the python API is really piss poor.

Project bookbrainz-site build #3968: errored in 1 min 44 sec: https://travis-ci.org/bookbrainz/bookbrainz-sit...

behold this SHIT code, _lucifer : https://github.com/influxdata/influxdb-python/b...

you know what? screw this. I will submit the data via the HTTP api and sod this API.

lol!

the http api seems more informative in case of errors than the python library

definitely better off using HTTP then

on it

ruaok, https://sentry.metabrainz.org/metabrainz/metric... errors showing up now

influx just never threw any errors at all.

makes sense.

> r = requests.post("http://%s:%d/write" % (config.INFLUX_SERVER, config.INFLUX_PORT), params=params, data=lines)

hmm. requests still interprets data as KV data, not as text.

ruaok, maybe lines.encode() works?

trying that now.

huh.

list += item seems to work as list.append(item)

Wut. My whole python had been a lie

Doesnt work for ints

Only for strings

`line = r.lpop(REDIS_METRICS_KEY)` apparently sometimes fetches data, removes it from redis and returns None. how that hell a, I doing THAT wrong?

ruaok, so the data exists but redis returns None?

seems so, but right now I am fighting docker since it no longer updating the container.

what's the issue?

williams doesn't pull the updated images I push.

Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit ?

where do I see that error?

when doing docker pull on williams

right. clearly time for a nap.

and maybe time for us to have our own docker image repo. this isn't workable anymore.

i was under the impression docker was logged in on prod machines.

not necessarily.

we can use github container registry maybe?

I'll just build from williams then.

or let's login on williams?

we really need to make our start_services.sh more quiet. too much noise to see real error messages

yeah. for the time being you could try `. scripts/services.sh` and then just run the start service line for your particular service.

that's how i and alastairp do it usually.

oh clever, I should get in the habit of doing that.

`. scripts/services.sh start_metric_writer "prod"` like that?

(doesn't output anything)

`sudo su; cd ~/docker-server-configs; . scripts/services.sh; start_metric_writer "prod";`

ah, separate line.

something like this

yes

scripts/services.sh has the functions, we source them and then call the required one

_lucifer: https://github.com/metabrainz/metric-writer/blo...

it almost always returns None.

even when there is data.

but redis has the data?

yes.

see the llen past above.

hmmm.

strange. what does this while loop do? fetch all the items at that key and decode them?

I wonder if there is some other metric writer longering somehwere. because the data vanished even without my reader. lol

lol

🤦

https://www.eff.org/deeplinks/2021/05/outliving... via https://twitter.com/mutesplash/status/139391719... Good question, why is MB not mentioned?

_lucifer: much better now: https://stats.metabrainz.org/d/OGg5QUCGz/listen...