_lucifer: please give an overview of the problem and the fix in the PR body, and link to jira (or just list the issue name in the body) so that I can easily look up the ticket
2020-09-25 26922, 2020
_lucifer
alastairp: i'll add the information to the PRs. I did not give the ticket in the body because that is the subject of the PR.
2020-09-25 26950, 2020
_lucifer
*title of the PR
2020-09-25 26950, 2020
alastairp
_lucifer: but github doesn't make the ticket name in the title clickable. if it's in the body it turns it into a link to jira automatically
2020-09-25 26901, 2020
_lucifer
oh ok will fix
2020-09-25 26924, 2020
alastairp
currently I have to copy the ticket number from title and paste into url, if it's in the body I can just click it
2020-09-25 26925, 2020
alastairp
:)
2020-09-25 26932, 2020
supersandro2000 has quit
2020-09-25 26909, 2020
supersandro2000 joined the channel
2020-09-25 26930, 2020
_lucifer
alastairp: updated the PRs with the tickets and added some more info as well
2020-09-25 26902, 2020
_lucifer
#312 for frontend check is still a work in progress (I forgot to mark it as a draft) because I want to see how it interacts with the changes in #311
2020-09-25 26937, 2020
_lucifer
except these two PRs, I believe all others are ready to be merged
2020-09-25 26917, 2020
_lucifer
(all other PRs that concern tickets planned for the release)
shivam-kapila: i'd really appreciate if you could add the backend tests to your component PR. they'll be helpful in my PR.
2020-09-25 26900, 2020
iliekcomputers
the frontend tests can wait
2020-09-25 26920, 2020
alastairp
iliekcomputers: thanks for the review on the BU stats PR, I'll try a few different things for the api
2020-09-25 26935, 2020
iliekcomputers
alastairp: I was thinking about it
2020-09-25 26908, 2020
iliekcomputers
how does the grafana thing work. grafana pulls in a metric at intervals of time from the API endpoint we'll expose, right?
2020-09-25 26921, 2020
alastairp
zas will be in barcelona next week, if we have time I'll try and grab him for a while to see how the integration to granfa will work
2020-09-25 26922, 2020
alastairp
yes, right
2020-09-25 26930, 2020
ruaok
telegraf, not grafana, iliekcomputers.
2020-09-25 26939, 2020
ruaok
grafana is just the graphing tool.
2020-09-25 26940, 2020
iliekcomputers
right, thanks.
2020-09-25 26949, 2020
alastairp
so many technologies
2020-09-25 26903, 2020
ruaok
something, often telegraf, gets data into influx and grafana plots from influx.
2020-09-25 26913, 2020
iliekcomputers
do we need the time interval thing? Just increment a key in redis, telegraf pulls it in at intervals we configure and then we group them on grafana
2020-09-25 26916, 2020
alastairp
you're right about if we actually need the buckets
2020-09-25 26918, 2020
iliekcomputers
or is that not possible
2020-09-25 26950, 2020
ruaok
what is the grouping that is desired here?
2020-09-25 26900, 2020
iliekcomputers
like number of users who've signed up
2020-09-25 26911, 2020
iliekcomputers
we want to see number of users per day, per month and so on
2020-09-25 26912, 2020
alastairp
ruaok: we want to plot graphs, of things like "number of x per time period y"
2020-09-25 26923, 2020
ruaok
ok, then do nothing intelligent yet.
2020-09-25 26935, 2020
alastairp
the question is should LB/AB/etc store counts per minute/10 minutes/hour
2020-09-25 26936, 2020
ruaok
just say things like "in the last hour 15 people signed up"
2020-09-25 26950, 2020
alastairp
right, so you're proposing groupings
2020-09-25 26952, 2020
alastairp
of 1 hour
2020-09-25 26955, 2020
alastairp
(?)
2020-09-25 26912, 2020
ruaok
yes, but no further smarts are really needed.
2020-09-25 26929, 2020
iliekcomputers
>just say things like "in the last hour 15 people signed up"
2020-09-25 26930, 2020
ruaok
I mean you could send a message for each one, but that would have the potential for becoming too much data.
2020-09-25 26940, 2020
iliekcomputers
the 15 minutes here
2020-09-25 26943, 2020
alastairp
the way I implemented it is "give the stat a name", "choose a grouping duration" (with some defaults of 1 minute, 10 minutes, 60 minutes)
2020-09-25 26951, 2020
iliekcomputers
does that need to be configurable
2020-09-25 26904, 2020
alastairp
and then when telegraf hits the API, it gets the counts in those groups
2020-09-25 26905, 2020
ruaok
I think that will be good, yes.
2020-09-25 26907, 2020
iliekcomputers
maybe it's a case of choosing a reasonable default and still allowing it to be configurable
2020-09-25 26910, 2020
ruaok
(making it configurable)
2020-09-25 26921, 2020
ruaok
+1. good default and config
2020-09-25 26932, 2020
alastairp
iliekcomputers: cool, so sounds like we just want a wrapper that sets a default
2020-09-25 26937, 2020
iliekcomputers
Ideally, I'd want to just do `Metrics.increment('users_signed_up')` in the sign up flow, and then see graphs
2020-09-25 26945, 2020
iliekcomputers
alastairp: yep!
2020-09-25 26952, 2020
alastairp
perfect, that's easy
2020-09-25 26910, 2020
ruaok
this is great -- I've been wanting something like this.
2020-09-25 26922, 2020
ruaok
etsy makes it super simple to record data points like this.
2020-09-25 26949, 2020
ruaok
and zas has shown that it works, because he can see problems that will impact the service sometime in the future.
2020-09-25 26921, 2020
iliekcomputers
graphs like these can be really helpful.
2020-09-25 26930, 2020
ruaok
super helpful.
2020-09-25 26923, 2020
alastairp
cool, so this PR is really close then
2020-09-25 26916, 2020
alastairp
I'll talk with zas next week to close the loop to telegraf
2020-09-25 26931, 2020
ruaok
is your data stored in redis until it gets collected by telegraf?
2020-09-25 26934, 2020
alastairp
yes
2020-09-25 26958, 2020
alastairp
iliekcomputers: mmm. what about `metrics.onehour.increment('users_signed_up)` ?
2020-09-25 26923, 2020
alastairp
I guess that's not much different than `.increment(users_signed_up, 60)`, which we're trying to get away from
2020-09-25 26929, 2020
iliekcomputers
i think the time period will just be extra overhead for devs in most cases.
2020-09-25 26908, 2020
alastairp
yeah, it's possible that we really never need any other periods than 1 hour, and we won't really be able to tell that until we release and start gathering stats
2020-09-25 26924, 2020
iliekcomputers
if i'm reading code in the sign up flow, `metrics.increment('user_signed_up')` fits in well, but thinking about what the `onehour` means here makes things more complicated.
2020-09-25 26948, 2020
alastairp
agreed
2020-09-25 26953, 2020
iliekcomputers
the 1 hour here means that telegraf will fetch things from redis every hour right?
2020-09-25 26909, 2020
iliekcomputers
i think the default should probably be lower, if so. maybe 15 minutes.
2020-09-25 26928, 2020
alastairp
this is distinct from how often it fetches. it's just how we store it in redis (and return the json)
2020-09-25 26918, 2020
alastairp
if we return a key {'date-14:00': 59} and then on the next fetch {'date-14:00': 145}, and influx does the right thing, that's probably fine
2020-09-25 26938, 2020
iliekcomputers
ah
2020-09-25 26939, 2020
iliekcomputers
ok
2020-09-25 26956, 2020
alastairp
but if it requires all of the stats for the time period to be fixed at the time that it collects it, then sure, we can make the default lower and group it in granfa
2020-09-25 26915, 2020
alastairp
again, we'll have to release it and see what happens
2020-09-25 26921, 2020
iliekcomputers
if we return {'1400': 59} and then {1400: 180}, we're assuming that the final value in influx would be 180
2020-09-25 26929, 2020
iliekcomputers
is that correct?
2020-09-25 26902, 2020
ruaok
ok, as far as having telegraf read the data from redis, you'll want to use this:
if so, then 1 hour essentially becomes the resolution of the data in influx, which is a bit on the higher side imo
2020-09-25 26953, 2020
ruaok
there is an input plugin to monitor redis, but that is not what you want. we already have that. we want to pull data from redis and it seems the easiest way to do that is to write an HTTP query and use this plugin to fetch it periodically.
2020-09-25 26920, 2020
ruaok
then one machine at hetzner needs to have its /etc/telegraf/telegraf.conf file modified to use the HTTP plug in.
2020-09-25 26936, 2020
ruaok
how exactly to do that, we'll need zas for that.
2020-09-25 26926, 2020
alastairp
yeah, I briefly discussed this with zas back when I started this, and we said that we'd talk again once I had the data being stored
