is there anything we can learn from the post (haven't read it yet)
ruaok
success is scary. :)
Gazooo joined the channel
good tips for operating at scale, which isn't our problem... yet/
I've skimmed it, but will read fully later.
iliekcomputers
>We processed ~5X the amount of data compared to the Wrapped 2018 campaign while spending 25% less overall for processing This was achieved by designing our system in a way that reduced group-by key operations, and reused the output of yearly jobs to produce top of the decade metrics.
iliekcomputers: so, good news. the upgrade to influxdb 1.7.9 seemed to work well. I need to make a few tests, to ensure that it did, but it seems so.
I also make a backup of the data and loaded the data back into a 1.2.4 instance, without problems.
the new version has more features for analyzing our data structure and ensure if we're doing things in a scalable way.
sumedh has quit
it will be good to use those tools to see what we need for greater scalability. and it appears that 1.7 is already more scalable than previous versions.
I'll prepare a migration doc today.
iliekcomputers
ruaok: how did you test it?
ruaok
just a cursory query so far. I hope to actually load a full version of LB (including PG) on this test server so I can run tests and actually use the data.
iliekcomputers
ah
one thing to consider
the production version of LB is probably running an older version of the client
(when compared to master)
ruaok
yes, indeed.
shivam-kapila joined the channel
rough outline for starters.
iliekcomputers
looking sane to me
to upgrade the client, you'll make a pr against the production branch ?
if we need to hotfix something, we have a mess on our hands.
iliekcomputers
seems not a good*
ruaok: yeah, agreed.
however, we should prioritize it.
ruaok
but I think this should be a top priority goal for us.
lol.
iliekcomputers
yep.
promehul joined the channel
mzfr[m] has quit
divyanshumahajan has quit
divyanshumahajan joined the channel
mzfr[m] joined the channel
promehul has quit
while we're at it, it might be a good idea to discuss ways to remove this toil when i'm in bcn.
promehul joined the channel
ruaok
in general I need to bug others about joining us that weekend...
iliekcomputers: also, do we want to focus on bog standard LB stuff and push to make progress there, or should we also see about having Mr_Monkey join us and focus on some of the recommendation tasks that could benefit from his help?
D4RK-PH0ENiX has quit
D4RK-PH0ENiX joined the channel
promehul has quit
iliekcomputers
ruaok: haven't thought about it much yet.
what is the recommendation stuff?
madmouser1 joined the channel
yvanzo
iliekcomputers: I found libxslt1-dev is installed in many of our Python projects, do you have any idea why?
git log didn’t really help as it is present in initial commit of Dockerfiles.
I found it is a requirement of lxml which is not used anymore but with mb-rngpy.