_lucifer: is error_message a string, or a dictionary?
2021-04-20 11009, 2021
_lucifer
string
2021-04-20 11028, 2021
alastairp
so in this case you're only using json.dumps to add quotes to it?
2021-04-20 11058, 2021
_lucifer
yes, i was. but format thing worked as well so i'll go ahead with it,
2021-04-20 11018, 2021
alastairp
I've done this before where I use json.JSONEncoder().encode(val)
2021-04-20 11008, 2021
alastairp
because, consider:
2021-04-20 11015, 2021
alastairp
s = 'foo"bar'; json.dumps(s)
2021-04-20 11020, 2021
alastairp
'"foo\\"bar"'
2021-04-20 11011, 2021
_lucifer
ah! makes sense.
2021-04-20 11050, 2021
ruaok
alastairp: got a sec for a sanity check?
2021-04-20 11009, 2021
alastairp
give me 5, just fighting with haproxy
2021-04-20 11030, 2021
ruaok
perfect. it will take me 5 minutes to explain. ;D
2021-04-20 11033, 2021
_lucifer
is there any difference between using dumps or the encoder?
2021-04-20 11013, 2021
ruaok
the listen count are very expensive to calculate and when we first push this branch into production, the site will appear effectively broken because nothing is loading.
2021-04-20 11026, 2021
ruaok
to mitigate that, I plan to write a script that:
2021-04-20 11040, 2021
ruaok
1) Picks the last inserted timestamp from the listen table.
2021-04-20 11040, 2021
ruaok
2) For each user, set a zero listen count, zero timestamp.
2021-04-20 11047, 2021
ruaok
3) iterate over every listen from begin of time to this timestamp. tabulate listen counts and time.
2021-04-20 11020, 2021
_lucifer
how much expensive are we talking about?
2021-04-20 11043, 2021
ruaok
4) At the end of this script INCREMENT each of the users counts by the calculated total. Update timestamps to ensure that calculated timestamps won't overwrite any timestamps that may have been calculated since this process started.
2021-04-20 11016, 2021
ruaok
This process allows for the timescale_writer to keep writing new listens and for the update script and the timescale writer to coexist peacefully.
2021-04-20 11031, 2021
ruaok
In theory we should catch all the listens as they come in.
2021-04-20 11034, 2021
ruaok
thoughts?
2021-04-20 11041, 2021
ruaok
_lucifer: I saw one take 85s.
2021-04-20 11028, 2021
_lucifer
😵
2021-04-20 11037, 2021
alastairp
here
2021-04-20 11001, 2021
ruaok
exactly, which is why I really want to stop computing them.
2021-04-20 11058, 2021
alastairp
yeah, doing a script to pre-compute these things makes a lot of sense. does this mean we'll have to make a deploy with the infrastructure available but disabled? or will you be able to do it before the big deploy?
2021-04-20 11024, 2021
alastairp
tabulate listen counts and time - this will just find min/max time and num listens for each user?
2021-04-20 11044, 2021
ruaok
> tabulate listen counts and time - this will just find min/max time and num listens for each user?
2021-04-20 11045, 2021
ruaok
yes
2021-04-20 11059, 2021
ruaok
> or will you be able to do it before the big deploy?
2021-04-20 11041, 2021
ruaok
I think that if we deploy the timescale_writer first, then we can run the tabulate script, then push the web container, then I think the right thing will happen.
2021-04-20 11051, 2021
ruaok
More correctly:
2021-04-20 11054, 2021
ruaok
1. tabulate.
2021-04-20 11003, 2021
ruaok
2. deploy timescale writer
2021-04-20 11011, 2021
ruaok
3. tabulate again
2021-04-20 11018, 2021
ruaok
4. deploy web
2021-04-20 11038, 2021
ruaok
then the release should be seamless.
2021-04-20 11038, 2021
alastairp
what service updates max and num listens when necessary? timescale writer?
2021-04-20 11056, 2021
ruaok
was timescale writer, but a lot of that logic has moved to timescale listenstore.
2021-04-20 11001, 2021
ruaok
where it belongs.
2021-04-20 11015, 2021
alastairp
but the writer container, not the web container?
2021-04-20 11021, 2021
ruaok
there is a lot of really needed cleanup in this PR.
2021-04-20 11022, 2021
alastairp
do you need any special code running in a container in order to do 1 ?
2021-04-20 11052, 2021
ruaok
yes,its runs in the writer container with no special code needed.
2021-04-20 11011, 2021
_lucifer
beta, and prod share the cache container so we could run the script in beta if needed?
2021-04-20 11013, 2021
ruaok
just python module speaking that logic moved to the listenstore.
2021-04-20 11033, 2021
ruaok
_lucifer: yeah, sure. good idea.
2021-04-20 11045, 2021
alastairp
ruaok: yeah, that's what I was trying to get at. running in beta should be fine
2021-04-20 11055, 2021
ruaok
the key being, the writer needs to be the first thing to move as part of the deployment.
2021-04-20 11013, 2021
ruaok
you can test all this one test.lb right now.
2021-04-20 11019, 2021
ruaok
load your feed page to find out what I mean.
2021-04-20 11030, 2021
ruaok
it will appear broken, I assure you.
2021-04-20 11046, 2021
alastairp
what's the purpose of 3? because of stuff that might come in between tabulate and when we shut down the old timescale writer?
2021-04-20 11030, 2021
ruaok
#1 is needed for #2 to work right. #3 is the part that does the actual work.
2021-04-20 11043, 2021
ruaok
I suppose #1 could be reduced to "set all redis keys to 0."
2021-04-20 11052, 2021
ruaok
but that is more code to write, lol
2021-04-20 11008, 2021
alastairp
ah, I follow now
2021-04-20 11046, 2021
_lucifer
another thing, how about getting the list of all users and then querying the api in beta container for all users.
2021-04-20 11007, 2021
_lucifer
instead of writing another script for the same task.
2021-04-20 11056, 2021
ruaok
that could work, but it would tie up the DB for hours. and it would make N passes over the data, as opposed to 1.
2021-04-20 11041, 2021
ruaok
and this other script I am talking about, all its pieces already exist.
2021-04-20 11054, 2021
ruaok
its just a matter of conjuring them into one script.
there's definitely something to be said for just sshing into a computer and running your webserver in a screen
2021-04-20 11053, 2021
_lucifer
nice but why the move to kubes?
2021-04-20 11005, 2021
alastairp
because our IT department supports it
2021-04-20 11002, 2021
_lucifer
ah! :D
2021-04-20 11049, 2021
yvanzo
I would love to read more your experience about it :)
2021-04-20 11007, 2021
alastairp
🤮
2021-04-20 11009, 2021
alastairp
lol
2021-04-20 11025, 2021
alastairp
nah, it's not too bad. we're lucky that IT gave us a set of templates to copy and fill out.
2021-04-20 11031, 2021
yvanzo
:D
2021-04-20 11008, 2021
alastairp
there are a lot of moving parts, and coming into it with no knowledge about how everything fits together there was a lot of guessing. I'm sure that as we migrate more services to it we'll come to understand better how everything works
