so just from looking at the error message it appears that it might be complaining about such a large number of parameters passed in?
2021-05-17 13734, 2021
_lucifer
probably. here the parameters are recording ids i think?
2021-05-17 13715, 2021
alastairp
yes
2021-05-17 13720, 2021
alastairp
recording id, offset
2021-05-17 13715, 2021
alastairp
so, I think I left a comment on the PR - maybe we need to load this in chunks
2021-05-17 13751, 2021
alastairp
if there are more than x items (10,000?) then load them in chunks
2021-05-17 13737, 2021
alastairp
however, I don't understand the error, because max_stack_depth appears to be related to recursive functions, and `load_many_low_level` doesn't seem to do anything unexpected
2021-05-17 13740, 2021
bitmap
yvanzo: reosarevok: hey, we'll start at meeting time in a couple hours if that's okay. I'll set up a banner message
2021-05-17 13704, 2021
reosarevok
Sure! You mean preparations or release?
2021-05-17 13723, 2021
reosarevok
(as in, do we start the preparations nowish or in 2h)
2021-05-17 13757, 2021
bitmap
we can start preparations nowish, but finish dinner first :)
2021-05-17 13717, 2021
_lucifer
alastairp, yeah right. i haven't dived very deep in the code yet so not i am not sure either what's happening. i'll start looking into the PR and then see how we can chunk it.
2021-05-17 13736, 2021
alastairp
I think chunking is probably going to be the easiest way, the question will be how big can we make the limit to minimise number of round trips to pg. there is a function called `chunks` copied in a few places in that source, we should move it to a utils module so that we can use it from other places
we don't have temperature before they changed the fan (I fixed it, a kernel module was missing)
2021-05-17 13746, 2021
ruaok
oh.
2021-05-17 13708, 2021
ruaok
well, I'm going to keep it under heavy load for 1000s and if it passes, we'll resume moving services to it.
2021-05-17 13730, 2021
zas
the graph looks normal to me, it follows load, cpu isn't throttling (I just checked)
2021-05-17 13755, 2021
ruaok
yeah, me too.
2021-05-17 13702, 2021
ruaok
I'll let it finish.
2021-05-17 13734, 2021
zas
hetzner buys crappy cpu fans, likely they have a whole stock of those....
2021-05-17 13705, 2021
ruaok
I bet they replace them with better fans than the factory fans.. I just don't understand why they dont do it on install.
2021-05-17 13727, 2021
ruaok
it would save them a lot of money/customer aggrevation.
2021-05-17 13712, 2021
ruaok
and it does jive with what we were doing -- loads of heavy compression just before crash.
2021-05-17 13758, 2021
ruaok
alastairp: I'm going to move redis back to lemmy. our timestamps are all invalid on boingo and now will need to be recomputed. I'll do that on lemmy now that it looks like lemmy is happy again.
2021-05-17 13712, 2021
alastairp
👍
2021-05-17 13742, 2021
alastairp
I forgot about that data. not much we could have done about it in the moment
2021-05-17 13746, 2021
ruaok
1000s test complete. all good.
2021-05-17 13711, 2021
alastairp
possible advantage to having a dedicated redis cluster spread over a few machines to ensure that we don't lose all of the data at once in case of an outage
Is there an environment variable or something that can be used to set Picard’s debug level (overriding config)? I know there’s the `-d` flag, but I’m looking for something that will lower the level, but also potentially customise it depending on other things.
2021-05-17 13759, 2021
_lucifer
ruaok, i have set the tempdir. pyspark picks it up correctly. i think request consumer will as well. should we import an incremental dump first to confirm?
2021-05-17 13739, 2021
ruaok
not sure it will do that, since we only have out-of-sequence incrementals.
