ruaok: import failed. no diskspace left on device.
2021-03-05 06438, 2021
sumedh joined the channel
2021-03-05 06452, 2021
sampsyo has quit
2021-03-05 06440, 2021
sampsyo joined the channel
2021-03-05 06412, 2021
sumedh has quit
2021-03-05 06446, 2021
Rohan_Pillai joined the channel
2021-03-05 06404, 2021
ruaok
Mooooiin!
2021-03-05 06424, 2021
ruaok
_lucifer: any idea how to clean up?
2021-03-05 06436, 2021
zas
bitmap: postgres-williams on paco needs more diskspace, it should go back to williams imho (and few containers on williams should prolly run on paco instead)
2021-03-05 06405, 2021
zas
bitmap: I truncated pg log file on floyd, we still need to restart docker to control log file size there
2021-03-05 06430, 2021
Rohan_Pillai has quit
2021-03-05 06449, 2021
zas
log file was doing 172Gb
2021-03-05 06411, 2021
_lucifer
ruaok: what's the size of the dump? we could clear the incomplete dumps and other things from hdfs.
2021-03-05 06441, 2021
_lucifer
that drive has 216G at max available and docker is using that images and other containers as well
2021-03-05 06446, 2021
_lucifer
just a docker prune can yield ~20G. clearing the temp files and incomplete dump should yield another ~125G
2021-03-05 06455, 2021
_lucifer
how much disk space do other nodes in the cluster have?
2021-03-05 06447, 2021
ruaok
they should all have the same specs.
2021-03-05 06451, 2021
ruaok
let's clean up then!
2021-03-05 06454, 2021
ruaok
do you know how?
2021-03-05 06441, 2021
_lucifer
hdfs -rm -r -skipTrash `path` inside the namenode should do that
2021-03-05 06444, 2021
_lucifer
let me try it
2021-03-05 06433, 2021
_lucifer
ruaok, i am trying to delete but there are some issues with namenode. can take some time to diagnose.
2021-03-05 06447, 2021
_lucifer
in the meanwhile, you might want to take a look at this
ruaok: i am trying to figure why it went on looking way back in the past for listens. ideally it should go from the start of the range to its end. which job was in the queue when you cleared it?
2021-03-05 06413, 2021
ruaok
it was a stats job that was working.
2021-03-05 06442, 2021
ruaok
I forget the exact one, but we can guess that it was the first one running according to the daily crontab
2021-03-05 06445, 2021
_lucifer
👍
2021-03-05 06419, 2021
_lucifer
the sentry stack trace is sparticularly unhelpful :(
ruaok: i am unable to debug the issue using the info present in sentry. is it possible to view the logs of the request conusmer before it was restarted and also is it fine if i change spark logging level for sentry to debug?
2021-03-05 06433, 2021
ruaok
sorry no, in order to free diskspace, I purged old containers. :(
2021-03-05 06432, 2021
_lucifer
no i mean the logs of the container you started after that but before clearing the queue
2021-03-05 06412, 2021
ruaok
one sec. let me finish this €66,000 task real quick.
2021-03-05 06425, 2021
_lucifer
sure
2021-03-05 06427, 2021
reosarevok
ruaok: dunno if you saw the mail to (I assume) modbot?
