ruaok: import failed. no diskspace left on device.
sumedh joined the channel
sampsyo has quit
sampsyo joined the channel
sumedh has quit
Rohan_Pillai joined the channel
ruaok
Mooooiin!
_lucifer: any idea how to clean up?
zas
bitmap: postgres-williams on paco needs more diskspace, it should go back to williams imho (and few containers on williams should prolly run on paco instead)
bitmap: I truncated pg log file on floyd, we still need to restart docker to control log file size there
Rohan_Pillai has quit
log file was doing 172Gb
_lucifer
ruaok: what's the size of the dump? we could clear the incomplete dumps and other things from hdfs.
that drive has 216G at max available and docker is using that images and other containers as well
just a docker prune can yield ~20G. clearing the temp files and incomplete dump should yield another ~125G
how much disk space do other nodes in the cluster have?
ruaok
they should all have the same specs.
let's clean up then!
do you know how?
_lucifer
hdfs -rm -r -skipTrash `path` inside the namenode should do that
let me try it
ruaok, i am trying to delete but there are some issues with namenode. can take some time to diagnose.
in the meanwhile, you might want to take a look at this
ruaok: i am trying to figure why it went on looking way back in the past for listens. ideally it should go from the start of the range to its end. which job was in the queue when you cleared it?
ruaok
it was a stats job that was working.
I forget the exact one, but we can guess that it was the first one running according to the daily crontab
_lucifer
👍
the sentry stack trace is sparticularly unhelpful :(
ruaok: i am unable to debug the issue using the info present in sentry. is it possible to view the logs of the request conusmer before it was restarted and also is it fine if i change spark logging level for sentry to debug?
ruaok
sorry no, in order to free diskspace, I purged old containers. :(
_lucifer
no i mean the logs of the container you started after that but before clearing the queue
ruaok
one sec. let me finish this €66,000 task real quick.
_lucifer
sure
reosarevok
ruaok: dunno if you saw the mail to (I assume) modbot?