#metabrainz

ruaok: import failed. no diskspace left on device.

Mooooiin!

_lucifer: any idea how to clean up?

bitmap: postgres-williams on paco needs more diskspace, it should go back to williams imho (and few containers on williams should prolly run on paco instead)

bitmap: I truncated pg log file on floyd, we still need to restart docker to control log file size there

log file was doing 172Gb

ruaok: what's the size of the dump? we could clear the incomplete dumps and other things from hdfs.

that drive has 216G at max available and docker is using that images and other containers as well

just a docker prune can yield ~20G. clearing the temp files and incomplete dump should yield another ~125G

how much disk space do other nodes in the cluster have?

they should all have the same specs.

let's clean up then!

do you know how?

hdfs -rm -r -skipTrash `path` inside the namenode should do that

let me try it

ruaok, i am trying to delete but there are some issues with namenode. can take some time to diagnose.

in the meanwhile, you might want to take a look at this

why if we just reformatted our HDFS and start over?

it seems all other datanodes are offline

that is supposed to be a valid use case. we need to reimport all the data.

it really sounds like the cluster needs a complete reboot. so let's do that.

yeah let's try that

did something happen on March 2? new datanode containers came up on leader that day and other workers went offline the same day

not that i know of, but the problem is that these systems aren't monitored, so hard to know.

yeah :(

I'm really warming up to your suggestion of using yarn and not docker to run the cluster.

once we are able to do that, then let's get 4 new machines, have zas monitor them and restart the cluster.

yeah makes sense

ok, 11G free.

ok, cluster stopped.

name node volumes dropped, recreated. now to do that to each datanode.

_lucifer: ok, reset complete. can you see if the cluster looks healthy?

on it

yup all datanodes are up

yay. lets start the dump import anew

yup

ehhh uhm.

look at the logs of the request consumer.

seems that its trying to clean up stuff that doesn't exist.

now trying to erase listens from the medival times.

1500s. :)

lets see if it stops when it gets to baby jesus.

lol

or its trying to execute an old task in the queue. list generate dataframes, but no data present.

nope.

I cleared the queue and re-entered the import command.

we need to debug why its doing what its doing. :(

ah. its trying to calculate stats.

maybe the queue purge failed.

yvanzo: ping

confirmed queue not purged.

previous magic to do so doesn't work on new install. need assistance from yvanzo

ruaok: pong

hiya!

I'm trying to purge a listenbrainz queue, but it doesn't seem to work right.

where is rabbitmqadmin installed on prince?

I copied over the file from trille, which may be the first mistake.

on trille: docker cp rabbitmq-prince:/usr/local/bin/rabbitmqadmin .

oops, on prince ^

this is not the same version of rabbitmq(admin)

ok, same error persists

see pm

lolfuss.

_lucifer: ^^

👏

ruaok: i am trying to figure why it went on looking way back in the past for listens. ideally it should go from the start of the range to its end. which job was in the queue when you cleared it?

it was a stats job that was working.

I forget the exact one, but we can guess that it was the first one running according to the daily crontab

👍

the sentry stack trace is sparticularly unhelpful :(

zas: https://www.cloudflare.com/galileo/

what do you think?

ruaok: i am unable to debug the issue using the info present in sentry. is it possible to view the logs of the request conusmer before it was restarted and also is it fine if i change spark logging level for sentry to debug?

sorry no, in order to free diskspace, I purged old containers. :(

no i mean the logs of the container you started after that but before clearing the queue

one sec. let me finish this €66,000 task real quick.

sure

ruaok: dunno if you saw the mail to (I assume) modbot?

reosarevok: see about 10 lines above. :)

Oh

I guess you did :D

can one view logs of a container that is now stopped? anyone know?

yes

correction, its been deleted

it not longer appears in docker ps --all

ah, in that case no, depending on the logging configuration

it sounds like you use file, in which case it got deleted with the container

yeah.

_lucifer: I suppose we can reproduce it sometime. once things calm down, we can reset the cluster again and try to generate stats.

yeah, sure. let's change the logging level to debug before that.

that should show up some more useful data in sentry.

what it did appear to be doing was to look for data and when it found none, the exit condition was never reached.

> now trying to erase listens from the medival times.

Yiikes.

ruaok: figured it out. `get_latest_listens_ts` is the culprit and every job recs or stats calls that.

we use it to find the `to_date` of the time range