14:10 PM
iliekcomputers
not sure why rabbitmqadmin wouldn't work though
2020-07-24 20612, 2020
14:11 PM
pristine___
What are these Bigquery jobs. I thought we don't use it anymore 🤷‍♀️
2020-07-24 20631, 2020
14:11 PM
iliekcomputers
rabbitmqctl purge_queue spark_request -p /listenbrainz
2020-07-24 20645, 2020
14:11 PM
iliekcomputers
pristine___: đź‘» ghost jobs
2020-07-24 20615, 2020
14:12 PM
ruaok
2020-07-24 20619, 2020
14:12 PM
ruaok
uhm, what?
2020-07-24 20658, 2020
14:12 PM
djwhitey joined the channel
2020-07-24 20607, 2020
14:13 PM
iliekcomputers
🤔
2020-07-24 20635, 2020
14:13 PM
sumedh joined the channel
2020-07-24 20622, 2020
14:14 PM
iliekcomputers
there's gonna be a new message at in around a min
2020-07-24 20635, 2020
14:14 PM
iliekcomputers
we have jobs every 15 mins starting 12 UTC
2020-07-24 20653, 2020
14:14 PM
iliekcomputers
not sure where the 8th message came from
2020-07-24 20617, 2020
14:15 PM
ruaok
> DEBUG in request_consumer: Pushing result to RabbitMQ...
2020-07-24 20627, 2020
14:15 PM
ruaok
still there. not sure if it is still doing stuff or got stuck
2020-07-24 20647, 2020
14:15 PM
ruaok
one python process using 100% CPU
2020-07-24 20618, 2020
14:16 PM
pristine___
Jobs every 15 minutes. Where can I look what these jobs are and how it all works?
2020-07-24 20619, 2020
14:16 PM
iliekcomputers
yeah, it is. spark does lazy evaluation, so it only runs the query when the data is needed, which is why logs aren't always the most accurate
2020-07-24 20638, 2020
14:16 PM
iliekcomputers
2020-07-24 20620, 2020
14:17 PM
pristine___
Woooo. Have not visited the script for long. It has become kinda huge <3
2020-07-24 20658, 2020
14:17 PM
ruaok
biggest crontab I've ever seen
2020-07-24 20620, 2020
14:18 PM
pristine___
Nice
2020-07-24 20654, 2020
14:18 PM
pristine___
> there's gonna be a new message at in around a min
2020-07-24 20658, 2020
14:18 PM
iliekcomputers
we'll probably consolidate it to send stats requests for all time ranges in one request.
2020-07-24 20601, 2020
14:19 PM
pristine___
Where these messages will show up?
2020-07-24 20615, 2020
14:19 PM
pristine___
E mail?
2020-07-24 20621, 2020
14:19 PM
ruaok
spark -request is still at 8
2020-07-24 20621, 2020
14:19 PM
iliekcomputers
but the granularity is helpful right now for debugging.
2020-07-24 20611, 2020
14:22 PM
iliekcomputers
rabbitmqderp probably
2020-07-24 20634, 2020
14:22 PM
iliekcomputers
the cronjob is at 14:15 UTC
2020-07-24 20642, 2020
14:22 PM
iliekcomputers
not sure what's up
2020-07-24 20657, 2020
14:22 PM
iliekcomputers
oh wait
2020-07-24 20658, 2020
14:22 PM
iliekcomputers
sorry
2020-07-24 20658, 2020
14:22 PM
ruaok
could the request have been early?
2020-07-24 20604, 2020
14:23 PM
iliekcomputers
it was at 14:10, my bad
2020-07-24 20612, 2020
14:23 PM
iliekcomputers
2020-07-24 20608, 2020
14:24 PM
iliekcomputers
so the message increase makes sense.
2020-07-24 20617, 2020
14:24 PM
iliekcomputers
what doesn't make sense is why it's not getting purged
2020-07-24 20635, 2020
14:24 PM
iliekcomputers
>Where these messages will show up?
2020-07-24 20644, 2020
14:24 PM
iliekcomputers
just in request consumer.
2020-07-24 20614, 2020
14:25 PM
pristine___
Thanks
2020-07-24 20642, 2020
14:26 PM
ruaok
9 items now.
2020-07-24 20609, 2020
14:27 PM
ruaok
I wonder if the request consumer has read the messages, but not acked them.
2020-07-24 20622, 2020
14:27 PM
ruaok
if bet if we kill the consumer the queue would get purged, but that may not be smart.
2020-07-24 20600, 2020
14:28 PM
iliekcomputers
ruaok: shouldn't be a problem, feel free
2020-07-24 20612, 2020
14:28 PM
ruaok
YOLO!
2020-07-24 20637, 2020
14:28 PM
iliekcomputers
it's not like we have some SLA on stats anyways :P
2020-07-24 20615, 2020
14:29 PM
ruaok
ok, stopped. queue was not empty, purged again (via rabbitmqadmin) and its now empty, leader is quiet.
2020-07-24 20621, 2020
14:29 PM
ruaok starts new
2020-07-24 20628, 2020
14:29 PM
iliekcomputers
👍🏽
2020-07-24 20634, 2020
14:29 PM
sumedh has quit
2020-07-24 20607, 2020
14:31 PM
ruaok
when i start the consuer, without having sent any messages, I lget this in the logs:
2020-07-24 20610, 2020
14:31 PM
ruaok
2020-07-24 20621, 2020
14:31 PM
ruaok
and now back to "[2020-07-24 14:31:09,664] DEBUG in request_consumer: Pushing result to RabbitMQ..."
2020-07-24 20637, 2020
14:31 PM
ruaok
thoughts?
2020-07-24 20615, 2020
14:32 PM
iliekcomputers
the logs mean it's processing some request...
2020-07-24 20633, 2020
14:32 PM
iliekcomputers
we're using rabbitmq over the internet. could that cause lag or something?
2020-07-24 20647, 2020
14:35 PM
ruaok
I doubt it in this context.
2020-07-24 20650, 2020
14:36 PM
iliekcomputers
restart the consumer once more?
2020-07-24 20644, 2020
14:37 PM
ruaok
that's what I just did.
2020-07-24 20604, 2020
14:38 PM
ruaok
this time I ensured the queue was empty and reentered the requests with the 3 users (not all)
2020-07-24 20618, 2020
14:38 PM
ruaok
now the log says: `INFO in download: Downloading msid-mbid-mapping-with-matchable-20200630-525068.tar.bz2 from FTP...`
2020-07-24 20627, 2020
14:38 PM
ruaok
which is exactly as we are expecting.
2020-07-24 20631, 2020
14:38 PM
ruaok
woo, using hammer works.
2020-07-24 20632, 2020
14:38 PM
ruaok
grunt
2020-07-24 20625, 2020
14:43 PM
v6lur_ has quit
2020-07-24 20621, 2020
14:46 PM
BrainzGit
2020-07-24 20623, 2020
14:46 PM
BrainzBot
2020-07-24 20623, 2020
14:46 PM
BrainzBot
2020-07-24 20640, 2020
14:46 PM
ruaok
importing the mapping failed.
2020-07-24 20649, 2020
14:46 PM
ruaok
do you have access to sentry, pristine___ ?
2020-07-24 20600, 2020
14:47 PM
iliekcomputers
did the consumer die too?
2020-07-24 20606, 2020
14:47 PM
ruaok
> raise DumpInvalidException("Received SHA256 checksum doesn't match the calculated checksum, aborting.")
2020-07-24 20623, 2020
14:47 PM
ruaok
no, the consumer is fine.
2020-07-24 20658, 2020
14:47 PM
pristine___
ruaok: hmm.
2020-07-24 20659, 2020
14:47 PM
ruaok
let me download the dump and check the checksum
2020-07-24 20652, 2020
14:48 PM
pristine___
Okay.
2020-07-24 20659, 2020
14:48 PM
pristine___
Keep me posted :)
2020-07-24 20613, 2020
14:50 PM
ruaok
looks like the download somehow failed.
2020-07-24 20623, 2020
14:50 PM
ruaok
the checksums on williams match.
2020-07-24 20619, 2020
14:51 PM
iliekcomputers
tagging ishaanshah here, he wrote the sha256 checks
2020-07-24 20634, 2020
14:51 PM
pristine___
But the mappings were successfully downloaded and uploaded to HDFS. Dataframes were generated :)
2020-07-24 20635, 2020
14:51 PM
iliekcomputers
might know what could be going wrong
2020-07-24 20659, 2020
14:51 PM
ruaok
dataframes are being generated with old data then.
2020-07-24 20607, 2020
14:52 PM
ruaok
maybe the redownload was not necssary?
2020-07-24 20616, 2020
14:52 PM
ishaanshah
ruaok: can you re-download it once
2020-07-24 20619, 2020
14:52 PM
ruaok
should I request another download and see if it passes this time?
2020-07-24 20624, 2020
14:52 PM
ruaok
:)
2020-07-24 20636, 2020
14:52 PM
pristine___
> maybe the redownload was not necssary?
2020-07-24 20636, 2020
14:52 PM
ishaanshah
I am pretty sure the sha 256 code is working as expected
2020-07-24 20640, 2020
14:52 PM
pristine___
No they aren't
2020-07-24 20609, 2020
14:53 PM
ishaanshah
Coz it works for importing data dump everyday
2020-07-24 20612, 2020
14:53 PM
ruaok
oh, perfect timing, the other requests just finished. new download started.
2020-07-24 20628, 2020
14:53 PM
ruaok
ishaanshah: pristine___ might be good to print which sha256 was generated in the error message.
2020-07-24 20639, 2020
14:53 PM
pristine___
Because we have changed the mapping schema. If they were being generated using old mapping, dataframes would error out.
2020-07-24 20644, 2020
14:53 PM
ruaok
it won't help in this case, but it might in future cases
2020-07-24 20650, 2020
14:53 PM
pristine___
So we know mappings were downloaded and unloaded
2020-07-24 20611, 2020
14:54 PM
ishaanshah
ruaok: makes sense, I will update it
2020-07-24 20614, 2020
14:54 PM
ruaok
2020-07-24 20627, 2020
14:54 PM
ruaok
ishaanshah: thanks
2020-07-24 20644, 2020
14:54 PM
ruaok
pristine___: I dont think they did get updated. looks very much like something aborted.
2020-07-24 20631, 2020
14:55 PM
pristine___
that is weird.
2020-07-24 20640, 2020
14:55 PM
iliekcomputers
ishaanshah: should we add a flag to the download function and only check integrity for listen dumps for now
2020-07-24 20641, 2020
14:55 PM
pristine___
wait
2020-07-24 20655, 2020
14:55 PM
pristine___
ruaok: did you pulled the latest changes from master?
2020-07-24 20650, 2020
14:56 PM
ruaok
request consumer is at a2a025f2b969a and has no newer changes.
2020-07-24 20648, 2020
14:57 PM
iliekcomputers
ishaanshah: the sha256 file in the mapping dir has 2 empty lines at the end of it, could that cause the equality check to fail?
2020-07-24 20613, 2020
14:59 PM
pristine___
ruaok: I mean the PRs we merged today
2020-07-24 20623, 2020
14:59 PM
pristine___
those changes shuld be there
2020-07-24 20603, 2020
15:00 PM
ruaok
a2a025f2b969a is the HEAD commit on listenbrainz-server
2020-07-24 20636, 2020
15:00 PM
ruaok
and all of the PRs have been merged, so I would conclude that the latest code is running.
2020-07-24 20647, 2020
15:00 PM
ruaok
unless there is a build step that I do not know about.
2020-07-24 20605, 2020
15:01 PM
ruaok
but the code get munged into a zip, so build shouldn't be necessary.
2020-07-24 20614, 2020
15:01 PM
pristine___
hmm...it is weird then.
2020-07-24 20615, 2020
15:01 PM
ruaok
should I go edit the sha256 file?
2020-07-24 20630, 2020
15:01 PM
pristine___
Let me see the candidate html first.
2020-07-24 20633, 2020
15:01 PM
ruaok
sha256 died again.
2020-07-24 20653, 2020
15:03 PM
ruaok
iliekcomputers: I don't see extra lines in any of the sha256 files.
2020-07-24 20653, 2020
15:03 PM
iliekcomputers
ruaok: couldn't hurt to try to edit the sha256 file
2020-07-24 20625, 2020
15:04 PM
iliekcomputers
you sure, it has 2 newlines for me at the end of it
2020-07-24 20627, 2020
15:04 PM
iliekcomputers
?
2020-07-24 20639, 2020
15:04 PM
ruaok
which exact file?
2020-07-24 20649, 2020
15:04 PM
iliekcomputers
2020-07-24 20624, 2020
15:05 PM
iliekcomputers
ah!
2020-07-24 20635, 2020
15:05 PM
iliekcomputers
2020-07-24 20651, 2020
15:05 PM
iliekcomputers
have the filename in the same line, the code doesn't expect that, just the sha256
2020-07-24 20615, 2020
15:06 PM
ruaok
ahhh, ok.
2020-07-24 20654, 2020
15:06 PM
ruaok
fixed.
2020-07-24 20641, 2020
15:07 PM
ruaok
ishaanshah: can we please fix the checsum checker to be ok with filenames at the end? because that is the output of sha256sum...
2020-07-24 20647, 2020
15:07 PM
BrainzGit
2020-07-24 20602, 2020
15:09 PM
ruaok
ok, re-issued commands so that if the import works this time it does the rec stuff.
2020-07-24 20626, 2020
15:11 PM
pristine___
the candidate_sets html looks good. ruaok can you check if you have a playlist for similar artists?
2020-07-24 20616, 2020
15:15 PM
ishaanshah
iliekcomputers, ruaok will do
2020-07-24 20650, 2020
15:16 PM
iliekcomputers
thanks!
2020-07-24 20604, 2020
15:20 PM
ishaanshah
iliekcomputers: I will clip everything after a space, does that sound good?
2020-07-24 20622, 2020
15:20 PM
iliekcomputers
yeah, i think that's reasonable.
2020-07-24 20641, 2020
15:20 PM
iliekcomputers
maybe extract the logic into a function and add tests for it too
2020-07-24 20600, 2020
15:22 PM
iliekcomputers
ruaok: the dumps doc changes are on your list, right?
2020-07-24 20624, 2020
15:25 PM
ruaok
Yes
2020-07-24 20635, 2020
15:25 PM
iliekcomputers
cool, thanks.
2020-07-24 20646, 2020
15:25 PM
pristine___
I checked similar artist playlist for shivam-kapila iliekcomputers and ruaok . They were generated. So we crossed one milestone. lemme know how much do you find it useful
2020-07-24 20652, 2020
15:26 PM
shivam-kapila
Someone called me?
2020-07-24 20659, 2020
15:26 PM
shivam-kapila
Oo new playlist
2020-07-24 20608, 2020
15:27 PM
iliekcomputers
alastairp: any chance we can chat today about the `implicit` stuff?
2020-07-24 20641, 2020
15:27 PM
shivam-kapila
ruaok: hi. How do you generate the HTML from cf rec mbids
2020-07-24 20647, 2020
15:27 PM
shivam-kapila
Troi?
2020-07-24 20604, 2020
15:28 PM
ruaok
Yes.
2020-07-24 20622, 2020
15:28 PM
ruaok
If you follow the instructions in the readme you can do the same.
2020-07-24 20633, 2020
15:28 PM
shivam-kapila
Noice
2020-07-24 20649, 2020
15:28 PM
shivam-kapila goes to get a new playlist
2020-07-24 20653, 2020
15:28 PM
ruaok
The example script listed need no local data sources.
