12:05 PM
pristine___
Yup.
2020-07-16 19828, 2020
12:21 PM
jmp_music joined the channel
2020-07-16 19820, 2020
12:31 PM
pristine___
Df creation is taking longer than I expected. 25 min up still no email :( but yeah data used this time (180 days) is double as compared to last time (60 days)
2020-07-16 19811, 2020
12:51 PM
d4rkie joined the channel
2020-07-16 19813, 2020
12:51 PM
Nyanko-sensei has quit
2020-07-16 19818, 2020
12:56 PM
iliekcomputers
pristine___: i think one of the stats jobs might be running first.
2020-07-16 19855, 2020
13:08 PM
pristine___
Oooo. Forgot about that. Thanks.
2020-07-16 19813, 2020
13:18 PM
iliekcomputers
>During the incident, a shared database table’s auto-incrementing ID column exceeded the size that can be represented by the MySQL Integer type (Rails int(11)): 2147483647. When we attempted to insert larger integers into the column, the database rejected the value and Rails raised an ActiveModel::RangeError, which resulted in 500s from our API endpoint.
2020-07-16 19827, 2020
13:18 PM
iliekcomputers
2020-07-16 19831, 2020
13:18 PM
iliekcomputers
can this happen with postgres?
2020-07-16 19812, 2020
13:19 PM
iliekcomputers
also 3/4 of their incidents are mysql
2020-07-16 19830, 2020
13:28 PM
iliekcomputers
looks like serial is int so yeah it can happen to postgres as well
2020-07-16 19857, 2020
13:29 PM
alastairp
iliekcomputers: yes, this is why you have to vacuum
2020-07-16 19843, 2020
13:31 PM
alastairp
bigserial is 64 bits, but it can still roll over
2020-07-16 19836, 2020
13:32 PM
alastairp
and I've seen reports of bugs where postgres ran out of tuples because a vacuum wasn't happening
2020-07-16 19814, 2020
13:38 PM
yvanzo
2020-07-16 19825, 2020
13:48 PM
yvanzo
zas: everything is down, but no error is shown for jira in cloud console ^
2020-07-16 19817, 2020
13:49 PM
yvanzo
zas: never mind, it seems to be back :)
2020-07-16 19814, 2020
14:39 PM
v6lur has quit
2020-07-16 19856, 2020
14:55 PM
milkii has quit
2020-07-16 19857, 2020
14:55 PM
gr0uch0mars has quit
2020-07-16 19826, 2020
14:57 PM
milkii joined the channel
2020-07-16 19826, 2020
14:57 PM
gr0uch0mars joined the channel
2020-07-16 19840, 2020
14:57 PM
MusicbrainzB0T has quit
2020-07-16 19854, 2020
14:57 PM
MusicbrainzB0T joined the channel
2020-07-16 19836, 2020
15:27 PM
yvanzo
updating test.mb.o
2020-07-16 19827, 2020
15:45 PM
ruaok
> also 3/4 of their incidents are mysql
2020-07-16 19844, 2020
15:45 PM
ruaok
yeah, people still *love* mysql and think PG is... weird.
2020-07-16 19851, 2020
15:45 PM
ruaok
fine, enjoy your downtimes.
2020-07-16 19802, 2020
15:46 PM
ruaok
any progress, pristine___ ?
2020-07-16 19831, 2020
15:46 PM
pristine___
Nah. No emails :(
2020-07-16 19832, 2020
15:47 PM
pristine___
Not sure how much time the stats job gonna take if there is any.
2020-07-16 19813, 2020
15:51 PM
ruaok
new leader is idle, so no idea what is happening.
2020-07-16 19805, 2020
15:52 PM
pristine___
:(
2020-07-16 19828, 2020
15:52 PM
pristine___
Is there any way to ensure that the request was sent
2020-07-16 19830, 2020
15:52 PM
pristine___
?
2020-07-16 19825, 2020
15:55 PM
iliekcomputers
`docker logs -f spark-request-consumer`
2020-07-16 19836, 2020
15:55 PM
iliekcomputers
and see if there's anything in sentry
2020-07-16 19844, 2020
15:59 PM
sumedh joined the channel
2020-07-16 19856, 2020
15:59 PM
sumedh has quit
2020-07-16 19814, 2020
16:00 PM
ishaanshah
iliekcomputers: Hi
2020-07-16 19848, 2020
16:00 PM
iliekcomputers
ishaanshah: hey
2020-07-16 19813, 2020
16:01 PM
ruaok
2020-07-16 19821, 2020
16:01 PM
ruaok
is that bad? 🤣
2020-07-16 19823, 2020
16:01 PM
iliekcomputers
:)
2020-07-16 19844, 2020
16:01 PM
ishaanshah
I worked on the improvement on charts page today
2020-07-16 19850, 2020
16:01 PM
iliekcomputers
did we keep a backup of the original config.py by any chance
2020-07-16 19800, 2020
16:02 PM
iliekcomputers
ishaanshah: awesome!
2020-07-16 19814, 2020
16:02 PM
ishaanshah
I will open a Pr for that tomorrew
2020-07-16 19819, 2020
16:02 PM
iliekcomputers
sounds good.
2020-07-16 19828, 2020
16:02 PM
ishaanshah
Thats all I did, today
2020-07-16 19843, 2020
16:02 PM
iliekcomputers
i fixed up the smallish things with the reports page, to get some load off you.
2020-07-16 19855, 2020
16:02 PM
iliekcomputers
other than that, i have nothing either.
2020-07-16 19856, 2020
16:02 PM
ishaanshah
Yeah saw that, thanks :)
2020-07-16 19811, 2020
16:03 PM
iliekcomputers
anything blocking you?
2020-07-16 19819, 2020
16:03 PM
ishaanshah
No nothing at all
2020-07-16 19823, 2020
16:03 PM
iliekcomputers
cool.
2020-07-16 19832, 2020
16:03 PM
iliekcomputers
nice quick meeting then. :)
2020-07-16 19844, 2020
16:03 PM
ishaanshah
yep :D
2020-07-16 19850, 2020
16:03 PM
ishaanshah
have a nice day!
2020-07-16 19854, 2020
16:03 PM
iliekcomputers
you too!
2020-07-16 19804, 2020
16:09 PM
ruaok
iliekcomputers: got a sec to debug the request consumer?
2020-07-16 19818, 2020
16:09 PM
iliekcomputers
yeah. sure
2020-07-16 19821, 2020
16:09 PM
ruaok
config.py and config.py.sample are identical as per pristine___ 's instuctions.
2020-07-16 19855, 2020
16:09 PM
iliekcomputers
:/
2020-07-16 19820, 2020
16:10 PM
iliekcomputers
which means that the config.py file doesn't have prod rabbitmq host port etc
2020-07-16 19844, 2020
16:10 PM
ruaok
did I just inadvertendly blow away critical config bits?
2020-07-16 19858, 2020
16:10 PM
iliekcomputers
hmm, probably.
2020-07-16 19815, 2020
16:11 PM
iliekcomputers
shouldn't be hard to recreate. i'll check them into `docker-server-configs` this time, i guess
2020-07-16 19817, 2020
16:11 PM
ruaok really should have setup is own setup.
2020-07-16 19823, 2020
16:11 PM
ruaok
good plan.
2020-07-16 19851, 2020
16:11 PM
iliekcomputers
👍🏽
2020-07-16 19828, 2020
16:12 PM
pristine___
My bad. There are some config values that should have been there which were missing earlier and caused the error :(
2020-07-16 19824, 2020
16:16 PM
pristine___
2020-07-16 19843, 2020
16:16 PM
pristine___
iliekcomputers: these configs should be in config.py
2020-07-16 19851, 2020
16:16 PM
iliekcomputers
i'll actually just create the request_consumer user while i'm at it.
2020-07-16 19803, 2020
16:17 PM
iliekcomputers
pristine___: thanks.
2020-07-16 19827, 2020
16:27 PM
iliekcomputers
pristine___: is there a difference between the gist you linked and config.py.sample in master?
2020-07-16 19844, 2020
16:31 PM
pristine___
Na. Just gave you recommendation engine specific commands
2020-07-16 19853, 2020
16:31 PM
pristine___
Configs*
2020-07-16 19802, 2020
16:32 PM
iliekcomputers
great, thanks.
2020-07-16 19807, 2020
16:32 PM
iliekcomputers
do you see the errors in sentry
2020-07-16 19807, 2020
16:32 PM
iliekcomputers
?
2020-07-16 19842, 2020
16:32 PM
iliekcomputers
2020-07-16 19845, 2020
16:35 PM
pristine___
No email from sentry. But can you check if html_files folder exists?
2020-07-16 19859, 2020
16:35 PM
iliekcomputers
2020-07-16 19834, 2020
16:36 PM
iliekcomputers
LISTENBRAINZ-B9
2020-07-16 19856, 2020
16:36 PM
iliekcomputers
where should the html_files folder be
2020-07-16 19856, 2020
16:36 PM
iliekcomputers
?
2020-07-16 19825, 2020
16:37 PM
pristine___
listenbrainz_spark/recommendations/
2020-07-16 19857, 2020
16:37 PM
iliekcomputers
2020-07-16 19837, 2020
16:39 PM
iliekcomputers
i'll just run the commands in order again
2020-07-16 19835, 2020
16:40 PM
pristine___
Should be a folder in there. I wonder why it doesn't exist. It was there when I checked last time.
2020-07-16 19837, 2020
16:40 PM
pristine___
Cool
2020-07-16 19813, 2020
16:41 PM
shivam-kapila
I see `cannot save similar artist empty candidate sets` and one `file doesnt exist`
2020-07-16 19821, 2020
16:42 PM
iliekcomputers
shivam-kapila: yeah, LISTENBRAINZ-B9 and LISTENBRAINZ-BA
2020-07-16 19850, 2020
16:42 PM
pristine___
I will open a ticket for the folder.
2020-07-16 19850, 2020
16:46 PM
iliekcomputers
k, it died again, same error
2020-07-16 19805, 2020
16:47 PM
iliekcomputers
let's purge the queue, fix the error and try again later.
2020-07-16 19827, 2020
16:47 PM
pristine___
Hmmm.
2020-07-16 19839, 2020
16:48 PM
pristine___
No mail. No. I will login and check
2020-07-16 19849, 2020
16:48 PM
pristine___
For sentry
2020-07-16 19856, 2020
16:51 PM
iliekcomputers
It's in the listenbrainz project, not beta
2020-07-16 19846, 2020
16:54 PM
killme has quit
2020-07-16 19813, 2020
16:56 PM
killme joined the channel
2020-07-16 19857, 2020
17:07 PM
pristine___
iliekcomputers: were the requests going theough `request_consumer` account on leader ?
2020-07-16 19805, 2020
17:08 PM
pristine___
Through*
2020-07-16 19806, 2020
17:09 PM
iliekcomputers
Yes
2020-07-16 19845, 2020
17:09 PM
pristine___
i did this `ls /home/request_consumer/listenbrainz-server/listenbrainz_spark/recommendations`
2020-07-16 19802, 2020
17:10 PM
pristine___
couldn't find `html_files`
2020-07-16 19800, 2020
17:11 PM
pristine___
I thought you created the folder earlier.
2020-07-16 19817, 2020
17:12 PM
iliekcomputers
I didnt
2020-07-16 19825, 2020
17:13 PM
iliekcomputers
Was I supposed to?
2020-07-16 19844, 2020
17:13 PM
pristine___
yes.
2020-07-16 19804, 2020
17:15 PM
iliekcomputers
Can you write up a doc on the exact list of commands that need to be run?
2020-07-16 19804, 2020
17:15 PM
BrainzGit
2020-07-16 19838, 2020
17:15 PM
iliekcomputers
I didn't see any message about *creating* a folder myself.
2020-07-16 19852, 2020
17:15 PM
iliekcomputers
Generally I'd assume this stuff would be done by the code.
2020-07-16 19812, 2020
17:16 PM
pristine___
right. the folder isn't there, i wonder why.
2020-07-16 19826, 2020
17:16 PM
pristine___
> Generally I'd assume this stuff would be done by the code.
2020-07-16 19807, 2020
17:18 PM
dseomn1 has quit
2020-07-16 19819, 2020
17:18 PM
iliekcomputers
Let's try to keep the deploy process more clear next time
2020-07-16 19829, 2020
17:18 PM
pristine___
right. this need not be done manually. I will check why the folder isnt't there. But for now if you can create the folder, the scripts will work. Otherwise I will open a PR and once it is merged you can run the commands I gave to you.
2020-07-16 19812, 2020
17:19 PM
iliekcomputers
Write a Google doc about what files need to be updated, what folders need to be created etc and what commands need to be run, in the exact order that it needs to happen
2020-07-16 19820, 2020
17:19 PM
pristine___
The process was clear, creating a folder wasn't a part of it. I will open a PR. let us leave it for now.
2020-07-16 19836, 2020
17:19 PM
iliekcomputers
k
2020-07-16 19827, 2020
17:26 PM
jmp_music has quit
2020-07-16 19815, 2020
17:33 PM
ruaok
the mogrifier has been mogrified finally. gonna run the test on gaga next.
2020-07-16 19802, 2020
17:35 PM
ruaok
I've done this
2020-07-16 19802, 2020
17:35 PM
ruaok
2020-07-16 19830, 2020
17:45 PM
ruaok
iliekcomputers: create-dump.sh is now running
2020-07-16 19834, 2020
17:47 PM
reosarevok
bitmap: if you have some time you could check
https://github.com/metabrainz/musicbrainz-server/… so we can release that bug fix :) Reviews for any small bug patches appreciated really, just that I didn't even remember that patch until yvanzo reviewed it now :D
2020-07-16 19815, 2020
17:48 PM
bitmap
sorry 'bout that, I must've overlooked it
2020-07-16 19818, 2020
17:48 PM
bitmap
approved now
2020-07-16 19802, 2020
17:50 PM
reosarevok
np!
2020-07-16 19811, 2020
17:50 PM
BrainzGit
2020-07-16 19812, 2020
17:50 PM
BrainzBot
2020-07-16 19818, 2020
17:53 PM
BrainzGit
2020-07-16 19812, 2020
17:57 PM
iliekcomputers
ruaok: nice! Let's try to import it tomorrow.
2020-07-16 19830, 2020
18:07 PM
BrainzGit
2020-07-16 19825, 2020
18:22 PM
BrainzGit
2020-07-16 19827, 2020
18:22 PM
BrainzBot
2020-07-16 19829, 2020
18:31 PM
BrainzGit
2020-07-16 19830, 2020
18:31 PM
BrainzBot
2020-07-16 19830, 2020
18:50 PM
BrainzGit
2020-07-16 19832, 2020
18:50 PM
BrainzBot
2020-07-16 19811, 2020
18:56 PM
reosarevok
Another small bugfix ^
2020-07-16 19817, 2020
19:06 PM
rdswift
Is MBS-10963 going to be difficult to do? It's not a show stopper by any stretch, but it is annoying.
2020-07-16 19818, 2020
19:06 PM
BrainzBot
2020-07-16 19853, 2020
19:06 PM
reosarevok
bitmap: is your code directly reusable on the release editor? ^
