hi nelgin, I can have a look at it, please PM for details.
2021-04-30 12047, 2021
nelgin
Hi. Sure.
2021-04-30 12037, 2021
nelgin
details sent. Thank you.
2021-04-30 12026, 2021
yvanzo
Custom memory settings have been correctly applied, there is no issue there.
2021-04-30 12007, 2021
yvanzo
About missing queues: rabbitmq is creating a new database directory on startup, based on hostname.
2021-04-30 12039, 2021
yvanzo
That is a new behaviour for rabbitmq Docker image, compared to what it used to be.
2021-04-30 12057, 2021
yvanzo
That explains I could not reproduce the issue: I was still using previous image.
2021-04-30 12019, 2021
yvanzo
So, we have to set a fixed hostname for 'mq' service and that should solve it.
2021-04-30 12017, 2021
nelgin
ok, can I do that here to test?
2021-04-30 12036, 2021
nelgin
or not sure if you're able to push a fix?
2021-04-30 12050, 2021
v6lur joined the channel
2021-04-30 12005, 2021
yvanzo
added local/compose/fix-mq-persistence.yml
2021-04-30 12040, 2021
nelgin
OK, do I need to configure add that or did you do it?
2021-04-30 12024, 2021
nelgin does it anyway it can't hurt
2021-04-30 12049, 2021
nelgin
yvanzo ok, that fixed that problem I'm seeing the queues after the restart of docker-compose
2021-04-30 12011, 2021
nelgin
Running replication.sh and then I'll check the queue
2021-04-30 12001, 2021
yvanzo
I did add that, stopped live indexing, ran amqp_setup again, down everything, readd live indexing, and up everything again.
2021-04-30 12057, 2021
yvanzo
'admin/configure add' handle duplicates, so no worry about having done that more than once.
2021-04-30 12008, 2021
nelgin
Very good. It's working the queue right now. I forgot to increase the indexer.ini value. I'll do that once this has finished, and we'll see if the settings stick.
2021-04-30 12037, 2021
yvanzo
Feel free to stop/remove the indexer, update its configuration and start it again. RabbitMQ will keep storing reindex messages in the meantime, so that live indexing is able to resume.
yvanzo, bitmap^ seems to be a new regression, might not be worth rushing it into this next release since it's relatively minor, but let's get it for the next :)
2021-04-30 12017, 2021
zas
ruaok: there was an alert on cpu temp for gaga, the load at this time was higher than usual, but not that high, I suspect a cpu fan failure (surprise!) (and too bad we can't gather cpu fan speed on those machines). It runs mainly lb stuff. We should ask for a tech to look at it. Can it be stopped for few minutes? preferred time frame?
2021-04-30 12021, 2021
nelgin
yvanzo, I got 3 of these messages in my indexer log:
2021-04-30 12021, 2021
nelgin
Index limit exceeded. Entity: release, Total Rows: 210617, Extra Data: {'path': 'artist_credit', 'table_name': u'artist_credit', 'select_query': 'SELECT release_1.id AS release_1_id \nFROM musicbrainz.release AS release_1 JOIN musicbrainz.artist_credit ON musicbrainz.artist_credit.id = release_1.artist_credit \nWHERE musicbrainz.artist_credit.id = :id_1'}
2021-04-30 12026, 2021
ruaok
well, this pretty much means downtime for LB.
2021-04-30 12010, 2021
nelgin
search.failed | 3
2021-04-30 12056, 2021
ruaok
zas: I am responsible for the extra load, I can stop that.
2021-04-30 12013, 2021
zas
well, the extra load isn't the issue, it was something like 40% cpu, but the cpu temp raised > 95°C so very likely a cooling problem (=cpu fan, we are used to that now...)
2021-04-30 12035, 2021
ruaok
I'd rather not take the whole site down for the fan upgrade. I'm considering some code improvements to allow the timescale to be offline, without having to take LB down. that way we can still accept listen submissions, but not show listens on the site for the duration.
2021-04-30 12046, 2021
zas
that's not urgent, but the hardware has to be fixed at some time
2021-04-30 12022, 2021
ruaok
ok, I'll stop the process that puts on the extra load, and start working on this so we can make the fix early next week.
2021-04-30 12030, 2021
ruaok
how is that for a plan?
2021-04-30 12036, 2021
zas
ok for me :)
2021-04-30 12040, 2021
ruaok
kewl.
2021-04-30 12057, 2021
nelgin
Gets my vote ;)
2021-04-30 12036, 2021
nelgin
yvanzo, really need to sort this crontab problem out, if you have the time. It;s 4am here so I can't stop up much longer.
Mr_Monkey: please to pare down GSoC proposals soon, ok?
2021-04-30 12016, 2021
Mr_Monkey
Yes yes, I've been reading the proposals throughout the week, will make final decisions now
2021-04-30 12045, 2021
ruaok
can you start setting proposals that you are clearly not going to accept to "ignore"?
2021-04-30 12052, 2021
Mr_Monkey
Sure thing
2021-04-30 12055, 2021
ruaok
thx!
2021-04-30 12026, 2021
adhawkins has quit
2021-04-30 12008, 2021
adhawkins joined the channel
2021-04-30 12059, 2021
Mr_Monkey
ruaok: I had tried earlier but thought it was too early: I can't set proposals as ignored. Maybe I don't have rights?
2021-04-30 12007, 2021
nelgin
yvanzo did you add the crontab manually?
2021-04-30 12042, 2021
ruaok
I can't give perms for that.
2021-04-30 12053, 2021
ruaok
PM me the titles of proposals to mark as ignore.
2021-04-30 12008, 2021
_lucifer
only org admins can do that afaik (star or ignore proposals)
2021-04-30 12031, 2021
yvanzo
nelgin: it still works after restart, after stop/start, after down/up
2021-04-30 12041, 2021
yvanzo
no, I did not add it.
2021-04-30 12050, 2021
nelgin
Hmm, maybe I was being too hasty.
2021-04-30 12008, 2021
nelgin
Ok. I'm going to change it to replicate every hour then.
2021-04-30 12028, 2021
yvanzo
That is possible: it can take a few minutes before the container sets the crontab and starts cron.
2021-04-30 12001, 2021
_lucifer
I released an update for MuiscBrainz Android yesterday which is now live on Play Store with akshaaatt[m]'s bugfix for a long standing bug in the Tagger.
2021-04-30 12004, 2021
_lucifer
!m akshaaatt[m]
2021-04-30 12004, 2021
BrainzBot
You're doing good work, akshaaatt[m]!
2021-04-30 12017, 2021
_lucifer
shivam-kapila: want to give it a try? :)
2021-04-30 12011, 2021
ljunkie has quit
2021-04-30 12015, 2021
ruaok
outsidecontext: zas: is that picard GSoC proposal on your radar?
2021-04-30 12056, 2021
ljunkie joined the channel
2021-04-30 12000, 2021
nelgin
yvanzo I created local/compose/replication.cron and added the path into .env
and there's no crontab. Checked for the last 2 minutes.
2021-04-30 12012, 2021
alastairp
ruaok: I commented on the proposals relevant to me
2021-04-30 12002, 2021
outsidecontext
ruaok: nope, haven't seen it yet :( It's on my radar to look into the GSoC dashboard, though.I think I had hoped there would be some kind of notification if there is a proposal
2021-04-30 12013, 2021
nelgin
Interesting, as soon as I login to the bash shell, there it is.
2021-04-30 12039, 2021
nelgin
I don't get it, but it's 430am so I'm going to bed, but something's not right, going to restart the container and then check when I get up.
2021-04-30 12020, 2021
yvanzo
Ok, good night.
2021-04-30 12043, 2021
nelgin
meh, forgot to use -d when I started it. Oh well. It'll be ok.
2021-04-30 12004, 2021
_lucifer
ruaok: alastairp: regarding lb username change, till we do not support it i was thinking we could provide a script for users. like the user could download their export from their profile delete their listens, point the script to the exported json file and put their new username token etc. and submit their listens.
2021-04-30 12020, 2021
ruaok
hmm. we could do that. but how important is that for us right now? enough to write this script?
2021-04-30 12044, 2021
nelgin
Welp, the crontab is there already. I dunno, man. something definitely odd. I've been through this several times before, I'm sure. Right anyway, did you want to tinker with the vm any more, if not I'll shut the ssh port down for now.
2021-04-30 12009, 2021
zas
ruaok, outsidecontext: I read it yesterday (got no notification either), as is, not very detailed, but we don't have any other Picard-related proposal afaik, so I'm ok to co-mentor (with outsidecontext?). Is proposal's author hanging around (dhhruv) ?
2021-04-30 12041, 2021
alastairp
_lucifer: this should be something that we do in the backend. if someone requests it we can just perform an sql query, or write a quick migration script ourselves to do it
2021-04-30 12047, 2021
alastairp
without having to delete/reimport all of the content
2021-04-30 12003, 2021
alastairp
(although you raise a good point, perhaps this is an easier way of doing it without locking the database)
2021-04-30 12028, 2021
ruaok sets a preliminary slot request
2021-04-30 12040, 2021
ruaok
alastairp: did you see zas' comment about the fan replacement on gaga?
2021-04-30 12047, 2021
alastairp
I just did
2021-04-30 12059, 2021
alastairp
I agree with your proposal to make LB run without the database
2021-04-30 12005, 2021
ruaok
great.
2021-04-30 12023, 2021
alastairp
which is great, because it ties in nicely with our discussion on wednesday about being about to start up services without dependencies, too
2021-04-30 12024, 2021
ruaok
I'm wondering on how to go about doing that.
2021-04-30 12048, 2021
_lucifer
yeah there are multiple ways to do it, i was looking into the easiest one without affecting the backend. i am not sure about the urgency of this though, i raised this because of a support email reo asked me about.
2021-04-30 12038, 2021
ruaok
I guess it should just be automatic. If the listenstore is not available, then set a flag and return 503 for API calls that touch the listenstore.
2021-04-30 12009, 2021
alastairp
ruaok: I'm just saying the first thing that comes to mind: if no db configuration, set an empty listenstore or set it to None; then have a @decorator which checks if the listenstore is available and if not, returns an error early
2021-04-30 12029, 2021
ruaok
OH. great idea.
2021-04-30 12037, 2021
alastairp
that should work well for API endpoints. some website endpoints also query listenstore directly
2021-04-30 12043, 2021
alastairp
so that'll need some additional checks
2021-04-30 12049, 2021
ruaok
agreed.
2021-04-30 12015, 2021
ruaok
I'll start working on that in a bit.
2021-04-30 12029, 2021
ruaok
now that gsoc is somewhat tucked away.
2021-04-30 12047, 2021
yvanzo
nelgin: local/compose/ is supposed to contain .yml compose files, not crontab file.
Then I should be able to do sudo docker-compose up -d right?
2021-04-30 12014, 2021
yvanzo
yes
2021-04-30 12037, 2021
yvanzo
at least, 'crontab -l' currently returns the same content as this file.
2021-04-30 12050, 2021
nelgin
Yup, it's good.
2021-04-30 12048, 2021
outsidecontext
zas, rdswift: FYI, I rebased the 2.7.x branch and merged it into master. As we don't plan another 2.6.x release we can continue there (if we should need to do a 2.6.3 unexpectedly we'll have a release branch for that)
2021-04-30 12007, 2021
zas
ok, thanks
2021-04-30 12019, 2021
zas
outsidecontext: I sent a mail to Dhruv (about GSoC/Picard), asking him to contact us here (may be he's around, but I don't know under which nickname)
Mr_Monkey: got an error on beta while testing error handling of youtube player, the error didn't show up in sentry but console has some info
2021-04-30 12037, 2021
ruaok
_lucifer: I'm not having any luck getting spark-request-consumer on michael running again. the container is created, but not started. then it just hangs. any ideas?
2021-04-30 12055, 2021
_lucifer
ruaok: let me login and take a look.
2021-04-30 12056, 2021
zas
community.meb is back
2021-04-30 12010, 2021
ruaok
the dump has finished and I want to do a full import now. (even though we'll generate a new dump tomorrow, but hey)
2021-04-30 12013, 2021
ruaok
_lucifer: thanks
2021-04-30 12045, 2021
_lucifer
ruaok: request consumer is up.
2021-04-30 12059, 2021
_lucifer
you need to run to start it correctly ./docker/spark-new-cluster/start-request-consumer-container.sh
2021-04-30 12028, 2021
_lucifer
and ./docker/spark-new-cluster/stop-request-consumer-container.sh to stop and cleanup
Mr_Monkey, I was testing what happens if user externally revokes the youtube access. the server should handle it and probably respond with 401 or 403, i think?
