#metabrainz

yvanzo setup-amqp-triggers: cannot uninstall: file '/tmp/indexer-sql' does not exist

ruaok: zas: `wget 10.2.2.42:62673` fails from kiss but works from boingo. further, on running `ufw status verbose` i see that boingo (and prince) have the default `allow (routed)` but on kiss it is ` deny (routed)`.

yvanzo: oh ok I tried that. I went to (https://sentry.metabrainz.org/settings/metabrai...) i see a JIRA Atlassian Connect option there (the only jira related option) and I enabled that. However, when I go to a ticket in the project, and I am still shown a Setup issue tracking button which redirects me to organization wide integration page.

yvanzo: i joined the mb team on sentry to see how it configures JIRA. I see it has a JIRA plugin option, this option in not available in the LB project. don't know why it is so.

yvanzo: i manually modified MB project's url for JIRA integration and to get at the JIRA integration page for LB and configured it there. there's probably some bug in sentry's UI. anyways works now thanks!

lucifer: yes, I'll finish this after coffee

nelgin: that happens when the 'musicbrainz' container has been recreated in the meantime, to solve this: admin/setup-amqp-triggers clean && admin/setup-amqp-triggers install

nelgin: then: admin/setup-amqp-triggers uninstall && admin/configure rm live-indexing-search

nelgin: and finally: sudo docker-compose up -d # to restart indexer without live mode.

lucifer: cool, please update the sysadmin wiki with your findings :)

Added SEARCH-653 to work on improving uninstall scripts.

SEARCH-653: Improve Drop SQL scripts to support previous versions too https://tickets.metabrainz.org/browse/SEARCH-653

rebooting kiss

thanks zas!

hi. how are we doing?

lucifer: see if lb containers work now on kiss

lucifer: in case you weren't notified, officebrainz is getting a new internet connection some time between midday-3pm (starting in about 2h)

I'm around from home, but in a meeting during those hours

yvanzo: sure. but i just found i created a big mess. while trying to leave mb team i accidentally deleted it 🤦. extremely sorry. how can i help clean this up?

zas: on it

alastairp: ah yes, i knew about it but wasn't sure of the timing. thanks for reminding.

ruaok: zas: working now :D similar users successfully requested.

good

lucifer: great. can we slowly move other services back?

yay!

i think yes!

how about as a test we spin up test or beta?

let's request an inc to make sure dumps work

that'll verify all of the service discovery

*inc dump

sure, i'll start beta containers

but we need redis for that, alastairp

redis is a service, so the copy on boingo _should_ be available from kiss

will the boingo one work if i start web beta on kiss?

ah ok makes sense

beta web up

sweet, so that confirms that service discovery to other hosts for the website works, and that routing from the gateway works

looks fine.

sounds good to me

ruaok: shall we do the redis dance again?

should i request the incremental dump?

lucifer: lets do one thing at a time.

lets move things back and then request an incremental.

alastairp: yes, lets.

makes sense, 👍

OK, give me 3m to prepare

I'm not even logged in yet. still in coffee mode. lucifer can you work with alastairp?

just a matter of starting/stopping containers.

yup

thx

shut down web boingo; save redis boingo; copy to kiss; start up redis kiss; start up web kiss

lucifer: if you can take care of web, I'll do redis. I'll tell you when I'm ready for you to take down web, then you tell me when it's done

sure, i'll take care of web.

I'll tweet.

i should also take down beta web, right?

theoretically new startup scripts should take it down and send error notifications while the redis service is down ;)

but yes, we could take it down to be sure

let's test the former then, if doesn't work out, i'll bring beta down manually

lucifer: ok, I'm ready

:+!

alastairp: web stopped

redis running on kiss with data

go ahead with startup

web up on kis

Hi there. Me again :-) ... is there any option to show/log output from WS code (in perl) so i can check for errors or debug messages ?

lucifer: btw, this is how I migrate the redis datbase from one machine to another. Maybe we should document it somewhere for future use if needed

alastairp: yup, makes sense to add this to syswiki.

alastairp, also i hadn't touched beta during the process so we can look at logs and sentry to find if the improvements work as expected.

looks like it

we need to add some more info to the sentry messages. Exit status 11 is "service not found"

yeah makes sense.

it'd be great to try and pass the missing information to the sentry message. unfortunately the missing information is identified in run-lb-command (part of run->consul) and the error message is sent in `finish`

and there's no way to pass data from one to another. what we could do is re-do the grep in `finish`, because the config file should still be there

makes sense, sure let's grep in finish as well.

should we move back other services now?

yes, let's finish moving everything over

can you take care of that?

sure will do

thanks!

ruaok: I used a trick for grafana vs kiss interface names, I added a specific processor in telegraf to rename external and internal interface metrics so it's consistent with the rest. In kiss dashboard you now have network stats

cool

all lb containers on kiss now.

😘 thanks

let's do incremental dump now? :)

one moment first

I just want to have a look over the permissions and directories in the dumps volume to check that it'll run as expected

👍

docker ps on kiss now shows ipv6 port mappings as well

I might make an alias for ps that has a custom --format

with less information

alastairp: shall we copy some of the data from the storage box back to FTP/backup volumes before generating the incremental dump?

I'm also itching to dump the storage box.

sure, good idea

mounted at /mnt/storagebox/

no 'lbdumps' user on kiss, I used uid=900,gid=900 in mount ops

good plan

just logged out of root, so the mount command should be in history now

I installed cifs-utils package for mount.cifs

just installed tmux too

this is what I was worried about - there's still some permissions stuff going on here. We'll have to manually change the permissions of this folder for now. I know that it's possible to make docker automatically change the owner of this folder on first startup (the redis image does it), but I don't know exactly how

s/permissions/owner/

chown?

right, but where? If you do it in the Dockerfile then that information is in the image

service file?

no, it should happen during docker run

if you specify docker run with --volume, then on the first startup docker will chown the volume's _data directory

as an example, try this:

docker volume create --driver local --name test-redis-volume

let's copy stuff off the storage box by manually correcting permissions if needed and then move this all to the root user?

instead of a separate lbdumps user

docker run --volume "test-redis-volume:/data" --name redis-test redis:5.0.3

I think so.

the lbdumps user was created because of the storage box.

then look at /var/lib/docker/voumes/test-redis-volume/_data - its owner will be 999, not root

we shouldn't have to do anything special anymore.

if we're OK to keep everything as root, then I guess we can do that

everything else runs as root anyways, so why not dumps as well

agreed.

ok, let's do that then. what do we need to adjust here? remove the users from Dockerfile, change the crontab to run as root, move the locations of backup and ftp dirs (in Dockerfile and in consul config and in startup scripts [volume mount location]) ?

yeah, i think so that sums it up.

alastairp: can you take a look at this https://github.com/metabrainz/listenbrainz-serv... ? i'll open a new PR targetting this branch to remove lbdumps user meanwhile.

hmm. the compat api seems to be borked somehow.

see nagios messages.

lucifer: it looks OK, but I'll have another look. I don't have much time today to work on MeB stuff but let me try and fit it in

ruaok: service container is up, and seems to be running. odd

nginx issue maybe??

api-compat-prod logs are empty which either means nobody is using it or it is receiving no traffic?