yvanzo setup-amqp-triggers: cannot uninstall: file '/tmp/indexer-sql' does not exist
2021-05-28 14833, 2021
Sophist_UK joined the channel
2021-05-28 14817, 2021
elomatreb has quit
2021-05-28 14825, 2021
anonranonnn joined the channel
2021-05-28 14815, 2021
anonranon_ has quit
2021-05-28 14849, 2021
akashgp0968 joined the channel
2021-05-28 14855, 2021
akashgp0968 has left the channel
2021-05-28 14845, 2021
lucifer
ruaok: zas: `wget 10.2.2.42:62673` fails from kiss but works from boingo. further, on running `ufw status verbose` i see that boingo (and prince) have the default `allow (routed)` but on kiss it is ` deny (routed)`.
2021-05-28 14805, 2021
thomasross has quit
2021-05-28 14856, 2021
lucifer
yvanzo: oh ok I tried that. I went to (https://sentry.metabrainz.org/settings/metabrainz…) i see a JIRA Atlassian Connect option there (the only jira related option) and I enabled that. However, when I go to a ticket in the project, and I am still shown a Setup issue tracking button which redirects me to organization wide integration page.
2021-05-28 14845, 2021
lucifer
yvanzo: i joined the mb team on sentry to see how it configures JIRA. I see it has a JIRA plugin option, this option in not available in the LB project. don't know why it is so.
2021-05-28 14820, 2021
lucifer
yvanzo: i manually modified MB project's url for JIRA integration and to get at the JIRA integration page for LB and configured it there. there's probably some bug in sentry's UI. anyways works now thanks!
2021-05-28 14852, 2021
dgw has quit
2021-05-28 14852, 2021
dgw joined the channel
2021-05-28 14813, 2021
zas
lucifer: yes, I'll finish this after coffee
2021-05-28 14842, 2021
yvanzo
nelgin: that happens when the 'musicbrainz' container has been recreated in the meantime, to solve this: admin/setup-amqp-triggers clean && admin/setup-amqp-triggers install
lucifer: in case you weren't notified, officebrainz is getting a new internet connection some time between midday-3pm (starting in about 2h)
2021-05-28 14852, 2021
alastairp
I'm around from home, but in a meeting during those hours
2021-05-28 14813, 2021
lucifer
yvanzo: sure. but i just found i created a big mess. while trying to leave mb team i accidentally deleted it 🤦. extremely sorry. how can i help clean this up?
2021-05-28 14822, 2021
lucifer
zas: on it
2021-05-28 14807, 2021
lucifer
alastairp: ah yes, i knew about it but wasn't sure of the timing. thanks for reminding.
2021-05-28 14843, 2021
lucifer
ruaok: zas: working now :D similar users successfully requested.
2021-05-28 14859, 2021
zas
good
2021-05-28 14801, 2021
alastairp
lucifer: great. can we slowly move other services back?
2021-05-28 14803, 2021
ruaok
yay!
2021-05-28 14807, 2021
lucifer
i think yes!
2021-05-28 14823, 2021
alastairp
how about as a test we spin up test or beta?
2021-05-28 14833, 2021
ruaok agrees
2021-05-28 14833, 2021
lucifer
let's request an inc to make sure dumps work
2021-05-28 14838, 2021
alastairp
that'll verify all of the service discovery
2021-05-28 14838, 2021
lucifer
*inc dump
2021-05-28 14849, 2021
lucifer
sure, i'll start beta containers
2021-05-28 14832, 2021
lucifer
but we need redis for that, alastairp
2021-05-28 14802, 2021
alastairp
redis is a service, so the copy on boingo _should_ be available from kiss
2021-05-28 14807, 2021
lucifer
will the boingo one work if i start web beta on kiss?
2021-05-28 14811, 2021
lucifer
ah ok makes sense
2021-05-28 14847, 2021
lucifer
beta web up
2021-05-28 14807, 2021
alastairp
sweet, so that confirms that service discovery to other hosts for the website works, and that routing from the gateway works
2021-05-28 14808, 2021
ruaok
looks fine.
2021-05-28 14816, 2021
alastairp
sounds good to me
2021-05-28 14824, 2021
alastairp
ruaok: shall we do the redis dance again?
2021-05-28 14825, 2021
Sophist_UK has quit
2021-05-28 14843, 2021
lucifer
should i request the incremental dump?
2021-05-28 14800, 2021
anonranon joined the channel
2021-05-28 14804, 2021
ruaok
lucifer: lets do one thing at a time.
2021-05-28 14816, 2021
ruaok
lets move things back and then request an incremental.
2021-05-28 14825, 2021
ruaok
alastairp: yes, lets.
2021-05-28 14828, 2021
lucifer
makes sense, 👍
2021-05-28 14835, 2021
alastairp
OK, give me 3m to prepare
2021-05-28 14843, 2021
ruaok
I'm not even logged in yet. still in coffee mode. lucifer can you work with alastairp?
2021-05-28 14852, 2021
ruaok
just a matter of starting/stopping containers.
2021-05-28 14854, 2021
lucifer
yup
2021-05-28 14858, 2021
ruaok
thx
2021-05-28 14803, 2021
alastairp
shut down web boingo; save redis boingo; copy to kiss; start up redis kiss; start up web kiss
2021-05-28 14838, 2021
anonranon_ has quit
2021-05-28 14851, 2021
alastairp
lucifer: if you can take care of web, I'll do redis. I'll tell you when I'm ready for you to take down web, then you tell me when it's done
2021-05-28 14810, 2021
lucifer
sure, i'll take care of web.
2021-05-28 14818, 2021
ruaok
I'll tweet.
2021-05-28 14840, 2021
lucifer
i should also take down beta web, right?
2021-05-28 14809, 2021
alastairp
theoretically new startup scripts should take it down and send error notifications while the redis service is down ;)
2021-05-28 14813, 2021
alastairp
but yes, we could take it down to be sure
2021-05-28 14802, 2021
lucifer
let's test the former then, if doesn't work out, i'll bring beta down manually
we need to add some more info to the sentry messages. Exit status 11 is "service not found"
2021-05-28 14808, 2021
lucifer
yeah makes sense.
2021-05-28 14835, 2021
alastairp
it'd be great to try and pass the missing information to the sentry message. unfortunately the missing information is identified in run-lb-command (part of run->consul) and the error message is sent in `finish`
2021-05-28 14804, 2021
alastairp
and there's no way to pass data from one to another. what we could do is re-do the grep in `finish`, because the config file should still be there
2021-05-28 14846, 2021
lucifer
makes sense, sure let's grep in finish as well.
2021-05-28 14858, 2021
lucifer
should we move back other services now?
2021-05-28 14833, 2021
alastairp
yes, let's finish moving everything over
2021-05-28 14835, 2021
alastairp
can you take care of that?
2021-05-28 14841, 2021
lucifer
sure will do
2021-05-28 14803, 2021
anonranon has quit
2021-05-28 14816, 2021
alastairp
thanks!
2021-05-28 14813, 2021
zas
ruaok: I used a trick for grafana vs kiss interface names, I added a specific processor in telegraf to rename external and internal interface metrics so it's consistent with the rest. In kiss dashboard you now have network stats
2021-05-28 14837, 2021
ruaok
cool
2021-05-28 14859, 2021
lucifer
all lb containers on kiss now.
2021-05-28 14819, 2021
alastairp
😘 thanks
2021-05-28 14824, 2021
lucifer
let's do incremental dump now? :)
2021-05-28 14837, 2021
alastairp
one moment first
2021-05-28 14802, 2021
alastairp
I just want to have a look over the permissions and directories in the dumps volume to check that it'll run as expected
2021-05-28 14810, 2021
lucifer
👍
2021-05-28 14824, 2021
lucifer
docker ps on kiss now shows ipv6 port mappings as well
2021-05-28 14852, 2021
alastairp
I might make an alias for ps that has a custom --format
2021-05-28 14855, 2021
alastairp
with less information
2021-05-28 14816, 2021
ruaok
alastairp: shall we copy some of the data from the storage box back to FTP/backup volumes before generating the incremental dump?
2021-05-28 14829, 2021
ruaok
I'm also itching to dump the storage box.
2021-05-28 14829, 2021
alastairp
sure, good idea
2021-05-28 14837, 2021
ruaok grumbles about a piece of shit
2021-05-28 14839, 2021
alastairp
mounted at /mnt/storagebox/
2021-05-28 14853, 2021
alastairp
no 'lbdumps' user on kiss, I used uid=900,gid=900 in mount ops
2021-05-28 14804, 2021
ruaok
good plan
2021-05-28 14807, 2021
alastairp
just logged out of root, so the mount command should be in history now
this is what I was worried about - there's still some permissions stuff going on here. We'll have to manually change the permissions of this folder for now. I know that it's possible to make docker automatically change the owner of this folder on first startup (the redis image does it), but I don't know exactly how
2021-05-28 14833, 2021
alastairp
s/permissions/owner/
2021-05-28 14847, 2021
ruaok
chown?
2021-05-28 14815, 2021
alastairp
right, but where? If you do it in the Dockerfile then that information is in the image
2021-05-28 14832, 2021
ruaok
service file?
2021-05-28 14842, 2021
alastairp
no, it should happen during docker run
2021-05-28 14805, 2021
alastairp
if you specify docker run with --volume, then on the first startup docker will chown the volume's _data directory
2021-05-28 14858, 2021
alastairp
as an example, try this:
2021-05-28 14807, 2021
alastairp
docker volume create --driver local --name test-redis-volume
2021-05-28 14808, 2021
lucifer
let's copy stuff off the storage box by manually correcting permissions if needed and then move this all to the root user?
2021-05-28 14820, 2021
lucifer
instead of a separate lbdumps user
2021-05-28 14826, 2021
alastairp
docker run --volume "test-redis-volume:/data" --name redis-test redis:5.0.3
2021-05-28 14828, 2021
ruaok
I think so.
2021-05-28 14838, 2021
ruaok
the lbdumps user was created because of the storage box.
2021-05-28 14851, 2021
alastairp
then look at /var/lib/docker/voumes/test-redis-volume/_data - its owner will be 999, not root
2021-05-28 14857, 2021
ruaok
we shouldn't have to do anything special anymore.
2021-05-28 14814, 2021
alastairp
if we're OK to keep everything as root, then I guess we can do that
2021-05-28 14837, 2021
lucifer
everything else runs as root anyways, so why not dumps as well
2021-05-28 14823, 2021
ruaok
agreed.
2021-05-28 14829, 2021
alastairp
ok, let's do that then. what do we need to adjust here? remove the users from Dockerfile, change the crontab to run as root, move the locations of backup and ftp dirs (in Dockerfile and in consul config and in startup scripts [volume mount location]) ?
