I tried adding 127.0.0.53 to docker daemon config too.
2021-11-01 30555, 2021
ruaok
zas: you still about?
2021-11-01 30504, 2021
ruaok
we could use your expertise for a second.
2021-11-01 30528, 2021
ruaok
maybe a firewall issue?
2021-11-01 30506, 2021
ruaok
making changes to docker daemon requires a docker restart.
2021-11-01 30514, 2021
zas
Eating right now, can it wait few minutes ?
2021-11-01 30518, 2021
ruaok
but containers have been up for 40 minutes... so.
2021-11-01 30521, 2021
ruaok
zas: k
2021-11-01 30523, 2021
yvanzo
I did restart docker service
2021-11-01 30534, 2021
ruaok
bitmap: can I restart docker?
2021-11-01 30540, 2021
yvanzo
(on purple)
2021-11-01 30540, 2021
bitmap
ruaok: sure
2021-11-01 30542, 2021
ruaok
it didn't restart.
2021-11-01 30546, 2021
ruaok
k
2021-11-01 30528, 2021
ruaok
oh, I see, times didn't reset on the containers.
2021-11-01 30531, 2021
ruaok
hmm.
2021-11-01 30514, 2021
ruaok
yeah, not working.
2021-11-01 30518, 2021
ruaok
I suspect firewall.
2021-11-01 30532, 2021
yvanzo
failed again
2021-11-01 30538, 2021
zas
Docker is set up to not use iptables, if you use docker networks it I'd likely the problem
2021-11-01 30550, 2021
ruaok
yes, we are.
2021-11-01 30528, 2021
ruaok
are we forced to hose hostmode for networking?
2021-11-01 30547, 2021
zas
Check etc docker daemon.json
2021-11-01 30538, 2021
yvanzo
I did add 127.0.0.53 to dns in /etc/docker/daemon.json
2021-11-01 30541, 2021
ruaok
actually, we can do iptable for now.
2021-11-01 30502, 2021
ruaok
restarting docker.
2021-11-01 30542, 2021
ruaok
working now.
2021-11-01 30547, 2021
ruaok
go ahead, yvanzo
2021-11-01 30553, 2021
yvanzo
I cannot even ping github from container (but fetching repo from github worked fine while building the image worked fine)
2021-11-01 30501, 2021
ruaok
you should be able to now.
2021-11-01 30524, 2021
yvanzo
same
2021-11-01 30543, 2021
ruaok
I can access the outside world from other containers.
2021-11-01 30551, 2021
ruaok
maybe stop all containers and start again?
2021-11-01 30557, 2021
yvanzo
I did that already
2021-11-01 30506, 2021
yvanzo
even rebuilt images
2021-11-01 30558, 2021
yvanzo
ruaok: works now, thanks
2021-11-01 30506, 2021
ruaok
great.
2021-11-01 30523, 2021
ruaok
ok, I'm idle. ping me if you need anything bitmap, yvanzo
2021-11-01 30535, 2021
yvanzo
I had to remove dns settings for containers
2021-11-01 30511, 2021
yvanzo
(fetching db dump for real)
2021-11-01 30547, 2021
bitmap
artwork-indexer container is running. it seems to be able to connect to PG fine.
2021-11-01 30519, 2021
bitmap
(gives FATAL: database "musicbrainz_db" does not exist for now)
2021-11-01 30538, 2021
bitmap
artwork-redirect*
2021-11-01 30525, 2021
ruaok
lets see if the gateway picked it up
2021-11-01 30557, 2021
ruaok
OH JEEZ MAKE IT STOP.
2021-11-01 30529, 2021
ruaok
it picked it up and there is a TON of traffic.
2021-11-01 30531, 2021
ruaok
scary.
2021-11-01 30536, 2021
bitmap
oof, stopped
2021-11-01 30547, 2021
ruaok
bitmap: sorry, not you.
2021-11-01 30556, 2021
ruaok
that was badly expressed, sorry.
2021-11-01 30501, 2021
ruaok
put that back it was doing great. :)
2021-11-01 30512, 2021
ruaok
docker logs -f 73b2abdd60b2
2021-11-01 30518, 2021
ruaok
is a bit of an assault.
2021-11-01 30546, 2021
bitmap
haha yes I see what you mean
2021-11-01 30533, 2021
bitmap
I wonder why the artwork-redirect_caa_1 logs aren't showing errors for these
2021-11-01 30535, 2021
ruaok
ah, there is some cert problem. I guess I need to restart the service to pick up the changes.
2021-11-01 30555, 2021
ruaok
its not quite working yet. the cert wasn't correctly generated. hang on
2021-11-01 30528, 2021
ruaok
`[Mon Nov 1 21:10:11 UTC 2021] coverartartchive.org:Verify error:DNS problem: NXDOMAIN looking up A for coverartartchive.org - check that a DNS record exists for this domain
2021-11-01 30540, 2021
ruaok
fat fingered that one. :)
2021-11-01 30554, 2021
ruaok
can you please fix that extra t in the domain, bitmap ?
2021-11-01 30557, 2021
ruaok
and restart your container?
2021-11-01 30523, 2021
ruaok
cover art art chive. way to go mr. kaye.
2021-11-01 30526, 2021
bitmap
whoops, ok one sec
2021-11-01 30537, 2021
ruaok
there may be two instances
2021-11-01 30518, 2021
bitmap
yeah
2021-11-01 30502, 2021
ruaok
I bet you have traffic now.
2021-11-01 30510, 2021
bitmap
indeed...
2021-11-01 30518, 2021
ruaok
piles and piles of it.
2021-11-01 30535, 2021
bitmap
I probably *should* stop the containers for now until these tracebacks stop
2021-11-01 30514, 2021
zas
I'll switch back from herb to kiki
2021-11-01 30559, 2021
bitmap
huh. there actually is no index_listing view in the database
2021-11-01 30536, 2021
ruaok
oh dear, the load is already 20 and we have no DB in service.
2021-11-01 30552, 2021
bitmap
yvanzo: I assume the schema is still being created?
2021-11-01 30509, 2021
ruaok
we can stop the proxy too. that will make the import go faster.
2021-11-01 30512, 2021
ruaok
let me do that
2021-11-01 30521, 2021
yvanzo
bitmap: yes
2021-11-01 30529, 2021
bitmap
okay
2021-11-01 30550, 2021
bitmap
I stopped the caa container for now
2021-11-01 30527, 2021
ruaok
yvanzo: the whole server is yours now.
2021-11-01 30546, 2021
ruaok
once the import is done we'll bring everything up and hope that one server can cope.
2021-11-01 30552, 2021
ruaok
if not, we'll get a bigger server tomorrow.
2021-11-01 30514, 2021
yvanzo
almost loaded db dump
2021-11-01 30549, 2021
ruaok
yvanzo: musicbrainz-docker_search_1 is continually restarting
2021-11-01 30513, 2021
yvanzo
I stopped it
2021-11-01 30518, 2021
yvanzo
(not needed)
2021-11-01 30506, 2021
ruaok
k
2021-11-01 30531, 2021
ruaok
this poor server is going to get hammered.
2021-11-01 30513, 2021
ruaok
zas: you about? is this server already configured for high load?
2021-11-01 30556, 2021
yvanzo
I guess I should allocate more than 4GB to PG shared buffers then?
2021-11-01 30501, 2021
zas
I don't think so
2021-11-01 30534, 2021
zas
I guess nginx & sysctl will need tuning
2021-11-01 30543, 2021
ruaok
can you please work on that?
2021-11-01 30502, 2021
ruaok
yvanzo: yes, give it 33%, I would say
2021-11-01 30516, 2021
ruaok
20Gb
2021-11-01 30536, 2021
ruaok
because almost nothing else is going to take ram, might as well give it all to pg.
2021-11-01 30505, 2021
yvanzo
you just need the DB, not MB server?
2021-11-01 30535, 2021
bitmap
we'll need replication running at least
2021-11-01 30511, 2021
ruaok
no MB server, but yes replication.
2021-11-01 30534, 2021
ruaok
but don't work harder to turn off MB server. its just not needed.
2021-11-01 30526, 2021
yvanzo
I will reduce the number of processes to 1 at least
2021-11-01 30534, 2021
ruaok
puurfect.
2021-11-01 30545, 2021
yvanzo
bitmap: do we need materialized tables?
2021-11-01 30519, 2021
ruaok
bitmap: ^^
2021-11-01 30522, 2021
bitmap
yvanzo No I don’t think so
2021-11-01 30545, 2021
ruaok
zas: how can we get the nginx and sysctl turning done soon?
2021-11-01 30518, 2021
yvanzo
creating PKs
2021-11-01 30526, 2021
zas
let me see
2021-11-01 30520, 2021
zas
where's your nginx.conf ?
2021-11-01 30520, 2021
ruaok
thx
2021-11-01 30535, 2021
ruaok
there is no additional nginx.conf as of yet
2021-11-01 30555, 2021
ruaok
the whole setup lives in ~musicbrainz/nginx-proxy
2021-11-01 30503, 2021
ruaok
and by whole, I mean that one file.
2021-11-01 30556, 2021
zas
hmmm
2021-11-01 30543, 2021
zas
it will not be easy to configure
2021-11-01 30530, 2021
ruaok
what do you want to do?
2021-11-01 30547, 2021
zas
well, we need to configure a lot of things at all levels in the config file, mainly set workers (at top level) and backlog (in server blocks)
