#metabrainz

`listenbrainz-timescale-pg13-data` has been in use since.

is that mounted anywhere? if not, nuke it.

i can delete the volume to free up 700g

how about typesense-data and typesense-new-data ?

that sounds like there is something duplicated.

yes that can also be removed

:D

I'll ask hetzner if we can get a AC102 with only two 7.68TB drives.

sounds great but might be an overkill for now. (7.68 drives seem to be expensive and we won't hit the limit for quite a while)

ok, then lets leave this issue be for a little while. I don't feel that there are good options for what we currently need.

sure sounds good.

but lets keep our eyes peeled on this.

yup makes sense, thanks!

to be clear, so also postponing the LB backup server thing for now?

it doesn't quite feel that the options are right for us now. I understand the need, but the path looks kinda meh.

for all the reasons mentioned above.

yup makes sense

we can't easily get another gaga class machine without it costing close to 300/month.

true

i guess buddy has enough backup so that we can setup barman at least?

<lucifer> "try this instead" <- still not working :(

what error do you get rimskii[m] ?

the same

onnection to server at "127.0.0.1", port 5433 failed: Connection refused

Is the server running on that host and accepting TCP/IP connections?

it will be different somehow because i changed the ports, share the new error.

buddy: /dev/md1 7.3T 959G 6.3T 14% /data2

great,

looks that way.

rimskii[m]: can you share the ssh command output?

to check if the port forwarding worked.

mybe I shoud have run first "ssh rimskii@wolf.metabrainz.org"; then "ssh -L 5432:localhost:5432 rimskii@wolf.metabrainz.org"?

rimskii[m]: the command is outdated.

you have to do `ssh -L 5433:localhost:5432 rimskii@wolf.metabrainz.org`

note the 5433:

That was the command rimskii ran, from what I can tell from the logs.

lucifer: yep i tried it too

oh i see, i didn't look at the logs but just the command.

hmm, that should work.

according to logs, the port forwarding worked.

Could it be something regarding "127.0.0.1:5432" vs. "localhost:5432" ?

I mean, I assume not, but ...

shuoldn't be. but its a mac

I tried to connect in the same way, works for me (on linux)

One thing I had to adjust to be able to connect to the DB was username and password

the latest error points to a port forwarding error

you can see from the logs that it is listening on 127.0.0.1 and ::1

so localhost vs. 127.0.0.1 is irrelevant

I will test on my MacBook but never had any issues on Linux vs. MacOS in this area

Same here, used to do port forwarding fine on macos

Maybe something related to the macos firewall?

or a university/home router configuration?

not sure how that could make any difference, it's a local port forward

so the router isn't involved

rimskii: what terminal are you using?

Sorry if I'm way out, but where is the connection error coming from? Inside Docker? if so, does the local port 5433 need to be added to the docker container ports?

atj[m]: the default one? mac?

mayhem[m]: atj[m]: i just hopped on a call with rimskii port forwarding is working. but the service that needs to access the port is inside docker.

monkey[m]: yeah that

Might be good to try connecting with psql to remove any potential docker-related issue

rimskii[m]: you mean terminal

ah

[mb-solr] 14yvanzo opened pull request #58 (03master…solr-9.6.1): Upgrade Solr version to 9.6.1 and other dependencies https://github.com/metabrainz/mb-solr/pull/58

atj[m]: yeah

iTerm is way better FWIW :)

psql --U musicbrainz --host localhost --port 5433

lucifer: ah, so that's an extra layer of fun

s//`/, s/--/-/, s//`/

atj[m]: I second that

atj[m]: oh i will try it then

not really relevant at this point but just a suggestion

rimskii[m]: can you replace `localhost` in MB_DATABASE_URI with `host.docker.internal` ?

wow, that's shitty hetzner: "It´s possible to remove the default drives but there will be no price reduction for it."

lucifer: omg it works!

now i got a new error haha

one step forward....

rimskii[m]: i see, we don't have the mapping tables on wolf but i can run a script to create them there.

will take a few hours for it complete.

atj: Is a new snapshot needed for upgrading Solr version in the cluster? Or should we test it separately?

lucifer: mybe i can do that

yvanzo[m]: I don't think so, if it works in 9.6.0 it will work in 9.6.1

If we do it separately I can write docs on that too :)

rimskii[m]: sure. so there are two ways to do it, 1 is running the command locally with the databases connected over ssh but that means you need a stable network connection for a few hours at least.

OK, let’s try upgrading Solr 9.6.1 at first then.

alternative is cloning the listenbrainz-server on wolf. and running the commands from there.

I’m building the snapshot in parallel.

i think this way will work better for you

lucifer: ok

yvanzo: just FYI, IME Solr can sometimes refuse to stop in an orderly fashion and the stop script waits 180 seconds before killing it

so if you see the restart task hanging, don't worry it's normal (ish)

just wait

atj: I pushed a commit to the `solr` branch and update the node 8 as suggested.

rimskii[m]: let me know when have cloned the repo and i can help update the script and let you know the commands to build the cache.

yvanzo: LGTM, want to upgrade the entire cluster?

Those are transient errors that occurred when updating the node 8.

lucifer: okay

i have to build the docket too right?

yvanzo[m]: that's normal AFAIU

rimskii[m]: yes but different containers.

atj: Should we pause between updating each node to preserve service availability?

-f1 is certainly useful

lucifer: which containers?

cloned the repo & updated config

atj: Ideally we should have a `wait_for` the node is available again before moving to another task.

atj: That would be for production though, I’ll proceed as documented for now.

yvanzo: https://github.com/metabrainz/ansible-role-solr...

yvanzo: do you want to wait a second and we can try using the serial strategy: https://docs.ansible.com/ansible/latest/playboo...

OK.

it should only need a minor update to the playbook

yvanzo: can I squash your commit into mine?

I need to force push to the solr branch really otherwise it gets messy and I just want to merge one big commit

atj: No problem.

rimskii[m]: `cd mbid_mapping` then `cp config.py.sample config.py`.

edit `config.py`, set `MBID_MAPPING_DATABASE_URI` and `MB_DATABASE_MASTER_URI` to `dbname=musicbrainz_db user=musicbrainz host=db port=5432 password=musicbrainz`

monkey: My MetaBrainz Tabler icon [pull request](https://github.com/tabler/tabler-icons/pull/1171) is up, FYI. Let's see if it is really that simple to get an icon added.

yvanzo: i've added support for setting the batch size using `serial` and updated the documentation: https://github.com/metabrainz/metabrainz-ansibl...

this is better than -f1 because it means the entire play will be run on each node serially rather than each task

so the "restart" and the "wait for start" handlers will run before moving on to the next host

<kellnerd[m]> "monkey, mayhem: Since my GSoC..." <- ^ Reminder just in case you have missed my message mayhem, last year an org admin was required to do that change on the SoC page IIRC (CC monkey).

<lucifer> "edit `config.py`, set `MBID_MAPP..." <- done !

rimskii[m]: run `./build.sh`

atj: Now testing!

lucifer: done !

now "python mapper/manage.py canonical-data"?

run `tmux`

then `docker run --rm -it --network musicbrainz-docker_default metabrainz/mbid-mapping python3 manage.py create-all`

lucifer: should I change sqlaclhemy one too?

rimskii[m]: hmm i see, okay try this instead. `docker run --rm -it --network musicbrainz-docker_default metabrainz/mbid-mapping python3 manage.py canonical-data --use-mb-conn`

atj: Oops, I cannot edit solr.yml at the same time.

(?)

what do you mean?

I was editing solr.yml in the meantime and realized that ansible-playbook was still running, so interrupted it, restored the file, and started it again.

Not sure if it is watching changes to solr.yml though.

oh right, yes it's read when the process starts

It is actually downloading the JAR again when updating Solr.

<lucifer> "rimskii: hmm i see, okay try..." <- done ! it works

rimskii[m]: did it finish already?

atj: Would there be a way to prevent routing requests to a node during the update?

lucifer: well seems so