#metabrainz

nothing out of the ordinary in the sir logs, all successful

can we bring services back up while search indexes are rebuilding?

Deleting search indexes manually

bitmap: yes

ok, I'll do caa-indexer

That includes the main servers?

starting with beta at least

bitmap: are you?

we can check beta real quick and then do prod

no, just caa-indexer rn

Ok

I can take betas

Can you prepare a commit for d-s-c?

but I don't think there's anything to index rn :)

sure

pushed it to a schema-27-restore-beta-services branch

fixing the accesslog thing

Huh

"Status: Image is up to date for metabrainz/musicbrainz-website:beta"

ok, paco

zappa is downloading a ton of stuff

So is serge

bitmap: is pink already available for use with sir?

yes

Digest: sha256:8cf61e264f98e8bf06c6042889db2180e2359f0dce4896f0f875807f6f1e95ae does seem correct, tho

So maybe it did have the right stuff already?

Weird

jenkins runs on paco, so the image was built there I assume

Oh

I guess so then :D

Ok, all betas should be ready

bitmap: can you turn them on?

bitmap: However I guess that Solr is not yet available: 503 from floyd

ok

Or should we wait until the indexer?

solr will come on with the consul config update, I guess?

https://beta.musicbrainz.org/ is up

yvanzo: is solr available now?

still 503

solr cloud is still down, at least from solr cloud load balancer -> https://stats.metabrainz.org/d/T4MODrIiz/solr-c...

but was it taken down ?? in which step?

zas: down in consul config

but solr cloud doesn't used consul afaik

i'm not sure what the solr key is for in consul/config.prod.json but it has "down": "" since 7 minutes ago

zas: where does this metric comes from? I can still query its Collections API locally.

it comes from solr cloud load balancer (haproxy)

it means backends aren't available (503s)

The time of the alert matches the time I deleted search indexes.

solr backends are using no cpu ... so I guess the problem is on their side

maybe we can bring musicbrainz prod back up with a banner saying that search indexes are still building?

they are not using cpu because they don't need to, there is no data to handle

👍

reosarevok: were prod containers updated yet?

No

On it

thanks

yvanzo: any estimation how long indexing might take?

aphex done

reosarevok: from 4h to 8h

Ok, let's not wait :D

hip done

ludwig done

patton done

rakim done

zas: in mb-solr-cloud, how to get the right value for $zk_addr ?

trille done, that should be it

bitmap: your turn

ok

yvanzo: I think you can use any of zk_nodes IP, but see https://github.com/metabrainz/mb-solr-cloud/blo... (it uses first one in ZK_NODES)

https://musicbrainz.org/ is back

zas: thanks

bitmap, yvanzo: should we announce we're back? And what should be the banners?

Well done everyone!

Nice to see things going fairly smoothly!

!m MB team

You're doing good work, MB team!

yeah, from here it looked like it was smooth sailing

alastairp: lucifer will LB automatically restart and enable logins again? Or do we need to restart anything?

looks like lucifer just did it

Ok, cool.

yup just did it a few mins ago

bringing CB back

!m team

You're doing good work, team!

Let me restart the mapping containers.

reosarevok: Is the docs updated?

yvanzo: what with?

zas: mbsssss.upconfig.sh just completed

Or you mean the update docs? I'm still doing the rest of the bits :)

Sitemaps now

reosarevok: just I see many lines not crossed out

Oh, wait, sitemaps was brought up with the mb servers :p

wikidata-bot started

Tailing it for a mo

CB up, login works

Same for BookBrainz

alastairp: lucifer

ah, I know what that is

when restarting the mapping containers. is this on your radar?

yvanzo, bitmap: updating prod-cron

mayhem: try running with v-2022-05-16.0 image

oh right - that's the old listenbrainz code before I fixed the listenstore-down stuff last week

I missed those containers when I deployed

yvanzo, bitmap: time to start the full dump, or any other small bits to do first from either of you?

nothing from me

bitmap, reosarevok, zas: just started to rebuild search indexes

I'll work on upgrading the json dump db

bitmap: can you document it as you do it, too? :p

yep

Screen running, you can probably tail it if you want to or whatever :)

ok, that image started, lucifer . queue cleared

wikidata-bot is doing its thing, so that's good

https://musicbrainz.org/search?type=area&qu... :)

Yay

(indexes rebuilt in alphabetical order)

I thought something would have appeared under this by now :D

mayhem: great!

reosarevok: I think it's just a quirk with admin/run

So I won't know when it's done or? :D

it should still be doing stuff but the progress won't update

reosarevok: you can know from processes activity, open a new window in your screen session

C-a c (to create a new window)

C-a " (to navigate through windows list)

bitmap, reosarevok: Suggestion to update the website banner: https://www.irccloud.com/pastebin/3Qkt53KD/

I guess

Update: MusicBrainz is getting back on track since <a href="https://timeanddate.com/countdown/gen...; target="_blank">19 UTC</a> after a successful <a href="https://blog.metabrainz.org/2022/03/1...; target="_blank">database schema change upgrade</a>! The database itself is back, and the website and the API are back except for the (indexed) search, which will

take a few more hours. Thank you for your patience!

maybe?

posted

looking into the WP draft now

yvanzo: posted the same to beta

Thanks!

I'll quickly check translators

reosarevok: “There is an autosave of this post that is more recent than the version below.” I never made any change, still the last autosave is credited to me...

Wordpress is stupid, I dunno

reosarevok: https://musicbrainz.org/report/ReleasesWithAmaz... is broken atm