#metabrainz

/

      • bitmap
        nothing out of the ordinary in the sir logs, all successful

        • 2022-05-16 13656, 2022

      • bitmap
        can we bring services back up while search indexes are rebuilding?

        • 2022-05-16 13628, 2022

      • yvanzo
        Deleting search indexes manually

        • 2022-05-16 13638, 2022

      • yvanzo
        bitmap: yes

        • 2022-05-16 13649, 2022

      • hackerman8 joined the channel

        • 2022-05-16 13650, 2022

      • bitmap
        ok, I'll do caa-indexer

        • 2022-05-16 13612, 2022

      • reosarevok
        That includes the main servers?

        • 2022-05-16 13628, 2022

      • bitmap
        starting with beta at least

        • 2022-05-16 13636, 2022

      • reosarevok
        bitmap: are you?

        • 2022-05-16 13640, 2022

      • bitmap
        we can check beta real quick and then do prod

        • 2022-05-16 13644, 2022

      • bitmap
        no, just caa-indexer rn

        • 2022-05-16 13646, 2022

      • reosarevok
        Ok

        • 2022-05-16 13648, 2022

      • reosarevok
        I can take betas

        • 2022-05-16 13659, 2022

      • reosarevok
        Can you prepare a commit for d-s-c?

        • 2022-05-16 13605, 2022

      • bitmap
        but I don't think there's anything to index rn :)

        • 2022-05-16 13621, 2022

      • bitmap
        sure

        • 2022-05-16 13657, 2022

      • hackerman has quit

        • 2022-05-16 13620, 2022

      • bitmap
        pushed it to a schema-27-restore-beta-services branch

        • 2022-05-16 13648, 2022

      • bitmap
        fixing the accesslog thing

        • 2022-05-16 13604, 2022

      • reosarevok
        Huh

        • 2022-05-16 13607, 2022

      • reosarevok
        "Status: Image is up to date for metabrainz/musicbrainz-website:beta"

        • 2022-05-16 13610, 2022

      • reosarevok
        ok, paco

        • 2022-05-16 13622, 2022

      • reosarevok
        zappa is downloading a ton of stuff

        • 2022-05-16 13652, 2022

      • hackerman8 has quit

        • 2022-05-16 13653, 2022

      • reosarevok
        So is serge

        • 2022-05-16 13606, 2022

      • hackerman0 joined the channel

        • 2022-05-16 13615, 2022

      • yvanzo
        bitmap: is pink already available for use with sir?

        • 2022-05-16 13621, 2022

      • bitmap
        yes

        • 2022-05-16 13623, 2022

      • reosarevok
        Digest: sha256:8cf61e264f98e8bf06c6042889db2180e2359f0dce4896f0f875807f6f1e95ae does seem correct, tho

        • 2022-05-16 13630, 2022

      • reosarevok
        So maybe it did have the right stuff already?

        • 2022-05-16 13633, 2022

      • reosarevok
        Weird

        • 2022-05-16 13634, 2022

      • bitmap
        jenkins runs on paco, so the image was built there I assume

        • 2022-05-16 13605, 2022

      • reosarevok
        Oh

        • 2022-05-16 13611, 2022

      • reosarevok
        I guess so then :D

        • 2022-05-16 13621, 2022

      • reosarevok
        Ok, all betas should be ready

        • 2022-05-16 13631, 2022

      • reosarevok
        bitmap: can you turn them on?

        • 2022-05-16 13633, 2022

      • yvanzo
        bitmap: However I guess that Solr is not yet available: 503 from floyd

        • 2022-05-16 13633, 2022

      • bitmap
        ok

        • 2022-05-16 13642, 2022

      • reosarevok
        Or should we wait until the indexer?

        • 2022-05-16 13655, 2022

      • bitmap
        solr will come on with the consul config update, I guess?

        • 2022-05-16 13642, 2022

      • bitmap
        https://beta.musicbrainz.org/ is up

        • 2022-05-16 13621, 2022

      • bitmap
        yvanzo: is solr available now?

        • 2022-05-16 13612, 2022

      • yvanzo
        still 503

        • 2022-05-16 13647, 2022

      • v6lur joined the channel

        • 2022-05-16 13636, 2022

      • zas
        solr cloud is still down, at least from solr cloud load balancer -> https://stats.metabrainz.org/d/T4MODrIiz/solr-clo…

        • 2022-05-16 13654, 2022

      • zas
        but was it taken down ?? in which step?

        • 2022-05-16 13628, 2022

      • yvanzo
        zas: down in consul config

        • 2022-05-16 13646, 2022

      • zas
        but solr cloud doesn't used consul afaik

        • 2022-05-16 13637, 2022

      • bitmap
        i'm not sure what the solr key is for in consul/config.prod.json but it has "down": "" since 7 minutes ago

        • 2022-05-16 13640, 2022

      • yvanzo
        zas: where does this metric comes from? I can still query its Collections API locally.

        • 2022-05-16 13653, 2022

      • zas
        it comes from solr cloud load balancer (haproxy)

        • 2022-05-16 13619, 2022

      • zas
        it means backends aren't available (503s)

        • 2022-05-16 13624, 2022

      • yvanzo
        The time of the alert matches the time I deleted search indexes.

        • 2022-05-16 13624, 2022

      • zas
        solr backends are using no cpu ... so I guess the problem is on their side

        • 2022-05-16 13650, 2022

      • bitmap
        maybe we can bring musicbrainz prod back up with a banner saying that search indexes are still building?

        • 2022-05-16 13651, 2022

      • yvanzo
        they are not using cpu because they don't need to, there is no data to handle

        • 2022-05-16 13610, 2022

      • yvanzo
        đź‘Ť

        • 2022-05-16 13646, 2022

      • HorusHorrendus has quit

        • 2022-05-16 13651, 2022

      • bitmap
        reosarevok: were prod containers updated yet?

        • 2022-05-16 13606, 2022

      • reosarevok
        No

        • 2022-05-16 13608, 2022

      • reosarevok
        On it

        • 2022-05-16 13612, 2022

      • bitmap
        thanks

        • 2022-05-16 13622, 2022

      • reosarevok
        yvanzo: any estimation how long indexing might take?

        • 2022-05-16 13636, 2022

      • reosarevok
        aphex done

        • 2022-05-16 13622, 2022

      • yvanzo
        reosarevok: from 4h to 8h

        • 2022-05-16 13632, 2022

      • reosarevok
        Ok, let's not wait :D

        • 2022-05-16 13635, 2022

      • reosarevok
        hip done

        • 2022-05-16 13647, 2022

      • reosarevok
        ludwig done

        • 2022-05-16 13619, 2022

      • reosarevok
        patton done

        • 2022-05-16 13647, 2022

      • reosarevok
        rakim done

        • 2022-05-16 13658, 2022

      • yvanzo
        zas: in mb-solr-cloud, how to get the right value for $zk_addr ?

        • 2022-05-16 13626, 2022

      • reosarevok
        trille done, that should be it

        • 2022-05-16 13633, 2022

      • reosarevok
        bitmap: your turn

        • 2022-05-16 13637, 2022

      • bitmap
        ok

        • 2022-05-16 13619, 2022

      • zas
        yvanzo: I think you can use any of zk_nodes IP, but see https://github.com/metabrainz/mb-solr-cloud/blob/… (it uses first one in ZK_NODES)

        • 2022-05-16 13621, 2022

      • bitmap
        https://musicbrainz.org/ is back

        • 2022-05-16 13655, 2022

      • yvanzo
        zas: thanks

        • 2022-05-16 13602, 2022

      • zas
        https://github.com/metabrainz/mb-solr-cloud/blob/…

        • 2022-05-16 13627, 2022

      • reosarevok
        bitmap, yvanzo: should we announce we're back? And what should be the banners?

        • 2022-05-16 13638, 2022

      • mayhem
        Well done everyone!

        • 2022-05-16 13646, 2022

      • reosarevok
        "We're back from the upgrade, but expect issues with new entities not appearing in search for a few hours"?

        • 2022-05-16 13655, 2022

      • mayhem
        Nice to see things going fairly smoothly!

        • 2022-05-16 13608, 2022

      • mayhem
        !m MB team

        • 2022-05-16 13608, 2022

      • BrainzBot
        You're doing good work, MB team!

        • 2022-05-16 13624, 2022

      • alastairp
        yeah, from here it looked like it was smooth sailing

        • 2022-05-16 13646, 2022

      • mayhem
        alastairp: lucifer will LB automatically restart and enable logins again? Or do we need to restart anything?

        • 2022-05-16 13655, 2022

      • alastairp
        looks like lucifer just did it

        • 2022-05-16 13602, 2022

      • mayhem
        Ok, cool.

        • 2022-05-16 13604, 2022

      • lucifer
        yup just did it a few mins ago

        • 2022-05-16 13610, 2022

      • alastairp
        bringing CB back

        • 2022-05-16 13611, 2022

      • monkey
        !m team

        • 2022-05-16 13611, 2022

      • BrainzBot
        You're doing good work, team!

        • 2022-05-16 13613, 2022

      • mayhem
        Let me restart the mapping containers.

        • 2022-05-16 13617, 2022

      • yvanzo
        reosarevok: Is the docs updated?

        • 2022-05-16 13648, 2022

      • reosarevok
        yvanzo: what with?

        • 2022-05-16 13653, 2022

      • yvanzo
        zas: mbsssss.upconfig.sh just completed

        • 2022-05-16 13657, 2022

      • reosarevok
        Or you mean the update docs? I'm still doing the rest of the bits :)

        • 2022-05-16 13604, 2022

      • reosarevok
        Sitemaps now

        • 2022-05-16 13630, 2022

      • yvanzo
        reosarevok: just I see many lines not crossed out

        • 2022-05-16 13631, 2022

      • v6lur has quit

        • 2022-05-16 13648, 2022

      • reosarevok
        Oh, wait, sitemaps was brought up with the mb servers :p

        • 2022-05-16 13614, 2022

      • reosarevok
        wikidata-bot started

        • 2022-05-16 13627, 2022

      • v6lur joined the channel

        • 2022-05-16 13630, 2022

      • reosarevok
        Tailing it for a mo

        • 2022-05-16 13634, 2022

      • alastairp
        CB up, login works

        • 2022-05-16 13648, 2022

      • monkey
        Same for BookBrainz

        • 2022-05-16 13643, 2022

      • mayhem
        alastairp: lucifer

        • 2022-05-16 13603, 2022

      • mayhem
        https://www.irccloud.com/pastebin/ppQt16Se/

        • 2022-05-16 13628, 2022

      • alastairp
        ah, I know what that is

        • 2022-05-16 13631, 2022

      • mayhem
        when restarting the mapping containers. is this on your radar?

        • 2022-05-16 13659, 2022

      • reosarevok
        yvanzo, bitmap: updating prod-cron

        • 2022-05-16 13622, 2022

      • lucifer
        mayhem: try running with v-2022-05-16.0 image

        • 2022-05-16 13624, 2022

      • alastairp
        oh right - that's the old listenbrainz code before I fixed the listenstore-down stuff last week

        • 2022-05-16 13642, 2022

      • alastairp
        I missed those containers when I deployed

        • 2022-05-16 13656, 2022

      • reosarevok
        yvanzo, bitmap: time to start the full dump, or any other small bits to do first from either of you?

        • 2022-05-16 13612, 2022

      • bitmap
        nothing from me

        • 2022-05-16 13638, 2022

      • yvanzo
        bitmap, reosarevok, zas: just started to rebuild search indexes

        • 2022-05-16 13648, 2022

      • bitmap
        I'll work on upgrading the json dump db

        • 2022-05-16 13630, 2022

      • reosarevok
        bitmap: can you document it as you do it, too? :p

        • 2022-05-16 13606, 2022

      • bitmap
        yep

        • 2022-05-16 13613, 2022

      • reosarevok
        Screen running, you can probably tail it if you want to or whatever :)

        • 2022-05-16 13600, 2022

      • mayhem
        ok, that image started, lucifer . queue cleared

        • 2022-05-16 13612, 2022

      • reosarevok
        wikidata-bot is doing its thing, so that's good

        • 2022-05-16 13637, 2022

      • yvanzo
        https://musicbrainz.org/search?type=area&quer… :)

        • 2022-05-16 13645, 2022

      • reosarevok
        Yay

        • 2022-05-16 13655, 2022

      • yvanzo
        (indexes rebuilt in alphabetical order)

        • 2022-05-16 13630, 2022

      • reosarevok
        https://www.irccloud.com/pastebin/yyPuVhMO/

        • 2022-05-16 13639, 2022

      • reosarevok
        I thought something would have appeared under this by now :D

        • 2022-05-16 13614, 2022

      • lucifer
        mayhem: great!

        • 2022-05-16 13632, 2022

      • bitmap
        reosarevok: I think it's just a quirk with admin/run

        • 2022-05-16 13632, 2022

      • reosarevok
        So I won't know when it's done or? :D

        • 2022-05-16 13625, 2022

      • v6lur has quit

        • 2022-05-16 13648, 2022

      • v6lur joined the channel

        • 2022-05-16 13641, 2022

      • bitmap
        it should still be doing stuff but the progress won't update

        • 2022-05-16 13642, 2022

      • yvanzo
        reosarevok: you can know from processes activity, open a new window in your screen session

        • 2022-05-16 13610, 2022

      • yvanzo
        C-a c (to create a new window)

        • 2022-05-16 13624, 2022

      • yvanzo
        C-a " (to navigate through windows list)

        • 2022-05-16 13642, 2022

      • yvanzo
        bitmap, reosarevok: Suggestion to update the website banner: https://www.irccloud.com/pastebin/3Qkt53KD/

        • 2022-05-16 13604, 2022

      • reosarevok
        I guess

        • 2022-05-16 13655, 2022

      • reosarevok
        Update: MusicBrainz is getting back on track since <a href="https://timeanddate.com/countdown/gener… target="_blank">19 UTC</a> after a successful <a href="https://blog.metabrainz.org/2022/03/15/… target="_blank">database schema change upgrade</a>! The database itself is back, and the website and the API are back except for the (indexed) search, which will

        • 2022-05-16 13655, 2022

      • reosarevok
        take a few more hours. Thank you for your patience!

        • 2022-05-16 13658, 2022

      • reosarevok
        maybe?

        • 2022-05-16 13609, 2022

      • yvanzo
        posted

        • 2022-05-16 13632, 2022

      • yvanzo
        looking into the WP draft now

        • 2022-05-16 13638, 2022

      • reosarevok
        yvanzo: posted the same to beta

        • 2022-05-16 13643, 2022

      • reosarevok
        Thanks!

        • 2022-05-16 13656, 2022

      • reosarevok
        I'll quickly check translators

        • 2022-05-16 13612, 2022

      • yvanzo
        reosarevok: “There is an autosave of this post that is more recent than the version below.” I never made any change, still the last autosave is credited to me...

        • 2022-05-16 13626, 2022

      • reosarevok
        Wordpress is stupid, I dunno

        • 2022-05-16 13625, 2022

      • yvanzo
        reosarevok: https://musicbrainz.org/report/ReleasesWithAmazon… is broken atm