#metabrainz

/

      • d4rkie has quit
      • d4rkie joined the channel
      • bitmap
        on jimmy block_size is also 8192, so not sure
      • I guess it's also affected by compression
      • yvanzo
        If that is related to `block_size` (could be something else), 8192 is just the default value, it can be changed when compiling.
      • bitmap
      • ansh has quit
      • ansh joined the channel
      • MonkeyPython
        fwiw the "/who name" command was new to me and helped
      • yellowhatpro joined the channel
      • atj
        https://www.postgresql.org/message-id/8326.1289... "the 8K limit applies after possible in-line compression, so the actual data value causing the failure was likely considerably longer than 10K"
      • bitmap
      • atj
        confused how a 44kb value doesn't trigger this error?
      • nothing i've found indicates that the limit can be increased
      • huhridge joined the channel
      • bitmap
      • that should return the "Number of bytes used to store a particular value (possibly compressed)"
      • lucifer
        bitmap, atj: there is no index on track.name in prod afais
      • atj
        lucifer: well that would explain it :P
      • bitmap was just checking that
      • do we have any indexed text / unlimited varchar fields?
      • actually, the insert would just fail
      • bitmap
        we might
      • atj
        so not a DoS
      • lotooo
        Does your prod missing an index or does my setup have an extra one ?
      • bitmap
        the error message doesn't indicate which index is triggering the error AFAICT
      • lucifer
        lotooo: `\d musicbrainz.track` in your local database and share the output probably.
      • TOPIC: MetaBrainz Community and Development channel | MusicBrainz non-development: #musicbrainz | BookBrainz: #bookbrainz | Channel is logged and not empty as it is bridged to IRC; see https://musicbrainz.org/doc/IRC for details | Agenda: Reviews, Group Contacts on IRC (yvanzo)
      • using psql that is.
      • MonkeyPython
        heh
      • yvanzo
        MonkeyPython: your issue cannot be resolved beforehand ^
      • bitmap
        it could be on a table other than track, I suppose, since we have some mirror triggers than run on track updates
      • MonkeyPython
        yep. we (myself & reo) asked about it in libera
      • yvanzo
        That would be nice to share about it here.
      • lotooo
        I learnt my lesson : https://paste.ee/p/s1lFO
      • MonkeyPython
        basically asked about it and was told that the community manager msut deal with it. reo is the defacto guy so he asked about it
      • he's not listed in our "contact people" even if founder
      • monkey[m] has quit
      • yvanzo
        reosarevok: feel free to drop the last topic from the agenda if you are already updating the list of group contacts as in https://libera.chat/chanreg/#group-contacts
      • MonkeyPython
        liberastaff member was going ot poke some people about it
      • bitmap
        lotooo: did you create this index yourself? because we used to have one with the same name, but it was dropped years ago.
      • MonkeyPython
        so first we gotta figure out how to put reo in that list
      • bitmap
        but I guess that may explain why nobody else has come forth to report replication issues (?)
      • reosarevok
        yvanzo: I just asked a bit ago, yeah :) Still nothing coming out of it. Maybe I just need to send the email :)
      • lotooo
        nope. we have a lot of mirrors at plex and we don't touch the database. We just grab the db from you guys and then replicate. But we built that setup years ago. We just update it to new versions but I usually don't rebuild nodes from scratch
      • last time I rebuilt everything was during the pg upgrade to pg12
      • yvanzo
        reosarevok: my intend was to ask for volunteers to join the list that we would mail to the staff
      • reosarevok
        Yes, that seems sensible too, anyway :)
      • MonkeyPython
        ++ that idea
      • bitmap
        lotooo: strange, it's explicitly dropped in admin/sql/updates/20200518-pg12-before-upgrade.sql which should have been run during that upgrade
      • yvanzo
        atj, bitmap: A similar title is set for the release, the track, and the recording.
      • MonkeyPython
        the "public contacts" secton on /ChanServ INFO #metabrainz
      • yvanzo
        yes
      • MonkeyPython
        reo, and yvanzo, and possibly monkey seem like good candidates imho
      • yvanzo
        they advise to have 2 or 4 contacts
      • MonkeyPython
        good idea
      • yvanzo
        volunteering is proposing yourself, not others :)
      • MonkeyPython
        hahaha
      • lotooo
        @bitmap : I just have to replay that script and call it a day ? (I'm planning on rebuilding our setup soon-ish anyway)
      • MonkeyPython
        didn't want to seem too ostentatious
      • bitmap
        lotooo: I wouldn't run that script by itself since it depends on steps in later scripts, but you can just drop the problematic index for now
      • lotooo
        @bitmap which index specificatly ?
      • bitmap
        DROP INDEX track_idx_txt;
      • lotooo
        :thumbsup:
      • mmm that doesn't seem to have helped :thinking_face: triple_checking I am on the good server
      • bitmap
        lotooo: my bad, track_idx_musicbrainz_collate too
      • lotooo
        I see that track_idx_musicbrainz_collate is based on name as well
      • ah ok perfect
      • bitmap
        yeah, it's alaso dropped in the script I mentioned
      • lotooo
        Thanks that seemed to have helped ! Letting this server catch up and I'll execute the same on our other mirrors. Thanks for your help
      • bitmap
        glad it's working šŸ‘
      • yvanzo
        bitmap: When you need it, you can try resuming the replication on wolf (it is only running at 03:00 UTC so it didnā€™t encounter this issue yet).
      • bitmap
        yvanzo: thanks, I'll check
      • reosarevok
        Should we then close the ticket and say that if anyone has similar issues they should drop those two indexes?
      • yvanzo
        hi yellowhatpro: Sorry I misspelled your nickname earlier: https://chatlogs.metabrainz.org/libera/metabrai...
      • reosarevok: no because it is just a temporary workaround
      • reosarevok
        Well, I thought the point was those were *not* supposed to exist anymore :)
      • From what bitmap said
      • yvanzo
        I read ā€œfor nowā€
      • bitmap
        yes, if the schema change was applied correctly they should not exist, although we may have other indexes on TEXT columns that exhibit this issue
      • sorry, "for now" was as opposed to "rebuilding our setup soon-ish"
      • yvanzo: I guess the mirror on wolf would only help checking that the upgrade scripts worked correctly, and only if it was originally built prior to the PG12 upgrade in 2020.
      • yvanzo
        It isnā€™t that old.
      • yellowhatpro
        hi @yvanzo, I'll be there!
      • Maxr1998_ joined the channel
      • Maxr1998 has quit
      • yvanzo
        bitmap: any TEXT or VARCHAR actually, right?
      • bitmap
        right
      • minimal joined the channel
      • reosarevok starts going back to Belgium
      • reosarevok
        mayhem: I should be available for the meeting (waiting in the airport) but if I see something goes wrong I'll let you know so you can take over :)
      • mayhem
        k.
      • reosarevok goes back to try to fit all this Asturian cheese in the bag šŸ˜…
      • MonkeyPython
        :V that's what he said
      • yvanzo
        reosarevok: Iā€™ve been through the same hardshipsā€¦ šŸ§€
      • bitmap: I checked that trying to enter such a title for a currently indexed field currently fails with error 500, it could be possibly improved but it isnā€™t at risk for mirrors. Are they other things I can help you with checking?
      • there
      • MonkeyPython
        man there is literally no image of this bagpipe on commons, and image in the article, linked to wikidata etc. is just a regular Great Highlands....
      • any armenians that wanna take some image of a parkapzuk :DD
      • BrainzGit
        [listenbrainz-server] 14MonkeyDo merged pull request #2821 (03masterā€¦docs/windows-docker-installation): Docs: Add windows docker installation instruction https://github.com/metabrainz/listenbrainz-serv...
      • huhridge has quit
      • bitmap
        yvanzo: thanks, I got the same failure too
      • reosarevok
        "We and ourĀ 699Ā partners store and/or access information on a device, such as unique IDs in cookies to process personal data."
      • Jesus, Discogs
      • bitmap
        I just checked the JSON dump mirror on hendrix, and it's failing with the reported "index row ..." error
      • reosarevok
        That's... a lot of partners
      • MonkeyPython
        i'd usually not slutshame but
      • bitmap
        however, this DB was already running PG12 in 2019, so I don't think I had actually run the developed upgrade scripts during the schema change
      • huhridge joined the channel
      • Sophist-UK has quit
      • Sophist-UK joined the channel
      • Lotheric_ joined the channel
      • kilos_8 joined the channel
      • Leftmost_ joined the channel
      • d4rk-ph0enix joined the channel
      • kepstin has quit
      • d4rkie has quit
      • Leftmost has quit
      • eharris has quit
      • kilos_ has quit
      • minimal has quit
      • Leftmost_ is now known as Leftmost
      • iconoclast_hero has quit
      • Lotheric has quit
      • kilos_8 is now known as kilos_
      • eharris_ joined the channel
      • minimal joined the channel
      • iconoclast_hero joined the channel
      • kepstin joined the channel
      • v6lur joined the channel
      • mayhem
        reosarevok: should I be preparing for the meeting?
      • yvanzo
        bitmap: I have a mirror with ā€œThe collation in the database was created using version 153.88, but the operating system provides version 153.120.ā€ for testing MBS-13360.
      • BrainzBot
        MBS-13360: Detect which indexes need to be rebuilt for collations https://tickets.metabrainz.org/browse/MBS-13360
      • lotooo has quit
      • yvanzo
        Iā€™m drafting a blog post about now broken replication for those of the long-standing mirrors that still have obsolete indexes.
      • mayhem
        can an instance be fixed or does it require a reload?
      • bitmap
        yvanzo: great, thank you x2
      • lucifer
        mayhem: can you try the tags endpoint on beta.lb and see if it works as expected?
      • yvanzo
        mayhem: it can be fixed with a few DROP INDEX
      • mayhem
        ohhh, thanks lucifer. I will have a look after the meeting.
      • lucifer
        popularity goes from 0 to 100, just to be clear.
      • mayhem
        first things I see, I was hoping to just collapse all the tags into one entity. breaking them out into entities is not actually useful.
      • reosarevok
        mayhem: I'm good, just sat down :)
      • mayhem
        heh, I just got all set.
      • bitmap
        I guess the 2020 schema change steps did work for most people (I think we would've heard from adhawkins by now otherwise :))
      • reosarevok
        <BANG>
      • yvanzo
        ouch
      • reosarevok
        Welcome to another Monday Meeting, this time with me on time and remembering it!
      • MonkeyPython
        šŸ‘