#metabrainz

/

      • aerozol
        I finally found it again! (though it doesn't work anymore)
      • now that MB is looking at making its data *work* I think this is a great method of getting people excited about data. sowing some seeds...
      • Lotheric
        mayhem, when French magazines do an article on life in Quebec or Montreal... They think we live in igloos and have dog sleighs etc
      • stereotypes are funny :)
      • aerozol
        NZ = hobbits. But usually people just get it mixed up with Australia anyway. I don't really mind to be honest, my geography sucks as well
      • intrnl joined the channel
      • intrnl
        seems like the incoming queue on LB has been growing and growing
      • intrnl has quit
      • Shubh joined the channel
      • lucifer
        uh wtf.
      • (psycopg2.errors.DataCorrupted)
      • bitmap: hi! are you around?
      • bitmap
        lucifer: hey, what's up
      • lucifer
        bitmap: debugging issue with LB, i am seeing this strange error in sentry https://sentry.metabrainz.org/organizations/met...
      • initial searches point to database corruption. any ideas what's going on?
      • bitmap
        hrm, that doesn't sound good
      • lucifer
        fwiw, its looks like only the messybrainz database is affected. writes to listenbrainz database are working.
      • bitmap
        this is on floyd?
      • lucifer
        yes
      • bitmap
        let me see if the corruption exists on pink too
      • never seen this before, kinda scary
      • lucifer
        indeed :(
      • bitmap
        since it's an insert I can't reproduce it on pink (read-only) but if I'm following the steps on https://pganalyze.com/docs/log-insights/server/S6 correctly to view the affected area of the data file, it appears to be identical on pink, so I'm not sure performing a failover would help
      • lucifer: is this causing downtime on any service rn?
      • lucifer
        yes listens cannot be inserted in LB due to this.
      • but those will happily pile up in the rmq so its not the end of world.
      • bitmap: you used `sudo xxd /var/lib/docker/volumes/pgdata-floyd/_data/12/main/base/16422/104743859.6 | grep -A 512 a45e000` ?
      • bitmap
        lucifer: yes, but I used the wrong segment number -- now I see the corruption there
      • lucifer
        ah ok.
      • bitmap
        (though I don't know how to interpret what I'm seeing)
      • lucifer
        yeah me neither. i am looking for sample output to compare with.
      • bitmap
        i.e. whether this is a bug in postgres, or some kind of disk failure?
      • lucifer
        oh!
      • bitmap: ^ it appears to be an index that is corrupted, how about dropping it for now?
      • should probably get things working for now
      • goldenshimmer joined the channel
      • bitmap
        lucifer: let's try it and hope for the best
      • lucifer
        šŸ‘
      • bitmap
        other options will likely require downtime :\
      • gcrkrause1 has quit
      • lucifer
        looks it worked. no error in sentry and queue is falling \o/
      • thanks bitmap!
      • gcrkrause1 joined the channel
      • bitmap
        phew! thank god it was an index and not the table itself
      • lucifer
        indeed
      • but probably an underlying issue still exists.
      • bitmap
        still concerned about what caused this, but...yeah
      • zas: atj: ^ just so you're aware, one of the postgres data files on floyd was corrupted somehow, but not sure yet if this was due to an actual hardware issue or a postgres bug
      • BrainzGit
        [musicbrainz-server] 14reosarevok merged pull request #2428 (03masterā€¦document-tests-6): Improve test documentation (VI): Controller::ISRC/ISWC https://github.com/metabrainz/musicbrainz-serve...
      • [musicbrainz-server] 14reosarevok merged pull request #2426 (03masterā€¦document-tests-5): Improve test documentation (V): Controller::Instrument (adds multiple tests) https://github.com/metabrainz/musicbrainz-serve...
      • lucifer
        riksucks: hi! yes, we had considered that idea but think it adds more complexity. we are thinking of instead doing full imports after every few incremental imports and let the db handle the duplicates/
      • riksucks
        I see lucifer, we can also do one thing, since a part of spotify listens are from BrainzPlayer, we can maybe retrieve data from there too
      • lucifer
        riksucks: not sure i understand that. how would that work?
      • riksucks
        lucifer: like, everytime brainzplayer plays something from spotify, we can call an API at the backend?
      • so like, we won't have to completely rely on gradual imports. we would have few data real time
      • lucifer
        but that's unreliable because 1) lot of users probably don't use BU but spotify apps etc. 2) data loss still happens for non BU submitted listens
      • riksucks
        ohh I see, I didn't know that. That makes sense
      • btw lucifer has the gradual import thing been implemented (or is being worked on), I would like to probably work on it
      • lucifer
        riksucks: its currently pending, not started working on it. feel free to work on it. let me see if i can find the previous discussion around it.
      • riksucks
        thanks!
      • alastairp
        !m lucifer
      • BrainzBot
        You're doing good work, lucifer!
      • alastairp
        lucifer: do we have periodic msb dumps?
      • BrainzGit
        [musicbrainz-server] 14rinsuki opened pull request #2435 (03masterā€¦mbs/12226): MBS-12226: Add allowed prefix of niconico video ID https://github.com/metabrainz/musicbrainz-serve...
      • lucifer
        riksucks: https://chatlogs.metabrainz.org/libera/metabrai... may need to scroll a bit. this describes the solution we discussed the last time.
      • alastairp: afaik no. but its on floyd so maybe getting backed up barman.
      • alastairp
        didn't we talk about that at the summit, and we realised that it wasn't? :)
      • lucifer
        i don't recall it distinctly now šŸ˜“
      • i guess we could add msb dumps as well or make it part of barman backup? i do recall that we finally decided except LB listens and AB, everything should be barman backed up
      • alastairp
        yeah, right. I think that dumps are a good idea anyway, maybe people can do something with them
      • lucifer
        btw i asked in the pg channel as well, they too suggested its mostly probably a hardware issue.
      • Mineo has quit
      • Mineo joined the channel
      • yeah dumps sound good then.
      • Mineo has quit
      • alastairp
        lucifer: do you understand this failing test in CB? "AssertionError: Popped wrong app context. (<flask.ctx.AppContext object at 0x7f7017eba460> instead of <flask.ctx.AppContext object at 0x7f7017f20280>)"
      • agatzk has quit
      • agatzk joined the channel
      • BrainzGit
        [critiquebrainz] 14alastair merged pull request #398 (03masterā€¦CB-401-nosuchentity): CB-401, CB-408: Return 404 for entities that don't exist in the MB database https://github.com/metabrainz/critiquebrainz/pu...
      • atj
        zas: FYI I'm dealing with a couple of work issues at the moment, hopefully shouldn't be too long
      • zas
        ok, np
      • Mineo joined the channel
      • bitmap: I checked smart status of disks on floyd (hardware raid), no error were logged, they all look healthy. Also nothing obvious in system logs. That said, it doesn't rule out an hardware issue, it can be anything (ram/cpu/chipset). We'll eventually need to stop this server and run a full hardware check from rescue mode. Ping me when you're around.
      • monkey
        aerozol: showing the user's total number of listens in the column to the right sound like a good idea indeed. We currently show it on the page but it would definitely be better on the right: https://usercontent.irccloud-cdn.com/file/bHeVn...
      • And maybe we could add a simple monthly listens total as a low hanging fruit?
      • lucifer: Do we have an easy way to get a user's listen count for the current month?
      • mayhem
        moooin!
      • mayhem is getting a late start today
      • monkey
        aerozol: We also talked a while back about having a calendar-based navigation similar (but maybe not quite the same) to the one lastfm uses, that would be another good candidate to have on the right hand side column https://usercontent.irccloud-cdn.com/file/QMOzd...
      • I love this component of the year in music report we made, and a time navigation based on that would be pretty rad https://usercontent.irccloud-cdn.com/file/YTLEX...
      • aerozol
        really cool, there's heaps of stuff from reports + charts that could be pulled over, but might need some tweaking to not just be obviously doubling up? some kind of summary?
      • this is what I meant re listen stats by the way, if the pin/now listening moves to the right it could be an opportunity to add some track-specific stats/data: https://raw.githubusercontent.com/Aerozol/Test/...
      • but total listens etc is definitely an important e-wang kind of thing to put at the top
      • anyway, night all!
      • monkey
        Ah, i see. I don't think we currently have those stats available, but that would indeed be nice
      • Night night!
      • zas
        mayhem: moooin
      • mayhem
        ok, time to spend a pile of money. woo!
      • riksucks
        hi monkey , so should I just put `title={ariaLabel}` inside the button component of `ListenControl`?
      • or do you want me to pass title of the button in some other manner
      • mayhem
        akshaaatt: zas: yvanzo: alastairp : invoices please!
      • alastairp
        yes sir
      • akshaaatt
        On it mayhem
      • monkey
        riksucks: That would solve the issue at hand, but seems confusing.
      • I'd prefer, if it's not too much work, to have a separate `text` property for the rendered text, and only use title for the tile property. How does that sound to you?
      • BrainzGit
        [brainzutils-python] 14alastair opened pull request #80 (03masterā€¦musicbrainzdb-docs): Clarify return values in musicbrainz_db documentation https://github.com/metabrainz/brainzutils-pytho...
      • lucifer
        alastairp: uh weird, i think i have seen that before not sure where. maybe something is created another flask app inside that but why is it erring now would be a mystery then. :/ maybe try and see output with https://stackoverflow.com/a/28139033
      • alastairp
        lucifer: yeah, but it doesn't fail for me locally either. it _could_ be related somehow to cached dependencies or versions in the test workflow, but who knows
      • I'll see if it keeps on coming up as I merge other stuff
      • lucifer
        monkey: re get a user's listen count for the current month?, yes should be doable.
      • alastairp: oh. makes sense.
      • mayhem, alastairp: oh! i forgot. i had dropped the index of the messybrainz table in the morning but didn't recreate it later because afaik we don't use it currently. it was the index on meta hash, we use data hash for dedup purposes. thoughts on whether to recreate it?
      • alastairp
        what was the meta hash? it's a hash over a smaller set of the provided fields, right?
      • lucifer
        yes
      • just the title and artist name.
      • alastairp
        I think the idea was to be able to do a lookup based on just artist/recording name to see if we should return a dup msid
      • but with the mapping table, that's not used. we could consider simplifying the messybrainz db too - perhaps some things like this column can be removed
      • atj
        zas: just pushed a commit with a new role to manage /etc/aliases
      • lucifer
        yes makes sense.
      • alastairp
        mayhem: lucifer: I'm fixing some bugs in CB and keep coming across broken code for the spotify mapping. In light of our other mapping projects, maybe this part should be removed and replaced with somethign better. what do you think?
      • mayhem
        nuke it. spend no time on it.
      • lucifer
        alastairp, yeah makes sense to remove it. i think the replacement would be BP.
      • alastairp
        šŸ‘ agreed
      • Ansh: let me know if you want a new CB task :)
      • atj
        zas: I see some non-zero uncorrectable errors / bad blocks / reallocated sectors on the Samsung SSDs on floyd, however I'm not convinced the SMART data attributes are correct
      • zas
        Nor I, a full offline check might tell more
      • alastairp
        yvanzo: hi there, I understand you've been working on the new weblate setup, can we talk about it this afternoon or later this week (for critiquebrainz)
      • riksucks
        monkey: that's exactly what I had in mind, but I wanted to know your approach to be double sure. Thanks btw!
      • monkey
        Thank you !
      • BrainzGit
        [bookbrainz-site] 14MonkeyDo merged pull request #779 (03masterā€¦search-page-empty-query): fix(search page): Allow empty search queries https://github.com/metabrainz/bookbrainz-site/p...
      • [bookbrainz-site] 14MonkeyDo merged pull request #777 (03masterā€¦dependabot/npm_and_yarn/superagent-7.1.1): chore(deps): bump superagent from 6.1.0 to 7.1.1 https://github.com/metabrainz/bookbrainz-site/p...
      • [bookbrainz-site] 14dependabot[bot] opened pull request #788 (03masterā€¦dependabot/npm_and_yarn/babel/register-7.17.0): chore(deps-dev): bump @babel/register from 7.16.0 to 7.17.0 https://github.com/metabrainz/bookbrainz-site/p...
      • [bookbrainz-site] 14MonkeyDo closed pull request #762 (03masterā€¦dependabot/npm_and_yarn/faker-6.6.6): chore(deps-dev): bump faker from 4.1.0 to 6.6.6 https://github.com/metabrainz/bookbrainz-site/p...
      • monkey
      • Let me know if you want to discuss any of this this afternoon
      • akshaaatt
        Hi monkey! Thanks for the reminder regarding this :) Will get this done in a few hours
      • monkey
        No rush, just wanted to see if there was a blocker
      • akshaaatt
        No blocker, just needs to be worked on
      • monkey
        šŸ‘
      • Happy to help if needs be
      • akshaaatt
        Thank you!
      • Ansh
        alastairp: For the commit 'Don't show a review if the entity that it reviews isn't in the database', If we are hiding the reviews, we have to find a way to adjust the pagination some way. I work on the sample MB data, and now it is showing total of 6 reviews but the pagination on bottom shows that there are 372 pages.
      • alastairp
        Ansh: yes, you're right. I added a comment to the code indicating that this happens in very few cases, so I didn't want to add extra code for something that isn't used very often
      • but I did realise that perhaps during development it might cause problems with showing reviews? I have an entire musicbrainz database with my CB environment, but if you only have a small subset database then you won't be able to see as much - is that what you're seeing?
      • Ansh
        Yes that was the problem I was facing