#metabrainz

I finally found it again! (though it doesn't work anymore)

now that MB is looking at making its data *work* I think this is a great method of getting people excited about data. sowing some seeds...

mayhem, when French magazines do an article on life in Quebec or Montreal... They think we live in igloos and have dog sleighs etc

stereotypes are funny :)

NZ = hobbits. But usually people just get it mixed up with Australia anyway. I don't really mind to be honest, my geography sucks as well

seems like the incoming queue on LB has been growing and growing

uh wtf.

(psycopg2.errors.DataCorrupted)

bitmap: hi! are you around?

lucifer: hey, what's up

bitmap: debugging issue with LB, i am seeing this strange error in sentry https://sentry.metabrainz.org/organizations/met...

initial searches point to database corruption. any ideas what's going on?

hrm, that doesn't sound good

fwiw, its looks like only the messybrainz database is affected. writes to listenbrainz database are working.

this is on floyd?

yes

let me see if the corruption exists on pink too

never seen this before, kinda scary

indeed :(

since it's an insert I can't reproduce it on pink (read-only) but if I'm following the steps on https://pganalyze.com/docs/log-insights/server/S6 correctly to view the affected area of the data file, it appears to be identical on pink, so I'm not sure performing a failover would help

lucifer: is this causing downtime on any service rn?

yes listens cannot be inserted in LB due to this.

but those will happily pile up in the rmq so its not the end of world.

bitmap: you used `sudo xxd /var/lib/docker/volumes/pgdata-floyd/_data/12/main/base/16422/104743859.6 | grep -A 512 a45e000` ?

lucifer: yes, but I used the wrong segment number -- now I see the corruption there

ah ok.

(though I don't know how to interpret what I'm seeing)

yeah me neither. i am looking for sample output to compare with.

i.e. whether this is a bug in postgres, or some kind of disk failure?

oh!

bitmap: ^ it appears to be an index that is corrupted, how about dropping it for now?

should probably get things working for now

lucifer: let's try it and hope for the best

👍

other options will likely require downtime :\

looks it worked. no error in sentry and queue is falling \o/

thanks bitmap!

phew! thank god it was an index and not the table itself

indeed

but probably an underlying issue still exists.

still concerned about what caused this, but...yeah

zas: atj: ^ just so you're aware, one of the postgres data files on floyd was corrupted somehow, but not sure yet if this was due to an actual hardware issue or a postgres bug

[musicbrainz-server] 14reosarevok merged pull request #2428 (03master…document-tests-6): Improve test documentation (VI): Controller::ISRC/ISWC https://github.com/metabrainz/musicbrainz-serve...

[musicbrainz-server] 14reosarevok merged pull request #2426 (03master…document-tests-5): Improve test documentation (V): Controller::Instrument (adds multiple tests) https://github.com/metabrainz/musicbrainz-serve...

riksucks: hi! yes, we had considered that idea but think it adds more complexity. we are thinking of instead doing full imports after every few incremental imports and let the db handle the duplicates/

I see lucifer, we can also do one thing, since a part of spotify listens are from BrainzPlayer, we can maybe retrieve data from there too

riksucks: not sure i understand that. how would that work?

lucifer: like, everytime brainzplayer plays something from spotify, we can call an API at the backend?

so like, we won't have to completely rely on gradual imports. we would have few data real time

but that's unreliable because 1) lot of users probably don't use BU but spotify apps etc. 2) data loss still happens for non BU submitted listens

ohh I see, I didn't know that. That makes sense

btw lucifer has the gradual import thing been implemented (or is being worked on), I would like to probably work on it

riksucks: its currently pending, not started working on it. feel free to work on it. let me see if i can find the previous discussion around it.

thanks!

!m lucifer

You're doing good work, lucifer!

lucifer: do we have periodic msb dumps?

[musicbrainz-server] 14rinsuki opened pull request #2435 (03master…mbs/12226): MBS-12226: Add allowed prefix of niconico video ID https://github.com/metabrainz/musicbrainz-serve...

riksucks: https://chatlogs.metabrainz.org/libera/metabrai... may need to scroll a bit. this describes the solution we discussed the last time.

alastairp: afaik no. but its on floyd so maybe getting backed up barman.

didn't we talk about that at the summit, and we realised that it wasn't? :)

i don't recall it distinctly now 😓

i guess we could add msb dumps as well or make it part of barman backup? i do recall that we finally decided except LB listens and AB, everything should be barman backed up

yeah, right. I think that dumps are a good idea anyway, maybe people can do something with them

btw i asked in the pg channel as well, they too suggested its mostly probably a hardware issue.

yeah dumps sound good then.

lucifer: do you understand this failing test in CB? "AssertionError: Popped wrong app context. (<flask.ctx.AppContext object at 0x7f7017eba460> instead of <flask.ctx.AppContext object at 0x7f7017f20280>)"

https://github.com/metabrainz/critiquebrainz/ac... it even failed when I re-ran it

[critiquebrainz] 14alastair merged pull request #398 (03master…CB-401-nosuchentity): CB-401, CB-408: Return 404 for entities that don't exist in the MB database https://github.com/metabrainz/critiquebrainz/pu...

zas: FYI I'm dealing with a couple of work issues at the moment, hopefully shouldn't be too long

ok, np

bitmap: I checked smart status of disks on floyd (hardware raid), no error were logged, they all look healthy. Also nothing obvious in system logs. That said, it doesn't rule out an hardware issue, it can be anything (ram/cpu/chipset). We'll eventually need to stop this server and run a full hardware check from rescue mode. Ping me when you're around.

aerozol: showing the user's total number of listens in the column to the right sound like a good idea indeed. We currently show it on the page but it would definitely be better on the right: https://usercontent.irccloud-cdn.com/file/bHeVn...

And maybe we could add a simple monthly listens total as a low hanging fruit?

lucifer: Do we have an easy way to get a user's listen count for the current month?

moooin!

aerozol: We also talked a while back about having a calendar-based navigation similar (but maybe not quite the same) to the one lastfm uses, that would be another good candidate to have on the right hand side column https://usercontent.irccloud-cdn.com/file/QMOzd...

I love this component of the year in music report we made, and a time navigation based on that would be pretty rad https://usercontent.irccloud-cdn.com/file/YTLEX...

really cool, there's heaps of stuff from reports + charts that could be pulled over, but might need some tweaking to not just be obviously doubling up? some kind of summary?

this is what I meant re listen stats by the way, if the pin/now listening moves to the right it could be an opportunity to add some track-specific stats/data: https://raw.githubusercontent.com/Aerozol/Test/...

but total listens etc is definitely an important e-wang kind of thing to put at the top

anyway, night all!

Ah, i see. I don't think we currently have those stats available, but that would indeed be nice

Night night!

mayhem: moooin

ok, time to spend a pile of money. woo!

hi monkey , so should I just put `title={ariaLabel}` inside the button component of `ListenControl`?

or do you want me to pass title of the button in some other manner

akshaaatt: zas: yvanzo: alastairp : invoices please!

yes sir

On it mayhem

riksucks: That would solve the issue at hand, but seems confusing.

I'd prefer, if it's not too much work, to have a separate `text` property for the rendered text, and only use title for the tile property. How does that sound to you?

[brainzutils-python] 14alastair opened pull request #80 (03master…musicbrainzdb-docs): Clarify return values in musicbrainz_db documentation https://github.com/metabrainz/brainzutils-pytho...

alastairp: uh weird, i think i have seen that before not sure where. maybe something is created another flask app inside that but why is it erring now would be a mystery then. :/ maybe try and see output with https://stackoverflow.com/a/28139033

lucifer: yeah, but it doesn't fail for me locally either. it _could_ be related somehow to cached dependencies or versions in the test workflow, but who knows

I'll see if it keeps on coming up as I merge other stuff

monkey: re get a user's listen count for the current month?, yes should be doable.

alastairp: oh. makes sense.

mayhem, alastairp: oh! i forgot. i had dropped the index of the messybrainz table in the morning but didn't recreate it later because afaik we don't use it currently. it was the index on meta hash, we use data hash for dedup purposes. thoughts on whether to recreate it?

what was the meta hash? it's a hash over a smaller set of the provided fields, right?

yes

just the title and artist name.

I think the idea was to be able to do a lookup based on just artist/recording name to see if we should return a dup msid

but with the mapping table, that's not used. we could consider simplifying the messybrainz db too - perhaps some things like this column can be removed

zas: just pushed a commit with a new role to manage /etc/aliases

yes makes sense.

mayhem: lucifer: I'm fixing some bugs in CB and keep coming across broken code for the spotify mapping. In light of our other mapping projects, maybe this part should be removed and replaced with somethign better. what do you think?

nuke it. spend no time on it.

alastairp, yeah makes sense to remove it. i think the replacement would be BP.

👍 agreed

Ansh: let me know if you want a new CB task :)

zas: I see some non-zero uncorrectable errors / bad blocks / reallocated sectors on the Samsung SSDs on floyd, however I'm not convinced the SMART data attributes are correct

Nor I, a full offline check might tell more

yvanzo: hi there, I understand you've been working on the new weblate setup, can we talk about it this afternoon or later this week (for critiquebrainz)

monkey: that's exactly what I had in mind, but I wanted to know your approach to be double sure. Thanks btw!

Thank you !

[bookbrainz-site] 14MonkeyDo merged pull request #779 (03master…search-page-empty-query): fix(search page): Allow empty search queries https://github.com/metabrainz/bookbrainz-site/p...

[bookbrainz-site] 14MonkeyDo merged pull request #777 (03master…dependabot/npm_and_yarn/superagent-7.1.1): chore(deps): bump superagent from 6.1.0 to 7.1.1 https://github.com/metabrainz/bookbrainz-site/p...

[bookbrainz-site] 14dependabot[bot] opened pull request #788 (03master…dependabot/npm_and_yarn/babel/register-7.17.0): chore(deps-dev): bump @babel/register from 7.16.0 to 7.17.0 https://github.com/metabrainz/bookbrainz-site/p...

[bookbrainz-site] 14MonkeyDo closed pull request #762 (03master…dependabot/npm_and_yarn/faker-6.6.6): chore(deps-dev): bump faker from 4.1.0 to 6.6.6 https://github.com/metabrainz/bookbrainz-site/p...

akshaaatt: Friendly ping about https://github.com/metabrainz/bookbrainz-site/p...

Let me know if you want to discuss any of this this afternoon

Hi monkey! Thanks for the reminder regarding this :) Will get this done in a few hours

No rush, just wanted to see if there was a blocker

No blocker, just needs to be worked on

👍

Happy to help if needs be

Thank you!

alastairp: For the commit 'Don't show a review if the entity that it reviews isn't in the database', If we are hiding the reviews, we have to find a way to adjust the pagination some way. I work on the sample MB data, and now it is showing total of 6 reviews but the pagination on bottom shows that there are 372 pages.

Ansh: yes, you're right. I added a comment to the code indicating that this happens in very few cases, so I didn't want to add extra code for something that isn't used very often

but I did realise that perhaps during development it might cause problems with showing reviews? I have an entire musicbrainz database with my CB environment, but if you only have a small subset database then you won't be able to see as much - is that what you're seeing?

Yes that was the problem I was facing