fwiw, its looks like only the messybrainz database is affected. writes to listenbrainz database are working.
2022-03-01 06004, 2022
bitmap
this is on floyd?
2022-03-01 06007, 2022
lucifer
yes
2022-03-01 06041, 2022
bitmap
let me see if the corruption exists on pink too
2022-03-01 06012, 2022
bitmap
never seen this before, kinda scary
2022-03-01 06000, 2022
lucifer
indeed :(
2022-03-01 06045, 2022
bitmap
since it's an insert I can't reproduce it on pink (read-only) but if I'm following the steps on https://pganalyze.com/docs/log-insights/server/S6 correctly to view the affected area of the data file, it appears to be identical on pink, so I'm not sure performing a failover would help
2022-03-01 06041, 2022
bitmap
lucifer: is this causing downtime on any service rn?
2022-03-01 06000, 2022
lucifer
yes listens cannot be inserted in LB due to this.
2022-03-01 06038, 2022
lucifer
but those will happily pile up in the rmq so its not the end of world.
2022-03-01 06018, 2022
lucifer
bitmap: you used `sudo xxd /var/lib/docker/volumes/pgdata-floyd/_data/12/main/base/16422/104743859.6 | grep -A 512 a45e000` ?
2022-03-01 06022, 2022
bitmap
lucifer: yes, but I used the wrong segment number -- now I see the corruption there
2022-03-01 06004, 2022
lucifer
ah ok.
2022-03-01 06013, 2022
bitmap
(though I don't know how to interpret what I'm seeing)
2022-03-01 06040, 2022
lucifer
yeah me neither. i am looking for sample output to compare with.
2022-03-01 06043, 2022
bitmap
i.e. whether this is a bug in postgres, or some kind of disk failure?
bitmap: ^ it appears to be an index that is corrupted, how about dropping it for now?
2022-03-01 06017, 2022
lucifer
should probably get things working for now
2022-03-01 06004, 2022
goldenshimmer joined the channel
2022-03-01 06033, 2022
bitmap
lucifer: let's try it and hope for the best
2022-03-01 06043, 2022
lucifer
š
2022-03-01 06050, 2022
bitmap
other options will likely require downtime :\
2022-03-01 06000, 2022
gcrkrause1 has quit
2022-03-01 06049, 2022
lucifer
looks it worked. no error in sentry and queue is falling \o/
2022-03-01 06029, 2022
lucifer
thanks bitmap!
2022-03-01 06033, 2022
gcrkrause1 joined the channel
2022-03-01 06043, 2022
bitmap
phew! thank god it was an index and not the table itself
2022-03-01 06053, 2022
lucifer
indeed
2022-03-01 06009, 2022
lucifer
but probably an underlying issue still exists.
2022-03-01 06017, 2022
bitmap
still concerned about what caused this, but...yeah
2022-03-01 06048, 2022
bitmap
zas: atj: ^ just so you're aware, one of the postgres data files on floyd was corrupted somehow, but not sure yet if this was due to an actual hardware issue or a postgres bug
riksucks: hi! yes, we had considered that idea but think it adds more complexity. we are thinking of instead doing full imports after every few incremental imports and let the db handle the duplicates/
2022-03-01 06024, 2022
riksucks
I see lucifer, we can also do one thing, since a part of spotify listens are from BrainzPlayer, we can maybe retrieve data from there too
2022-03-01 06020, 2022
lucifer
riksucks: not sure i understand that. how would that work?
2022-03-01 06046, 2022
riksucks
lucifer: like, everytime brainzplayer plays something from spotify, we can call an API at the backend?
2022-03-01 06016, 2022
riksucks
so like, we won't have to completely rely on gradual imports. we would have few data real time
2022-03-01 06058, 2022
lucifer
but that's unreliable because 1) lot of users probably don't use BU but spotify apps etc. 2) data loss still happens for non BU submitted listens
2022-03-01 06047, 2022
riksucks
ohh I see, I didn't know that. That makes sense
2022-03-01 06038, 2022
riksucks
btw lucifer has the gradual import thing been implemented (or is being worked on), I would like to probably work on it
2022-03-01 06045, 2022
lucifer
riksucks: its currently pending, not started working on it. feel free to work on it. let me see if i can find the previous discussion around it.
alastairp: afaik no. but its on floyd so maybe getting backed up barman.
2022-03-01 06027, 2022
alastairp
didn't we talk about that at the summit, and we realised that it wasn't? :)
2022-03-01 06059, 2022
lucifer
i don't recall it distinctly now š
2022-03-01 06022, 2022
lucifer
i guess we could add msb dumps as well or make it part of barman backup? i do recall that we finally decided except LB listens and AB, everything should be barman backed up
2022-03-01 06056, 2022
alastairp
yeah, right. I think that dumps are a good idea anyway, maybe people can do something with them
2022-03-01 06038, 2022
lucifer
btw i asked in the pg channel as well, they too suggested its mostly probably a hardware issue.
2022-03-01 06057, 2022
Mineo has quit
2022-03-01 06005, 2022
Mineo joined the channel
2022-03-01 06054, 2022
lucifer
yeah dumps sound good then.
2022-03-01 06010, 2022
Mineo has quit
2022-03-01 06015, 2022
alastairp
lucifer: do you understand this failing test in CB? "AssertionError: Popped wrong app context. (<flask.ctx.AppContext object at 0x7f7017eba460> instead of <flask.ctx.AppContext object at 0x7f7017f20280>)"
[critiquebrainz] 14alastair merged pull request #398 (03masterā¦CB-401-nosuchentity): CB-401, CB-408: Return 404 for entities that don't exist in the MB database https://github.com/metabrainz/critiquebrainz/pullā¦
2022-03-01 06034, 2022
atj
zas: FYI I'm dealing with a couple of work issues at the moment, hopefully shouldn't be too long
2022-03-01 06051, 2022
zas
ok, np
2022-03-01 06002, 2022
Mineo joined the channel
2022-03-01 06014, 2022
zas
bitmap: I checked smart status of disks on floyd (hardware raid), no error were logged, they all look healthy. Also nothing obvious in system logs. That said, it doesn't rule out an hardware issue, it can be anything (ram/cpu/chipset). We'll eventually need to stop this server and run a full hardware check from rescue mode. Ping me when you're around.
2022-03-01 06015, 2022
monkey
aerozol: showing the user's total number of listens in the column to the right sound like a good idea indeed. We currently show it on the page but it would definitely be better on the right: https://usercontent.irccloud-cdn.com/file/bHeVnPoā¦
2022-03-01 06055, 2022
monkey
And maybe we could add a simple monthly listens total as a low hanging fruit?
2022-03-01 06053, 2022
monkey
lucifer: Do we have an easy way to get a user's listen count for the current month?
2022-03-01 06058, 2022
mayhem
moooin!
2022-03-01 06005, 2022
mayhem is getting a late start today
2022-03-01 06013, 2022
monkey
aerozol: We also talked a while back about having a calendar-based navigation similar (but maybe not quite the same) to the one lastfm uses, that would be another good candidate to have on the right hand side column https://usercontent.irccloud-cdn.com/file/QMOzd69ā¦
really cool, there's heaps of stuff from reports + charts that could be pulled over, but might need some tweaking to not just be obviously doubling up? some kind of summary?
2022-03-01 06053, 2022
aerozol
this is what I meant re listen stats by the way, if the pin/now listening moves to the right it could be an opportunity to add some track-specific stats/data: https://raw.githubusercontent.com/Aerozol/Test/maā¦
2022-03-01 06025, 2022
aerozol
but total listens etc is definitely an important e-wang kind of thing to put at the top
2022-03-01 06031, 2022
aerozol
anyway, night all!
2022-03-01 06014, 2022
monkey
Ah, i see. I don't think we currently have those stats available, but that would indeed be nice
2022-03-01 06020, 2022
monkey
Night night!
2022-03-01 06051, 2022
zas
mayhem: moooin
2022-03-01 06014, 2022
mayhem
ok, time to spend a pile of money. woo!
2022-03-01 06032, 2022
riksucks
hi monkey , so should I just put `title={ariaLabel}` inside the button component of `ListenControl`?
2022-03-01 06033, 2022
riksucks
or do you want me to pass title of the button in some other manner
riksucks: That would solve the issue at hand, but seems confusing.
2022-03-01 06041, 2022
monkey
I'd prefer, if it's not too much work, to have a separate `text` property for the rendered text, and only use title for the tile property. How does that sound to you?
alastairp: uh weird, i think i have seen that before not sure where. maybe something is created another flask app inside that but why is it erring now would be a mystery then. :/ maybe try and see output with https://stackoverflow.com/a/28139033
2022-03-01 06041, 2022
alastairp
lucifer: yeah, but it doesn't fail for me locally either. it _could_ be related somehow to cached dependencies or versions in the test workflow, but who knows
2022-03-01 06048, 2022
alastairp
I'll see if it keeps on coming up as I merge other stuff
2022-03-01 06053, 2022
lucifer
monkey: re get a user's listen count for the current month?, yes should be doable.
2022-03-01 06011, 2022
lucifer
alastairp: oh. makes sense.
2022-03-01 06049, 2022
lucifer
mayhem, alastairp: oh! i forgot. i had dropped the index of the messybrainz table in the morning but didn't recreate it later because afaik we don't use it currently. it was the index on meta hash, we use data hash for dedup purposes. thoughts on whether to recreate it?
2022-03-01 06027, 2022
alastairp
what was the meta hash? it's a hash over a smaller set of the provided fields, right?
2022-03-01 06032, 2022
lucifer
yes
2022-03-01 06041, 2022
lucifer
just the title and artist name.
2022-03-01 06052, 2022
alastairp
I think the idea was to be able to do a lookup based on just artist/recording name to see if we should return a dup msid
2022-03-01 06024, 2022
alastairp
but with the mapping table, that's not used. we could consider simplifying the messybrainz db too - perhaps some things like this column can be removed
2022-03-01 06001, 2022
atj
zas: just pushed a commit with a new role to manage /etc/aliases
2022-03-01 06002, 2022
lucifer
yes makes sense.
2022-03-01 06051, 2022
alastairp
mayhem: lucifer: I'm fixing some bugs in CB and keep coming across broken code for the spotify mapping. In light of our other mapping projects, maybe this part should be removed and replaced with somethign better. what do you think?
2022-03-01 06023, 2022
mayhem
nuke it. spend no time on it.
2022-03-01 06032, 2022
lucifer
alastairp, yeah makes sense to remove it. i think the replacement would be BP.
2022-03-01 06034, 2022
alastairp
š agreed
2022-03-01 06042, 2022
alastairp
Ansh: let me know if you want a new CB task :)
2022-03-01 06031, 2022
atj
zas: I see some non-zero uncorrectable errors / bad blocks / reallocated sectors on the Samsung SSDs on floyd, however I'm not convinced the SMART data attributes are correct
2022-03-01 06032, 2022
zas
Nor I, a full offline check might tell more
2022-03-01 06038, 2022
alastairp
yvanzo: hi there, I understand you've been working on the new weblate setup, can we talk about it this afternoon or later this week (for critiquebrainz)
2022-03-01 06056, 2022
riksucks
monkey: that's exactly what I had in mind, but I wanted to know your approach to be double sure. Thanks btw!
[bookbrainz-site] 14MonkeyDo merged pull request #777 (03masterā¦dependabot/npm_and_yarn/superagent-7.1.1): chore(deps): bump superagent from 6.1.0 to 7.1.1 https://github.com/metabrainz/bookbrainz-site/pulā¦
2022-03-01 06037, 2022
BrainzGit
[bookbrainz-site] 14dependabot[bot] opened pull request #788 (03masterā¦dependabot/npm_and_yarn/babel/register-7.17.0): chore(deps-dev): bump @babel/register from 7.16.0 to 7.17.0 https://github.com/metabrainz/bookbrainz-site/pulā¦
Let me know if you want to discuss any of this this afternoon
2022-03-01 06018, 2022
akshaaatt
Hi monkey! Thanks for the reminder regarding this :) Will get this done in a few hours
2022-03-01 06023, 2022
monkey
No rush, just wanted to see if there was a blocker
2022-03-01 06035, 2022
akshaaatt
No blocker, just needs to be worked on
2022-03-01 06032, 2022
monkey
š
2022-03-01 06042, 2022
monkey
Happy to help if needs be
2022-03-01 06044, 2022
akshaaatt
Thank you!
2022-03-01 06025, 2022
Ansh
alastairp: For the commit 'Don't show a review if the entity that it reviews isn't in the database', If we are hiding the reviews, we have to find a way to adjust the pagination some way. I work on the sample MB data, and now it is showing total of 6 reviews but the pagination on bottom shows that there are 372 pages.
2022-03-01 06023, 2022
alastairp
Ansh: yes, you're right. I added a comment to the code indicating that this happens in very few cases, so I didn't want to add extra code for something that isn't used very often
2022-03-01 06015, 2022
alastairp
but I did realise that perhaps during development it might cause problems with showing reviews? I have an entire musicbrainz database with my CB environment, but if you only have a small subset database then you won't be able to see as much - is that what you're seeing?
