20:05 PM
BrainzGit
2023-11-02 30610, 2023
20:06 PM
mayhem
thx!
2023-11-02 30641, 2023
20:06 PM
lucifer
this shouldn't change anything but lets' see
2023-11-02 30658, 2023
20:06 PM
lucifer
because the actual image was built and pushed to dockerhub 2 days ago
2023-11-02 30620, 2023
20:08 PM
mayhem
hm. random CB tests failing on my pr. seems unrelated.
2023-11-02 30628, 2023
20:14 PM
mayhem
lucifer: spark tests still failing. I merged your branch into mine
2023-11-02 30631, 2023
20:14 PM
mayhem
2023-11-02 30610, 2023
20:15 PM
lucifer
mayhem: yes i the previous runs, its a new error. looking into it.
2023-11-02 30613, 2023
20:15 PM
lucifer
*saw the
2023-11-02 30608, 2023
20:33 PM
aerozol
reosarevok: had to get this out of my head, been floating around for months… if we could assign a ‘product’ attribute to works then I think we could display something like this without having to get new data? It’s already all there, just doesn’t display in a way that’s useful for me
https://usercontent.irccloud-cdn.com/file/bT0ubPr…
2023-11-02 30631, 2023
20:33 PM
aerozol
The niggly bit would be the year, but this would be useful for me even if that column didn’t exist
2023-11-02 30652, 2023
20:46 PM
BrainzGit
2023-11-02 30638, 2023
20:47 PM
monkey
There an issue with LB, I get this when I try to load my dashboard:
2023-11-02 30642, 2023
20:47 PM
monkey
2023-11-02 30656, 2023
20:47 PM
mayhem
ok, shifted work-day, but I still got stuff done. except paying some stragglers.
2023-11-02 30659, 2023
20:47 PM
mayhem
bitmap: ^^
2023-11-02 30607, 2023
20:48 PM
mayhem
on prod, monkey ?
2023-11-02 30613, 2023
20:48 PM
monkey
yvanzo:
2023-11-02 30619, 2023
20:48 PM
monkey
Woopp,s sorry for the ping
2023-11-02 30623, 2023
20:48 PM
monkey
I mean Yes
2023-11-02 30630, 2023
20:48 PM
mayhem
seems related to the DB move.
2023-11-02 30647, 2023
20:48 PM
mayhem
yep. got the same
2023-11-02 30617, 2023
20:50 PM
mayhem restarts the web container
2023-11-02 30643, 2023
20:50 PM
mayhem
zas: atj : bitmap : we've got some DB issues!
2023-11-02 30651, 2023
20:50 PM
bitmap
I'll stop some of the pgbouncer instances for now
2023-11-02 30659, 2023
20:50 PM
bitmap
we might have to increase pg's max_connections
2023-11-02 30608, 2023
20:51 PM
mayhem
MB seems unaffected.
2023-11-02 30629, 2023
20:52 PM
mayhem is all for raising number of connections.
2023-11-02 30644, 2023
20:52 PM
mayhem
they are not expensive, so why are we so conservative with them?
2023-11-02 30621, 2023
20:53 PM
bitmap
yeah, we should next time we restart PG
2023-11-02 30610, 2023
20:54 PM
mayhem
the problem is still happening.
2023-11-02 30646, 2023
20:54 PM
mayhem
anything I can help with bitmap?
2023-11-02 30604, 2023
20:57 PM
bitmap
I stopped the pgbouncer on hendrix which should reduce the # of connections
2023-11-02 30622, 2023
20:57 PM
bitmap
hard to tell what's going on though 'cause I think the pgbouncer stats aren't working since the move
2023-11-02 30636, 2023
20:57 PM
mayhem
no change. should I restart the web container again?
2023-11-02 30623, 2023
20:58 PM
bitmap
you can try, I'll try increasing LB's connection allotment in the pgbouncer config
2023-11-02 30620, 2023
20:59 PM
mayhem
no change. :(
2023-11-02 30633, 2023
20:59 PM
monkey
Some alerts on telegram about gaga. Related?
2023-11-02 30611, 2023
21:00 PM
mayhem
ah, we're looking at the wrong db. :)
2023-11-02 30639, 2023
21:00 PM
mayhem
bitmap: its gaga, not jimmy/hendrix
2023-11-02 30643, 2023
21:00 PM
mayhem
lucifer: ping
2023-11-02 30650, 2023
21:00 PM
mayhem
gaga PG is quite unhappy.
2023-11-02 30604, 2023
21:01 PM
bitmap
ah, no wonder nothing was taking
2023-11-02 30605, 2023
21:01 PM
lucifer
pong
2023-11-02 30616, 2023
21:01 PM
mayhem
gaga's PG is very unhappy.
2023-11-02 30620, 2023
21:01 PM
mayhem
help me look , please?
2023-11-02 30642, 2023
21:01 PM
lucifer
sure, unhappy as in?
2023-11-02 30628, 2023
21:02 PM
bitmap undoes the main cluster changes then
2023-11-02 30639, 2023
21:02 PM
mayhem
high load, many PG procs.
2023-11-02 30652, 2023
21:02 PM
mayhem
2023-11-02 30606, 2023
21:03 PM
mayhem
this error was found a lot in the logs before we ran out of connections.
2023-11-02 30614, 2023
21:03 PM
mayhem
that might be eating the connections.
2023-11-02 30616, 2023
21:03 PM
lucifer
2023-06-06 02:38:15.409 UTC [3991072] STATEMENT:
2023-11-02 30621, 2023
21:03 PM
lucifer
5 months ago
2023-11-02 30659, 2023
21:03 PM
mayhem
went too far, lol.
2023-11-02 30624, 2023
21:04 PM
mayhem
2023-11-02 30632, 2023
21:04 PM
mayhem
ok, this was in the logs before.
2023-11-02 30643, 2023
21:04 PM
mayhem
do we have a spark process or anything writing to gaga right now?
2023-11-02 30601, 2023
21:06 PM
lucifer
not really
2023-11-02 30616, 2023
21:06 PM
lucifer
checking pg_stat_activity i see mostly listen read requests
2023-11-02 30653, 2023
21:06 PM
mayhem
we should check where the connections are coming from.
2023-11-02 30658, 2023
21:06 PM
mayhem
how do we do that?
2023-11-02 30610, 2023
21:07 PM
lucifer
its back to normal now i think
2023-11-02 30644, 2023
21:07 PM
mayhem
did you change anything?
2023-11-02 30658, 2023
21:07 PM
lucifer
nope
2023-11-02 30604, 2023
21:08 PM
mayhem
odd. wtf?
2023-11-02 30620, 2023
21:08 PM
lucifer
16824 | listenbrainz_ts | 1179026 | | | | | | | | 2023-11-02 20:28:19.08004+00 | 2023-11-02 20:42:10.394548+00 | 2023-11-02 20:42:10.394548+00 | 2023-11-02 20:42:10.394548+00 | Timeout | VacuumDelay | active | | 406250699 | autovacuum: VACUUM ANALYZE
2023-11-02 30620, 2023
21:08 PM
lucifer
_timescaledb_internal._hyper_41_6573_chunk | autovacuum worker
2023-11-02 30631, 2023
21:08 PM
lucifer
a very wild guess.
2023-11-02 30648, 2023
21:08 PM
mayhem
plausible.
2023-11-02 30651, 2023
21:08 PM
lucifer
autovaccum locked listen table, leading to accumulation of connections
2023-11-02 30600, 2023
21:09 PM
mayhem nods
2023-11-02 30607, 2023
21:09 PM
lucifer
but then autovaccum timed out so it went back to normal again
2023-11-02 30620, 2023
21:09 PM
lucifer
so went from immediate bad to long term bad
2023-11-02 30622, 2023
21:09 PM
monkey
Although LB is accessible, it shows that I haven(t listened to anything on my profile
2023-11-02 30623, 2023
21:09 PM
mayhem
I wonder what spooked autovacuum.
2023-11-02 30639, 2023
21:09 PM
mayhem
monkey: give it a moment to recover
2023-11-02 30643, 2023
21:09 PM
monkey
Oki
2023-11-02 30650, 2023
21:09 PM
lucifer
it couldn't complete within the timelimit
2023-11-02 30652, 2023
21:09 PM
lucifer
not sure why
2023-11-02 30618, 2023
21:10 PM
mayhem
all my listens are accounted for, but they are coming from navidrome
2023-11-02 30624, 2023
21:11 PM
lucifer
its showing some listens now.
2023-11-02 30640, 2023
21:11 PM
lucifer
(not for me, aerozol's profile)
2023-11-02 30622, 2023
21:14 PM
lucifer
tomorrow, i think we should run `VACUUM VERBOSE listen` and it may list if there are any obvious problems.
2023-11-02 30654, 2023
21:14 PM
lucifer
if not, let's upgrade PG/TS on gaga to latest and then look into tuning it.
2023-11-02 30612, 2023
21:15 PM
mayhem
good plan
2023-11-02 30604, 2023
21:18 PM
aerozol
lucifer: my ‘older’ button is not working now, I can’t check if the covers are fixed 🫣
2023-11-02 30633, 2023
21:18 PM
lucifer
aerozol: yup, its related to the db issues.
2023-11-02 30633, 2023
21:18 PM
aerozol
“You have listened to 8 songs so far”
2023-11-02 30635, 2023
21:18 PM
aerozol
uh oh
2023-11-02 30638, 2023
21:18 PM
lucifer
the older button that is.
2023-11-02 30641, 2023
21:18 PM
aerozol
Ah okay, you’re onto it. Thanks
2023-11-02 30651, 2023
21:18 PM
lucifer
covers are broken due to a different db issue
2023-11-02 30659, 2023
21:18 PM
lucifer
which will take longer to fix
2023-11-02 30658, 2023
21:19 PM
aerozol
allgood
2023-11-02 30641, 2023
21:37 PM
aerozol
2023-11-02 30654, 2023
21:37 PM
aerozol
2023-11-02 30603, 2023
21:38 PM
mayhem
yerp
2023-11-02 30650, 2023
21:38 PM
mayhem
maybe they also don't know about the "save to spotify" option?
2023-11-02 30650, 2023
21:38 PM
aerozol
Happy to ask follow up questions if wanted, otherwise happy to leave it as “something for users to live with” :)
2023-11-02 30634, 2023
21:39 PM
aerozol
Since I don’t use Spotify I don’t have any idea what’s meant tbh, I assumed it was the player, but maybe not?
2023-11-02 30633, 2023
21:41 PM
mayhem
2023-11-02 30651, 2023
21:41 PM
mayhem
this option exists on a playlist if the user has connected spotify
2023-11-02 30659, 2023
21:41 PM
mayhem
I wonder if this person knows about it
2023-11-02 30617, 2023
21:42 PM
fivesheep joined the channel
2023-11-02 30633, 2023
21:42 PM
mayhem
gaga is still under quite a load.
2023-11-02 30647, 2023
21:42 PM
aerozol
Right - I don’t even know if they are talking about radio, playing stuff from elsewhere on LB, or submitting listens from Spotify...
2023-11-02 30603, 2023
21:43 PM
mayhem
doesn't matter.
2023-11-02 30633, 2023
21:43 PM
mayhem
its brainzplayer flakiness that they are upset about. so do as we do and send playlists to spotify.
2023-11-02 30637, 2023
21:43 PM
lucifer
mayhem: i tried cancelling autovacuum but it seems it isn't going away that easy.
2023-11-02 30638, 2023
21:43 PM
aerozol
Okay, I’ll pass that on
2023-11-02 30606, 2023
21:44 PM
mayhem
lucifer: do you need to cancel it for each chunk? 🙄
2023-11-02 30615, 2023
21:44 PM
lucifer
yeah
2023-11-02 30637, 2023
21:44 PM
lucifer
well for the chunks its currently running at the least
2023-11-02 30639, 2023
21:44 PM
lucifer
20 or sor
2023-11-02 30656, 2023
21:44 PM
fivesheep
2023-11-02 30649, 2023
21:45 PM
lucifer
fivesheep: hi! yes, our db is currently under high load so some glitches on the site. the data is still there.
2023-11-02 30652, 2023
21:45 PM
mayhem
your listens are fine fivesheep. the listen counts got borked.
2023-11-02 30610, 2023
21:46 PM
fivesheep
Ah great thank you for the reassurance!!
2023-11-02 30601, 2023
21:49 PM
lucifer
2023-11-02 30612, 2023
21:49 PM
lucifer
the second one looks redundant, i'll drop it
2023-11-02 30656, 2023
21:49 PM
fivesheep has quit
2023-11-02 30637, 2023
22:06 PM
fivesheep joined the channel
2023-11-02 30622, 2023
22:07 PM
lusciouslover joined the channel
2023-11-02 30615, 2023
22:12 PM
lusciouslover has quit
2023-11-02 30657, 2023
22:12 PM
fivesheep has quit
2023-11-02 30605, 2023
22:29 PM
kellnerd
2023-11-02 30628, 2023
22:29 PM
kellnerd
Scared LB users on the forums as well
2023-11-02 30607, 2023
22:32 PM
Maxr1998_
Thanks for the update & the hard work on resolving this! 🙂
2023-11-02 30613, 2023
22:32 PM
mayhem
2023-11-02 30609, 2023
22:34 PM
mayhem
the data in the DB appears to be fine.
2023-11-02 30620, 2023
22:34 PM
mayhem
time to restart labs api, methinks.
2023-11-02 30618, 2023
22:35 PM
mayhem
ah no, that is not labs. huh, odd.
2023-11-02 30637, 2023
22:39 PM
mayhem
well, all seems normal now, except for playcounts and top-recordings. the latter is not so important now.
2023-11-02 30603, 2023
22:41 PM
relaxoMob has quit
2023-11-02 30644, 2023
22:46 PM
monkey
mayhem: it's not just the play counts that are borked from a user point of view. no listen is shown that is older than 2h ago, so it looks to users as if they lost all their listens. See mine for example:
https://listenbrainz.org/user/mr_monkey/
2023-11-02 30659, 2023
22:46 PM
mayhem
i understand.
2023-11-02 30620, 2023
22:47 PM
mayhem
but pagination is borked because of the play counts.
2023-11-02 30602, 2023
22:49 PM
monkey
2023-11-02 30615, 2023
22:50 PM
mayhem
its all connected, monkey.
2023-11-02 30619, 2023
22:50 PM
petitminion joined the channel
2023-11-02 30625, 2023
22:50 PM
relaxoMob joined the channel
2023-11-02 30630, 2023
22:50 PM
monkey
So weird
2023-11-02 30624, 2023
23:00 PM
mayhem
listen counts, minimum timestamps and maximum timestamps are all calculated in the same process.
2023-11-02 30638, 2023
23:00 PM
mayhem
and without min/max ts you get no working pagination.
2023-11-02 30643, 2023
23:08 PM
petitminion has quit
2023-11-02 30643, 2023
23:08 PM
relaxoMob has quit
2023-11-02 30651, 2023
23:11 PM
outsidecontext
What a shock seeing the LB data missing. I really had enough data loss shit this week. But glad to hear the listens themselves are fine. Wish you the best getting everything back up
2023-11-02 30651, 2023
23:19 PM
petitminion joined the channel
2023-11-02 30647, 2023
23:24 PM
relaxoMob joined the channel
