yvanzo: ListenBrainz data dump is happening which may explain the increased load on gaga.
2022-12-01 33504, 2022
yvanzo
When did it start?
2022-12-01 33551, 2022
lucifer
12 hours ago actually.
2022-12-01 33552, 2022
lucifer
hmm
2022-12-01 33518, 2022
lucifer
pg_stat_activity is empty otherwise and the timescale container is the one consuming most resources currently.
2022-12-01 33524, 2022
yvanzo
My bad, it was not a surge of queries, it was just an inflation of logs due to 35 queries having failed on timeout.
2022-12-01 33551, 2022
lucifer
load is still higher than usual on gaga but falling now. no idea why though
2022-12-01 33541, 2022
lucifer
mayhem: `2022-12-01 13:35:41.821 UTC [1597618] LOG: refreshing continuous aggregate "listened_at_max" in window [ -9223372036854720000, 1669852800 ]` ts container logs have this. but i checked and didn't find any cont. agg. maybe a ts bug or bad migration i guess, i'll ask in ts forums.
2022-12-01 33503, 2022
wargreen joined the channel
2022-12-01 33553, 2022
yvanzo
I checked mb cron logs, was not even running at that time
2022-12-01 33510, 2022
mayhem
I bet the continuous aggregate was not dropped properly, problaby due to it being so old.
2022-12-01 33522, 2022
yvanzo
I checked queried MB URLs that timed out, nothing special about it either.
2022-12-01 33532, 2022
lucifer
should we try creating it again and dropping?
2022-12-01 33513, 2022
mayhem
either that or figure out what needs to be dropped manually... its all just a thin veneer over PG, right?
2022-12-01 33527, 2022
lucifer
yeah but all information views of ts are empty.
2022-12-01 33537, 2022
lucifer
*cont agg related views
2022-12-01 33551, 2022
yvanzo
I checked musicbrainz-redis containers, nothing either.
2022-12-01 33529, 2022
lucifer
do we know which requests 5xx'ed?
2022-12-01 33526, 2022
yvanzo
yes
2022-12-01 33531, 2022
yvanzo
reosarevok, bitmap: timeouts mainly occurred on Pg.pm but someitmes on Redis.pm instead.
pgbouncer connections' average wait time also increased around that time.
2022-12-01 33543, 2022
lucifer
zas: it seems consul agent container on herb does not expose port 8500 at all.
2022-12-01 33516, 2022
reosarevok
We have issues with timeouts also giving generally terrible feedback (direct ISEs, often) to users, even when they should fail a lot more gracefully
2022-12-01 33550, 2022
reosarevok
Freso: around?
2022-12-01 33554, 2022
yvanzo
reosarevok: You might want to check Harry Botter.
2022-12-01 33506, 2022
reosarevok
yvanzo: ok, I'll take a look
2022-12-01 33512, 2022
reosarevok
Is it doing something weird?
2022-12-01 33553, 2022
reosarevok
(in MB I mean)
2022-12-01 33502, 2022
zas
lucifer: try on 10.2.2.26 8500
2022-12-01 33554, 2022
yvanzo
reosarevok: I found a lot searches for its edits at the time of the load peek.
2022-12-01 33522, 2022
yvanzo
a lot +of
2022-12-01 33526, 2022
reosarevok
Hmm
2022-12-01 33500, 2022
reosarevok
The bot does check its own edit count, but I can't imagine that'd cause issues... do you know what IP was searching? Was it from zappa?
2022-12-01 33511, 2022
yvanzo
There were also some searches for edits which note contains "musicbrainz.org/track". These searches typically take over 10s.
2022-12-01 33527, 2022
reosarevok
That was jesus2099, probably
2022-12-01 33550, 2022
lucifer
zas: ah ok, thanks. i see that `ui` needs to be explicitly enabled in prod using `-ui` cli flag and currently its not.
2022-12-01 33551, 2022
reosarevok
(given jesus' comment on the /track redirect PR)
2022-12-01 33504, 2022
reosarevok
And yeah, edit note contains searches take a fair amount of time
2022-12-01 33536, 2022
zas
lucifer: wait I'll change it on herb (I thought I did, let me check)
2022-12-01 33530, 2022
zas
lucifer: try again, on herb
2022-12-01 33554, 2022
lucifer
zas: working now, thanks!
2022-12-01 33515, 2022
reosarevok
Freso: I entered removals for stuff from the "URGENT REMOVAL" support email, but it suggests the editor entering those might be part of some sort of scammy at worst and spammy at best thing
2022-12-01 33556, 2022
reosarevok
Also, we might need to reconsider the "all stuff published as music belongs" because it seems the sort of "musical spam" where people put spam up on bandcamp, iTunes and whatnot as music just to get SEO is growing
2022-12-01 33505, 2022
wargreen has quit
2022-12-01 33524, 2022
yvanzo
reosarevok, bitmap: This search condition should probably not be allowed alone as it is taking too much time otherwise.
2022-12-01 33544, 2022
reosarevok
Is it? Timeout is set at 60s, not 10s
2022-12-01 33509, 2022
reosarevok
There's plenty of other conditions which seem to timeout consistently, which means this isn't even that bad :/
2022-12-01 33519, 2022
reosarevok
Tbh, edit search is just generally a badly-performing mess
2022-12-01 33500, 2022
yvanzo
Any condition that takes too much time should not be used alone.
2022-12-01 33551, 2022
reosarevok
I guess I just dunno what is "too much time" - but then we'd need to decide what that means and figure how to limit them (using it combined with a very wide search, like "editor is not me", probably won't help much, for example)
2022-12-01 33557, 2022
yvanzo
We could even preselect search condition “created after last month” so that it doesn’t require additional action from editors in most common cases.
2022-12-01 33534, 2022
yvanzo
reosarevok: too much time as in it needlessly impacts response time
2022-12-01 33506, 2022
reosarevok
Except for example, for edit note content, that's not really the most common need. Every time I've ever used the edit note content search, I've literally wanted to find any edits using the content, from any time :)
2022-12-01 33521, 2022
yvanzo
It's not about making the edit search less usable, it's the contrary.
2022-12-01 33529, 2022
reosarevok
I understand that it's messy though, and if we can find a way to do it better, that'd be good
2022-12-01 33524, 2022
yvanzo
Selecting additional conditions that would make sense just takes too many clicks for users to bother about it.
2022-12-01 33518, 2022
wargreen joined the channel
2022-12-01 33557, 2022
yvanzo
reosarevok: so, does restricting to last month by default seem sensible?
2022-12-01 33520, 2022
reosarevok
As I said earlier, that doesn't seem to make any sense for edit note content, for example - the main use case for that is to find all edits mentioning a specific word (for example to find rude editors), so combining with temporal restrictions at all is not really a good idea IMO.
2022-12-01 33549, 2022
reosarevok
It might be very good for other timing out searches, though - stuff like relationship types I think struggles and might work fine with that default?
2022-12-01 33503, 2022
yvanzo
Are you most often searching for old edits or recent edits?
2022-12-01 33503, 2022
reosarevok
(it might also still struggle, haven't tested :) )
2022-12-01 33526, 2022
reosarevok
Whenever I've used that specific filter, all edits by a certain editor or all edits at all
2022-12-01 33534, 2022
yvanzo
In general.
2022-12-01 33535, 2022
reosarevok
For most other filters, recent edits :)
2022-12-01 33543, 2022
reosarevok
Hence saying that it might work fine for other stuff
2022-12-01 33554, 2022
yvanzo
So it makes sense to preselect it in general?
2022-12-01 33509, 2022
reosarevok
I dunno, depends on how other people use edit search :)
2022-12-01 33516, 2022
reosarevok
Maybe ask the community?
2022-12-01 33533, 2022
reosarevok
That's kind of a band-aid for the horrible performance of edit search, but not sure how much that can be improved with our millions of edits without adding a bunch of extra indexes and possibly even materialized tables :/
2022-12-01 33515, 2022
yvanzo
It just avoids wasting resources and time with not refined edit searches.
2022-12-01 33500, 2022
reosarevok
Yes, we should just ask whether people feel that'd be helpful or annoying
2022-12-01 33517, 2022
reosarevok
Most edit lists default to 2 weeks, so if you're going to add a default I'd pick that
2022-12-01 33559, 2022
reosarevok
But maybe it turns out people want more wide searches and they dislike the pre-selection - or it might turn out it's actually helpful and not an issue
2022-12-01 33515, 2022
reosarevok
That's why I suggested asking the community, maybe in the forum
2022-12-01 33529, 2022
reosarevok
Did you test a bit to make sure there's a significant improvement?
2022-12-01 33522, 2022
reosarevok
I haven't played with it enough, honestly - for example, does the ordering of the conditions affect speed because it filters things differently?
2022-12-01 33526, 2022
yvanzo
where do you see that "edit lists default to 2 weeks"?
2022-12-01 33532, 2022
yvanzo
reosarevok: it is just a preselection, it takes one click to discard
Yes, and if you use searches a lot and have to discard them every time, that seems very annoying - but it might not be a problem because maybe only one or two users use searches a lot and they don't mind :)
2022-12-01 33505, 2022
reosarevok
Alternatively, of course, we could have a cookie to discard it on editor request, so that experienced editors who don't want preselections don't have to deal with them while most others have them there by default
2022-12-01 33516, 2022
reosarevok
But I'm just thinking I know very little about how people use edit search
2022-12-01 33543, 2022
reosarevok
Maybe most uses are from the "Refine this search" links, in which case those are already full of pre-selections?
2022-12-01 33501, 2022
reosarevok
I expect there's only a small amount of users actively building their own searches often, but I haven't researched it
2022-12-01 33524, 2022
yvanzo
"2 weeks ago" is already used for some edits voting lists indeed.
2022-12-01 33535, 2022
yvanzo
I did use search a lot (and I guess you did as well) and I’m always annoyed when it takes a lot of time during which I’m thinking that I should have been more specific.
2022-12-01 33533, 2022
reosarevok
I'm usually specific already when I think it's going to time out, but I don't otherwise mind waiting 10 seconds or whatever :)
2022-12-01 33541, 2022
reosarevok
But yeah, I'm not necessarily a good example
2022-12-01 33507, 2022
wargreen has quit
2022-12-01 33513, 2022
aerozol
Mōrena! Did people look at the yim mockup? Any thoughts? monkey, mayhem
2022-12-01 33523, 2022
aerozol
monkey: oops, we were going to do a regular meeting but I had COVID last week and today I slept through it! Not a good start
2022-12-01 33519, 2022
mayhem
not a chance yet, but will this evening.
2022-12-01 33526, 2022
lucifer
aerozol: the color scheme felt bit out of touch with usual LB theme but otherwise lgtm.
2022-12-01 33543, 2022
aerozol
Open to color thoughts! Would be nice to do something other than white for this special occasion imo
2022-12-01 33518, 2022
monkey
Sorry too aerozol, totally forgot aout it with all the USA family stuff
2022-12-01 33545, 2022
aerozol
Ooh are you still in the states monkey?
2022-12-01 33510, 2022
lucifer
monkey, mayhem, alastairp: first monday of month, should we have LB meet next week?
2022-12-01 33518, 2022
jivte_ joined the channel
2022-12-01 33513, 2022
yvanzo
Opened a ticket and a community post.
2022-12-01 33531, 2022
monkey
Yes aerozol, and lucifer I'm going to be travelling this monday 5th, flying back to europe so won't be available for a meeting
2022-12-01 33553, 2022
jivte_
lucifer: Hey I went through the examples you sent and found if the example feature is to be implemented then everything is working right like if there is no MBID/MSID then other options will also get affected as they all depend on the MBID/MSID
2022-12-01 33554, 2022
lucifer
ah cool, maybe later in the week then
2022-12-01 33557, 2022
monkey
I'm landing in Spain Tuesday evening
2022-12-01 33509, 2022
monkey
But yeah after tha later in the week is OK for me
2022-12-01 33510, 2022
aerozol
Safe travels monkey!
2022-12-01 33521, 2022
wargreen joined the channel
2022-12-01 33549, 2022
lucifer
jivte_: yes, i suspect the issue is that we are entirely forgetting to add some buttons on some pagees.
2022-12-01 33539, 2022
jivte_
lucifer: When I read the issue it was about to have some options across all pages
2022-12-01 33556, 2022
jivte_
so implemented that way only
2022-12-01 33507, 2022
jivte_
so is there anything I could do
2022-12-01 33533, 2022
lucifer
right, the issue is that some options are missing entirely from pages. think that recording mbid is present but the open in musicbrainz link is still absent.
