yvanzo: ListenBrainz data dump is happening which may explain the increased load on gaga.
yvanzo
When did it start?
lucifer
12 hours ago actually.
hmm
pg_stat_activity is empty otherwise and the timescale container is the one consuming most resources currently.
yvanzo
My bad, it was not a surge of queries, it was just an inflation of logs due to 35 queries having failed on timeout.
lucifer
load is still higher than usual on gaga but falling now. no idea why though
mayhem: `2022-12-01 13:35:41.821 UTC [1597618] LOG: refreshing continuous aggregate "listened_at_max" in window [ -9223372036854720000, 1669852800 ]` ts container logs have this. but i checked and didn't find any cont. agg. maybe a ts bug or bad migration i guess, i'll ask in ts forums.
wargreen joined the channel
yvanzo
I checked mb cron logs, was not even running at that time
mayhem
I bet the continuous aggregate was not dropped properly, problaby due to it being so old.
yvanzo
I checked queried MB URLs that timed out, nothing special about it either.
lucifer
should we try creating it again and dropping?
mayhem
either that or figure out what needs to be dropped manually... its all just a thin veneer over PG, right?
lucifer
yeah but all information views of ts are empty.
*cont agg related views
yvanzo
I checked musicbrainz-redis containers, nothing either.
lucifer
do we know which requests 5xx'ed?
yvanzo
yes
reosarevok, bitmap: timeouts mainly occurred on Pg.pm but someitmes on Redis.pm instead.
pgbouncer connections' average wait time also increased around that time.
zas: it seems consul agent container on herb does not expose port 8500 at all.
reosarevok
We have issues with timeouts also giving generally terrible feedback (direct ISEs, often) to users, even when they should fail a lot more gracefully
Freso: around?
yvanzo
reosarevok: You might want to check Harry Botter.
reosarevok
yvanzo: ok, I'll take a look
Is it doing something weird?
(in MB I mean)
zas
lucifer: try on 10.2.2.26 8500
yvanzo
reosarevok: I found a lot searches for its edits at the time of the load peek.
a lot +of
reosarevok
Hmm
The bot does check its own edit count, but I can't imagine that'd cause issues... do you know what IP was searching? Was it from zappa?
yvanzo
There were also some searches for edits which note contains "musicbrainz.org/track". These searches typically take over 10s.
reosarevok
That was jesus2099, probably
lucifer
zas: ah ok, thanks. i see that `ui` needs to be explicitly enabled in prod using `-ui` cli flag and currently its not.
reosarevok
(given jesus' comment on the /track redirect PR)
And yeah, edit note contains searches take a fair amount of time
zas
lucifer: wait I'll change it on herb (I thought I did, let me check)
lucifer: try again, on herb
lucifer
zas: working now, thanks!
reosarevok
Freso: I entered removals for stuff from the "URGENT REMOVAL" support email, but it suggests the editor entering those might be part of some sort of scammy at worst and spammy at best thing
Also, we might need to reconsider the "all stuff published as music belongs" because it seems the sort of "musical spam" where people put spam up on bandcamp, iTunes and whatnot as music just to get SEO is growing
wargreen has quit
yvanzo
reosarevok, bitmap: This search condition should probably not be allowed alone as it is taking too much time otherwise.
reosarevok
Is it? Timeout is set at 60s, not 10s
There's plenty of other conditions which seem to timeout consistently, which means this isn't even that bad :/
Tbh, edit search is just generally a badly-performing mess
yvanzo
Any condition that takes too much time should not be used alone.
reosarevok
I guess I just dunno what is "too much time" - but then we'd need to decide what that means and figure how to limit them (using it combined with a very wide search, like "editor is not me", probably won't help much, for example)
yvanzo
We could even preselect search condition “created after last month” so that it doesn’t require additional action from editors in most common cases.
reosarevok: too much time as in it needlessly impacts response time
reosarevok
Except for example, for edit note content, that's not really the most common need. Every time I've ever used the edit note content search, I've literally wanted to find any edits using the content, from any time :)
yvanzo
It's not about making the edit search less usable, it's the contrary.
reosarevok
I understand that it's messy though, and if we can find a way to do it better, that'd be good
yvanzo
Selecting additional conditions that would make sense just takes too many clicks for users to bother about it.
wargreen joined the channel
reosarevok: so, does restricting to last month by default seem sensible?
reosarevok
As I said earlier, that doesn't seem to make any sense for edit note content, for example - the main use case for that is to find all edits mentioning a specific word (for example to find rude editors), so combining with temporal restrictions at all is not really a good idea IMO.
It might be very good for other timing out searches, though - stuff like relationship types I think struggles and might work fine with that default?
yvanzo
Are you most often searching for old edits or recent edits?
reosarevok
(it might also still struggle, haven't tested :) )
Whenever I've used that specific filter, all edits by a certain editor or all edits at all
yvanzo
In general.
reosarevok
For most other filters, recent edits :)
Hence saying that it might work fine for other stuff
yvanzo
So it makes sense to preselect it in general?
reosarevok
I dunno, depends on how other people use edit search :)
Maybe ask the community?
That's kind of a band-aid for the horrible performance of edit search, but not sure how much that can be improved with our millions of edits without adding a bunch of extra indexes and possibly even materialized tables :/
yvanzo
It just avoids wasting resources and time with not refined edit searches.
reosarevok
Yes, we should just ask whether people feel that'd be helpful or annoying
Most edit lists default to 2 weeks, so if you're going to add a default I'd pick that
But maybe it turns out people want more wide searches and they dislike the pre-selection - or it might turn out it's actually helpful and not an issue
That's why I suggested asking the community, maybe in the forum
Did you test a bit to make sure there's a significant improvement?
I haven't played with it enough, honestly - for example, does the ordering of the conditions affect speed because it filters things differently?
yvanzo
where do you see that "edit lists default to 2 weeks"?
reosarevok: it is just a preselection, it takes one click to discard
Yes, and if you use searches a lot and have to discard them every time, that seems very annoying - but it might not be a problem because maybe only one or two users use searches a lot and they don't mind :)
Alternatively, of course, we could have a cookie to discard it on editor request, so that experienced editors who don't want preselections don't have to deal with them while most others have them there by default
But I'm just thinking I know very little about how people use edit search
Maybe most uses are from the "Refine this search" links, in which case those are already full of pre-selections?
I expect there's only a small amount of users actively building their own searches often, but I haven't researched it
yvanzo
"2 weeks ago" is already used for some edits voting lists indeed.
I did use search a lot (and I guess you did as well) and I’m always annoyed when it takes a lot of time during which I’m thinking that I should have been more specific.
reosarevok
I'm usually specific already when I think it's going to time out, but I don't otherwise mind waiting 10 seconds or whatever :)
But yeah, I'm not necessarily a good example
wargreen has quit
aerozol
Mōrena! Did people look at the yim mockup? Any thoughts? monkey, mayhem
monkey: oops, we were going to do a regular meeting but I had COVID last week and today I slept through it! Not a good start
mayhem
not a chance yet, but will this evening.
lucifer
aerozol: the color scheme felt bit out of touch with usual LB theme but otherwise lgtm.
aerozol
Open to color thoughts! Would be nice to do something other than white for this special occasion imo
monkey
Sorry too aerozol, totally forgot aout it with all the USA family stuff
aerozol
Ooh are you still in the states monkey?
lucifer
monkey, mayhem, alastairp: first monday of month, should we have LB meet next week?
jivte_ joined the channel
yvanzo
Opened a ticket and a community post.
monkey
Yes aerozol, and lucifer I'm going to be travelling this monday 5th, flying back to europe so won't be available for a meeting
jivte_
lucifer: Hey I went through the examples you sent and found if the example feature is to be implemented then everything is working right like if there is no MBID/MSID then other options will also get affected as they all depend on the MBID/MSID
lucifer
ah cool, maybe later in the week then
monkey
I'm landing in Spain Tuesday evening
But yeah after tha later in the week is OK for me
aerozol
Safe travels monkey!
wargreen joined the channel
lucifer
jivte_: yes, i suspect the issue is that we are entirely forgetting to add some buttons on some pagees.
jivte_
lucifer: When I read the issue it was about to have some options across all pages
so implemented that way only
so is there anything I could do
lucifer
right, the issue is that some options are missing entirely from pages. think that recording mbid is present but the open in musicbrainz link is still absent.