#metabrainz

/

      • lucifer
        no. looking now.
      • yvanzo
        lucifer: 9:05 IST? :)
      • lucifer
        uh sorry! yes 9:05 IST
      • 15:35 UTC
      • yvanzo
        Actually there was a surge of queries to MB website at 15:34.
      • lucifer
        i see. i read the file in syswiki but still not sure where to find pgbouncer logs or how consul is connected to pgbouncer.
      • the doc explains how to start/stop/change maintenance mode of the database but not why/how it can automatically restart etc.
      • yvanzo
        I checked pg_stat_activity but didn’t find anything suspicious.
      • gaga load is still high
      • lucifer
        hmm
      • mayhem: there are 2 active timescale dbs on gaga currently?
      • mayhem
        uh?????
      • lucifer
        and it shows as up since 6 months?
      • 7 months.
      • mayhem
        zero clue, sorry.
      • is it getting traffic?
      • lucifer
        checking that.
      • yes
      • according to LB config, it is currently connected to the old timescale db container.
      • mayhem
        🤯
      • lucifer
        and my listen history seems fine so has it been this way all along?
      • well at least for 7 weeks. when the newer pg container came up.
      • mayhem
        I'd like to know where this traffic is coming from.
      • and if we just stop the container, whatever is using it might just update and start using the correct DB?
      • or does the universe implode?
      • lucifer
        let me check the databases.
      • which one has what data etc
      • zas: is there a consul admin ui to check all running services?
      • zas
      • lucifer
        mayhem: oh nvm, false alarm. we started the newer db during summit, `SERVICE_5432_NAME=listenbrainz-timescale-pg14-test`
      • zas: yes but i was thinking about something which has more consul related info available.
      • mayhem
        ahhh, phew.
      • zas
        lucifer: nothing directly accessible, but you can ssh forward a port to access consul port 8500 (on herb for example), see https://developer.hashicorp.com/consul/tutorial...
      • lucifer
        zas: thanks! that sounds good
      • yvanzo: ListenBrainz data dump is happening which may explain the increased load on gaga.
      • yvanzo
        When did it start?
      • lucifer
        12 hours ago actually.
      • hmm
      • pg_stat_activity is empty otherwise and the timescale container is the one consuming most resources currently.
      • yvanzo
        My bad, it was not a surge of queries, it was just an inflation of logs due to 35 queries having failed on timeout.
      • lucifer
        load is still higher than usual on gaga but falling now. no idea why though
      • mayhem: `2022-12-01 13:35:41.821 UTC [1597618] LOG: refreshing continuous aggregate "listened_at_max" in window [ -9223372036854720000, 1669852800 ]` ts container logs have this. but i checked and didn't find any cont. agg. maybe a ts bug or bad migration i guess, i'll ask in ts forums.
      • wargreen joined the channel
      • yvanzo
        I checked mb cron logs, was not even running at that time
      • mayhem
        I bet the continuous aggregate was not dropped properly, problaby due to it being so old.
      • yvanzo
        I checked queried MB URLs that timed out, nothing special about it either.
      • lucifer
        should we try creating it again and dropping?
      • mayhem
        either that or figure out what needs to be dropped manually... its all just a thin veneer over PG, right?
      • lucifer
        yeah but all information views of ts are empty.
      • *cont agg related views
      • yvanzo
        I checked musicbrainz-redis containers, nothing either.
      • lucifer
        do we know which requests 5xx'ed?
      • yvanzo
        yes
      • reosarevok, bitmap: timeouts mainly occurred on Pg.pm but someitmes on Redis.pm instead.
      • lucifer
      • pgbouncer connections' average wait time also increased around that time.
      • zas: it seems consul agent container on herb does not expose port 8500 at all.
      • reosarevok
        We have issues with timeouts also giving generally terrible feedback (direct ISEs, often) to users, even when they should fail a lot more gracefully
      • Freso: around?
      • yvanzo
        reosarevok: You might want to check Harry Botter.
      • reosarevok
        yvanzo: ok, I'll take a look
      • Is it doing something weird?
      • (in MB I mean)
      • zas
        lucifer: try on 10.2.2.26 8500
      • yvanzo
        reosarevok: I found a lot searches for its edits at the time of the load peek.
      • a lot +of
      • reosarevok
        Hmm
      • The bot does check its own edit count, but I can't imagine that'd cause issues... do you know what IP was searching? Was it from zappa?
      • yvanzo
        There were also some searches for edits which note contains "musicbrainz.org/track". These searches typically take over 10s.
      • reosarevok
        That was jesus2099, probably
      • lucifer
        zas: ah ok, thanks. i see that `ui` needs to be explicitly enabled in prod using `-ui` cli flag and currently its not.
      • reosarevok
        (given jesus' comment on the /track redirect PR)
      • And yeah, edit note contains searches take a fair amount of time
      • zas
        lucifer: wait I'll change it on herb (I thought I did, let me check)
      • lucifer: try again, on herb
      • lucifer
        zas: working now, thanks!
      • reosarevok
        Freso: I entered removals for stuff from the "URGENT REMOVAL" support email, but it suggests the editor entering those might be part of some sort of scammy at worst and spammy at best thing
      • Also, we might need to reconsider the "all stuff published as music belongs" because it seems the sort of "musical spam" where people put spam up on bandcamp, iTunes and whatnot as music just to get SEO is growing
      • wargreen has quit
      • yvanzo
        reosarevok, bitmap: This search condition should probably not be allowed alone as it is taking too much time otherwise.
      • reosarevok
        Is it? Timeout is set at 60s, not 10s
      • There's plenty of other conditions which seem to timeout consistently, which means this isn't even that bad :/
      • Tbh, edit search is just generally a badly-performing mess
      • yvanzo
        Any condition that takes too much time should not be used alone.
      • reosarevok
        I guess I just dunno what is "too much time" - but then we'd need to decide what that means and figure how to limit them (using it combined with a very wide search, like "editor is not me", probably won't help much, for example)
      • yvanzo
        We could even preselect search condition “created after last month” so that it doesn’t require additional action from editors in most common cases.
      • reosarevok: too much time as in it needlessly impacts response time
      • reosarevok
        Except for example, for edit note content, that's not really the most common need. Every time I've ever used the edit note content search, I've literally wanted to find any edits using the content, from any time :)
      • yvanzo
        It's not about making the edit search less usable, it's the contrary.
      • reosarevok
        I understand that it's messy though, and if we can find a way to do it better, that'd be good
      • yvanzo
        Selecting additional conditions that would make sense just takes too many clicks for users to bother about it.
      • wargreen joined the channel
      • reosarevok: so, does restricting to last month by default seem sensible?
      • reosarevok
        As I said earlier, that doesn't seem to make any sense for edit note content, for example - the main use case for that is to find all edits mentioning a specific word (for example to find rude editors), so combining with temporal restrictions at all is not really a good idea IMO.
      • It might be very good for other timing out searches, though - stuff like relationship types I think struggles and might work fine with that default?
      • yvanzo
        Are you most often searching for old edits or recent edits?
      • reosarevok
        (it might also still struggle, haven't tested :) )
      • Whenever I've used that specific filter, all edits by a certain editor or all edits at all
      • yvanzo
        In general.
      • reosarevok
        For most other filters, recent edits :)
      • Hence saying that it might work fine for other stuff
      • yvanzo
        So it makes sense to preselect it in general?
      • reosarevok
        I dunno, depends on how other people use edit search :)
      • Maybe ask the community?
      • That's kind of a band-aid for the horrible performance of edit search, but not sure how much that can be improved with our millions of edits without adding a bunch of extra indexes and possibly even materialized tables :/
      • yvanzo
        It just avoids wasting resources and time with not refined edit searches.
      • reosarevok
        Yes, we should just ask whether people feel that'd be helpful or annoying
      • Most edit lists default to 2 weeks, so if you're going to add a default I'd pick that
      • But maybe it turns out people want more wide searches and they dislike the pre-selection - or it might turn out it's actually helpful and not an issue
      • That's why I suggested asking the community, maybe in the forum
      • Did you test a bit to make sure there's a significant improvement?
      • I haven't played with it enough, honestly - for example, does the ordering of the conditions affect speed because it filters things differently?
      • yvanzo
        where do you see that "edit lists default to 2 weeks"?
      • reosarevok: it is just a preselection, it takes one click to discard
      • reosarevok
      • But I might misremember the duration :)
      • Yes, and if you use searches a lot and have to discard them every time, that seems very annoying - but it might not be a problem because maybe only one or two users use searches a lot and they don't mind :)
      • Alternatively, of course, we could have a cookie to discard it on editor request, so that experienced editors who don't want preselections don't have to deal with them while most others have them there by default
      • But I'm just thinking I know very little about how people use edit search
      • Maybe most uses are from the "Refine this search" links, in which case those are already full of pre-selections?
      • I expect there's only a small amount of users actively building their own searches often, but I haven't researched it
      • yvanzo
        "2 weeks ago" is already used for some edits voting lists indeed.
      • I did use search a lot (and I guess you did as well) and I’m always annoyed when it takes a lot of time during which I’m thinking that I should have been more specific.
      • reosarevok
        I'm usually specific already when I think it's going to time out, but I don't otherwise mind waiting 10 seconds or whatever :)
      • But yeah, I'm not necessarily a good example
      • wargreen has quit
      • aerozol
        Mōrena! Did people look at the yim mockup? Any thoughts? monkey, mayhem
      • monkey: oops, we were going to do a regular meeting but I had COVID last week and today I slept through it! Not a good start
      • mayhem
        not a chance yet, but will this evening.
      • lucifer
        aerozol: the color scheme felt bit out of touch with usual LB theme but otherwise lgtm.
      • aerozol
        Open to color thoughts! Would be nice to do something other than white for this special occasion imo
      • monkey
        Sorry too aerozol, totally forgot aout it with all the USA family stuff
      • aerozol
        Ooh are you still in the states monkey?
      • lucifer
        monkey, mayhem, alastairp: first monday of month, should we have LB meet next week?
      • jivte_ joined the channel
      • yvanzo
        Opened a ticket and a community post.
      • monkey
        Yes aerozol, and lucifer I'm going to be travelling this monday 5th, flying back to europe so won't be available for a meeting
      • jivte_
        lucifer: Hey I went through the examples you sent and found if the example feature is to be implemented then everything is working right like if there is no MBID/MSID then other options will also get affected as they all depend on the MBID/MSID
      • lucifer
        ah cool, maybe later in the week then
      • monkey
        I'm landing in Spain Tuesday evening
      • But yeah after tha later in the week is OK for me
      • aerozol
        Safe travels monkey!
      • wargreen joined the channel
      • lucifer
        jivte_: yes, i suspect the issue is that we are entirely forgetting to add some buttons on some pagees.
      • jivte_
        lucifer: When I read the issue it was about to have some options across all pages
      • so implemented that way only
      • so is there anything I could do
      • lucifer
        right, the issue is that some options are missing entirely from pages. think that recording mbid is present but the open in musicbrainz link is still absent.