#metabrainz

/

      • piwu has quit
      • piwu joined the channel
      • jivte has quit
      • aerozol
        mayhem: lucifer: I know this is entry level, but in case there’s something useful: http://playlistmachinery.com/examples.html
      • A really clear example of comboing various playlists
      • jivte joined the channel
      • jivte has quit
      • jivte joined the channel
      • jivte has quit
      • q3lont joined the channel
      • q3lont has quit
      • Pokey has quit
      • Pokey joined the channel
      • Maxr1998 has quit
      • Maxr1998_ joined the channel
      • v6lur joined the channel
      • zas
        bitmap: after I blocked lidarr-extended UA, we didn't get any issue. It was around 3k queries per minute, and 25% of them were for recordings of [unknown] artist (well, parts of it, since they are using offset)
      • not sure what to do from here, I think we should suggest app's author to use its own mirrors. But it doesn't solve the core issue, which is high load on main db due to those specific queries, is there room for improvement in this field?
      • bitmap, yvanzo, lucifer, mayhem: we have to reboot servers to ensure docker upgrades are fully applied. After investigation it happens recent upgrades to latest major docker version didn't play well with live-restore (it is documented, live restore behavior is only guaranteed for minor versions upgrades).
      • with atj, we'll change the Ansible config to prevent such issue in the future.
      • I'll start with few servers that are not critical, but ofc we need to plan a (short) downtime for critical ones. A reboot isn't strictly needed, but safer, and it will also apply kernel changes.
      • aerozol
        monkey: you’re not going to believe it, but I have done the first design mockup for the LB recs pages: https://www.figma.com/file/YRbCOtFHBez8XmMdCKbG...
      • Having dinner now, but will be back on the compy later for other work, so might catch you if you’re around
      • petitminion joined the channel
      • zas
        rebooting aphex
      • done
      • rebooting ludwig
      • reosarevok
        Hmm
      • zas
        morning reo
      • ludwig rebooted
      • reosarevok
        I wonder if [unknown] is our entity with the most recordings. It might be the case
      • 1700 pages is a lot
      • zas
        yes, and requests are done over and over from different IPs
      • reosarevok
        Not sure how much can be done to speed it up though, in the sense that those recordings probably *do* belong there...
      • zas
        because each user has its own instance (that's a docker image)
      • reosarevok
        Is the problem that they're all hitting at the same time of day again?
      • Because I assume one or two at random times wouldn't be a big issue
      • zas
        not really, hits come every few seconds, I didn't notice "peaks"
      • reosarevok
        Hmm
      • And they're otherwise legit hits which respect waiting times and whatnot, but there's still so many of them that they cause issues?
      • zas
        yes
      • reosarevok
        Or is it a small amount at a time but our query is so unperformant it causes issues?
      • zas
        a bit of both I think
      • heavy queries crossed with high number of hits
      • reosarevok
        I mean
      • "Lidarr is a music collection manager for Usenet and BitTorrent users"
      • suggests that probably this is all useless for them too
      • Since I doubt someone is specifically caring whether there's new and exciting [unknown] music
      • zas
        I didn't look at their code, but yes, somehow the same problem as QNAP, useless queries, many users
      • reosarevok
        And it's a new issue, right? Wonder if it's a recent code change, or just a lot of new users
      • aerozol
        I've thought about setting up Lidarr, you can set rules to download stuff automatically, that you don't have yet, from artists you set. No idea why that extension would settle query [unknown]...
      • reosarevok
        I would of course happily block the whole thing on grounds that it's clearly just a pirate tool anyway
      • But 🤷‍♂️
      • zas
        reosarevok: not really new, there are hitting us since a while, but for some reason, we reached a point where it becomes a problem. I didn't investigate details, just identified the main source, blocked, in order to preserve services for other users over the week-end.
      • mayhem
        if we're getting too much traffic from lidarr or if the queries are BS, let me know and I'll go back to complain to them.
      • aerozol
        Lidarr is a pretty big source of editors for us, as it uses MB exclusively afaik
      • reosarevok
        mayhem: it's an extension apparently, not lidarr itself
      • zas
        in this case, that's lidarr-extended
      • reosarevok
        aerozol: of pirate editors, yes. I find it hard to care for someone who still pirates hardcore in the era of affordable streaming tbh
      • I mean, I get "I can't afford music so I download some stuff from artists I love", but this is "I automate piracy at a massive scale2 :p
      • mayhem
        ah, that's better. but still, if there is a complaint you'd like me to pass along to lidarr, who might be able to put pressure on the plugin devs.
      • zas
        reosarevok: totally agree, that said, it reveals a weakness in our web services, and a more legit tool would have caused the same issue
      • aerozol
        reosarevok: I find it hard to respect people who use sites like Spotify to destroy artists but still feel fine about it, because huge behemoth
      • mayhem
        aerozol: reosarevok : covering the two sides of this argument perfectly.
      • reosarevok
        At least that involves *some* money getting to the artist, not saying it's great, but it's clearly not worse than piracy :)
      • Not that it matters anyway, soon I expect Spotify will be full of AI-generated artists by major labels (or Spotify itself) so they can get the listens without having a pesky artist to manage and pay
      • aerozol
        If someone using Lidarr buys one t-shirt they've paid more to musicians than my partner will have in a decade of Spotify use
      • reosarevok
        Of course. Just you don't need to use Lidarr for that :D
      • mayhem
        amen
      • aerozol
        I'm not saying stealing is good... It's just complicated
      • reosarevok
        I mean, I assume superfans who stream also buy merch, and they're probably unlikely to buy more merch because they use Lidarr instead
      • But eh
      • yvanzo
        O’Moin
      • reosarevok
        musicbrainzArtistRecordings=$(curl -s -A "$agent" "$musicbrainzMirror/ws/2/recording?artist=$lidarrArtistMusicbrainzId&limit=1&offset=0&fmt=json")
      • That doesn't have the includes...
      • Wonder where they call this
      • zas: agreed re: a more legit tool, I just wonder if any tool, legit or not, has a real use case for repeatedly requesting [unknown], lol
      • aerozol
        Yeah it's a weird one
      • reosarevok
        Oh, found it
      • curl -s -A "$agent" "$musicbrainzMirror/ws/2/recording?artist=$lidarrArtistMusicbrainzId&inc=artist-credits+url-rels+recording-rels+release-rels+release-group-rels&limit=100&offset=$offset&fmt=json" | jq -r ".recordings[]" >> "/config/extended/cache/musicbrainz/$lidarrArtistId--$lidarrArtistMusicbrainzId--recordings.json"
      • So
      • "Searches Lidarr Artists (musicbrainz) video recordings for videos to download"
      • Even less use for them hitting [unknown], lol
      • That said, it seems we could get rid of the issue by adding inc=video_only
      • Or something
      • But of course that won't help when someone wants something *else* from [unknown]
      • zas
        rebooting hip
      • v6lur has quit
      • jivte joined the channel
      • now rebooting rudi then rex
      • reosarevok
        zas, bitmap, yvanzo: Added MBS-13093 for a new feature for the API since piracy aside, this seems like a legitimate use case
      • BrainzBot
        MBS-13093: Allow querying for video / audio recordings only https://tickets.metabrainz.org/browse/MBS-13093
      • reosarevok
        Happy to get suggestions on better inc= naming / options
      • I think for this situation, they would want to *both* use something like this and skip [unknown], but it seems to be more widely useful?
      • yvanzo: remind me, does a search query already allow to say "show me only videos"?
      • yvanzo
        probably
      • there is a video boolean flag
      • reosarevok: I updated Edit_Note wikidoc to match the terms after the PR: modify/remove.
      • zas
        yvanzo: can we reboot rakim? it runs sir3 and a pg instance
      • yvanzo
        zas: let me stop the two MB prod containers first
      • aerozol
        reosarevok: Are you fixing their code? :)
      • zas
        yvanzo: not strictly needed as health checks will remove them from gateways, stopping them manually isn't much better
      • yvanzo
        zas: all set
      • zas
        ok, rebooting
      • yvanzo
        zas: I just set the service-maintenance flag.
      • aerozol
        reosarevok: missed your ticket link somehow, now I get what you’re up to
      • zas
        remaining servers will need coordination: aretha clash flobot floyd gaga kiss patton paco pink quest serge trille wolf zappa
      • moroder is hosting grafana & prometheus, so it might generate alerts, but can be rebooted
      • petitminion has quit
      • mingus is discourse, so we need to announce the downtime
      • mayhem
        I rebooted wolf, and it is still behaving ok. probably ok to remove from the list, zas.
      • zas
        mayhem: ah yes
      • thanks
      • yvanzo: rakim is back
      • I excluded jackson5 servers, as they don't run docker
      • and there's also sentry server, but I don't know much about it
      • BrainzGit
        [troi-recommendation-playground] 14mayhem merged pull request #98 (03main…weekly-patches): Turn daily-jams patch into periodic-jams https://github.com/metabrainz/troi-recommendati...
      • yvanzo
        zas: thanks, re-enabled MB prod on rakim.
      • zas: jackson5 (at least) isn’t listed in syswiki
      • zas
        yvanzo: they are ;) tito michael etc...
      • yvanzo
        Ok, thanks :)
      • CatQuest
        I was about to ask "jackson5 servers" ?
      • neat
      • yvanzo
        reosarevok: does wikidata-bot require any attention to be stopped?
      • bitmap, reosarevok, zas: also we tend to forget filling the incident logs recently (since March).
      • zas
        I added a banner to forums, I'll reboot the server in 15 minutes
      • yvanzo
        👍
      • CatQuest
        also i think you said "quest" about one of the servers so probably it's in the "is in use2 part now?
      • zas
        atj: consul-server service weren't restarted after reboot on rex & rudi
      • I did it manually
      • yvanzo
        looking into cron tasks on trille
      • zas
        atj: same for consul-client on ludwig
      • aerozol
        MeB team running like a well oiled machine
      • mayhem
        well oiled rickroll machine.
      • atj
        zas: will investigate
      • zas
        atj: restarting manually works without issue, so I guess that's the config preventing auto restart
      • I guess we'll have the same issue with mingus (which runs both a server & a client instances)
      • atj
        the role explicitly enables the services
      • CatQuest
        [11:34] <mayhem> well oiled rickroll machine. <---
      • zas
        atj: but Restart=on-failure
      • atj
        that's unrelated to whether the service starts on boot
      • it controls when systemd restarts an already started service if it dies
      • CatQuest
        anyway if anyone else minds i'll mov "quest" line from "suggested" to "in use" on https://wiki.musicbrainz.org/Hosting/Server_names but someone else will have to blup the jackson5
      • zas
        atj: if on-failure, it will not restart on clean exit, I guess it should be Always or on-success