#metabrainz

/

      • CatQuest joined the channel
      • lucifer
        !m aerozol
      • BrainzBot
        You're doing good work, aerozol!
      • CatQuest
        yes it seems i have 2 hours and in those thre was an aerozol's important message
      • aerozol
        Haha lucifer was helping me not break stuff with a test server setup
      • Finally pull request was done, no conflicts 🎉
      • monkey
        Good work aerozol !
      • aerozol
        Yay thanks monkey!
      • BrainzGit
        [listenbrainz-server] 14MonkeyDo merged pull request #2150 (03master…follower-following-modal): Changing default modal view to ‘following’ https://github.com/metabrainz/listenbrainz-serv...
      • monkey
        Stoked to see you enact change on LB aerozol :)
      • lucifer
        bitmap: hi! while parsing the dbmirror2 packets, python's json decoder threw an invalid json error on 1 line. in case its relevant, here it is:
      • alastairp: if my counts are correct then at max 60 laus relevant to us were deleted in the past month.
      • i'd be fine with incrementally updating the rest and updating link fully every week or so. maybe also try to optimize the full metadata cache generation to the extent possible. i have some ideas to try out in that direction.
      • alastairp
        lucifer: awesome
      • BrainzGit
        [troi-recommendation-playground] 14amCap1712 merged pull request #67 (03main…daily-jams-accept-day): Accept jam_date as optional argument in daily-jams patch https://github.com/metabrainz/troi-recommendati...
      • alastairp
        lucifer: interesting that you said that the query to get a new set of artist rels is pretty quick too - but I think artist rels are part of an artist, not as a top level key in the recording json?
      • so we'd need to get a full artist blob for each recording, right? and then replace the "artists" key in _all_ recordings
      • lucifer
        alastairp: yes, indeed. but we can restructure the cache or do a partial update instead of insert.
      • alastairp
        lucifer: ok, yeah, right.
      • so, I like this idea of moving things around to make updates easier, or change the way that we do updates
      • lucifer
        we can also build a temp table of artist rels and index on artist mbid and then join to it instead of having it in CTE.
      • and in any case everything should happen at once when we run the mb_metadata_cache script.
      • alastairp
        yes, right
      • you said that the cte for artist rels is pretty quick, does that speed carry over to actually writing the data to a table and building the index too?
      • just putting another idea out there - is splitting each of these CTEs into a separate temp table and then running the final query as slow as running the entire query in one go?
      • lucifer
        wrtiting the data takes 1 hour for full table. it only begins after the query has finished executing iiuc.
      • yes, splitting each cte and indexing before running final query is one of the approaches i want to check.
      • BrainzGit
        [troi-recommendation-playground] release 03v-2022-09-29 has been published by 14amCap1712: https://github.com/metabrainz/troi-recommendati...
      • lucifer
        mayhem: does run_daily_jams sound good for https://github.com/metabrainz/listenbrainz-serv... ?
      • mayhem
        yeah
      • lucifer
        👍
      • BrainzGit
        [listenbrainz-server] 14MonkeyDo merged pull request #2171 (03master…dependabot/npm_and_yarn/json-schema-and-jsprim-0.4.0): Bump json-schema and jsprim https://github.com/metabrainz/listenbrainz-serv...
      • [listenbrainz-server] 14dependabot[bot] opened pull request #2186 (03master…dependabot/npm_and_yarn/postcss-and-stylelint-less-8.4.16): Bump postcss and stylelint-less https://github.com/metabrainz/listenbrainz-serv...
      • [listenbrainz-server] 14MonkeyDo merged pull request #2172 (03master…dependabot/npm_and_yarn/nth-check-and-cheerio-2.1.1): Bump nth-check and cheerio https://github.com/metabrainz/listenbrainz-serv...
      • [listenbrainz-server] 14dependabot[bot] opened pull request #2187 (03master…dependabot/npm_and_yarn/node-fetch-and-nivo/bar-and-nivo/calendar-and-nivo/core-and-nivo/geo-and-nivo/heatmap-and-nivo/legends-and-nivo/tooltip--removed): Bump node-fetch, @nivo/bar, @nivo/calendar, @nivo/core, @nivo/geo, @nivo/heatmap, @nivo/legends and @nivo/tooltip https://github.com/metabrainz/listenbrainz-serv...
      • [listenbrainz-server] 14MonkeyDo closed pull request #2187 (03master…dependabot/npm_and_yarn/node-fetch-and-nivo/bar-and-nivo/calendar-and-nivo/core-and-nivo/geo-and-nivo/heatmap-and-nivo/legends-and-nivo/tooltip--removed): Bump node-fetch, @nivo/bar, @nivo/calendar, @nivo/core, @nivo/geo, @nivo/heatmap, @nivo/legends and @nivo/tooltip https://github.com/metabrainz/listenbrainz-serv...
      • reosarevok
        mayhem: did you mean the normalizing email or the Italian one? :)
      • Anyway, I'm dealing with both, but can you check the cover art one?
      • (heh, or the Spanish answer)
      • mayhem
        both. :)
      • reosarevok
        Answered all 3 now
      • mayhem
        thanks
      • lucifer
        mayhem: i updated the PR to move the debug playlists back to post recommendation step, only daily jams run hourly now. that was the easiest way to keep debug playlists and not generate them everyhour in my understanding.
      • mayhem
        makes sense
      • lucifer
        also, tested it just not on cron by changing my time zone to US/Hawaii. will merge once tests pass.
      • BrainzGit
        [listenbrainz-server] 14amCap1712 merged pull request #2183 (03master…troi-hourly): Run troi bot hourly https://github.com/metabrainz/listenbrainz-serv...
      • monkey
        lucifer: I spotted a weird mapping issue in my listens, full report here: https://gist.github.com/MonkeyDo/d6652584e0fa11...
      • Ah, well there's already LB-1116 open for that I guess
      • BrainzBot
        LB-1116: Use Album/Release info when matching https://tickets.metabrainz.org/browse/LB-1116
      • lucifer
        monkey: check again. its fixed now as in showing the same mbid as shown on the labs.api page. i am not sure why it chose the compilation earlier
      • BrainzGit
        [listenbrainz-server] 14amCap1712 opened pull request #2188 (03master…daily-jams-playlist): Fix daily jams notification url https://github.com/metabrainz/listenbrainz-serv...
      • monkey
        Huh.
      • lucifer
        monkey: to be clear, i invalidated the match so that the mapper attempted to find a new match and this time it found the right once.
      • monkey
        Ah, I see.
      • Any way we could improve the odd ones by using the release name in the listen?
      • lucifer
        the mapper should be deterministic and find the same match each time unless there are some code changes or new additions to the database. its neither in this case so not sure why it chose the wrong one earlier.
      • yes using release name in the listen is a planned improvement. mayhem can probably tell you more about the exact plan.
      • monkey
        Thanks for the explanations !
      • lucifer
        mayhem: can you review LB#2188. i'll do a release after that.
      • BrainzBot
      • lucifer
        monkey: any other PRs to merge?
      • reosarevok
        bitmap: you're already doing MBS-12644 but now there's a ticket ;) In case you want to rename the commit or something
      • BrainzBot
        MBS-12644: Beta: Shorten link phrases in Autocomplete2 https://tickets.metabrainz.org/browse/MBS-12644
      • bitmap
        lucifer: hi, do you know which packet # that line is from by chance? and are you importing the packets into postgres first, or parsing the files directly?
      • lucifer
        bitmap: parsing directly by reading the file as csv with excel-tab dialect and then trying to parse olddata/newdata column as json. i don't know the packet number currently but can find it.
      • bitmap: packet number 152588
      • oh i see in the packet its like this, so maybe it is csv parsing that is messing it?
      • bitmap
        lucifer: yea, I think the parser might be confused? the escaping in the csv looks correct to me
      • lucifer
        makes sense. thanks for checking
      • bitmap
        if I do \copy dbmirror2.pending_data from 'mbdump/pending_data'; then postgres imports it ok
      • (the pending_data table does use the json data type, so it would fail if the json was invalid at least)
      • reosarevok: thanks :) I'll update the commit then
      • lucifer
        i see, makes esnse
      • mayhem returns from a business lunch
      • monkey
        lucifer: I can finish up LB##2182 , just need to merge and update snapshots
      • LB#2182
      • BrainzBot
        LB-1125: Add a button to reduce YT player: https://github.com/metabrainz/listenbrainz-serv...
      • lucifer
        mayhem: C-Suites meeting i presume ;)
      • monkey: 👍
      • BrainzGit
        [listenbrainz-server] 14amCap1712 merged pull request #2188 (03master…daily-jams-playlist): Fix daily jams notification url https://github.com/metabrainz/listenbrainz-serv...
      • lucifer
        alastairp: if you have time, can you please review BU#95
      • BrainzBot
        Enable sqlalchemy warnings and update mbdata: https://github.com/metabrainz/brainzutils-pytho...
      • mayhem
      • One of the people at lunch was the chief behind the Free Music Archive.
      • do you see the "support this artist" button?
      • as you can see we're kindred spirits.
      • lucifer
        ah nice!
      • alastairp
        ah, great. nice looking page too!
      • I didn't know how much I was missing one of these
      • mayhem
        I'm going to meet with them again in the morning, brainstorming about collaboration
      • lucifer
        alastairp, mayhem, monkey: the lfm loved tracks import is live on test.lb and seems to work fine in case you want to try it out.
      • monkey
        Sweeeet! :)
      • alastairp
        thanks lucifer !
      • not sure I ever loved any tracks on lfm
      • lucifer
        `curl 'https://test-api.listenbrainz.org/1/feedback/import' -H 'Authorization: Tpken YOUR_TOKEN' -H 'Content-Type: application/json' -d '{"service": "lastfm", "user_name": "YOUR_LFM_USERNAME"}'`
      • currently, no UI available so have to use api directly.
      • alastairp
        hah
      • Tpken?
      • lucifer
        ah yes, typo. should be Token
      • monkey
        One pain I have in my personal workflow is I use the Pano Scrobbler app on my phone which connects to LFM/LB, but loved tracks are only sent to LFM. Not I'll be able to import them :)
      • lucifer
        currently we only import loved tracks that have a mbid assignned to them by LFM.
      • we could change it to lookup mbids ourself from mapper for tracks that do not have a mbid assigned or maybe look it up for all.
      • alastairp: LFM uses track mbids in this endpoint. maybe that's where the confusion between track mbid and recording mbid in MLHD came from ?
      • alastairp
        lucifer: hmm, no, we were definitely seeing both recording and track mbids in the same field from the response of some API
      • I think the field name wwas "track_mbid" or something (as a fallback to pre-NGS)
      • lucifer
        ah ok, i see.
      • alastairp
        lucifer: I'll try and take a look at the BU PR when I get home
      • lucifer
        for a couple of random users, i imported on a test account. i get: `{"inserted":3990,"invalid_mbid":0,"mbid_not_found":291,"missing_mbid":3745,"total":8026}`
      • alastairp
        though, actually. maybe I'll just do it now, I trust all of your code :)
      • lucifer
        its possible some of those 291 not found are recording mbids
      • alastairp
        ohhh, interesting
      • lucifer
        hehe. no hurry :)
      • kepstin
        i think i actually noticed recently that last.fm switched from returning recording ids to track ids in the same field, yeah
      • so older tracks imported from last.fm before some date will have recording ids, after they'll have track ids.
      • alastairp
        thanks kepstin
      • kepstin
        iirc i opened an issue back when i noticed that
      • BrainzBot
        LB-431: Last.fm api returns track mbid instead of recording mbid for new scrobbles
      • alastairp
        so, I think we may have known this - it's why I was originally against importing the recording ids from lastfm in the first place
      • BrainzGit
        [listenbrainz-server] 14MonkeyDo merged pull request #2182 (03master…lb-1125-reduce-youtube-player): LB-1125: Add a button to reduce YT player https://github.com/metabrainz/listenbrainz-serv...
      • alastairp
        then we decided to add them anyway, but after identifying this, we changed them importer to put them in another field which wasn't the main LB recording mbid field
      • I don't think we've done anything since then - we could definitely go back and look at all the fields we have in our listens and see which ones can map to recordings and which to trakcs
      • lucifer
        with the mapper its much less relevant anyway.
      • alastairp
        yes, right
      • still, it'd be curious to see where lfm and mapper differ
      • lucifer
        yes makes sense
      • BrainzGit
        [listenbrainz-server] 14MonkeyDo merged pull request #2170 (03master…lb-1020): LB-1020: Remove 12-hour formatting from listencard timestamps https://github.com/metabrainz/listenbrainz-serv...
      • monkey
        That's all the PRs I had ready lucifer, thanks for waiting
      • lucifer
        np, thanks. will do a release later today
      • alastairp
        !m monkey
      • BrainzBot
        You're doing good work, monkey!
      • alastairp
        monkey: how goes the upgrade?
      • I saw you closed dependabot, not sure if it was getting noisy due to changes you made, or was just being annoying
      • monkey
        Took a break from diving into nivo internals.
      • I closed one dependabot PR that was doing the same upgrade I've started, but without implementing the required changes (i.e. still considered a point release despite some APIs and props completely changed
      • But I'm getting there
      • alastairp
        nice
      • monkey
        I'm gonna have to deploy that one to test.LB to ensure all the graphs are working properly
      • testicon has left the channel
      • lucifer
        monkey: oh lfm also an api to fetch user's tags as well. https://www.last.fm/api/show/user.getPersonalTags
      • antlarr has quit
      • bitmap
        zas: hey, when you are around can you check the openresty logs for prod.caa-redirect.access.log? there seems to be a lot of requests being spammed from a particular user agent
      • related to CAA-141
      • BrainzBot
        CAA-141: Too many 429 when viewing cover art https://tickets.metabrainz.org/browse/CAA-141
      • SothoTalKer has quit
      • SothoTalKer joined the channel
      • antlarr joined the channel