#metabrainz

/

      • alastairp
        personally, I think that a deleted listens table is one more moving part in the system and that if we can get away without doing it, that would be better (until someone complains I guess)
      • lucifer
        alastairp: we already have that table.
      • alastairp
        oh, well then. that changes things
      • :)
      • zas
        Yes, we do. One limitation is arising: we cannot use 10.2.2.x network, so it comes as an additional network, but all nodes will be connected through it.
      • lucifer
      • alastairp
        sure, so if we have the table, and the direct db connection (soon), and it just requires one more query in spark, then let's do it
      • lucifer
        this table is currently cleared hourly.
      • alastairp
        sorry, I wasn't sure from your original comment how much of this was already in place
      • lucifer
        i'd change that to clear weekly so that there is ample time to copy over data + more impl details.
      • ah no worries. should've explained in more detail.
      • cool, i'll work on this when the virtual network is in place.
      • alastairp, mayhem: if you have some time, comments on this would be nice: https://www.irccloud.com/pastebin/yK7jxqbb/
      • new listens table schema
      • alastairp
        right, I thought you were asking "should we start storing deleted listens so that we can improve stats?" rather than "should we periodically copy our existing deleted listens to spark so that we can improve stats?"
      • mayhem
        sorry
      • alastairp
        lucifer: yeah, just finishing up deleted listens first :)
      • lucifer
        alastairp: ah yes makes sense
      • alastairp
        lucifer: why is track_name gone?
      • mayhem
        it exists in data.
      • lucifer
        it'll be part of jsonb data
      • BrainzGit
        [listenbrainz-android] 14dependabot[bot] opened pull request #26 (03main…dependabot/gradle/androidx.compose.material3-material3-1.1.0-alpha04): Bump material3 from 1.1.0-alpha03 to 1.1.0-alpha04 https://github.com/metabrainz/listenbrainz-andr...
      • alastairp
        lucifer: btw, I guess we didn't delete this after migrating to user_id: https://github.com/metabrainz/listenbrainz-serv...
      • BrainzGit
        [listenbrainz-android] 14dependabot[bot] opened pull request #27 (03main…dependabot/gradle/androidx.compose.material3-material3-window-size-class-1.1.0-alpha04): Bump material3-window-size-class from 1.1.0-alpha03 to 1.1.0-alpha04 https://github.com/metabrainz/listenbrainz-andr...
      • [listenbrainz-android] 14dependabot[bot] opened pull request #28 (03main…dependabot/gradle/com.google.android.material-material-1.8.0-rc01): Bump material from 1.8.0-beta01 to 1.8.0-rc01 https://github.com/metabrainz/listenbrainz-andr...
      • mayhem
        this looks ok to me lucifer .
      • alastairp
        and recording_msid is the reverse - pull it out of `data` because we use it often?
      • lucifer
        i intend to use recording msid instead of track name for unique index henceforth.
      • alastairp
        excellent
      • lucifer
      • its because of this
      • consistency in how we deduplicate listens
      • alastairp
        ah, cool. thanks
      • lucifer on top of things, as usual
      • lucifer
        hehe :)
      • alastairp
        yes, the schema looks fine to me too. my intuition is that the listened_at and tz_offset is correct for the types of computations that we need to do, but it'd be good to try and actually get the required data from a test db before we go ahead with the full migration
      • lucifer
        makes sense
      • alastairp
        when you do `at time zone 'x'` in an sql query, do you know in what formats x can be?
      • does it need to be 'Region/City'?
      • or would we just do `SELECT listened_at + tz_offset as local_time` ?
      • lucifer
        the latter
      • alastairp
        we need to check what timezone pysopg2 will put into the resulting datetime object in this case
      • lucifer
        should be utc i think but need to check.
      • alastairp
        specifically, we should check if it's utc or server time
      • because `SELECT listened_at + tz_offset at time zone 'UTC' as local_time` is wrong too
      • anwyay, we can try it out and see what the results are
      • lucifer
        yeah makes sense.
      • alastairp
        I have a feeling that it might be better to extract listened_at always as UTC, and update the listen object with a datetime at the correct timezone (using `.astimezone()`) in python - just to avoid getting an object out of the database with an attached tz, but then it actually being the date at a different tz
      • but as I said, we can test
      • lucifer
        i don't think we can get a correct timezone without storing timezone instead of offset.
      • alastairp
        yes, I suspect you're right
      • lucifer
        or i guess we could make a table to map INTERVAL 5h 30 mins to UTC +530 time zone and use that
      • alastairp
        but I think that's fine, I have no problem with trying it out and seeing what the code looks like
      • atj
        what about DST changing offsets?
      • lucifer
        the client would submit an offset so we are hoping that the client is applying DST on its end.
      • atj
        ah ok
      • jivte_ has quit
      • alastairp
        atj: this is why we settled on storing +0100 instead of 'BST' or 'Europe/London'
      • atj
        yeah, i see the logic, as the api client will know the offset at the time of the listen
      • alastairp
        yep
      • (also, I tried to work out how to automatically calculate 'Europe/Madrid' from python without user input... not easy
      • monkey
        Nothing is ever easy when it comes to time, is it?
      • alastairp
        monkey: let me take some time and get back to you on that...
      • monkey
        And I guess we should be thankful that we don't have to handle relativistic time (yet)
      • alastairp
        well, we allow people to import listens from the past
      • lucifer
        what about listens from the future
      • atj
        "how to automatically calculate 'Europe/Madrid' from python" you mean going from offset -> zone name?
      • antlarr has quit
      • alastairp
        atj: yes
      • jivte_ joined the channel
      • antlarr joined the channel
      • atj
        not sure that's possible, given each offset may match multiple timezones?
      • alastairp
        atj: right, and another motivation to only use UTC hour offsets instead of named time zones
      • Sophist_UK has quit
      • antlarr has quit
      • antlarr joined the channel
      • antlarr has quit
      • antlarr joined the channel
      • vibhoo_24 joined the channel
      • bitmap
        reosarevok: yeah, that doc is out of date, we no longer pass $c as a prop (you have to use react context or getCatalystContext if not inside a component)
      • reosarevok
        Thought so. I'll update it
      • antlarr has quit
      • antlarr joined the channel
      • bitmap
        re MBS-8668 do you have an example of which error no longer triggers?
      • BrainzBot
        MBS-8668: Adding duplicate entities to a series is still allowed https://tickets.metabrainz.org/browse/MBS-8668
      • vibhoo_24 has quit
      • antlarr has quit
      • reosarevok
        See the PR comments :)
      • vibhoo_24 joined the channel
      • antlarr joined the channel
      • bitmap
        thanks
      • zas
        note: we are deploying vswitch changes to all nodes right now, a new IP in 10.10.10.x network will be added
      • antlarr has quit
      • antlarr joined the channel
      • agatzk has quit
      • antlarr has quit
      • TOPIC: MetaBrainz Community and Development channel | MusicBrainz non-development: #musicbrainz | BookBrainz: #bookbrainz | Channel is logged; see https://musicbrainz.org/doc/IRC for details | Agenda: Reviews
      • antlarr joined the channel
      • atj
        vswitch changes applied and nothing appears to be on fire
      • zas
        (yet) ;)
      • alastairp
        I can change that if you want?
      • jasje_ joined the channel
      • antlarr has quit
      • antlarr joined the channel
      • jasje_
        akshaaatt: in EntityTestUtils, what is the loadResourceAsString function?
      • antlarr has quit
      • zas
        I just restarted sir-prod, same error as usual
      • antlarr joined the channel
      • antlarr has quit
      • yvanzo
        alastairp, lucifer, mayhem: I added LB-1106 to the LB section of Docsprint.
      • BrainzBot
        LB-1106: Improve LB's documentation about RabbitMQ https://tickets.metabrainz.org/browse/LB-1106
      • antlarr joined the channel
      • yellowhatpro
        akshaaatt: I'll be afk for a while, won't be active for a week or so. I'll try to wrap up work on tickets for aerozol.
      • antlarr has quit
      • agatzk joined the channel
      • antlarr joined the channel
      • serial-ata joined the channel
      • BrainzGit
        [listenbrainz-server] 14alastair opened pull request #2338 (03master…canonical-release-connection): Generate canonical release redirect only in the mb schema https://github.com/metabrainz/listenbrainz-serv...
      • alastairp
        lucifer: ^ I think this is the only change we need for the mapper bug from the other month
      • BrainzGit
        [listenbrainz-server] 14alastair closed pull request #2288 (03master…canonical-data-mb-conn): Remove lb_conn option from canonical musicbrainz data dump https://github.com/metabrainz/listenbrainz-serv...
      • antlarr has quit
      • antlarr joined the channel
      • antlarr has quit
      • antlarr joined the channel
      • jivte has quit
      • Freso
        <BANG>
      • It’s Nothing Monday!
      • No mailed in reviews or notes, so…
      • People up: atj, bitmap, lucifer, monkey, reosarevok, alastairp, yvanzo, mayhem, zas, akshaaatt, CatQuest, Pratha-Fish, yellowhatpro, riksucks, Freso – anyone else who wish to give review, let me know ASAP. :)
      • atj: Go!
      • Sophist-UK joined the channel
      • atj
        hi
      • CatQuest
        fish
      • Pratha-Fish
        🐟
      • aerozol
        Mōrena! I’m here too Freso :)
      • CatQuest
        :D
      • vibhoo_24
        Freso: count me in
      • Pratha-Fish
        ^_^
      • atj
        last week I worked with zas to test deploying a Hetzner vSwitch to all servers so they can communicate securely
      • as part of that work i simplified the netplan and shorewall configuration in ansible so it should be a bit easier to understand
      • we just added the vSwitch VLAN to all servers today and no issues have been reported (so far)
      • aerozol: go
      • monkey
        vWell done
      • aerozol
        Kia ora! I had a lovely break and finally finished Hades (the game)
      • mayhem
        atj: thanks for working on that, thats going to be hugely helpful for us.
      • aerozol
        Apart from that I finally tidied some wiki pages (esp the userscripts one), kept on top of discussions, and have been doing lots of looking at other YIM-type stuff to inform the next one. Hoping to get onto the design early early
      • jivte_
        freso add me too :)
      • jasje_
        Freso: count me in