#metabrainz

/

      • ishaanshah[m]
        and then collect the result it should work>
      • 2020-05-14 13555, 2020

      • ishaanshah[m]
        ?*
      • 2020-05-14 13509, 2020

      • iliekcomputers
        no, the query won't change the df you have registered
      • 2020-05-14 13519, 2020

      • iliekcomputers
        each query returns a new df
      • 2020-05-14 13539, 2020

      • iliekcomputers
        new_df = select * from view where ...
      • 2020-05-14 13541, 2020

      • ishaanshah[m]
        Ah, so again createTempView(new_df)
      • 2020-05-14 13545, 2020

      • iliekcomputers
        yes.
      • 2020-05-14 13553, 2020

      • ishaanshah[m]
        and run and collect
      • 2020-05-14 13556, 2020

      • iliekcomputers
        right
      • 2020-05-14 13500, 2020

      • ishaanshah[m]
        So we collect only once
      • 2020-05-14 13516, 2020

      • iliekcomputers
        yeah.
      • 2020-05-14 13540, 2020

      • ishaanshah[m]
        Ohk, got it thanks
      • 2020-05-14 13501, 2020

      • iliekcomputers
        there's a book called high performance spark by o'reilly that i'm planning on reading.
      • 2020-05-14 13550, 2020

      • ishaanshah[m]
        Hmm, I will also have a look at it
      • 2020-05-14 13512, 2020

      • ishaanshah[m]
        I found the spark documentation a bit confusing, maybe the book might help
      • 2020-05-14 13526, 2020

      • iliekcomputers
        our setup right now is pretty shit in terms of optimizations
      • 2020-05-14 13535, 2020

      • iliekcomputers
        we can probably improve it a lot
      • 2020-05-14 13544, 2020

      • iliekcomputers
        but it needs some expertise
      • 2020-05-14 13532, 2020

      • ishaanshah[m]
        Another qs
      • 2020-05-14 13551, 2020

      • ishaanshah[m]
        What do the files in the updates folder in admin mean?
      • 2020-05-14 13509, 2020

      • ishaanshah[m]
        admin/sql*
      • 2020-05-14 13525, 2020

      • iliekcomputers
        those are schema updates to the postgres db
      • 2020-05-14 13547, 2020

      • iliekcomputers
        if we want to make a change to the schema in the production db, we need a script we can run against it.
      • 2020-05-14 13555, 2020

      • ishaanshah[m]
        So, if I make changes to the stats schema should I write one too?
      • 2020-05-14 13515, 2020

      • iliekcomputers
        yes.
      • 2020-05-14 13527, 2020

      • iliekcomputers
        what change are you making?
      • 2020-05-14 13546, 2020

      • ishaanshah[m]
        Adding columns for stats
      • 2020-05-14 13500, 2020

      • iliekcomputers
        umm
      • 2020-05-14 13501, 2020

      • ishaanshah[m]
        I am thinking we should do <entity>_<time_range>
      • 2020-05-14 13526, 2020

      • iliekcomputers
        can you give an example?
      • 2020-05-14 13539, 2020

      • iliekcomputers
        i'd rather just put it in the artist jsonb column we have right now.
      • 2020-05-14 13551, 2020

      • ishaanshah[m]
        artist_all_time, artist_last_month, recording_all_time
      • 2020-05-14 13515, 2020

      • iliekcomputers
        and those are jsonb columns?
      • 2020-05-14 13523, 2020

      • ishaanshah[m]
        We decide to have different columns yesterday right
      • 2020-05-14 13525, 2020

      • ishaanshah[m]
        yes
      • 2020-05-14 13540, 2020

      • iliekcomputers
        did we? i'm sorry, i must have gotten confused.
      • 2020-05-14 13500, 2020

      • iliekcomputers
        what's the benefit of creating different columns vs putting it in the artist json?
      • 2020-05-14 13540, 2020

      • ishaanshah[m]
        We dont have to process the whole json?
      • 2020-05-14 13558, 2020

      • ishaanshah[m]
        If I want only all_time stats I just get them
      • 2020-05-14 13559, 2020

      • iliekcomputers
        you can write queries in postgres to get certain subparts of the json
      • 2020-05-14 13520, 2020

      • iliekcomputers
      • 2020-05-14 13531, 2020

      • iliekcomputers
        the `->>` operator
      • 2020-05-14 13553, 2020

      • ishaanshah[m]
        Oh
      • 2020-05-14 13506, 2020

      • ishaanshah[m]
        I did not know that
      • 2020-05-14 13506, 2020

      • iliekcomputers
        the insert will be a bit complicated.
      • 2020-05-14 13540, 2020

      • iliekcomputers
        however, the idea is that when it'll get too complicated, we'll look into designing an actual better rdbms style schema
      • 2020-05-14 13500, 2020

      • iliekcomputers
        until then, let's just keep doing our jsonb things in the columns we already have.
      • 2020-05-14 13510, 2020

      • ishaanshah[m]
        Is there any con for having multiple columns?
      • 2020-05-14 13512, 2020

      • iliekcomputers
        i don't want to do schema changes until we know all the stats that we're computing, because then we can actually build tables.
      • 2020-05-14 13521, 2020

      • ishaanshah[m]
        Ohk
      • 2020-05-14 13540, 2020

      • ishaanshah[m]
        So ig the schema of json shoud be-
      • 2020-05-14 13503, 2020

      • ishaanshah[m]
        {all_time:{....}, last_month:{...}, ...}?
      • 2020-05-14 13508, 2020

      • iliekcomputers
        yes.
      • 2020-05-14 13517, 2020

      • iliekcomputers
        imo
      • 2020-05-14 13558, 2020

      • ishaanshah[m]
        Ohk, I will have a update my code a bit then
      • 2020-05-14 13535, 2020

      • Mr_Monkey
        CatQuest: That would be useful, but I don't currently have the bandwidth to investigate userscripts
      • 2020-05-14 13539, 2020

      • iliekcomputers
        a complicated schema in the jsonb also gives us a nice usecase to setup jsonschema and see if it'll fit our other usecases as well.
      • 2020-05-14 13553, 2020

      • Mr_Monkey
        Maybe if some userscript wiz want to take a stab at it…?
      • 2020-05-14 13514, 2020

      • ephemer0l_ joined the channel
      • 2020-05-14 13532, 2020

      • CatQuest
        🙏 please someone
      • 2020-05-14 13505, 2020

      • ishaanshah[m]
        Yep, makes sense
      • 2020-05-14 13521, 2020

      • ishaanshah[m]
        I will first work with the spark side Ig
      • 2020-05-14 13538, 2020

      • iliekcomputers
        cool, thanks.
      • 2020-05-14 13544, 2020

      • CatQuest
        I'm going to add so many tickets Mr_Monkey but first: works have no way of saying "this was written in [year]"
      • 2020-05-14 13544, 2020

      • CatQuest
        also "revisionnotes have these
      • 2020-05-14 13544, 2020

      • CatQuest
        it's kind of annoying"
      • 2020-05-14 13544, 2020

      • CatQuest
        and links aren't made int olinks in revision notes
      • 2020-05-14 13520, 2020

      • ishaanshah[m]
        Those were all my qs, thanks for your time
      • 2020-05-14 13528, 2020

      • iliekcomputers
        awesome, no worries.
      • 2020-05-14 13545, 2020

      • iliekcomputers
        i don't think i'll be able to make it to our catchup later today.
      • 2020-05-14 13500, 2020

      • iliekcomputers
        ping me if you have any questions, otherwise let's catch up tomorrow.
      • 2020-05-14 13509, 2020

      • ishaanshah[m]
        Ok sure
      • 2020-05-14 13550, 2020

      • v6lur has quit
      • 2020-05-14 13557, 2020

      • Chinmay3199 joined the channel
      • 2020-05-14 13535, 2020

      • BrainzGit
        [musicbrainz-docker] yvanzo opened pull request #148 (master…expose-db-config): Set Postgres configuration parameters from command https://github.com/metabrainz/musicbrainz-docker/…
      • 2020-05-14 13545, 2020

      • yvanzo
        ruaok: this way is easier to customize and supported upstream ^
      • 2020-05-14 13537, 2020

      • pristine__
        ruaok: how do we want the data to look like when sent over lemmy?
      • 2020-05-14 13512, 2020

      • pristine__
      • 2020-05-14 13514, 2020

      • CatQuest
        Mr_Monkey: can we create a jira component "test server" maybe?
      • 2020-05-14 13518, 2020

      • pristine__
        Something like this?
      • 2020-05-14 13544, 2020

      • iliekcomputers
        pristine__: it is up to you, you'll have to change spark_reader.py to read it and save in the db as well
      • 2020-05-14 13553, 2020

      • Mr_Monkey
        CatQuest: Maybe a tag?
      • 2020-05-14 13559, 2020

      • CatQuest
        hm
      • 2020-05-14 13519, 2020

      • CatQuest
        i can't add tags on jira 😹
      • 2020-05-14 13528, 2020

      • jmp_music joined the channel
      • 2020-05-14 13542, 2020

      • pristine__
        iliekcomputers: right, wanted to discuss before moving forward. If ruaok has some input :)
      • 2020-05-14 13515, 2020

      • Mr_Monkey
        Ah, I thought they were free for all
      • 2020-05-14 13554, 2020

      • Mr_Monkey
        What would the tag/component mean CatQuest ?
      • 2020-05-14 13509, 2020

      • Mr_Monkey
        (Label, not tag)
      • 2020-05-14 13520, 2020

      • CatQuest
        uh behaviour that's valid on the test server (but not for prod/beta) (ex merging right now)
      • 2020-05-14 13513, 2020

      • yvanzo
        this could be a fixVersion then.
      • 2020-05-14 13518, 2020

      • Mr_Monkey
        There's now a testing-server label
      • 2020-05-14 13553, 2020

      • yvanzo
        there is a version 'beta' we use in MBS to address regressions that occur on beta.mb.o but are fixed before releasing to mb.o.
      • 2020-05-14 13506, 2020

      • CatQuest
        ja
      • 2020-05-14 13528, 2020

      • CatQuest
        .. I stil ldon't se any way to add a "label" on add ticket
      • 2020-05-14 13517, 2020

      • Mr_Monkey
        Once the ticket is created you can add a label.
      • 2020-05-14 13501, 2020

      • CatQuest
        hm. anyway it seems that al the werid name and alias and such non-sticking for mmy initial merging test isn't valid anymore. you've been working on them?
      • 2020-05-14 13525, 2020

      • Mr_Monkey
        I have indeed been working on fixing all the issues I could find. Not sure I remember exactly which one you're referring to, but I did fix an issue about multiple default aliases that could match what you describe.
      • 2020-05-14 13552, 2020

      • ruaok
        pristine__: yes, that approach looks good, but please let me review the actual schema for the CF filtered recordings.
      • 2020-05-14 13526, 2020

      • CatQuest
        Mr_Monkey: I tried merging soem stuff and choosing a differnt name and such. and it seems it always selects the name i chose now so that issue seems gone atleast :D
      • 2020-05-14 13555, 2020

      • pristine__
      • 2020-05-14 13506, 2020

      • ruaok
        yvanzo: so the goal is to allow people who want to change the default PG config to create a custom docker-compose file and then put that into place with the admin/config command?
      • 2020-05-14 13524, 2020

      • pristine__
        ruaok: this looks good. Just need to change recording_msid to recording_mbid
      • 2020-05-14 13528, 2020

      • pristine__
        Imo
      • 2020-05-14 13537, 2020

      • CatQuest
        Mr_Monkey: I created https://tickets.metabrainz.org/browse/BB-471 instead (since it's a pretty big issue. imho)
      • 2020-05-14 13538, 2020

      • BrainzBot
        BB-471: Search Issues and the Parameters of Invisibility
      • 2020-05-14 13520, 2020

      • ruaok
        pristine__: yes, agreed. I'm glad that we already had all of this in mind. :)
      • 2020-05-14 13503, 2020

      • pristine__
        <3
      • 2020-05-14 13522, 2020

      • Mr_Monkey
        CatQuest: Thanks for the ticket. Yeah, that's something that has been missing for sure on all pages with pagination
      • 2020-05-14 13536, 2020

      • CatQuest
        i'm creating several tickets :D
      • 2020-05-14 13545, 2020

      • yvanzo
        ruaok: yes, for example a file local/compose/custom-db-options.yml like https://gist.github.com/yvanzo/2c42a8bf8fd78853c4…
      • 2020-05-14 13525, 2020

      • Lotheric_ joined the channel
      • 2020-05-14 13536, 2020

      • CatQuest
        Mr_Monkey: also leaving votes and notes on other tickets
      • 2020-05-14 13525, 2020

      • CatQuest
        Mr_Monkey: should I create a ticket each for all these identifiers or make one big ticket or jsut maybe we cna modify this one for "add more work identifiers: https://tickets.metabrainz.org/browse/BB-462 ?
      • 2020-05-14 13526, 2020

      • BrainzBot
        BB-462: Allow VIAF identifiers for Work entities
      • 2020-05-14 13539, 2020

      • CatQuest
        (see my comment)
      • 2020-05-14 13553, 2020

      • Lotheric has quit
      • 2020-05-14 13548, 2020

      • Mr_Monkey
        One big ticket is probably best, And they can be added in one go
      • 2020-05-14 13546, 2020

      • CatQuest
        yash. so do I link VIAF ticket to it or edit VIAF ticket?
      • 2020-05-14 13547, 2020

      • loujine has quit
      • 2020-05-14 13519, 2020

      • Mr_Monkey
        You can link to the VIAF
      • 2020-05-14 13514, 2020

      • Mr_Monkey
        ruaok: Still can't play anything on LB? (which browser & version, by the way?)
      • 2020-05-14 13530, 2020

      • ruaok
        still cant: `NetworkError when attempting to fetch resource.` posting to lb.org nor beta.lb.org works. FF 76.0.1 Mac 10.15.4
      • 2020-05-14 13502, 2020

      • Mr_Monkey
        Care to try another browser, see if that's the issue?
      • 2020-05-14 13506, 2020

      • ruaok
        even though the data seems to load fine, I bet there is something with my posted file....
      • 2020-05-14 13513, 2020

      • Mr_Monkey
        Does it say what is throwing that error?
      • 2020-05-14 13531, 2020

      • Mr_Monkey
        I tried the file you sent last night, it's opening fine for me
      • 2020-05-14 13531, 2020

      • ruaok
        can you try this file and see if it works for you?
      • 2020-05-14 13537, 2020

      • ruaok
        ah. heh.
      • 2020-05-14 13542, 2020

      • ruaok
        ok, lemme try chrome.
      • 2020-05-14 13531, 2020

      • BrainzGit
        [musicbrainz-server] reosarevok merged pull request #1516 (master…icon-updates): Updated Facebook, Google Play, and Spotify icons. https://github.com/metabrainz/musicbrainz-server/…
      • 2020-05-14 13505, 2020

      • ruaok
        FF is the culprit.
      • 2020-05-14 13518, 2020

      • Mr_Monkey
        OK, let me install FF and see what's waht
      • 2020-05-14 13540, 2020

      • ruaok
      • 2020-05-14 13550, 2020

      • Mr_Monkey
        Yeah, I was testing that.
      • 2020-05-14 13556, 2020

      • ruaok
        I think I am cursed. brainzpayer really doesn't like me.
      • 2020-05-14 13502, 2020

      • reosarevok
        PAY ALL THE BRAINZ
      • 2020-05-14 13513, 2020

      • Mr_Monkey
        Most of the songs aren't on spotify, so logical it shouldn't find them. Some of them you can find, but the title is a bit different
      • 2020-05-14 13544, 2020

      • Mr_Monkey
        ruaok: Are none of the songs playing?
      • 2020-05-14 13548, 2020

      • ruaok
        and this is the reason why I was suggesting that we work on a spotify limited dataset. most of my shit is just not there.
      • 2020-05-14 13504, 2020

      • ruaok
        in chrome, I get 3 songs to play.
      • 2020-05-14 13504, 2020

      • Mr_Monkey
        I'm working on adding other datasources
      • 2020-05-14 13521, 2020

      • ruaok
        geting YT in there might really help.
      • 2020-05-14 13549, 2020

      • Mr_Monkey
        Yep, that what I'm looking at
      • 2020-05-14 13519, 2020

      • m00n joined the channel
      • 2020-05-14 13526, 2020

      • m00n
        o7 all
      • 2020-05-14 13544, 2020

      • m00n
        im wondering, is it possible to copy files upon saving rather than moving them?
      • 2020-05-14 13509, 2020

      • ruaok
        m00n? I just to love in a place calle dhte moon and it spelled exactly like that.
      • 2020-05-14 13523, 2020

      • ruaok
        I take it you're talking about picard?
      • 2020-05-14 13530, 2020

      • m00n
        yeah sorry