#metabrainz

/

      • D4RK-PH0ENiX joined the channel
      • d4rkie joined the channel
      • D4RK-PH0ENiX has quit
      • naught101_ joined the channel
      • D4RK-PH0ENiX joined the channel
      • d4rkie has quit
      • jsturgis joined the channel
      • jsturgis has quit
      • jsturgis joined the channel
      • jsturgis has quit
      • exogen joined the channel
      • jsturgis joined the channel
      • bitmap
        samj1912: sry I wasn't around, I'm looking at the pastes you linked and trying to wrap my head around things now :)
      • samj1912
        bitmap: o/
      • bitmap
        hey :)
      • samj1912
        I reduced it to a series of joins
      • The only problem is the last statement
      • I think we can convert it to an outer left join and figure and filter on the aliased join field
      • That will reduce the entire headache with deleted fields
      • But I am not sure how efficient that join would be
      • naught101_ has quit
      • jwf
        iliekcomputers: Quick question again… is this InfluxQL or PostgreSQL queries? https://github.com/metabrainz/listenbrainz-serv...
      • s/is this/are these
      • bitmap
        samj1912: is https://hastebin.com/raw/qonifutoka still relevant & can you give an example of a problematic statement?
      • samj1912
        bitmap those are the joins I constructed by considering only the second to last paths
      • iliekcomputers
        jwf: BigQuery standard sql
      • jwf
        Ooh, okay.
      • Have any recommended reads on BigQuery other than the upstream docs?
      • samj1912
        A problematic statement in that is if let's say we delete a release status. We won't know which releases to update.
      • bitmap
        but we can't delete a release status without the releases being updated first...unless I misunderstand you
      • samj1912
        Yes
      • That is correct
      • Although, I am not sure if I can 100% prove we will cover all the things, in case we add new paths
      • Another problem is tables with multiple FKs
      • *PKs
      • Let's say we update area, see how the statement for annotation update triggered by it looks like
      • On updating: `area`, We update entity: `annotation` via path: `areas`, ['type'] SELECT musicbrainz.annotation.id AS musicbrainz_annotation_id FROM musicbrainz.annotation JOIN musicbrainz.area_annotation ON musicbrainz.annotation.id = musicbrainz.area_annotation.annotation WHERE musicbrainz.area_annotation.area IN (:area_1) AND musicbrainz.area_annotation.annotation IN (:annotation_1)
      • bitmap
        to generalize the thing above, when a row is deleted...it doesn't makes sense to join with other tables that reference that row
      • d4rkie joined the channel
      • samj1912
        Hmm, yes
      • D4RK-PH0ENiX has quit
      • But we are joining them simply as a way to get ids
      • Which are already emitted from the trigger
      • Hmm bitmap, so you are saying if we are deleting something, we don't need to care about updating entities which are not emitted as FKs(so we discard those SQL statements) and update the rest of the paths?
      • Since they will be updated before we can delete the given row
      • bitmap
        well, the direction of the FKs has to be taken into account...if a release_status is deleted, that should be completely ignored, because it's only referenced *by* other tables, and those will have been updated already
      • if a release_alias is deleted, the release has to be updated because it references the release, not the other way around
      • if a row has FKs in both directions, only those need to be followed
      • samj1912
        Hmm, but the join method from the sql alchemy abstracts all of that, it is figures out which direction it is is from the path
      • I'll have to manually inspect each path and figure out the fk direction again
      • exogen has quit
      • bitmap
        I assume you only have to look at the direction of the first item in the path if it's a delete
      • samj1912
        Direction of the last item and the deleted item you mean?
      • bitmap
        CatQuest: hi, https://test.musicbrainz.org is now up & able to be edited again
      • samj1912: right, last item. for each path of the deleted row's table
      • samj1912
        Yup
      • I'll figure something out
      • bitmap
        I guess it should be checked in the index callback and when we're generating the triggers, since some delete triggers shouldn't even be generated
      • well, I guess changing how it generates the update map fixes both, nvm
      • I keep forgetting how this works
      • samj1912
        bitmap, meanwhile should I replace the generate selections code with the joins?
      • Dalek_ joined the channel
      • jsturgis has quit
      • bitmap
        samj1912: if you think it's an improvement, yes
      • samj1912
        not sure if its an improvement
      • just that it will generate bug free expressions and clean up the code, making it easier to understand
      • bitmap
        sounds like an improvement :)
      • samj1912
      • bitmap: all the entities and their manytoone rels
      • I guess these are the only ones that are to be updated on delete?
      • iliekcomputers
        jwf: not really, all I've been reading are docs :/
      • zas
        moiiin
      • bitmap: caa.org is naughty since ~1 hour, any known reason ?
      • alastairp: ping
      • adhawkins has quit
      • adhawkins joined the channel
      • alastairp
        hi zas
      • just looking now
      • load is pretty high; disk is pretty full. not sure what's using all of the space
      • /tmp is empty - not sure if you just cleaned it up, or if we have a similar issue to what happened last time
      • zas
        i didnt clean up anything yet
      • d4rkie has quit
      • D4RK-PH0ENiX joined the channel
      • D4RK-PH0ENiX has quit
      • D4RK-PH0ENiX joined the channel
      • UmkaDK_ joined the channel
      • UmkaDK has quit
      • yokel has quit
      • yokel joined the channel
      • samj1912
        bitmap: made some more progress
      • but the join thing wont work
      • its too inefficient
      • hmm, even the select statements are pretty bad
      • bitmap: ruaok we really need to take a look at our schema
      • *search schema
      • I managed to fix live indexing I think
      • afaict
      • bitmap: take a look at this https://github.com/metabrainz/sir/pull/55/commi... when you have time
      • the worst culprit in terms of reindexing is recording
      • we are indexing too much data for it
      • solr cant keep up, even when making reindexing queries
      • *not solr, psql
      • reosarevok has no idea about https://community.metabrainz.org/t/can-the-listenbrainz-db-be-updated-to-the-latest-musicbrainz-db/340633/12 - iliekcomputers and ruaok, I think we should have a clear pinned post about "Why only some LB listens are linked to MB", any takers? :p
      • MajorLurker has quit
      • Freso
        reosarevok: The Last.FM imported only matches to MB if Last.FM itself has their track linked to an MB Recording ID (sometimes they're linked wrongly!). Last.FM importing doesn't give MBIDs for Artists, Releases, etc.
      • pingupingu joined the channel
      • pingupingu has quit
      • pingupingu joined the channel
      • gcilou joined the channel
      • jwf|matrix
        iliekcomputers: Okay, no problem! If the docs are good, I'll work my way through them.
      • jsturgis joined the channel
      • pingupingu has quit
      • CatQuest
        +1 about linked post
      • Thanks bitmap!
      • jwf
        yvanzo: Oops, sorry, didn't realize MBS-9549 was a duplicate. But happy to see it's already fixed. :)
      • BrainzBot
        MBS-9549: "Homepage" field in user profile printed twice https://tickets.metabrainz.org/browse/MBS-9549
      • yvanzo
        No worry, I am glad that people are actually using beta and report bugs. :)
      • CatQuest
        usually i use beta anyway
      • jsturgis has quit
      • SothoTalKer
        when are changes to beta usually applied? :)
      • jwf
        beta++
      • SothoTalKer
        its suprising that something gets fixed in production earlier than in beta (:
      • CatQuest
        I've never heard of that happening
      • alastairp
        Freso: hi, I won't be around tonight, no news
      • TOPIC: MetaBrainz Community and Development channel | MusicBrainz non-development: #musicbrainz| Google Code-In https://goo.gl/8xSZem | Meeting agenda: reviews
      • TOPIC: MetaBrainz Community and Development channel | MusicBrainz non-development: #musicbrainz| Google Code-In https://goo.gl/8xSZem | Meeting agenda: reviews, Design Tickets(chhavi_)
      • Freso
        <BANG>
      • iliekcomputers
        Hi!
      • chhavi_
        hello
      • Freso
        Hello :D
      • Freso: go!
      • 🙋
      • I went back to Barcelona last week and had a meeting with Rob and then got my contract renewed, so you're stuck with me a while longer. :)
      • CatQuest
        hi!
      • iliekcomputers
        :D
      • CatQuest
        Freso: yay!
      • Freso
        I also helped review reosarevok's Mahler debriefing blog post, but I'm sure he'll say more about that.
      • Leo_Verto
        hey!
      • Freso
        I dealt a small bit with reported editors.
      • jwf is here
      • A large bit with Google Code-in.
      • And otherwise kept up with forums and IRC. (I think maybe there was also a support@ mail this week I answered.)
      • fin!
      • jwf: Go!
      • (Still up: reosarevok, bitmap, zas, ruaok, yvanzo, samj1912, iliekcomputers; maybe up (please confirm): LordSputnik, Leftmost, Leo_Verto, CatQuest, chhavi_, ListMyCDs, ferbncode; anyone else?)
      • jwf
        I was a little quiet, but getting back to speed. Working more with ListenBrainz statistics. I want to work with iliekcomputers and ruaok to add more example metrics.
      • ListMyCDs
        I'm following but have nothing to report this time. Just skip me.
      • CatQuest
        hey
      • jwf
        Once I get the hang of one, I want to write out a document to plan more so we can make sure it will scale.
      • samj1912
        chhavi_ you might wanna go early since you might fall asleep any moment :P