#metabrainz

/

      • alastairp
        what if, when we got a new model and needed to re-dump an old mbid, we just re-dump the entire file again
      • 2017-10-24 29739, 2017

      • alastairp
        that way, when we uncompress over the top of an existing dump, it'll just overwrite the old one
      • 2017-10-24 29707, 2017

      • alastairp
        actually, that's almost the same as saying "when we release a new model we make a new initial high-level dump"
      • 2017-10-24 29746, 2017

      • iliekcomputers
        Yes. But how often are models released?
      • 2017-10-24 29703, 2017

      • alastairp
        which I like - it removes a huge amount of complexity, and we don't plan to release new models very often
      • 2017-10-24 29729, 2017

      • alastairp
        I don't think we'd make new model releases more than 2 or 3 times per year
      • 2017-10-24 29710, 2017

      • alastairp
        it means we'd have to keep track of highlevel and lowlevel json dumps differently
      • 2017-10-24 29722, 2017

      • alastairp
        (currently we use just one row in `incremental_dumps` for everything)
      • 2017-10-24 29756, 2017

      • iliekcomputers
        That wouldn't be hard to do i guess.
      • 2017-10-24 29714, 2017

      • iliekcomputers
        This sounds like a good solution to me.
      • 2017-10-24 29754, 2017

      • alastairp
        me too - it sounds familiar
      • 2017-10-24 29702, 2017

      • alastairp
        I think we came up with the same solution
      • 2017-10-24 29709, 2017

      • iliekcomputers
        Lol :)
      • 2017-10-24 29734, 2017

      • iliekcomputers
        alastairp: I'll open a few tickets and start working on them, I guess. We can supersede AB-97 with a new one saying no dump file should have more than X recordings.
      • 2017-10-24 29734, 2017

      • BrainzBot
        AB-97: Provide small JSON exports, for testing https://tickets.metabrainz.org/browse/AB-97
      • 2017-10-24 29747, 2017

      • iliekcomputers
        And one for the hl Json stuff
      • 2017-10-24 29714, 2017

      • alastairp
        hold of on closing 97 for now
      • 2017-10-24 29730, 2017

      • iliekcomputers
        Right, not closing it. Will just open a new one.
      • 2017-10-24 29731, 2017

      • alastairp
        I'd like to see how big a 100k or 200k or 500k file is first
      • 2017-10-24 29701, 2017

      • alastairp
        because I don't think we should go less than 100k in a dump-part
      • 2017-10-24 29716, 2017

      • alastairp
        but if that's 5-10gb, then it's too big for a sample dump
      • 2017-10-24 29749, 2017

      • alastairp
        (having said that, we should be able to make a sample dump of x size or n number using the dump script)
      • 2017-10-24 29735, 2017

      • jsturgis joined the channel
      • 2017-10-24 29758, 2017

      • iliekcomputers
        Right.
      • 2017-10-24 29741, 2017

      • iliekcomputers
        A sample dump would be at most 1-2 GB, I guess.
      • 2017-10-24 29700, 2017

      • alastairp
        I think that the absolute maximum we should have is 1gb
      • 2017-10-24 29741, 2017

      • alastairp
        maybe even less compressed - 2GB uncompressed hl and ll combined
      • 2017-10-24 29742, 2017

      • alastairp
        or something
      • 2017-10-24 29722, 2017

      • alastairp
        this is for people who might want to set up a dev environment to look at the server; or who want to play with the data for running experiments
      • 2017-10-24 29743, 2017

      • alastairp
        they don't need lots, just enough to make a basic dataset, etc
      • 2017-10-24 29731, 2017

      • iliekcomputers
        I get the use case, just having set up AB too :P
      • 2017-10-24 29734, 2017

      • iliekcomputers
        AB-314, AB-315
      • 2017-10-24 29734, 2017

      • BrainzBot
        AB-314: Don't allow more than X recordings in a data dump file https://tickets.metabrainz.org/browse/AB-314
      • 2017-10-24 29734, 2017

      • BrainzBot
        AB-315: Fix high level json dumps https://tickets.metabrainz.org/browse/AB-315
      • 2017-10-24 29738, 2017

      • CatQuest
        "D/T" it is then I guess
      • 2017-10-24 29748, 2017

      • CatQuest
        ugh you know what is annoying?
      • 2017-10-24 29713, 2017

      • CatQuest
        the "spam" measurments we have on the wiki not cikcing in until I'm half way trought typing a sentence
      • 2017-10-24 29714, 2017

      • Leftmost
        If you say "Leftmost", that will be not-nice. :P
      • 2017-10-24 29734, 2017

      • iliekcom- has quit
      • 2017-10-24 29735, 2017

      • ferbncode has quit
      • 2017-10-24 29716, 2017

      • CatQuest
        i am sure I'm more annoying thna leftmost anyway
      • 2017-10-24 29730, 2017

      • CatQuest
        @I'm QuestMost
      • 2017-10-24 29732, 2017

      • alastairp
        great, thanks iliekcomputers
      • 2017-10-24 29718, 2017

      • CatQuest
        uhm, hey have you done anything to the stuff behind the tagging ability?
      • 2017-10-24 29734, 2017

      • CatQuest
        i cant tag fro mthe sidebar anymore (the box in the "tags" tab still works)
      • 2017-10-24 29722, 2017

      • CatQuest
        I could tag things just yesterday so..
      • 2017-10-24 29747, 2017

      • CatQuest
        ListMyCDs: thanks for already finding this and making a bug ticket <3
      • 2017-10-24 29724, 2017

      • samj1912
        bitmap: this is the current SOLR setup (https://docs.google.com/document/d/1ZZfpnXSUhMVWh…)
      • 2017-10-24 29735, 2017

      • samj1912
        me and zas are trying to dockerize this
      • 2017-10-24 29749, 2017

      • samj1912
        zas: said you'd know about how to handle the pgsql plugins part
      • 2017-10-24 29724, 2017

      • bitmap
        I thought it was already mostly dockerized
      • 2017-10-24 29746, 2017

      • samj1912
        nope
      • 2017-10-24 29755, 2017

      • samj1912
        Is it?
      • 2017-10-24 29708, 2017

      • samj1912
      • 2017-10-24 29729, 2017

      • bitmap
        yes
      • 2017-10-24 29744, 2017

      • samj1912
        but its missing components
      • 2017-10-24 29744, 2017

      • bitmap
        what's missing?
      • 2017-10-24 29747, 2017

      • samj1912
        mainly sir
      • 2017-10-24 29755, 2017

      • samj1912
        and the indexing components
      • 2017-10-24 29712, 2017

      • samj1912
        although there is a docker folder in sir https://github.com/metabrainz/sir
      • 2017-10-24 29722, 2017

      • samj1912
        hmm
      • 2017-10-24 29732, 2017

      • agentsim_ has quit
      • 2017-10-24 29748, 2017

      • bitmap
        right, Gentlecat had already deployed this stuff using docker, so I assume something is working
      • 2017-10-24 29759, 2017

      • agentsim joined the channel
      • 2017-10-24 29743, 2017

      • samj1912
        zas is not aware of the current config/parts and neither am I (docker noob here)
      • 2017-10-24 29759, 2017

      • bitmap
        that shouldn't be hard to figure out
      • 2017-10-24 29741, 2017

      • samj1912
        I have listed out all the parts in the manual setup, can you help us figure out if they are already dockerized
      • 2017-10-24 29707, 2017

      • samj1912
        zas: also pointed out a couple of mistakes with the existing config (data files in the same directory as config files)
      • 2017-10-24 29725, 2017

      • samj1912
        and also we might want to run a solr cloud setup rather than the normal solr
      • 2017-10-24 29752, 2017

      • bitmap
        solr, mbsssss, mmd-schema, and mb-solrquerywriter are all in https://github.com/metabrainz/docker-solr/blob/ma…
      • 2017-10-24 29701, 2017

      • samj1912
        okay
      • 2017-10-24 29708, 2017

      • bitmap
        those are meant to be in the same container
      • 2017-10-24 29734, 2017

      • samj1912
        okay
      • 2017-10-24 29724, 2017

      • samj1912
        what about rabbitmq and connection to mbdb?
      • 2017-10-24 29733, 2017

      • bitmap
        and afaiu mbdata and rngpy are just libraries used in the sir container
      • 2017-10-24 29747, 2017

      • samj1912
        yes
      • 2017-10-24 29751, 2017

      • samj1912
        they are installed with sir
      • 2017-10-24 29709, 2017

      • bitmap
      • 2017-10-24 29709, 2017

      • bitmap
        that config gets used by https://github.com/metabrainz/sir/blob/master/con… in the container
      • 2017-10-24 29743, 2017

      • bitmap
        it's a config template, rendererd by consul-template (which pulls that above data from consul)
      • 2017-10-24 29717, 2017

      • samj1912
        cool and where do we install the pg_amqp plugin in the main db?
      • 2017-10-24 29732, 2017

      • samj1912
        pointing to the above rabbitmq instance
      • 2017-10-24 29736, 2017

      • bitmap
        that's already installed
      • 2017-10-24 29706, 2017

      • bitmap
      • 2017-10-24 29730, 2017

      • bitmap
        but you'll need to insert a row into the amqp.broker table
      • 2017-10-24 29746, 2017

      • samj1912
        okay, where will I have to do that?
      • 2017-10-24 29759, 2017

      • samj1912
      • 2017-10-24 29713, 2017

      • bitmap
      • 2017-10-24 29740, 2017

      • bitmap
        has to be done manually currently (but it's just a one-time thing)
      • 2017-10-24 29751, 2017

      • samj1912
        okay
      • 2017-10-24 29710, 2017

      • samj1912
        the host remains the same always?
      • 2017-10-24 29747, 2017

      • bitmap
        yeah, it's where rabbitmq is running, which is always on serge right now
      • 2017-10-24 29710, 2017

      • bitmap
        there should already be a row in the broker table for the setup Gentlecat had, you can just use/modify that
      • 2017-10-24 29723, 2017

      • samj1912
        okay
      • 2017-10-24 29752, 2017

      • samj1912
        I think that takes care of everything
      • 2017-10-24 29709, 2017

      • samj1912
        I will co-ordinate with zas to get a test setup via docker up and running
      • 2017-10-24 29703, 2017

      • samj1912
        thanks for the help :D
      • 2017-10-24 29728, 2017

      • bitmap
        sure, let me know if you need help with the amqp setup
      • 2017-10-24 29748, 2017

      • samj1912
        yup
      • 2017-10-24 29709, 2017

      • samj1912
        I haven't even looked into that part of SIR yet
      • 2017-10-24 29725, 2017

      • samj1912
        will have to understand how it works first
      • 2017-10-24 29717, 2017

      • bitmap
        if you see queries like 'PERFORM amqp.publish(2, ...)' in sir, the '2' is actually the row id in the broker table
      • 2017-10-24 29734, 2017

      • bitmap
        that tells it where/how to connect to rabbitmq and where to publish things
      • 2017-10-24 29747, 2017

      • samj1912
        okay
      • 2017-10-24 29737, 2017

      • samj1912
        bitmap: one last thing, how do I add the triggers generated by SIR to MB DB?
      • 2017-10-24 29743, 2017

      • bitmap
        cat sql/CreateFunctions.sql | psql -U musicbrainz db_name
      • 2017-10-24 29758, 2017

      • bitmap
        cat sql/CreateTriggers.sql | psql -U musicbrainz db_name
      • 2017-10-24 29702, 2017

      • bitmap
        something like that :)
      • 2017-10-24 29714, 2017

      • samj1912
        okay, so I just run in manually once?
      • 2017-10-24 29719, 2017

      • bitmap
        (those files are in the sir repo)
      • 2017-10-24 29732, 2017

      • bitmap
        yes, unless the functions/triggers change and you need to update them
      • 2017-10-24 29743, 2017

      • samj1912
        I haven't looked at the code yet, but I am not sure if the current triggers handle all the relationships and attributes I added over the last 2 weeks
      • 2017-10-24 29704, 2017

      • bitmap
        you can regenerate them with the Makefile and see what changes
      • 2017-10-24 29718, 2017

      • bitmap
        `make createsql` I guess
      • 2017-10-24 29736, 2017

      • github joined the channel
      • 2017-10-24 29736, 2017

      • github
        [acousticbrainz-server] alastair closed pull request #235: Upgrade javascript dependencies (master...js-upgrade) https://git.io/v5ht6
      • 2017-10-24 29736, 2017

      • github has left the channel
      • 2017-10-24 29717, 2017

      • bitmap
        samj1912: are you setting this up on the production database (bowie)? or somewhere else
      • 2017-10-24 29751, 2017

      • bitmap
        I mean if you're doing a test setup somewhere else first or something
      • 2017-10-24 29701, 2017

      • samj1912
        I will first do it on a mbdb replicae
      • 2017-10-24 29729, 2017

      • samj1912
        solr-test.mb.org is where I will replicate the solr/sir setup
      • 2017-10-24 29736, 2017

      • samj1912
        its a google cloud vm
      • 2017-10-24 29702, 2017

      • samj1912
        and there is mbs-repl-1.metabrainz.org for the db
      • 2017-10-24 29718, 2017

      • bitmap
        okay, not sure how postgres is setup there, you may need to install pg_amqp by hand
      • 2017-10-24 29727, 2017

      • samj1912
        okay
      • 2017-10-24 29738, 2017

      • bitmap
        which is easy, just clone https://github.com/omniti-labs/pg_amqp && make && make install
      • 2017-10-24 29754, 2017

      • samj1912
        yup, installed it on my local setup
      • 2017-10-24 29709, 2017

      • samj1912
        I have the entire thing running locally on solr-test
      • 2017-10-24 29717, 2017

      • samj1912
        I just need to shift it to docker
      • 2017-10-24 29723, 2017

      • samj1912
        bitmap: how do I call the GenerateSQLScript with correct arguments?
      • 2017-10-24 29745, 2017

      • samj1912
        I dont think I did it correctly in https://github.com/metabrainz/sir/pull/50
      • 2017-10-24 29745, 2017

      • github joined the channel
      • 2017-10-24 29745, 2017

      • github
        [sir] samj1912 opened pull request #50: Update Triggers (master...updatesql) https://git.io/vdjr1
      • 2017-10-24 29745, 2017

      • github has left the channel
      • 2017-10-24 29702, 2017

      • samj1912
        because I can see a lot of missing statements in the dropsql part
      • 2017-10-24 29727, 2017

      • samj1912
        I called it by `./GenerateSQLScripts.pl ../../sir/sql/ `
      • 2017-10-24 29744, 2017

      • agentsim_ joined the channel
      • 2017-10-24 29742, 2017

      • agentsim has quit
      • 2017-10-24 29711, 2017

      • samj1912
      • 2017-10-24 29715, 2017

      • samj1912
        good night people o/
      • 2017-10-24 29730, 2017

      • aaa_ joined the channel
      • 2017-10-24 29732, 2017

      • aaa_ has quit
      • 2017-10-24 29705, 2017

      • CatQuest
        natta samj1912 !
      • 2017-10-24 29714, 2017

      • Somasis is now known as squawk
      • 2017-10-24 29721, 2017

      • squawk is now known as Somasis
      • 2017-10-24 29733, 2017

      • bitmap
        samj1912: it looks like it removed some stuff from the Drop* files that no longer exists, but nothing current is missing I don't think?
      • 2017-10-24 29719, 2017

      • kyan has quit
      • 2017-10-24 29710, 2017

      • bitmap
        upgrading jenkins to fix some security vulnerabilities
      • 2017-10-24 29737, 2017

      • MajorLurker joined the channel
      • 2017-10-24 29706, 2017

      • ferbncode_ has quit