#metabrainz

/

      • pristine--
        Can send you the complete logs if you want
      • 2019-03-20 07956, 2019

      • iliekcomputers
        ooh
      • 2019-03-20 07907, 2019

      • iliekcomputers
        do you have an extracted dump in the playground folder
      • 2019-03-20 07928, 2019

      • iliekcomputers
        we need a dockerignore file
      • 2019-03-20 07958, 2019

      • iliekcomputers
        which branch are you working in?
      • 2019-03-20 07902, 2019

      • Mr_Monkey
        baniket: if you read the backlog, I've updated the User collections project description: https://wiki.musicbrainz.org/Development/Summer_o…
      • 2019-03-20 07928, 2019

      • pristine--
        iliekcomputers: yeah. There is. But I am not sure if it was there or was created by ./rec and stopped in middle.
      • 2019-03-20 07945, 2019

      • pristine--
        Batch-queries
      • 2019-03-20 07926, 2019

      • iliekcomputers
        one sec.
      • 2019-03-20 07951, 2019

      • iliekcomputers
        lemme try to push
      • 2019-03-20 07925, 2019

      • pristine--
        What are you pushing
      • 2019-03-20 07933, 2019

      • iliekcomputers
        master and worker
      • 2019-03-20 07936, 2019

      • iliekcomputers
        with numpy
      • 2019-03-20 07942, 2019

      • pristine--
        iliekcomputers: If you could do explain me why it didn't work on my side?
      • 2019-03-20 07950, 2019

      • iliekcomputers
        i am not sure.
      • 2019-03-20 07958, 2019

      • pristine--
        why a storage issue
      • 2019-03-20 07900, 2019

      • iliekcomputers
        which is why i'm trying to reproduce the error
      • 2019-03-20 07909, 2019

      • pristine--
        rest everything was working fine.
      • 2019-03-20 07911, 2019

      • pristine--
        okay :)
      • 2019-03-20 07943, 2019

      • akhilesh
        Mr_Monkey: what about the API project?
      • 2019-03-20 07956, 2019

      • iliekcomputers
        i wonder where the 160G is being used
      • 2019-03-20 07957, 2019

      • rvedotrc joined the channel
      • 2019-03-20 07909, 2019

      • Mr_Monkey
        reosarevok, yvanzo, bitmap: I'm clarifying the BB API project for GSOC and would love your input on this https://gist.github.com/MonkeyDo/2bb3add231979834…
      • 2019-03-20 07917, 2019

      • pristine--
        iliekcomputers: me too
      • 2019-03-20 07925, 2019

      • reosarevok
        culinko: I'm looking at this and I'm confused, so I'll have to get bitmap involved :)
      • 2019-03-20 07952, 2019

      • culinko
        alright. did you manage to repro the bug with my repro steps?
      • 2019-03-20 07915, 2019

      • Mr_Monkey
        akhilesh: Quite the timing. I'll need to wait for some input from the MB API pros, but there are some aspects of the gist above that you could look at already
      • 2019-03-20 07943, 2019

      • akhilesh
        Mr_Monkey: thanks :
      • 2019-03-20 07913, 2019

      • Mr_Monkey
        akhilesh: Note that the API structure related stuff (lines 23 onwards) might change significantly
      • 2019-03-20 07949, 2019

      • Mr_Monkey
        But I hope this allows you to move forward. I'm here if you have more questions as well
      • 2019-03-20 07923, 2019

      • iliekcomputers
        /var/lib/docker takes up 67G
      • 2019-03-20 07939, 2019

      • iliekcomputers
        pristine--: can you paste the fulllogs
      • 2019-03-20 07902, 2019

      • Mr_Monkey
        iliekcomputers: I seem to remember something about Docker not getting rid of old logs by default, so log files keep growing until you run out of disk space :/
      • 2019-03-20 07914, 2019

      • Mr_Monkey
        Not sure it's related, but there.
      • 2019-03-20 07930, 2019

      • alastairp
        not just log files, but intermediate images too
      • 2019-03-20 07921, 2019

      • alastairp
        but it's unclear from pristine--'s error message exactly what moving part is breaking
      • 2019-03-20 07940, 2019

      • reosarevok
        culinko: yeah
      • 2019-03-20 07910, 2019

      • alastairp
        iliekcomputers: `docker system prune -a` is your friend
      • 2019-03-20 07931, 2019

      • iliekcomputers
        i did a docker system prune and cleaned up 6 GB
      • 2019-03-20 07938, 2019

      • iliekcomputers
        would the -a help?
      • 2019-03-20 07940, 2019

      • alastairp
        -a should clean up more
      • 2019-03-20 07959, 2019

      • alastairp
        you may need to pull/build some images again after that
      • 2019-03-20 07929, 2019

      • alastairp
        are there any stopped but not removed containers hanging around? they may also keep some images from being deleted
      • 2019-03-20 07951, 2019

      • akhilesh
        Mr_Monkey: In line 19, Did you mean that , We will implement search end point on elasticsearch for now?
      • 2019-03-20 07919, 2019

      • culinko
        reosarevok: so far i have found 3 places where the artist name is used instead of AC (all 3 are in the recordings tab) but there might be more, idk whether the same code is reused or such
      • 2019-03-20 07933, 2019

      • Mr_Monkey
        akhilesh: Yes. One GSOC proposal should notrely on another proposal
      • 2019-03-20 07949, 2019

      • reosarevok
        culinko: Pretty sure it's just there, because that's taken from the ws recording search
      • 2019-03-20 07905, 2019

      • reosarevok
        It's just weird in how it only happens in some specific cases
      • 2019-03-20 07918, 2019

      • culinko
        what do you mean by specific cases? i have seen it happen every time the artist has been credited with an AC that is different from the artist name
      • 2019-03-20 07959, 2019

      • culinko
        but only with recording AC (so no RG, release AC, etc.)
      • 2019-03-20 07947, 2019

      • iliekcomputers
      • 2019-03-20 07946, 2019

      • akhilesh
        Mr_Monkey: Thanks for giving such clear view on the project, I will submit new proposal as soon as possible :)
      • 2019-03-20 07952, 2019

      • alastairp
        you could try and restart docker - it might still have an open file handle on some removed files (and so it won't free it up)
      • 2019-03-20 07904, 2019

      • alastairp
        I assume that `df` on that machine shows 0 bytes free?
      • 2019-03-20 07943, 2019

      • iliekcomputers
        yep
      • 2019-03-20 07954, 2019

      • iliekcomputers
        tbh, the images need to be fixed.
      • 2019-03-20 07954, 2019

      • alastairp
        and this is even after running system prune?
      • 2019-03-20 07911, 2019

      • alastairp
        restarting docker seems like a really good next step then
      • 2019-03-20 07918, 2019

      • iliekcomputers
        yes.
      • 2019-03-20 07934, 2019

      • alastairp
        going to lunch
      • 2019-03-20 07953, 2019

      • pristine--
      • 2019-03-20 07958, 2019

      • pristine--
        iliekcomputers: ^
      • 2019-03-20 07921, 2019

      • alastairp
        > Sending build context to Docker daemon 35.95GB
      • 2019-03-20 07925, 2019

      • alastairp
        this is way too huge
      • 2019-03-20 07948, 2019

      • pristine--
        iliekcomputers: earlier the error was about copying the file
      • 2019-03-20 07948, 2019

      • alastairp
        iliekcomputers knows about this, he said that he'd add a .dockerignore file - wait until he does this before trying to build again
      • 2019-03-20 07915, 2019

      • iliekcomputers
        pristine--: i removed the big dir
      • 2019-03-20 07917, 2019

      • alastairp
        what image is this building (what command are you running that causes this error?)
      • 2019-03-20 07923, 2019

      • iliekcomputers
        the image needs to be rethunk
      • 2019-03-20 07926, 2019

      • iliekcomputers
        rethought
      • 2019-03-20 07939, 2019

      • iliekcomputers
      • 2019-03-20 07954, 2019

      • pristine--
        when you deleted some containers or whatever, the tar file was copied and the now hits the error whilst processing
      • 2019-03-20 07954, 2019

      • iliekcomputers
        i can see the /apt/list removal etc.
      • 2019-03-20 07902, 2019

      • pristine--
        it must be space issue
      • 2019-03-20 07906, 2019

      • pristine--
        not sure
      • 2019-03-20 07921, 2019

      • pristine--
        alastairp: ./docker/push-master.sh
      • 2019-03-20 07924, 2019

      • alastairp
        iliekcomputers: btw, spark should be installed before the COPY, to stop it from invalidating the cache all the time
      • 2019-03-20 07956, 2019

      • iliekcomputers
        alastairp: yeah, those dockeriles were created in a two day hack and never fixed since
      • 2019-03-20 07900, 2019

      • alastairp
        right
      • 2019-03-20 07935, 2019

      • iliekcomputers
      • 2019-03-20 07959, 2019

      • iliekcomputers
        not sure why it was moved below the copy
      • 2019-03-20 07925, 2019

      • alastairp
        what's in the master image?? why is it so big, and why does it need to be continually rebuilt?
      • 2019-03-20 07934, 2019

      • alastairp
        (I don't know anything about this setup)
      • 2019-03-20 07947, 2019

      • alastairp
        is there a file in the master image that it copied to workers?
      • 2019-03-20 07955, 2019

      • iliekcomputers
        the master image runs the spark master node
      • 2019-03-20 07905, 2019

      • iliekcomputers
        we could probably do a base spark image
      • 2019-03-20 07913, 2019

      • iliekcomputers
        and then use just that.
      • 2019-03-20 07920, 2019

      • alastairp
        sure, but that's a next thing
      • 2019-03-20 07933, 2019

      • iliekcomputers
        i'm not sure why it is so big
      • 2019-03-20 07934, 2019

      • alastairp
        I still want to know why we're continually building and pushing this image
      • 2019-03-20 07945, 2019

      • alastairp
        well, it's so big because there's 30gb of files in that directory :)
      • 2019-03-20 07949, 2019

      • alastairp
        anyway, lunch for real now
      • 2019-03-20 07909, 2019

      • iliekcomputers
        it shouldn't be pushed frequently. we just came across a use case needing numpy
      • 2019-03-20 07915, 2019

      • iliekcomputers
        and numy wasn't installed.
      • 2019-03-20 07927, 2019

      • iliekcomputers
        in what directory?
      • 2019-03-20 07953, 2019

      • iliekcomputers
      • 2019-03-20 07918, 2019

      • alastairp
        1:20 PM <alastairp> > Sending build context to Docker daemon 35.95GB
      • 2019-03-20 07929, 2019

      • alastairp
        I guess you since deleted stuff?
      • 2019-03-20 07949, 2019

      • iliekcomputers
        not really. I think it was building from some incorrect context.
      • 2019-03-20 07956, 2019

      • iliekcomputers
        let me check the script.
      • 2019-03-20 07958, 2019

      • rvedotrc has quit
      • 2019-03-20 07909, 2019

      • iliekcomputers
        alastairp: it was a context problem somehow.
      • 2019-03-20 07927, 2019

      • iliekcomputers
        i don't know where it was building from
      • 2019-03-20 07959, 2019

      • iliekcomputers
      • 2019-03-20 07906, 2019

      • reosarevok
        culinko: testing locally it seems to only happen sometimes, but maybe I'm missing something. Does it also happen if the artist is not the release artist?
      • 2019-03-20 07924, 2019

      • iliekcomputers
        when i ran this from inside the repo root dir via `./docker/push-master.sh` it errored out.
      • 2019-03-20 07947, 2019

      • iliekcomputers
        but when I added a `cd ..` and ran it from the docker dir (`./push-master.sh`) it worked.
      • 2019-03-20 07948, 2019

      • culinko
        reosarevok: i think so, because first time it happened on a VA release, let me try to reproduce
      • 2019-03-20 07948, 2019

      • iliekcomputers
        weird.
      • 2019-03-20 07915, 2019

      • culinko
        reosarevok: yeah i reproduced it on a VA release just now, re-used the same recording as in my ticket
      • 2019-03-20 07934, 2019

      • reosarevok
        Ok. I'm not sure why it doesn't always work (or well, break) locally
      • 2019-03-20 07946, 2019

      • reosarevok
        I'll talk to bitmap :)
      • 2019-03-20 07958, 2019

      • pristine--
        iliekcomputers: what did that cd .. from inside docker so that it worked out
      • 2019-03-20 07907, 2019

      • culinko
        reosarevok: thanks a lot!
      • 2019-03-20 07922, 2019

      • iliekcomputers
        pristine--: in the push-master.sh file, I added a `cd ..`
      • 2019-03-20 07929, 2019

      • iliekcomputers
        then from inside the docker dir
      • 2019-03-20 07936, 2019

      • iliekcomputers
        `./push-master.sh`
      • 2019-03-20 07906, 2019

      • pristine--
        How did it change things?
      • 2019-03-20 07914, 2019

      • iliekcomputers
        i have no idea.
      • 2019-03-20 07916, 2019

      • pristine--
        Why wasn't there an error after this change
      • 2019-03-20 07919, 2019

      • pristine--
        Okay
      • 2019-03-20 07923, 2019

      • iliekcomputers
        it made the docker image smaller for some reason.
      • 2019-03-20 07947, 2019

      • pristine--
        Let me know whenever you know the reason
      • 2019-03-20 07949, 2019

      • iliekcomputers
        ruaok: when you're back, let's talk about these docker images. :P
      • 2019-03-20 07906, 2019

      • iliekcomputers
        pristine--: workers and masters both have numpy now
      • 2019-03-20 07911, 2019

      • pristine--
        How
      • 2019-03-20 07931, 2019

      • pristine--
        I would like to understand the prob and sol
      • 2019-03-20 07942, 2019

      • pristine--
        If you could explain :)
      • 2019-03-20 07955, 2019

      • pristine--
        iliekcomputers: ^
      • 2019-03-20 07901, 2019

      • iliekcomputers
        i don't particularly understand it myself yet.
      • 2019-03-20 07912, 2019

      • iliekcomputers
        the basic idea is that the images being build were too big.
      • 2019-03-20 07917, 2019

      • iliekcomputers
        i'm not sure yet why.
      • 2019-03-20 07946, 2019

      • pristine--
        So in push-master and worker you added cd ..
      • 2019-03-20 07948, 2019

      • pristine--
        ?
      • 2019-03-20 07925, 2019

      • KassOtsimine
        https://wordpress.com/me is where you can fill out some little bio or info about for the blog
      • 2019-03-20 07930, 2019

      • iliekcomputers
      • 2019-03-20 07932, 2019

      • iliekcomputers
        yes
      • 2019-03-20 07943, 2019

      • iliekcomputers
        it fixed the context somehow and the image got smaller.
      • 2019-03-20 07947, 2019

      • iliekcomputers
        i'll have to look more into it.
      • 2019-03-20 07917, 2019

      • pristine--
        But then you might have thought about something before adding cd..
      • 2019-03-20 07943, 2019

      • iliekcomputers
        i wanted to make sure that the script ran in the correct dir.
      • 2019-03-20 07946, 2019

      • pristine--
        A directory up from docker dir is?
      • 2019-03-20 07951, 2019

      • iliekcomputers
        yes
      • 2019-03-20 07918, 2019

      • rvedotrc joined the channel
      • 2019-03-20 07928, 2019

      • ruaok
        iliekcomputers: ok, sounds good. Tuesday sometime? :-/
      • 2019-03-20 07942, 2019

      • iliekcomputers
        cool.
      • 2019-03-20 07957, 2019

      • yvanzo
        reosarevok: do you know if we have minute notes for ws/3 from MeB summit?
      • 2019-03-20 07908, 2019

      • yvanzo
        because Mr_Monkey’s proposal “to batch multiple separate requests” is more or less what we discussed, right?
      • 2019-03-20 07917, 2019

      • Freso
        yvanzo: There's nothing in https://docs.google.com/document/d/1lhMtpPrayk3AP… at least…
      • 2019-03-20 07952, 2019

      • yvanzo
        Thanks, there are two lines about “Web Service URL Structure” that also relates to the topic, but nothing about splitting requests indeed.