#metabrainz

/

      • ruaok
        now I have to upgrade my instance, lol
      • 2020-11-20 32525, 2020

      • outsidecontext
        Ha
      • 2020-11-20 32547, 2020

      • Lotheric
        Is there a MB funkwhale yet ?
      • 2020-11-20 32554, 2020

      • Lotheric wants to join :)
      • 2020-11-20 32556, 2020

      • supersandro2000 has quit
      • 2020-11-20 32517, 2020

      • supersandro2000 joined the channel
      • 2020-11-20 32553, 2020

      • ruaok
        no. but people have created their own instances.
      • 2020-11-20 32526, 2020

      • ruaok
        zas: dumps finally finished. nearly 24 hours. Thats quite a lot longer than last time. Not sure what changed.
      • 2020-11-20 32553, 2020

      • zas
        did it work properly?
      • 2020-11-20 32504, 2020

      • ruaok
        flawlessless from what I can tell.
      • 2020-11-20 32505, 2020

      • ruaok
      • 2020-11-20 32507, 2020

      • zas
        nothing changed performance-wise
      • 2020-11-20 32527, 2020

      • ruaok
        not, not for your part. the actual dumping/mogrification was a lot slower than last time.
      • 2020-11-20 32558, 2020

      • ruaok
        https://github.com/metabrainz/listenbrainz-server… -- have another look so we can merge, zas?
      • 2020-11-20 32553, 2020

      • zas
        reviewed
      • 2020-11-20 32525, 2020

      • ruaok
        thx
      • 2020-11-20 32510, 2020

      • ruaok
        I guess that wasn't ready to merge, heh.
      • 2020-11-20 32514, 2020

      • ruaok just wants to be done with it
      • 2020-11-20 32545, 2020

      • ruaok pushes the fixes.
      • 2020-11-20 32544, 2020

      • v6lur joined the channel
      • 2020-11-20 32552, 2020

      • Mr_Monkey
        "Pushafix" sounds like an Astérix et Obélix character.
      • 2020-11-20 32513, 2020

      • ruaok
        srsly
      • 2020-11-20 32539, 2020

      • ruaok
        zas: tested the fixes with an incremental dump. all is well. should be good to go now.
      • 2020-11-20 32507, 2020

      • ruaok
        pristine___: iliekcomputers : We've fixed the dump issues and generated a full new dump. I've triggered an import into spark just to make sure we have fresh data.
      • 2020-11-20 32544, 2020

      • ruaok
        meh. no more interest to fuck with dump code now. I'll just forget it about it for the time being.
      • 2020-11-20 32510, 2020

      • iliekcomputers
        Isn't that what we all do, forget about dump code? 🙈
      • 2020-11-20 32531, 2020

      • zas
        ruaok: I approved the PR, I don't want to be too picky ;) but you have at least to fix PEP8 before merging
      • 2020-11-20 32544, 2020

      • ruaok
        pretty much. I'm just miffed that it stole a day from me this week.
      • 2020-11-20 32504, 2020

      • ruaok
        merge what?
      • 2020-11-20 32508, 2020

      • ruaok has forgotten
      • 2020-11-20 32525, 2020

      • zas
      • 2020-11-20 32533, 2020

      • ruaok was joking
      • 2020-11-20 32549, 2020

      • zas
        damn, missed this one... ;)
      • 2020-11-20 32558, 2020

      • ruaok
        I'll get back to it late next week when I think I might have time next
      • 2020-11-20 32513, 2020

      • michaelqm joined the channel
      • 2020-11-20 32541, 2020

      • michaelqm
        I'm having an issue setting up a mirror server on digitalocean. I'm using the docker image, and the import fails when postgres dies at this query:
      • 2020-11-20 32507, 2020

      • michaelqm
        SELECT musicbrainz.recording_alias.recording AS musicbrainz_recording_alias_recording, musicbrainz.recording_alias.type AS musicbrainz_recording_alias_type, musicbrainz.recording_alias.id AS musicbrainz_recording_alias_id, musicbrainz.recording_alias.name AS musicbrainz_recording_alias_name, anon_1.musicbrainz_recording_id
      • 2020-11-20 32508, 2020

      • michaelqm
        AS anon_1_musicbrainz_recording_id FROM (SELECT musicbrainz.recording.id AS musicbrainz_recording_id FROM musicbrainz.recording WHERE musicbrainz.recording.id >= 233449 AND musicbrainz.recording.id < 249336) AS anon_1 JOIN musicbrainz.recording_alias ON anon_1.musicbrainz_recording_id = musicbrainz.recording_alias.recording ORDER BY
      • 2020-11-20 32508, 2020

      • michaelqm
        anon_1.musicbrainz_recording_id
      • 2020-11-20 32527, 2020

      • michaelqm
        Which results in the error:
      • 2020-11-20 32534, 2020

      • michaelqm
        Failed to import recording with id in bounds (149699, 159990)2020-11-20 21:13:32,859: (psycopg2.OperationalError) server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request.
      • 2020-11-20 32537, 2020

      • ruaok
        how much ram does your VM have?
      • 2020-11-20 32540, 2020

      • michaelqm
        16GB
      • 2020-11-20 32559, 2020

      • michaelqm
        I've increased postgres and SOLR to use 4GB each (over the 2GB initial value)
      • 2020-11-20 32522, 2020

      • michaelqm
        That should be enough, no?
      • 2020-11-20 32538, 2020

      • ruaok
        yes. did you modify the postgres config and restart postgres?
      • 2020-11-20 32513, 2020

      • michaelqm
        Yeah, I created local/compose/memory-settings.yml and then added it to the .env
      • 2020-11-20 32522, 2020

      • michaelqm
        Here's what's in that file:
      • 2020-11-20 32534, 2020

      • michaelqm
        services:
      • 2020-11-20 32535, 2020

      • michaelqm
        db:
      • 2020-11-20 32537, 2020

      • michaelqm
        command: postgres -c "shared_buffers=4GB" -c "shared_preload_libraries=pg_amqp.so"
      • 2020-11-20 32541, 2020

      • michaelqm
        search:
      • 2020-11-20 32544, 2020

      • michaelqm
        environment:
      • 2020-11-20 32546, 2020

      • michaelqm
        - SOLR_HEAP=4g
      • 2020-11-20 32517, 2020

      • ruaok
        in theory that should be good.
      • 2020-11-20 32530, 2020

      • ruaok
        have you looked at the logs of the postgres container?
      • 2020-11-20 32522, 2020

      • michaelqm
        Yeah, the logs show that query as being the one that kills the server.
      • 2020-11-20 32511, 2020

      • ruaok
        I have no idea what that might be. Seen this before yvanzo ?
      • 2020-11-20 32533, 2020

      • michaelqm
        I found chatlogs from here from a user 'nelgin' who had the same issue, but other than that I can't find much
      • 2020-11-20 32525, 2020

      • michaelqm
        If i do a `docker stats` while running the recording indexing, I'm seeing system_indexer_1 and system_search_1 both grow to about 3GB of memory each.
      • 2020-11-20 32545, 2020

      • michaelqm
        `system_db_1` stays at a relatively low memory usage.
      • 2020-11-20 32530, 2020

      • michaelqm
        Actually, I'm watching `system_indexer_1` grow quite a bit. It's up to 9GB now.
      • 2020-11-20 32527, 2020

      • michaelqm
        system_search_1 is sticking around 3GB
      • 2020-11-20 32512, 2020

      • michaelqm
        And then it crashes.
      • 2020-11-20 32542, 2020

      • michaelqm
        So it never looks like I max out on memory, ~ 12GB for all docker images.
      • 2020-11-20 32546, 2020

      • michaelqm
        Out of 16GB
      • 2020-11-20 32538, 2020

      • michaelqm
        If it is a memory issue, is there anything I can do short of increasing my RAM?
      • 2020-11-20 32523, 2020

      • ruaok has quit
      • 2020-11-20 32533, 2020

      • ruaok joined the channel
      • 2020-11-20 32557, 2020

      • prabal has quit
      • 2020-11-20 32505, 2020

      • prabal joined the channel
      • 2020-11-20 32532, 2020

      • iliekcomputers has quit
      • 2020-11-20 32543, 2020

      • iliekcomputers joined the channel
      • 2020-11-20 32521, 2020

      • michaelqm
        I think I might have gotten it to work, by doing something counter-intuitive.
      • 2020-11-20 32549, 2020

      • michaelqm
        The 4GB for PG and SOLR that they recommend in the readme, I put that back to the 2GB default.
      • 2020-11-20 32504, 2020

      • michaelqm
        It's still running, but it's about 10x farther than it's ever gotten.
      • 2020-11-20 32505, 2020

      • v6lur has quit
      • 2020-11-20 32535, 2020

      • BrainzGit
        [listenbrainz-server] mayhem merged pull request #1178 (master…critical-dump-fixes): Critical LB dump fixes https://github.com/metabrainz/listenbrainz-server…