#metabrainz

/

      • D4RK-PH0ENiX joined the channel
      • Slurpee joined the channel
      • Slurpee has quit
      • Slurpee joined the channel
      • Slurpee has quit
      • Nyanko-sensei joined the channel
      • D4RK-PH0ENiX has quit
      • Nyanko-sensei has quit
      • D4RK-PH0ENiX joined the channel
      • spellew
        ferbncode: I can't get the tests you mentioned in your comment to run. It timeouts "waiting for dependencies to become available."
      • BrainzGit
        [listenbrainz-server] vansika opened pull request #578 (master…recommednation-table): table to store recommended playlist of every user https://github.com/metabrainz/listenbrainz-serv...
      • Darkloke joined the channel
      • Rotab
        CatQuest: wut
      • Darkloke has quit
      • Darkloke joined the channel
      • ferbncode
        spellew: the tests use a redis docker container and need to get the docker image. Tests should run properly once the image is downloaded and containers are launched. Could you paste the output the next time you try to run the tests?
      • Mr_Monkey
        Moin!
      • yvanzo
        mo’’in’
      • akhilesh
        Moin!
      • ruaok
        mo'
      • ruaok is all he can manage with the jetlag
      • travis-ci joined the channel
      • travis-ci
        Project bookbrainz-site build #2225: failed in 4 min 26 sec: https://travis-ci.org/bookbrainz/bookbrainz-sit...
      • travis-ci has left the channel
      • Freso
        "01:35:08 <zarcade_droid> It's just a bit sad that that link isn't shown on the sidebar" - note that the relationship type and whether a link is shown in the sidebar are generally not related ("official homepage" excepted). Sub-Fandom pages could be added to the sidebar while still using "fan page" or "biography" links.
      • D4RK-PH0ENiX has quit
      • D4RK-PH0ENiX joined the channel
      • D4RK-PH0ENiX has quit
      • D4RK-PH0ENiX joined the channel
      • iliekcomputers
        alastairp: hi, we on for the release today?
      • BrainzGit
        [musicbrainz-server] yvanzo opened pull request #1093 (master…mbs-10212): MBS-10212: Display artist name for SoundCloud URL with trailing slash https://github.com/metabrainz/musicbrainz-serve...
      • BrainzBot
        MBS-10212: SoundCloud URL with trailing slash is not displayed with user name in artist sidebar https://tickets.metabrainz.org/browse/MBS-10212
      • alastairp
        I'm still making changes, what does your availability look like today?
      • iliekcomputers
        alastairp: around most of the day
      • alastairp
        ok, thanks. I'll ping you when I'm ready
      • iliekcomputers
        Cooo
      • sampsyo has quit
      • sampsyo joined the channel
      • bukwurm joined the channel
      • BrainzGit
        [musicbrainz-server] yvanzo merged pull request #566 (master…mbs-9512-verify-name): Reject edits with empty name to mitigate MBS-9512 https://github.com/metabrainz/musicbrainz-serve...
      • BrainzBot
        MBS-9512: Changing recording name to empty string should not be allowed https://tickets.metabrainz.org/browse/MBS-9512
      • BrainzGit
        [musicbrainz-server] yvanzo merged pull request #1093 (master…mbs-10212): MBS-10212: Display artist name for SoundCloud URL with trailing slash https://github.com/metabrainz/musicbrainz-serve...
      • BrainzBot
        MBS-10212: SoundCloud URL with trailing slash is not displayed with user name in artist sidebar https://tickets.metabrainz.org/browse/MBS-10212
      • reosarevok
        yvanzo: is that enough to actually close https://tickets.metabrainz.org/browse/MBS-9512 ?
      • BrainzBot
        MBS-9512: Changing recording name to empty string should not be allowed
      • yvanzo
        yes
      • reosarevok
        Ok :)
      • (was wondering since it said "mitigate" at first)
      • yvanzo
        ya but bitmap found a definitive fix after all :)
      • Darkloke has quit
      • akhilesh
        Mr_Monkey: did you tried endpoint implemented until. Only relationship of work lookup is left. I am reading about caching which is our next task. I understand the storing and getting data from cache but still confused with invalidation of cache? Have you any idea about cache invalidation for existing setup?
      • Mr_Monkey
        akhilesh: Haven't tried the endpoints since yesterday, I was waiting for work to be done to review more thoroughly.
      • As for cache, I'm not the best person to ask. My knowledge of it is limited. zas has recently been testing setting up cache at the gateway level, so I think it would be better to keep cache implementation for a later date and move forward on the endpoints.
      • akhilesh
        Sounds good to me for now!
      • Mr_Monkey
        We might end up not needing to set up cache for any endpoint
      • You will need a cursory understanding of redis as the cache layer for credentials, but api/app.js is already set up for that as it is now.
      • Hi bukwurm ! You wanted to talk shop?
      • reosarevok
        yvanzo: thanks for the hints re: the constants for select! :)
      • Should be good now
      • akhilesh
        Mr_Monkey: Ok, but it is good to me if you will check implemented endpoint and see the output and give your valuable suggestions please. It will help me to speed up the project.
      • bukwurm
        Mr_Monkey: Hey
      • Mr_Monkey
        OK akhilesh. I'll do a full review tomorrow
      • bukwurm
        Are you available right now?
      • Mr_Monkey
        Hi bukwurm
      • Yup
      • bukwurm
        Great
      • Mr_Monkey
        So what can I help you with for the import project?
      • bukwurm
        I was thinking of moving in the direction of final import process.
      • First of all, do you have anything specific direction for the project?
      • Mr_Monkey
        Not at this point, no
      • bukwurm
        Alright
      • Mr_Monkey
        But the direction you were going previously seemed right to me
      • bukwurm
        Ok
      • Mr_Monkey
        I think there are some questions regarding the UI I would propose to make sketches for
      • bukwurm
        Ok
      • First, I'd like to talk about the importer
      • The changes since then that have been made are:
      • 1. Moving the project under a new repository - bookbrainz-utils
      • 2. Adding a `cleaner` sub-project (apparantly in typescript) which doesn't appear to have much functionality as of now
      • Mr_Monkey
        Yep, was just looking at it
      • bukwurm
        Ok
      • So there are some fundamental changes I am thinking of making
      • Back then I had just started out with nodejs , so a number of issues had crept in
      • First change I'd like to have is a separate micro-service for consumer (one that pushes the data into db)
      • Mr_Monkey
        How is it currently run?
      • bukwurm
        Currently, it's coupled with a lot of things
      • Presently it's just a script, which can fail silently
      • It can be very easily set up as a separate service/worker
      • Nothing much needs to be changed
      • It can be even set up as a cron
      • Mr_Monkey
        Yup
      • bukwurm
        We can even run it at will though right now
      • But I imagine a situation where 3rd party individuals can use the API to import data from any place
      • The server pushes the request and data to queue, and the workers listening can feed that into the db
      • But it's not really really needed right now
      • Mr_Monkey
        It's a potential future goal, yes
      • bukwurm
        The next thing is - we should ideally have an error queue which we can monitor
      • In case an import fails, it could be possibly due to some bug. Instead of logging it - we can ideally push it to the queue.
      • Mr_Monkey
        OK, sounds sane
      • Easy to monitor
      • bukwurm
        If errors exceed while monitoring and in case we find there's an issue - we can just start some worker to refeed the message back to the original queue
      • after fixing it
      • Else we can simply flush the queue
      • Again, not a critical change but good to have
      • Next thing - and a major one
      • The other end of importer
      • Is the auto import we were running
      • By reading and parsing the dumps from OpenLibrary
      • Mr_Monkey
        Right
      • bukwurm
        When I was writing the code I had little knowledge of event loop and all
      • Cyna
        bitmap: I still dont understand how do i get $form
      • bukwurm
        So I wrote a fairly complicated code for multiprocessing and what now
      • Cyna
        I've put the statement you told me, in the required place.
      • bukwurm
        I don't think it's effective in a resource constraint project such as bookbrainz
      • Mr_Monkey
        I was looking through it, yeah, it seemed pretty complex
      • bitmap
        Cyna: push the code again and I'll check :)
      • bukwurm
        It's clumsy, and as there were no threads then in js
      • Cyna
        Sure
      • bukwurm
        It's not async
      • I think we can get rid of the module
      • And replace it with a much cleaner worker threads now stable
      • Mr_Monkey
        OK, that sounds like an important job, and a good idea
      • Cyna
        Just pushed the code
      • bukwurm
        If we at all want threading
      • Mr_Monkey
        Resource wise certainly
      • Cyna
      • bukwurm
        Mr_Monkey: Yeah, it was consuming very high amounts of memory and CPU as well
      • That issue was effectively left unaddressed then
      • The problem was reading file stream was faster than pushing into queue
      • As all I/O is non blocking
      • It was consuming huge amount of resources
      • Mr_Monkey
        Right
      • So it seems to me like this would be a good candidate for the first job
      • bukwurm
        Yeah
      • Fixing producer is priority
      • Other changes I felt was
      • Surrounding how global connections are made
      • Say I create connection object for db or queue
      • In bookbrainz server, we simply attach that to all requests etc iirc
      • bitmap
        Cyna: you're still overriding all the props (including form) there https://github.com/anirudhjain75/musicbrainz-se...
      • bukwurm
        But we don't have such thing in importer
      • bitmap
        as I said yesterday, edit_action doesn't return $form :)
      • bukwurm
        So I was thinking of maintaining a global state for the application
      • Mr_Monkey rereads a second time
      • It might be helpful for inversion of control tests as well
      • bitmap
        Cyna: I'd set component_props in the stash before it calls edit_action, too
      • bukwurm
        Right, so let me give you an example
      • Cyna
        It still feels strange not having $form but passing it
      • Mr_Monkey
        In BB server we instantiate once at the beginning and pass that object along, yeah. What are you suggesting to do for the importer?