#metabrainz

ferbncode: I can't get the tests you mentioned in your comment to run. It timeouts "waiting for dependencies to become available."

[listenbrainz-server] vansika opened pull request #578 (master…recommednation-table): table to store recommended playlist of every user https://github.com/metabrainz/listenbrainz-serv...

CatQuest: wut

spellew: the tests use a redis docker container and need to get the docker image. Tests should run properly once the image is downloaded and containers are launched. Could you paste the output the next time you try to run the tests?

Moin!

mo’’in’

Moin!

mo'

Project bookbrainz-site build #2225: failed in 4 min 26 sec: https://travis-ci.org/bookbrainz/bookbrainz-sit...

"01:35:08 <zarcade_droid> It's just a bit sad that that link isn't shown on the sidebar" - note that the relationship type and whether a link is shown in the sidebar are generally not related ("official homepage" excepted). Sub-Fandom pages could be added to the sidebar while still using "fan page" or "biography" links.

alastairp: hi, we on for the release today?

[musicbrainz-server] yvanzo opened pull request #1093 (master…mbs-10212): MBS-10212: Display artist name for SoundCloud URL with trailing slash https://github.com/metabrainz/musicbrainz-serve...

MBS-10212: SoundCloud URL with trailing slash is not displayed with user name in artist sidebar https://tickets.metabrainz.org/browse/MBS-10212

I'm still making changes, what does your availability look like today?

alastairp: around most of the day

ok, thanks. I'll ping you when I'm ready

Cooo

[musicbrainz-server] yvanzo merged pull request #566 (master…mbs-9512-verify-name): Reject edits with empty name to mitigate MBS-9512 https://github.com/metabrainz/musicbrainz-serve...

MBS-9512: Changing recording name to empty string should not be allowed https://tickets.metabrainz.org/browse/MBS-9512

[musicbrainz-server] yvanzo merged pull request #1093 (master…mbs-10212): MBS-10212: Display artist name for SoundCloud URL with trailing slash https://github.com/metabrainz/musicbrainz-serve...

MBS-10212: SoundCloud URL with trailing slash is not displayed with user name in artist sidebar https://tickets.metabrainz.org/browse/MBS-10212

yvanzo: is that enough to actually close https://tickets.metabrainz.org/browse/MBS-9512 ?

MBS-9512: Changing recording name to empty string should not be allowed

yes

Ok :)

(was wondering since it said "mitigate" at first)

ya but bitmap found a definitive fix after all :)

Mr_Monkey: did you tried endpoint implemented until. Only relationship of work lookup is left. I am reading about caching which is our next task. I understand the storing and getting data from cache but still confused with invalidation of cache? Have you any idea about cache invalidation for existing setup?

akhilesh: Haven't tried the endpoints since yesterday, I was waiting for work to be done to review more thoroughly.

As for cache, I'm not the best person to ask. My knowledge of it is limited. zas has recently been testing setting up cache at the gateway level, so I think it would be better to keep cache implementation for a later date and move forward on the endpoints.

Sounds good to me for now!

We might end up not needing to set up cache for any endpoint

You will need a cursory understanding of redis as the cache layer for credentials, but api/app.js is already set up for that as it is now.

Hi bukwurm ! You wanted to talk shop?

yvanzo: thanks for the hints re: the constants for select! :)

Should be good now

Mr_Monkey: Ok, but it is good to me if you will check implemented endpoint and see the output and give your valuable suggestions please. It will help me to speed up the project.

OK akhilesh. I'll do a full review tomorrow

Are you available right now?

Hi bukwurm

Yup

Great

So what can I help you with for the import project?

I was thinking of moving in the direction of final import process.

First of all, do you have anything specific direction for the project?

Not at this point, no

Alright

But the direction you were going previously seemed right to me

Ok

I think there are some questions regarding the UI I would propose to make sketches for

Ok

First, I'd like to talk about the importer

The changes since then that have been made are:

1. Moving the project under a new repository - bookbrainz-utils

2. Adding a `cleaner` sub-project (apparantly in typescript) which doesn't appear to have much functionality as of now

Yep, was just looking at it

Ok

So there are some fundamental changes I am thinking of making

Back then I had just started out with nodejs , so a number of issues had crept in

First change I'd like to have is a separate micro-service for consumer (one that pushes the data into db)

How is it currently run?

Currently, it's coupled with a lot of things

Presently it's just a script, which can fail silently

It can be very easily set up as a separate service/worker

Nothing much needs to be changed

It can be even set up as a cron

Yup

We can even run it at will though right now

But I imagine a situation where 3rd party individuals can use the API to import data from any place

The server pushes the request and data to queue, and the workers listening can feed that into the db

But it's not really really needed right now

It's a potential future goal, yes

The next thing is - we should ideally have an error queue which we can monitor

In case an import fails, it could be possibly due to some bug. Instead of logging it - we can ideally push it to the queue.

OK, sounds sane

Easy to monitor

If errors exceed while monitoring and in case we find there's an issue - we can just start some worker to refeed the message back to the original queue

after fixing it

Else we can simply flush the queue

Again, not a critical change but good to have

Next thing - and a major one

The other end of importer

Is the auto import we were running

By reading and parsing the dumps from OpenLibrary

Right

When I was writing the code I had little knowledge of event loop and all

bitmap: I still dont understand how do i get $form

So I wrote a fairly complicated code for multiprocessing and what now

I've put the statement you told me, in the required place.

I don't think it's effective in a resource constraint project such as bookbrainz

I was looking through it, yeah, it seemed pretty complex

Cyna: push the code again and I'll check :)

It's clumsy, and as there were no threads then in js

Sure

It's not async

I think we can get rid of the module

And replace it with a much cleaner worker threads now stable

OK, that sounds like an important job, and a good idea

Just pushed the code

If we at all want threading

Resource wise certainly

Mr_Monkey: Yeah, it was consuming very high amounts of memory and CPU as well

That issue was effectively left unaddressed then

The problem was reading file stream was faster than pushing into queue

As all I/O is non blocking

It was consuming huge amount of resources

Right

So it seems to me like this would be a good candidate for the first job

Yeah

Fixing producer is priority

Other changes I felt was

Surrounding how global connections are made

Say I create connection object for db or queue

In bookbrainz server, we simply attach that to all requests etc iirc

Cyna: you're still overriding all the props (including form) there https://github.com/anirudhjain75/musicbrainz-se...

But we don't have such thing in importer

as I said yesterday, edit_action doesn't return $form :)

So I was thinking of maintaining a global state for the application

It might be helpful for inversion of control tests as well

Cyna: I'd set component_props in the stash before it calls edit_action, too

Right, so let me give you an example

It still feels strange not having $form but passing it

In BB server we instantiate once at the beginning and pass that object along, yeah. What are you suggesting to do for the importer?