#metabrainz

[critiquebrainz] release v-2021-02-24.0 has been published by github-actions[bot]: https://github.com/metabrainz/critiquebrainz/re...

[listenbrainz-server] mayhem opened pull request #1292 (master…add-mbid-mapping-to-labs-api): Add mbid mapping to labs api https://github.com/metabrainz/listenbrainz-serv...

!m _lucifer

You're doing good work, _lucifer!

new version of cb has been released :D

*sigh* execute failed: ERROR: relation "cover_art_archive.index_listing" does not exist

alastairp: should i update beta as well?

yeah, go for it

done :D

yay, for releasing stuff on your first full day.

!m _lucifer

You're doing good work, _lucifer!

alastairp: Sorry, it seems it cannot be implemented with consul-template 0.16 being used by LB containers, it does not support keyOrDefault either.

yvanzo: ok, no problem. I'll finish the consul-template 0.18 PR today and we'll release that too

Only solution that worked for listenbrainz-websockets-test is to hardcode "rabbitmq-prince" in template for now.

thanks for folowing up the cause

Hi roger_that, I can try to help you with if you provide more info please: raw log excerpt (please use pastebin or gist) and the output of "admin/configure show".

alastairp: quick question: where it the config.py file that is used by jenkins for testing?

ruaok: in LB? I think it's just copied from the example file

ah, ok.

just looking

thx

I see now -- one test fails because typescript expects an API key -- which key doesn't matter since the actual search call is mocked.

putting a garbage key in config.py.sample seems bad.

mocking the typesense setup call is doable, but also less good.

how would you solve this, alastairp ?

actually other config items set dummy keys too. I'll just do that for now.

right, so it's looking for config.TYPESENSE_KEY ?

yeah, I don't think there's a problem setting a dummy (or empty) value in config.py.sample

We may fill it like we do for spotify token `typesense needs a non-empty key to run tests`

oh yes, I keep forgetting. MetaBrainz has dropbox for business for free.

if anyone would like to have a MeB dropbox account for free, its better equipped than a free account.

hit me up!

🤣

ruaok, i would like one :)

ruaok: Will it be available to non team members 😅

sorry, no.

np

does Voluntary Project Leaders count? :D

I've just made it so that anyone with a @metabrainz.org address can ask to join the team automatically.

CatQuest: do you have a @meb addy?

right on.

no ruaok :)

would you like one?

uh. wow sure?

catcat@ ?

(I think I can hear the squee all the way over here in Spain)

CatQuest pls

alastairp: :P

pm me the email address where I should send the invite.

atj: whats your LB username?

_lucifer: congrats on your new job :D

ok https://github.com/metabrainz/listenbrainz-serv... is ready for review for anyone who has time.

I have one more endpoint to add, but I will do so in another PR, this one is large enough already.

thanks outsidecontext!

shivam-kapila: atj

gotcha thanks

Anyone seeing "Failed process was running: CREATE INDEX recording_idx_name ON recording (name);" when attempting to import the sample data into the docker db instance?

check diskspace?

unfortunately not - 932GiB free

are you trying for a clean DB import?

if you really want to full start over, consider removing the postgres volume.

yep, I've tried that and it happens again

exact same message

I've tried about 4 times now

yvanzo: ^^ any idea?

Docker on MacOS

using master on MB server and MB docker

atj: I was able to reproduce your issue.

shivam-kapila: that's good :)

So the WS listen has quite much of redundant info like PRIVATE_KEYS SUPPORTED KEYS etc. Shall I nuke it too?

shivam-kapila: fixing on the server-side?

cleint side

client*

👍

atj: Never since this before, I ran it successfully again just on Monday.

[musicbrainz-server] atj opened pull request #1932 (master…MBS-10312-fix): MBS-10312: Fix tracking parameter removal in Firefox 44 https://github.com/metabrainz/musicbrainz-serve...

yvanzo: how often are the dumps generated?

there was a new one this morning, I’m testing it

yvanzo: I'm using a sample by the way, not a full dump

it's generated on every Wednesday and Saturday

ho, ok, sample dump is monthly generated, so it cannot the problem here.

+be

The index it fails to create has been added in 2013, it's not the problem either.

Mr_Monkey: ping

pong

Can you take a look at the gist I posted above

atj: Is 932GiB free for your disk or the partition being used by Docker?

yvanzo: it's MacOS, so it's just one big FS

Why do the unnecessary fields persist even after serialising it as a Listen in frontenf

df -h $( sudo docker system info --format '{{.DockerRootDir}}' )

These fields are not declared in the Listen type

yvanzo: I'm using docker desktop so that doesn't work

I'll try on my Linux box and see if it happens there

Ok, it could be helpful to watch your system resources when trying to run it on macOS.

The Docker prefs shows "Disk image size: 59.6 GB (16.2 GB used)"

shivam-kapila: TBH I think the cleaning up should be done server-side befoere sending through websockets. Where can I see the code you talk about in "even after serialising it as a Listen in frontend"?

I can see that track_metadata doesn't have recording_msid in the WS listen (it's instead one level up)

its top level

and there are a lot of things like SUPPORTED_KEYS, date, dedup_tag etc which we dont need

Here we convert the ws message to listen

I just ran it succesfully again (in a linux box).

That line does no transformation, it's just a typescript hint about what it should expect the type to be.

So as far as I know we don't transfrom or sanitize the arriving WS message

yvanzo: thanks appreciate you checking, it must be something on my side. I seem to spend more time battling with getting the docker instances working than actually doing any coding, so it's a bit frustrating!

Mr_Monkey: oh okay my bad. Is there a better way to discard these unnecessary keys rather than deleting them individually

That should be done server-side to avoid sending a whole bunch of data we don't need

and also to fix the original issue which is the misplaced recording_msid key

okay. IIRC ruaok had some concerns about doing this server side when _lucifer was onto it.

that was the original intent. but it adds another serializing/deserializing step so it was decided to do on the client side

Imo we can move the recording_msid to the required position and add a comment that ws message contains a lot of unnecessary info. Does this sound okay?

it is intended to use protobuf or some other solution and then do this server side

_lucifer: how does the above solution sound to you?

shivam-kapila: sounds good to me. but Mr_Monkey might have a better solution

Can't really think of anything better to do if we don't want to handle it server-side.

We're already deserializing so there's that.

okay. I will add a comment till then and also add a test for moving keys in the ws message

Is JSON sent over the WS connection?

yes

yep

So you're instantiating an object from JSON sent by via WS?

s/by//

atj: did you sort out the docker+mac issue?

alastairp: no not yet

I've seen errors importing the full MB database before when the docker machine didn't have enough memory allocated to it

alastairp: I left it for a while to reduce the risk of me throwing my Macbook out of the window :)

alastairp: thanks, good point, I've bumped the memory up to 5GB

> So you're instantiating an object from JSON sent by via WS?

atj: yeah

shivam-kapila: on that line you linked to you have "json as Listen', where is the JS Listen object defined? (sorry might be a stupid question)

atj: did you check that? https://github.com/metabrainz/musicbrainz-docke...

atj: the listen typescript type is declared here: https://github.com/metabrainz/listenbrainz-serv...

yvanzo: sorry no I didn't. I never ran into any memory issues in Docker on Linux so I didn't think about it.

shivam-kapila: thanks :) just trying to understand how the JS side of things work.

atj: No worry, it is normally not needed to create database, but maybe it will unblock something on your macOS, or not.

yvanzo: the sample import just completed without error. Please accept my apologies for wasting your time! *embarrassed*

thanks for the tip alastairp