#metabrainz

lol

does anyone know what a real world size limit of an RMQ message is? iliekcomputers, I am thinking about the data returned from the user similarity algorithm... can we return everything in one giant chunk of data?

the default limit imposed by rabbitmq is 128Mib but not sure if you were asking about this

that mostly answers my question... now we know the default limit, what is sensical?

one message will suffice for now, I think. we don't have that many users.

yvanzo: what do you guys have set for your indexer limit? What are the implications of not processing certain messages?

thanks for the review _lucifer. If you have time, the larger PR needs a review too: https://github.com/metabrainz/listenbrainz-serv...

ruaok: yeah, sure. how about merging the small PR into this one first?

sure, let me do that.

[listenbrainz-server] mayhem merged pull request #1293 (add-mbid-mapping-to-labs-api…add-recording-search-to-labs-api): Add user facing recording search https://github.com/metabrainz/listenbrainz-serv...

👍

ok, lets see if the tests still pass.

_lucifer: are you following along in ghe planning doc?

I've written more on the second page, have a look when you can.

yes, i am. will do.

thx

alastairp: _lucifer iliekcomputers shivam-kapila : I've updated the LB social features doc with what I understand to be the current state of things. Please edit/leave comments for anything that isn't quite right.

left a few comments

thanks, integrated those already.

it currently doesn't feel like we have enough work to keep all of us busy for an entire week, but I am sure this task will guppy up.

I could take on the troi-bot posting as a stretch goal.

hmm. the feature to recommend a track -- is that part of the timeline or is that separate?

i don't think it was, originally.

ok, adding a new section.

oh btw

alastairp: should we work on lb consul upgrade today?

yes, that'd be a great idea

I'm in a meeting for 1-1.5 hours more, but I'm kind of available here from time to time if I'm not paying too much attention to the meeting

let me open the PRs

legoktm: should we be using the Wikimedia REST API for getting the extracts rather than the PHP API?

I see https://en.wikipedia.org/w/api.php?action=query... vs https://en.wikipedia.org/api/rest_v1/page/summa... returns different content - the REST API removes more unneeded formatting and whatnot but it also returns less paragraphs

What's the general expectation of old exintro vs new summary?

bitmap: is https://tickets.metabrainz.org/browse/MBS-9858 something your timeline rewrite affects at all?

MBS-9858: Statistics/Time Graph/Rate of change: incorrect handling 31/30/28 day months?

_lucifer: sorry, some stuff came up. do you want to do this now?

alastairp: no worries. sure, let's do it.

[listenbrainz-server] alastair closed pull request #1282 (startup…startup-dump-logging): Improved dump logging and dump hardening (lol), try II https://github.com/metabrainz/listenbrainz-serv...

OK, I merged that PR into mine. I'm just opening some tickets to add some future tasks

can you check out the startup branch and merge it? After that I'll show you how to set up consul on your local machine to test each of the services

s/merge/build/

on it.

OK, I've opened tickets for the outstanding comments at https://github.com/metabrainz/listenbrainz-serv...

[musicbrainz-server] reosarevok opened pull request #1943 (master…MBS-9857): MBS-9857: Make merging HTTP->HTTPS versions of URL an auto edit https://github.com/metabrainz/musicbrainz-serve...

to me, the key config items are here: https://github.com/metabrainz/listenbrainz-serv...

_lucifer: can you look at each of these sections and double-check that it appears that 1) each config block acts on a single directory in /etc/service/, 2) each block takes files from a single directory in ./docker/services/, and 3) each block adds a `down` file to /etc/service/[service]

yes, sure.

Good evening. ruaok: I have added some comments to the planning doc. overall it looks fine

alastairp, one nitpick, follow_dispatcher service is now called websockets. we should probably rename that here as well.

good catch, please update it

👍

this has already been updated in docker-server-configs. The container name is now `listenbrainz-websockets-$DEPLOY_ENV`

the build failed with this error

i think this was fixed in master recently

mmm, I think I saw that once, and I think it might be related to a fix that Mr_Monkey made

can you merge master into that branch?

yup, will do that.

_lucifer: where are you at?

was away for a few mins. resolving merge conflicts currently.

where are there conflicts?

a few were in the Dockerfile. those are done.

remaining are in uswgi-*.ini files

ah yes, because I added the logging items?

roger_that: We use index_limit = 2000000; Requeuing some messages just delay updating search indexes.

alastairp, yes

those files should be deleted right?

_lucifer: we should be left with 1 for each flask-based service (app, compat app, labs)

:+1

alastairp, updated the PR.

cool, let me review it again

looks good to me. Do you want to build and test it locally?

how can we make sure if the port here is set correctly?

i took a look at the changes. everything else seems to be in order.

SERVICE_3031_NAME -> 3031 is the port

ah, makes sense. thanks!

the other thing we need to do before we merge is undo the changes to docker-compose.yml

however, before we do that, do you want to set up consul locally and test config generation/app startup?

yes sure

I made this quick script to put things into consul for quick testing, I wasn't sure if there is a better tool for using, but this is fine for me for now

I installed consul, and in a venv python-consul

I start consul with `consul agent -dev`

and then I put some stuff into it with `python publish_consul.py LB/config.prod.json consul/LB/config.prod.json` (from in the docker-server-configs checkout)

what OS/docker stack are you running?

Ubuntu 20.04

ah, because I have `CONSUL_HOST: host.docker.internal` in docker-compose.yml

so that I can run consul on my local machine, however this dns name only works on mac/windows

we might have to run the consul server in docker-compose

oh ok

I wonder if this might help: https://github.com/metabrainz/mbspotify/blob/94...

should i remove the CONSUL_HOST variable as well?

let me show you how this all fits together

i am bit a confused here. yes please :D

it has `runit` installed. this manages the startup scripts that we put in /etc/service

when a docker container starts, it first executes /etc/rc.local (this is why we put stuff here), and then it looks in /etc/service for any directory _without_ a `down` file, and executes its `run` file

In most cases this executes run-consul-template: https://github.com/metabrainz/docker-consul-tem...

makes sense

this connects to a consul server (think of consul as a key/value store for application configuration, it has some other magic that isn't necessary to know now)

consul-template takes a config file (e.g. https://github.com/metabrainz/listenbrainz-serv...), performs susbstitution on the template, generating our config.py, and then runs the script in the `exec {}` block

so on MeB servers there's a centralised consul server, it loads its config magically from the contents of the consul directory in docker-server-configs

so if we want to run the production server workflow locally, we need to start up a consul server, populate it with relevant config information, and then make the `webserver` container connect to it (with the CONSUL_HOST env variable)

that clears it up. thanks for the clear explanation :D

one question though where does git2consul come into play/

git2consul is the magic that reads those files and pushes them into consul

ok, makes sense.

now to run consul server locally, i should add the consulagent and git2consul service to my docker-compose.yml and modify CONSUL_HOST accordingly?

I _think so_. I've not done this before sorry

no worries. let me try this.

[musicbrainz-server] yvanzo opened pull request #1944 (master…mbs-10416-cntrl-chars): MBS-10416: Remove invalid characters from newly entered annotations https://github.com/metabrainz/musicbrainz-serve...

alastairp: getting many errors like the one above

ah, that appears to the be the issue I was seeing when I tried to deploy

and it appeared to be because we were using consul-template 0.16 instead of 0.18. Check that the python base image you're using is correct