#metabrainz

we have quest which has some diskspace, but it needs some preparation, I can have a look tomorrow (1AM here)

no problem, it can wait until tomorrow

actually there's not much to do, I'll try to prepare it right now

hmmm, nope, I forgot this machine has no local net 10.2.2 so it is a bit trickier to setup

I'll have a look tomorrow

👍

yvanzo: if you're around tomorrow morning and have time to prepare a copy of https://github.com/metabrainz/docker-server-con... for trille -> quest that'd be great

bitmap: I managed to have picard website (beta) on quest, see https://github.com/metabrainz/docker-server-con...

still experimental, but I think we are slowly moving towards getting rid of 10.2.2

I'll continue on this tomorrow, sleepy now

mooooin!

Does the MusicBrainz API has any sort of API Terms of use like this, i can link to? Maybe just to the API doc? https://www.themoviedb.org/api-terms-of-use

I am working on the privacy policy for my android app which uses the MusicBrainz API. Does this also apply to the API? https://wiki.musicbrainz.org/About/Privacy_Policy

Which data is stored, in example in server logs? IP addresses?

Anything else I need to mention in the app? Something like "this is not a part of MetaBrainz"?

O’Moin

Hi relaxoMob: Good questions. As a developer, make sure to follow https://musicbrainz.org/doc/MusicBrainz_API and more particularly “What can I do with”, “Who can use”, “Application rate limiting and identification”

Our privacy policy should be updated to include the API, so yes link to it but no it doesn’t cover it at the moment.

Basically, just link to https://metabrainz.org/privacy

API server “logs include: your IP address, your browser's "User-Agent" string, and which page you requested” as in https://wiki.musicbrainz.org/About/Privacy_Poli...

And yes, you can specify that the MusicBrainz API is provided by a third-party, the MetaBrainz Foundation.

bitmap, zas: available now, just reading the back logs.

yvanzo: moooin

[musicbrainz-server] 14reosarevok opened pull request #3094 (03master…MBS-13368): MBS-13368: Properly compare series relationships if series is target https://github.com/metabrainz/musicbrainz-serve...

yvanzo: what are the exact dependencies for this stuff? I was thinking about quest, but this one isn't on 10.2.2.0/24 (but it is on 10.10.10.0/24) and not sure if we can run everything there, but for sure it has disk space and free to use right now;

also, we have many services only listening on 10.2.2.0/24, it can be an issue if there are to be accessed from 10.10.10.0/24 on machines not having 10.2.2.x IP.

zas: It needs to have access to pg (jimmy/hendrix) and possibly redis (aphex) through consul, and to sshd (aretha) through local config.

s/possibly// (for sure)

(pg through consul too)

well, consul will report IP/port as provided by serviceregistrator, and that's usually 10.2.2.x IP, which is an issue for quest

so we need to find a way to expose needed services on 10.10.10.x too

that's tricky

atj: around?

the experiment I did yesterday shows we can get gateways using backends on 10.10.10.x (because they are parts of this network, and backend service run on it too, plus serviceregistrator provides correct IP/port)

would it be possible to use a mirror db (instead of directly accessing to jimmy/hendrix) ? Why does it need redis (which one?) ?

Those were the dependencies for mb production cron, I will check those for meb.o website too

zas: no, because it is also making changes to the DB.

hmmm, then we need to get db accessible from 10.10.10.x, what needs to be done for it to happen? and it will not suffice... ok. Can't we run this stuff on hendrix and/or jimmy ? they have everything needed

It needs access to musicbrainz-redis-store-4 to expire some sessions, and access to musicbrainz-redis-cache-4 for caching.

hendrix is running the secondary DB iiuc, it could be the best choice.

yes, and it's much simpler to set up

zas: where are the primary/secondary roles defined atm? (somewhere in ansible iirc)

i'll be around in 30 mins or so

why does the dump fail on trille?

disk space

trille doesn't have much of it, it wasn't meant to host huge dumps

yvanzo: it isn't defined in ansible, and with recent changes bitmap did on pg, I think it is more dynamic (see https://github.com/metabrainz/syswiki/blob/mast...)

relaxoMob: Actually, this is a web API, and our privacy policy already cover web access logs: https://metabrainz.org/privacy :)

Primary and secondary is set in docker AFAIU

the docker image, sorry

yes, found it, it is set by IP address: docker-postgres-cluster/Makefile

Weird Indian pizzas. Lunchtime break at the Indian summit.

monkey: I need my breakfast now!

oops, mayhem ^

weird but at least no pineapple ;)

how much disk space is required?

More than 100G

(disk space currently available on trille)

is that compressed size?

+ the current volumes, let me check

more than 250G, total

it contains a lot of already compressed archives.

(250G is what trille provides for those volumes and free disk space left)

that's not too bad then

(I mean 250G is currently not enough but not too far.)

If we can have 350G, we should be fine until we split those into separate containers that can run on different servers.

yvanzo: I was informed that these are not the weird pizzas we wanted. These are standard fare. Super tasty.

They did look pretty normal

zas: just noticed that there is currently no temperature data for jimmy and hendrix

(I was trying to compare their respective activity as hendrix is also running the search indexer)

but otherwise they use about the same amount of memory, so hendrix definitely seems to be a good choice, I will prepare PRs

yvanzo: yes, it is because the hardware is very new, and it seems there's no support yet for temp sensors on those. I guess we'll be able to get it work after some time passed.

👍

yvanzo: I will create a dedicated ZFS dataset for the dumps that will need to be bind mounted in the containers

yvanzo: the primary/secondary status is mainly stored in the database cluster (and you can query PG to determine whether a server is primary). ideally it isn't hard-coded anywhere, but there are a couple places still: the barman and repmgr configuration files

the PRIMARY var in docker-postgres-cluster/Makefile actually isn't used, I'll remove it

bitmap: Ok thanks!

btw, I nearly have a patch ready that will cause MB to direct a lot more read-only traffic to the standby...but if the current load average on jimmy is any indication, that hopefully shouldn't be any problem for running the dumps and whatnot

bitmap: Is there any reason for defining `RSYNC_FULLEXPORT_PORT` in `admin/config.incremental-json-dump.sh`instead of `script/services.sh`? Is it okay if I change that?

yvanzo: yes, I think it makes senes to move them

it makes it easier to change the configuration without restarting the container (if you are setting the env variable on container start)

but restarting a cron container isn't an issue

It’s just that this has to be changed for moving stuff from trille to hendrix

right, having it all in one repo makes things easier too.

So a new image is needed anyway.

[musicbrainz-server] 14reosarevok opened pull request #3095 (03master…MBS-13366): MBS-13366: Also show medium data on Edit medium preview https://github.com/metabrainz/musicbrainz-serve...

bitmap: Also why do we have production cron on trille and the sshd for fullexport on aretha, while we have json dumper on aretha and the sshd for it on trille?

heh

well, json dumps are on aretha because it had the most space available for it

and ftp is on aretha, which is where the full exports are synced to

the incremental json dumps are synced back to trille because that's where metabrainz.org is hosted

the full json dumps are sent to ftp

Ok, I didn’t notice that meb.o had a direct dependency on this volume.

I can see it from `services.sh` now.

Thanks!

reosarevok: just remembered that for https://github.com/metabrainz/musicbrainz-serve... since prod and beta share a cache, we will have to deploy the {Data,Entity}::LinkAttributeType types to both prod and beta before we can put the rest of the patch on beta

Oh, ok. Do you see any reason why that'd be problematic?

(other than the annoyance of putting them on prod early)

nope, I mainly just have to put it in a separate commit/PR that targets the production branch

Ok :) Feel free

btw, for https://github.com/metabrainz/musicbrainz-serve... maybe you can look into adding some basic tests for the whole duplicates setup :)

(seems no test explicitly tests compareRelationships but that might be because it's implicit elsewhere)

heh

not sure if I will get to it today but I can have a look

Oh, no need at all for it to be today

> console.log('it thinks it is the same');

It doesn't need to be even before this is merged IMO if we document needing them eventually, but we might just want them

Oh ffs, I didn't push :D

(I did change that locally)

On it

I'm glad I usually don't accidentally commit my console.logs, because I use much less professional messages

[musicbrainz-server] 14yvanzo opened pull request #3096 (03production…move-cron-to-hendrix): Drop hard-coded IP address for incrental JSON dump https://github.com/metabrainz/musicbrainz-serve...

[musicbrainz-server] 14yvanzo merged pull request #3096 (03production…move-cron-to-hendrix): Drop hard-coded IP address for incrental JSON dump https://github.com/metabrainz/musicbrainz-serve...

aerozol, kellnerd, your input in https://tickets.metabrainz.org/browse/STYLE-2416 would be lovely

STYLE-2416: Add "Broadcasted By" relationship for Labels or Places

(see my comment)

bitmap: well, I do too once I get frustrated enough, luckily I didn't here ;p

yes, sometimes I have to grep for profanities before I commit :P

Hehe, my temporary log messages are usually only too short to be understood by anyone else :)

Serious talk about doing the summit in India next year.... :-)

reosarevok: Looks great :) How about making https://musicbrainz.org/relationship-attribute/... (time) available for this relationship?

I considered it, but