Which data is stored, in example in server logs? IP addresses?
2023-11-17 32148, 2023
relaxoMob
Anything else I need to mention in the app? Something like "this is not a part of MetaBrainz"?
2023-11-17 32154, 2023
prout has quit
2023-11-17 32106, 2023
prout joined the channel
2023-11-17 32125, 2023
yvanzo
O’Moin
2023-11-17 32123, 2023
yvanzo
Hi relaxoMob: Good questions. As a developer, make sure to follow https://musicbrainz.org/doc/MusicBrainz_API and more particularly “What can I do with”, “Who can use”, “Application rate limiting and identification”
2023-11-17 32123, 2023
yvanzo
Our privacy policy should be updated to include the API, so yes link to it but no it doesn’t cover it at the moment.
yvanzo: what are the exact dependencies for this stuff? I was thinking about quest, but this one isn't on 10.2.2.0/24 (but it is on 10.10.10.0/24) and not sure if we can run everything there, but for sure it has disk space and free to use right now;
2023-11-17 32124, 2023
zas
also, we have many services only listening on 10.2.2.0/24, it can be an issue if there are to be accessed from 10.10.10.0/24 on machines not having 10.2.2.x IP.
2023-11-17 32119, 2023
yvanzo
zas: It needs to have access to pg (jimmy/hendrix) and possibly redis (aphex) through consul, and to sshd (aretha) through local config.
2023-11-17 32141, 2023
yvanzo
s/possibly// (for sure)
2023-11-17 32138, 2023
yvanzo
(pg through consul too)
2023-11-17 32122, 2023
zas
well, consul will report IP/port as provided by serviceregistrator, and that's usually 10.2.2.x IP, which is an issue for quest
2023-11-17 32146, 2023
zas
so we need to find a way to expose needed services on 10.10.10.x too
2023-11-17 32149, 2023
zas
that's tricky
2023-11-17 32154, 2023
zas
atj: around?
2023-11-17 32105, 2023
zas
the experiment I did yesterday shows we can get gateways using backends on 10.10.10.x (because they are parts of this network, and backend service run on it too, plus serviceregistrator provides correct IP/port)
2023-11-17 32110, 2023
zas
would it be possible to use a mirror db (instead of directly accessing to jimmy/hendrix) ? Why does it need redis (which one?) ?
2023-11-17 32116, 2023
yvanzo
Those were the dependencies for mb production cron, I will check those for meb.o website too
2023-11-17 32132, 2023
yvanzo
zas: no, because it is also making changes to the DB.
2023-11-17 32141, 2023
zas
hmmm, then we need to get db accessible from 10.10.10.x, what needs to be done for it to happen? and it will not suffice... ok. Can't we run this stuff on hendrix and/or jimmy ? they have everything needed
2023-11-17 32109, 2023
yvanzo
It needs access to musicbrainz-redis-store-4 to expire some sessions, and access to musicbrainz-redis-cache-4 for caching.
2023-11-17 32117, 2023
yvanzo
hendrix is running the secondary DB iiuc, it could be the best choice.
2023-11-17 32134, 2023
zas
yes, and it's much simpler to set up
2023-11-17 32118, 2023
yvanzo
zas: where are the primary/secondary roles defined atm? (somewhere in ansible iirc)
2023-11-17 32118, 2023
atj
i'll be around in 30 mins or so
2023-11-17 32125, 2023
atj
why does the dump fail on trille?
2023-11-17 32133, 2023
zas
disk space
2023-11-17 32106, 2023
zas
trille doesn't have much of it, it wasn't meant to host huge dumps
Weird Indian pizzas. Lunchtime break at the Indian summit.
2023-11-17 32157, 2023
yvanzo
monkey: I need my breakfast now!
2023-11-17 32105, 2023
yvanzo
oops, mayhem ^
2023-11-17 32128, 2023
yvanzo
weird but at least no pineapple ;)
2023-11-17 32132, 2023
atj
how much disk space is required?
2023-11-17 32158, 2023
yvanzo
More than 100G
2023-11-17 32110, 2023
yvanzo
(disk space currently available on trille)
2023-11-17 32131, 2023
atj
is that compressed size?
2023-11-17 32131, 2023
yvanzo
+ the current volumes, let me check
2023-11-17 32109, 2023
yvanzo
more than 250G, total
2023-11-17 32124, 2023
yvanzo
it contains a lot of already compressed archives.
2023-11-17 32118, 2023
yvanzo
(250G is what trille provides for those volumes and free disk space left)
2023-11-17 32147, 2023
atj
that's not too bad then
2023-11-17 32135, 2023
yvanzo
(I mean 250G is currently not enough but not too far.)
2023-11-17 32114, 2023
yvanzo
If we can have 350G, we should be fine until we split those into separate containers that can run on different servers.
2023-11-17 32132, 2023
mayhem
yvanzo: I was informed that these are not the weird pizzas we wanted. These are standard fare. Super tasty.
2023-11-17 32116, 2023
monkey
They did look pretty normal
2023-11-17 32146, 2023
fletchto99 has quit
2023-11-17 32130, 2023
fletchto99 joined the channel
2023-11-17 32129, 2023
yvanzo
zas: just noticed that there is currently no temperature data for jimmy and hendrix
2023-11-17 32159, 2023
yvanzo
(I was trying to compare their respective activity as hendrix is also running the search indexer)
2023-11-17 32156, 2023
yvanzo
but otherwise they use about the same amount of memory, so hendrix definitely seems to be a good choice, I will prepare PRs
2023-11-17 32116, 2023
zas
yvanzo: yes, it is because the hardware is very new, and it seems there's no support yet for temp sensors on those. I guess we'll be able to get it work after some time passed.
2023-11-17 32130, 2023
yvanzo
đź‘Ť
2023-11-17 32143, 2023
atj
yvanzo: I will create a dedicated ZFS dataset for the dumps that will need to be bind mounted in the containers
2023-11-17 32141, 2023
Maxr1998_ joined the channel
2023-11-17 32139, 2023
Maxr1998 has quit
2023-11-17 32104, 2023
bitmap
yvanzo: the primary/secondary status is mainly stored in the database cluster (and you can query PG to determine whether a server is primary). ideally it isn't hard-coded anywhere, but there are a couple places still: the barman and repmgr configuration files
2023-11-17 32127, 2023
bitmap
the PRIMARY var in docker-postgres-cluster/Makefile actually isn't used, I'll remove it
2023-11-17 32149, 2023
yvanzo
bitmap: Ok thanks!
2023-11-17 32112, 2023
bitmap
btw, I nearly have a patch ready that will cause MB to direct a lot more read-only traffic to the standby...but if the current load average on jimmy is any indication, that hopefully shouldn't be any problem for running the dumps and whatnot
2023-11-17 32130, 2023
yvanzo
bitmap: Is there any reason for defining `RSYNC_FULLEXPORT_PORT` in `admin/config.incremental-json-dump.sh`instead of `script/services.sh`? Is it okay if I change that?
2023-11-17 32101, 2023
bitmap
yvanzo: yes, I think it makes senes to move them
2023-11-17 32100, 2023
bitmap
it makes it easier to change the configuration without restarting the container (if you are setting the env variable on container start)
2023-11-17 32106, 2023
bitmap
but restarting a cron container isn't an issue
2023-11-17 32149, 2023
yvanzo
It’s just that this has to be changed for moving stuff from trille to hendrix
2023-11-17 32112, 2023
bitmap
right, having it all in one repo makes things easier too.
bitmap: Also why do we have production cron on trille and the sshd for fullexport on aretha, while we have json dumper on aretha and the sshd for it on trille?
2023-11-17 32144, 2023
bitmap
heh
2023-11-17 32118, 2023
bitmap
well, json dumps are on aretha because it had the most space available for it
2023-11-17 32121, 2023
bitmap
and ftp is on aretha, which is where the full exports are synced to
2023-11-17 32129, 2023
relaxoMob has quit
2023-11-17 32108, 2023
bitmap
the incremental json dumps are synced back to trille because that's where metabrainz.org is hosted
2023-11-17 32135, 2023
bitmap
the full json dumps are sent to ftp
2023-11-17 32147, 2023
yvanzo
Ok, I didn’t notice that meb.o had a direct dependency on this volume.
2023-11-17 32127, 2023
yvanzo
I can see it from `services.sh` now.
2023-11-17 32134, 2023
yvanzo
Thanks!
2023-11-17 32140, 2023
relaxoMob joined the channel
2023-11-17 32118, 2023
bitmap
reosarevok: just remembered that for https://github.com/metabrainz/musicbrainz-server/… since prod and beta share a cache, we will have to deploy the {Data,Entity}::LinkAttributeType types to both prod and beta before we can put the rest of the patch on beta
2023-11-17 32148, 2023
reosarevok
Oh, ok. Do you see any reason why that'd be problematic?
2023-11-17 32155, 2023
reosarevok
(other than the annoyance of putting them on prod early)
2023-11-17 32149, 2023
bitmap
nope, I mainly just have to put it in a separate commit/PR that targets the production branch
