#metabrainz

I've seen e.g. https://github.com/rembo10/headphones/wiki/Musi...

... which doesn't have much to do with the MB community, but presumably it helps /somebody/ :-)

Project musicbrainz-server_master build #394: ABORTED in 13 sec: https://ci.metabrainz.org/job/musicbrainz-serve...

grrr I keep getting logged out of ci so I can't cancel things before they start

if I accidentially visit http, then switch to https, I'm logged out

there's a "Whitelist Target Branches" option

maybe we can just set it to master?

ok, done

what does it do?

ideally not build pull requests to schema-change-2016-q2

cool

"Adding branches to this whitelist allows you to selectively test pull requests destined for these branches only. Supports regular expressions (e.g. 'master', 'feature-.*')."

ah, so only prs to master will be built, that sounds good

or ^(master|beta|production)$ if that actually works

I read the policy regarding the Internet Archive storage, which lists it for historical, research purposes and fair use - sounds good to me. Then the MB site says use images at your own risk, I'm assuming they are referring to commercial interests?

basically any redistribution or non-private use of the images is at your own risk, since nobody owns the copyrights to them

nobody meaning MB or the IA

Maybe I could add a coffee stain to the images and call it my art ;-)

ah, I wouldn't worry about that. if the IA gets a complaint, they'll take the images down (it's happened before)

ahh so there's no guarantee that if I spend time to do this, it will *stick*. Can images be added to a release through an API? This way I could more easily upload it in a batch sort of way?

well, by happened before, I mean 0.something% of all uploads :)

there's not an API to do this, unfortunately

who doesn't want to be archived? that's crazytalk.

OK, got it. Does the original uploader of an image get notified if it get's deleted? Actually how does this work, doesn't MB just store a link to to the image at IA (CAA). How does MB know that the IA has deleted an image?

we don't, actually. there's a way to get a list of deleted images from the IA (in a bug report somewhere), but it hasn't been implemented yet

it hasn't happened frequently enough to be a development priority, I guess

makes sense

bitmap: added beta and production to that list

thx

about an API for uploads, the main problem is that the CAA was designed so that images never pass through MB servers

but only MB has the keys necessary to sign uploads directly to the IA

Gentlecat: seems to be working :)

\o/

bitmap: ping

ah nvm, astro / was full, i removed >280Gb of nginx logs (only .gz),

How come astro alone was using so high bandwidth ?

And that's not nginx according to http://stats.metabrainz.org/dashboard/db/all-ng...

bitmap: can you take a look at https://github.com/metabrainz/pytools/pulls if you have some time?

Gentlecat: sure thing

thanks!

zas: maybe a full db export?

there was a --with-full-export running, so I guess

the 30d range makes that easier to see

super minor formatting issue in notification mail, missing space : "Label "2 Swords"(Copy Paste Soul's personal imprint) - merged by edit #38369836

there should be a space after " and before (

bitmap: then this export may have failed, due to lack of disk space, did you verify it ?

so, uhm, who is responsible for the local weather. I'd like to file a report. :(

Hey good morning Rob !

morning!

Finally at home ?

yep, got home last night.

and all I want to do is go for a nice long ride in the spanish sun. except there is no sun today. oh well, manaña. :)

ruaok: i thought about something about number of IPs we need at NewHost, having more IPs is good when it comes to the number of sockets restrictions, which are (source ip, source port, dest ip, dest port) tuple (do you remember ernie/bert issues related to this at some point). Since we want to be able to handle a lot of simultaneous connections, better

spread services on different IPs (and hostnames).

Also i think about redis HA setup, we'll need another server for that (one redis master and, at least, one redis slave, basically the setup is based on HAProxy + redis sentinel, and is quite simple. It will solve a reliability issue we currently have (that is losing redis master). bitmap said mbs doesn't handle well (at all) redis master failures.

make sense to me. just keep adding these thoughts to the doc.

ok

now as far as redis masters... the use case for redis is so bizarre that most machines don't fit really well for that.

ideally we'd need 64GB with one or two cores.

really weird setup.

i think redis makes good use of more cores, to be verified

so, the question in my mind is this: what do we use that is CPU intensive, but not memory intensive?

making good use and using them all are two different things no?

Yes ;)

a memcached machine is bored most of the time, but with massive memory use.

I'd like to find a complementary task, that also needs to be HA, that we can stick on those servers with redis for better utilization.

This is why we can run cpu intensive processes (but not memory intensive processes) on the same machines

exactly.

indexer is one (on jiji)

building indexes is one, but that will go away hopefully soon.

what are our future, long term use cases?

perhaps compression (backups, logs ?)

logs, quite possibly.

logs would fit

we need HA centralized logs storage

if logs are written through network, and not stored on most machines, it means we can go for small and fast non-RAID SSD for web servers ie?

just as long as we don't impement that before the move. after the move, yes.

Same goes for gateways

why SSD?

HDD should be sufficient.

Oh yes, i mean HDD

ok.

astro disk is usually < 5%

but yes, non RAID HDD on app servers.

:)

We need to think about backups

not coffee??

NewHost may not have someone to rotate our USB drives ;)

oh, yes that concept is dead.

but, given that EACH machine has a massive bandwitdh budget, we take our backups machine and give it a public IP.

then blast backups as fast as we can to google cloud or glacier.

probably both.

google cloud and then once a week to glacier as well

Yup, may be a soluton

solution*

ok coffee ;)

for super fast retrieval from google cloud, and archival on glacier.

bai!

About number of web servers... i did some rough calculation, i would like to redo with you

ok.

the good thing about the vrack is that we can start smaller and expand when we want to.

search, web servers and postgres can scale easily with the new setup.

gateways, when done properly will have loads of capacity and will scale nicely on a 2 host setup for quite some time.

we also need to think about what traffic we want to serve.

I'm not interested in spending a lot of hosting resources on headphones.

speaking of headphones, the new rate limiting needs to have per-app limits.

Coffee ready.

Ok, based on current incoming reqs

we have 1k req/s incoming

around 65% hit our web servers

and 55-60% of those are rate limited (503s)

so we actually return ~270 req/s as 20x

so let's say we want to double that, and convert most 503s in 200s

it will double load on web servers, so if we want to keep same load with same hardware it would mean having x2 web servers (we have 5) -> 10

but new web servers will be more performant, likely at least 30%, -> 7

considerably more, I would think.

these machines were given to us in 2010 and were several years old.