grrr I keep getting logged out of ci so I can't cancel things before they start
2016-04-30 12106, 2016
D4RK-PH0ENiX joined the channel
2016-04-30 12125, 2016
bitmap
if I accidentially visit http, then switch to https, I'm logged out
2016-04-30 12117, 2016
Gentlecat
there's a "Whitelist Target Branches" option
2016-04-30 12128, 2016
Gentlecat
maybe we can just set it to master?
2016-04-30 12153, 2016
Gentlecat
ok, done
2016-04-30 12104, 2016
bitmap
what does it do?
2016-04-30 12121, 2016
Gentlecat
ideally not build pull requests to schema-change-2016-q2
2016-04-30 12139, 2016
bitmap
cool
2016-04-30 12148, 2016
Gentlecat
"Adding branches to this whitelist allows you to selectively test pull requests destined for these branches only. Supports regular expressions (e.g. 'master', 'feature-.*')."
2016-04-30 12100, 2016
bitmap
ah, so only prs to master will be built, that sounds good
2016-04-30 12155, 2016
bitmap
or ^(master|beta|production)$ if that actually works
2016-04-30 12131, 2016
linuxrocks
I read the policy regarding the Internet Archive storage, which lists it for historical, research purposes and fair use - sounds good to me. Then the MB site says use images at your own risk, I'm assuming they are referring to commercial interests?
2016-04-30 12132, 2016
bitmap
basically any redistribution or non-private use of the images is at your own risk, since nobody owns the copyrights to them
2016-04-30 12116, 2016
bitmap
nobody meaning MB or the IA
2016-04-30 12132, 2016
linuxrocks
I just want to upload much of the content I have, encouraging others to as well - to make a more complete archive. But I don't want to cause any issues with MB or IA.
2016-04-30 12122, 2016
linuxrocks
Maybe I could add a coffee stain to the images and call it my art ;-)
2016-04-30 12135, 2016
bitmap
ah, I wouldn't worry about that. if the IA gets a complaint, they'll take the images down (it's happened before)
2016-04-30 12109, 2016
JesseW joined the channel
2016-04-30 12158, 2016
linuxrocks
ahh so there's no guarantee that if I spend time to do this, it will *stick*. Can images be added to a release through an API? This way I could more easily upload it in a batch sort of way?
2016-04-30 12153, 2016
bitmap
well, by happened before, I mean 0.something% of all uploads :)
2016-04-30 12106, 2016
bitmap
there's not an API to do this, unfortunately
2016-04-30 12149, 2016
CallerNo6
who doesn't want to be archived? that's crazytalk.
2016-04-30 12107, 2016
linuxrocks
OK, got it. Does the original uploader of an image get notified if it get's deleted? Actually how does this work, doesn't MB just store a link to to the image at IA (CAA). How does MB know that the IA has deleted an image?
2016-04-30 12109, 2016
bitmap
we don't, actually. there's a way to get a list of deleted images from the IA (in a bug report somewhere), but it hasn't been implemented yet
2016-04-30 12150, 2016
bitmap
it hasn't happened frequently enough to be a development priority, I guess
2016-04-30 12111, 2016
linuxrocks
makes sense
2016-04-30 12101, 2016
Gentlecat
bitmap: added beta and production to that list
2016-04-30 12110, 2016
bitmap
thx
2016-04-30 12139, 2016
bitmap
about an API for uploads, the main problem is that the CAA was designed so that images never pass through MB servers
2016-04-30 12131, 2016
bitmap
but only MB has the keys necessary to sign uploads directly to the IA
2016-04-30 12114, 2016
bitmap
Gentlecat: seems to be working :)
2016-04-30 12122, 2016
Gentlecat
\o/
2016-04-30 12109, 2016
JesseW has quit
2016-04-30 12159, 2016
JesseW joined the channel
2016-04-30 12139, 2016
JesseW has quit
2016-04-30 12152, 2016
bitmap has quit
2016-04-30 12113, 2016
bitmap joined the channel
2016-04-30 12133, 2016
QuoraUK has quit
2016-04-30 12147, 2016
zas
bitmap: ping
2016-04-30 12144, 2016
zas
ah nvm, astro / was full, i removed >280Gb of nginx logs (only .gz),
there was a --with-full-export running, so I guess
2016-04-30 12157, 2016
bitmap
the 30d range makes that easier to see
2016-04-30 12140, 2016
yeeeargh joined the channel
2016-04-30 12158, 2016
neersighted has quit
2016-04-30 12108, 2016
neersighted joined the channel
2016-04-30 12108, 2016
neersighted has quit
2016-04-30 12124, 2016
JesseW joined the channel
2016-04-30 12136, 2016
kanha has quit
2016-04-30 12124, 2016
neersighted joined the channel
2016-04-30 12104, 2016
darwin
super minor formatting issue in notification mail, missing space : "Label "2 Swords"(Copy Paste Soul's personal imprint) - merged by edit #38369836
2016-04-30 12118, 2016
darwin
there should be a space after " and before (
2016-04-30 12102, 2016
linuxrocks is now known as linuxrocks_
2016-04-30 12148, 2016
linuxrocks_ is now known as linuxrocks
2016-04-30 12130, 2016
linuxrocks has left the channel
2016-04-30 12128, 2016
linuxrocks joined the channel
2016-04-30 12146, 2016
linuxrocks has left the channel
2016-04-30 12128, 2016
linuxrocks joined the channel
2016-04-30 12102, 2016
JesseW has quit
2016-04-30 12128, 2016
regagain joined the channel
2016-04-30 12101, 2016
kanha joined the channel
2016-04-30 12128, 2016
UmkaDK has quit
2016-04-30 12118, 2016
zas
bitmap: then this export may have failed, due to lack of disk space, did you verify it ?
2016-04-30 12123, 2016
ruaok
so, uhm, who is responsible for the local weather. I'd like to file a report. :(
2016-04-30 12150, 2016
zas
Hey good morning Rob !
2016-04-30 12157, 2016
ruaok
morning!
2016-04-30 12118, 2016
zas
Finally at home ?
2016-04-30 12147, 2016
ruaok
yep, got home last night.
2016-04-30 12110, 2016
ruaok
and all I want to do is go for a nice long ride in the spanish sun. except there is no sun today. oh well, manaña. :)
2016-04-30 12142, 2016
zas
ruaok: i thought about something about number of IPs we need at NewHost, having more IPs is good when it comes to the number of sockets restrictions, which are (source ip, source port, dest ip, dest port) tuple (do you remember ernie/bert issues related to this at some point). Since we want to be able to handle a lot of simultaneous connections, better
2016-04-30 12142, 2016
zas
spread services on different IPs (and hostnames).
2016-04-30 12157, 2016
zas
Also i think about redis HA setup, we'll need another server for that (one redis master and, at least, one redis slave, basically the setup is based on HAProxy + redis sentinel, and is quite simple. It will solve a reliability issue we currently have (that is losing redis master). bitmap said mbs doesn't handle well (at all) redis master failures.
2016-04-30 12134, 2016
ruaok
make sense to me. just keep adding these thoughts to the doc.
2016-04-30 12155, 2016
zas
ok
2016-04-30 12101, 2016
ruaok
now as far as redis masters... the use case for redis is so bizarre that most machines don't fit really well for that.
2016-04-30 12115, 2016
ruaok
ideally we'd need 64GB with one or two cores.
2016-04-30 12119, 2016
ruaok
really weird setup.
2016-04-30 12139, 2016
zas
i think redis makes good use of more cores, to be verified
2016-04-30 12140, 2016
ruaok
so, the question in my mind is this: what do we use that is CPU intensive, but not memory intensive?
2016-04-30 12159, 2016
ruaok
making good use and using them all are two different things no?
2016-04-30 12109, 2016
zas
Yes ;)
2016-04-30 12114, 2016
ruaok
a memcached machine is bored most of the time, but with massive memory use.
2016-04-30 12139, 2016
ruaok
I'd like to find a complementary task, that also needs to be HA, that we can stick on those servers with redis for better utilization.
2016-04-30 12150, 2016
zas
This is why we can run cpu intensive processes (but not memory intensive processes) on the same machines
2016-04-30 12101, 2016
ruaok
exactly.
2016-04-30 12113, 2016
zas
indexer is one (on jiji)
2016-04-30 12125, 2016
ruaok
building indexes is one, but that will go away hopefully soon.
2016-04-30 12134, 2016
ruaok
what are our future, long term use cases?
2016-04-30 12138, 2016
zas
perhaps compression (backups, logs ?)
2016-04-30 12157, 2016
ruaok
logs, quite possibly.
2016-04-30 12113, 2016
zas
logs would fit
2016-04-30 12131, 2016
zas
we need HA centralized logs storage
2016-04-30 12153, 2016
zas
if logs are written through network, and not stored on most machines, it means we can go for small and fast non-RAID SSD for web servers ie?
2016-04-30 12102, 2016
ruaok
just as long as we don't impement that before the move. after the move, yes.
2016-04-30 12115, 2016
zas
Same goes for gateways
2016-04-30 12125, 2016
ruaok
why SSD?
2016-04-30 12135, 2016
ruaok
HDD should be sufficient.
2016-04-30 12154, 2016
zas
Oh yes, i mean HDD
2016-04-30 12158, 2016
ruaok
ok.
2016-04-30 12105, 2016
ruaok
astro disk is usually < 5%
2016-04-30 12125, 2016
zas still under the needed coffee level
2016-04-30 12126, 2016
ruaok
but yes, non RAID HDD on app servers.
2016-04-30 12131, 2016
ruaok
:)
2016-04-30 12148, 2016
zas
We need to think about backups
2016-04-30 12101, 2016
ruaok
not coffee??
2016-04-30 12125, 2016
zas
NewHost may not have someone to rotate our USB drives ;)
2016-04-30 12135, 2016
ruaok
oh, yes that concept is dead.
2016-04-30 12105, 2016
ruaok
but, given that EACH machine has a massive bandwitdh budget, we take our backups machine and give it a public IP.
2016-04-30 12120, 2016
ruaok
then blast backups as fast as we can to google cloud or glacier.
2016-04-30 12125, 2016
ruaok
probably both.
2016-04-30 12134, 2016
ruaok
google cloud and then once a week to glacier as well
2016-04-30 12140, 2016
zas
Yup, may be a soluton
2016-04-30 12145, 2016
zas
solution*
2016-04-30 12155, 2016
zas
ok coffee ;)
2016-04-30 12155, 2016
ruaok
for super fast retrieval from google cloud, and archival on glacier.
2016-04-30 12158, 2016
ruaok
bai!
2016-04-30 12138, 2016
zas
About number of web servers... i did some rough calculation, i would like to redo with you
2016-04-30 12157, 2016
ruaok
ok.
2016-04-30 12110, 2016
ruaok
the good thing about the vrack is that we can start smaller and expand when we want to.
2016-04-30 12125, 2016
ruaok thinks scalability
2016-04-30 12143, 2016
ruaok
search, web servers and postgres can scale easily with the new setup.
2016-04-30 12106, 2016
ruaok
gateways, when done properly will have loads of capacity and will scale nicely on a 2 host setup for quite some time.
2016-04-30 12126, 2016
ruaok
we also need to think about what traffic we want to serve.
2016-04-30 12138, 2016
ruaok
I'm not interested in spending a lot of hosting resources on headphones.
2016-04-30 12149, 2016
ruaok
speaking of headphones, the new rate limiting needs to have per-app limits.
2016-04-30 12112, 2016
zas
Coffee ready.
2016-04-30 12128, 2016
zas
Ok, based on current incoming reqs
2016-04-30 12132, 2016
ruaok needs to pop out before the old-catalan-lady-brigade ddos'es every available veggie shop in the area
2016-04-30 12134, 2016
zas
we have 1k req/s incoming
2016-04-30 12129, 2016
zas
around 65% hit our web servers
2016-04-30 12156, 2016
zas
and 55-60% of those are rate limited (503s)
2016-04-30 12118, 2016
zas
so we actually return ~270 req/s as 20x
2016-04-30 12106, 2016
zas
so let's say we want to double that, and convert most 503s in 200s
2016-04-30 12126, 2016
zas
it will double load on web servers, so if we want to keep same load with same hardware it would mean having x2 web servers (we have 5) -> 10
2016-04-30 12107, 2016
zas
but new web servers will be more performant, likely at least 30%, -> 7
2016-04-30 12119, 2016
ruaok
considerably more, I would think.
2016-04-30 12141, 2016
ruaok
these machines were given to us in 2010 and were several years old.
