First successful attempt was 2 days after image files upload, then index.json was apparently rewritten few times
2016-04-29 12047, 2016
bitmap
might be multiple index events for each image
2016-04-29 12019, 2016
zas
Ok, makes sense
2016-04-29 12007, 2016
zas
We were discussing a bit about the move to NewHost (check long backlog)
2016-04-29 12017, 2016
bitmap
ruaok: zas: about the CAA, it needs access to the same postgres DB as musicbrainz, so it'll have to be moved with it
2016-04-29 12008, 2016
zas
bitmap: ok
2016-04-29 12044, 2016
bitmap
for Jenkins/CI stuff...my only concern about google cloud stuff is how fast it is compared to hobbes, given that MB takes 19 minutes to build already :)
2016-04-29 12022, 2016
zas
Ok, this is quite something to take in account, we need fast machine for CI
2016-04-29 12040, 2016
zas
Does it need to be moved together the rest ?
2016-04-29 12056, 2016
bitmap
nope, that's all self-contained
2016-04-29 12008, 2016
zas
Ok, i'll update the doc
2016-04-29 12034, 2016
ruaok
Yeah, I figured that.
2016-04-29 12052, 2016
zas
bitmap: about redis vs memcached, i couldn't find matching ticket if it exists already
2016-04-29 12016, 2016
bitmap
I don't believe a ticket was filed
2016-04-29 12021, 2016
zas
the main idea is to get rid of memcached, and use redis + sentinel
2016-04-29 12034, 2016
bitmap
yea
2016-04-29 12015, 2016
bitmap
chirlu had something on his sandbox to do that. apparently it was pretty easy since the memcached/redis perl modules use the same cache protocol
2016-04-29 12036, 2016
bitmap
s/protocol/API/
2016-04-29 12039, 2016
zas
Yes, i remember that, perhaps chirlu could tell us more
2016-04-29 12014, 2016
zas
But on the general idea we agree that this move has to be done, can we expect to do it before the NewHost move ?
2016-04-29 12008, 2016
bitmap
we can make that a priority if it makes $NewHostMove easier
2016-04-29 12021, 2016
zas
I think it will
2016-04-29 12000, 2016
kanha has quit
2016-04-29 12006, 2016
zas
Since using Sentinel implies having more RO redis slaves, how mb server can benefit of this ?
2016-04-29 12037, 2016
bitmap
unclear, since I don't think cache latency is a bottleneck for us
2016-04-29 12005, 2016
bitmap
we could be caching more things than we are, though
2016-04-29 12031, 2016
zas
well, my main point is that we target multiplying number of requests per second by 10
2016-04-29 12026, 2016
zas
so it means more web servers (and more powerful), it wouldn't be wise to keep only one redis server ...
2016-04-29 12043, 2016
bitmap
right, makes sense
2016-04-29 12016, 2016
zas
We need to make the whole system far more scalable, because we target an increase of bandwidth by minimum a factor of 3 to start with, but i prefer to target a factor 10. NewHost move is the right time to rethink the whole infrastructure, identify weak points.
2016-04-29 12033, 2016
zas
Gateway server handles 800 reqs/s atm, that means 8k req/s at NewHost, and we are not sure it can handle it (it also means 10x log lines, 10x more disk writes, etc...)
2016-04-29 12049, 2016
zas
Same goes for redis server
2016-04-29 12011, 2016
zas
And any single server we have, like db
2016-04-29 12050, 2016
zas
New servers will faster, but not x10 faster
2016-04-29 12010, 2016
zas
Hmmm, gateway server is more like 1k req/s atm
2016-04-29 12034, 2016
zas
Each web server handles between 150 and 200 req/s
2016-04-29 12005, 2016
zas
And they are all at max reasonable load
2016-04-29 12013, 2016
zas
Also, the move to NewHost will be the right time to split web service and web site among different servers
2016-04-29 12037, 2016
zas
This is where wsN.mb.o would be a superior approach, we can use one gateway for ws, and one for the rest, and even use DNS RR
