#metabrainz

and we're going to buy bigger specs. I think 50% more is a fair guess.

so, keep at 5, i would say

for starters. knowing that we can order more at any time.

ok, but we don't want to stay on current level of number of requests, we actually want to use our bandwidth ;)

if we go for x3 bandwidth it would mean x3 reqs/s

I still think we should start with a smaller setup, study its performance and then grow it.

initial goal: 2x.

so we can start with 5 + 1 (to be safe, in case of failure of 1)

Can we put dbopm in the penalty box please? ruaok

and we are safe for a while

then observe for a while. and really understand how the load is working on the new setup.

He is being very rude to me.

discussing with bitmap, it seems it is reasonable to think we can improve mbs performance (software) by 20-30%

chrisskye: remind me about them. I keep seeing the emails, but haven't really followed in detail.

also i wonder how much of the current load is due to local disk logging

on web servers? very little I would imagine.

http://www.dbopm.com See the email I forwarded.

not that sure, but i could just stop logging on one web server for few minutes to verify it

I asked some questions regarding his project. Very defensive. Says you spoke to him, then didn’t reply, then you ruaok called him a ‘dumbass’ on IRC (unaware he is on here).

Asking someone to sign-up and what they are doing with our database, I believe is fair. We don’t ask all users to give $$. I do ask for confirmation of non-profit status or general details on the goals in order to determine fairness. -sigh-

sigh.

well, clearly this isn't going anywhere. putting him in the penality box is only going to make it worse.

I think a polite email asking him to stop using musicbrainz and not bother with the link might be the best way forward.

Also, if you are building a database of popular music...and somewhere link to us...is it not fair to ask how that metadata was put together? I mean, just for interest even...

Yes...but he included a veiled threat to blog about us in a negative way...

About db server, using SSD will speed up things by, in the very worst case, a factor x3 (and prolly up to x20 with recent SSD/kernel/postgresql) (based on old HDD/SSD comparison at http://it-blog.5amsolutions.com/2010/08/perform...)

And also our local network should be much faster

so start with 1 DB server one failover?

Yes, one master, one slave

Those machines have the biggest spec, they need fast CPU/fast and big RAM/fast and big SSD/fast network card

About search servers, we can just add another one

I wonder if we can start with two.

if we have SOLR in place.

because when not freaking out, they are at 20% CPU.

doubling would be 40%.

Still they are slow

I think SSDs might help

I think software is just damn inefficient

yes.

which is why I want to start small, observe and then grow.

I mean with vrack, adding a new machine should be near trivial.

Ok. so 2, with better specs than current ones, and a third one if not sufficient

perfect.

I think redis machines could be used for stats too

stats is cpu/disk intensive

and influx db has a relay now, so we can do HA

(storing collected metrics on 2 machines)

http://stats.metabrainz.org/dashboard/db/musicb... -> looks at the 503s at the beginning of each hour

we need to educate people to NOT do that.

metabrainz/picard#1333 (master - e30179a : Laurent Monin): The build passed.

Change view : https://github.com/metabrainz/picard/compare/65...

Build details : https://travis-ci.org/metabrainz/picard/builds/...

This is going to be the backdrop in my office. :)

speaking of which, time to head there and see if we have power and see if the net works, if we do have power.

👍

Pretty!

Not MeB though? :p

Also not done yet, reosarevok. :-)

Do you have THE POWER?

ruaok / zas, I still think you should consider using proxmox, it will allow you to officially utilise the servers, especially for something like the redis servers, as you can give cpu priority and assign memory

(meh, it's hard to pronounce stuff properly METAL-y when writing)

-officiall + efficiently

also with servers with a lot of memory you might not notice that big a difference in speed between ssd and hdd if most of you operations are read and the databse fits into ram

and if something crashes in 1 container it's not going to bring the whole server down

another benefit is the public ip of the container isn't the public ip of the actual server

so you could disable ssh completely to the public ip for example and only connect to it via the actual server then "pct enter 100" for example to enter the container

no power, reosarevok.

Aww

only some more clues for the next puzzle.

we contracted power a week ago.

over a week and it was supposed to take a week.

A Spanish week? :p

now, if this is anything like with the water, I need to go to the meter and turn it on. they didn't do that.

see, I don't know. :)

there is a door where the power meters might be, but none of the keys that were supposed to open everything open everything.

so, now we need to go back to the rental agency and figure out how to get access to that. or call the power company and figure out if they actually did turn shit on.

all I know is that after 2 hours, my fukkits are gone for the day.

I really want to try the net connection, but I have nothing that can produce 12V for any length of time.

any neighbour ?

you can also get an overall picture of your system: http://i.imgur.com/In0zBOi.png

zas: yeah, whenever I want to ask, no one is around.

You can easily get 12V from a bunch of 1.5V batteries

I'd need D cells for that to work well.

and I dont really care to spend that money on this

KodeStar: proxmox looks very interesting

zas I use it on my servers (though I haven't attempted a cluster) and it makes things so much easier, if a particular container is using too many resources you can either restrict the resources the container is allowed or move it to another node that has more spare capacity

KodeStar: do you use proxmox HA cluster ? it is said it requires 3 physical nodes at least

zas, no, I only have 2 servers currently

If we go for proxmox, i will set it up for HA cluster

so I run them as stand alone

it means I cant transfer between nodes without some down time, but that's ok for my needs

ruaok: the more i read about proxmox the more i think it would be absolutely great for us, as a far better catchall VM

good because that xen setup seems to fragile.

with 3 physical nodes we'll get a very solid solution, and we can provide sandboxes to our devs (as VMs)

our xen machine is a very weak point

you can also use kvm if you want to guarantee resources (similar to xen) or you can use lxc containers which are much more efficient

tbh i'd only ever use kvm vms if I needed to, like for a windows vm

lxc containers would be perfect for mbs dev envs

yeah

ruaok: i wonder about network links, ideally they have to be redundant, with no single point of failure

and they have to support IP multicast

not sure how OVH vRack play with this

many open questions

why do we need multicast?

internally, you mean?

zas, maybe with the ip load balancer?

corosync uses IP multicast

KodeStar: i mean between private network machines, ideally we should have 2 network cards, with 2 switchs to interconnect them, to prevent any serious failure of the whole system if a switch fails

hmm, you might struggle to get that from any of the budget providers

We aren't *that

on budget

\o/ https://github.com/LordSputnik/passport-musicbr...

Well, i mean the most stable and performant is our system, the more we can provide services to big players, so...

!m LordSputnik

You're doing good work, LordSputnik!

even without a budget it might be difficult, if you want 2 network cards between private network machines, must providers either have a single card for public network or 2 cards, 1 for public network and 1 for private

and by budget providers, one you go away from the big 3, you end up paying 2-3x he amount for worse specs

KodeStar: atm we have 3 (public + 2 privates) or 2 (private) network cards on most machines

lots of typos though, but typing with 1 hand and trying to calm a crying 6 month old with the other, lol

ahah, i know that ;)

zas, those are your own machines though, you can put what you like in them

ovh might allow some customisation though

you would have to ask them

vRack means we don't have any switch etc.. so the single point of failure is the network card (and those are quite reliable), so till we have redundant machines/services not much an issue

Our search servers are very stable, only 7 crashes within 3 hours...

sweet, you should talk to google about that, sure they would love to see howyou made it that stable

I mean, personally i'd try shooting for a crash every 20 minutes, but that's just me