optimization suggestions for springrts.com

[abma]

set /etc/default/mdadm:

Code: Select all

AUTOCHECK=false

(to disable monthly raid checks)


/etc/apache2/mods-enabled/fcgid.conf:

Code: Select all

MaxProcessCount 20

to reduce memory usage of php processes

[dansan]

/etc/apache2/mods-enabled/fcgid.conf

Why not use "libapache2-mod-php5"? It's _much_ faster at executing PHP. Security considerations?

The web server was completely down (at least) two times today...

[Tobi]

Because libapache2-mod-php5 isn't supported with mpm-worker (or did that change?), and with mpm-prefork we need many processes that include the whole php runtime, even though many apache requests are just static files.

So this way uses less memory, probably indeed at a speed cost.

(Though the proper solution would be to move static content to a different domain, and host it from a different (virtual) box with a web server optimized for static file serving, so the current apache can be optimized for php..)

[koshi]

Apache2 + mod_fastcgi + PHP-FPM ?
Would also make it possible to assign different worker pools (each with its own resource allocation) to seperate vhosts afaics.

[dansan]

My suggestions for the NextServer(TM):

* Use Xen or KVM for virtualization. VirtualBox does have a bad I/O behavior, as it seems to bypass the kernels I/O scheduler or does O_SYNC - both is autsch. With Xen or KVM there is no problem if one VM abuses the disk - the other VMs will also get their I/O bandwidth. Do not run anything in the host system (except SSH and management stuff). Pin a cpu-core to the host exclusively, so it can never be interrupted in its work by other VMs.
* Create a VM for each task: web server, database, lobby masta, build bot (already somewhere else?), more? Use a internal (private) network for inter-VM-communication. Maybe other separations can make more sense: Put MySQL and springrts.com together (low latency services) and put springfiles.com (I/O intensive) in an extra VM. Put mail service of all domains together in an extra VM.
* Do NOT allow for remote initialization of rsync - it takes control of CPU+I/O usage out of your hand, actually that's true for all kind of backups
* Replicate all data to secondary host(s) and allow people to pull backups from those
* Advanced stuff - only if necessary: Use master-master or master-slave MySQL replication and have the forums and different websites as well as the lobbyserver run on different hardware to distribute the load. Use multiple machines and prepare system for live migration of VMs.

I can help with / do the setup of the virtualization, VMs, disks, backup system, as I have experience with that.

Never did VM live migration and MySQL replication, but did read enough about it to be sure, that if needed I could implement it. I don't think it's necessary, as a modern server has enough power to handle all that I know about spring-infra (which is prob not very complete :).

The Achilles heel of root servers are their slow / lack of HDDs - that's where our brains must work.

[Licho]

But Xen cannot host windows..

And KVM was slower than virtualbox in my tests - mainly in IO.

I dont think that VBox skips kernel, it uses ordinary process to access disk and you can ionice it fine..

[Licho]

Tobi decreased worker count.
I think its safer to keep mdadm check Nobody knows when one of the disks starts failing..

Also per JJ request i increased php post and upload limits to 2GB ..

[dansan]

But Xen cannot host windows..

It can (for example http://wiki.xensource.com/xenwiki/HowTo ... sOnCentOS5).

And KVM was slower than virtualbox in my tests - mainly in IO.

That's strange... did you install paravirt-drivers (http://www.linux-kvm.org/page/WindowsGuestDrivers) in the Windows VM? Did you test it with the Windows-VM running alone, or under load? Did you test it with a Linux-VM - those will be most VMs, and the most demanding ones.

BTW: I prefer Xen much to KVM...

I dont think that VBox skips kernel, it uses ordinary process to access disk and you can ionice it fine..

Hmm... then it must be the O_DIRECT calls... I didn't measure it, but I always get the feeling starting a WinXP VM in VBox puts to much IO-load on my PC... but it's subjective...

Anyway: To efficiently use virtualization on a server with multiple VMs (almost all Linux) I suggest to use Xen or KVM. For management libvirt/virt-manager (http://virt-manager.et.redhat.com/) can be used.

[Licho]

Yes i tried with paravirt drivers and linux wasnt loaded.
VBox was faster even without paravirtual drivers .. dunno why..
VBox instead had some strange network issues and randomly freezed to death, but its fixed now.

This table suggests there are better alternatives to Xen in terms of performance..
http://en.wikipedia.org/wiki/Comparison ... l_machines

It claims it suffers disk slowdowns

[abma]

Also per JJ request i increased php post and upload limits to 2GB ..

this seems to require 2GB memory for such an upload:
http://stackoverflow.com/questions/5463 ... le-uploads

so better lower it back, and we will find an other solution... (maybe some cgi script or so...)

[dansan]

Yes i tried with paravirt drivers and linux wasnt loaded.
VBox was faster even without paravirtual drivers .. dunno why..
VBox instead had some strange network issues and randomly freezed to death, but its fixed now.

Did you use pre-allocated RAW disks?

Anyway: I guess exp. differs. I suggest to test on the actual hardware, as CPU/IO virt. extensions features, hardware drivers and chipsets differ. Probably that's the reason for our different experiences.

This table suggests there are better alternatives to Xen in terms of performance..
http://en.wikipedia.org/wiki/Comparison ... l_machines

Afaik higher performance is only archived by using solutions that only use 1 kernel for all VMs, like virtuozzo, containers and vserver do. They are the most efficient virtualizers, but for example do not allow to run Windows, BSD or fully-virtualized VMs - well they run just 1 kernel and separate namespaces for each VM - like an advanced chroot :)

It claims it suffers disk slowdowns

The claim needs citation.

My exp. is, that it archives near-host performance. On a dedi-server the hardware is to slow to properly test this. The biggest loss I encountered recently was on a server where the hosts RAID archived 600 MB/s and the KVM-VM (Linux) only 500 MB/s. But on my dedi it's ~80 MB/s in both host and VM (Xen/Linux). 10-20% inaccuracy in the tests are due to hosters using commodity HDDs.

VirtualBox cannot prioritize VMs, nor can it pin host CPUs to VMs. Is it possible to see accounting of CPU and I/O somewhere?

With virt-manager it's possible to manage VMS graphically like with VBox.

.... OK - I've read around the Internet a little, and the overall result was: For Windows-guests the CPU+IO performance winner is VMWare, followed very closely by VirtualBox, then KVM and Xen. For KVM the on-disk-format matters a lot. For Linux guests KVM and Xen are the winners in both CPU and IO. The biggest problem with all tests was, that they test only 1 VM at a time, while in reality what matters is the preformance with multiple VMs...

BTW: This explains our diff exp - I use on servers only Linux, and Windows only in VBox on my PC :)

Let's change the discussion to: 1st a concept, then the implementation (technique).

[koshi]

One of the biggest optimization opportunities imo: don't have 6+ people with root access that not always document/share their doings with each other.

[Licho]

Ok I will order next server at least 1 month ahead so we have time to test different virtualization options and check performance..
I would order it only for few months in case hw sux and then extend to full period if its ok

[dansan]

One of the biggest optimization opportunities imo: don't have 6+ people with root access that not always document/share their doings with each other.

Having multiple VMs and using groups and good sudo configurations could restrict admins to administer only the stuff they are supposed to administer.

Additionally a mailing list with archive / closed subforum and a wiki/etherpad/svn+$EDITOR for persistent documentation could help.

[hoijui]

the thing is, that we need multiple people for every part, cause nobody is around all the time, and sometimes stuff can not wait 2 weeks, till one guy comes back from holidays.

[koshi]

To be clear, I'm not saying having 6+ people with access is bad on itself. No close teamwork with little planning and ad-hoc measures that aren't documented/communicated are.

[Masure]

I heard "virtualization" several times here. I know licho uses it to run Windows applications on the linux server.

But why on earth a virtual machine per application (web server, database, ...) should be better than all the apps on the same machine ?

Operating systems are made to host multiple apps, using an OS per application sounds really odd guys.

Could you explain more about it ?

[dansan]

I heard "virtualization" several times here. I know licho uses it to run Windows applications on the linux server.

He runs a complete Windows OS in a VM.

But why on earth a virtual machine per application (web server, database, ...) should be better than all the apps on the same machine ?

Operating systems are made to host multiple apps, using an OS per application sounds really odd guys.

There can be various reasons for that:
* apps require incompatible library versions
* you need different OS
* you need same OS but different kernel
* you need 32 bit and 64 bit software
* it enables high(er) availability setups
-- you can migrate a running VM from one hardware to another (to fix/upgrade/modify the previous hardware or to distribute load)
-- if a hardware fails hard, the VM can _quickly_ be restarted on another host
* it is "easy" to scale horizontally and vertically (more hosts for a cluster or more CPU/mem/HDD for a host)
-- this scaling can often be done without rebooting
* kernel upgrade requires reboot - you want to stop only as few service as possible - thus rebooting one VM at a time
* migrating a VM from one hardware to another only requires to setup the host system, as the VM does not (need to) know about the real hardware
* security:
-- restrict admins to work only in the VM they need to
-- easy and fast backups with snapshots
-- cracking/infecting 1 VM does not affect the other systems (web servers are more or less open doors - don't put your mail server on them, maybe not even the database)
-- possibly stricter firewall configurations
* distributing hardware differently for different work loads: web servers need lots but slow CPUs, databases fast CPUs and small, low latency disks, file servers need few and slow CPU and big, high throughput disks

In the end the overhead of a extra Linux-VM is just around 16-64 MB and additional admin time - that's not relevant when you have 8192 MB. It pays off in terms of manageability, uptime and security.

[Licho]

I see VM main benefit as scalability .. it easy to swap VM to better one.. but..
there are lots of potential issues - lower performance, higher maintenance overhead, unforseen issues etc..

So yeah imo no reason to overspam those.. Frankly i think that current system with VM and one windows works pretty welll ..

But if there is virtualization option that gives us better performance and better resource sharing -> no reason not to switch to it.

[Licho]

I reverted upload limits back to 200M, jj said he would leave springrts.com server in few weeks and increased limit caused massive apache memory spam