optimization suggestions for springrts.com - Page 2

optimization suggestions for springrts.com

For the discussion of infrastructure improvements and changes.

Moderator: Moderators

User avatar
koshi
Lobby Developer
Posts: 1059
Joined: 14 Aug 2007, 16:15

Re: optimization suggestions for springrts.com

Post by koshi »

Is the windows vm is needed to run something besides the ZeroK site / planetwars and the springies?
Is renting two (smaller) boxes, one of which could then run windows as host OS, a possibility?
User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: optimization suggestions for springrts.com

Post by Licho »

It is a possibility but:

* its not as cost efficient. Hardware like CPU is mostly underused and its better to share it on same machine
* there is windows licensing issue. With OVH if you want windows on dedicated server you have to use one of their editions which is then leased at relatively high price. With VM you can use your own edition (in my case old but working win2k3)
User avatar
koshi
Lobby Developer
Posts: 1059
Joined: 14 Aug 2007, 16:15

Re: optimization suggestions for springrts.com

Post by koshi »

Licho wrote: there is windows licensing issue. With OVH if you want windows on dedicated server you have to use one of their editions which is then leased at relatively high price. With VM you can use your own edition (in my case old but working win2k3)
Never realized the additional per month cost cause it's not immediately visible on their site. The rate for the "smallest" win2008 server edition (whateverthefuck that exactly is) is 20E/month which looks like retail price/24.
koshi wrote:Is the windows vm is needed to run something besides the ZeroK site / planetwars and the springies?
And in broader terms, does anyone actually have a complete picture of what services/sites currently run?
Masure
Posts: 581
Joined: 30 Jan 2007, 15:23

Re: optimization suggestions for springrts.com

Post by Masure »

dansan wrote:There can be various reasons for that:
* apps require incompatible library versions
* you need different OS
* you need same OS but different kernel
* you need 32 bit and 64 bit software
* it enables high(er) availability setups
-- you can migrate a running VM from one hardware to another (to fix/upgrade/modify the previous hardware or to distribute load)
-- if a hardware fails hard, the VM can _quickly_ be restarted on another host
* it is "easy" to scale horizontally and vertically (more hosts for a cluster or more CPU/mem/HDD for a host)
-- this scaling can often be done without rebooting
* kernel upgrade requires reboot - you want to stop only as few service as possible - thus rebooting one VM at a time
* migrating a VM from one hardware to another only requires to setup the host system, as the VM does not (need to) know about the real hardware
* security:
-- restrict admins to work only in the VM they need to
-- easy and fast backups with snapshots
-- cracking/infecting 1 VM does not affect the other systems (web servers are more or less open doors - don't put your mail server on them, maybe not even the database)
-- possibly stricter firewall configurations
* distributing hardware differently for different work loads: web servers need lots but slow CPUs, databases fast CPUs and small, low latency disks, file servers need few and slow CPU and big, high throughput disks

In the end the overhead of a extra Linux-VM is just around 16-64 MB and additional admin time - that's not relevant when you have 8192 MB. It pays off in terms of manageability, uptime and security.
I know what you can do with virtualization as I know we have to host windows apps but I asked for performance concerns.

All your points are about compatibility, manageability, security, scalability. There is nothing about pure performance benefits from virtualized architecture no matter what the topic title says.

When I read 1 app per VM, I understood your point was not performance cause fully splitted ressources can never achieve better performances for a given hardware. You tell about application needs in very simple way and yet makes mistakes (databases require memory before disk).

Learn to tune an OS and its apps before playing with VMs because it's the last thing in vogue.
User avatar
hoijui
Former Engine Dev
Posts: 4344
Joined: 22 Sep 2007, 09:51

Re: optimization suggestions for springrts.com

Post by hoijui »

scalability is about performance.
you act as if the goal is, to maximize performance on a single set of HW, but that is not the case. it would be per cost, not per single set of HW, plus there are many other concerns then performance, namely.. everything that dansan listed.
Masure
Posts: 581
Joined: 30 Jan 2007, 15:23

Re: optimization suggestions for springrts.com

Post by Masure »

Scalability is one way to improve performance.

What I say is paying for huge misused hardware is bad. I have the per cost/performance in mind and it doesn't seem to be the goal when I read one app per VM.

Am I the only one against virtualization for virtualization ?
dansan
Server Owner & Developer
Posts: 1203
Joined: 29 May 2010, 23:40

Re: optimization suggestions for springrts.com

Post by dansan »

Well.. first of all work time (management, administration) and security is much more expensive than hardware. So the rest of this post is just to please you (no offense intended), not because I think single-host-performance is an issue. Performance must be good ofc, but other things are as or more important (manageability, availability and security).
Masure wrote:What I say is paying for huge misused hardware is bad. I have the per cost/performance in mind and it doesn't seem to be the goal when I read one app per VM.
dansan wrote:In the end the overhead of a extra Linux-VM is just around 16-64 MB [RAM]
BTW: 1 app / VM is in the case of limited resources a little bit extreme. As Linux and DB performance benefit a lot from free RAM, a compromise is the normal case: put some stuff together in a VM to save RAM to boost performance by having less disk-IO.

But back to why performance can even be better with more VMs:
* a hung process using all CPU/memory/disk on one VM cannot kill another VM
How important is this? --> The lobby server on Lichos server would not get killed sometimes by apache/backup overload --> higher performance and availability for the lobby server!
* Sometimes you have to lock entire databases to make a consistent backup. That really hurts performance as no write access is possible at that moment and reading is slow. If you replicate the DB to another VM and make the backup there --> no locking on the main DB --> higher performance and availability for the primary DB.
Masure wrote:Am I the only one against virtualization for virtualization ?
I made very clear, that anything that is not needed should be avoided - that includes OS virtualization.
Masure
Posts: 581
Joined: 30 Jan 2007, 15:23

Re: optimization suggestions for springrts.com

Post by Masure »

I understand 100% CPU usage by a process penalizes the others. I'm not a linux user but I assume Linux has some fine process prioritization configuration. That should avoid 1 single process killing the others.

Sorry for obviousness but why don't you focus on the hanging process ? An hanging process is not something that should happen and I would first dig hard into this instead of buying hardware and using virtualization to bypass the problem.
* Sometimes you have to lock entire databases to make a consistent backup. That really hurts performance as no write access is possible at that moment and reading is slow. If you replicate the DB to another VM and make the backup there --> no locking on the main DB --> higher performance and availability for the primary DB.
:lol: I don't wanna get into details but you won't achieve better backup consistency with VM replication. The problem is the same with a different backup target. If you're able to make a snapshot without lock to replicate your VM, you can do the same with database data.

No offense (really) but do you really know what you're talking about with backup consistency & locking concerns ? Giving virtual solutions doesn't solve real issues... Some database providers implement consistent and non blocking backups.
User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: optimization suggestions for springrts.com

Post by Licho »

Most databases allow non blocking backup or replication.

If not you can use shadow volume copy (in windows at least) to preserve current consistent disk state and copy it while allowing continuous operation.
Masure
Posts: 581
Joined: 30 Jan 2007, 15:23

Re: optimization suggestions for springrts.com

Post by Masure »

Licho wrote:Most databases allow non blocking backup or replication.

If not you can use shadow volume copy (in windows at least) to preserve current consistent disk state and copy it while allowing continuous operation.
That's what I said
dansan
Server Owner & Developer
Posts: 1203
Joined: 29 May 2010, 23:40

Re: optimization suggestions for springrts.com

Post by dansan »

Masure wrote:I understand 100% CPU usage by a process penalizes the others. I'm not a linux user but I assume Linux has some fine process prioritization configuration. That should avoid 1 single process killing the others.
A thing that happens when you have 1 process @100% is, that the latency climbs. Ofc the other processes will not get killed by that, but your desired quality level will maybe not be met anymore.

But that is still not scary... the biggest thread comes from processes eating your memory. First your server begins to swap - everything becomes slow, especially remotely logging in and administering. Then at some point the OOM-killer (Out Of Memory) becomes active and it's not a smart piece of software...

Another thing crazy software does is write GBs of log files - until the partition is full. Then suddenly lots of services begin to fail... some on purpose (for security considerations) others for being lazily programmed... in this situation data loss is common.
Masure wrote:Sorry for obviousness but why don't you focus on the hanging process ? An hanging process is not something that should happen and I would first dig hard into this instead of buying hardware and using virtualization to bypass the problem.
Ofc you should dig into this, but: It's already to late when it happens. And there is simply no software without bugs.

Administrators must be paranoiac, because everything has bugs, and everything that can go wrong will go wrong at some point.
Masure wrote:
dansan wrote: * Sometimes you have to lock entire databases to make a consistent backup. That really hurts performance as no write access is possible at that moment and reading is slow. If you replicate the DB to another VM and make the backup there --> no locking on the main DB --> higher performance and availability for the primary DB.
:lol: I don't wanna get into details but you won't achieve better backup consistency with VM replication. The problem is the same with a different backup target. If you're able to make a snapshot without lock to replicate your VM, you can do the same with database data.
Please read carefully - I don't want to snapshot or replicate the VM. I want to contiguously replicate the DB to another VM (cluster style). There the locking/backup can take place.
Masure wrote:Some database providers implement consistent and non blocking backups.
Exactly: some. If you come across a DB that does not (some storage engines in MySQL, OpenLDAP), then you need locking.
Also: What if your DB has a size of 50 GB? You think you can make a backup of this on your main server without heavy performance loss?
Masure wrote:Giving virtual solutions doesn't solve real issues...
Given you have no idea what you're talking about as you obviously have no real world admin experience, and then write a sentence like this, I guess you're a troll and so I stop talking to you.
Licho wrote:shadow volume copy
LVM can do that on Linux, but unfortunately it doesn't help with non-blocking, consistent DB backup.

I use it a lot to make backups of all running VMs: snapshot the VM-disk in the host, mount the snapshot, make backup.
User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: optimization suggestions for springrts.com

Post by Licho »

Server is still getting apache memleak!
Its not as severe as with SF, but it still means 3-4 apaches with 200mb ram each..

Why is it happening? What exactly was changed?
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: optimization suggestions for springrts.com

Post by abma »

i guess there is some mem-leak in php.
so lowering MaxRequestsPerProcess could help in /etc/apache2/mods-enabled/fcgi.conf

imo also disabling/removing some php-modules could fix this too.

( /etc/php5/cgi/conf.d )
dansan
Server Owner & Developer
Posts: 1203
Joined: 29 May 2010, 23:40

Re: optimization suggestions for springrts.com

Post by dansan »

I found this page: http://serverfault.com/questions/88997/ ... hp-based-w - maybe it helps...
They advise to lower MaxRequestsPerChild (relation to MaxRequestsPerProcess?) and to find the problem do strace - have fun with that :P
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: optimization suggestions for springrts.com

Post by abma »

found a solution that allows replacement of suPHP:

apache2-mpm-itk

seems to work at springfiles.com + its easy to setup. it also allows to user mod_php... this way you also can see, which user is using what resources. so exactly what you want! :-)
User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: optimization suggestions for springrts.com

Post by Licho »

Ok thats cool, i will try looking into it in a few months when i get more time (unless someone else will).

I also reduced max requests to 50. Does not seem to be helping much :(
Post Reply

Return to “Infrastructure Development”