Server backup

Server backup

For the discussion of infrastructure improvements and changes.

Moderator: Moderators

Post Reply
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Server backup

Post by abma »

Currently no backup is made of this server. Some scripts dump db contents to some dir on this server, but if hardware fails data is gone forever (wiki, forum, some custom stuff).

What needs a backup?

Imo:

- db (website, lobbyserver), files from website (images, attachments in wiki/forum/mantis, ...?!)
- rapid (versions.gz at least)
- things i forgot?!
dansan
Server Owner & Developer
Posts: 1203
Joined: 29 May 2010, 23:40

Re: Backup needed

Post by dansan »

Better backup entire system. That's the only way you'll be able to restore it in an acceptable time frame on another machine (or even the same one) if this one dies. It will also be good for your nerves and above all it makes sure, that nothing is forgotten. If someone else installs or stores something somewhere and forgets to add it to your list of things-to-backup, it'll be gone. If you backup everything, you'll catch it too. All you need is a blacklist of things you don't want.

I can provide some 100 GBs of SSH-accessible RAID1-storage on a 1gbit link, if you need it. You can gpg-encrypt everything before sending, if you like... or just a tar.bz2.gpg of the sensitive data, and rsync the rest.
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: Backup needed

Post by abma »

dansan wrote:Better backup entire system.
atm not possible, there is many stuff on this server which isn't related to spring :-(.

(still +1 for everything, but atm some exclusions have to be made)

i prefered whitlist / most important stuff as i don't have the bandwith.
User avatar
cheapsheep
Lobby Developer
Posts: 69
Joined: 31 Dec 2011, 16:42

Disaster Recovery Plan

Post by cheapsheep »

I have seen the scary topic about lack of backup, and the hours long interruption this night was quite bad for PR...

https://en.wikipedia.org/wiki/Disaster_recovery_plan

Please address the following risks:

- system suffers from hardware failure, all data is lost
- some angry noob decide to poke the server with a stick, all data is lost (or worse, silently corrupted...)
- company (the one renting the server, or one hosted on the server) suffers from a legal attack/bankruptcy/whatever, server is seized or "administratively unavailable"

How long does it take to recover?

Who is required for recovery?
- is access to systems shared by enough people, or only 1 unavailable person will make recovery near impossible? "access" being either technical or administrative
- is knowledge about server setup shared by enough people?
- is knowledge about server setup stored on the potentially failed system?
- in case of extreme/cascading failure... who has DNS control over the key domain(s) to re-establish another server elsewhere?
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: Backup needed

Post by abma »

(merged topics, please don't create a new thread for each post...)
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: Server backup

Post by abma »

fyi: backup script is currently running, adding exclusions to stuff thats not needed in the backup.
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: Disaster Recovery Plan

Post by abma »

cheapsheep wrote:I have seen the scary topic about lack of backup, and the hours long interruption this night was quite bad for PR...

https://en.wikipedia.org/wiki/Disaster_recovery_plan

Please address the following risks:

- system suffers from hardware failure, all data is lost
- some angry noob decide to poke the server with a stick, all data is lost (or worse, silently corrupted...)
- company (the one renting the server, or one hosted on the server) suffers from a legal attack/bankruptcy/whatever, server is seized or "administratively unavailable"
Backup is made to dansans server, so we should have some redundancy.
cheapsheep wrote: How long does it take to recover?
hard to say without testing. in theory it should run directly from the backupserver after fixing permissions. imo ~24 hours to get most important stuff up and running on a new server.

cheapsheep wrote: Who is required for recovery?
depends on what is broken. dansan for accessing the backup i guess and tim for fixing dns-entries.
cheapsheep wrote: - is access to systems shared by enough people, or only 1 unavailable person will make recovery near impossible? "access" being either technical or administrative
dansan and i have access to the backup-server.

cheapsheep wrote: - is knowledge about server setup shared by enough people?
mostly its no fancy setup. for the important stuff a ~/bin/run.sh (or some similar script) exists to start it.
cheapsheep wrote: - is knowledge about server setup stored on the potentially failed system?
depends on the service. for most services there exist docs in the github repository how to set it up.
cheapsheep wrote: - in case of extreme/cascading failure... who has DNS control over the key domain(s) to re-establish another server elsewhere?
afaik tim.

i guess we need some wiki page for this...
Post Reply

Return to “Infrastructure Development”