Forum on New Server

[KlaxonCow]

Fair enough.

I thought it best to file the bug report, though, just in case something had all gone horribly wrong somewhere.

[gravy]

No, cheers man. It slipped my mind.

Just tried something else and I have something else to try before bed

[gravy]

Whole load of testing going on here

[Istar]

The storage I have is 10gb in EBS, that's persistent isn't it? I thought it was Just took a snapshot in case it's not

As for scaling I was thinking of having a much around with using nginx to serve the site; it looks like it's much lighter on RAM and processor load than Apache

If the instance is an EBS-backed type, then it depends on the 'delete on termination' setting of the EBS volume - it always used to default to 'on' (i.e. when the instance terminates it deletes its' storage tidily). That might have changed, in which case the storage will persist independently of the instance and can therefore be attached to another if the first dies.

Note: snapshots are ideally taken when the storage is quiesced (i.e. either the instance is stopped or the storage is detached from the instance) as although you can do hot snaps it's possible that in-flight updates might be caught in an inconsistent state. It all depends on what the IO load pattern of the storage is - if it's not the OS boot/swap volume and gets updates relatively infrequently (i.e. less than 1 transaction per second) it's probably OK to hot snap.

[Cloudane]

Say this really fast out loud, it helps with your testing

Tent
Tent
Tentacles

Barn
Barn
Barnacles

Test
Test
Test...

[gravy]

Testing a bit more
Post here please

[Little stoned kiwi]

Okay.

[gravy]

And that was the confirmation I needed, email notifications now working

[gravy]

And the favicon should be good now. You may need to do some cache clearing stuff to refresh it, depends on your browser

[gravy]

Hey Istar

This is a little worrying, my assumption is that if the instance is terminated I'll still have my EBS available to me do a recovery from.

Here's how I -think- it works, more than happy to be corrected

- I'm running an EC2 instance that boots from EBS backed storage
- I'm using no instance based storage
- The EBS is backed onto S3
- I can stop and restart the instance and it'll boot fine (haven't tried this yet)
- If I terminate the instance as opposed to stopping my EBS will still be available to me
- And I could re-attach it as a volume to a new instance
- And maybe even boot from it if the EBS isn't corrupt

I had a good look around for any configuration details on what happens to the EBS used by an instance on termination but couldn't find anything in the management console.

Any help greatfully recieved

[gravy]

Little follow on.

If that's all correct the best backup strategy would probably be some sort of automated process that will take the forum offline, get AWS to make a snapshot and then reattach it after?

[Istar]

Hey Istar

This is a little worrying, my assumption is that if the instance is terminated I'll still have my EBS available to me do a recovery from.

Here's how I -think- it works, more than happy to be corrected

- I'm running an EC2 instance that boots from EBS backed storage
- I'm using no instance based storage
- The EBS is backed onto S3
- I can stop and restart the instance and it'll boot fine (haven't tried this yet)
- If I terminate the instance as opposed to stopping my EBS will still be available to me
- And I could re-attach it as a volume to a new instance
- And maybe even boot from it if the EBS isn't corrupt

I had a good look around for any configuration details on what happens to the EBS used by an instance on termination but couldn't find anything in the management console.

Any help greatfully recieved

When the EBS-backed instance spins-up, it creates a single EBS volume (10GB sounds like a Linux instance - Windows usually gets at least a 30GB volume). That's the boot volume, attached as /dev/sda1. It has a setting called 'delete on termination', which always used to default to 'on'. If it is 'on', set it to 'off' (click the volume, look under 'actions' menu, change setting). Job done - if your instance terminates the EBS volume stays put and you can spin up another instance, stop it, detach and delete that new EBS volume, attach the old one as /dev/sda1, and boot from it. This is how my estate works - we have a suite of customised snapshots from which we create volumes, and attach these to instances replacing their default storage.

You can reboot, stop and start your instance at any time with no ill effect. If you terminate the instance, it's gone - and the EBS volume will be deleted or retained depending on the volumes' 'delete on termination' switch. Worried you might accidentally terminate the instance? Switch 'termination protection' on in the actions menu - it won't stop Amazon killing it if the host is b0rked, but you won't be able to terminate it until you turn that setting off again.

EBS volumes are in the EBS subsystem. This is not S3. When you taken a snapshot of a volume, that IS stored in S3 - but not in any bucket you can access from outside (it's a sort of 'hidden' bucket dedicated to EBS snapshots). You can create volumes from snapshots - this is how my estate backup process works (stop instance, snap EBS, start instance). Recovery from instance disaster is: create new instance, stop new instance, detach/delete EBS, create EBS from snap, attach EBS, start instance. This is also useful from cloning instances for testing - just spin-up a new instance and attach a copy of the EBS made from a snap of the original.

EC2 has no automated backup facility. There are a few products out there, including a couple of free ones, which offer varying degrees of functionality overlaying what AWS gives you in terms of scheduling, backups, etc. Suck and see, or just do it manually, or do what I did and write your own tool which uses the API to handle that sort of stuff.

EDIT: don't forget the two golden rules:

1. Backups. Don't be 'that guy' on the EC2 forums who has 25 years of financials locked solely in an instance that he now can't get into.
2. You are the sysadmin. Amazon cannot and will not help you deb0rk an instance you've misconfigured. Accidentally killed the network adapter by screwing with the config, or locked the VM into an endless reboot sequence, or damaged a MySQL database with an ill-judged script? Kiss the instance goodbye. See rule 1.

[gravy]

Cheers I think I get all that. The only issue I have is I can't find any reference to a setting about what to do with EBS store on termination. I have one instance and one EBS volume and below are pics of the actions menu for both of them.






Turned on protection against clumsy thumbed accidental termination

[gravy]

Okay. Reading up it doesn't look like Delete On Termination is a EBS related flag, it's set in the AMI you create your instance from and it doesn't look like you can alter that post spinning up the instance from the AMI. You need to edit the AMI you're using

As for backup? Daily routine that will stop the instance, create a snapshot and the restart it should work well. The backups are in S3. If the instance is hosed I can always go back a couple of backups to a working copy and if the filesystem isn't hosed I can still likely access the data on later backups.

The snapshots are only incremental in size and we're not adding much data.

[Istar]

Ah yes, the setting is on the AMI - I use AMIs so rarely I'd forgotten that.

So go through the spin-up process again, switching 'DoT' off. If it's already off, your current instance must have the same setting, so you're done - abort creation of this new instance, nothing to see here, move along.

If DoT is on, turn it off and continue spinning the new instance up. Once it's running, terminate it and observe its new volume not disappear.

Happy? Delete that new volume.

Stop the YY instance, snap the EBS, create a new instance with DoT off and Termination Protection on, stop it and switch the volumes so the YY volume is attached as the boot volume to the new instance (maybe make a copy from the snap and use that, just to play safe - if something goes wrong, your old instance is still there waiting to restart. This also proves to you that recovery from snapshots works).

Check the new instance is good, then assign the YY EIP to the new instance (you are using Elastic IP for the public address, right? On which note, don't forget you have to re-assign the EIP manually after a stop/start cycle when you do backups).

You now have a stopped, old YY instance, and a new running YY instance with DoT off. Once you're happy the new instance is working as expected and responding to web requests, tidy-up by terminating the old instance. If it had DoT on (which is why we're doing all this) it'll delete the volume too. The new one won't.