Forum on New Server

[gravy]

Wow. And now we're on a new instance where the EBS isn't deleted on termination.

That's amazing. Thanks Istar

Now I need to get the backup stuff working.

[Istar]

No worries - shout if you need a hand with anything, happy to help.

I don't have a recommendation for a backup manager as I use our bespoke tool which automates instance up/down activity, EIP re-allocation, and EBS snapshot management. I evaluated a bunch of tools about 18 months ago, but none quite matched our requirements list - though I seem to recall Ylastic was quite cheap (possibly had a free tier) and might suit a small estate like YY. The one that came closest for us was RightScale, but even that didn't tick all the boxes, and it's not cheap.

[gravy]

I'm going to try using snapshots as backup. I can take a snapshot while the system is running and the only really precious data is the mysql database.

So! The process will be:

- Chuck up YY Backin' Up holding page
- Turn off mysql
- Snapshot the current instance
- Turn on mysql
- Take down holding page

It looks like I can script that on the server fine

[Istar]

I'd just re-iterate that I don't recommend taking a hot snap of a running instances' boot volume - the OS is still going to be doing disk I/O, even if the primary services are stopped. It's unlikely but not impossible for the snap process to read the block store just as the OS is making a change, which could conceivably leave you with a snapshot that isn't completely atomic (or even bootable, worst-case).

Options:

1. Leave the data store on the boot volume, do hot snaps, keep fingers crossed.
2. Leave the data store on the boot volume, stop for snapshots, no risk.
3. Create another EBS volume and attach/mount as new drive, move transient data to that, take one cold snap of the new OS configuration (for instance recovery as already understood, then take hot snaps of the data volume after stopping services and making sure nothing is writing to it.

[zip accident]

Can we have a favicon now?

They're back.

Yay!

<edit>

(for forgetting to say the following the first time around)

Many Thanks gravy.

[gravy]

I'd just re-iterate that I don't recommend taking a hot snap of a running instances' boot volume - the OS is still going to be doing disk I/O, even if the primary services are stopped. It's unlikely but not impossible for the snap process to read the block store just as the OS is making a change, which could conceivably leave you with a snapshot that isn't completely atomic (or even bootable, worst-case).

Yeah I understand the concern but feel comfortable with the risk. I'm basing that on to spin up the AMI I have of YY and attach a non booting instance as another device and then migrate the data back. It doesn't matter if the EBS is bootable.

It is possible in that scenario for YY to be utterly screwed but as far as I can see this is what would have to happen:

- Node main instance is on terminates
- Somehow the instance EBS is toasted as well
- Everyone of the previous xxxx days snapshots of a working setup must have had write activity from the OS that invalidates the mysql directory and below
- And it can't be fixed on any of those snapshots

I'll get this working and then might have a look at moving to an instance where the mysql database is stored on EBS. Wouldn't need the instance to be booted from EBS either then. It'd make backing up a lot easier

[Istar]

Agreed. Just wanted to make sure you were cognisant of the risks.

Risk assessment and contingency planning only work when you know what the risks -are-.

[Barney]

Awww, yeah

I go away for a weekend and come back to a cow! Excellent, many thanks. My OCD is sated.

[Kelloggs]

If it helps, the way we do it in work is to have a custom AMI set up with the OS on and preconfigured, and an EBS solely with the database on; from elsewhere, a Python script snapshots the DB volume via the API every 15 minutes (and keeps three) and every midnight and noon (and keeps fourteen - i.e. a week). I've done it from elsewhere because - as I understand it - the API needs the Secret Key stuff wielding (which makes sense to me), but I'm not overly happy burning that into an AMI. This way, if the instance dies, we can spin up a new one and attach the old database to it, and if the database itself has a problem we can roll back up to an hour(ish) and try again.

BTW... AWS is cooool. Especially when you throw (e.g.) Puppet into the mix, and User Data, and get an instance that can come up and configure itself from a standard base OS. Then you stick in a demand-based auto-scaler, which spins up new instances (and passes User Data to them) and adds them into a load-balancer...

[gravy]

Hey that's good to read, it's basically the same kind of thing I was thinking of doing with keeping the mysql files on it's own EBS run by an instance that only has ephemeral storage. And yeah, I'm probably going to initiate the process from an outside server.

Not sure what language I want to script this in yet Just had a quick played aws-api front ends for node / coffeescript and Haskell. Clojure next I'll sort something.

[Istar]

BTW... AWS is cooool. Especially when you throw (e.g.) Puppet into the mix, and User Data, and get an instance that can come up and configure itself from a standard base OS. Then you stick in a demand-based auto-scaler, which spins up new instances (and passes User Data to them) and adds them into a load-balancer...

This.

It always makes me laugh when people disparage Cloud Computing. I instantly know the commentator neither understands it nor has any direct experience with it. As a Solution Architect, AWS has made it possible for me to do stuff in minutes that would otherwise have taken weeks to organise.

[Cloudane]

It's all greek to me, which is bad as I'm an IT technician. (And one with cloud in my name haha) I've stagnated in the same dead-end job for far too long...

I'll probably gain more of an interest in cloud computing when we get internet in the area with a decent upload speed. Thankfully, FTTC is imminent...

[gravy]

You should give the amazon stuff a go. If you register you can have a free micro instance on the go for a year

Found all of this absolutely fascinating

[Kelloggs]

I've been using them in work for about two years, after being quite cold about the whole idea, and more than a little cautious (we started with an in-house Eucalyptus cloud server, just to see what's what, and that's a good way to get very paranoid about persistence - we didn't have any) but I've got myself a t1.micro instance a few months ago, just so I've actually got "my own internet server" (no, the work fleet of 30 instances, 90 volumes / 128 snapshots and 7 load-balancers just aren't enough, why do you ask?) It's the free-for-a-year one, but I've just gone and calculated that, for a "play" server (one that's actually capable of quite a bit of playing) it should be just under £10 per month of run-time, if I don't go spewing spam everywhere or torrenting with it. Petrol is costing me just under £200 per month, and in comparison a t1.micro will probably get lost in the noise when it stops being free.

I think the main problem initially was that we'd tried virtualization before, on the VMware free setup, and found we just couldn't throw enough hardware at the thing to get any benefit over physical servers (and this was just as they switched from a Windows-native management console to a Java-based crashy one...) and I really didn't want to be pinning the servers' performance to a platform like that. What we actually got with EC2 was rather different; we're happily paying for computational capacity, where such is "guaranteed" for each instance (or, at least, added servers aren't fratricidal to the performance of what we're already running), and a lot easier to rely on because of that, even before you consider bells, whistles, comedy horns, custard-pie launchers and doors that fall off. So to speak.

[Colonius]

Yes this thread has got me keen to take a look too and that t1.micro looks ideal - thanks for writing about it gravy and to Istar for the war stories. Like Kellogs I'd had bad experiences with VMware where as the user of the service I found the shared clusters we were on were being loaded with more and more hosts and a lack of 'real' understanding from the infrastructure team on how to set the thing up for performance. I loved VMware for making our HA and BCP planning 100x easier than it was with dedicated boxes but the performance could be a real killer.

One area I need to understand better is how well 3rd party vendors have adjusted their pricing models to the cloud world. The print engine product in my old place had its pricing model very firmly fixed to the CPU capacity of the hardware (and not in a good flexible way with respect to virtualization). Something to look into when play times over though as that's not my job any more.