Forum on New Server

[Istar]

Quite a few outages over the last few days - issues with EC2?

[fog]

It's been down more than it's been up the last week. Same with the forum.

[gravy]

FNARRR!

Just had a fiddle, let's see if it's a bit better

[fog]

Just had a fiddle, let's see if it's a bit better

Err no.

Now go and ask an adult for help.

[Deckard]

I blame Nigel Farrage.

[Firefox]

Yeah, importing all that mucky yurrupean bandwidth from Ireland!

Anyway, phpBB always seems solid, it's the database that seems to be having problems.

[gravy]

Okay I've had a proper much about and decreased the amount of memory mysql can use.

Let's see if that helps

[gravy]

Okay I've been less lazy and had a proper look. For those interested here's what was happening and here's what I did

The server is using nginx / mysql / php-fpm

php-fpm is pretty cool (http://php-fpm.org/) and makes thing run a bit faster. To do that it can use a bunch of memory and the other part of the equation is we're hosted on a low memory EC2 instance.

The problem was php-fpm reserving a bunch of memory pools over time and eventually causing the server to run out of memory. Luckily there's a lot of levers you can pull in the config file regarding process management and memory usage and I"ve changed a few to reduce memory load and force memory pool release over time. I've also set an upper limit on mysql's innodb buffer.

It'll probably work We'll see. If it doesn't then the options are (in order of less appealing)

* Fiddle some more with memory usage in the mysql / php-fpm modules
* Replace php-fpm with plain old php
* Pony up some more money so we upgrade from an EC2 micro instance

Hey you linux bods. Something the newer versions of OSX do is in memory compress pages of memory not recently used and decompress on demand. It works out pretty well and could be good for this kind of situation. Is there anything like that on Linux? Apple usually nick their best ideas from *nix

[gumboot]

As a back-up plan for the future, I suggest a web2.0 version of this embedded in the error page.

[gumboot]

And I assume you can swap to a file on a compressed filesystem on a ramdisk under Linux.

[gravy]

Well I wrote this back in the day

[gravy]

Just noticed something odd in the logs.

I had an old version of tapatalk still installed on the site and for some reason the forum is getting spams with POST requests to tapatalk API

There's a handful of different IP addresses doing but they're all EC2 instances. I've turned tapatalk off for bit as this seems to be what was causing php-fpm to chew up a load of resources.

I'll have a look into flood protection on the weekend

[gravy]

Yep, sorry tapatalk fans but I'm removing it for good.

The tapatalk team seem fairly responsive in fixing security exploits but there's been a pattern of new issues popping up regularly.

[Zoot]

Boo! (But I don't blame you really )

[Misfit]

Yep, sorry tapatalk fans but I'm removing it for good.

The tapatalk team seem fairly responsive in fixing security exploits but there's been a pattern of new issues popping up regularly.

Bugger. That'll cut down my forum activity then.

Just had to change device to see what was going on as currently computerless in Crete.