Why is the site going down?

[Portland]

Perhaps the server was feeling very depressed?

[KlaxonCow]

What about a cron task to restart the the SQL server once a day?

Actually, if you try to browse the forums in the wee early hours of the morning, then it's often reporting an SQL error.

Specifically, that it can't connect to the UNIX domain socket (I don't recall the exact name but it's something like "/var/lib/mysql/mysql.sock").

This has, in fact, been happening most nights that I can see, but how long it stays down varies. So, sometimes, I'll only catch it for five minutes. Other times, it cab be an hour or so.

Then, every so often, it can last many hours - so much so, that it's still down when everyone else wakes up and looks at the forums, reporting the problem here.

But I've noticed this many more times than seems to be being reported here, as I'm often up at an early hour (basically, I've been getting up early in order to fit in a morning run before having to get ready for work).

So, actually, I think the problem is that there is already some sort of cron job that's firing most nights, which is taking the SQL server offline - perhaps for "database optimisation" or something - and this is a variable-length process.

And I suspect, if it is some sort of "optimisation / defragmentation" cron job to sort out and organise the day's posts in the database, then when it hits some of the stupidly big threads we've got on here - "hitting refresh", "youtube", "doctor who", etc. - then it's just taking so long to shuffle and optimise the data in those threads, that it ends up taking all morning or even all day to get the job done.

So rather than create a new cron job, I'd actually be checking if there's already one enabled somewhere - for phpBB or the SQL server in general - which is triggering a "wee early hours of the morning" optimisation / defragmentation operation to keep things tidy while we're all assumed to be sleeping.

Although, this is kind of tricky, because I don't know if it'd be entirely wise to just disable it. I mean, yes, that would prevent the outages but such a cron job would exist with good reason; a forum would be adding / changing / removing data to the database all the time, as we all use the forums, so a regular optimisation / clean up keeps it spritely and disabling such optimisation might just be trading the outages for overall worse performance all the time.

Already, browsing to the "hitting refresh", "youtube" and "doctor who" threads can take many seconds - where other threads are otherwise loading up straight away - so I've just got a feeling that those particular threads have got to a size that phpBB was never really designed to handle (these threads are, in some cases, well over a decade of regular postings, after all) and it's just straining and struggling under the weight.

(So maybe what might have to be done is splitting the "hitting refresh" / "doctor who" thread into smaller chunks and "archiving off" the earlier chunks - so they can still be preserved for posterity and nostalgia reading - but locking them, so that only the current "latest chunk" is actually active and the post count kept more reasonable at < 250, say.)

[zip accident]

Again.

Is there nothing that can be done?

Oops, didn't spot this page before posting.

And I suspect, if it is some sort of "optimisation / defragmentation" cron job to sort out and organise the day's posts in the database, then when it hits some of the stupidly big threads we've got on here - "hitting refresh", "youtube", "doctor who", etc. - then it's just taking so long to shuffle and optimise the data in those threads, that it ends up taking all morning or even all day to get the job done.

...

(So maybe what might have to be done is splitting the "hitting refresh" / "doctor who" thread into smaller chunks and "archiving off" the earlier chunks - so they can still be preserved for posterity and nostalgia reading - but locking them, so that only the current "latest chunk" is actually active and the post count kept more reasonable at < 250, say.)

Yesterday Yakyak was down for over 24 hours. The day concerned was a bank holiday, which suggests to me that the problem is one of physical access to the hardware, not software related at all.

On the other hand, Giant in the Playground forums limit all threads to fifty pages (of 30 posts per page) I don't know whether they're running the same version of the software or something completely different.

[gravy]

It's because the site is on a micro instance when really it should be on something bigger.

I'll have a look at the cron job as that would likely work fine.

[Baggypants]

I figured the forum was doing its own Passion play.

[gravy]

In terms of the amount of posts it does feel like lent

[Istar]

Still up and down like the proverbial. I’m a 30-year IT veteran and I’ve seen this pattern before - if we don’t do something about it, it’ll keep getting worse until either something breaks beyond repair, or there’s no one who can (or wants to) spend time keeping it limping along. And then it’ll be dead.

As I’ve said before, I know my way around AWS, relational databases, and general IT mechanics so if it’s a manpower shortage I’m available to assist (as indeed I guess are a number of others if asked).

[venusian]

And if it's a money thing, I'm happy to chip in a few quid.

[Misfit]

And if it's a money thing, I'm happy to chip in a few quid.

Yeah, me too.

Be nice if we could get that mobile skin sorted too...

[zip accident]

WTF is going on?

[Istar]

Yep, down again this morning.

[Istar]

Down again for a chunk of yesterday evening.

[merman]

There is also a lot of spam hitting phpBB boards right now, a new exploit to break Captchas is out there...

[zip accident]

Down again for a chunk of yesterday evening.

Afternoon and evening

[Portland]

Lot of boards being hit.