IPS Driver Error

[white90]

Understood it is unsatisfactory and Yes this is number one priority here to get sorted.

I have been on the phone again today to the suppliers, (at my cost in case anyone wonders)

they have been told in no uncertain terms that the issue is addressed asap or we ( the whole forum) will be moving home.

whereas it may seem simple looking in, the site has increased in traffic so we moved to dedicated server but too many issues have still occured

but rather than just move we are letting the suppliers with our assistance try to resolve the issues.

[Yostumpy]

Im getting problems onlots of sites, especially yahoo, ran a check and no prob with connection, so unplugged the router and plugged it back in and bingo! bin like this for days. Summitsup!!!!!!!!!!!!!

[Holyzeus]

.

I have been on the phone again today to the suppliers, (at my cost in case anyone wonders)

Shouldn't be at your expense in my view

[LR90]

Had hoped to ring Invision myself but after another 12hr day Tony had beaten me to it. This really is an issue we all want to get resolved.

For those that don't know the history our choice of IPB was somewhat accidental in the rush to get a new space set up but was poorly supported with our initial host. We switched to Invision, the authors of IPB, as it gives us a hosting supplier with (we hope!) knowledge both of hosting (apache, php and sql) plus they should know IPB better than anyone. The big advantage of this to the members is that as a forum we are not reliant on any one individual. So if I'm busy and Geoff is otherwise occupied Tony or any of the mods/admins can raise a ticket or cal Invision and get a patch put on, config changed, or whatever.

Having spent a day last week merging code bases for another forum to address a security issue I am very aware of the added 'costs' that would be incurred and risks if we just took any old hosting deal and stuck our copy of IPB on it. Another supplier, (any recommendations?) that offers dedicated hosting and managed IPB service would be an option (IPB is a perfectly capable piece of software, see the s2ki forum) and the migration would be relatively simple. Changing forum software would be far more complex but doable if push comes to shove though again we should find a supplier that doesn't just offer a Ubuntu host with ssh access but offers a fully managed forum hosting service. Pricing is unlikely to differ much from now but if they can maintain forum performance it might be worth the upheaval.

We have a lot of (IT) expertise on here but however well intentioned any of us are we are not structured/funded to ensure 24x7 specialized support which for a few dollars a month the service providers are. At the end of the day I feel this model serves the forum membership far better.

[lansalot]

I don't understand it really but from what I do understand we are paying for a dedicated server so there bluddy well shouldn't be anybody hammering it.

Out of interest, anyone know if this is actually the case ? I'd be very surprised if so.. Virtualization would be my guess, and not at the hardware level...

[Les Henson]

The latest from Invision (for those of you who know what the heck he's on about )

Hello,

There was something really wrong with your topic markers table.. I'm not sure what it was.. But I pruned it back a little bit.. Anyone who hasn't logged in for 6 months will probably notice a problem with their read posts being marked as unread.. But the way I figure.. If they haven't logged in for 6 months, they have no room to complain... They probably don't remember what they have read anyway.

But even after that.. It just wasn't working right.. So I backed the table up.. Which took about 10 minutes.. Far longer than it should have.. And then blew the table up.. Then restored it from the backup.. Which took about 10 seconds.. That's how long it should have taken to backup in the first place.. Now it seems to be working much better..

So, I think that was alot of the problem... BUT.. Those errors in your log file.. Something about those I don't like.. The subselect ones specifically..

I cleared those logs.. Let's let them build back up.. I want to see what we get AFTER I did the work.. That'll give us a better idea of where we stand.. I think we've likely solved the slowdown problem.. But maybe not ALL of the problems.. Those logs will likely pop back up, and there's something there that needs to be looked at.

Jason

IPS Support

Les.

[lansalot]

Can we see these error logs ?

Mentioning subselect errors sounds like badly written SQL code, which wouldn't affect how long the database took to backup (database could take a while to backup or drop, if some thread had a lock on it for example, but I'm not that familiar with mySQL). SQL either parses or it doesn't, so I'm curious as to what he means by subselect errors...

Clearing logs - yeah, I've done that occasionally (when I had something to hide.. )

[Cat_J]

Yeah that does seem a bit funny to wipe the log files after an attempted fix. Doesn't leave anything for comparison.

[Les Henson]

Can we see these error logs

I just copied and pasted - all that is gibberish to me. The only logs I'm familiar with are ones that come from chopping a tree up?

Les.

[Cat_J]

Do you have any update on this? It still seems to be dropping off now and then.

[geoffbeaumont]

Okay - update:

We've been doing some monitoring of the server, and to cut a long story short we need more memory to cope with the current site traffic. With the site ticking along nicely, fairly typical load and no problems we were using 88% of available memory this evening. That leaves no headroom at all, and will cause things to come grinding to a halt if there's a bit of a spike.

There may still be other issues to sort out after this, but first step is to solve the known problem.

editedit: seems like phpBB can import invision database - this would make an interesting exercise, I'd be willing to have a go if moderators want to contact me ?

So you're suggesting we ditch a feature rich, well written forum package with a decent (albeit not perfect) security record for a basic one with one of the worst security records out there and (last time I looked, which was admittedly a few years ago - I haven't bothered since) a really horrible code base...?

Sorry - I integrated phpbb with some custom code a few years ago for a government agency (don't think it ever saw the light of day, but that was political not technical), and it scarred me for life

I'd say there's something more substantial than a loading issue going on. My experience is that the site works in one of two states: perfectly, as it is at time of writing, all quick and zippy - and "not at all" (IPS driver errors, timeouts etc).

Nothing unusual about that - the IPS driver errors occur when the PHP scripts can't connect to the database. It's quite likely this is simply because the server has run out of memory and everything has ground to a halt. My usual example is a busy motorway - when it's close to capacity it'll flow along nicely until something happens - slow vehicle, breakdown, just a few more cars pushing it over the limit - and then it'll rapidly become a car park as no vehicle can move because of the vehicles in front of it.

There's not really a middle ground (site works but is pretty slow) that I'm observing, and this leads me to think there's some other factor that Invision aren't telling you about. Whether this might be because you're sharing resources in a virtualized environment and some other customer is giving it a good hammering, whether a shared mysql instance that someone else is performing some major operation on, whether site maintenance (eg backup) or whatever, can't say.

This is something that could be verified (to a degree) with some shell access and a bit of linux performance monitoring.

What I do think is that a fully-controlled system would likely be preferable. Black-box solutions are pretty good - when they work. And a nightmare when they don't.

The server isn't virtualised (or at least, it may be a virtual machine, as that's a common way to manage servers in a hosting environment, but it's on dedicated hardware - we have the machine to ourselves).

It isn't a black box - we have full admin shell access to the server and the database, and the forum software itself is unencrypted PHP so we have access to all the code too.

I think it's a networking/routing issue, I've used traceroute while it's down and it always fails or is delayed at the same IPSDirect.net router. If it'll help I can paste an output here next time it does it.

Interesting - you can always PM that to either Trev, Mark or myself. However, the common error we're getting (IPS driver errors sometimes followed by no response at all) is caused by the forum PHP scripts being unable to connect to the database.

[LR90]

Long and the short of it seems to be we're going to have to stump up more cash to increase our server memory from the meagre 1GB to something more appropriate to the forums level of usage. Something we had hoped to avoid as on a managed server with 24x7 staff coverage (coz us admins might prefer a night in the pub to waiting by the phone) will be rather more than a few chips from pcworld.

[BogMonster]

So the upshot of all that is that all you lot that are moaning had better get your wallets out because we need funds to pay for some more memory

That isn't entirely true (forum funds will meet the immediate need) but what is definitely true is that the server is leased, and the cost of leasing more memory is (or seems to me) exorbitant but may be necessary, hence the cost of hosting goes up and the cost of the forum as a whole goes up and we will therefore need more money coming in over the longer term. Such is the price of popularity.... rattle rattle

[Cat_J]

How many people do you get on at once at peak times?

[Les Henson]

Another 75 bucks per month for 4GB Ram

Les.

[Cat_J]

Would it not be sufficient to upgrade to 2? Or is it just not worth it?

[geoffbeaumont]

Would it not be sufficient to upgrade to 2? Or is it just not worth it?

2 might be sufficient for our immediate needs (but only just) but it wouldn't take much of an increase in traffic (or just a spike in popularity because someone linked to a thread on here from another forum) to bring the server to its knees again - it's much better to bite the bullet now and get enough to be sure we won't be back for another upgrade (with attendant downtime, forum unreliability in the run up and general hassle) in the near future.

PS. lancalot - sorry, just reread my replies above after a wrap on the knuckles PM from Trev. They weren't meant to sound negative! Really shouldn't post when I'm that tired and cranky

[Fatboy]

These conversations are so "beyond me"

Guys - I as always, really appreciate the work, time and effort you put into this forum. The Tech Forum is phenomenal - I just wish someone would take a P38 to Les for fixing or a Freelander TD4 for the Fuel Filter... - A job I'm dreading.

Sorry for OT, I've Paypal'd a donation to make up.

[Dunc]

So the upshot of all that is that all you lot that are moaning had better get your wallets out because we need funds to pay for some more memory

Bugger! More donationing time then is it? Wait until payday and i'll raid the paypal account again.

I imagine it'll only be a tenner again, but every little helps, eh?

I'll update my sig to read something like "there's a whole bunch of us help to keep this forum going and if every one of the 14,419 members did the same we'd own the sodding server let alone lease space on it!"

[V8CAMEL]

ok so the forum spends 75 $ per month extra. If that does not solve the issue is that money refundable?

[Hybrid_From_Hell]

OK

I'll answer this, cos as a complete' Server Mumpty' (My idea of a server is a nice lady bringing me a medium rare Steak Chips and Lager )

if I explain my understandings of the issue(s) then many like me here will understand....

So here goes

We are to a degree victims of our own success on LR4x4, the "Traffic" we get is such that the server at times cannot cope,

although the technicities of why what and how are waaaaay over my head the server due to this chucks its teddy out .........and down we go.

We are tiold by Invision, and also the view of the admins Trev / Geoff who also know about this sort of stuff,

is that the RAM we have in our server at 1GB is waaaaaaaaaaay underspec'd for the forum now and its traffic,

we are seeing 500, 600, even 800+++ at peak times, which is huge, .....some are members, .....some are Guests.....

(and some guests are members not logged in) .....and some are the search engines - which as has been explained to us is a good thing

as it moves us up the search engines and popularity ....and hence we grow more and so on.

But now with this traffic volume we need around 4GB, which means we need 3GB more (we have 1) and the 3 x more is another $75,

per month on top of the current bill - which with the dollar to Pound rate at the mo hurts more than say a year ago,

I also however I asked about UK severs, and amazingly what we have with Invision now,

even with the $ to £ rate is actually very good value for money at the spec and standards we have

So, we have asked for the upgrade,

This will cost more every month, if it solves it then yippe doo, if it doesn't then we can either have the RAM removed

and the costs drop, as we only pay while we use it - so 'no' to a refund as such.... but 'yes' to costs while we have it,

we have studied and looked at other options an arrived at this as a "Must Try" solution..

I think the basic logic here is that if this sorts it (and we all hope it will) then thats all we need, as everything else is fine,

but be under no doubt the entire A&M team have this as the absolute No 1 issue to fix, but we now need to try this as everyone

is saying that the traffic and server spec is just too far away from it being able to cope.

We will let you know when this is done, then we can all keep are eyes open, for hopefully NOTGHING to then report

Re the costsings as Boggy says we are in good financial shape, we are fortunate to have members regaularly sending in small donations and gifts,

we have a balance which will allow us to not only have the upgrade and stay with the additional costs (prov it works) and the forum is safe financially

for some time, .....but we will use up all the donations and balance 'faster' if we up the monthly spend, which is exactly what we are about to do

Thats why we have looked carefully at everything before committing to additional monies, but we have reached the crossroads now of action.. and all fingers point to RAM

Hope the above helps - as I said from a mumpties point a view

Nige

[BogMonster]

Yep every little helps, thank you

[lansalot]

I find it a little saucy that we are basically renting the extra memory per month - should be a one-off cost of upgrade, imo ! ;o)

[geoffbeaumont]

ok so the forum spends 75 $ per month extra. If that does not solve the issue is that money refundable?

As Nige said above, we can have it removed and stop paying the extra for it - however, even if it doesn't stop all the problems, we need it anyway to cope with current traffic levels.

I find it a little saucy that we are basically renting the extra memory per month - should be a one-off cost of upgrade, imo ! ;o)

We lease the server as a whole, increasing the spec of the server increases the charges. Much the same as would happen if you decide your leased repmobile isn't big enough any more and upgrade to a different model. You don't just pay a one off upgrade charge, you pay more each month for your car.

As with cars, the way to avoid this is to buy your own server. To do this we'd have to move away from Invision, though, who's support is generally pretty good, and we'd still have to lease space for the server in a data centre, along with bandwidth and a backup solution. That can easily add up to as much as we're paying now, for a remotely reliable service. As Nige pointed out, we're actually getting pretty good value for money.