Continual problems with new site software
Thread Starter
Former Sponsor
Joined: Oct 2000
Posts: 9,081
Likes: 0
Continual problems with new site software
ERROR: Cannot find database s2ki
Is it continual hardware or software problems you are experiencing with the new site version?
In retrospect do you wish you'd stuck with vBulletin (the forum software of choice for just about every major site out there)?
Is it continual hardware or software problems you are experiencing with the new site version?
In retrospect do you wish you'd stuck with vBulletin (the forum software of choice for just about every major site out there)?
Administrator
Joined: Oct 2000
Posts: 20,274
Likes: 4
From: Toronto, Canada
The server that handles search crashed. the memory settings were set for a machine with 2GB of RAM and it only has 1.5GB. It's not a software issue. It's fixed now.
The newer software (it's not new anymore) doesn't cause any more issues than the old software did at this point. Yes, it was rough while we got the corners knocked off for the first couple months but at this point it's not any more problematic than any other.
If it's not one thing it's another. Let's see, this week...
- The main webserver's partition which stores web logs was found to be too small and running out of space causing Apache to croak. The logs were moved to a partition with more space.
- The scheduled task which rotates the logs on the main server (shark) wasn't running correctly causing the web logs to grow to over 2GB and apache croaked. Rebooting it solve the problem.
- Yogi synchronized changes from the development server to the production web servers which including in an in progress update to the ad software which broke payments and showed a half developed page those those who tried. He did this twice. Shit happens.
- Yogi also made changes to the skin to fix the really annoying quick reply which broke a different skin which is used for the Vendor Center and Group Buys and caused the topic list not to display. We figured out the problem and fixed it. Again, shit happens.
These things aren't yogi's fault. He does the most of the work on the software and therefore naturally makes the most screwups. Those are just the few I can think of off the top of my head. There have been a number of issues to deal with, some more than once. A simple oversight can take down the site or part of it. We have been trying to get updates out as quickly as possible and sometimes that causes problems especially when things aren't right.
There is validity to your post. I don't think incremental changes to the site should be made to the site in such an ad-hoc fashion. On the other hand when someone sends an email saying HELP!!!! I can't read my PMs!! we feel compelled to do something for that person right away and not tell them that we've fixed the issue but he has to wait till XX o'clock for the fix to be published. Sometimes, more often than not, when a problem is reported and a fix is in hand which would help someone out we will publish it immediately. On occasion we screw up, put a semicolon wrong or not test (or can't test) something as well as it should be, and there are bad consequences.
There is a price to be paid for quick turn-around and that's some blips. The price for fewer blips and bumps is longer turn-around. For example we could say we will only make changes once a day at midnight. That means that if you hit a problem then it won't be fixed until midnight at the earliest, go have a nap. If we did this then there could be a scheduled period of instability but it would be consistently timed.
A lot has gone on behind the scenes with respect to stability as well. Servers have been poked, prodded and tuned. We've installed and configured 2 different software packages in just the last week to better monitor the servers and alert us of problems. We have implemented an internal tracking system, knowledge-base if you will, to identify issues and record solutions and translate those to the monitoring software so we don't end up fixing the same problems over and over.
Do I wish we had stuck with the old software? Sometimes I do. I really didn't want or need all the issues of the past 6 months. If my only concern was the short-term then maybe. I think though that memory tends to be kinder that it should. The old software didn't work very well either if you recall. The search didn't work, that's not new. It was slow and unstable. You can see images in peoples signatures of the S2KI in unavailable page which turned up far more often that we care to remember. In a sense we traded the old problems for new ones but we also traded software which was at the end of the road and completely beyond repair for software which could be fixed and had a future as a platform, now and 2 years from now.
Historians can write what ever version of events sounds the best to them. I know in my own mind that the software we have now is better than the old v3 software ever was. It's getting better all the time and will continue to get better. I couldn't say that in March or April, it was all downhill. There are still people who think DOS was the best thing ever to be or has been. Those before them said the same about VAX/VMS or CP/M or even the VIC 20. The good ol' days huh?
The newer software (it's not new anymore) doesn't cause any more issues than the old software did at this point. Yes, it was rough while we got the corners knocked off for the first couple months but at this point it's not any more problematic than any other.
If it's not one thing it's another. Let's see, this week...
- The main webserver's partition which stores web logs was found to be too small and running out of space causing Apache to croak. The logs were moved to a partition with more space.
- The scheduled task which rotates the logs on the main server (shark) wasn't running correctly causing the web logs to grow to over 2GB and apache croaked. Rebooting it solve the problem.
- Yogi synchronized changes from the development server to the production web servers which including in an in progress update to the ad software which broke payments and showed a half developed page those those who tried. He did this twice. Shit happens.
- Yogi also made changes to the skin to fix the really annoying quick reply which broke a different skin which is used for the Vendor Center and Group Buys and caused the topic list not to display. We figured out the problem and fixed it. Again, shit happens.
These things aren't yogi's fault. He does the most of the work on the software and therefore naturally makes the most screwups. Those are just the few I can think of off the top of my head. There have been a number of issues to deal with, some more than once. A simple oversight can take down the site or part of it. We have been trying to get updates out as quickly as possible and sometimes that causes problems especially when things aren't right.
There is validity to your post. I don't think incremental changes to the site should be made to the site in such an ad-hoc fashion. On the other hand when someone sends an email saying HELP!!!! I can't read my PMs!! we feel compelled to do something for that person right away and not tell them that we've fixed the issue but he has to wait till XX o'clock for the fix to be published. Sometimes, more often than not, when a problem is reported and a fix is in hand which would help someone out we will publish it immediately. On occasion we screw up, put a semicolon wrong or not test (or can't test) something as well as it should be, and there are bad consequences.
There is a price to be paid for quick turn-around and that's some blips. The price for fewer blips and bumps is longer turn-around. For example we could say we will only make changes once a day at midnight. That means that if you hit a problem then it won't be fixed until midnight at the earliest, go have a nap. If we did this then there could be a scheduled period of instability but it would be consistently timed.
A lot has gone on behind the scenes with respect to stability as well. Servers have been poked, prodded and tuned. We've installed and configured 2 different software packages in just the last week to better monitor the servers and alert us of problems. We have implemented an internal tracking system, knowledge-base if you will, to identify issues and record solutions and translate those to the monitoring software so we don't end up fixing the same problems over and over.
Do I wish we had stuck with the old software? Sometimes I do. I really didn't want or need all the issues of the past 6 months. If my only concern was the short-term then maybe. I think though that memory tends to be kinder that it should. The old software didn't work very well either if you recall. The search didn't work, that's not new. It was slow and unstable. You can see images in peoples signatures of the S2KI in unavailable page which turned up far more often that we care to remember. In a sense we traded the old problems for new ones but we also traded software which was at the end of the road and completely beyond repair for software which could be fixed and had a future as a platform, now and 2 years from now.
Historians can write what ever version of events sounds the best to them. I know in my own mind that the software we have now is better than the old v3 software ever was. It's getting better all the time and will continue to get better. I couldn't say that in March or April, it was all downhill. There are still people who think DOS was the best thing ever to be or has been. Those before them said the same about VAX/VMS or CP/M or even the VIC 20. The good ol' days huh?
Administrator
Joined: Oct 2000
Posts: 20,274
Likes: 4
From: Toronto, Canada
to clarify I'm not implying that there aren't going to be other problems. If I were a betting man, and I am, I'd bet on the fact that sometime in the middle of dinner tonight some other damn thing will go down the crapper and I'll probably spend the better part of the evening trying to figure out what and trying to fix it. It happened yesterday and the day before and before...
Oh wait, people are showing up as other people. I'm ahead of schedule...
Oh wait, people are showing up as other people. I'm ahead of schedule...
Trending Topics
Registered User
Joined: Apr 2003
Posts: 2,693
Likes: 0
From: Boston Baby!
test here, Im logged in as my self, then it changes with the screen and I then am logged in as 2 other people! Yikes. Im not myself 
Second thought.....I needed a break from this body 
hmm lets see who I am with this post
Second thought.....I needed a break from this body hmm lets see who I am with this post
Joined: Nov 2001
Posts: 11,594
Likes: 7
From: Bumpass, VA
Cool...I'm still me 
. I like being me 
cthree...j/k aside, I have a question that would be perfectly related here I suppose. When things "get goofy" on the site, and I start noticing burps, long load times, and an eventual "page cannot be displayed"...what I also seem to catch at those times is info at the bottom of the particular webpage at that particular time indicating the % (I suppose of load) being put on a particular server hosting the particular page). So when I see 126%, etc., and "goofy stuff" going on...is that par for the course? And if so, what does that percentage mean? RAM overloaded? CPU? Just too many users hitting at the same time? All I know is when things get "un-goofy", I find that the percentage has dropped below 100% & the particular server seems to get all happy & non-goofy
.
Care to elaborate? Or point me to another thread if you've already poured over this particular explanation.
Now April...about taking that break, you're kidding, right?.
- Dave
. I like being me cthree...j/k aside, I have a question that would be perfectly related here I suppose. When things "get goofy" on the site, and I start noticing burps, long load times, and an eventual "page cannot be displayed"...what I also seem to catch at those times is info at the bottom of the particular webpage at that particular time indicating the % (I suppose of load) being put on a particular server hosting the particular page). So when I see 126%, etc., and "goofy stuff" going on...is that par for the course? And if so, what does that percentage mean? RAM overloaded? CPU? Just too many users hitting at the same time? All I know is when things get "un-goofy", I find that the percentage has dropped below 100% & the particular server seems to get all happy & non-goofy
.Care to elaborate? Or point me to another thread if you've already poured over this particular explanation.
Now April...about taking that break, you're kidding, right?
.- Dave
