Server Upgrades: Meh, next week some time

News updates, status reports, and version information from the FC Team

Server Upgrades: Meh, next week some time

Postby weasel on Sun Aug 17, 2008 11:17 pm

We're having some more server problems. :cry:

The source of which you can see via my very scientific graph:

Untitled.jpg
Scientific Graph
Untitled.jpg (27.13 KiB) Viewed 3059 times


I hate to use the old expression, "victim of our own success," but it really applies here. We've gone from a teensy little indie-friendly webhost, to an enterprise-level webhost, to a dedicated server... And as of today, the fastest, beefiest, best possible server hardware we could get our hands on simply isn't up to the task of serving all your database requests. Sometime in the next few days we'll have setup a server cluster to run this darned game. And that shocks the hell out of me! :o

Seriously, when Pacifist launched this little game he was hoping he'd please a few people and be a casual amusement to people of like mind. Nobody thought it would explode so quickly, and so beautifully. Hell, we all thought about running the server off of one of our HOME boxes!

Every time we upgrade the server equipment, it isn't a tiny incremental upgrade - it's a huge beefy upgrade. We jumped from a shared host on a fast server, to a dedicated quad-core box with so many RAM chips it challenged a can of pringles. And you guys still broke it within a week! :shock:

Every day is setting new traffic records. Every day there are more emails pouring in with praise and support. Heck, take a look at this forum - the one I've sworn to protect and keep safe from trolls and jerks. With nearly two pages of new posts every day, it's busier than most game forums I frequent.. and I frequent a lot! You know how much work I've had to do? None. I locked one thread. That's it. This community is so polite, understanding, and fun-loving. It's great to see such a dedicated crowd gathered here, all congregating to share their fantastic contraptions.

And that's why it hurts so much to let you guys down. The save-game timeouts; the 12 hours of no server being up at all last week; the constantly slow connections and page-load times... It's really heartbreaking to all of us here on the FCrew. The last thing we want to do is let down our loyal fans.

I mean seriously, we look at some of the work out there. Check this out: S-Rank Hard Mode. It's a level designed in the "Level 21" contest thread. Check out the beauty and the work behind that. Those sweeping curves are actually stacks and stacks of squares, just angled so to look like a curve. Check out the decoration on that bad boy. The level has so many hours of effort it almost brings a tear to my eye. It is so heartbreaking to think that people are out there designing these things, and having the game crash on them at the last second.

And the worst part is, that's just one tiny example off the top of my head from the last 12 hours or so. Nevermind the reams of levels so fantastic, that I PM them to Pacifist all day long and we oogle over them.

Please, understand we want these problems to go away. We want everything to run SO smoothly for you guys. We are actively NOT advertsing and spreading the word for the game just to keep it stable for another week.

Bear with us, we will be through the storm soon! 8-) Thanks so much for your loyal support and not throwing your monitors out the window when things go down. We love you guys! :mrgreen:
User avatar
weasel
 
Posts: 2469
Joined: Sat Jul 05, 2008 4:03 pm

Re: Server Woes: ETA: A few days

Postby HexOxide on Mon Aug 18, 2008 3:34 am

Have you considered recruiting/hiring a professional/skilled DBA? From the sounds of it a lot of performance could be enhanced from tweaking the database itself as opposed to constantly beefing up the hardware?
HexOxide
 
Posts: 129
Joined: Fri Jul 25, 2008 4:10 am

Re: Server Woes: ETA: A few days

Postby onion on Mon Aug 18, 2008 5:38 am

SCIENCE!

I had figured all the database issues came from giant spike in users joining and contributing to FC...it's easy to see when you look at the fact that user-made level ID's were in the 1000's not too long ago, and now they're beyond 5000!

well I think you guys are doing a fantastic (lolpun) job at dealing with the database issues, and I salute you for giving all of us the opportunity to be able to play and contribute to this great game.

Keep up the great work, but don't forget to relax and have fun :>
User avatar
onion
 
Posts: 309
Joined: Mon Jul 28, 2008 9:45 am
Location: Chula Vista, CA

Re: Server Woes: ETA: A few days

Postby Twilightzero on Mon Aug 18, 2008 6:10 am

Just out of curiosity, who are you guys hosting with? I'll put in a shameless plug for Rackspace hosting, they're highly fantastic and they do have the 24x7x365 support it sounds like you need :D
Twilightzero
 
Posts: 206
Joined: Tue Aug 05, 2008 9:53 am
Location: Rochester, Hell, MN

Re: Server Woes: ETA: A few days

Postby pacifist on Mon Aug 18, 2008 8:06 am

HexOxide wrote:Have you considered recruiting/hiring a professional/skilled DBA?


Database changes, massive and small, have accompanied all stages of server upgrade as well.

But if you're volunteering (or know a guy) we can definitely talk. My email is on the credits page.

Twilightzero wrote:Just out of curiosity, who are you guys hosting with?


We are right now with WiredTree. We started bootlegging a friend's account. Now we've pretty much gone as far as we can with them. Their support really struggled to keep up with our growth. I guess for most applications it's ok for a server upgrade to take place a week after the request comes in. But for us a week is a lifetime.

We are almost definitely moving to Dexterity Networks in the next few days (expect some outages when the switch over happens). They aim themselves at the gaming industry specifically and have set us up a realy nice package. With them we have more room to grow.
User avatar
pacifist
Site Admin
 
Posts: 170
Joined: Fri Jul 04, 2008 8:38 pm

Re: Server Woes: ETA: A few days

Postby Twilightzero on Mon Aug 18, 2008 9:06 am

Cool, Dexterity seems to be pretty decent guys. Best of luck, I for one will stick with you through the upgrades as I'm too damn addicted to quit...:twisted:
Twilightzero
 
Posts: 206
Joined: Tue Aug 05, 2008 9:53 am
Location: Rochester, Hell, MN

Re: Server Woes: ETA: A few days

Postby marcelluss on Mon Aug 18, 2008 6:54 pm

I translated so all the addicts in here can understand:

Image

Can you beat the traffic?: http://FantasticContraption.com/?levelId=5669

Thanks to all who keep this excellent game online and work the forums.

~m
marcelluss
 
Posts: 4
Joined: Mon Aug 04, 2008 2:48 pm

Re: Server Woes: ETA: A few days

Postby onion on Mon Aug 18, 2008 7:49 pm

marcelluss wrote:Can you beat the traffic?: http://FantasticContraption.com/?levelId=5669


argh! pacifist's stress level is too difficult to overcome!
User avatar
onion
 
Posts: 309
Joined: Mon Jul 28, 2008 9:45 am
Location: Chula Vista, CA

Re: Server Woes: ETA: A few days

Postby NightStryke on Mon Aug 18, 2008 10:08 pm

onion wrote:
marcelluss wrote:Can you beat the traffic?: http://FantasticContraption.com/?levelId=5669


argh! pacifist's stress level is too difficult to overcome!

I'm sure he'd agree. :lol:


I cheated. :xd:

http://FantasticContraption.com/?designId=597611
User avatar
NightStryke
 
Posts: 623
Joined: Wed Jul 23, 2008 9:10 pm
Location: The Ether

Re: Server Woes: ETA: A few days

Postby weasel on Mon Aug 18, 2008 10:31 pm



oh god you killed our traffic! ;)
User avatar
weasel
 
Posts: 2469
Joined: Sat Jul 05, 2008 4:03 pm

Re: Server Woes: ETA: A few days

Postby NightStryke on Mon Aug 18, 2008 10:52 pm

weasel wrote:


oh god you killed our traffic! ;)

And yet pacifist is still stressed. Has he tried therapy?
User avatar
NightStryke
 
Posts: 623
Joined: Wed Jul 23, 2008 9:10 pm
Location: The Ether

Re: Server Woes: ETA: A few days

Postby lootsorrow on Mon Aug 18, 2008 11:00 pm

I think the best quick-fix to address people losing their work when the DB times out on saving is to make offline saving possible. A simple CRC scheme will prevent 'save-hacking' of the levels (hello script injection exploits! goodbye script injection exploits!). With offline saving, not only will people not lose their design in a crash, they'll be able to save much faster (which will let people save frequently and mitigate the OOPS I HIT BACK disasters), AND people can save their progress as they work without filling the actual game DB up with incomplete levels (also reduces traffic requests to the server). This would also make it easier to have a dedicated server for handling just upload requests (or if you're really SPIFFY AND COOL you can PRETEND to do this with MULTITHREADING! fellow programmers in the house cringe in unison). Just some thoughts.

Also, it might be nice if users could try to do their part by being a bit more selective about what they upload for the next little while. I freaking love marcellus's contraption-interpretation of the :pseudo: SCIENCE graph though.
User avatar
lootsorrow
 
Posts: 83
Joined: Sun Jul 27, 2008 7:19 pm

Re: Server Woes: ETA: A few days

Postby lootsorrow on Mon Aug 18, 2008 11:02 pm

NightStryke wrote:And yet pacifist is still stressed. Has he tried therapy?


Only the kind that comes in a bottle I'll bet :D

(relax, it's not offensive. all programmers are heavy drinkers and abusers of energy drinks. this is a well document fact)
User avatar
lootsorrow
 
Posts: 83
Joined: Sun Jul 27, 2008 7:19 pm

Re: Server Woes: ETA: A few days

Postby HexOxide on Tue Aug 19, 2008 7:18 am

Me? Volunteering? hah... hahah.. hahahahahahahaa... AHAHAHAHAHAHAHAHAHAHAHAHAHAHA! HAH! HAH HAH! HAAAAAAAAAAAAAAA..... Yeah.... NO. I hate databases, can't stand them, and avoid them when possible, no I like to code, I don't like to f**k(are we actually allowed to swear? I've been avoiding it just in case we're not.) with databases. Hooray for delegation! (And I'm not talking about programming delegates :D)
HexOxide
 
Posts: 129
Joined: Fri Jul 25, 2008 4:10 am

Re: Server Woes: ETA: A few days

Postby C4FF on Tue Aug 19, 2008 12:58 pm

I just wanted to get a chance to say thank you. It's not often an indie web game developer becomes an overnight success, so fantastic work so far regardless of downtime! I'm sure most people here would agree that in the long run its worth these small sacrifices. :cool:

You guys truly are an inspiration to all designers out there, you souly prove that it is still possible to smash the market straight in the face with an original idea and make it work! :woop:

If you guys need any help with forum moderation or website design feel free to give me a shout, id be happy to part with some time for you guys!

Once again, thank you all for the unimaginable amount of work you guys have put in over SUCH a small amount of time.

Regards.

Caff.
C4FF
 
Posts: 95
Joined: Tue Aug 12, 2008 5:50 pm
Location: London, UK

Re: Server Woes: ETA: A few days

Postby dillybar1 on Tue Aug 19, 2008 2:09 pm

Could you PLEASE make the cancel button work during a save, so we can abort when the server isn't responding? I know it's not your fault your game is successful, but if I have to re-build any more contraptions because everything froze up when I saved i'm going to pull my hair out.
dillybar1
 
Posts: 5
Joined: Tue Aug 12, 2008 2:41 pm

Re: Server Woes: ETA: A few days

Postby Twilightzero on Tue Aug 19, 2008 2:36 pm

dillybar1 wrote:Could you PLEASE make the cancel button work during a save, so we can abort when the server isn't responding? I know it's not your fault your game is successful, but if I have to re-build any more contraptions because everything froze up when I saved i'm going to pull my hair out.


When you're saving and it freezes up, just WAIT and it'll clear up in a few minutes. I know it's not the best solution in the world but it does work :D
Twilightzero
 
Posts: 206
Joined: Tue Aug 05, 2008 9:53 am
Location: Rochester, Hell, MN

Re: Server Woes: ETA: A few days

Postby NightStryke on Tue Aug 19, 2008 2:48 pm

Twilightzero wrote:When you're saving and it freezes up, just WAIT and it'll clear up in a few minutes. I know it's not the best solution in the world but it does work :D

Not if the server's gone down, though. You can wait for however long it takes to come back up, but it'll still be stuck.

But yes, the recent slow saving due to the need for server upgrades can be solved by just waiting for it.
User avatar
NightStryke
 
Posts: 623
Joined: Wed Jul 23, 2008 9:10 pm
Location: The Ether

Re: Server Woes: ETA: A few days

Postby Twilightzero on Tue Aug 19, 2008 3:12 pm

Yup just had a save bomb on me, returned a designID of 0... :(
Twilightzero
 
Posts: 206
Joined: Tue Aug 05, 2008 9:53 am
Location: Rochester, Hell, MN

Re: Server Woes: ETA: A few days

Postby MadConan on Tue Aug 19, 2008 3:39 pm

Twilightzero wrote:Yup just had a save bomb on me, returned a designID of 0... :(
ditto.
MadConan
 
Posts: 35
Joined: Mon Aug 04, 2008 3:01 pm

Next

Return to The Dev Dish



Who is online

Users browsing this forum: Google [Bot]

cron