Outage

The data center informs me that it's going to take more than a simple reboot.

At this point, I'm going to guess that it may be noon before everything is working again - but that's just a guess. I should have a better idea in a few hours, and I'll keep you updated.

Life is a beta...

There are pivot root errors on the primary drive. I don't expect you to know what that is; I had to google it myself. I suspect this was a hardware problem, but I don't know for sure yet.

In any case, I've elected to simply *replace* the primary drive. It's a lot of work, but better safe than sorry.

I tried to keep it quiet at the time, so that nobody would worry, but I got a severe MRSA infection (that's a drug-resistant staph infection on my legs and arms) last December, and it took until April before we really got must of a handle on it. I'm mostly recovered - there are a few spots of infection here and there - but my stamina has not yet returned. I've rebuilt an entire server in 24 hours before, but I've been up 23 hours at this point, and I need to get some sleep before I start on this job. I normally work at a fairly fast clip, but I want to do this slowly and carefully, because mistakes would be very unfortunate. I now suspect that the server will be back in operation about noon Sunday.

You obviously are wondering if your files are safe. I certainly believe they are. The D21C.COM accounts are on the primary drive, with DOMANIA.US accounts on a secondary drive, and all the accounts backed up on a tertiary drive, as daily, weekly, and monthly backups. The daily backups are current as of late Thursday night, so the only thing that wouldn't be present *twice* (or more) on the server would be changes that were made on Friday.

It'll take them a few hours to install a new drive, then install the operating system and systems software. I'm going to try to sleep right now, so I'm alert when they finish.

Life is a beta...

I guess,this includes me at web-user.net too Paul.

I get an error message when I go to my editor.

Thanks,

Bud aka web-user

Bud, you're having DNS problems. A lot of the work in rebuilding the server involves telling the server what to do, and then waiting on it to finish, so I'm trying to fix your problem along with rebuilding the server. You should be up late tonight.

Some D21C.COM files are appearing now, and some DOMANIA.US files will start appearing in the next hour. However, I have disabled the editing software at this time, because the underlying libraries aren't all in place, and it's possible that editing files right now could damage them. It's still going to be around noon tomorrow before D21C.COM and DOMANIA.US are really "back".

Life is a beta...

I've learned what caused the server to die; it was probably NOT due to attacks.

At this point, I believe all D21C.COM and DOMANIA.US files are being served. The server can do only so many things at once, and by spending server resources on this, it's slowing down the rebuilding process. However, some users have auctions running on ebaY, and this is critical for them. Since there is visibly no loss of files, that also is allaying people's fears, and I'm getting a lot fewer phone calls, which is speeding up the rebuilding process.

Blondie was getting sorta testy there for a while. Sorta like a "How many times do I have to tell you" feeling, although for the most part, it was a different person calling each time. I took her off phone duty and gave her a glass of wine and suggested she take a nap. I could use the same, but although the phone calls have slowed down, they haven't stopped.

The subdomains and the mail will be the last thing to come back up. In some cases, we will need to have you tell us your password in order to set up the subdomain and mail. If you want a head start on that, mail your password to support@domania.org with "My Password" in the subject line. Be sure to include your username, and whether it's d21c or domania.us.

Life is a beta...

just wanted to say thanks, paul, for informing us about the outage & keeping us apprised of the situation.. it's much appreciated.. you da man!

karen :)

I originally guessed that the server would be basically up at noon today. It's now 12:34 PM, so I have missed my deadline. There are always unexpected problems, but I haven't run into any insurmountable problems. I'm trying to work slowly and carefully, and that is taking more time than I anticipated.

The server will be unplugged tonight, presumably sometime after midnight, so the techs in the data center can remove some equipment I had temporarily installed. I'm not sure of the exact time, and I'm not sure of exactly how long it will take, but I estimate 15-30 minutes.

I still think you're going to be able to edit your website later this afternoon. There are a few subdomains set up, but they're set up wrong; I'm going to have to go through and fix them, and then set up the rest by hand. That could take me a couple of days. Mail won't work right until the subdomains are working right.

The phone stopped ringing long enough that I actually got about four hours of sleep last night; I'm feeling bushy-eyed and bright-tailed, (or something like that) and very optimistic.

The bad news: I bought some ribs at Costco before this all happened, and I was going to barbecue them yesterday. They won't last forever in the fridge, so Blondie is making them today, but in the oven. I have to say that's a disappointment. On the other hand, I don't suppose that matter much to anyone else. And if that's the worst news I have to share, I guess things are going well.

Life is a beta...

We are aware that the server is not, at the moment, serving files.

I think it's a minor configuration problem, but at the moment *I* am locked out of the server. I expect the techs at the data center will have it up and going in jig time.

Life is a beta...

Ouch. REALLY ouch.

I'm sorry, folks. You have NO idea how sorry I am. I was looking forward to zonking out, but that's going to have to wait....

Sometimes life is a doggoned ALPHA...

Thanks Paul for keeping us informed. All my pages are now available and stuff is working okay from my end. :)

As long as you're replacing that drive again, I told them, let's upgrade it.

Except that it's the weekend, and this being one of the largest datacenters in the world, there are a lot of people working there. When you have a lot of people, you end up developing policies, so that everybody does everything in a uniform manner.

This server dates to March 2005. That's positively ancient. Their automated ordering system doesn't list drives for that server, only for the servers they are currently selling. The theory is that it would just confuse their new customers; we're supposed to go through the sales department to get prices and place orders.

The sales department used to be around 8 AM to Midnight, seven days a week. They have a new guy running the company now, and now the sales department only works 9-9, weekdays.

There's another policy, too; the sales ticket has to be completed before the work order can be issued for an upgrade.

None of those policies are unreasonable by themselves, but you combine the three policies, and it means that it is impossible to get an upgrade during a weekend emergency.

I could have gotten a different server - but that would mean different IPs, and even if the server was running like a top, it'd take days for some people's ISPs to find that we were located at a different numeric address on the internet. (That's the biggest single reason we're still using a 2005 server.)

Finally, though, Janelle W. displayed an immense amount of courage, and bypassed the policies. I don't know how she came up with a price, and I don't know how she entered a sales ticket, but they'll be installing our new *improved* boot drive, bigger (which is nice) and faster (which is nicer) and more reliable (which is best of all).

So it looks like we'll be up Monday night after all.

Garrison Keilor says they've tried again and again, but even sex isn't as good as fresh sweet corn. I'm not sure where this ranks in there; possibly better than sex, not as good as sweet corn.

Or maybe it's like hitting your head against the wall, where it feels so good when you stop.

In any case, I'm dancing a jig, as well as this gimp can dance, and drinking a toast to Janelle W., a woman of courage and honor. She's my hero of the day.

Life is a beta...

Nothing new to report. Just dropping in to let you know there's nothing new to report.

Well, actually, things seem to be going slow, but that's probably a side-effect of frustration and fatigue. What I mean is that there have been no real disasters this time.

Life is a rerun...

I have two things to report, this time.

First off, friends of Rick56Fla already are aware of this, but Webbies trying to go to d21c.com see a page saying we were hacked. Since WebTV caches pages, ONLY Webbies can see that page. If computer users go to d21c.com, the browser will report that no page was returned.

In the process of restoring a site, I bring back the files, then install the protection for our site editing software, then install the password file that allows you to log in. There was a period of a few hours where the server got hung up on restoring mail messages, and I was so busy trying to unjam that process that I didn't install the protection for our site editing software right away. It was at *that* point that the front page for d21c.com was edited.

I guess I should be complimented that somebody *cares*. The fact is, it wasn't a hacker that took the server down, it was bad hardware.

But I'm not sure what else the vandal might have touched.

So the second thing I'm announcing is that I'm scrubbing that hard drive, and rebuilding the server once again, from the operating system up. I said late tonight, people would be able to edit their sties, and while I suspect that's still a good guess, "late" in this case is going to be late, late, late.

Life is sometimes a
mother (if you're a dog)...

I don't have the subdomains and mail working for everybody yet, but I think most of the site-editing capabilities are back at this point.

It's possible that I missed installing a library that's required for something functions. In that case, the software shouldn't do any damage to your site; it should instead simply refuse to function at all.

Let me know if you run across anything that's not working right, though.

Sometimes, life is just plugging away...

Thanx for all your hard work and labors over the past days.

Hope you can get some much needed and deserved rest now.....;)

Sunny

KATbrdi

Just re-read what Sunny said. That goes for me, too.

Thanks, and we hope you are doing some good R & R about now.

KAT/brdi

Paul, everything seemed ok this morning except that the color grabber was not working...unfortunately have not been able to log in for the past hour or so....please advise....thank you, Annemarie

Domania is NOT down, and HAS not been down since last night.
Both d21c.com and domania.us are up and working.

If you're using a computer, you're having a problem with DNS. Many ISPs have deliberately misconfigured their DNS boxes so as to save bandwidth. Unfortunately, that screws their customers.

Set your Domain Name Servers to 208.67.222.222 and 208.67.220.220, and you'll not only get *correct* domain name service, but you'll be able to surf a little faster, and you'll have added protection from phishing sites. These are the OpenDNS name servers.

There are instructions for a dozen different brands of routers or if you don't have a router, there are links to click on for Windows, Linux, mobile devices, game consoles - even Mac.

Life is a beta...

Thanks Paul for getting us back up and running, thanks to Blondie for her help, and thanks to Janelle for getting things done faster!

I could get in yesterday but last night couldn't, so I used the OpenDNS and it works. I'm back in and surfing faster. Thanks much for the info on that too.

Also thanks for the setup wizard in cpanel at AH; there's something I'm going to use later. Off to bed now.

BTW Hope you got your much deserved zonk out,

Life's a beta, then you alpha

Thanks Paul,
Just check and my account is up and running.

Have a good one.
Dave

Hi Paul,
At this time I can't get into my account..have Msntv2...

account:

Have a good one.
Dave