Service Outage Summary
What happened?
We’ve restored service to almost all the Intuit customer websites affected by this week’s outage. The disruption occurred during a routine maintenance procedure Tuesday evening. An accidental but severe power failure during that procedure affected our primary and backup systems, taking a number of Intuit’s websites and services offline.
We continue to look into the root of the problem to prevent a recurrence.
Why did it take so long?
We understand your frustration and are deeply sorry it took so long . We worked to restore our sites and services as quickly as possible while bringing services back in the safest, most reliable way for our customers. We understand the important role our services play in your Business And we take that responsibility very seriously.
We know you expect more from us, and we continue to look into the cause of the problem so that we can learn from this outage and prevent it from happening again.
Why didn’t you have backup?
We did. Unfortunately it too was impacted by the severe nature of the power outage. We’re still looking into this so we can prevent this in the future. As we learn more about what happened we will share it.
Was my data affected?
We have no evidence of a security breach or attack on our servers. And it this time, we do not believe there was any damage or loss to customer data.
What are you doing to compensate customers for business losses?
This outage has impacted your business and for that we genuinely apologize. In order for us to process any refunds…you’ll want to contact our trained support team. The only way we can help process these is through this support channel. Just log into the product and choose the way you want to contact support (call, schedule a call back, etc).
Again, the team at QuickBooks Online and the Intuit team apologize for the impact this has had on your business.
Explore posts in the same categories: Uncategorized
June 17, 2010 at 5:21 pm
Let me see if I understand this … ALL the servers (primary and backup) are housed in ONE Data Center? If not, then how did a power outage affect the 2nd Data Center? Are the 2 data centers so close that they are on the same power grid?
Why dont you have multiple data centers in different regions of the country that can act as backup for your primary data center?
What happens during a natural disaster, fire, earthquake, tornado?
I was led to believe that the backup servers were in a seperate data center — was I mislead?
Thanks
Rob
June 17, 2010 at 5:39 pm
We are certainly not a GM or an American Airlines but we are a small business that nevertheless depends upon our information systems available from QuickBooks Online.
Over the past 5 years we have been impressed with the reliability of information and availability of data. Your support is far beyond anything in recent memory and reminiscent of my early days in IBM.
You know…stuff happens! You throw all the resources you can at the problem and pray it is resolved quickly. If it takes some time you pray your clients understand. And if the problems are rare with limited impact your clients will certainly be more forgiving.
We wish you good luck, a speedy recovery and new and improved plans for future contingencies. We prize our work with you and look forward to many years of continued success.
Gary Smith
V.P. Broadway Bound
June 17, 2010 at 7:31 pm
For those of us that rely heavily on our accounting functions, “inconvenience” seems a bit cavalier. We were literally held hostage by this “inconvenience”. I guess my concern is what prevents this from happening again and to Rob’s point why did the backup go down at the same time.
June 17, 2010 at 9:50 pm
Intuit,
Thank you for correcting the issue and keeping us informed every step of the way. Yes, it did delay our work-productivity and our day-to-day administrative work at Macy Maid. There are not many companies out there that show they really make an effort in caring about a company’s data and online products and services, but you. As an owner for an IT company, I can see you kept us informed about the outage, restoration was in place, showed a duty of care and that you work closely with your clients to make sure they are whole again.
Thanks again,
Brian
Macy Maid, Inc.
June 17, 2010 at 10:14 pm
While we appreciate the apology, I am requesting a credit to both my accounts for the lost we incurred from not being able to finish financial reports that were due and the lost productivity for my staff person who was not able to perform her duties for two days.
June 17, 2010 at 10:31 pm
I guess no one can never say this is not going to happen again.
We as customers basically lost a day of work.
I feel that a free month or two of service from Intuit should suffice.
June 18, 2010 at 6:19 am
I appreciate you keeping us updated during the outage. I’m sorry this happened and I wish you luck in ensuring this does not happen again. Of course some things happen beyond our control. We must only learn something from it, if it is indeed within our control. We didn’t lose a day of work. We only went to OUR backup plan of using paper, manual invoicing, then met a mound of paperwork to catch up on the next day! EVERYONE should have a backup plan, not just Quickbooks Online!
June 18, 2010 at 7:58 am
We were unable to produce financial documents, and there may be possible monetary fines because of this.
I am looking to QuickBooks Online for reimbursement of these fines.
June 18, 2010 at 9:08 am
Thank you for your hard work in restoring the system as quickly as you could. I know that stuff happens and when working with computer systems anything can happen on a daily basis. I admit I was a bit annoyed at first but was able to work on other stuff around the office waiting for QBO to come back up. Finally I resorted to my back-up plan and typed my invoices on an old system that I had used in the past. I just had to enter all those invoices into QBO the following day when your system was working again. Yes it did add additional work to my already busy schedule, but I understand that stuff happens and it’s just called “LIFE”. I don’t understand why some people demand refunds or to be compensated for their inconvenience. How would these business owners like it if their customers demanded compensation from them when their system accidentally goes down for a day or two. I treat everyone the same way I would like to be treated when faced with the same situation. I wish everyone could do the same!!
June 18, 2010 at 1:50 pm
I totally agree with you. I also have a backup plan and I don’t wait until the last minute to take care of business (documents, taxes, billing, etc.) We should all, as business owners, have a backup plan so that one little glitch in our routine doesn’t throw us all into a pit where we demand money for our inconvenience. I totally love Quickbooks Online!
June 18, 2010 at 9:33 am
I understand that these types of things happen and also the presure everyone is under to get things back up and running.
I am concerned that the primary servers and the backup system are in 1 location. It was my understanding that backups were in a different location.
Please update me on your plan for backups if this happens in the future.
June 18, 2010 at 11:15 am
SOME STUFF=
-intuit server and backups are on the same building(that’s what intuit apology letter says basically)
-email notification for restored services took more to arrive than the outage itself.
-since a power outage cause this imagine what can happen if a natural disaster attacks intuit server and backup(only 1?), or if a hacker gain access to servers. then i guess is time for intuit to provide an option for download REAL BACKUPS and in case of a major event that ALL GET LOST then we can upload the REAL BACKUP and continue working.
-power outage or similar OK, we can live with that, but if data changes at least 0.01% THEN NOW WAY!! (this is based on CEO letter that as far as they are aware data was not touched)
-intuit online services (server,backups,security,continuity) are way bellow average
June 18, 2010 at 4:16 pm
To Intuit: Please provide the details of QBOEs datacenter locations. As many others have indicated, the communications thus far imply primary and backup servers are co-located. If this is correct, it is not acceptable. If incorrect, please clarify.
To all Readers: everyone is assuming that the “accidental power loss” was beyond Intuit’s control like a service interruption or outage by the utility company. The word “Accidental” alone doesn’t eleiminate the possibility that the power outage was caused by Intuit. The wording in the communications strikes a nice balance between specific and vague, but I just thought I’d point out that its very possible the power loss was caused by Intuit’s own actions based on the communications thus far. Forgive me if I missed a communication that was more specific about the power loss- I’m not sure I’ve read all of the numerous memos from Intuit on this topic.
June 19, 2010 at 4:48 pm
SATURDAY AFTERNOON, ANOTHER SERVICE OUTAGE, WHEN I CAN CATCH UP WITH THE MISSING DAY IF THE OUTAGES ARE SO FREQUENT????, THIS IS GETTING ME UPSET, HAVING THE IMPOTENCE OF EVEN CONCLUDING MY ACCOUNTING TASKS FROM THIS WEEK THURSDAY HAVING THIS ANOTHER OUTAGE
June 19, 2010 at 6:43 pm
It does indeed seem we have a new long, outage on Saturday. It is a now a few, but pushing several hours long so far. I have to assume the problem is not resolved, and Intuit does not yet have the redundancy needed the backup system to cover be 100% of possible down scenarios on the main data center. We’ll see what happens, but I am getting quite concerned, not just about down time issues but also about potential data loss. If they can have outages this long perhaps we have all (Intuit included) incorrectly assumed they have other issues covered properly. Our business is at their mercy and all I can do is hope there get these issues resolved.
June 19, 2010 at 10:25 pm
I am so very sorry that some of our services (QuickBooks Online, ProLine Tax Online & Merchant Services as well as others) have been affected by an outage. We’re working aggressively to resolve a hardware system issue and we’ve made some progress correcting the problem. We are beginning to bring some applications back online. However, there is still more work to be done. It continues to be all-hands on deck.
We have failed to live up to the high standard of dependability that we set for ourselves and for you. There is no excuse for the negative impact we’ve had your livelihoods this past week.
Apologies,
Heather Kirkby
heather_kirkby@intuit.com
Director Product Management and Marketing
June 20, 2010 at 7:30 am
QuickBooks came back online in the middle of the night. All our other services are back online too. I’m deeply sorry for the effect that our service outage had on you and your business. I know many of you were trying to catch up from our outage earlier in the week. We appreciate your patience in what has been, understandably, a very frustrating experience.
The Intuit team continues to monitor our systems closely. We are still looking into these outages to fully understand the root cause and what we can do to get better. Thank you for your patience.
Apologies,
Heather Kirkby
heather_kirkby@intuit.com
Director Product Management and Marketing, QuickBooks Online
June 20, 2010 at 12:05 pm
A quick update to let folks know that our support lines are open until 4pm PST today (Sunday 6/20). You can reach us at 1800 286 6800.
Heather Kirkby
heather_kirkby@intuit.com
Director Product Management and Marketing, QuickBooks Online
June 21, 2010 at 5:55 am
You say it was only a day, but I do a lot of my accounting at night when I return from a busy day. And, my staff access the time tracker at night as well. It really was 3 evenings that I could not track time or bill clients. It really hurt and delayed my revenue stream. As a small business, I think you can understand the importance of cash flow. In addition, I had to delay paying my employees because I could not tell how many hours they worked. This hurt quite a bit. What can we do to back up our own company so we can at least access our data if this happens again?
By the way, I’ve loved QB Online overall. It is wonderful to be able to access it from anywhere. I’m not giving up, but want a little more security.
June 21, 2010 at 8:52 pm
It is very plain that Intuit has there servers and backup servers in the same building and on the same power source. They have to be linked. If they were not they both would not go down at the same time for days at a time. Then go offline for another half day 2 days later. By not answering a very easy question about servers and backups tells us that they are not being up front with us. How does not giving us a straight answer about this jell with keeping us informed about what is going on and doing every thing they can to inform us. One thing i can’t stand for in a company is disingenuousness.
June 22, 2010 at 9:00 am
The question is not “does Intuit have all there servers in 1 location” but, after this disaster, “will Intuit continue to use 1 location for all their server(s) in the future”. If we give them the benefit of the doubt their mistake was likely that they had remote back up but do not remote fail over. Anybody in IT knows that It takes time and money to migrate a massive infrastructure like this to one that is designed for a remote failover (and can only be accomplished while the system is operational else you don’t have the time to do the job right).
If the answer is NO, perhaps there is a great opportunity for one of you IT visionaries out there.
Heather, Intuit needs to come clean, address the future, and stop apologizing for the past. The only real compensation for a mistake of this magnitude is the knowledge that the likelihood of this ever happening again are astronomically slim.
June 22, 2010 at 6:14 pm
As a small business owner I rely on QBO for all my business tasks all day long.I understand that things beyond our control happen and asking for compensation for a days loss of quickbooks is not fair to QBO as they happen in our own place of business, what if our customers ask for compensation what would you do? Please QBO look into other ideas so this might not happen again in the future and best of luck in the future to help us all who use your product.
June 23, 2010 at 9:19 am
Our ability to bill and receive payments was completely down. Luckily we have a backup system (freshbooks). This is the 3rd shutdown during regular business hours of quickbooks online this year. No longer a viable option. We do hosting and we have 5 minutes a year we can be down.