By Dr. Jim Anderson
Mizuho bank is one of the three largest banks in Japan. Forget everything that you may have heard about the earthquake, tsunami, and the radiation leaks in Japan - Mizuho has even bigger problems: they can't keep their IT systems up!
What Went Wrong At Mizuho
Mizuho Bank is in the trust business. People are willing to turn their money over to Mizuho and have them hold it for them because they believe that they can get it back anytime that they want it. Right now, that does not seem to be the case.
As I write this, Mizuho is dealing with an IT crisis of an unprecedented scope. Something has gone seriously wrong in their back office and they appear to be unable to complete requests to transfer funds.
This means that the unprocessed requests to transfer funds are starting to build up. In just a few days this backlog has grown to be over US$9.4B. For their customers this means that 620,000 salary payment transactions have not been processed. If you are one of the millions of Japanese who, like the rest of us, live from paycheck to paycheck, not getting paid on time is a very big deal indeed.
Where things currently stand is that Mizuho is saying that they have been able to process roughly 250,000 transactions. However, this leaves 910,000 unprocessed transactions and that number is only going to continue to grow.
The source of Mizuho's problems is currently unknown. Although Japan has recently experienced some dramatic natural and man-made events: an earthquake, a tsunami, and a damaged nuclear power plant, none of these events seems to be the root cause of Mizuho's problems.
Rather, a surge in deposit transactions at some of the bank's branches in Tokyo caused the bank's IT systems to become stressed and has led to the current situation. The bank reported that it was going to take at least 3 days to restore their IT systems and this was helped out by the lucky arrival of a 3-day Japanese holiday weekend.
Why This Is A Big Deal
The CIO at Mizuho has one single primary task: he is responsible for: making sure that the company's IT systems support the bank's daily operations. Although this is by no means a simple task, it should be fairly straight forward by now.
In the 21st Century all banks now rely on their IT systems as a sort of "nervous system" for the bank - they can't operate without their computer systems. Realizing this, the Mizuho CIO should have taken steps to ensure that the current situation would never happen.
Based on the information that is currently available, it appears as though the Mizuho systems have simply become overwhelmed. This could be due to a surge in volume - although that would seem to be unlikely since how many transactions could a single bank branch actually generate?
Rather, it's looking like there's been a failure somewhere in the Mizuho IT "supply chain" and this has caused the transaction processing process to slow down dramatically. This slowdown has allowed a backlog to develop and the problem has just grown from there.
Mizuho is now taking some of the right steps. They are going to have to shut down their network of ATM machines while they attempt to correct their IT problems. While that is happening, they are going to set up branches that will allow customers to withdraw money using their IT and cash cards.
Doing disaster planning and anticipating just exactly what can go wrong in a company's IT infrastructure is a key part of any CIO's job. Somehow this appears to have not happened at Mizuho. Once this crisis has been resolved, this needs to be job #1 for Mizuho's new CIO.
What All Of This Means For You
With a little luck, a CIOs day-to-day job is actually fairly boring. However, it sure looks like the CIO over at Mizuho has been anything but bored over the past few days. This can't be good news.
The bank is experiencing a system-wide breakdown in its IT processing systems. This has forced them to commit the ultimate violation of their customer's trust: they have not been able to dispense money. This is a clear case where an IT failure has harmed the company's overall reputation.
A CIO needs to ensure that this kind of company wide failure can never occur. However, since we can never be sure that all possible ways for a failure to occur have been prevented, we also need to ensure that there is a back-up plan in place that we can quickly execute. Let's learn what not to do from Mizuho's troubles!
What Went Wrong At Mizuho
Mizuho Bank is in the trust business. People are willing to turn their money over to Mizuho and have them hold it for them because they believe that they can get it back anytime that they want it. Right now, that does not seem to be the case.
As I write this, Mizuho is dealing with an IT crisis of an unprecedented scope. Something has gone seriously wrong in their back office and they appear to be unable to complete requests to transfer funds.
This means that the unprocessed requests to transfer funds are starting to build up. In just a few days this backlog has grown to be over US$9.4B. For their customers this means that 620,000 salary payment transactions have not been processed. If you are one of the millions of Japanese who, like the rest of us, live from paycheck to paycheck, not getting paid on time is a very big deal indeed.
Where things currently stand is that Mizuho is saying that they have been able to process roughly 250,000 transactions. However, this leaves 910,000 unprocessed transactions and that number is only going to continue to grow.
The source of Mizuho's problems is currently unknown. Although Japan has recently experienced some dramatic natural and man-made events: an earthquake, a tsunami, and a damaged nuclear power plant, none of these events seems to be the root cause of Mizuho's problems.
Rather, a surge in deposit transactions at some of the bank's branches in Tokyo caused the bank's IT systems to become stressed and has led to the current situation. The bank reported that it was going to take at least 3 days to restore their IT systems and this was helped out by the lucky arrival of a 3-day Japanese holiday weekend.
Why This Is A Big Deal
The CIO at Mizuho has one single primary task: he is responsible for: making sure that the company's IT systems support the bank's daily operations. Although this is by no means a simple task, it should be fairly straight forward by now.
In the 21st Century all banks now rely on their IT systems as a sort of "nervous system" for the bank - they can't operate without their computer systems. Realizing this, the Mizuho CIO should have taken steps to ensure that the current situation would never happen.
Based on the information that is currently available, it appears as though the Mizuho systems have simply become overwhelmed. This could be due to a surge in volume - although that would seem to be unlikely since how many transactions could a single bank branch actually generate?
Rather, it's looking like there's been a failure somewhere in the Mizuho IT "supply chain" and this has caused the transaction processing process to slow down dramatically. This slowdown has allowed a backlog to develop and the problem has just grown from there.
Mizuho is now taking some of the right steps. They are going to have to shut down their network of ATM machines while they attempt to correct their IT problems. While that is happening, they are going to set up branches that will allow customers to withdraw money using their IT and cash cards.
Doing disaster planning and anticipating just exactly what can go wrong in a company's IT infrastructure is a key part of any CIO's job. Somehow this appears to have not happened at Mizuho. Once this crisis has been resolved, this needs to be job #1 for Mizuho's new CIO.
What All Of This Means For You
With a little luck, a CIOs day-to-day job is actually fairly boring. However, it sure looks like the CIO over at Mizuho has been anything but bored over the past few days. This can't be good news.
The bank is experiencing a system-wide breakdown in its IT processing systems. This has forced them to commit the ultimate violation of their customer's trust: they have not been able to dispense money. This is a clear case where an IT failure has harmed the company's overall reputation.
A CIO needs to ensure that this kind of company wide failure can never occur. However, since we can never be sure that all possible ways for a failure to occur have been prevented, we also need to ensure that there is a back-up plan in place that we can quickly execute. Let's learn what not to do from Mizuho's troubles!
