Recalculating Odds of RAID5 URE Failure
Alright, my normal RAID-5 caveats stand here. Pretty much every RAID level other than 0 is better than a single parity RAID, until RAID goes away. If you care about your data and speed, go with RAID-10. If you’re cheap, go with RAID-6. If you’re cheap and you’re on antique hardware, or if you just like arguing about bits, keep reading about RAID-5.
I got into an argument discussion with the Linux admin at work (who is actually my good friend, and if you can’t argue with your friends, who can you argue with?). What sucks is that he was right, and I was wrong. I think I hate being wrong more than, well, pretty much anything I can think of. So after proving to myself that I was wrong, I figured I’d write a blog entry to help other people who are in the same boat as me. Then you can get in arguments with your friends, but you’ll be right. And being right is way better.
So, first some groundwork. Hard drives usually write data and are then able to read it back. But not always. There is such a thing as an “Unrecoverable Read Error”, or URE. The likelihood of this happening is called the URE Rate, and is published in bits read. For instance, the super cheap hard drives that you get in your NewEgg or Microcenter ads probably have a URE rate of 1 in 10¹⁴ bits. As Google will tell us, that’s 12.5 terabytes.
That sounds like a lot of bytes until you realize that it’s 2014, and you can totally buy a 6TB hard drive
. So if you’ve got a home NAS (like this sweet looking 4-bay NAS
backing a media center, you have 24TB worth of hard drives sitting there. What a wonderful world we live in.
The popular argument by me until now, is that as your RAID array approaches the number of bits stated in the URE, you approach a virtual certainty that you will lose data during a rebuild. And that’s kind of true, but only after you go past the array size by a bit, and mathematically, it’s not what I thought it would look like at all. That’s what I’m here to explain.
Common sense says, “If I have a 12 TB array, and the URE rate for the drives in the array are 1 in 10¹⁴, then I’m almost guaranteed to experience a URE when reading all the data.”. The math tells the story differently, though.
The big assumption here is that a URE is an independent event, and I believe that to be the case. It’s like dice, rather than bingo. That is, if you have 65 dice, each with 65, and you roll them all at the same time, how likely is it that there is at least one die that rolled a 1? Around 63%, which I’ll explain shortly. Now, take 65 bingo balls, and pull 65 balls out of the basket. How likely is it that you pulled the #1 ball? 100%. See the difference? No one is saying, “ok, we’ve almost written the entire hard drive, time to throw in an error”. So the assumption is that it happens randomly, and that it happens at the rate published by the manufacturer. Those are the only numbers I have, so that’s what we’re going to use. (Also, if you have evidence that they do not happen randomly, and that they are, in fact, dependent on the amount of data written, then please comment with a link to the evidence. I’d love to see it!).
The equation that we care about is this:
Probability = 1-((X-1)/X)^R
where X is the number of outcomes, and R is the number of trials. So, if we roll a normal die (a D6) once, what are the odds that it comes up 6? The equation looks like this:
1-(5/6)¹
(That is, there are 5 sides that AREN’T the one we’re looking for, and we’re rolling it once). That equation comes out to 0.16666…, so there is a 16% chance that the number we roll is a 6. Since we’re only rolling once, an easier way is to say that we have a 1 in 6 chance, or 1/6, which also comes out to 0.16666… so the math checks out.
Lets complicate things by rolling the die 6 times. This is a simple enough experiment (and you probably have enough experience with Risk
that you intuitively know that your odds are not 1 in 1 — you KNOW there’s a chance that you won’t roll a six out of all 6 dice. SO lets use the equation:
1-(5/6)⁶
The only difference is that, instead of rolling it once, we’re rolling it six. Google will tell us that the odds of rolling at least one six is about 66.5%.
You should start having a creeping suspicion by this point that the likelihood of 10¹⁴ bits being read in a hard drive doesn’t mean that there’s a guaranteed failure, even if the rate is 1 in 10¹⁴, but the difference is probably still bigger than you imagine. Lets do the equation:
1-(((10¹⁴-1)/10¹⁴)¹⁰¹⁴
What we get is 0.63182…. This is actually very, very close to an interesting number: 1–1/e (e is Euler’s number, for the people who, like myself, sadly, are not mathematically inclined). As you keep increasing die sizes, from 6 to 6 billion, if you roll the die a number of times equal to the number of sides, the odds of a single side popping up get closer to 1–1/e. Weird, huh?
So, does that mean the odds of failure during a RAID array will never get worse than 63%? No way. Remember, that’s if the number of sides remain equal to the number of rolls. But as the RAID array size increases, the odds increase, too.
Lets look at the 4-bay NAS that I linked to, and the 6TB hard drives, because who wouldn’t want that? The scenario is that we have 4 6TB drives, and one of them has died. That means that when we replace the drive, we will have to read every bit on the other 3 6TB drives in order to calculate the parity. 12TB in 9*10¹³ bits. Lets plug that into the equation:
1-(((10¹⁴-1)/10¹⁴)^(9*10¹³)
which comes to 0.5931…, so 59% chance of failure. Still worse chances than a coin flip, but definitely not a certainty.
How about if we had a 6 bay array
? Well, during a drive failure, we’d be reading 5 drives worth, or 30 TB (if the drives are 6TB), and 30TB is 2.4e14. Lets try THAT equation:
90.9%. I’m not sure what your degree of certainty is, but I’d be getting nervous by this point. OK, well before this point. Another drive?
1-(((10¹⁴-1)/10¹⁴)^(2.88*10¹⁴)
94%. It doesn’t actually hit 99% until you get to 10 6TB drives. But really, how close do you need to go?
In conclusion, I just wanted to write about what I found, and that it was counter-intuitive, and I thought that the probabilities might help other people too. If you have questions, or if I did the math wrong, just comment below to let me know.
Also, I don’t know if they’re going to cover this, but at LISA this year, there is a a Statistics for Ops course and an R for SysAdmins course that you might check into, if you’re interested in this sort of thing. I’m definitely going to be in those classes. Hopefully I’ll see you there!
