Playing with NetApp … After Rightsizing

It has been a tough week for me and that’s why I haven’t been writing much this week. So, right now, right after dinner, I am back on keyboard again, continuing where I have left off with NetApp’s usable capacity.

A blog and a half ago, I wrote about the journey of getting NetApp’s usable capacity and stopping up to the point of the disk capacity after rightsizing. We ended with the table below.

Manufacturer Marketing Capacity NetApp Rightsized Capacity
36GB 34.0/34.5GB*
72GB 68GB
144GB 136GB
300GB 272GB
600GB 560GB
1TB 847GB
2TB 1.69TB
3TB 2.48TB

* The size of 34.5GB was for the Fibre Channel Zone Checksum mechanism employed prior to ONTAP version 6.5 of 512 bytes per sector. After ONTAP 6.5, block checksum of 520 bytes per sector was employed for greater data integrity protection and resiliency.

At this stage, the next variable to consider is RAID group sizing. NetApp’s ONTAP employs 2 types of RAID level – RAID-4 and the default RAID-DP (a unique implementation of RAID-6, employing 2 dedicated disks as double parity).

Before all the physical hard disk drives (HDDs) are pooled into a logical construct called an aggregate (which is what ONTAP’s FlexVol is about), the HDDs are grouped into a RAID group. A RAID group is also a logical construct, in which it combines all HDDs into data or parity disks. The RAID group is the building block of the Aggregate.

So why a RAID group? Well, first of all, (although likely possible), it is not prudent to group a large number of HDDs into a single group with only 2 parity drives supporting the RAID. Even though one can maximize the allowable, aggregated capacity from the HDDs, the data reconstruction or data resilvering operation following a HDD failure (disks are supposed to fail once in a while, remember?) would very much slow the RAID operations to a trickle because of the large number of HDDs the operation has to address. Therefore, it is best to spread them out into multiple RAID groups with a recommended fixed number of HDDs per RAID group.

RAID group is important because it is used to balance a few considerations

  • Performance in recovery if there is a disk reconstruction or resilvering
  • Combined RAID performance and availability through a Mean Time Between Data Loss (MTBDL) formula

Different ONTAP versions (and also different disk types) have different number of HDDs to constitute a RAID group. For ONTAP 8.0.1, the table below are its recommendation.

So, given a large pool of HDDs, the NetApp storage administrator has to figure out the best layout and the optimal number of HDDs to get to the capacity he/she wants. And there is also a best practice to set aside 2 HDDs for a RAID-DP configuration with every 30 or so HDDs. Also, it is best practice to take the default recommended RAID group size most of the time.

I would presume that this is all getting very confusing, so let me show that with an example. Let’s use the common 2TB SATA HDD and let’s assume the customer has just bought a 100 HDDs FAS6000. From the table above, the default (and recommended) RAID group size is 14. The customer wants to have maximum usable capacity as well. In a step-by-step guide,

  1. Consider the hot sparing best practice. The customer wants to ensure that there will always be enough spares, so using the rule-of-thumb of 2 HDDs per 30 HDDs, 6 disks are set aside as hot spares. That leaves 94 HDDs from the initial 100 HDDs.
  2. There is a root volume, rootvol, and it is recommended to put this into an aggregate of its own so that it gets maximum performance and availability. To standardize, the storage administrator configures 3 HDDs as 1 RAID group to create the rootvol aggregate, aggr0. Even though the total capacity used by the rootvol is just a few hundred GBs, it is not recommended to place data into rootvol. Of course, this situation cannot be avoided in most of the FAS2000 series, where a smaller HDDs count are sold and implemented. With 3 HDDs used up as rootvol, the customer now has 91 HDDs.
  3. With 91 HDDs, and using the default RAID group size of 14, for the next aggregate of aggr1, the storage administrator can configure 6 x full RAID group of 14 HDDs (6 x 14 = 84) and 1 x partial RAID group of 7. (91/14 = 6 remainder 7). And 84 + 7 = 91 HDDs.
  4. RAID-DP requires 2 disks per RAID group to be used as parity disks. Since there are a total of 7 RAID groups from the 91 HDDs, 14 HDDs are parity disks, leaving 77 HDDs as data disks.

This is where the rightsized capacity comes back into play again. 77 x 2TB HDDs is really 77 x 1.69TB = 130.13TB from an initial of 100 x 2TB = 200TB.

If you intend to create more aggregates (in our example here, we have only 2 aggregates – aggr0 and aggr1), there will be more consideration for RAID group sizing and parity disks, further reducing the usable capacity.

This is just part 2 of our “Playing with NetApp Capacity” series. We have not arrived at the final usable capacity yet and I will further share that with you over the weekend.

About cfheoh

I am a technology blogger with 20 years of IT experience. I write heavily on technologies related to storage networking and data management because that is my area of interest and expertise. I introduce technologies with the objectives to get readers to *know the facts*, and use that knowledge to cut through the marketing hypes, FUD (fear, uncertainty and doubt) and other fancy stuff. Only then, there will be progress. I am involved in SNIA (Storage Networking Industry Association) and presently the Chairman of SNIA Malaysia. My day job is to run 2 companies - Storage Networking Academy and ZedFS Systems. Storage Networking Academy provides open-technology courses in storage networking (foundation to advanced) as well as Cloud Computing. We strives to ensure vendor-neutrality as much as we can in our offerings. At ZedFS Systems, we offer both storage infrastructure and software solution called Zed OpenStorage. We developed Zed OpenStorage based on the Solaris kernel, ZFS file system and DTrace storage analytics.

Posted on October 14, 2011, in NetApp, RAID, Reliability and tagged , , , , , , , . Bookmark the permalink. 5 Comments.

  1. the root volume is not put in its own aggregate for performance reasons. A 3-disk aggregate can’t provide much performance, and the root volume doesn’t need any performance anyways. The ONLY reason why it’s recommended to put it on a separate aggregate is because of the slight possibility that you have to runa WAFL_check on the system. Since the root volume (and thus the root aggregate) needs to be online for the system to boot, the WAFL_check has to run during bootup on the whole root aggr. This can take a while, and for older versions of OnTap on smaller systems could even panic the storage controller (the WAFL_check that is run on bootup is a limited version of the one available when the system is online). After the system has booted, all other aggregates can be checked concurrently and are brought online as soon as they have been checked. Thus, if you have a smaller root aggregate, you get your system back online quicker.

    NetApp recommends a separate root aggregate “for larger systems”, while they don’t say what “larger” means. We, for example, configure a separate aggr0 if the system has at least 48 disks


    • Hi Michael

      Sorry for the reply. This went into spam (Bad Akismet!)

      The balance between performance, WAFL_check and MTTDL will always be in play when designing the root volume for ONTAP. It pretty much depends on the configurations. In the initial configuration of smaller systems, the limited disk counts will require sysadmins to consider putting all the disks into one single root volume. Options of putting the root volumes into SSDs can help but NetApp has a quirky and inflexible way of selling their SSDs in their boxes.

      The larger configuration will give flexibility to house the root volume, but I wouldn’t go very large for a “larger” aggr0. It is best to use 1 RAID group with the default RAID group size for aggr0. For example if the RGS is 8, then we would just go with aggr0 of 8 disks. In terms of performance, availability, MTTDL, and recovery, this, in my point of view, is an ideal configuration. In real life, many factors determine how aggr0 should be configured and will not turn out to be so ideal over time.

      So, my 2cents worth, is that there isn’t going to be an ideal configuration for the root volume over time.

      Thanks for sharing.


  2. Interesting to read your article, but not sure I agree completely on your RG sizing (also isn’t the maximum aggregate size 100TB)?

    Ignoring the aggregate size for now, by having one RAID Group with only 7 disks won’t it lead to an uneven & inefficient aggregate? Wouldn’t it be better to have something along these lines (leaving the number of spares and root volumes as you have designed, so still 91 disks available):

    Basically, we could have 7 RAID groups of 13 disks. The aggregate is then perfectly distributed across the 7 RAID groups.

    The recommended practice for RAID Group sizing to have a RAID group size that is within the range of 12 (10+2) to 20 (18+2) and that achieves an even RAID group layout. If an even layout isn’t possible then the aggregate should not be defficient more than a number of drives equal to one less than the number of RAID Groups.(

    • Hello Andrew

      I thank you for your comment. Much of the things I wrote were based on my experience before ONTAP 7.3 and I realized that there are inaccuracies relative to the new ONTAP versions, in which I am in the midst of updating.

      ONTAP is always evolving and changing and your comments and insights are helping me and my readers to learn.

      Please visit rather than this WordPress blog and I appreciate more of your comments.

      Thank you

  1. Pingback: Playing with NetApp … final usable capacity « Storage Gaga

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: