Blog Archives
Enhanced Commodity Storage – Do You Believe in Magic?
With predictable regularity someone surfaces on the Web, claiming they have discovered a way to turn slow SATA arrays into high performance storage. Their method usually involves adding complex and sophisticated software to reallocate and optimize system resources. While there may a few circumstances where this might work, in reality it is usually just the opposite.
The problem with this concept is similar to the kit car world several decades ago. At the time, kit-build sports cars were all the rage. Automobile enthusiasts were intrigued by the idea of building a phenomenal sports car by mounting a sleek fiberglass body on the chassis of a humble Volkswagen Beetle. Done properly, the results were amazing! As long as their workmanship was good, the end results would rival the appearance of a Ferrari, Ford GT-40, or Lamborghini!
However, this grand illusion disappeared the minute its proud owner started the engine. Despite its stunning appearance, the kit car was still built on top of an anemic VW bug chassis, power train, and suspension!
Today we see a similar illusion being promoted by vendors claiming to offer “commodity storage” capable of delivering the same high performance as complex SAN and NAS systems. Overly enthusiastic suppliers push the virtues of cheap “commodity” storage arrays with amazing capabilities as a differentiator in this highly competitive market. The myth is perpetuated within the industry by a general lack of understanding of the underlying disk technology characteristics, and a desperate need to manage shrinking IT budgets, coupled with a growing demand for storage capacity.
According to this technical fantasy, underlying hardware limitations don’t count. In theory, if you simply run a bunch of complex software functions on the storage array controllers, you somehow repeal the laws of physics and get “something for nothing”.
That sounds appealing, but it unfortunately just doesn’t work that way. Like the kit car’s Achilles heel, hardware limitations of underlying disk technology govern the array’s capabilities, throughput, reliability, scalability, and price.
• Drive Latencies – the inherent latency incurred to move read/write heads and rotate disks until the appropriate sector address is available can vary significantly.
For example, comparing performance of a 300GB, 15K RPM SAS disk to a 3TB 7200 RPM SATA disk produces the following results:
• Controller Overhead – Masking SATA performance by adding processor capabilities may not be the answer either. Call it what you will – Controller, SP, NAS head, or something else. A storage controller is simply a dedicated server performing specialized storage operations. This means controllers can become overburdened by loading multiple sophisticated applications on them. More complex processes also means the controller consumes additional internal resources (memory, bandwidth, cache, I/O queues, etc.). As real-time capabilities like thin provisioning, automated tiering, deduplication and data compression applications are added, the array’s throughput will diminish.
• “Magic” Cache – This is another area where lots of smoke-and-mirrors can be found. Regardless of the marketing hype, cache is still governed by the laws of physics and has predictable characteristics. If you put a large amounts of cache in front of slow SATA disk, your systems will run really fast – as long as requested data is already located in cache. When it isn’t you must go out to slow SATA disk and utilize the same data retrieval process as every disk access. The same is true when cache is periodically flushed to disk to protect data integrity. Cache is a great tool that can significantly enhance the performance of a storage array. However, it is expensive, and will never act as a “black box” that somehow makes slow SATA disk perform like 15K RPM SAS disks.
• Other Differences – Additional differentiators between “commodity storage” and high performance storage include available I/Os per second, disk latency, RAID level selected, IOPS per GB capability, MTBF reliability, and the Bit Error Rate.
When citing the benefits of “tricked out” commodity storage, champions of this approach usually point to obscure white papers written by social media providers, universities, and research labs. These may serve as interesting reading, but seldom have much in common with production IT operations and “the real world”. Most Universities and research labs struggle with restricted funding, and must turn to highly creative (and sometimes unusual) methods to achieve specific functions from a less-than-optimal equipment. Large social media providers seldom suffer from budget constraints, but create non-standard solutions to meet highly specialized, stable, and predictable user scenarios. This may illustrate an interesting use of technology, but have little value for mainstream IT operations.
As with most things in life, “you can’t get something for nothing”, and the idea of somehow enhancing commodity storage to meet all enterprise data requirements is no exception.
Rumors of Fibre Channel’s Death are Greatly Exaggerated
If you believed all the media hype and vendor pontifcations three years ago, you would have thought for sure that Fibre Channel was teetering on the edge of oblivion. According to industry hype, 10Gbps Ethernet and the FCoE protocol were certain to be the demise of Fibre Channel. One Analyst even went so far as to state, “IP based storage networking technologies represent the future of storage”. Well as they say, “Don’t believe everything you read”.
In spite of a media blitz designed to convince everyone that Fibre Channel was going extinct, industry shipments and FC implementation by IT storage professionals continued to blossom. As 16Gbps Fibre Channel rapidly grew in acceptance, the excitement around 10GbE diminished. In a Dell’Oro Group report for 4Q12, fibre channel Director, switch, and adapter revenues surpassed $650 million, while FCoE champion Cisco suffered through soft quarterly results.
So what makes Fibre Channel network technology so resilient?
• Simplicity – FCP was designed with a singular purpose in mind, and does not have to contend with a complex protocol stack.
• Performance – a native 16Gbps FC port is 40% faster than a 10GbE network, and it too can be trunked to provide aggregate ISL bandwidth up to 128 Gbps.
• Low Latency – FC fabric is not penalized by the additional 2-hop latency imposed by routing data packets through a NAS server before it’s written to disk.
• Parity of Cost – The dramatic reduction in expense promised by FCoE has failed to materialize. The complexity and cost of pushing data at NN_Ghz is fairly consistent, regardless of what protocol it used.
• Efficiency – Having a Fibre Channel back-end network supports such capabilities as LAN-less backup technology, high speed data migration, block-level storage virtualization, and in-fabric encryption.
An excellent indicator that Fibre Channel is not falling from favor is Cisco’s recent announcement of their new 16Gbps MDS 9710 Multilayer Director and MultiService Fabric Switch. Cisco was a major proponent of 10GbE and the FCoE protocol, and failed to update their aging MDS 9500 family of Fibre Channel Directors and FC switches. (http://searchstorage.techtarget.com/news/2240182444/Cisco-FC-director-and-switch-moves-to-16-Gbps-new-chassis) This left Brocade with a lion’s share of a rapidly growing 16 Gbps Fibre Channel market. For Brocade, it produced a record quarter for FC switch revenues, while Cisco struggled with sagging sales.
Another influencing factor in FC longevity of is the average IT department’s need for extremely high-bandwidth storage network capabilities. Prior to 10GbE technology, Ethernet LANs performed quite well at 1GbE (or some trunked variation of 1GbE). The majority of the fibre channel world still depends upon 4Gbps FC, with 8Gbps technology recently starting to make significant inroads in the data center. Given the fairly leisurely pace of migration to higher performance for the SAN and NAS fabric technology. Except for a fairly small percent of IT departments that actually require high performance / high throughput, the lure of a faster interface alone has a limited amount of allure.
So which network technology will win? Who knows (or even cares)? There are usually bigger issues to overcome than what the back-end “plumbing” is made of. It’s far more important to implement the most appropriate technology for the task at hand. That could be Ethernet, Fibre Channel, Infiniband, or some other future network scheme. The key is to select your approach based on functionality and efficiency, not what is being hyped as “the next great thing” in the industry. In spite of all the hyperbole, Fibre Channel isn’t going away any time soon.
As Samuel Clemens (aka Mark Twain) said after hearing that his obituary had been published in the New York Journal, “The reports of my death are greatly exaggerated”.
Consultant, Contractor, or Staff Augmentation – Do You Know the Difference?
The world of IT is becoming remarkably complex, and companies grow increasingly reliant on outside knowledge and skills for assistance. But when you enter into really uncharted waters and need someone you can trust, who will you call? Unfortunately there are a lot of companies in the industry claiming to be technical experts for everything from “Big Data” to “Desktop Virtualization”. How do you identify the serious resources from the technical wannabe’s?
That is an interesting question. Every since the industry’s rush to identify and fix Y2K problems over a decade ago, the line between Consultant, Contractor, and Staff Augmentation has blurred. The recession of the past few years further masked the distinction between roles, since many laid-off IT employees simply re-branded themselves as “Independent Consultants” in an attempt to secure short-term project work.
So what are the differences between Staff Augmentation, Contractors, and Consultants? Let’s start with a definition. According to Wikipedia:
“Staff Augmentation is an outsourcing strategy which is used to staff a project and respond to business objectives. The technique consists of evaluating the existing staff and then determining which additional skills are required. One possible advantage of this approach is that it may leverage existing resources as well as utilize outsourced services and contract workers.”
“An Independent Contractor is a natural person, business, or corporation that provides goods or services to another entity under terms specified in a contract or within a verbal agreement. Unlike an employee, an independent contractor does not work regularly for an employer but works as and when required, during which time he or she may be subject to the Law of Agency. Independent contractors are usually paid on a freelance basis.”
“A Consultant (from Latin: consultare “to discuss”) is a professional who provides professional or expert advice in a particular area such as security (electronic or physical), management, accountancy, law (tax law, in particular), human resources, marketing (and public relations), finance, engineering, or any of many other specialized fields. A consultant is usually an expert or a professional in a specific field and has a wide knowledge of the subject matter.”
…Wikipedia Online Dictionary
Staff augmentation is based on the concept of a “faceless, replaceable skill” that is available for an entire category of labor (Administrator, Engineer, Programmer, Database Administrator, Web Designer, etc.). Since IT relies on a large labor pool of technical skills, these are relatively low priced roles. Since participants are required to have only prerequisite skills in their specialty and no other unique capabilities, they can be hired and released pretty much on demand. Rates are dictated by current market prices, and range from $35 – $95 per hour.
IT contractors are further up the scale in capabilities and value. They are typically companies that deliver a complete service or system to solve a clearly defined problem. This may be a particular operation, type of application, virtualized infrastructure, or network operation. In many instances it is delivered as a complete package, including hardware, software, utilities, installation, configuration, and testing. Contractor services may be purchased on a per-project or a time-and-materials basis and are consistent with similar projects. Bundled labor rates within a specified package or service are in the $125 to $185 per hour range.
At the top of the pyramid is IT consultant. This is a professional service offering highly developed skills and extensive experience in a specialized field. In addition to being a Subject Matter Expert for a particular technology or service, IT consultants typically have an extensive knowledge of related activities that include business operations, project management, associated technologies, industry best practices, quality assurance, security, and other operations. They are sought out by organizations for their comprehensive understanding of business-critical operations or other activity than can have industry-changing ramifications. Since these are highly specialized skills, they command rates from $225 to $450 per hour or more. Although consultants are expensive, they return value to the company that can far exceed their billable rate.
Clearly it’s in the client’s best interests to understand the differences and capabilities of each category. Unfortunately, these titles are frequently intermixed and tossed around somewhat indiscriminately by organizations. Unless due diligence is performed beforehand, occasionally some hapless company will think they landed a senior Consultant for $85 per hr. (plus expenses), when they actually contracted Staff Augmentation. This can quickly becomes the root cause of poor performance, lack-luster productivity, poor organization, missed objectives, and ultimately a failed project.
Technical personnel do not automatically become senior consultants just because that’s a label they’ve anointed themselves with. Buyer beware! Engaging the proper skill-set can either be a game-changer, or a “boat anchor” for the project.
FCoE? Thanks, but No Thanks!
I may be a bit “slow on the uptake”, but I’m struggling to understand industry claims that FCoE (Fibre Channel over Ethernet) is superior to having storage traffic sent over Fibre Channel. As a 34-year IT industry veteran and SAN storage specialist, it is my belief the only thing Ethernet data communications and SAN fabric transmissions may have in common is the label “network”. Therefore I’m puzzled why anyone feels “Unified Computing” is a more desirable solution for either Ethernet or SAN traffic. (Other than vendors who want you to buy their FCoE products.)
For the past couple of years we’ve been flooded with claims that “Unified Computing” (A.K.A. – Fibre Channel over Ethernet, or FCoE) is superior to separate Ethernet and SAN fabric networks. Webcasts and the trade press are awash with comments about the benefits and advantages of this new technology. If you believe everything you read, then FCoE should simply be sweeping the industry, making segregated Ethernet and SAN fabric channels a thing of the past. It’s not.
But will it? When I examined some of the claims in greater detail, they just don’t add up. The following is a matrix of popular “benefits” presented for FCoE, and my corresponding response as to why I question the validity of their claims.
| Claimed Benefit | Response |
| Reduces the number of adapters and cables that are deployed | On the surface this sounds logical, but it really doesn’t make much sense if you think about it. If a network (LAN or SAN) is designed for 30% average throughput with spikes of up to 70%, then it will still need (2) cables to support the configuration (70% + 70% = 140% of a single cable’s capacity). Unless your system is relatively small and/or the network is seriously underutilized, multiple cables will still be required. In addition FCoE will require some type of Quality-of-Service utility to ensure one service will not “starve” another, adding both additional complexity and greater expense. |
| Higher performance from 10Gbps network | This is also a compelling argument if performance is compared to 4Gbps Fibre Channel. But why, when 8Gbps FC is the current standard? Due to its more efficient protocol, 8Gbps performance is very similar to that of 10Gbps Ethernet. More significantly, now that 16Gbps Fibre Channel is shipping FCoE over 10Gbps Ethernet is the technology playing “catch up” now. |
| 40Gbps and 100Gbps Ethernet interfaces are coming | This is a meaningless claim unless you’re doing extreme computing. 8Gbps Fibre Channel has been shipping for a couple of years, yet it is still being adopted at a leisurely pace. If there is no rush to upgrade from 4Gbps to 8Gbps FC (a 100% increase), why then will there be a rush to deploy 40Gbps Ethernet (a 400% increase) or 100Gbps (1000% increase) over 10Gbps Ethernet? Even 16Gbps Fibre Channel is a 160% over 10Gbps Ethernet.20Gbps and 40Gbps Infiniband have also been around for quite awhile. If raw channel speed is a major industry requirement, then why hasn’t Infiniband become a dominant network technology? |
| More efficient 64/66 encoding | If throughput is crucial, there is a logical argument for using 10Gbps FCoE (that uses 64/66 encoding) rather than 4Gbps or 8Gbps Fibre Channel (which has the less efficient 8/10 encoding). However, the latest 16Gbps Fibre Channel (and above) employs 64/66 encoding too, so this “benefit” is no longer relevant. |
| Greater flexibility | Hmmm… I’m not certain how merging two dissimilar technologies onto a single network medium will provide “greater flexibility”. In most cases just the opposite occurs. |
| Lower power and cooling | Since their component count, general circuit layout, and optical drivers are very similar, just what is it that makes FCoE have “lower power and cooling”? (Please don’t say that it’s because it needs fewer cables. Passive Fibre cabling really doesn’t consume much power!) 🙂 |
| Simplified Infrastructure | This might be true, as long as you’re running low demand systems that only require a single cable. However, if traffic load needs two or more cables, then all bets are off. |
| Better compatibility with virtualized servers | Why? How is running multiple virtual servers over FCoE provide better compatibility than running multiple virtual servers over NPIV? What unique attribute is it that makes FCoE more compatible? |
| Availability of network security tools | This is an interesting argument. The reason we have more Ethernet security tools is that as an external facing technology, more people are trying to hack it. It is true that fibre channel has fewer security tools, but if they are sufficient to provide excellent storage security, why does having more of them matter? |
| Lower cost | Really? What numbers were they looking at? A quick search on Google Shopping shows both FCoE NICs and 8Gbps HBAs are in roughly priced the same.Several months ago we also estimated the total cost of an enterprise architecture using the both technologies, and found that the FCoE configuration ran about 50% higher than 8Gbps Fibre Channel! So much for being less inexpensive! |
| Familiarity within the enterprise | True, but what does familiarity have to do with it? There are lots of people familiar with copying data to DVDs, but that doesn’t make DVDs a better choice for data center backup and recovery. A specialized application like NetBackup or TSM will do a far better job of enterprise backup and recovery, even if only a few IT backup specialists are familiar with them. “Dumbing down” an IT operation to save money is a questionable tactic if user performance is sacrificed in the process. |
| Interface with the Cloud | In what way? The TCP/IP protocol is not native to WAN communications infrastructure, so 10Gbps Ethernet must be converted into something else on each end, just like Fibre Channel. For an internal Cloud connection, TPC/IP is not native to the SAN storage either, so 10Gbps Ethernet must be converted into a block storage format and back in the array, as well. |
| Simplified management and integration with tools | Whoever claimed this as a “benefit” apparently knew little about the breadth and depth of storage management tools available on the market today. |
| No proprietary tools needed to install | I have no idea what proprietary tools they’re referring to for installing Fibre Channel. Last time I did a Fibre Channel installation we used exactly the same tools that were used for high-speed Ethernet interconnections. |
| Lossless Ethernet | Hmmm… If I push the Ethernet standard far enough to compensate for its inherent “best effort” characteristics, doesn’t it just end up looking a lot like the Fibre Channel Protocol (Which is a well established, proven technology)? |
| Operational efficiencies and performance enhancements | If I run FCP (or any protocol) over any other protocol I incur two types of delays – conversion latency, and the consumption of extra CPU cycles. How does adding overhead improve either efficiency or performance? |
| People and skill consolidation | This is an argument typically presented by people with a limited understanding of the complexity of modern SAN storage. Ethernet LANs and SAN FC Fabric have very little in common, other than both support data traffic. Assigning Ethernet LAN specialists to manage enterprise SAN fabric makes no more sense than having SAN specialists manage corporate network communications. |
| Ubiquitous computing | This is a benefit? Stored data is the most valuable asset a corporation or Agency owns. While it may be important to offer ubiquitous computing to the user community, maintaining, protecting, and optimizing data assets should be carefully orchestrated activity provided by highly trained storage specialists! |
| Cost-effective network | Do your own comprehensive cost comparison and see if you agree. My estimate indicated identical functionality from 10Gbps FCoE would cost around 150% more than an equivalent 8Gbps Fibre Channel configuration. |
| Pervasive skill set | Like the “people and skill consolidation” myth above, this is based on a misguided assumption that operating a SAN fabric is somehow similar to operating an Ethernet data communications network. It is not. |
| Simplified interoperability | This may be true – if you can tolerate the latency and performance penalties associated with having one technology host another. As long as server farms are fairly small and storage requirements are modest, making performance compromises for the sake of convenience isn’t an issue. However, it rapidly grows in difficulty as stored data volume increases. |
| Reduces capital and operational costs | As above, do your own price estimates for identical functionality from 10Gbps Ethernet and 8Gbps FC. I think you may be surprised. |
What seems to be missing from these discussions is:
- Vulnerability created by having both data communications and storage traffic over the same medium. If there is an external attack on the Ethernet network, all computing activities will be brought to a halt. If there is a critical firmware bug, both data and SAN traffic is impacted. Troubleshooting becomes much more complex and time-consuming.
- The importance of keeping dissimilar technologies separate so they’re allowed to evolve at their own pace. If both storage traffic and data communications are dependent upon Ethernet, then each is constrained by the evolution of the other. If one requires more capacity and the other doesn’t, you’re forced to buy the consolidated infrastructure in its entirity.
- Dissimilar skill sets and areas of responsibility managed by different IT specialists. Ask most LAN specialists how to zone a fabric or allocate LUNs and you’ll get a blank stare. Ask most SAN specialists how the configure a router or use a packet sniffer, and you’ll probably get a similar response. SAN storage and SAN fabric management are activities that are inextricably linked. Splitting areas of responsibility between a LAN Group and SAN Group is a recipe for operational inefficiency, troubleshooting complexities, and reduced staff productivity.
- If industry adoption of FCoE has been widespread, then why do IT industry research Groups keep reporting sluggish sales? Also, why do Fibre Channel equipment sales remain robust?
I have no illusions there being lots of things I didn’t know, so I could be wrong about this too. If you feel there are other compelling reasons why FCoE will dominate the industry, I’d love to hear them.
Randy Cochran
Big Data Challenges 