@CloudExpo Authors: Stefana Muller, Pat Romanski, Karthick Viswanathan, Elizabeth White, Zakia Bouachraoui

Related Topics: Containers Expo Blog, Java IoT, Microservices Expo, @CloudExpo, Cloud Security, SDN Journal

Containers Expo Blog: Blog Feed Post

Bare Metal Blog: Mean Time Between Failures

MTBF has meaning well beyond storage

If you are new to the Bare Metal Blog series, find them all here

When assembling a model – any model, from a highly detailed functional replica of an engine to a mass produced plastic model of an airplane – there are several places where things can go wrong. The final product is only as good as the model kit, the glue used, the tools used, and the skill of the craftsman. I’ve seen the same exact model assembled and painted by two different people that look completely different, simply because of the array of variables and how they interact.

This is true of high tech equipment also, and like modeling, it is often overlooked. Interestingly, in my entire IT career, MTBF has only been a measure that meant a ton in two circumstances: When designing hardware and scoping the parts to go in it, and when talking about storage. In all other endeavors, MTBF if mentioned was a side note.

And yet it matters. It can matter a lot. Like most hardware companies (because we spec our own parts and monitor our own quality), we track MTBF both computed from the sum of the parts with average environmental considerations, and actual tracking based upon support cases involving hardware and RMAs. For us, knowing helps us improve quality. For customers, knowing helps gauge the bounds of useful life for the equipment being purchased. Of course, MTBF is a mean, not a fact, and it is entirely possible for a device to last much longer than its MTBF, in fact the fact that it is a mean kind of implies that roughly half of the devices out there will last longer. But it’s the mean, not the median, and most IT shops do not want to plan like a device will last well beyond its MTBF value. MTBF can offer a bit of guidance when it is fairly calculated, and another tool in the evaluation toolbox never hurt an IT shop.

As mentioned earlier in this series, F5 sets quality standards for suppliers to meet, if they wish to continue supplying. This allows a bit better control over MTBF than doing something like “lowest bidder” or similar procurement, simply because the standards set include the quality of parts used, which all rolls into the MTBF calculations – and more importantly for most IT shops, the MTBF reality. While MTBF is a complex set of equations, you can generalize to “the MTBF of a device is as low as or lower than the MTBF of its weakest part”. That means supplier quality standards matter in a very real way. I had a RAID array fail on me once – several drives down all at the same time. The array vendor had to count that as a failure, since RAID no longer worked (thank heavens for backups!), but the failure was on the part of one of their suppliers. That’s how it is in the manufacturing world whomevers’ name is on the box gets the bad rep for quality, regardless of whose handiwork was slipshod. That is why F5’s non-stop quality monitoring program (devices are tested from before release until EOL is announced) matters a lot. It’s also why quality standards for parts suppliers matter more then getting the absolute cheapest part, as some manufacturers are wont to do.

I will not replicate our entire knowledge base article here, if you have an ask.f5.com account, you can click here to read it. I’ll just summarize and pull bits out for the readers’ enjoyment.

F5 gear runs the gauntlet from entry level to massive blade systems. As such, MTBF varies from device to device. The worst calculated MTBF for an F5 device is over three years. And our quality team tells me that the calculated value is far lower than the real-life-experience value they get from watching returns and such. The best calculated MTBF is over 21 years. It’s a rare piece of computer gear that is used that long, but Lori and I have got some pretty old F5 gear that’s still clipping away like it was new, so no surprises there. Most F5 devices fall somewhere in between.

Why the large variance in MTBFs if we control for quality? A valid question. The fact is that it is not all about the quality of parts. Airflow inside the device, number of redundant parts, number of removable parts… there are a zillion other things that go into MTBF, and they all tend to get better as the device gets physically larger. Entry level devices are small, restricting airflow and cutting down on available space for redundant power supplies, etc. While the top end blade servers have room for all of that, and since cards are replaceable, tend to less failures. You will find a similar spread with any other vendor that covers such a wide range of hardware. And all of those numbers are likely to beat out a COTS server running a software product.

So when looking at any electronic gear, ask about MTBF. Alone it simply gives you insight into the priorities for the device you’re looking at, when combined with the MTBF numbers from several different devices (the same manufacturer or multiple), it gives you an idea of what you are buying in terms of quality. Of course with a large chunk of any given appliance handled in software, MTBF is not as meaningful as it once was, but it is still the underlying bedrock for that software to run on.

Read the original blog entry...

More Stories By Don MacVittie

Don MacVittie is founder of Ingrained Technology, A technical advocacy and software development consultancy. He has experience in application development, architecture, infrastructure, technical writing,DevOps, and IT management. MacVittie holds a B.S. in Computer Science from Northern Michigan University, and an M.S. in Computer Science from Nova Southeastern University.

CloudEXPO Stories
Containers, microservices and DevOps are all the rage lately. You can read about how great they are and how they’ll change your life and the industry everywhere. So naturally when we started a new company and were deciding how to architect our app, we went with microservices, containers and DevOps. About now you’re expecting a story of how everything went so smoothly, we’re now pushing out code ten times a day, but the reality is quite different.
The hierarchical architecture that distributes "compute" within the network specially at the edge can enable new services by harnessing emerging technologies. But Edge-Compute comes at increased cost that needs to be managed and potentially augmented by creative architecture solutions as there will always a catching-up with the capacity demands. Processing power in smartphones has enhanced YoY and there is increasingly spare compute capacity that can be potentially pooled. Uber has successfully been able to harness the excess capacity of privately owned vehicles and turned into a meaningful business. This concept can be step-functioned to harnessing the spare compute capacity of smartphones that can be orchestrated by MEC to provide cloud service at the edge.
Using new techniques of information modeling, indexing, and processing, new cloud-based systems can support cloud-based workloads previously not possible for high-throughput insurance, banking, and case-based applications. In his session at 18th Cloud Expo, John Newton, CTO, Founder and Chairman of Alfresco, described how to scale cloud-based content management repositories to store, manage, and retrieve billions of documents and related information with fast and linear scalability. He addressed the challenges of scaling document repositories to this level; architectural approaches for coordinating data; search and storage technologies, Solr, and Amazon storage and database technologies; the breadth of use cases that modern content systems need to support; how to support user applications that require subsecond response times.
When building large, cloud-based applications that operate at a high scale, it’s important to maintain a high availability and resilience to failures. In order to do that, you must be tolerant of failures, even in light of failures in other areas of your application. “Fly two mistakes high” is an old adage in the radio control airplane hobby. It means, fly high enough so that if you make a mistake, you can continue flying with room to still make mistakes. In his session at 18th Cloud Expo, Lee Atchison, Principal Cloud Architect and Advocate at New Relic, will discuss how this same philosophy can be applied to highly scaled applications, and can dramatically increase your resilience to failure.
While some developers care passionately about how data centers and clouds are architected, for most, it is only the end result that matters. To the majority of companies, technology exists to solve a business problem, and only delivers value when it is solving that problem. 2017 brings the mainstream adoption of containers for production workloads. In his session at 21st Cloud Expo, Ben McCormack, VP of Operations at Evernote, discussed how data centers of the future will be managed, how the public cloud best suits your organization, and what the future holds for operations and infrastructure engineers in a post-container world. Is a serverless world inevitable?