Photo: Courtesy of Red Hat.
Recently, Red Hat has made Red Hat Enterprise Linux (RHEL) usage free to individuals, including some production use cases, and for certain qualifying teams of developers. This represents a sea-change for the RHEL user community and brings additional benefits to individuals and developers that had previously used downstream or “respin” releases of RHEL source code. The specifics of the RHEL development subscriptions are amply covered elsewhere. This article focuses on the added benefit to developers using RHEL, principally being that Free RHEL is RHEL.
What?!! That’s pretty obvious.
Maybe, but it also means a lot. Red Hat for years has publicly released our operating system source code, exceeding the requirements of the various open source licenses including the GNU Public License (GPL). Others have been able to use that source code to build alternatives to RHEL. This list includes CentOS Linux, Scientific Linux, and others. Developers and individuals used these “respin” releases to build applications that could then be deployed in production on RHEL, if desired. With the opening up to individual and development team use, RHEL can now be used to build the solutions as well as deploy them in production for individuals for free.
Over the last twenty-five years, Red Hat has accumulated extensive operational knowledge of our operating system from numerous customer interactions. This knowledge has been distilled into an AI/ML based cloud service called Red Hat Insights to ease the administrative effort for RHEL. By using RHEL, even via the free developer program, individuals and development teams gain the benefits of Insights and also comply with various accreditations and certifications.
If the source is the same and RHEL is certified, isn’t the derivative rebuild also certified?
No, it’s not, and this is a common misconception around how many certifications actually work. As a company, Red Hat spends significant time and money meeting the stringent requirements of multiple industrial, international, and governmental certifications and accreditations. These are often tied to specific binary releases of RHEL or specific testing and validation conditions. Many times neither the binaries nor specific test conditions and results can be easily reproduced — and so these certifications are not transferable to alternate builds of the same source code. And it does not help merely to make sure configuration settings are the same, since there are potentially a lot of libraries and modules under the hood that you’d need to check as well — and that the configuration is enforced and honors your intent and aren’t merely “no-ops” that might give you false assurances.
But, isn’t the source code the same?
It is, mostly, but the binaries are not. The difference comes down to the build processes, compiler versions and switches, and other optimizations that go into the RHEL release. These can have a significant impact on the produced binaries. There’s many technical details that cover why that is, which is beyond the scope of this discussion. These articles do a great job of highlighting some of those differences. Red Hat also invests significant effort to employ robust processes and protect the entire supply chain needed to convert source into binaries — the compiler and tools as well as the clean-room physical infrastructure to execute the builds. In the end it’s about confidence, so you know your core operating system can be trusted.
So I carefully rebuild then?
Yes, definitely do that. But also realize that your binaries will be different unless you guess the exact right combination of compiler versions, optimizations, and flags across hundreds of packages. CentOS Linux packages for example are similar to the RHEL packages, but differences occur in the compiled libraries and executables and not just package metadata. Similar is not the same. And this matters when talking about certifications – which after all give you the confidence that your platforms will work reliably with RHEL certified hardware, software, and applications. The new developer program lets you start with the exact same binaries you would use with RHEL otherwise — meaning when it’s time to go to production or you need to get support, you can update your subscriptions without having to reinstall your operating systems, applications, or anything else. You are all ready to go.
Recently, the GSA’s FedRAMP program acknowledged this and reiterated this for CentOS Linux. As a downstream derivative of RHEL, the Centos Linux community relied on rebuild and recompilation of its cryptographic modules. But, Red Hat cryptographic modules running on any version of CentOS thus lack FIPS-140 validation, and FedRAMP cannot accept FIPS-140 validation assertions of these modules on the CentOS platform, including CentOS 7.
What?!! C’mon, source is the same, binaries are close, what’s the problem?
Source is the same and binaries are close, but the packages are still different. This matters. Certifications are often tied to very specific binaries and binary compatibility. One way to think about this is how a cake and the cake recipe differ. Given a recipe, anyone can try to make the cake but the result often falls short of the picture in the cookbook. Certifications can be thought of as applying to a specific cake and not the recipe in general. Claims made by professional reviews of a master chef’s work wouldn’t apply to the results in your own kitchen (if you’re like me). Making claims about your own work carries less weight than third parties giving high praise for the work of another. The recipe may be the same but the results are different. Trust increases when neutral third parties attest that the item is good, where good typically means the item meets or exceeds some previously agreed set of criteria. Attestations from neutral parties increase trust. Unless a derivative rebuild of RHEL source code also engages in the same certification processes, there are no third-party attestations.
Security is all about trust and binary assurance is critical. Let’s look at some specific examples. The National Institute of Standard and Technology (NIST) in the US administers the Cryptographic Module Validation Program (CMVP). This program focuses on verifying module binaries compliance with Federal Information Processing Standard 140-2 and 140-3. Cryptographic modules submitted to this program undergo third-party test and review to validate that they meet the established FIPS criteria. These attest to the correct behavior of the module that can then be relied on.
The important thing is that Red Hat Enterprise Linux appears many times across multiple releases and libraries having undergone this process. And it matters! As NIST states the following on the CMVP main page (emphasis is theirs):
Use of Unvalidated Cryptographic Modules by Federal Agencies and Departments
Unvalidated cryptography is viewed by NIST as providing no protection to the information or data—in effect the data would be considered unprotected plaintext. If the agency specifies that the information or data be cryptographically protected, then FIPS 140-2 (until September 22, 2026) or FIPS 140-3 (beginning September 22, 2020) is applicable. In essence, if cryptography is required, then it must be validated. Should the cryptographic module be revoked, use of that module is no longer permitted.
Regardless of arguments that FIPS validation is not needed or that adequate alternative protections are in place to mitigate the need, no one can claim that a non-validated cryptographic module meets the FIPS requirements. Given heightened awareness of supply chain concerns, security officers will become more and more reluctant to grant waivers that accept the risk of technology from unvalidated sources from unknown origins.
Another example is the Defense Information Systems Agency (DISA) that works with vendors to issue Security Technical Implementation Guides (STIG) for the vendor’s respective products. DISA maintains a list of Linux/Unix STIG content with RHEL appearing multiple times. US DoD programs seeking interim or full authority to test and/or operate must typically demonstrate compliance with a relevant set of STIGs. This activity entails modifying systems and configuration to comply with various security controls. Of note, there are specific guideline entries that preclude use of the STIG with an alternative operating system — Meaning a RHEL STIG cannot be applied against CentOS Linux — RHEL security guidance is applicable only to RHEL and using it for an alternative operating system is a Category (CAT) 1 Severity Level finding, the most severe level. CAT 1 findings are not arguable and they cannot be ignored.
Ok. I get it.
That’s great! Free RHEL for individuals and development teams confers all the benefits of RHEL, benefits that aren’t present with a respin. As an example, RHEL Insights is immensely valuable to individuals and teams looking to simplify operations and maintain compliance. For some use cases, this is a distinction without a difference. But for use cases that require certifications and accreditations, this is important and that is why Red Hat makes RHEL available for free. Organizations producing derivative rebuilds can seek the same certifications and accreditations on their own, but they cannot legitimately claim that RHEL certifications apply to their efforts.
Authors:
- Rich Lucente, Principal Solution Architect to the DoD, Red Hat
- Michael Epley, Chief Architect, Public Sector, Red Hat
