Managing a Linux distribution for a long time requires immense community effort. But what is the key to success in a market that includes hundreds of competitive options? Also, what are the challenges in building an open source offering? Karanbir Singh, project leader, CentOS, answers these questions and outlines the future of the platform that has been leading Web developments, in an exclusive conversation with Jagmeet Singh of OSFY. Edited excerpts..
Q How did you start your journey with the CentOS project?
It was in late 2004. I was not one of the founders of CentOS but showed up on the scene in its early days. At that time, we had a small team and a lot of machines running Red Hat Linux 7.3 and Red Hat Linux 9.
With Red Hat moving down the path towards Red Hat Enterprise Linux, a model that didn’t work well for us, I started looking at options. We explored Debian and SUSE initially, but found management and lifecycle on each of them hard to map out workflow into.
It was during this time that I came across the Whitebox Linux effort and then the CentOS Project. Both had the same goal, but the CentOS team was more inclusive and seemed more focused on its goals. So, in late September 2004, I joined the CentOS IRC channel and then, in November, I joined the CentOS mailing list as a contributor. And I am still contributing 13 years down the road!
Q What were the biggest roadblocks that emerged initially while designing CentOS for the community, and how did its core development team overcome them?
A lot of our problems were about not getting off the ground. Initially, there was no clear aim. And then, we faced the challenge that the build systems and code audit tools in 2003/2004 were either primitive, absent entirely or the contributors were unaware of them. A Linux distribution is a large collection of a lot of code, written in many languages — each with its own licence, build process and management.
Three main strategies saw us past that painful process. The first was consistency. Whatever we did, we had to be consistent and uniform across the entire distribution, and make sure all developers had a uniform understanding of the process and flow. The second was a self-use focus. Regardless of what the other people were targeting, all developers were encouraged to focus on their own use cases and their personal goals. The third was the hardest, to try and disconnect commercial interests from developer and contributor work.
Q Why was there a need for CentOS Linux when Fedora and Red Hat Enterprise Linux already existed in the market?
The Fedora project was still getting sorted out around then. Its team had a clear mandate to try and build an upstream-friendly Linux distribution that was going to move fast and help the overall Linux ecosystem mature. Red Hat Enterprise Linux, on the other hand, has been built for the commercial medium to large organisations, looking for value above the code. This left a clear gap in the ecosystem for a community-centric, manageable, predictable enough Linux distribution that the community itself, small vendors, and niche users around the mainstream could consume.
Initially, the work we did was quite focused on the specific use cases that the developers and contributors had. All of us were doing specific things, in specific ways and CentOS Linux fitted in well. But as we started to mature, we saw great success in specific verticals, starting from academia and education institutions to Web hosting, VoIP (Voice over Internet Protocol) and HPC (high performance computing).
Q What were the major outcomes of the Red Hat tie-up?
Red Hat came on as a major sponsor for the CentOS Project in January 2014. From the CentOS Project’s perspective, this meant we were then able to start looking beyond just the platform and the Linux distribution. It allowed us to build the infrastructure and resources needed to support other projects above the platform, develop a great CI (continuous integration) service as well as a much better outreach effort than we were able to earlier.
The real wins have been from the user perspective. If today you are looking for an OpenStack user side install, the CentOS Project hosted on the RDO stack is the best packaged, tested and maintained option.
In a nutshell, the Red Hat relationship has allowed the CentOS Project to dramatically expand the scope of its operations beyond just the Linux distribution and enable many more user scenarios.
Q What makes CentOS Linux a perfect choice even 13 years after its first release in May 2004?
When users install a Linux distribution, they are almost always doing so in order to achieve a goal—either to run a website or to run a mail server. Helping users achieve their end goals easily has been our constant focus. It is a key metric we still track in order to reach our goals.
This means that as the user base adapts to the new world of cloud-native, container-based and dynamically orchestrated workloads, the CentOS Project continues to deliver the same level of user focus that we have had over the years.
Protecting the user’s investment in the platform across the base without giving up on existing processes is something we deliver till date. For instance, people can choose when and how they jump on the container process, or just entirely opt out. It is not something that will be influenced by a CentOS Linux release. It is this duality, which maintains an existing base while allowing the user to seamlessly move into emerging tech, that creates a great value proposition for CentOS Linux.
“We invest heavily in automation to enable long-term support”
Q How do you manage the diversification of different CentOS Linux versions and releases?
The way the contribution process and ownership works makes it relatively easy to manage the diversification. Primarily, the aim is to ensure that if we are doing something specific, the people doing the work are directly invested in the result of the work itself. This helps ensure quality as there are eyes scrutinising incoming patches and changes—since the developers’ own requirements could be impacted by shipping a sub-optimal release.
Participation from the target audience for the specific media or release is a very critical requirement. And because this typically comes through the common project resources, it also means that the people doing this work are well engaged in the core project scope and Linux distribution areas, allowing them to bridge the two sides nicely.
At the moment, there are dozens of different kinds of CentOS releases, including atomic hosts, minimal installs, DVD ISOs, cloud images, vagrant images and containers. Each of these comes through a group that is well invested in the specific space.
Q What are the various efforts that lead to a consistent developer experience on CentOS Linux?
Application developers working to consume CentOS Linux as their base can trust the API/ABI efforts where content deployed today will still work three or five years down the road. The interfaces into that don’t change (they can evolve, but won’t break existing content/scripts/code). Therefore, working with these interfaces also means that they work within the same process that the end user is already aware of and already manages for simple things like security patching, an area often overlooked by the casual developer.
Q How does the small team of developers manage to offer a long period of support for every single release?
We invest heavily in automation to enable long-term support. And that means that a very small group of people can actually look after a very large codebase. It changes the scope of what the contributors need to do. Rather than working on the code process, we work on the automation around it, and aggressively test and monitor the process and code.
The other thing is that we get community support. Developers and contributors don’t always have the time to work on each request, but if you look at the CentOS Forums, almost every question gets a great result.
There is also a lot of diversity in the groups. The CentOS IRC channel idles at over 600 users during the day, but a large number of users never visit the forums. Similarly, the CentOS mailing lists include over 50,000 people, but a large number of them are never reaching the IRC channel or the forums.
Q Is it solely the community feedback that helps you build new CentOS updates, or do you also consider feature requests from the Red Hat team?
CentOS Linux is built as a downstream from Red Hat’s sources. They are being delivered via git.centos.org, and then the additional layers and packages are built by various community members. We also encourage people to come and join the development process to build, test and deliver features to the audience that we target through the open source project. All this is entirely community focused.
So if someone at Red Hat wants to execute something, they would need to join the relevant community and work that route for engagement on CentOS.
Having said that, we have a concept of Special Interest Groups that can be started by anyone, with a specific target in mind. Of course, this is only above the Linux distribution itself.
Q Apart from your engagements related to being a CentOS Project member, what are your major tasks at Red Hat?
These days, I spend around 30 per cent of my time working on the CentOS Project. Rather than in the project itself, most of my focus is around enablement and making sure that contributors and developers have the resources they need to succeed. The other 70 per cent of my time is spent as a consulting engineer at Red Hat, working with service teams, helping build best practices in operations roles and modern system patterns for online services.
Additionally, I have been involved in some of the work going on in the containers world and its user stores, which includes DevOps, SRE-Patterns, CI and CD, among others.
Q What are the major differences you’ve observed being a part of a corporate entity like Red Hat and a community member? Which is tougher among the two?
I have been involved with open source communities for over 15 years. During such a long period, open source work has never been my primary job. It’s always been something that I do in my free time or in addition to what I was already doing, similar to my move with the CentOS Project. But what makes Red Hat unique in a way is that this isn’t an odd role.
A large number of people at Red Hat participate and execute their day job via open source communities. And that makes it a lot easier, being a long-term contributor.
There is only one key challenge that one needs to keep in mind when working on an open source project as a part of the day job, though. It is to set realistic expectations around community participation, and recognise that the community is there because its members often care about something far more than the people paid to work on a project. However, this typically isn’t a concern when a community comes together around smaller pieces of the code.
The CentOS Project has quite a widespread and extensive footprint. It involves talking to and participating in a wider community where a large majority is unknown personally. Managing
expectations and ensuring there is respect both ways has been a balancing act that’s never easy. It’s something I work quite hard on, and hope we can keep getting it better.
Q What is your advice to those who are planning to begin with CentOS?
The most important thing to remember for those planning to begin with CentOS is that you are not alone. There is lots of documentation as well as tremendous support and help from a vast community. So as a new user, make sure you engage with the existing user base, ask questions, and spend a bit of time to understand what and how things are. In the long-term, understanding the details will pay off.
CentOS Linux is built on a lot of past experience. Anyone starting down the path of adoption should keep this in mind. We’ve tried to build a culture of helping those that need the most help, but also encourage new users to learn and grow with the community.
Q What are the biggest features that we can see in the next CentOS release?
The Special Interest Groups (SIGs) are constantly releasing new content all the time. As an example, the Cloud SIG released OpenStack Ocata within a few hours of the upstream project release. There is also a lot of work being done in the Atomic SIG around containers, and in the Virt SIG on existing and upcoming virtualisation technologies.
Q Lastly, where do you see CentOS in the future of open source?
CentOS Linux, due to its characteristics, is a great fit for areas like Web hosting, Web services, cloud workloads and container delivery. Also, as a platform for long-term community centric workloads, it is a good option in areas like HPC and IoT. The Linux distribution also specifically suits the needs of the education sector—starting and supporting not only IT education but education as a whole. Moreover, I would like to see CentOS Linux extend its footprint as a vehicle for enablement.
And this is where the CentOS Project provides a great space for other open source projects—building services for CI/CD, interacting with the users, finding the niches that people care about and solving real-world problems.
If you are involved today with an open source project, I strongly encourage you to get in touch with me and discuss its development areas.
We measure our success on the basis of how successful CentOS has been for the people, the communities, the open source projects and the users who have invested their time, resources and support for the CentOS Project. And we look forward to solving more problems, building better solutions and bridging more gaps together.
You can reach Karanbir directly at kbsingh@centos.org or meet him on Twitter at @kbsingh.
