Carl George
Appearances
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
But those overlapping things are things that community projects have never had. CentOS never had them. And the new RHEL rebuilds that are trying to claim that they're the new CentOS, they don't have them either. They also have corporate sponsors that sell those extensions. They're trying to make their buck too, which is understandable. We're all trying to make money in open source.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
The big value prop that I talked about with Red Hat with the ecosystem stuff is that, not that you'll just go use this and it's a cheaper price than RHEL, it's that you can go to the people creating this software. A lot of times they're maintaining it in RHEL, they're maintaining it in CentOS, and often times they're maintaining it in Fedora too.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Not always, but there's a huge, huge participation from Red Hat in Fedora all the way. It is separate from Red Hat, but we're very involved at every step of the process. So if you can make a feature request and say, I wish this software did this thing, Red Hat can say, all right, that's a good idea. Here's how we'd go about it.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
First, we're going to put it in the upstream project where we're also participating. Then we'll build it in Fedora. And then it'll go into either the next minor version of RHEL or the next major version of RHEL, depending on how disruptive the change is. And then they put it in CentOS Stream Next. And then it goes into RHEL after that.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
So having people that are holistic across the entire pipeline, that's the expertise thing. From the engineering angle, that's the real value I see looking at it with a set of engineering eyes.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Sure.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
You mentioned about how it works differently now. I want to go into that a little more if I can. What do you mean by that? CentOS and working differently, right?
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
The IBM acquisition stuff is kind of tangential, right?
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
So CentOS started outside of Red Hat. And then I think it started around 2004. About 10 years later, the project was kind of on the ropes. Maintainers were burned out. They had day jobs. No one was getting paid to work on it. And what Red Hat saw was that... And it's kind of weird. It's a bit of an incompetence thing.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
We had inside Red Hat development teams using CentOS to build with because we couldn't get out of our own way and give our own teams free RHEL. It's super messy, and it's gotten better since then. But at the time, that was kind of the state of things. That's pretty funny. Yeah. Maybe I should talk about that. But I think it's hilarious.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
I'm just kidding. Nobody's told me I can't say that.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
uh but that kind of drove it they basically red hat was like we want this this project to keep existing and so we're gonna you know they made job offers to all of the developers most of them took it a few of them turned it down and then um they basically came into red hat partially they were still kind of kept off to the side they're like well you're still kind of duplicating this product but we want you to keep going and and uh exist and so they kind of sat in that limbo for a while where they weren't growing they weren't getting
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
They weren't getting people resources, but they had the resources they need to focus their full time on it, get a paycheck, and keep the project going. That was a little bit of an infusion, but we still had this problem around this whole bug for bug thing and also being a duplicate of the product.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
There would never be a business incentive to put the same engineering resources into your product and this project that is trying to match it as close as possible. That would never make sense. No business person would agree to that. but because of all the nuance around how it was being used as a development platform.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
But we also saw the pain points of it being a development platform that lagged behind the thing it was trying to match, right? CentOS would typically lag about a month behind on the minor versions, like RHEL 7.6 would come out, and then CentOS 7.6 would be, it'd be 7.5 for a while, they'd finish the rebuild and publish it, and about a month later you'd get it.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
So those rebuild gaps were real painful for the developers trying to use it as a platform to build on.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
That push wasn't about that reaction. That reaction came later. But yeah, I get you.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Yeah.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
That's where we're at now. It was just a really messy transition. Part of that was like rush timelines. That's a compression of a lot of time. Yeah, definitely.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
For sure. We don't have a lot of time. And that was the dream originally of it, right? We had CentOS lagging behind RHEL. It was painful for developers. It needed to exist, but we had developers frustrated that, okay, well, I'm making this change, but then it changed in the next minor version, and I didn't find out about it until a month later.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
So they wanted to get ahead of those things, and they basically wanted RHEL a little bit earlier than they were getting RHEL-like things in CentOS, in what I call classic or legacy CentOS. The official distro name is CentOS Linux. The way it should have gone down was we just did a clean break at a new major version and said, for example, CentOS 9 is here early and it's different now.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
But because of some compressed timelines and people were excited to get it out there, we ended up doing two variants in version 8. We had the classic variant, which was a rebuild following RHEL, CentOS Linux 8, and then we had to make a new name to distinguish the variant, which became CentOS Stream 8. It's still the same basic operating system, just released on a different cadence.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
And I can say that because at the time, that was my full-time job. I'm working on Apple now, but that was what I got hired by Red Hat to work on. I was doing those builds. It was still, and I mentioned earlier that the RHEL maintainers are taking over control and doing all that work in CentOS now. The early transition wasn't that way.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
The small group of people, like three or four of us that were building classic CentOS, started having to do two rebuilds. The rebuild of CentOS Linux following RHEL, and then the rebuild of CentOS Stream that was ahead of RHEL. and it was really messy for a while until we could get it actually properly onboarded in version nine.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
We ended up putting it on GitLab, and so all the rail maintainers would do their packages there, create them, and do all their development, and then there wouldn't be a rebuild process. They would just build it, and it would become CentOS Stream. But in the early days, we'd have builds, They were all rebuilds, we tagged them at different times, basically just released them at different times.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
And some of them would be classic CentOS Linux, and some of them would be CentOS Stream 8. But it was all from the same build system, all from the same people, all from the CentOS project. So that's one of the things that irks me when people say, this isn't the same CentOS. I'm like... No, but yes, it is. Like, it's the same people. It's the same project. CentOS isn't dead.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Technically, CentOS is the project. CentOS Linux and CentOS Stream were the distributions. But thankfully, we don't have that double thing anymore. We onboarded all the RHEL people, and it's just CentOS Stream. And I think my personal opinion is that we should one day just drop the stream and just say, yeah, this is just CentOS. Most people just call it CentOS. And let's avoid the confusion.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
We should have never had the overlap. It should have just been a clean break in a new major version and leave all the old major versions on the old model. That's not the way the transition happened. Clean breaks are good. Poorly executed transition, in my opinion. Some of it predated me. Some of it I was front row and center and doing what I could.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
So it's all open source. And everything in Fedora is just out there in the open, built in the open. There's nothing private. Everything in CentOS Stream is the same way. It's built in the public. It's all public. And you can contribute to it. RHEL, the contribution path into RHEL is through CentOS because functionally the way it works is it's the major version of RHEL.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
You've got like CentOS Stream 9 now is where all the RHEL 9 development happens and then periodically they branch that into 9.4, 9.5, 9.6. So you can't actually contribute directly into a RHEL minor version because those are built inside Red Hat. But then the major version, you can get it on there.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
So from the developer angle, like you can do pull request to master, but you can't do pull request to the 9.4 branch. Okay. Sometimes the RHEL maintainers will say, yeah, we also have customer pressure to get it in these older, minor versions, and then they can do that part internally. But then the After Effects, it's still all open source.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
It's still all published, all compliance with all the licenses. Once you have RHEL, you have access to the source for every package, even the ones with licenses that don't require it, like MIT or BSD license. So it's fully open source, top to bottom.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Totally cool, right? I'll push back on you a little bit. Okay. You tried real quick on your phone while we were drinking at the bar. I wasn't drinking. You were drinking. I wasn't drinking. I was drinking water. Well, very quick attempt on your phone. It's not the same as sitting down like, yeah, let me create this account. I won't create accounts on my phone.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
I'm going to wait until I get on my laptop again, right? Okay, let me push back to you then. There's a little bit of a barrier, yes. You have to sign up. Let me push back to you then.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Yes.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Download the ISO. LTS. Sure, your point.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
It's challenging. I'll give you a throwback to one of your older episodes when you interviewed Adam Jacob. Sure. Fantastic interview. And he brings up the point of, like, you make a product and you sell it. You don't give it away for free. I agree. Ubuntu's model is that they are giving their product away for free, which there are pros and cons to that.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
And I'm not going to... I don't want to criticize another company's business model. You know, I wish them all luck. I've got friends that work in Ubuntu and work for Canonical or ex-Canonical. But the... You know, it gets back to that problem. You can have all of the market share you want by giving away your product for free. And it's hugely successful and popular.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
But then I know that my canonical friends have told me before that Ubuntu's biggest challenger was always free Ubuntu. Like everyone that's getting it for free because they can and the conversion rate of people that are like should be paying for it to help sustain the engineering of that product is a vanishingly small number.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
And it's extremely hard sell to say, here's why you should pay us when you can just get the product for free. Right. So Red Hat tries to take a different stance.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Well, the access is the same thing, right? Because access is part of that subscription, part of that product.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
And a lot of that is confusion, right? People looked at it as, this is the free access version.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
What would I tell you if, what would you say if I told you that, one, it was never blessed for production, and two, that there's even a website... It was marketed as that. No, it definitely wasn't. Show me a page that says it was blessed for production. But anyways, that's a tangent.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
That's what people said. There was no blessing, right? But that's a minor point. Yeah. There's some nuance to it. There is nuance there. That's not the point I'm trying to get to, though. What would you say if I told you that I can show you a page right now on the Red Hat website that says RHEL is not intended for production? We had this conversation last night. I'm down for it. Yeah.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
It's because on the page I'm talking about, it's in the product store where they say it's a self-support rail where you can buy just access to rail and can't file support cases. And it says this is not intended for production. Because... Red Hat thinks that you should have production or support for your production instances. It's that simple.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
So when they say that, you know, there's also a blog post that says CentOS Stream is not designed for production or intended for production because it doesn't have support. It's around that part, but it's been misinterpreted to say even Red Hat says this isn't good enough for production. Right.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
And there's other interviews with other Red Hatters, like from the Fedora Flock Conference, Brian Axelbeard, he said that just because we don't say you should use it for production or we don't intend it for production, doesn't mean you can't, and there's lots of companies that do. I've got some friends over at Meta, Facebook, their fleet is probably the largest fleet of servers in the world.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
I think the last PR approved term they got to use was millions, plural, of instances. And they're running CentOS Stream everywhere. and they get on the new versions as soon as they can. They're active contributors, and they're deploying this stuff regularly. They use it at massive scale in production. So it certainly can be, it's still rail-like, and it can be used in production.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Right? And there's even more detail to that. We talked about that partner ecosystem stuff. The whole idea of being real compatible is because they want access to that ecosystem. The real brand name.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Yeah, a little bit of that. There's some of the confusion, and that's going on now with the whole automatic and WP Engine stuff around brand name and how you identify that. But the bigger thing is... They're like, oh, I don't care about having RHEL. I care about this app I can install, and it works on this hardware, that whole ecosystem.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
That is what they're buying into, and that is what Red Hat sells. As a product. Yeah. Which I'm cool with. I get that. The whole idea of being exactly RHEL compatible is the idea of getting a foot into that ecosystem and taking advantage of that ecosystem from people that did not spend decades building it and countless dollars building it. Right.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Yeah. And that conflation is a sticking point for a lot of people.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
What I realized around that angst is that We made all those changes, and some of it predates me, some of it was right around when I was getting hired. But what I learned about the CentOS community was they're basically two different personas. And it kind of splits evenly in the lifecycle. They were the people using CentOS in the first five years of the lifecycle.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
New version would come out, they would say, yes, I want these new features, I want these new capabilities, and I'm also frustrated. Those happened to be the same people that were frustrated that they couldn't contribute to it and make changes to it. Then there were people kind of using it in the last five years instead of using RHEL. For them, it was just the free, unbranded RHEL.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
They were never going to contribute. They don't care about being able to contribute. They just want to get the product for free, and they want it to be maintained for as long as possible. So those two personas were kind of where we unintentionally divided the community.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
People that liked what we were doing with CentOS Stream, being able to contribute, and it still has a five and a half year lifecycle, which, I mean, that's the same thing Ubuntu LTS gives you without the pro subscription, five years. So it's still a pretty long time. It's still an LTS. Those people, they're like, yeah, I like these changes. This makes a lot of sense to me. And the people that...
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
do not care about contributing, do not care about getting their bugs answered. They just want to get the product for free. They're like, oh, no, I'm going to go to these other guys that give me the same thing.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
The big change is that because it got actually harder on CentOS and Red Hat once the AquaHire thing happened and they were paying the CentOS maintainers because customers would come in and say, well, you're making both of these things, so why should I pay you for one and not the other? Or why should I pay you for the one when this other one's free?
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
And that conflation of having Red Hat sponsorship, it helped the project not fail and collapse, but it also made it harder to have those conversations, to draw that line between the product and the project. And so now the new rebuilds, like I heard one guy inside Red Hat described it as these changes are Red Hat getting out of the rebuild business.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Like we decided that's not where we want to spend our time. Here's the way that building an operating system works in our pipeline holistically to make a better product. And it's still really close to RHEL and you can still use it for whatever you want to, but it's not going to be trying to match RHEL identically anymore. It's getting six months ahead of RHEL on features and fixes.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
But like you said, a lot of those people that are going to different alternatives now, they're in that latter group, the five plus year usage where they just want the same thing. They don't want anything to change ever. And they don't want to think about being able to contribute being a benefit.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
They're self-supporting. They're active in the projects. They're contributing. They identify a feature that they want or something that's broken that they want to fix, a bug. and they're contributing that into CentOS Stream. They're active contributors there. They're contributing to upstream projects. I know they're heavily involved in SystemD. They participate there.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
A lot of times you'll find talks from them at conferences like Scale where they're talking about the internals of SystemD because they employ a lot of SystemD developers. They have kernel developers, ButterFS developers, all kinds of stuff. So they have a lot of that expertise in-house. Gotcha.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
So they're not really, they don't really need to leverage that support any more than just interacting with those communities already. All right. So the future stuff. Juicy future. Juicy. So the major version right now of RHEL is 9. Everyone knows that. Same for all these RHEL-likes and CentOS Stream, which is still RHEL-like. It's all major version 9.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Everyone can count and knows that the next number after that is 10. Is it 10? Yes. Was it eight, nine? I'm making this joke. Go ahead. It's a little silly because there was actually a time before I got hired where there's some weird marketing thing around it where they were telling engineers that they couldn't say that the next version was eight. And I don't know where it originated or why.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Oh, wow. But then, like, some real marketing folks showed up at the, I think it was the Fedora Flock Conference, with stickers with the rocket ship and the number 8 on it. And after, you know, all the messaging to the engineers was like, don't say the number 8. Just say, oh, whatever, you know, whatever the next version is. And so the engineers were all mad.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
They're like, oh, these guys showed up with the number 8 on a sticker, and they told us we can't say it? That's so stupid. Like, why do we even have this problem?
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
I missed that joke. Big company, inner things, whatever. Weird things. The next version's 10. Juicy stuff. Go. So RHEL's on a three-year major version cycle now, six-month minor version cycle. It'll be a little more reliable. It used to be kind of hit or miss, and one of the feedback we got from customers was bringing it back to Ubuntu.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
They have their schedule where they're like, yeah, we're publishing this month. You can count on it. And a lot of customers really value that. So eventually, version 8 was when they adopted that in 2019. So three-year cycles, you can see that RHEL 9 came out in 2021. Sorry, 2022. So 2025 is when RHEL 10 is going to come out.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
We can't officially say dates, but there's an event in 2025 in the spring that Red Hat puts on that might make sense for there to be product announcements at. Anyone can figure that out just by looking at public websites. It's not that hard. Not that that would be the exact day, but probably pretty close. It's a good time frame to expect it. CentOS Stream 10 has already branched off from Fedora.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
It's getting that initial productization to become, to stabilization, to become RHEL eventually. It's in a state now where you can get it and install it today, but we haven't announced it as, you know, ready's a weird word,
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
I think we usually use launched or released, but there's going to be a launch announcement or release announcement for CentOS Stream 10 pretty soon because it's getting to the point now, it's not that high pace of stabilization. It is, okay, well, we basically have all the features we want.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
We might make a few more changes before it gets released as RHEL 10, but it's basically stabilized, and this is what you can expect RHEL 10.0 to be whenever it comes out next year. So we're going to have that announcement pretty soon, probably next month or the month after, where we announce CentOS Stream 10 is here. You can use it now. It's pretty good. We like it.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Also, Eppleton, the thing that I work on directly, we're going to announce that about the same time. Usually when we've announced them separately, we usually have the feedback that, well, why would you announce, you know, If we announce one, immediately the question is, well, I want the other one to use them together. I want those extra packages, and I want the base operating system.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
They're useless without each other and a lot of people's opinions. So we're going to do kind of a joint announcement probably the same day or the same week where we say, yep, Apple 10 is here. We've got all these extra things you can add. The community has been building them for the last few months, and we've had the infrastructure online. But we're doing like a flag day, like here it is.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
It's as ready as it will be, you know. It's the thing, do we say it's ready at 2,000 packages? Do we say it's ready at 3,000? We're going to keep adding stuff, and even after we announce it, it doesn't stop growing. So we've got those things coming up. And timeline-wise, you can look at it as that's about six months before the RHEL 10 launch.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Yeah, so spring of 2025 is when RHEL 10 is going to be coming out. And then we're about a little bit more than six months before that right now. We're getting all this stuff buttoned up to say, yeah, CentOS Stream 10 is here. You can use it. It's a major version, stable operating system. It doesn't have minor versions, but it's going to be maintained for five and a half years. It's very RHEL-like.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
You can add all of these Apple packages we've been working on and use it right now, and it'll be good to go. I love it. That's the good stuff coming up.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Okay. That is just the mentality of it. It's only packages that you can't get in the base operating system. So I kind of mentioned that there's like 60-something thousand packages in Fedora, and only about 10% of those go into CentOS and then eventually go into RHEL. Everything else in Fedora that isn't that 10% is eligible to go into Apple. So, like, say I maintain, like, the Caddy web server.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
I maintain that package in Fedora, and I also maintain it in Apple branches. Up to date, I haven't seen anyone say, like, we need to put Caddy into RHEL. We have customers asking for Caddy. Maybe that changes in the future. But for now, I maintain it in Fedora, and I put it in the Apple branches for each release, Apple 7, Apple 8, Apple 9, and Apple 10 now.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Put it in there so people can use it on that RHEL release or that CentOS release or any of the other RHEL-like things that are out there. They use it there, but it's not a RHEL package. It's not maintained by Red Hat. You can't file a support case for it. So that's what the extra in the name is for. It's only additional things.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
If, for example, Caddy, if Red Hat decided to add that into RHEL, into the product, it would then become ineligible for Apple, and we'd retire it from there, and instead of getting it from the community repo, you'd get it from the main repos. Gotcha. Does that help clear that up?
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
I'm not going to argue with that point.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
They're trading on the RHEL brand.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Thanks, Carl.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
62%.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Should I introduce myself?
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Sound check. One, two. I like barbecue. How about that? Tacos are good, too. I love eating.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Oh, yeah. Do you? I'm a good amateur. I'm not professional level, right?
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Backyard barbecue. Tell me about your tools, your tooling, your cooking methods. So my smoker that I have, my father-in-law gave it to me before he passed away.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
It was in his backyard for a while, and I was picking up my kiddos from staying at Grandma and Grandpa's house one weekend, and my mother-in-law mentions, oh, I told Catherine, my wife, she's like, I told Catherine that you could have that smoker. And I'm like, what? She never told me that. And my wife denies it to this day. She's like, she never told me that. She's making that up.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
I was over there in like 12 hours. I had like four full grown men over there lifting this smoker in the back of my truck. Like, yes, I will take a free smoker. How many gallons? I don't know. It's not huge. Well, four guys to carry it. Yeah. I mean, it was like. 18 gauge steel, thick steel. Yeah, it was very thick and heavy. One of their cousins made it for him for an anniversary present, I think.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
That's cool. So very old, but very good smoker.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
I mean, better than paying, you know. It's already seasoned. Yeah, better than paying a grand or two for a brand new one, right? Oh, for sure.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Could you build it yourself? Do you know how to weld?
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
You want to build it on site so you don't have to move it?
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Yeah. I don't have any other tricks than that other than like you want to use a good smoker. Volume matters. Like you said, I don't know how many gallons this one is, but I noticed that where like you have like a backyard smoker compared to what you get at the restaurants, the real professional stuff. You have a tall stack.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
They've got like, you know, like 10,000 gallon or 1,000 gallon propane tanks that have been converted into smokers. And I think the volume makes a huge difference on that on how much you can control the temperature variation. It's huge.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
ongoing barbecue science yeah it's endless in texas the smaller it is the harder it is like i have trouble sometimes keeping the temperature even because it's not a huge smoker it's a decent size but yeah yeah the that's how i think the big that's the the real secret from the big professional joints is they can they can afford the massive smokers doing you know 20 briskets at a time and that volume helps them keep the temperature so consistent like one maybe two you know yeah i mean brisket alone is expensive so i'm gonna afford one to mess it up yeah i mean all right
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
And here we are. Good.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Apple. Extra packages for Enterprise Linux. It's an add-on repo. The closest analogy for people that are... I like to compare it to Ubuntu's universe. The main difference is Ubuntu, they enable their universe, their community packages out of the box. You just have it. They're available. But they're not...
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
I think they've changed it a little bit with the new Ubuntu Pro stuff, but for the longest time, Ubuntu's universe repo was these are the community things, Canonical doesn't handle these, and that's basically what Apple is for RHEL. It's just we don't have it enabled out of the box, we make it an opt-in thing.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
You have to go out of your way, add the Apple repository, and then install the community main team packages you want. And a good thing to note is Eppel, it's not its own project. It's part of the Fedora project. And the way the whole thing fits together, it's much easier visually with the diagram. So I'm trying to think how I can describe the picture in my head.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
But there's this line going across that's Fedora Rawhide. That's our rolling release. And that's where all the newest stuff goes right away, kind of like Debian SID. But after this point, the Debian analogies fall apart, it doesn't work. We do our Fedora releases every six months. Fedora 41, I think, just got released today. Those branch off of Fedora Rawhide.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
But then that's something like, I think the last time I looked, it was something like 60,000 packages that are in Fedora. Red Hat doesn't want to support all of those in the product, eventually where it gets into into RHEL, Red Hat Enterprise Linux. So it's only a subset. I think roughly around 10% of the Fedora packages, like 6,000 or so, actually make it into RHEL.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
And that happens by going through CentOS, or CentOS Stream, rather. There's a whole bunch of confusion around the name change we did. It's still the CentOS project. CentOS is not dead. It's just a little bit different now.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
It is there. There's been a lot of misleading messaging around, CentOS is dead or you have to replace CentOS. No. There's differences, you should understand them, but I think there are a lot of positive changes that people are missing out on it because they're not just buying the marketing line of somebody that says, I want to be the new CentOS. Well, That's kind of flawed.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Why don't you just be a distro on your own, make your own reputation, and then see what CentOS is doing. If it works for you, then keep using it. I think it would work for a lot of people. There are some people that, I think there's one guy I know at work that says that if you have a RHEL-sized hole, we want to sell you RHEL. 10 year life cycle, vendor escalation. Assurances.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Yeah, assurances, the partner ecosystem.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Before we started recording, I was telling Adam that one of the big value propositions that I know Red Hat talks about a lot, but I think a lot of people miss out on, whether it's just phrasing or it doesn't convey well, is that Red Hat has spent literal decades and countless amounts of money building a partner ecosystem with hardware vendors, software vendors, and upstream communities, right?
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
and the big value premise you're paying for when you buy RHEL, and I'm not a RHEL salesman, this is going to sound very sales pitchy. You're an engineer. I'm very low in the weeds.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Yeah, 2019.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
I think a good bit of nuance to that is that, yeah, I've only been there since 2019. Relatively short, I've been in the CentOS and Fedora and Apple communities before that. I got hired out of those communities to do it full time at Red Hat, which is another huge value that they do is employing people in open source projects to keep making open source, which we have.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
There's a whole track yesterday here at the conference about open source sustainability and sustainability versus freedom and choice and open source purists and things like that. And yeah, a lot of people, the dream is to get paid to work in open source. I feel great, I've achieved that dream.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Other people aren't as lucky or they get it like, I know my last employer had a thing where it was like, well you can do open source part time and then this much time you have to do these things inside the company. You have a lot of that. And I know a lot of companies, their OSPO offices, open source programs office or equivalent name,
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
They struggle around how do we get our engineers to be better open source citizens. They're consuming all this open source. How do we turn them from just consumers into making sure the things we depend on continue to exist long term? Which is a theme that I'd like to segue off of in DescentOS.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
So, big question, right? I started going on a little bit, started talking about how I wish I had a diagram of Fedora branching from Rawhide into its releases. Every three years or so, we'll take one of those Fedora releases and we'll branch it again and start building the next major version of RHEL. That starts as CentOS Stream, but before we've announced it. It's still very early.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
We're still forming pre-alpha days. We're putting all this stuff together. And then at a certain point, they have enough of the changes that they want to go into the next major version of RHEL. Like we want this version of Apache, this version of OpenSSL. Maybe it's the same ones at the exact time they branched. Maybe they go one forward, one back.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Maybe they add a few other features, build a few things differently. but that is the process of turning the Fedora fast-moving, innovative project into the enterprise product, and that happens through CentOS. There's a lot of chat about how they talk about RHEL compatible and like the Enterprise Linux standard, other people with other projects. There isn't really a standard.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
There's Red Hat making a product, and to whatever extent there is a standard of Enterprise Linux, CentOS defines that. That is where it happens. And so, because it's happening there, you can influence it. You can actually contribute to it.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
I know you all have a big developer audience, and the analogy I used earlier was that if you've got a choice between two libraries, one that is active development, getting features, you can contribute to it, whether or not you have the ability to or the intent to, the fact that you can contribute to a vibrant project that's growing and active,
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
Would you rather use that or something else that says, yeah, we're going to be exactly the same as the other thing. And if you send us a bug report, if it's in the other thing, we're just going to close it. And you can't contribute here. We are bug for bug compatible. There's this whole mythos about bug for bug compatible.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
And really, when someone says I want bug for bug compatible with RHEL, what they mean is I want RHEL without paying for it. That's really what it boils down to. It's a pretty blunt statement, but it's true. What's different from the past when CentOS originally started was that you can get just RHEL for free. There's a lot of free programs.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
There's the, and this is going to sound sales pitchy again, but I'm telling you how to get free stuff. There's the Red Hat developer subscription for individuals. Anyone can sign up and get 16 free RHEL instances to do whatever they want to with. No limits. You can even use it in a business. It's just a little fuzzy because it is individual, right? You can't agree to the terms on behalf of an org.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
So for most businesses of more than one person, it's not really going to work.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
There is also another program, Developer Subscription for Teams, that'll give you, I don't remember the exact number. It's high. It's in the thousands of free RHEL instances in your non-production environments if you're paying for RHEL in production. And then there's also programs for giving open source projects free RHEL.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
There's programs for giving educational institutions free or heavily discounted RHEL. There's tons of ways to get RHEL without paying for it. But there are definitely scenarios where Red Hat once thinks that, yes, this person should pay for RHEL. And a lot of those people are the ones that they use CentOS rather than just, I want an operating system.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
They wanted just to get RHEL without paying for it or get a discount on their RHEL. They'd use, you know, 10% of their fleet on RHEL and then the rest on CentOS to cut cost. That was never a good fit for it because of small, subtle differences in the engineering and how it's built. One of those is that Red Hat Enterprise Linux actually has overlapping minor versions.
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
You can stay on, say, 9.0 after 9.1 and 9.2 come out, still get security updates, and some third parties only certify on specific minor versions. So if you've got third-party vendor software that hard requires 9.2 using anything that's on, you know, like...
The Changelog: Software Development, Open Source
ANTHOLOGY — Packages, pledges & protocols (Interview)
one of the other rebuilds that's on 9.4 on Centro Extreme that basically has 9.6 content right now, it's a little bit ahead on minor versions, then if a vendor requires 9.0 strictly, then it might not work. But Red Hat will sell you 9.0 still with security updates. 9.2 might be a better example, because it doesn't last forever, you can't stay there forever, it's just an extension.