Ep 21 — Red Hat’s Emmy Eide on How To Build a Strong Software Supply Chain Security Program

Harshil: Hello, everyone. Welcome back to another episode of The Future of Application Security. Today I have Emmy with me from Red Hat. Emmy, welcome to the show.

Emmy: Hi, how are you doing?

Harshil: I am doing well. Such a pleasure to have you on this show today, Emmy. I've been following your work for a little bit especially since we've been all talking about supply chain security quite a bit. And this is going to be a great conversation, I can already tell. I was looking at some of the work you've done, some of your social posts and things like that on this topic, including things like the SLSA framework. Really looking forward to dive into those topics today. Before we go too far down, I would love to have our audience just to get to know you better. If you don't mind, can you tell us what you do at Red Hat, what has your career been, how did you arrive at the role you're at today?

Emmy: Yeah, so my name is Emmy, and my last name is pronounced Ay-dee. For those who have a hard time, it's the most complicated four letters in English language. So I'm Emmy Eide, I'm the Director at Red Hat. I lead the supply chain security group that falls under product security at Red Hat. Let's see here from a little career path, was in the Army for a little bit, was an intelligence officer over there in the US Army and then moved, I did a brief fellowship at Amazon where I kind of got more involved with information security and kind of found a passion there. And then I ended up at Cisco doing information security, then pivoted to product security at Cisco, and I think it was a year and a half ago, I moved over to Red Hat to start up and lead this supply chain security group, which is a kind of interesting bridge between InfoSec and ProdSec. It's been fun, though.

Harshil: Yeah, that's exciting. So you found passion, interest in this area of supply chain security. So tell us more about why does this matter now?

Emmy: Sure. To start, supply chain security, folks talk about supply chain security differently. Just to level set what does supply chain security mean when I'm talking about it, with us it is anything that, think about it like a capsule or even like if the code is water flowing through the pipes, we are the pipes, right? So supply chain is anything that touches, stores, manipulates, signs the code that eventually ends up with our customers, whether it's in an offering or in a product. So that is all the way from start to finish. In companies who use open source like we do, it's more complicated with what that source looks like. If you're a closed source company, then you can control all the way from start to finish. Open source companies like us have to figure out where does that control start and what parameters do you put in place and then what can you push back to the community to help to alleviate any issues they might be seeing, I would say, downstream, right?

Harshil: I love that definition. I love that analogy of this being similar to water flowing through pipes. So anything that touches, you said it better, but on the pipes and the security of the systems that are building the code. Which is kind of interesting because as you and I were talking about this earlier, a lot of the supply chain security, software supply chain security definitions, I've seen a lot of narrower versions of it. Sometimes people think software supply chain security means SBOMs or sometimes it means CI/CD security. What I'm hearing is it's all of it and more.

Emmy: Yeah, I don't think I actually answered your last question. I just kind of went off and defined supply chain security by itself. But yeah, it's all of those and more, right? Like it's the culmination of it. It is a piece of product security and information security that security practitioners, I believe, have been ignoring for many years because it's complicated. There's no fine line between what is information security, what is product security, what is secure development. It is this consistent gray area that I think is a challenging gray area because you're working on the same systems but for different end states, right? My purpose for being there is going to be different than someone from information security's purpose for being there. I think as security practitioners, we always need to walk that back up. Like, what's our purpose for looking at that specific system or that specific use case? Is your goal to secure the code? Is your goal to secure the company? Is it both? Why is it an important topic though, which was your first question, think about those pipes, if there is lead in your pipes, you're going to poison the water, right? You're going to poison anyone who drinks it or anywhere that water is used. If that water is used to fuel something that needs pure water. So CPAP machines need distilled water. Well, what if that water wasn't distilled and you have now it has lead in it and someone breathing that in. It's the same with supply chain security. If you interrupt that, if you interrupt that process, you can inject code into it, let's say you're a malicious attacker and you're putting it in a back door into, let's use Red Hat as an example. Use Red Hat’s Enterprise Linux. Now our customers' supply chains are also corrupted so it's this kind of trickle effect but it also is hugely lucrative for them, right? Like injecting into a supply chain, you're not only attacking the primary target, but you're attacking all of their customers as well, which we've seen, if we've seen any of the big breaches in the past, those are the ones that are the most systemic, the hardest to clean up and most lucrative for the attackers.

Harshil: Right. And a lot of times the primary target might be somebody downstream, but the upstream source might be an easier way to get into that. Yeah

Harshil: That's phenomenal. We've seen some of these breaches or some of these incidents happen around software supply chain. I think there is definitely a heightened awareness about those topics nowadays. And I believe you recently spoke at a conference and you were one of the keynotes on this topic. Maybe talk about what the topic was, what did you discuss, and what were the key highlights from that?

Emmy: Yeah. So I talked a bit about out why supply chain security is important, why is it such a hot topic. And then I went into, what are we going to do about it. I'm not going to sit here and just be apocalyptic all the time of “Oh, my gosh, there's all these bad things”. Bu wthat are we going to do about it? There's plenty of talks at that same conference in various conferences about using different tools. There's just a laundry list of tools that you can use to help your supply chain security depending on what your supply chain looks like. And that is a key component there is that no supply chain is the same as anybody else's. There's probably some repetition that you can do if you wanted to streamline things more, obviously. But they all look very different which means you have to have different solutions in place. So the idea is not that if I hand you a set of tools, it's going to automatically secure your supply chain. What my call in my keynote was we need to go beyond that and we need to move into partnership. And so in the past we've talked of weaving security community, we've talked about baking security and shifting left with security. And all of it really is a call to partnership but without kind of the key word I think that locks it in. Moving left with security does not mean that I take my clipboard of security check the box things to do and say “Ah, I'm going to tell you this in your planning stages versus telling you this right before you go to production”. And that's kind of what it implies. So I'm moving beyond that to save partnership, which is, I take my list of my check the box things to do and I as a security practitioner make the effort to draw what the business, what the risk is and clearly articulate that risk to the people making decisions about feature development and planning. So someone in security, and not Sec DevOps, I'm saying someone in the security group that is making the policies and they are implementing timelines for when you have to be compliant, goes to the product teams themselves and has this very in depth conversation to say “We are going to work on identity and access management. It's something that we're seeing systemically that needs to change. Here's some issues that we've seen. This is what we want to do. Now shoot holes in it. Tell me all the things that you as a product need for feature development. Tell me all the timelines that you have, because I don't want to interrupt product timelines. I want you to sell product just as much as you want to sell product. So tell me all of those things. Okay, now let's talk about the business risk. Do you understand what it is? Can I articulate it better? Is there a risk?”. Because not every risk is the same. There are plenty of vulnerabilities that have a higher risk and lower risks, and you want to make sure those are prioritized appropriately for your business. So going in and having that business conversation and bringing it back to a language that is clear between two people or between two organizations and then saying, “Great, I hear you. I hear what your program is, your productization timeline - let's say you plan a quarter out - your productization timeline is X. Can we get this implemented in that X time frame? We have given you enough notice. What can we do? You have this system. It's not always a plug and play. Okay, you have the system over here. How can you make sure that system can speak to the system? Oh, they're incompatible? Well, I'm going to find a different solution that's compatible with yours so you don't have to completely review your pipeline”, right?. So having that real partnership where you're actually working together on the what and the how is key. I think it's only through that kind of empathetic lens that we are going to be able to have real security and really reduce risk. Otherwise we're just going to fall back to check-the-box security because that's what's easiest to do. It's easy when you get all these government regulations coming out saying, “Here's your SBOM. This is what it has to look like”. Okay, I've got like a laundry list of 100 things I have to now comply with and supply chain security. I don't have time for this. I'm just going to blast out an email and try to get everyone to do what I want. It's not going to work. It's actually going to take people longer. There's no point in even doing that at all. While it seems like the path of least resistance, it isn't. But as long as it seems like the path of least resistance, that's where people are going to go. So what we want to do is we want to take that partnership, have everyone understand all of the risk associated with these changes that we're making or not making, and then make those decisions together, so that when you're not in the room, as a security practitioner, they can make follow on decisions without you. So that they can say “Something's got to give. Can we let this give?”. “ This is a random security test. Does anyone know what this is?”. “I don't know what it is. I don't know what the problem is. Let's just boot it to the next quarter, we've got something else to do”. You want to avoid that. So really, coming into partnership and that's really what my keynote is about, is like, embracing that. But I think I actually had more time in this talk than in my actual keynote. Haha”

Harshil: Hahaha. Also, the other thing that was different from the keynote was there was nobody asking questions in the middle of your conversation. I'm going to ask you a question.

Emmy: Sure.

Harshil: I love that idea of partnerships and understanding where developers are coming from and making them understand where we are coming from as security professionals. It's a true partnership. I think as an industry, we've tried multiple iterations of it originally more than a decade ago. It started as a Security Champions program, right? The idea was to build Champions who would champion the cause of security within the dev team. Because the fact is, security teams will never be able to scale as much as the engineering teams or dev teams, right?

Emmy: Right.

Harshil: But what ended up happening, at least this is just my opinion, in most cases, is that Security Champions turns into a training program. We will teach you how to code write, or not code, but how to write secure code rather. And that's where it tends to end. Not ideal, obviously, but I also love the fact that you're taking it even further in terms of building partnership and setting that as a part of the “job description”. I've seen some companies do that, especially. I was talking to a few other people in Seattle, and there's some companies here in the Bay Area as well who even have the title as a security partner. Their whole role is to be very closely embedded with the business and help understand. Now, the challenge is, if I'm a security team and how do I scale my resources, that I can build that partnership model. And on the other hand, if you're not able to scale effectively, if you don't have that model correct, I don't know what the correct way is, but if it doesn't really work, then the dev teams, on the other hand, are saying, “Well, these security people, they come to us once in six months, whereas I'm building new things every other week”. They're also working with six other teams. They don't really understand our architecture, they don't really understand our code. I'm not really going to wait for them when they come around. I'll just tell them “we'll just do whatever assessment they need to do”. So those are the two personas from a dev perspective. They need to maintain agility and flexibility, speed, whereas security will never have the resources that are enough to keep up with the dev team, right? I shouldn't say never, but in most cases. So how do you structure this in a way to be effective?

Emmy: I think you're right and that having embedded folks in an organization whose pure job is security, it's not necessarily the right way to go. The fundamental thing here is that anyone who develops can understand security concepts because they're going to have to be the ones making the changes anyways. You have system administrators who are going to have to make changes if it's to a system itself. So we don't want a security champion, we want systemic security. What we've done with the scalability approach is to make a partnership kind of a cohort-ish. That's not the right word. It really is just like a partnership group where we have folks that represent all different parts of development and engineering and they are the developers or the admins, or the folks that really need to be the ones making the changes or reading those changes. And we communicate to them, they go back, communicate to their groups, we supplement those communications. And it's not a “this is what you'll be doing”, it is questions, “Please answer these questions for us”. If we implement this on this timeline, what are we going to be impacting? And that idea of, oh, they're not here to just make us do a bunch of random security tasks. They're actually wondering, oh well actually you're going to affect this product that's going out the door if you try to make that change. We're going to have to do a full migration over to a new architecture if we make that change, right? That's stuff that we wouldn't know. I think that are the things that developers are sitting back and going like, “Why do I bother? They're not going to consider these anyways”. We do consider the things, they come back and go, “You're right, we're going to push this out to this quarter. We're going to help you plan, completely plan how this is going to take place”. And they're like, “Whoa, all right, well, they are listening. Okay, what else can I do for security?”. It's just like true partnership back and forth. We say, “Hey, if you want to enter this space over here, this market space over here, you're going to want to make this change sooner than later. If we're like reading the tea leaves, government is going to be asking for this. We see this coming down the road. We want you to be able to sell in that space. If we backwards plan, this is where we should kind of start working on it. How can we implement that with you?” We're really legitimately down on the ground, working with the teams to help make these changes and to push them through. I think how I've seen security champions being done is that you still have your list of SSDF requirements or whatever it is, and you have that security champion that's in there trying to point to this list and say, “No, you're supposed to, can you follow all these? I'm going to do an audit of you. Can you just follow all these?”. There's not like a rhyme order to it. It's not like we're all in this together. We're all kind of pushing through. Like all of the pipelines are making changes at the same time, or like, we're all working on it collectively. So it's just a different approach of instead of a representative or like a position where we're trying to compete with voices, where it's like one security voice to every 20 engineers or 20 developers. It's several security voices integrated and having a conversation with those 20 developers and hearing and listening and actually taking feedback and taking that in and adjusting plans accordingly.

Harshil: Right. So it's a very deep cultural shift where people think of it as a partnership, and they have to believe in that as a partnership.

Emmy: I will say that Red Hat has, like a little bit of an advantage with this because that's just how they work. Red Hat has this, like, open culture where everybody has a say, everyone has a seat at the table. If you think that this is not the right way to do it, say something. And so implore other organizations that also carry those models of everybody can contribute regardless of what their level is, can embrace something like this. But you do kind of have to have that deep rooted culture in it of feedback is a gift. I want your feedback, I'm not going to be offended by your feedback. Go ahead and blast an email out and I will be like, “Oh, well, that's great. I just learned something new!”.

Harshil: Right. That's amazing. It's a fantastic culture, especially for security people. When you're trying to get people to do stuff, it's in the interest of the business at the end of the day, but it's still difficult.

Emmy: And it’s not getting people to do stuff. It's getting everyone on the same page about risk. It's getting everyone to understand stuff and to really have it be like, “Okay, if I can understand what your end state is, then I can move forward with it. But if you're giving me a list of things I need to do, I'm going to forget that list.”

Harshil: Right. Yeah, you're 100% right. Have you seen any forcing functions work? I'm not seeing forcing function in a negative way, but more like people are humans, they tend to have multiple priorities and they may or may not remember the things that they should be doing even if they believe in it. And the reason I'm asking this is because I think the way I see the SLSA framework or some of that work around exposing the information about security controls directly to the developers as a way of making them aware of their decisions. So it sort of indirectly forces them to think about it at least. And they can make their own choices, obviously but do you have any thoughts on how you can build technical controls that may help the developers build secure code or make the right choices at the right time? Maybe you can talk about how SLSA framework fits in to enabling developers to secure the software supply chain.

Emmy: Yes, so SLSA is more of like a maturity model and best practices. I really do like SLSA and I think it's a really good framework, especially for kind of normalizing our conversation around supply chain security. But the 1.0 hasn't even been published yet and so we're still working on finding tuning that to make sure that we're not being overly prescriptive and how you do something, because that can also kick a lot of people out from being able to implement any of these controls. Because what I can do at Red Hat is different than what a startup can do.I have a whole team to work with versus startups might not have that. I don't have an awesome answer for you unfortunately. For that it really just depends on what your supply chain. So some folks do like a single supply chain, a single pipeline that all of their systems go through that have a lot of steps throughout the way. Other ones, if you are developing product code that is in different formats then you might not be able to do that, give different languages, you might not be able to do that. So it really just depends on that. But I will say that the SLSA community is working on tools that can go along with checking for SLSA and attesting to it. So I would say if an organization is just starting out in supply chain security, you can start with level one and go through that list and you can work to use I think they're involving Tekton Chains with it, but Tekton Chains to check those actions as you go. But I would say that it's more again, going back to that culturally, you need that systemic side of it to happen. Like when you wake up in the morning, you have a series of ritual actions that you don't think about, that you can think about the rest of your day while you're doing these actions, right? You're getting dressed, you're brushing your teeth, you're showering, you're shaving, whatever it might be. Those are not things that you think about every day and they don't overwhelm your brain. You're thinking about completely other things. Those are just part of how you do things. Similarly, that's what we have to get to with coding, is that it's not even thought about, this is how we do things, this is how we secure and we code in a secure way. This is how we have secure supply chains. When I get to the point of the check, it shouldn't kick you back to the beginning. You should have already done those things because that's just going all you down and set on your efficiency and no one wants that.

Harshil: Right. Yeah, so it's a lot of automation of security controls just as a native part of the dev workflows, right?

Emmy: Yeah. You shouldn't have to reach out to someone to do a code review every single time, right?

Harshil: Yeah. And I think there's been a lot of conversations on similar topics. I think some of the security teams, they call it security guardrails or security paved roads or what have you. It's all similar in concept, right? Implementing automated security controls. So it doesn't present a cognitive overload to whoever is dealing with it.

Emmy: Right.

Harshil: So going back to the topic of software supply chain, I think we're privileged in a way that we are in the business of building and deploying software with the latest and greatest technologies. A lot of times people are, especially senior security leaders when I talk to them, they're dealing with many different aspects of it. Then the question comes in, okay, if I'm buying a SaaS application, is that a part of the software supply chain? If I'm reviewing a third party vendor, is that a part of the software supply chain? Because technically it is software that they're buying and using. And if it's a company like yourself, which is in the business of building software that millions of users are using, how is it deployed? Is that deployment mechanism also a part of the software supply chain? If somebody's building an app…. 21:20

Emmy: Like, when does it leave my supply chain and turn into someone else's supply chain?

Harshil: Yeah, exactly. I don't know if you have thoughts on it. What should be the scope of software supply chain? 21:30

Emmy: That is the key question in solving what is an SBOM and what's going to go into the software bill of materials. That has been like a key part of that conversation is, okay, so if I hand over my product to another organization and I hand them the SBOM with it, do I need to include the SBOM from the companies that I used or the vendors that I use? And like , where does it stop, or does it stop, right?

Harshil: There’s hurdles all the way.

Emmy: Yeah. But there's also mitigations. Like if you are working on your own supply chain, you put mitigations in place. You onboard vendors, you make sure that they're following whatever your rules of the road are, your guardrails are. That's what's kind of nice about this is what I really like about SLSA and what I like about SSDF is that it's providing a kind of a common language for folks to talk about when we talk about supply chain security. Because what I shouldn't do is get there and say, “Well, why did you provide provenance that looks like this?” and they’re like “I've never heard of provenance that looks like that”. And that's going to cause just basically the conversation of crumble and not go anywhere at all. Yes, it's all partial, the supply chain, the cones, but everything is part of the supply chain. Your laptop could be considered part of the supply chain but I think it's what can you handle as an organization? So leaders, what can you handle within the organization that you have and the money and funding that you have? How far can that reach go? And then where your reach stops, pull the wall. Put a wall in there where you reach stops so that you can control what you can control. You can't control those other things but they shouldn't impact you. So you either have scanners that come in, you're watching the updates for whatever group, whatever vendor you're using. Like, you're not going to be able to go and change their code, but you can put a stop in place that prevents, hey, if we're going to have to shut down this tool because it has a massive security vulnerability that hasn't been patched yet, we're going to be aware of all those things so that we're not introducing more vulnerabilities into our supply chain.

Harshil: Right? Yeah. So let's imagine if you're a security leader at a company, you're responsible for AppSec, ProdSec in your organization and somebody from the executive leadership, let's say the board of directors, they come in and say, “Hey, we don't know what software supply chain security is but it sounds important, do something about it”. Now, if you haven't done anything about it yet, if you don't know what that means, where do you start? What are the resources you go to to learn more about it? What would steps one, two, three look like?

Emmy: If you know nothing about supply chain security, start with what does your development process look like? So go back to understand what your company is doing. That's going to be the first step, it’s really understanding what your business goals are and your objectives and how those are being done now. What tools are being used, what processes, like what policies do you have in place? Are people allowed to stand up whatever server they want and take it down whenever they want? Is there any control in place for that? And then what you do is you can use SSDF as a framework to start from and you can go through and say, “Are we doing X, Y, and Z?”, and you perform a risk assessment of your own organization, say, and then find your gaps. Before you fix anything, find your gaps. Then you, as a security practitioner, look through those gaps and say, “What's the low hanging fruit?”. Because if it's low hanging fruit for you, it's also low hanging fruit for an attacker. Close those gaps first. And also the highest risk ones and see and prioritize what that looks like and roadmap that for the business. And for goodness sake, don't turn it into apocalyptic, like the world is going to end, but ‘Hey, this is what we should be doing, this is what we're going to need to do it. How can we schedule this in?”. Again, going back to that partnership model, start with the outside, what can others outside access first? And then you think about insider threats and look at your end to end of your supply chain. What am I consuming and what am I pushing out? So you want to really kind of capture that in a big giant rectangle. What's the top, what's the bottom, what's left and right? And then as you go through that, it starts to kind of map out what your trajectory looks like for supply chain security. And kind of what I was talking about was mostly system based, but then there's also how is the code moving through your pipeline, which is also very important. So logging those actions, making sure you know how the code is moving, what actions are being done, at what point are users interacting with that code and manipulating, and who has access to that code and manipulation of that code, and how are you verifying the authenticity. And then making sure the customers don't think that verification of authenticity has anything to do with security other than to say it came from me, right?

Harshil: Fantastic. That's actually a really good overview. I don't know if you've written about this, but you should write it down somewhere. This is great.

Emmy: Hahaha. Throw this in the ChatGPT and make a blog. No I’m just kidding.

Harshil: Hahaha, yes. So I heard you mention SSDF. For those of us who don't know what SSDF is, for those in the audience, elaborate more in terms of what SSDF is and what does it do.

Emmy: Secure software development framework, I believe. You say an acronym enough, you forget what it actually means, but it is a framework for how you develop securely. In that framework, it involves what you can do with code, like coding standards that you can apply, but also includes supply chain security because it is such a blend when it comes to it. And it just came out this year, I think? It's pretty new.

Harshil: Yeah, 2022. Last year.

Emmy: Oh, right it’s 2023. Yeah, and there's a couple of different - it's NIST, right?

Harshil: Yes.

Emmy: Yeah. And there's a couple of different overlapping, SLSA directly aligns with it so you can map every SLSA best practice with an associated SSDF control.

Harshil: Yeah. And I'm hearing there's some talk about federal agencies making that sort of a contractual obligation for suppliers that are providing software to the federal agencies. I'm not exactly sure when that's going to happen, but I think that's going to happen at some point.

Emmy: Yeah, I think that's the intention right now, right? One of the big problems is the supply chain. A fun stat is the 742% average annual increase in supply chain attacks for the last three years. So every year there's been a 742% increase in attacks, which is a lot. So the US government at least is, I think, pretty worried about that, as they should be. And so there is a lot of government regulation coming out about it now. They are working pretty closely, it appears, with organizations. Like, Red Hat's involved in it - and making sure that they're clearly defining and articulating what critical software means, for example. So one of the things was, “Hey, you have to have an SBOM if you are considered critical software” and that was a whole entire working group that had to define what critical software meant because the first definition that came out was literally every software. So it was good that they went down and kind of dove a little deeper into that. Yeah, it's the direction it's going, who knows when it's going to happen for sure though.

Harshil: Yeah, we still have to figure out what we are going to do with all that SBOM that we collect at some point.

Emmy: Yeah, I mean it's not the human readable versus machine readable and setting up your organization to maybe having a minimum criteria that the SBOM have to run through in order to onboard a vendor into your supply chain. There's various things that involves automation and computers that can think faster than we can.

Harshil: Fantastic. So my key takeaway is the first step if you want to do something about software supply chain security, the first step is let's just start with understanding how your teams are developing software in the first place. And that is I'm surprised at how many security professionals don't know that or they have an assumption that is actually not factually correct, so it's pretty important. I mean, it's kind of obvious in a way that security teams are not very well staffed. Typically we have one security person for every 100, 200 developers. So it's really hard to keep up with all of that anyways. But yeah, that's a great first starting step, leveraging SSDF because it has a lot of supply chain related recommendations, frameworks requirements, however you want to call it, those parts are included in it. That's a good first step resource to understand it, and then step by step implementing the controls that make sense for your organization depending on how you're building, deploying software. What are the inputs and outputs, that big rectangle that they should draw, as you mentioned. Fantastic. Emmy, this has been such a fun conversation. I hope we can get you to this podcast again, because as software supply chain security develops further, as SLSA framework goes along further, there's going to be a lot more conversations about this. It's fun to see the new evolution of where AppSec is going, or where security software development is going. It's been a fantastic conversation. Thank you so much for your time.

Emmy: Yeah, thank you. Thank you for having me.