EP 25 — Navigating the Complex World of Software Supply Chain Security with Schneider Electric’s Cassie Crossley

Harshil: Hello everyone. Welcome back to another episode of the Future of Application Security. Today, I have a very interesting guest with me here. Welcome Cassie to the show.

Cassie: Thank you for having me.

Harshil: Cassie, now, I would love for you to introduce yourself to our guest. I think this is going to be a very exciting topic today. If you don't mind sharing just a little bit about yourself, what do you do, where do you work.

Cassie: Okay, I am a vice president for supply chain security at Schneider Electric. We are a 34 billion euro company headquartered in Paris, France. We specialize in energy management and industrial automation solutions. So it can be products in any manufacturing line in critical infrastructure. We even have the brand of APC. Those that are working in data center environments may recognize the APC brand and also Square D for electrical components. So a very big company with a large footprint of nearly just even smart products. I would say, they are over 20,000 smart products and most of them are connectable.

Harshil: That's a pretty large footprint of IoT things and smart products. And, of course, Schneider Electric. Schneider as a corporation has presence in pretty much every country on this planet.

Cassie: Yeah. 130,000 employees spread over 100 plus countries.

Harshil: That's phenomenal. How large is the security organization?

Cassie: Security organization just in the direct security organization is over 100. And then when you expand out to our even dedicated and then part time security folks, it's several, many hundred around the world. And even the folks that are doing specific IT topics or other areas, they have quite a number of security champions. Some security advisors, digital risk leaders, and we even have cyber services for a large OT environment around the world, whether it be in manufacturing or critical infrastructure.

Harshil: That's awesome. So now Cassie, we've got a lot of my previous guests on this podcast. They come from tech companies, SAS companies, cloud native companies. And what Schneider comes in with is a very different type of risk surface, very different type of soft threats that you have to worry about. Do you mind sharing some things that make Schneider or securing Schneider in your context a little bit different than any other typical tech company?

Cassie: Sure. Well, we absolutely have all those other things that you've mentioned. We have large cloud offerings, many software products in our eco-structure platform. But what makes us unique is not only is it IoT, but it's OT operational technology, which means that it's running 24/7 round the clock, running for years and decades throughout the world to keep just some of the different businesses and organizations and our infrastructure going. And with this kind of footprint, that means that a lot of the products are not immediately updatable, such as you might have for an iPhone app, or you might see on just a Windows platform with auto update features. And what this means is that the responsibility for the customers is just as important. They need to be able to secure their environment, have the correct compensating controls, because these OT products may be existing on platforms that are a bit older and you need them to operate with each other in these environments. Like a manufacturing floor, it's not uncommon to see products that might have been installed in the early 2000’s and still running with great speed and efficiency. And those are very important as a holistic, because anybody who's in security can understand that there could be compromises in the IT layer that can spill over into an OT environment. So IT and OT convergence is an area that we're heavily involved in and have been for before it became a buzzword.

Harshil: Right, right. Yeah, and I can imagine if the devices or the products are staying in their environments for many many years, many decades in some cases, then it would be absolutely very crucial, especially in these types of environments, crucial to manage the risk effectively. Because the things that get introduced, I can imagine it would be hard to mitigate, or it's not as easy as pushing a new AMI in the cloud, right? It's a little bit more difficult than that.

Cassie: Right.

Harshil: So then your title is also very interesting because now you're focused on software supply chain within the context of Schneider Electric, which is phenomenal. If you can tell a little bit more about the scope of your role, what's your objective, what are you trying to solve?

Cassie: Sure, sure. I've been passionate about the software supply chain topic for a number of years, actually, before SolarWinds. My background is I was a developer and developer a long time ago, but I moved between development and IT, so I've been in all the different technology areas in different parts of different companies throughout my background. And what I recognized, especially moving back into cybersecurity - I did work at McAfee in the mid 90s - what I recognized when I moved back into cybersecurity about nine years ago is that the cybersecurity functions, while a lot of it's focused on the IT side, is understanding the development side, and for example, how to set up labs. Because even before we had really cloud native organizations, you had build servers, you had development servers, people were spinning up instances and things like that. And I remember even just with WannaCry, people would be like, “Oh, well, we're still seeing it” because you only spin up that dev environment, for example, for it's based off of an image that you only bring out when you're doing a test, and same thing with labs. So I've always been very specific and directed toward how can the developers, which I was one of, do their job but also be secure both in their platforms but also of what they build? Because there is an overlap and a risk that we saw in something like SolarWinds that makes a significant difference. And generally IT groups tend to keep a hands off, they don't want to adjust the development environment. The development environments are generally owned by the development staff and accountable to the dev leadership and vice presidents and such. And what that means is that it becomes a bit of just a gray area or black hole, or whatever you want to refer to it as, where things can happen because they're not being logged, they're not being monitored, there's not an awareness. I mean, it takes a lot of background. Even just having, for example, if you want to do logging to make sure things are running from a trusted time server. You know, are they all connected? So those make a big difference for those of us if we're having to do forensics or anything of that level. And so supply chain security first on the software side is understanding that entire area. I wrote first a chapter for a digital book, that then I presented to O'Reilly Media and I'm publishing a book at the end of 2023 on software supply chain security because it's so much more than just, while I like what Google SLSA has come out with, it's much more than that, because there's so many different pieces and you have to look at every tool that's being used as part of the environment because it could create a compromise. Especially as developers, we love open source projects. We love open source tools. We bring them down. How do you know that open source tool unless you examine all of the code in it and when you built it doesn't have a keylogger hidden in it? Did you read all the code? We just don't know. And we have to treat everything that we bring down and are not developing proprietary that it has some level of risk associated with it.

Harshil: Yeah, it's about time somebody wrote an authoritative book on software supply chain because as we were discussing earlier, there's so much confusion about what it actually means. I talk to so many people and I hear all kinds of different opinions, and I got to tell you, like, to be transparent, a lot of this I can blame it on the AppSec, product vendors as well, because every company is trying to push their own narrative, right? So the SBOM companies will say the software supply chain is SBOMs, right? CI/CD security companies will say, the software supply chain is CI/CD security. Well, I think the market needs more consensus and alignment into what it actually means. But I'd love to hear your definition of software supply chain security. What is it?

Cassie: I consider it the cradle to grave risk that can be introduced from the moment there is a potential design or IP, or just bytes of code all the way until it's decommissioned and out of service. So in my, what I'm writing about past topics about people, how do you manage it, even just as simple as social engineering techniques. And I present it with a focus of what does that mean to a development team? Can they be socially engineered against, it's not just the financial side that you run into, it's any kind of manufacturing. Let's say that if you're developing an IoT product, because there's more and more every day, right? So if you're developing the next IoT light bulb, what about the gateway that you're producing that's part of that? We've got more than just the SBOM. I'm definitely involved in the SBOM community. I'm definitely involved in CI/CD pipeline topics. All of that is I think it gives you some transparency at different angles. But I'm concerned, and I simplify it down and say, if a one and a zero has been created, does it stay a one and a zero through the entire process? So that means that you have integrity throughout that process until it finishes. And that means the data, I'm covering IP topics, intellectual property, there's a lot of data breaches happening, and there's internal threats also that can happen both from human error, but also, unfortunately, it can happen where it's on purpose, and we have to be aware of that. I'm looking at the risks over the entire area. I introduce some of the frameworks that are out there, and a year or two from now, there's going to be new frameworks and new areas such as the NIST SSDF, which is a software secure development framework or secure software development framework. And that was a great improvement of what they did. But the average application security team is not going to go through and read that. The average CISO organization and let's just say third party suppliers - I can talk about that in a bit - they're not going to read every single one of those unless you have mandated FedRAMP in the US or some of those other ones. I work in a global company, I'm not expecting someone in Spain to go read all the US documentation. So where do they go to understand that? And so part of it is just a set of controls that I have in there where people can then expand and tie it in. But it's making, especially on the development side, if you come from an IT background and you were in charge of the networks, the network access control and all that, you don't know how a developer thinks and how they go through their processes and what do you need to be aware of? So I’m trying to highlight all of those things. I guess the best of in various different documents. But also everybody's risk is different. So third party assessment risk, I get questionnaires every day, hundreds of lines long from dozens every month and what are they doing with those answers? So I talk about in the book and what I'm looking for is my question and what I'm looking at is very different if it's an internal application or if I'm building it into a product.

Harshil: Right.

Cassie: And the same kind of conditions we need to look at for open source code when we're including it. Are you scanning it? Are you doing the appropriate tests? You can't add secure development life cycles after you've built it. So what are the mitigating controls that you have in place to make sure that you're scanning and appropriately looking at items and addressing it based off of that risk? Every risk for what you're using could be different. And that's what I'm trying to do when I talk with people all around is talk to them about it, is that you have to understand what your responsibility and accountability are, but then give them some definitions and insight. I mean, even the concept of provenance, which is changing every day, some people and governments interpret it one way for a definition. Others consider it where the person and developer was. And then you've got open source, you can't tell where everybody is. I like to comment that if you've got a Windows platform, you've got some of my code because I worked at a company back in the late 80s that sold Paint.exe to Microsoft. So there are lines of my code still in existence because as a community, we're not deprecating our code. We've got a lot of code bloat and that's going to continue until things like software transparency and SBOM and really how do we remove some of that code and really look at the lines of code that removes.

Harshil: So it's interesting that you mentioned NIST, SSDF and some of the other frameworks because if you really think about it, let's just take SSDF as an example, is that fundamentally different from the software security maturity models that we had before? Whether you take it BSIMM or OpenSM or some of those other frameworks and is SSDF or similar things, a newer iteration integrating some of the other newer concerns like code integrity for example, or provenance and some of those things? Or are those just completely different ways of looking at software security? What's your thought on that?

Cassie: I think it's different ways. I've done the BSIMM internally. I've had that company that released it come out and do it and it gave a lot of great feedback. It's a good feedback mechanism, good to see where you are comparison. So I don't consider truly BSIMM to be a maturity model. It's a maturity comparison to others in your peer space and with SAMM and the OWASP in different areas, I think from a maturity that gives good insight, but the one thing that SSDF, and I do like the controls in SSDF, is Schneider Electric. We are following IEC 62443 and that is an industrial control system, set of framework. Both have an SDL portion of it, which is -4-1 part of it, and then it has other areas like product requirements, how you actually maintain the systems and so on. So 4-1 before SSDF was to me the most robust set of security requirements from a process standpoint. And from there, along with something like SafeCode, has some good requirements and controls and SSDF has, let's just say, put it toward it's now accessible. 62443, you need to purchase it. Some companies, and it's quite extensive, SSDF has a set of requirements and controls that I think are pretty modern. It has some cloud activity, cloud items in there. So I consider that, I used to send suppliers, if they did not have, especially smaller suppliers, over to Microsoft SDL because it was free, it was available, it was documented and that was the way to go. But I do think that SSDF along with the Microsoft SDL, most companies can build really that layer that they need to. In fact, SSDF has a few things that I'm sure will get added into 62443 based off the build management. I mean, some of the build management pieces in SSDF are very specific because of the risk that we've seen through some attacks and techniques. I think that it will continue to evolve. I don't know what speed it will be, but I think from a foundation standpoint, it's got the informative references which lists where people can find more definitions and details. I definitely think that out of all of them right now, SSDF is the most current. And if you are working from a development standpoint where you're trying to build a set of controls in your organization, it has some good foundations. Not the full supply chain though. It is very focused on development and really doesn't have as much of a precursor in some areas. It doesn't cover third party assessment management at the level of some of the other NIST documents, which again is very exhausting for companies. But that's my opinion. When it comes down to a straight dev shop, let's say you're a smaller company or you're a mid sized company, a lot of what you need to do, you talked about vendors earlier, is seeing where some of those vendors fit in with the overall processes and techniques. You have to have policies, you have to have procedures, but it's all part of having the right tool sets in place too because you can't do everything by hand. So if you were trying to do everything that's listed in SSDF, or even when you're doing and evaluating some of the gaps you have in a BSIMM assessment, you're going to find that, well, if I have a tool, that's doing software composition analysis or just SAST and security testing tools that can alleviate a lot of the controls and help with some of those controls in those platforms.

Harshil: Right, right. So if you're a smaller security team that's doing some of the basic stuff like using some of these scanning tools to look at code, they've got them integrated into their CI pipeline. They're checking and testing all the PRs and things like that. Should they be embarking on a journey towards a more SSDF based adoption, or should they still be looking at a separate maturity model for broader application security?

Cassie: Yeah, I haven't seen the maturity models really be able to help mature organizations. I think that from what I've seen and what I've had, maturity models are great if you are trying to assess your internal organization against other groups. I mean, I have 500 or so projects that can be running at different times. I've got products being released all the time. Maturity models are good to evaluate products compared to each other. They're good for evaluating lines of businesses, business units in that level, and to see where your peers might be and where you want to continue maturing. But if you want more of a step by step guidance, I don't think that maturity models give you that. Because you don't know, you may not, I mean, there's pieces that you can totally ignore and there are pieces for example, a good CI/CD pipeline is automated. What happens when you have a company like mine where the CI/CD pipelines are all different for different products? Because you might have been different companies that were purchased and they execute differently. Some companies that we purchase are cloud native and so their CI/CD pipeline and how they go through are very different. So overall you're looking at just the control level, but the maturity may be much higher because of the level of automation. I mean, just because you don't have the technical debt from tools or processes or even just code. And you make an interesting point. Just for a small company, I think, again, looking at the risk when I'm evaluating suppliers, I'm going to look at the risk completely differently if I'm evaluating a cloud supplier versus if I'm evaluating someone who's creating a library where I have the accountability. So it's a bit almost a warranty. Am I responsible or is someone else accountable? And for anybody working with, especially a small company, you're working with a lot of open source. One of the key things I'm looking at in any of those is are they addressing and remediating the risks and can they explain the risks that they're seeing in there? One of which we've seen for the last couple of years is your architecture. That's what things like Log4J create that discussion of, do you really need it? Is there a better component? Well, probably this was the best choice at that time. Is it still the best choice? And it's creating that conversation that dev teams, when you're working on and having been a product manager, you're working toward the next feature and not concentrating on the tech debt. And so I think that cybersecurity is going to help us in that, being able to keep that as part of our sprints and really being able to focus on it as we continue evolving products for each, whether AppSec or whichever.

Harshil: Right right. That makes sense. I mean, I think the closer the collaboration between the IT dev teams and security folks that are championing improvements in certain areas, it’s obviously better. That's a no brainer. We've been talking about it for years now as a security community. But specifically looking at the topic that you're writing a book about, which is software supply chain security, where do you think is the industry, your peers, you talk to so many people, where do you think the general current state is for software supply chain security today, and where is it going over the next few years? If you can look at the crystal ball?

Cassie: Sure. I think there's a disconnect right now between what our customers are expecting and what the general development teams around can provide. I think that part of it is just unfamiliar with how development works. And If you hear someone say, “Oh, well, we shouldn't be using open source”, well, those of us in the industry that have, I mean, a lot of these are well tested open source projects, much better than can be written from a proprietary standpoint. They save a lot of time, they're rapidly being updated for any security risks. I think that there's a disconnect in what the customers are expecting for us to, as producers of software, for us to say we produce this software, so I think the key aspect of it is being able to say I'm verified and I can claim that the source has been vetted and examined and tested at this level and I will be the accountable party if something happens. It doesn't mean that I know where every line of code was written, when it was written, what tools were necessarily used to build certain objects originally, but from a standpoint of “Here's what I'm going to commit to and be responsible for”, I think that's where we have a way to go where there's a disconnect. And having been involved for a number of years now in the software bill of materials conversation for SBOMs, it doesn't give you the answer of whether or not you're impacted. That’s really what the customers are wanting to know is, am I affected? They want to know the second they hear something in the news or see it go across their newsfeed, they always want to know, is this going to impact me? It's not because they want to go and look at it themselves. It's because as software producers, we can't answer that in near real time. So future wise, I mean, I'm involved in a lot of software transparency discussions, vulnerability exploitability and the vex discussions that we have going on, what this means is in the future, can really the CISOs and the IT groups trust the information that they're getting from different platforms? Do the asset management tools have the answers to determine based off of my situation? Like if you're looking at CBS, there's the scores that are now off to the side of, I don't know your environment. If an environmental score is adjusted because this is part of your critical process, then asset management tools, the soc tools, all those tools should be able to use the data and the answers that supply and software publishers give to them to make the decision. That's where I see the future state will be, is that the moment the CBE is released, how quickly can we give almost a real time answer, which is difficult because you have to examine a lot of code and a lot of situations and use their test frameworks to see where that's going to happen. I just want to bring up like OpenSSL. We knew for a couple of weeks that something was going to come out, right, it was going to come out on November 1 of last year. We took that time, and I'm sure many others did, to find out where OpenSSL was. We did not have what the CBE was, but knowing and using the SBOMs to be able to say, “Okay, dev teams, it's not a Friday night at 5 pm that this is coming out. Let's be prepared to do the analysis and come out quickly”. And so that's what customers are expecting. They're not going to be able to always say, I'm affected. You're not going to have simply a solar wind saying shut down this component. It's going to be much more nuanced. That's where we need to help with that nuance to get faster at it in the future using these other areas like asset management and other tools, AppSec, tools to say, “Yes, we've now been alerted that this is impacted or it's not an impact”. And that answer may change a bit, but then they can focus on something that they know for sure is impacted. So our job is to give actionable events and help that process for CiSOs and business folks throughout the world so that they can make the right decision as quickly as possible.

Harshil: Right, right. If I was to summarize your recommendation towards the end, would it be accurate if I say our job as security professionals should leverage a lot of the context behind what is this piece of software, and how critical it is to the business? And then combine that with the vulnerability, CVSS score, what have you, to really focus on what matters, and have the dev teams focus on what matters.

Cassie: Yes.

Harshil: Obviously this overall framework of risk prioritization has been around for a long time, but what's different now is the fact that it's not just the software that your developers are building, it's software that's coming from multiple different, potentially unknown, untrusted sources, and going through the pipelines in a very dynamic environment, which could introduce more risks. It's being deployed very rapidly, which also potentially introduces other types of risks. So the fundamental framework of calculating risk, which is leveraging context and the raw vulnerability scores, that hasn't changed, but where it applies, that has changed, right? Just because you don't know what the software is that you're using, you don't know where it's going, you don't know how quickly it is being released and deployed.

Cassie: Yeah, that's right, because in some of these environments, let's just say a normal building, but what if it's a hospital? Every situation is going to be different. How they set up their perimeter defenses could be different. They're all going to have their own situation of where they find the risk in it. So yes, I agree with your summation. That's a really good way to put it.

Harshil: So is it reasonable to assume that this concerns pretty much any organization that builds software or do you think it's applicable to a subset of companies?

Cassie: I think it's every organization that builds software and also consumes software. So I don't see anybody that's not affected in some manner. Even if you are an organization that has never written your own line of code, but you are a consumer, then you are impacted by application security up the line, right? I mean, everybody has some form of email platform, right? So I don't think there's any difference. Now from an application security standpoint, when you're building lines of code, there's a level of process and rigor that you need to even put on low code platforms, right? What about API layers? I'm getting quite involved in AI threat modeling and how do we manage the models for AI, let alone the changing environments? So whether or not you're using a predetermined AI data model, that's part of the supply chain, either you create the data model from scratch and so you are source zero or you're getting it from somewhere else. So this is going everywhere. I mean application security, that's not the only part, even of software supply chain security and just supply chain security because it can happen at any point in time along the pipeline. Even just distributing the code, as we've seen, there's lots of man in the middle attacks. We all know the OWASP Top 10 and such. When you get to the point of how are we going to control the AI layers, how are we going to control the ChatGPTs, all of that is going to make a difference because that’s data, and I think the data aspect of supply chain security is going to be as important. We think data and we think privacy but how you're producing things honestly, I mean, source code is data and IP. We have to be worried about that just as much as anything else. So source code, governance, it's all there.

Harshil: Yeah, those are very interesting points, and I think it makes a lot of sense that those things need to be addressed because I can totally imagine the threats becoming so much more relevant now. On the other hand, when I talk to a lot of the CISOs and people who are making budgeting decisions, I've rarely seen people allocate or most companies, except for some very specific companies, I've rarely seen security teams allocate budgets towards software supply chain security as one of the top three things that a CISO has to worry about. Because if we expand ourselves, our spectrum, as a security leader, as a CISO in charge of a large organization, you have so many things you have to worry about. I don't know if software supply chain security is still in the top three. Would you agree to that?

Cassie: Oh, I agree. Yeah. It's only really concentrated at this moment on a lot of critical infrastructure or when there's geopolitical discussions going on. What that means, though, is everybody in some manner is affected. Anytime you have a CVE in your environment, it's somewhere related to a software supply chain problem. I think it's just a different way. They're already doing patch management, vulnerability management. They're addressing those concerns. But I want to say they may not consider it their top three, but if they're evaluating suppliers, then it is a concern for them, because otherwise we would just take it all as it comes, right? And we know that's not happening by the influx of questionnaires and tools to evaluate risk. And what's going to happen, though, is you can't evaluate every single supplier at the same level. Microsoft will not fill out questionnaires, large companies will not fill out questionnaires. Do they have due diligence and SDL? Do they have SOC 2 reports? So there's other ways that we as a community have been able to evaluate software supply chain security, but there are ways that are sort of avoiding the discussion. So now you're seeing, especially in critical infrastructure, they won't take items. Some of them are requiring full pen tests. Does every product right now get pen tested in the world? No, they don't. But for certain environments, you're going to see that more and more. I do think that CISOs are going to start to at least acknowledge as part of their software, as part of their supplier risk management practices, that they want more of the trusted vendors. Somebody that can say, “I've done SOC 2, I'm this and I'm that”. I think that's a ticket to play for most cloud vendors. What about those that aren't having cloud products? How are you assessing IoT? So the US government has been working, Singapore was the first to come up with an IoT cybersecurity baseline or labeling plan. And NIST has an IoT baseline and they're working toward a labeling plan. There's going to be that Good Housekeeping seal mark that the CSOs can begin to rely on as these concerns are elevated up beyond the critical infrastructure.

Harshil: Right, right. Yeah, and what I'm gathering is also some parts of the broader software supply chain security discussion are already being done by security teams, right? You talked about vulnerability management or third party supplier reviews. All those things are being done, but it's not put together as pieces of this jigsaw puzzle called software supply chain security. And there are some missing pieces, obviously, but these frameworks help us put those pieces together into a nice picture that we will eventually start calling a software supply chain security. Fantastic. So if you have a person who's asking you about “Cassie, what do I do today? My board just asked me to do something about software supply chain security”. What would you recommend that person do today?

Cassie: I say concentrate on your third party suppliers because you're only as good as you're upstream, and concentrate on what you're taking in. So open source, so putting in the right just checks and balances through tools and making sure that you feel comfortable with what you're bringing in and being then responsible for, that's the key thing. You can't do it all. You can't query every supplier and say, how are you managing your source code? That's just too much. But if you are responsible to your board on how you're going to present to your customers so it's whether you're having to make sure that the platforms your teams are running on are safe or something that you include and build for customers, it's the same conversation. Do you know what's going in? How well are you making sure that you understand that by examining the risk and then making the right decision. There is not going to be vulnerability free software. It just is not going to happen, despite what some people write down in laws or regulations and such, it isn't going to happen. We're going to always be confronted with more intelligent attacks and techniques and we have to respond. But the best way to do that is by prioritizing what you can do to make the biggest difference, both for your internal systems, but also as a responsibility if you have customers, whether that be through the proper data controls. It matters on your business, not necessarily what's in an SSDF or something of that nature. So that's where I tell people to focus on, is know what you have there and that you've scanned it and evaluated it. And that is one of the most key pieces for understanding, because then you know where your risks or gaps may be and then you can do something about them.

Harshil: Phenomenal. That's a fantastic piece of advice, Cassie, and I cannot wait to read your book on this topic. Thank you so much for being a part of this podcast.

Cassie: Yeah. Thank you so much for having me.