Ep 23 — Martin Nystrom: How Lumen Scales Product Security

Harshil: Welcome to another episode of the Future of Application Security. Today we have Martin with us from Lumen. Martin, it's such a pleasure to have you here.

Martin: Thanks, Harshill.

Harshil: All right, so let's begin with a really quick introduction. Martin, I would do severe injustice if I was to introduce you and your profile. I would love to have you talk to our audience about what you do today, how you came to this point of your career, and where do you work?

Martin: Okay, thank you. So, Lumen, we are the largest network provider in the world, and we provide both networks and security outcomes within those. And I'm a security guy, so I came to Lumen to lead our architecture, engineering, threat intelligence and operations functions to build this for customers. The way I got here was I was at Cisco for a long time, and at Cisco, the last thing I did was I led our XDR platform. I built our XDR platform, it's called SecureX, bringing all these security products together to find threats in customers' environments and help them hunt through it. And I came over to Lumen to kind of see the other side, which is, okay, it's one thing to be a product leader, now let me come over here and see what it's like to actually own architecture, engineering, threat intelligence, and build it, right? Build it for customers, working with product. So I would even start going back. It's painful for me to say this, but it's almost 20 years ago that I started getting involved with security. My job was really to basically secure applications that we’re writing. Make sure that there's no SQL injection, cross site scripting, some of the old gems that we used to deal with. And I had those skills because I was a Java application developer and I knew architecture. So mostly my job was to slow down everyone else's projects by making them do security. And then I saw that all the cool people were there working on incident response, so they're fighting bots, dealing with problems. And so I raised my hand and started dealing with that in the middle of the night, those threats. And sometimes those threats ended up being Application Security attacks, right? Like successful SQL injection compromise. So I started writing papers on it to help developers know how to secure their applications. And over time, I ended up building out our incident response functions, our global SOCs, all of our policies and architectures, and then went on to build a managed security business for Cisco, which had SOCs all over the world and was making a multimillion dollar solution, and then started building our platforms for SOCs and Application Security development. Well, really, that was mostly around SOC and Application Security. It's way easier to just find a Windows vulnerability and compromise that than it is to break in through the front door and application. So that's a lot of where I spent my time. So Harshil, there's a bit of my background for you.

Harshil: Fantastic. For those of us who don't know what Lumen does, can you elaborate more in terms of what made you interested in coming and doing this for Lumen? We were talking about this earlier before the podcast, but the extent of the data that you can see in Lumen and the interesting things that you all are doing over there.

Martin: Sure. So let me start by saying when I got the call to come over to Lumen and lead our product security organization, what fascinated me the most was this notion that you have this massive global network and all of these capabilities and all this multi gig fiber that we can deliver that with this visibility, we can do some extraordinary things. One of the cool things that we're doing is we're taking all these central offices. Do you remember central offices? We learned about them in school, at least I did, where you're routing phone switching and stuff, and we turn those into micro data centers, maybe little edge compute data centers that are within five milliseconds of 97% of all US businesses. So we can then deploy workloads into that space. The natural question is, all right, what are the security opportunities to layer into that with workload protection and RASP and all kinds of cool stuff that we can build, as well as SAS and other things I'm building. The other cool thing I lead within my organization is Black Lotus Labs. You take all this data and you eventually realize that we see 99.7% of IP addresses that are flying around in the internet, and from there we can apply, essentially, netflow capture. From there, we take that capture, we run that through our engineering team that can run machine learning algorithms to figure out how are these threats behaving in the network. We can actually interact, discover what privileged intelligence that we can apply to that to take threat actors and look for them in the space. The actors, as well as in some, I would say public sector spaces, without being too specific, we can do that in some highly sensitive areas, but we also turn that into indicators of compromise, IOCs, which means we're in a policy engine, slow down your application and make it perform.

Harshil: So, Martin, thanks for that overview of what Lumen does. It's phenomenal the amount of footprint that you all have and the analytics that can be derived from it. I'm super curious, what does Black Lotus labs do? A little bit more in detail. You mentioned the high level overview, but considering this is a very unique set of data that you all are positioned to be able to access, what are the benefits that Black Lotus Labs gives to its customers?

Martin: I think first and foremost we have, I mean the simplest thing to understand are the indicators of compromise. The IOCs, these are just atomic elements of IP addresses, DNS names, file hashes that can be just placed in front of any of the security solutions we offer and just speed up your protection. We update those every few seconds and so that keeps you protected. And I would just say one of the chief advantages there because this already comes with our Secure Application Services Edge solution, with our SASE solutions, it's in front of our DDoS solutions and then we're adding it to others as well. And it's really simple to understand. You don't want your policy engines all gummed up, evaluating files, trying to see if this is something they've seen before, even detonating in a sandbox before it decides what to do with it. So just blocking that traffic outright, it gives you a huge advantage to just keeping it clean, keeping the pipes itself clean. But the way this works is we're collecting netflow from everything we see. And as we collect that into our back end, we're then applying algorithms across that to figure out where this is headed, right? Where are the actors moving and how are they changing some of their techniques? We then can actually interact with those bots and learn more about where they're headed as well as what they're targeting their next victims, right? So in many cases, we actually know, based on some of the adversarial knowledge that we have, as well as what we can reverse, we can actually see where the adversaries are headed, like who they're targeting. Right. Which enterprises they're targeting. Which gives us a heads up that they're going to need protection in advance of the attack actually happening. And this just requires a team of analysts, a team of data scientists, reverse engineers, and then also we have people who do outreach who can help analyze, understand and publish about this on our blog, as well as on Twitter and other places to help the world know about what we're seeing.

Harshil: That's phenomenal. And what is sort of related is the title that you have, which got me super interested as well, which is related to product security. Now tell me more about what product security means to you and in your role. How do you define that?

Martin: I think the way I would answer it is this. We use the term product in front of security, there to differentiate between an outward and an inward focus. You have the chief security officer whose responsibility and organization is there to protect our footprint, to protect our own internal network and our customers with a focus on finding threats that are aimed at Lumen and Lumen customers. Product security is about building solutions that benefit our customers within and that they'll pay for, that they actually want to sign up for and pay for as an outcome. And that's why we created that description of product security. And sometimes it's a fuzzy line between the two roles, but we are back to back protecting both the internal network as well as the external network and our customers' offers.

Harshil: Right? So effectively the work that your team does is monetizable. You could sell it to your existing customers, newer customers, as a product. So you're effectively building security products. Now, I've talked to many many security teams and CISOs who are struggling with this differentiation, really, because fundamentally these are different things, right? Whether your job is to secure the company, its assets, its data and its customers, versus building products that you can then sell as security capability, security products or even security features. And it requires different types of skill sets, different types of operating function to do both of those things, at least in my opinion. The unfortunate reality is that a lot of the Chief Security Officers, they don't come from a product building and selling background. There are different things that are needed, things like product management, things like collaboration with sales and marketing and all that stuff. That's not the bread and butter of most CiSOs. So if there's a CISO who is looking to understand how to really differentiate those roles, or what are the differences between these two functions? One is the more inward looking responsibility of protecting the company, its data, its assets, its people, its employees, its customers, versus the more outward facing role where CSOs or the team is actually building security products. How would you differentiate them? What are the key things to think through if you're trying to solve this problem?

Martin: I think it's probably best to delineate this on the OKRs, on the organizational key results. What are your top line goals as an organization? If you think about what Chief Security Officer is measured on, it's really about risk, right? It's about risk and usually underneath risk, it's about a maturity evolution. So taking your measured maturity, and there's a term for that and it's not coming to me right now, but your measured, recognized maturity and building toward higher maturity that you can demonstrate that then helps the board understand that you are mitigating risk and you're bringing down risk. So the investment they're placing in the security organization is being repaid in a way that you could calculate with a return on investment, reducing that risk. The product security organization is really measured on revenue, right? Sales, revenue, earnings, margins, those kinds of things. How efficiently, first of all, are we building the right things that customers will buy? And once we do that, are we doing it in a cost effective way? That delivers the margins that our investors want to see. That's really what we're measured on, which means we're selective. We're going to end up building and offering security products that the market wants, even if it doesn't necessarily check a box on the maturity framework, right? Even if it doesn't necessarily bring down risk. If that's what customers want to buy, then that's what we're going to build and we're going to offer it if it takes advantage of our native strengths.

Harshil: Yeah. So that's a very easy way to think about it. Thanks for making it super crystal clear to our audience. This is great. Now, if you compare your with a traditional Cecil.org, what are the key differences? Whether it's structure, people, what have you?

Martin: Well, first I would highlight that there's a lot in common. So we work intimately together. For example, as we become aware, Black Lotus Labs becomes aware of threats, we pump that into our internal Sock and Incident Response Team to make sure that we protect them. We also collaborate with them to hunt and resolve these threats as they come in. Because there is often an overlap between what happens within our corporately controlled network and where our customers are so that we can then rapidly protect them. So the customers get a great outcome. They just get security by default because our organizations work together, right? And we have a really nice ability to bring people back and forth. So you take a great security architect who knows how to build Application Security frameworks. We can then bring them over to help us build products. We do this all the time, right? We cross them over one another. You actually do find that a lot of the roles can float back and forth just fine and they work great. I think where you're going to see the differences in leadership. As you move up the chain in leadership, there are just different focuses, different outcomes. Whereas the CSO organizations have to get really good at framing risk, calculating risk, and also going to make investments based on risk reduction. So they're going and asking the board for money to reduce that risk. In my case, we're looking to make investments based on projected revenue. And so knowing how to build a spreadsheet that shows this is how much they need to spend on this quarterly basis to bring in this revenue, here's what the total addressable market looks like. Here's how I know I'm going to pull it in. I think when you start getting to director and VP level, the difference really shows up there and it shows up much less so in those that are building the solutions.

Harshil: Right. So now if you're at a company where there is an existing Cisco organization, obviously a smaller company than Lumen’s size, but they don't have anybody, or they don't have a team similar to your team or product security, but they're trying to explore how to build security products, sell to their customers on additional capabilities. What are the things, what should a CISO org do if they are responsible for doing this? Who should they partner with in an existing organization to build out certain capabilities and feature sets of what we should call this product security?

Martin: So Hashil, I think the biggest gap that I've seen in a CSO's organization is a knowledge and appreciation for product management. Just old, hard product management, knowing how to look at the market. What does the market want? What will it pay for? Because what we tell ourselves as security professionals usually is we have an ideology, right? Like the bad guys, me against the bad guys are going to build these great tools. I need all the money you could ever possibly give me to build these tools, and I will make you the most, absolutely the most secure, right? Product management is going to say, no. We know what the market we talk to customers here's, their willingness to pay, they're willing to pay for these specific features. This is what we need to build. The absolute critical element here for a CSO organization, hire a great product manager who knows how to address the market and knows how to prioritize features, knows what willingness to pay looks like, and can get those features to market.

Harshil: That's phenomenal. We went through a similar exercise in one of my previous companies where our CISO organization was asked to build security capabilities because our large financial banking customers were asking for it. We tried to do similar things where we realized that, okay, product management is not something we do. We don't even know if customers will want the things that we want to build. The leadership recognized it, and we tried to go down the same path of hiring a product manager. Very quickly what we realized is, because we are not a security company, most of the product managers who built and delivered and sold security products, they were not willing to join us just because it was not a security product company. So I 100% agree with your direction. For us, in one of my previous companies, it didn't work out because of the situation in the environment over there. But what we ended up doing is we hired a senior engineering leader who had built security products. That person was wearing the hat of a product manager, talking to customers, like, you're a poor man's product manager, not really a product manager. At least that person was familiar with the risks of building something that customers may or may not want or may not pay for. So that's what we ended up doing. So I really appreciate your focus on product management and how important it is.

Martin: And if I may, Harshil. I would just say one more thing. All of us can talk to customers. You can even talk to friends, right? Go to a conference and ask people, what are the biggest problems that you face in security today? How are you solving it today? What are the cool tools out there that you see and why do you think that will solve your problem? If you're interested, if you ask questions, and if you ask open ended questions and you listen, you can now begin to extrapolate whether that will help you build something or not. What usually happens is we only talk to each other and we get really confident that we're building something cool and it flops.

Harshil: Yeah, that's great. I agree. Collaboration with peers, customers, hearing a lot more about what they're saying, what they're talking about, definitely helps. On that note, how do you see the future of product security? Where is it going? Any thoughts on that topic?

Martin: I put a lot of thought into this as I was preparing to come and speak with you. I think, if I may, I'd like to answer that for Application Security in this realm. The thing that I'm hearing from customers when I speak to them is multi cloud. So actually 15 years ago we said the cloud is tomorrow and it turns out it wasn't tomorrow, right? It still took a little while for us to get things out of the cloud and we're still moving workloads to the cloud. There's still stuff in data centers and go lows all over the place. I think we assumed we would just put it all in one cloud and then we use all the tools in that cloud. So it's AWS, they've got a bunch of great facilities all buy it from there. I think in the last couple of years what I'm hearing from customers is they're like, “Oh, I'm not going to be able to get it all down to one cloud, so I'm going to have to buy it from Azure and Google and AWS and Oracle and maybe Alibaba and all kinds of different cloud providers. I'm going to have to use them all”. That's right, maybe not all of them, but more than one. Multicloud now means that the tooling, the tool chains that I build, the DevOps processes I build, the Application Security protections I want, I probably can't put it all in one place. If I can, I probably can't put it in that single cloud, I'm going to have to figure out how to straddle these clouds and get my outcome. That's one of the challenges I'm seeing today that I think is here, it's going to be with us for a while.

Harshil: How do you think this will change the world of Application Security? Let's just say multi cloud developers are pushing similar stacks to multi cloud or to different cloud platforms, which inherently, I'm guessing inherently means that you can't really rely on the native tooling for each of the cloud providers or that creates a separate set of problems. Then does the security workload, the security policies or enforcements or controls, do they shift at a different layer of the stack, if not native to the cloud platforms? Do we have to think about multicloud implications of security controls in some way? How do you think that's going to affect it?

Martin: Well, first, I think it represents an opportunity. You look at some of these companies that are building some great API security solutions like Placework, and we're working with CORSIA on some machine to machine stuff, there's a real opportunity here because they can sit wherever and they can essentially work across the clouds to bring in their solution. You don't have to buy it from that single cloud provider. You've got the solution that can work across all of them. So I think it represents an opportunity. It represents an opportunity for Lumen too, because we're a cloud hosting provider on our edge. We can say to a customer, we can put it really close to where your users are with five millisecond latency, and then we can deploy within our DevOps process a solution that allows you to work and bring it in from Azure. We're deploying Azure ACI across that environment.

Martin: I would say it creates a problem, but it also creates a lot of opportunities for those of us building Application Security solutions because we can strand all these clouds and show that you can get that outcome. The nice thing about it too, is it gives you a sense of freedom. It comes to renegotiating your prices with your cloud provider, you're not entirely beholden and intertwined with them. You've actually placed your Application Security solutions outside of their control. If you do decide to move your workloads over, and if you can, you don't have to start all over with your Application Security solutions again, right?

Harshil: Maybe if you think about what that means to the day to day of an Application Security engineer or security engineer, are there different types of skills that you think they should start adopting as the world moves towards multi cloud decentralized architectures or edge computing, some of those things? Does that affect the types of skills we need in our teams today?

Martin: I'm not sure I've given that one enough thought. I think that it still comes down to knowing automation and orchestration, right? Knowing what tools you're going to use to do that. With the idea that you can address your containers wherever they sit, and it doesn't matter anymore where they are. You can address them whether they're sitting on a customer edge or on Lumen's edge, or within the Azure Big cloud or within Azure Gov cloud, wherever it is, you'll still be able to address those. I think we're already building the right skills and cadence here by leaning into our Cicd pipeline with good DevOps and DevSecOps, building everything with automation and orchestration so that we can burn down the machines and push them out wherever we want to send them. I think we're already doing the right things here. We can't make the assumption that things are sitting in a single cloud hosting provider. We actually have to assume that they're going to be sitting and deployed and possibly diverse cloud providers in the future.

Harshil: Right. What I'm hearing is the focus needs to be on the automation orchestration. Of the things that eventually end up in multicloud you can't really rely on runtime. I mean you have to rely on runtime for a lot of things. To really protect the workloads and the infrastructure and the software being built, you effectively have to shift left in a way because that's where everything emanates from. Which would imply that the traditional security skills of pushing buttons, running scans and finding vulnerabilities, while that is needed, there's a lot more focus that needs to happen. The skills need to be up-leveled into more developer centric security developer adaptable security.

Martin: That's right. The good news, we're already doing that. We've already been teaching ourselves to ship last for many years, let's say five years or more. We just lean into that and we keep doing it. The assumptions that we had made before, we know we can't take a note. This is really the long history of programming, right. Instead of static references to things, we know that they're dynamic references and they can live anywhere in memory and they can live in any workload in any container. And now we can just keep moving that assumption to the left which means we can't assume it's running in a known cloud provider, right? We can just rely on our layers of automation orchestration and get it done.

Harshil: Fantastic. If we think about younger security professionals who are looking to advance their career, do you have any advice for product security professionals?

Martin: Well, I think that what we're going to have to do for security professionals in the Application Security space is we're going to have to keep taking advantage of the tooling because within the tooling, I guess I'll say it this way, right? There was a time and I really hope no one out there is still living in this space. But maybe you are. Where we did things like were just relying on static Application Security testing, right? We just like to pass for a code. We looked for problems in the code and we fixed it until we got it right and we pushed it out. We're really at a whole different place now where we can do things like dynamic scanning, build it right into the pipeline so that it's actually part of the build process. We can do things like workload both container protections down at the container level in CWPP as well as actually sitting up runtime in there and we can dynamically pull that back and make decisions about what threats it's encountering and how it needs to dynamically change. We've got to embrace, I mean, we really just said it five minutes ago, right? We have to embrace the model and not assume that you can do things in a static way and be done with it. The great thing is that it's a lot more fun. You're not caught in the drudgery of running toad scans, fixing your code, staying up all night. So it's right. You're actually dynamically integrating with this. This also means that we can now start to pull AI into this, right? AI, that can take in far more runtime contextual signals and figure out, what this workload is currently encountering is not benign, right? Because I can see where it's coming from, how many queries it's trying. It doesn't look like a normal human interacting with this. It can take so many signals into account that runtime that we could never predict before, and it can now actually respond to that. I think we have to embrace that because if we don't, well, the adversaries are going to beat us.

Harshil: Yeah, that's a good point. I mean, I think we're at a place in technology where we have the data available. The technology stacks have changed, so you can collect this massive amount of data and at the same time, AI and ML has matured, where you can apply those techniques onto this vast amount of data available, in my opinion. You can tell me if that's how you see it or not, but AI and ML, there are like really strong ways of building valuable things, but at the same time, it has to be married to the right type of data, the right volume of data, the right data set. If you don't have the right data, then it doesn't make sense, it doesn't matter, you're not going to get the right output out of it. A lot of it is also about what data you have access to that you can apply these algorithms to. For somebody like Lumen, you're sitting on massive amounts of valuable data. You can really leverage it to a very good extent. It really becomes important on do you have access to that type of data that you can then use it for AI and ML?

Martin: Actually, what I would say to that too, is that it also means that we can imagine a future where the way we build our solution looks very different. What software as a service has brought to us is a consumption model. So in the old days, you bought software and you bought a license and you held on to your license key because you were going to use it for five years or until it died, right? But now you can buy it via a consumption model, which is great for almost everyone, right? We're making more money selling it that way, you then have our attention as a SAS provider because we have to keep making it better for you to renew your subscription. But we can also now start to think of the network in the same way. And this is a whole new way of thinking about networks because historically we thought of like, a 100 gig network pipe. You're going to pay thousands a month for that, for your business, right? Now we can start to say, “well, what if I can sell you a network as a service subscription with you're going to pay for the consumption, right, pay for what you use, but we're also going to programmatically deliver things into that solution with your applications and with the way traffic is routed to give you better outcomes. Built into what you're paying for as a subscription could be clean traffic, right? So you're actually going to pay more for clean traffic than I've already scrubbed all the bad stuff out of. You don't have to worry as much about encountering botnets and attacks and things like that, which we're already doing with our DDoS and LAF solutions that sit on the CDN edge. We could also layer in other outcomes and we're seeing hints of this now with SASE, right? Because with SASE, what we're basically saying is, I don't want to send all my traffic back and forth through my firewalls and through all my application engines. Let me just let me take known good applications and let you get straight up to them no matter where you are and then let me build my security policy around it. I can now much more efficiently route your traffic. I can build security policy into the solution. The next step is I can now get you to basically pay just for the outcomes you want as a subscription with network as a service and layered in security. And that's really where Lumen is headed.

Harshil: That's amazing. I think there are so many parallels of that, which is there's an existing technology provider or a footprint or a company that now natively adds security capabilities making it much more easier and simpler for the end customers to consume. The great thing as a security professional, in my opinion, is that security just becomes a native part of your tech stack so you don't have to do many different things. The example that I can also draw from the Application Security side of the world is we are seeing all of these development tooling companies like GitHub and Jfrog, all of them now are offering security capabilities, right? So. GitHub Advanced Security. Gitlab security package. JFrog has a security capability. AWS has a lot of security capabilities. Now that makes it incredibly easy for developers to just consume security as a part of their normal development workflows.

Harshil: It has worked really well in a lot of cases where now you don't have to run seven different tools separately. That security team is running in a complete off cycle with the actual dev process. Now GitHub Dependabot, for example, or similar tools from Gitlab. They're just natively included as a part of your development process and development workflows, which is overall a good thing for security professionals, right? Because security is now becoming a native part of the other stacks.

Martin: I think it isn't largely because most people don't want to wedge security in. They don't trust it, they don't know it. They'll often skip it because they'd rather get the code out the door. By bundling it into the pipeline, you're going to get a better outcome with obviously the big risk here is that you start disrupting normal traffic, right? You've got to have the confidence that your solutions, your scanning, your protections are robust enough that they don't disrupt legitimate traffic. And that has always been the challenge. But I think we're making progress. There used to be a time when a WAF was something you'd never put in front of your app. We're past that today.

Harshil: Yes, for sure. Martin, this was such a phenomenal conversation. Thank you so much for being on this podcast.

Martin: Thank you, Harshil.