Video: Thought Leadership Webcast: Essential DevOps Security Best Practices | Duration: 3580s | Summary: Thought Leadership Webcast: Essential DevOps Security Best Practices
Transcript for "Thought Leadership Webcast: Essential DevOps Security Best Practices": Awesome panelists. So we got Scott Hellman. Wave your hand, Scott. He is supervisory special agent from the FBI, the massive crime, and, just all around general fun guy. He's the one that you'll see on stage when the FBI wanna send out the fun, happy one. If you see the other ones, you should duck and cover. And we've got Brian Wald. He's a senior director, the global field CEO at GitLab. Him and his team, they are the masters. They're the ones that will send in for architecture, for best practices, great advice, and a lot more. And then finally, the practitioner in residence, Colin Rand. He is the EVP, product and engineering, cloud edge security, SonicWall, formerly Banyan. And Colin and I actually work together, so we will have a few stories and some laughs along the way. Well, everybody, let's get started. First of all, welcome. We appreciate your time. And as we go through the the webinar, folks, please throw questions your way. If you see questions, you can upvote questions. You can I don't know if you can downvote them, but you can probably get away with that? And then secondly, there's a docs tab in there, and courtesy of GitLab, they have given us a 1,000,000,000 documents. Well, not a 1,000,000,000. I can't count past 10. So there's a lot of documents in there, and I would suggest you look at those, download those, and get ready, and you can use them later. So let's jump in. I I wanna set the stage and some context. As as from a security perspective, let's hear some horror stories because we always we always do that fear, uncertainty, and doubt. So let's have Scott, why don't you share some some stories of a compromise that began because of poor development practices or poor configuration? So from the FBI standpoint, when we're looking at what's the result of a poor development practice, it may not display itself for us. We're always gonna see things at the very end point when they get broken and they're used for a breach. So I may not know exactly what the issue was at the time. Logs, when I'm doing an investigation, logs are just gonna show that there was a compromise, there there was a login somewhere, or maybe pivoting around a network, but I don't know what the initial vector was or if it was as a result of poor development practices. Was it stolen credentials? Was it a vulnerability in the product? And so, oftentimes, we'll only get that information when we catch people, we interview criminals, we interview hackers, or if they decide sometimes they'll post information online after they've breached a particular organization. So we had a case where it's very, very simple concept where we had criminals using the basics of, reusing passwords that they initially found through previous breaches, and then combine that with weak or no MFA, and using those to get access to cloud source code repositories. And so once they get access to the source code repo, they found that there were secrets or cloud storage keys inside the source code for database backups. And database backups, we're talking about database back customer database backups. So usernames, password hashes for many, many customers. So they rinsed and repeated this process of password reuse, figuring out various ways to bypass MFA, and they built up a database of 100 of millions of usernames and passwords. And then use this to target online advertising agencies, And once they got access to the advertising agencies, they essentially siphoned off advertising spend from various different, companies and used it to facilitate their own sort of nefarious activities. So they use these credentials ultimately in this complex online advertising scheme, stole 1,000,000 of dollars from some companies, and then made those those 1,000,000 of dollars from themselves. And ultimately, really, it all stemmed from having secrets inside the source code repo that led to database, database backup theft. Awesome. So passwords, passwords, password and code. Let's, jump into so a little controversy. I've I've had a lot of people talk about this one over the years. So there's DevOps, then there's DevSecOps. What what what what's the difference, and why do we care, Colin? Yeah. It's a fun question. So so DevOps, I think everybody's pretty familiar with the space. It's been around for a while. It's really, you know, trying to bridge the gap between developers and their infrastructure needs, getting the teams to collaborate early in the process so that we can push our systems into the production environments and, you know, generally have more stable applications. In that, everybody will tell you security is a concern of DevOps. It's it's, in fact, a really important one, but that still is not quite the same as the DevOps DevSecOps perspective. DevSecOps really is a layer on top of DevOps and the practices, again, and it's enhancing the collaboration. So now we're including security stakeholders as critical stakeholders in the process of in the collaboration. And now it's also adding their tooling, to be the entire dev DevOps tool chain, whether it's static analysis, different types of dynamic analysis, cloud scanning. And you're integrating this into the development process so that you can deploy your code into production and not only know you have your stability, checks in place from various testing, but you also have secure security, checks in place too. So it's an enhancement. I wouldn't say it's one or the other. It's an enhancement on top of basic DevOps practices. Hey, Colin. I have a question. Where in here, like, is the is the goal for DevOps or DevSecOps or both to to increase security in the final product? Or is there any part of that that increases security in the development environment itself? You know, that that's a great question because there's also there's often dramatically different security requirements in prod versus non prod. And the tooling can play a role in securing your non prod environments and prod environments. It can be the same tooling, but they'll also have they'll often have very different policies in place. So, for example, your your nonproduction environments, in order to access them and troubleshooting, might be much less restrictive. Right? And you might have more general policies because you're, you know, developing and iterating quickly. The whole development, team might have access. But for production, it might be a much reduced scope, you know, at least more of the least privileged access model for the production. And so your your tool chain can reflect that, but it's it's often a different set of considerations that you have for security non prod versus production. Awesome. And so so, Brian, I was just gonna say, so your team deal with 100 of customers. So I'd love to hear what your take is on this. And then also, what do you hear when you talk to customers about the difference? Yeah. Yeah. I mean, I think, as Colin was saying, you know, I think DevSecOps was is really the the natural evolution of of DevOps. Right? You know, and you can see my shirt is is now technically vintage at this point because it just says DevOps. But, you know, so what I mean by that is, you know, DevOps was really built around, hey. How do we build better collaboration between the development and operations teams and allowing us to build software faster? Right? And whatever happened is that, well, once you start to build the software, you still have this secure security aspect of it that happened serially after the development process. So what this meant is that developers, you know, would go off and go into their sprint, and they build the their application, and then they're, you know, move on to the their next task. And security then, you know, maybe 2 weeks later or whatever, goes and does all of these different security things comes back and says, hey. We found all these vulnerabilities. Well, I at this point, I I have no context into what I was doing there. I have to go back and and and figure out what was happening, fix these vulnerabilities, and now I'm, you know, kind of slowing down the development process that I'm on to my next tasks. Right? So the DevSecOps was really that shifting left the security practices to be part of the development process to keep you in context, to allow you to build a better development workflow that allows you to be fast and efficient, which we were doing with DevOps, but also now more secure at the same time. And, you know, what I'm seeing now in a lot of organizations who have adopted this practice of shifting left and moving security into the development process is really the next evolution of that is, okay. Well, that's great that I've moved that there, but now I've placed a huge burden on developers. Now they have to go and manage these tools. They have to understand what our policies are and all the other aspects of that. And the expectation from security is you're just doing that yourselves. Right? And so they're saying now, well, what if we just took that abstraction away and say, you know, almost kind of like a shift up, if you will, where you move the security practices to the platform level and you apply them down to the development teams? So this means that now you your developers can just go through a flow, which which is much easier for them than actually managing the actual security tasks themselves. And so that's really where a lot of organizations are starting to head with the next evolution of DevSecOps as well. Awesome. Awesome. Hey. So, audience, we have got so first of all, we've got some great polls. We'd still invite you guys to ask some questions. So why don't we start off with the first poll? And, so what intrigued you to attend the webcast today? We'd love to hear your your thoughts. Now if you go to the polls tab, you will see it there. Just click the polls tab, and, you can give us an answer. Now while you guys are answering, let's jump into the next question. This one's a this one's a doozy for me. I don't know if you guys know doozy. It's probably a Scottish word. Anyway, so how how is it different writing code that ships with a product versus writing code that's gonna be used as an actual service? So you got a platform that's up and running versus the old shrink-wrap, I'm gonna sell the product. Colin, you've you've done both, so why don't you share your thoughts? Yeah. Yeah. Yeah. It it's a it's a good question. So when you're when you're building a web service and you you are responsible for the operation, now you obviously have vendors involved, whether infrastructure, cloud vendors are involved. You're not doing it by yourself. And but, fundamentally, you, the owner of the service, are responsible for the security of the data, of the operation, start to finish. You can change it. Oh, I find a security vulnerability. I can patch it on whatever cadence my my business and team allows for. I can run my security tools as I come up with thing. I I'm in control of that. And that's, you know, with it's a great responsibility, right, that that you have to really focus on. However, if you are shipping software, and I think that's kind of a lot of the traditional mentality around security, where you ship software and somebody else is using it. There's all these CVE processes around it, where other people are going to take your software. They're going to use it, abuse it, run it on infrastructure that you have no idea about, and they're gonna blame you for all the security issues. And, they're not gonna update it when you give them updates. And so it's a very different world, whether you are operating the software that you write, or somebody else is. Both are challenging, both will give you gray hairs, but, you know, at the end of the day, there's a lot of commonality in the coding practices that you need to take in terms of the processes, but the actual security risk and the actual security vulnerabilities in the coding can be wildly different. You know, just in terms of SQL injection on something that you ship looks very different than protecting a SQL injection on something you host to maintain yourself. Awesome. Does anyone else wanna contradict Colin's, Colin's, sentences there? And if if not, let's, so okay. So architecture. Brian, this is this is your team's, space. So can you share examples of poor architecture configurations as poor security, or or created more risks in environments? Yeah. Yeah. I mean, so when you think about runtime security for applications that you deployed, I mean, you could get into countless number of ones or rings around, you know, anything from unsanitized forms to database connections, all that kind of stuff there. But, you know, for me, a lot of the times I think about this is there's a there's there's a much bigger focus today on how the development process impacts the architecture of the the development process impacts the application that goes into production. And, you know, this is what we oftentimes hear as, you know, software security supply chain. And this is a big gap that we're really focused on trying to fix now. You know, building this architecture for development is really, really important to help prevent, you know, bad actors or accidental compromising of applications. So for example, you know, one scenario that I've seen is that when when an application is built with CI and it runs all the security scans, you maybe even generated an s bomb, you're doing a lot of the right practices for doing secure coding. One of the risks is that, well, you know, what happens if in after you've built all of those those items that later in your pipeline, there's a job that, goes and, you know, install some additional packages. Maybe even these packages aren't related to the application but related to the server configuration. Right? Well, now those are untracked components that are getting injected into the application that will get deployed, and you don't have any sense of what that is or where it comes from. And this is a really easy attack vector. Right? And so this is where we oftentimes talk about, you know, when you build software security supply chain, this is around cryptographically signing each individual job and stage as you go through the process and having attestations associated to that as well. And then also, you know, on top of that, making sure that the the the actual promoted artifact is immutable. Right? So once you've created that artifact, nothing should change in that artifact between the point where you've you've created that and when it's distributed into production. And so between those two things, there's a lot of risk, and we've seen a lot of attacks. You see a lot of those in the news that happened in the last, you know, year or 2. And and this is an area where there's an increased effort inside organizations to really address, you know, not just applications running in production, but how the development process gets you that. And this is really where a platform kind of solves a lot of these issues there to solve, to make that easier. Awesome. Awesome. So, so passwords or secrets in code. Let's talk about some best practices on how you'd avoid on avoid this. And, Colin, I'd love to start with you. Oh, man. Best practices. So the the first thing you have to do is, and I'm I'll make this as obviously tongue in cheek. You beg your developers not to do it. You plead with them. You encourage them. Anything you could do not to do it. But at the end of the day, you have developmental pressures, and that's all around speed. They want local devire develop environments. They want it easy, because they're like, look, I'll deal with that later. I'll deal with that later. And it's, you know, the the best practice is convincing everybody of the mindset shift that security has to be first. Right? Security is can't be an afterthought. So, honestly, that is the human side of this. That is number 1. There's a lot of tools we can use for secrets management, whether it's anything from password sharing. Don't share your password in Slack as much as it, like, feels like that's a fast way to do. Scott's gonna tell you about, you know, pivoting attack vectors. Hey. I love the you know, I would love to be able to share password. Oh, I'll just post it in Confluence so all the QA team can get it. Right? No. No. No. No. No. You don't do that. Right? So there's a lot of tools and best practices, but at the end of the day, it's understanding, like, the human motivation to make things easy for themselves, and working with the teams involved to say, okay, security actually has to bump up in the priority list, because otherwise, you know, we're we're gonna be shooting ourself not even shooting ourselves in the foot. We're we're working against ourselves to what we're trying to accomplish. So I'd say, you know, it's, multifaceted. It's hard. Use your tools. Use industry best practices that are well understood. But at the end of the day, especially managers and senior people in companies, understand the mindset set shift that has to happen, and encourage teams to, you know, take the extra minute, take the deep breath, do what's right, and it'll pay off dividends. Yeah. And I I think I'll just add to that real quick. You know, we know that people are also not going to do that. So you also need to be building in a lot of the checks and balances to prevent that as much as possible at an organization level. Right? And so what this means is that you should be running a secret detection tool prior to any of your code getting pushed into your source code management system. So in Git, this would be like a pre receive hook that goes and, you know, runs the regex against the application to see, is there any secrets in this? And if it is, then reject that from even getting into the repository. Because once you get a secret in the repository, it's really hard to clean up. You have to, you know, first, you have to rewrite the history of Git if you really wanna remove it because even if it's in a feature branch, it still lives there somewhere. And then you have to go and rotate all of those keys, and someone has to go and make sure that that happens. Right? And so really, really making sure that you put the guardrails around your development organization to prevent those things on top of the people and culture aspect that Colin was talking about there as well. So, Brent, I'll give you a great anecdote, which a tool like you're describing is critical. So I I worked at a a small team once years ago, and, we we were under a financial pressure to build a new feature and yet save money at the same time. Very typical start up environment. One of my developers wanted to get another git repo, and the service provider would have cost us money. And the finance said no. And they said, well, while you guys deal with your financial nonsense, I don't care about it on developer. I'm gonna take this project, and I'm gonna publish it as a public project, which I can get for free. So he did the right thing. He looked at it, identified some secrets. He didn't really wanna spend time cleaning. So he asked everybody on the team, hey. Are these secrets good? Everybody's, like, nope. Those are old, but not used. They're defunct. You know you know where this is going. He publishes it. Sure enough, some crawler comes around, picks up, finds it. Next thing you know, we have a $20,000 bill from AWS over a $50, you know, increase in a license fee. So that's that's real. It happens. It's the human element that gets added in. You have to have the tooling to make sure that, you know, the the old joke I like to tell, ask for a bunch of engineers. I got a bunch of people. That, at the end of the day, the tooling is how you contend with our people. Exactly. Yeah. Yeah. Yeah. Yeah. So so here's, so we've done a poll earlier. We'd love, to share the results of the poll. So I would intrigue you to attend today's webcast. Let's go. We, we have some of the FBI perspective. There is a clear winner, though, getting DevSecOps best practices. So that means, team, we need to continue sharing some best practices. So, as we get into here, let's let's Scott, I'd love to to have your opinion on this. Right? So we build these dev environments, and people with remote access into these dev environments. Can you share the worst thing that you guys have seen from a security best practice or perspective, when people are accessing these dev environments? It's actually a pretty easy segue directly from what you and Brian and Tom were just talking about. And really, at least for me, when I see it again, I see it when it's already broken, and the reasons why things get broken in many situations, it really comes down to balance. It's the the old adage of the balance between security and availability. And when you're talking about an environment where people wanna work quickly, they're often gonna sacrifice that security. And Brian, you bring up, well, this the security then needs to be something that's built into the system. And one thing we see commonly is some of those security measures are are sort of coming from the top down, and companies end up over tightening security to the point where the resources aren't available, and then you have your developers get very frustrated. So what are they gonna do? They're gonna figure out a way to get their job done, and they're gonna come up with some sort of a workaround. And in that moment, it's done altruistically. I wanna get my work done. I can't get it done because this resource isn't available, so I'm going to download a backup. I mean, you know, do whatever I need to do. And that's all well and good right then and it doesn't become an issue until that employee leaves the company or becomes disgruntled or or both. Right? And so here's a couple of examples. Real simple one, employee, took a backup of a database, put it on an external hard drive, brought it home. Everything was all hunky dory right until, he gets fired, and is now very upset that he got fired. And so now he's got this piece of leverage that he attempts to extort the company with. I'm not gonna return this. I'm gonna do something. I'm not very honest with it. Another one, very similar. Employee, it's not necessarily over tightening security. I think it's just an example of employees wanting to get the developers wanted to get the job done quickly, so they all have shared API keys as opposed to each having their own. So what happens? The employee gets fired and now he's got this API key and logs back into his system. Whether or not it was an accident, I think remains to be seen, but blows away about 400 different production VMs or I'm sorry. Development VMs, and this ends up oh, no. I'm sorry. They were production. It ends up causing this massive customer outage of the, of the product, and it's, you know, 1,000,000 of dollars of loss. And so two best practices to avoid something like this, it's almost on the back end because employees are always gonna find some work around and and they're going to be there. So how do you find those things? Number 1, when you let someone go, don't just interview them. Everybody has sort of an, off boarding process, but you gotta interview some of the colleagues that they've worked with too and find out, was your team using any shared resources, shared keys, things of that nature? And also, did did they have some additional, let's say, admin account or whatever that they developed while they were working there that wasn't part of their initial onboarding package. That's really important because, colleagues will know those things. And second is it's really important to run proactive threat hunting, on the back end periodically throughout the year because that's how you're gonna find those additional don't necessarily wanna call them backdoors because that's sort of nefarious sounding, but if I'm struggling to get access to the resource and I create a second account because it's easier for me to do for whatever reason, that's not gonna show up in the normal logs that the company's gonna be looking at. And so threat hunting is a really great way to identify those, out of band, out of band additional workarounds that employees will develop. Awesome. So and it's it's funny because I I was just thinking about this one. It's, you know, long time ago, you'd have your labs and your engineering environments, and and you'd see people sometimes, I I mean, we call people trying to actually bridge and put a VPN in there so that they they could access the lab environment from their home, but totally out with the control of IT. And then you'd you'd find some of that stuff. And then the other one for us, as as cloud became more and more a thing, then people spinning up their own accounts and their own environments in the Cloud without actually tying it back to our IDP. So you totally lose the whole access management control piece of that, plus any additional things like MFA. So those those things for me always scared the the the the monkeys at me. Colin, what's what's your take on this one? Developers are gonna find a way. No. No way no way get around it. You know, you can put in your security controls. You can you're gonna have dedicated VPNs. Yeah. I I remember many many many times to Belfer saying, the VPN is slow. I don't wanna deal with it. The I keep on getting dropped. The authentication is hard. I'm just gonna install my own proxy somewhere out there, and I'm just gonna go in. Right? And then they leave, and they forget about it. And you've got who knows what running out there in what environment. So, you know, it's I mean, developers are gonna find a way. I mean, that's at the end of the day, they're gonna find a way to be productive. That's you hired them to be clever monkeys. We are clever monkeys. We're gonna find ways. So it's, it's a challenge. It was the worst thing I've ever seen. I don't know if I can say that live, but it's it's scary. Yeah. For legal reasons, you'd like not to share. Right? Hey, Brian, what I mean, you you guys, so in in your your career and also in your customers' environments, what are you what are you seeing out there? Yeah. I mean, I think look. I'll I'll take a kind of similar perspective here, but, you know, I think that there's look. With a with a dev environment, it's not necessarily a terrible idea to go in there and get access to it in certain scenarios. So forgive me an example. Like, if I'm working on a a microservices project and I'm working locally and I'm making my changes and everything looks all good and fine, the the first point in which I can actually see my part of the service running with all the other services inside this application is oftentimes in a dev environment that's running on some servers. Right? And so, by the time it gets up there and I see, you know, my test passing and there's maybe some issues I need to go solve, well, it's not a crazy idea to go in there and say, okay. Let me just tweak some things here while I have all this stuff running to see if it works, and then go back and, you know, you know, commit the code in there and rerun the pipelines and all the stuff that goes into it. The problem comes into if you have bad architecture for how you do your promotion process, this becomes an issue. And I see it all the time. So a lot of times people will say, here's my dev environment. Yes. Everything's passing. Well, I'm just gonna take this artifact, and I'm gonna move this up into staging or production. Right? Well, that's a a terrible idea. Right? What you're gonna wanna do is have everything be rebuildable for each stage. So if I go in and make a change to dev, that's fine because even if I don't go back and and, and and make those changes to, you know, the code and have it re redistributed to the development environment, if that were to move into staging or production, which it shouldn't, but if it does, in the staging and production instances, they're rebuilding from source. They're not building from what you have inside your dev environments. And so it's really around making sure you have the right architecture that's gonna support these things because, you know, you you wanna prevent any of these scenarios where someone goes in, make some changes, forgets about it, they get pulled into a a critical issue, and then someone says, yeah. All this looks good on dev. I just tested it out, and they they promote it up into staging production. Now all of a sudden, you have a vulnerability in production that's no one's aware of. Right? So I think for me, it's really around building the arc. Again, the the structure and architecture for the workflow for your your development life cycle is really comes down to, you know, setting those right guardrails and making sure that everything is immutable, but everything's rebuildable as well. Awesome. Awesome. Add in add in a flavor to this just to kind of help everybody understand the scale of the of the problem, you know, and and the challenges, and then, obviously, opportunity to fix it with the dev set of ops mindset. And and it's m and a. Right? So every startup out there is goal is, often m and a. And what happens is you bring in a myriad of tool chain, a myriad of practices together. And if you have a central governance team, you know, you're you're all of a sudden left with, like, really challenging dynamics of multiple teams coming together. And so, you know, you know, using the tooling is is critical, but it's I I just wanna highlight, like, sometimes you have, you know, you don't have consistency across your stack in your your production environments. You often will have fragmented systems accumulated over 10, 20 years that you have to contend with. And it and it it's, it takes time, it takes investment, it takes, you know, know how in order to get it right. Awesome. Thanks, guys. Now, so let let's have another poll, and while we put the poll on the screen, so you'll see the polls tab. So we'll throw a poll up there, and while we are, looking for the while we're, yeah, while we're, making that available for everybody, let's, let's ask a question from the audience. Hey, so Alex Smith, he he he asks us, how are open source dependency packages versioned, managed, and evaluated from a security perspective? Gives an example of using packages from the public PYPI repo. Colin, why don't you start? Oh, boy. The yeah. Let's poke the bear right out of the gate. So you're gonna get a violent reaction from different technology techno types of technologists. You have the immutable folks, the functional programming folks who really are just bent on making sure everything is reproducible, known state, you know, state management is front and center at the concept of the systems that they're building, versus the other camp, which is much more pragmatic. Hey, we're gonna get it done. Not everybody's gonna retrain themselves to learn functional programming. We're not all gonna go learn Rust or whatever it is. And so, it's it's a challenge. So you want you you often will have a heterogeneous environment. You have multiple technologies, multiple coding platforms in an environment. And so, yeah, I mean, you you do need to understand how versioning node and NPM works different from PyPI, different from other other, like, types of container open source containers you might pull down that are that are fully defined from a vendor. In terms of a best practice for specifically for Python, I probably can't do that off the top of my head other than I can tell you it's important. And, again, known state and reproducibility are going to win out at the end of the day for security and stability, but there's challenges. Yeah. I think, yeah, to add to that, you know, it's it's, you know, there's there's 2 aspects of this that I see, you know, to prevent a lot of the risk. 1 is, you're simply running dependency scanners to check for CVEs before every time you you you do any sort of build or deployment. Right? And so, you know, that one is is is pretty obvious one that is is important to evaluate all of your packages and their current pin state inside your your repositories. Now the other aspect of this is, you know, quarantining packages, that are, you know, go through a proxy before they get into your development environments. You're not pulling directly from the public open source repositories. Right? So there's a lot of organizations who are going down this route because there is a supply chain attack risk where some author makes a change into a package and and nobody realizes it, and it gets into into their production instances. So what they do is they say, okay. Well, what we're gonna do is, proxy the dependencies inside here, pinned to certain versions, and then those developers are gonna use those versions there. And if there's something that they need to change, then they just go through the process of getting that pulled into the proxy. Right? So, you know, that's a little bit slower and that's probably more you'd see more in enterprises. But, that tends to be the the the two routes where you can have, you know, a pretty good assurance that, you know, as you're managing your dependencies that there's no risk in upgrading that or having, you know, CVEs in your in your vulnerabilities and and every, you know, every build. And then the last piece, I guess, I would say is is is is running something as an agent in your runtime. Because if something's deployed and then a CVE comes out, well, you need to be aware that you have a deployed version of that package in production, and something has to tell you to trigger to go either upgrade the version or fix this or see if it's a false positive for your implementation of it. So I think the three areas, you know, preventative, blocking, and then and then reactive is the is the approach most people are are looking to take. Yeah. One of one of the things that you that you highlight there, I'm gonna jump in. And, Scott, you may have a a sense on this from seeing it out in the wild. But hearing about the attacks where abandoned open source projects or frankly bad actors buying rights to become, owners of open source projects, and they let them be normal looking projects for months to years. And then one day, they go in and they swap out the bad code in there and catch a lot of people unaware because you've it used to be an like, for 20 versions, this thing was hunky dory, and I'm just going along. And then all of a sudden, the attack comes, very nefarious and, but it's definitely live in the supply chain world. Awesome. And then, folks, we're gonna close this poll shortly, as as as I read out and we answer the next question from our audience. And then another reminder, please, send us your questions in, and, we're gonna answer one from Anthony Franciscovich. So, okay, I don't know if you guys can read this question as well in in the the panel, but how do you manage security for legacy applications or those that do not currently follow industry best practices? And then Anthony gives some exam an example that comes to mind is how we handle SCA dependency scanning, s bomb creation, and so on and so on. It's a bit it's a lot of words in there. So I'm hoping you guys can, jump in. And, why don't we start, Brian, with you this time? Yeah. Sure. This, I think, is, you know, the age old question. Right? Especially when you're talking about, yeah, my my dependency scanners and everyone that I'm aware of that's out there in the industry today depends on looking at, you know, some sort of manifest file that's going to, you know, whether it's a JSON lock file or whatever it looks like to see what my my dependencies are. But if you're using legacy application, you may be just, you know, physically loading those modules directly into your repository. You may be checking that in your repository. It doesn't It doesn't look like a dependency, and so it's hard to kind of know if their CDE comes out. Well, how is this related to what I'm doing? Because I don't have any way to test to see if mine's doing that without manually going and looking at it. Right? You know, I think the short answer of a lot of these is that you're, if you're if you have to do that, then, this is where your your you have to you kind of reduce the blast radius of what the the library is is providing for your application. And so what I mean by that is, you know, if if I'm pulling in a a library from, you know, something that exists external from my application, whether that's something that's hosted in, you know, repositories externally or, you know, other package management solutions and you're just not using a package manager, well, you know, you have to be able to, you know, build your application in a way that you're only pulling in the right components to to be used for that. So you're reducing that again, reducing that blast radius of, you know, the whole package. There might be a vulnerability in that package, but only my implementation of that is reduced from, from, being vulnerable to that attack. Right? I think the short answer a lot of times is that this is these are the types of applications organizations are looking at saying, yeah. Look. These are a high risk security, a place that we're looking to invest money and and time to either refactor, frankly, a lot of the AI stuff that's starting to come out, especially when I look at in the in the public sector space where they're looking at, you know, memory safe language changes to, move move applications over to be able to do that in a much faster and more cost effective way. There's options now to do that more efficiently. And before that was, like, yeah, this is too expensive to do and we're gonna take that risk. Now that's starting to change in saying, well, now we need to modernize some of these. Now we need to actually put this into your best practices that we can have reproducibility for it. So, not a not a great answer. I don't think there's a there's an easy answer for this problem. But I think, you know, mitigating risk and mitigating the blast radius and then looking at opportunities to upgrade at least components of that to a more modern approach so you can start to take advantage of security tools and get more standardized across your org. Oh, sometimes, Brian. I was gonna jump in and say the number one thing to do is to prioritize the risk of the various applications because sometimes, you know, there's really, really sensitive data in some, application, and sometimes there's not. And understanding that risk profile is going to help you focus your limited resources because oftentimes, the team will find themselves with a legacy application that's dumped on their lap, and nobody wants to spend the time to touch it. And you have to have somebody that is coming in and saying, yeah, this is important, or no, it's not. I get it, and this is an acceptable risk. Yeah. And that the the whole SBOM thing is funny because the last, I think, last 2 years, the SBOM term has been been gaining momentum and people seem to be caring more, especially as part of supplies chain security. So, Scott, I mean, from an SBOM perspective, do the FBI, hear this more and more and think that that's a bit of an issue? I'll be the first person to say, I don't know what s bomb stands for. And I think there's a very good reason for that. It's because most of what we see is in vulnerabilities in the development environment itself, not necessarily in the development process. The development process has loads of layers that all of you are discussing and many different building blocks that you can, put into the process to help develop a secure product on the end, on the back end, whether it's, you know, like you talked about code as a service or whether it's an actual product. Most of the time, we are not seeing the a failure in development security in the product itself until it's way down the road. A CVE, you know, a vulnerability is identified, and then it's too late because the product is already in the hands of the end user, whoever's obviously the end user. So for us, most of what we see is a lack of security in the development environment itself and it that's no different to any other internet facing environment. Right? So you're talking about the basics of phishing attacks, social engineering social engineering attacks, vulnerabilities in systems that the developers are using more so than the vulnerabilities in the product they're developing. It's just not something that we see. We're not exposed to that nearly as much in the development life cycle. Yeah. Just one more point on that I'll I'll just say about the SBOM is, you know, I I think an SBOM is a good start. But as, Anthony, you mentioned there, right, is there's a lot of edge cases where things are not going to get inside your software build material, Scott. That's what that stands for. And and so, you know, it's a manifest file that collects all this information. But the problem is a lot of times, it's just looking at your application code itself and the dependencies that are associated to that. Right? And so sometimes it gives you this false sense of security because there's a lot of other places where things can get injected into the process that you have no record of inside this s bomb. So you can create this great s bomb and it might help you find and identify vulnerabilities, but it's not going to be the holistic thing. I've I've I've had a lot of conversation with customers who say, yeah. What about and I even mentioned 2 of these today. What about when someone injects something in a pipeline after the s bomb has been created? What about when I need to modify libcurl enough that I actually have this as part of my repository and it's not classified as a deposit or as a as a dependency anymore? Don't do that, by the way, but they do I've this is a real example I've heard. Right? But so these are all edge cases. Sovereign saying, yeah. Okay. S bounce get us part of the way there. But I'm almost wondering if that's almost making things worse because you're expecting that to be the single source of truth when there's other ways to inject things in there that could potentially be dependencies that you're unaware of. Awesome. And, hey, Brian. Thanks for for explaining the the acronym. I was just about to do a shameless plug for 99 cyber and say, hey, we've got an acronym as a service product. You know, you could sign up and, you know, a software bill of materials. And it's funny when Colin and I were working together at Banyan, you know, we would we would talk about this because the minute someone says, oh, there's a vulnerability in this open source library or there's a code or whatever. Right? Or some other vendor's got some some issue in CVE, then we'd look and say, well, is that something that we're using as part of delivering our service? So the only thing that I was thinking, oh, this is valuable for was a quick list of do we use it or do we not, which in a smaller company is easy. When I was working at Adobe and Cisco and running enterprise security there, that's a gnarly task. I mean, if you wanna figure out, like, what is all used within products that ship at Adobe and Cisco, that that that for me is, like, mind blowing. You're trying to get a negative at that point because there's so many data points that it's I think you're highly likely to miss something. No. Yeah. Yeah. Well, that's that's the problem. Let's talk about security in the CICD pipeline. Can you guys share a few best practices on how you integrate security into the CICD pipeline? Brian, let's start with you. Yeah. So I'll start by saying, you know, I think there's there's a lot of tools and analyzers that you can be building into the development process. But for me, before I even start to think about the, you know, you get the best tools in the world about how those are actually running against your application's code, it's really around building the workflow that supports the security process the most effective way. And Colin mentioned this earlier, developers are gonna find the way. They're also they also want to write secure code, but they're also gonna take the path of least resistance. So it's on the organization to build out the development workflow that the secure path is the least resistant path. And what I mean by that is, well, okay, it's great if you're gonna run security tools, but I wanna know that when that when those security tools run, that it's it's only giving me vulnerable vulnerabilities that are in context to the code that I'm writing right now. I don't wanna know about all the vulnerabilities that someone else hasn't fixed that's in their backlog from the last few years. I wanna know what I'm doing right now. If I'm writing insecure code, tell me about it at the time I'm writing my code, and allow me to fix that today. Right? And I wanna make sure and I don't want it to be jumping between a bunch of different tools either. We know that context switching is is is a massive disruption to developers. So don't don't make me jump to 5 different tools dashboards to see all of the vulnerability reports to see if I have any vulnerabilities related to my code. Present that to me again back where I'm doing my work. So I'm giving the visibility, making it really easy to to run, and not having to own the security tools themselves in the implementation of my application. Right? Let me write my application code, apply the security tools, give me the findings in one place, and I'll write secure code for you. Right? So that's really what I think is the most important aspect of this is is finding that inflection point where you can introduce the security, scanning vulnerability, security scanning and and whatever compliance that you've built into your process and make it easy for them to see, understand, and fix those things and not disrupt their normal development workflow as part of that. Yeah. And I would you know, it's funny. I I just always learned that if you make something the easiest path because, you know, everyone Colin, you mentioned this earlier. Everyone's just trying to do their job. They're they're trying to push they're trying to push product out the door, build features, and and get moving quickly. So if you come up with policies and other other security things that just get in the way and add friction to that workflow and that process, they will find ways to work around security, or they'll find ways to work around governance. Colin, anything you wanna add to this one? Yeah. Yeah. I mean, it's hard to argue or quibble with anything that Brian said. It's all spot on. I will add that, again, adding the human element, and it's really to me, security comes straight to the CICD process during the it's a little bit before. It's the code review process and helping train folks to understand, you know, how to help each other while the code is still being authored and give the secure oriented feedback. So how do you do that? Well, you you identify you could go so far as identify security champions on the engineering side to help everybody understand it. You can do training courses. A lot of different ways you can kind of build that educational base, but deliver that in the code review process. Again, there's a spectrum like you can do explicit checklist, make sure you've checked with these. But, really, it's the spirit of the conversation, the mindset. Hey. Have put security as part of the code review process. So at least it's a topic of conversation, and it's raising awareness and of of this as a challenge. And once the awareness is there and people sign up to it, yeah, they'll generally work towards solving it. They're all nobody is intentionally writing insecure code. And so it's bringing that focal point of awareness in the code review process, I think, is a really big, step in the right direction. Awesome. So, look, this would not be a great webinar without one of those AI questions. So Joe McDougall posed a question for us. How are the panelists working towards securing DevOps with the introduction of AI? So, Colin, let's start with you. How am I looking to secure DevOps with AI? That is do you to quote then, that is a doozy. You know, it's it's nothing but controversy. You know? I'll just put it out there. So I would start with, first, we're exploring it, evaluating, understanding of what it can and can't do, and reasonable expectations. Second, if you're talking about generative AI in the DevOps world or coding world, you're talking about generating code. Anybody who's tried to read somebody else's code who's no longer at the company and trying to make sense of what they were thinking when they wrote it to know how to change it is terrified of AI producing code gen at a scale they've never seen before, so moving very carefully. And when you move into the DevOps space and you're dealing with physical infrastructure, it only adds to the complexity. That said, there is a very powerful time and place for for AI, which I think we're gonna see more in the code review process, the recommendations, the hints, and the feedback, more than the primary source of what to do. Because when you're building technology, 3 quarters of the time is spent figuring out what to build. The coding is the easy part, and the coding is all about helping another human understand what's going on. And because machines machines really code, machines don't need help in writing instructions for the machines. So I'd say I'm moving very carefully on it, evaluating it, keeping an eye out. But it's it's a if it's nothing else, it's controversial. Awesome. Anyone got anything else to add before we jump into some poll results? I just just a quick point on that. So, yeah, I mean, if I had 100 of these conversations in the last year, with a lot of different organizations at different maturity levels. Right? And, you know, I think when it comes to, generative AI for code development, the first thing I say is, well, if if you're gonna let developers go off and use this, then let's make sure that you've built in the right guardrails into your development process to prevent more code. Because if you're gonna be generating more code in theory, that's the idea. Right? It makes you more productive. Generating more code that we're we're testing and making sure that we're reducing the risk of those vulnerabilities as you're going through that process. Right? And so, you know, that's a big piece of it is is, you know, let's build again the right architecture to support, you know, more code coming in, more code review going into this. But it gets into a lot of other aspects too. But I actually think that, what I'm more excited about is how generabyte itself is actually helping us be more secure. Right? So I can you know, these the analyzers that are that are starting to come out for that are using GenII to look at applications code is is not only doing a better job of identifying risk, explaining to them what the vulnerability is, but, also, it's doing a really good job of of keeping the noise down of false positives. Because with a lot of these tools now, I can have context into your entire application. I can understand how you're using that dependency. There's a CVE and a dependency, but I haven't called any of those methods into my application. It's a false positive. So I can look up through that tree and see how you're using that really, really easily with GenAI. So, you know, kind of flipping that around is I think that there's gonna be a huge advantage of AI as a tool to support being more secure in your software development as much as there is risk of it generating bad code, or, you know, any of the risks of, you know, the, you know, polluted LLMs or any of those kind of things that we're starting to see from attack factors there. Those can be mitigated by just good process. But what's great about the the the AI tools out there now is they're starting to be more sophisticated, making developers' lives easier as part of that process. Awesome. Thanks, Brian. Now let's share the results of our poll. I'm always sharing the results. So what type of applications do your teams work on? Select all that apply. In house in house, third party. So in house developed, it's it's pretty pretty interesting to see that we are building our own tools. Maybe that's not our company's core competency, but, you know, people do love to build stuff, especially developers. That is that's an interesting one for sure. So, and, actually, so before we move on to the next question, why don't, we share the next poll question? Because, you know, we're getting close on time here. So, let's share the next poll question, and, we will leave that up and running while I ask, a couple of, questions here. So infrastructure as code. Now if you don't mind, can we, do do do yeah. Let's say yeah. Hide that to the side. The poll question on the side is a poll tab. So let so, James, let's talk about infrastructure as code. What is it and why is it important in DevSecOps? And Colin, we're gonna have you take this one. Yeah. So infrastructure as code is the practice of it's re re defining how your, infrastructure needs everything from their software networks, to load balancers to databases to, any type of auto scale that and, you know, everything that a cloud provider or a cloud style infrastructure provides, defining that declaratively preferably, but defining that as software that can be checked in, versioned, run, controlled, tested, and pushed to production because it it it's software. It's treating your, configuration of your production environments and nonproduction environments as core part of your application, codifying it, and then treating that as part of the full software development life cycle. So that's the gist of what infrastructure as code is. Was there another part to the question? Or just No. I I think it's well, I think there's the definition, but why is it important? It's important because if you don't define your infrastructure, you then are left documenting it, and you're left with an with a system where you go into production, and you make a change, and knowing what is actually running becomes a very difficult challenge. If you I mean, it's it's been so long since we've been doing this. At least since I've been doing this, it's it's kind of hard to put my my mind back in the camp. If you if you don't know what's running, you don't know what you have, you can't secure it. It's hard enough to be secure if you know what you have. It is impossible if you don't. And so I would say it is a it is a mandatory step in secure software development if you are hosting applications, kind of nonnegotiables. Yeah. Brian or Scott, anything else to add there? I think not much. Right? I think I think Colin more or less covered it from from, you know, the perspective of, you know, what it is and how we're utilizing. I think the the importance for me, when we talk about infrastructure as code is that, you know, this just makes it a lot more easy for you to go through a a repeatable process in the same way you would with building your applications. And so you have a lot more really the ability to make it easy to remediate things when something is wrong, rolling back, understanding what all the things were said, all the security I'm policies, everything associated to creating all of the, the infrastructure. So it's not just about, you know, making it easy to do that, but it's also how do you go back and figure out how do we debug this, or where are there there are vulnerabilities, or what happened inside this this attack, for example. Awesome. Awesome. So let's share the results of the poll while we ask a question from Alex about LLMs. So with LLMs training with public data off the Internet, what considerations can be done with ensuring Jenny I doesn't train on malicious instructions or later get injected into production environment using the LLM? Brian, do you have any thoughts on that one? Yeah. This is a, you know, this is a really tricky one to, to to do. But I think, you know, what it comes down to is that there there is absolutely always going to be insecure code that's that's that's created by LMS. There's just no way around it. Right? They're training off of public code, and in public code, there's a lot of vulnerabilities. Right? There's a lot of things that organizations are doing in mitigation of that for the training of the their applications. But, once it's once it's in there, you know, for from the the end user side of it, this comes down to, again, making sure you have those guardrails in place to run your your test against the application to detect if vulnerabilities are introduced, as part of that. There's the the other aspect of this is, you know, being able to use context and RAG and some other various fine tuning aspects of it to prevent, you know, known sources of areas where that may be a higher risk. Right? And and and speaking of, known sources, the other aspect of this is making sure you have some sort of attestations associated with the code that's being generated as well so that you can investigate where that comes from. Some some tools out there today are doing that. It's starting to become a new and more, required aspect of that. If you look at the EU AI act, for example, that's going to be part of the act that's introduced there to show all the attestations of where all the AI code is being generated from. So I think that between the three of those things, you're starting to reduce that risk. But really, you know, making sure you have the gating process, that you have the secure application tools against it to to, prevent, you know, things that you're developing going into production. Awesome. And and and sorry. I was just gonna say if we go back to the poll results, it seems to me that most people are still on the fence. You know, they're still they're still getting to learn things about AI. They're still figuring out how to use it. They're still figuring out how to use it securely, and I think there's a lot of experimenting going on. And the the big word of wisdom or word of caution is, you know, you're uploading code into these environments. And if that's your crown jewels, you've gotta be really thoughtful about what happens in the event one of these companies get breached, because I don't think it's a case of if, but a case of when. So, let's let's start the last poll that we've got. We've got a, another question, another poll here. I will leave that off to the side while I ask a final question for everybody. So if you had one piece of security advice for anyone starting out in their DevOps career, what would it be? And I'd love to start with Scott. I had an initial thought, but I I switched it. And I'm gonna say, it's just as important to protect your DevOps environment as it is the product that you're working. And so it's it's both the environment as well as the whole DevOps process itself. Awesome. Thank you. Thanks, Scott. Colin, what about you? Yeah. I would say anybody starting off on DevOps, learn the infrastructure. If you need to understand what these components do, you need to understand how they're built, you under need to understand their their histories, their evolution, and why they behave and do things the way they do, then you'll be able to secure them. If you start with controlling what they do first, you and you do not gain that deeper understanding, you're going to have a hard time on your hands because you you will be doing things, you don't know why, and you don't know the implications. Awesome. Brian? Yeah. Just to piggyback off that a bit, it's, you know, embracing automation where possible so that you have the visibility. You know, there's a lot of different vectors to manage when you're building secure software. And so having that visibility because your automation's running and giving you that in real time as you're doing your development, it makes it much easier to have that that that, you know, understanding of where things are coming from and what your next tasks are going to be, as well as helping you stay ahead of, you know, how do we do remediations? Well, if I have coverage across all of this stuff, it's easy for me to look back and figure out what my root cause is and what I needed to go and do and fix. Awesome. Thank you. And I'll I'll close with, hey. You can be the best coder in the world, you can be the best dev DevSecOps person in the world, but if you're sloppy, security practices on your laptop or the environment you clo code in, then you can bring down your company just as fast. So let's, wrap up. We're, just on the minute here. I've got a few housekeeping things. One is we are gonna send a survey at the end of this. The survey is gonna have some, great questions for you guys, so we'll appreciate it if you'd answer that. That'll help us as we move forward to continually improve, how we're doing these these events. And then secondly, there's a load of documents, courtesy of GitLab. So please, please download them. And then finally, we will share the recording out. Feel free to share the recording, plus all the assets with your friends, your families, your loved ones, or anybody else that you think might benefit. Give us a like, give us a thumbs up, subscribe, or whatever you do, but follow GitLab, on the the socials, and you will learn more about all of these events. Thank you, everybody, and have a great day. Thank you. See you all. Take care.