Cloud Control: Q&A with DryRun Security's Ken Johnson on Contextual Security Analysis

April 23, 2024

Want weekly newsletters featuring interviews with security and cloud leaders delivered right to your inbox? Sign up for Cloud Control here.

Unpacking Contextual Security Analysis with DryRun Security's Ken Johnson

Hi Cloud Control Readers👋

Grab a coffee and settle in—I just had a fascinating chat with Ken Johnson, the brains behind DryRun Security (more formally known as the Co-Founder and Chief Technology Officer). Ken gave us an in-depth look at Contextual Security Analysis and its practical applications in today’s software development environments. We explored how this approach is reshaping traditional security testing and discussed strategies for weaving security more naturally into the developer’s daily workflow. If you’re keen on the latest in cybersecurity and application security, you’ll find plenty of valuable insights in our conversation. Keep reading for the full interview👇

Question 1 💭

Ken, I’m excited to have you here! You started your career at GitHub on their Product Security Engineering team, and have now launched DryRun Security - which is incredible. What led you to pivot from leading internal security code reviews to launching a startup? What opportunities did you see that kicked you into action?

Answer 1 🎯

Great question! My career started almost 23 years ago when I joined the Navy as an IT. But, my first “real job”, in terms of application security, was at the Pentagon about 16 years ago. Since then, I’ve bounced between consulting, and internal defender positions at places like Fishnet Security, LivingSocial, and more.

You mentioned GitHub, and this was really the place that forged in my mind the need for a broader style of security analysis. At GitHub, we faced the same challenges as many other development organizations. One of those challenges was around how we tend to perform our due diligence in the initial launching of a product and we can say the application is fairly secure at that point in time. However, as code changes and the application evolves, issues get introduced. When that happens, in a best case scenario, we get reports from internal and external sources telling us about those issues. The incident response process starts, tickets get filed, etc. etc. But what’s the reason?

As I mentioned earlier, during initial delivery of a new application, humans perform many complex design and review tasks but when we talk about incremental code changes, the reality is that we security people just don’t tend to have the bandwidth to do this. Instead, we rely on tooling that most of the industry agrees is a “best effort” with many flaws in its results. We use SAST and SCA as our canaries in the coal mine and that’s about it. Two data points.

Some modern AppSec programs (like what Chime’s and Reddit’s teams are doing) have begun ingesting other data points like container scanning and custom rules from something like Semgrep and other tools but I have yet to see anyone bringing in more categories of information, some that might even be classified as “threat intel”.

The reality is, we have a lot of data available to us about who, what, and where things are changing that can be used in determining risk but those data points often go unused. Our goal, and one major reason I co-founded DryRun Security with James Wickett is to surface true risk using a multitude of factors. We call it Contextual Security Analysis (more on that later).

Question 2 💭

Over the years, you've trained over 10,000 developers on security testing and code reviews. What are some common misconceptions about security you see across developers? What’s DryRun’s approach to tackling these misconceptions?

...you have to take a risk based approach and use a really disciplined approach both in your note-taking and following of the methodology. It is very common for folks to start searching for easy wins in the code base, maybe run a tool, maybe chase a few leads, end up with very little, and then wonder how they should spend the rest of their time.

Answer 2 🎯

Another fantastic question. Training people is my passion. Thankfully I’m still able to take time out of my CTO role to provide this training because it really is something near and dear to my heart. We’re not necessarily trying to solve this particular problem at DryRun Security but one misconception I see, especially with code reviews, is that we’re trying to catch EVERY vulnerability possible during a review. Granted, I’m sure there are times when we’re talking about a super narrowly-scoped, artisan-ally hand-crafted, incredibly high paying engagement where you can take all the time in the world to thoroughly review every line of code… I just haven’t been involved in one in 16 years of doing this.

So for all the rest of the reviews, which make up the vast majority of reviews any of us will ever do, you have finite time and finite resources to accomplish the engagement. In this case, you need to take a very risk based approach to your reviews…. Ok I take that back… maybe this is EXACTLY what we’re trying to solve over here at DryRun Security. Jokes aside, you have to take a risk based approach and use a really disciplined approach both in your note-taking and following of the methodology. It is very common for folks to start searching for easy wins in the code base, maybe run a tool, maybe chase a few leads, end up with very little, and then wonder how they should spend the rest of their time. I’ve been that person. It took years of doing code reviews to create this methodology/approach. We (my Co-Host of Absolute AppSec and close friend - Seth Law) train people in this very practical approach, based in real-world experience, so they don’t have to follow that same path.

Question 3 💭

The concept of Contextual Security Analysis is quite new. Tell us how this approach differs from traditional security testing methodologies. Does it have any benefits - or drawbacks - for real-time code development environments?

Answer 3 🎯

Contextual Security Analysis utilizes a “SLIDE” methodology. Surface, Language, Intent, Detection, and Environment. The idea being, while SCA/SAST results are data points that tell us if the code change matches some pattern, those tools do very little to explain any other additional risk factors that might be important to us such as what is changing, who is changing it, and where they are making those changes.

Just as an example, imagine someone committing code that has never committed code to that repo before. Now imagine they are also touching sensitive authorization-related code paths, they’ve added new HTTP paths, and also introduced a security flaw according to the SAST results. To us, that seems like a risky code-change! You have to be careful to surface real risk and avoid inundating engineers and security people with noise. It is a challenge but I believe we’re perfectly positioned to solve it.

Question 4 💭

DryRun Security's philosophy of integrating security analysis within the developer's workflow is new and exciting. Could you share how this philosophy has been received by developers and security professionals? Considering this changes the traditional dynamics of code reviews, security testing, and probably more.

Answer 4 🎯

So far, we’ve helped detect PHI/PII before being merged into a code base, detected flaws in code as we developed our own SAST tool backed w/ LLM technology using a proprietary approach that heavily relies on context about the application in tandem with our own knowledge base, discovered secrets in source code pre-merge, and been visible to the developers in their workflow when this happens which we’ve been told is incredibly helpful. We also are used by the DefectDojo project, primarily to protect against changes to any sensitive code paths by unauthorized authors. Developers and security teams find it useful if their tool only notifies them when there are actionable results and we’ve been accomplishing just that.

I think supply chain attacks and attacks against devops infrastructure is probably the biggest evolution that I’ve seen. When I started, software development and infrastructure management were two different disciplines but in today’s day and age it's not uncommon to see developers performing both roles. As a result, there may be some knowledge gaps in how to securely deploy tooling as well as manage cloud infrastructure.

Question 5 💭

You have quite the background in web application hacking and security training. How has your perspective on the evolution of application security threats influenced the design and functionality of DryRun Security's solutions?

Answer 5 🎯

Thank you! I think supply chain attacks and attacks against devops infrastructure is probably the biggest evolution that I’ve seen. When I started, software development and infrastructure management were two different disciplines but in today’s day and age it's not uncommon to see developers performing both roles. As a result, there may be some knowledge gaps in how to securely deploy tooling as well as manage cloud infrastructure.

Supply chain attacks on the other hand have been fascinating to watch. The latest social engineering attack against xz (ssh) is a perfect example. When I worked at GitHub I had the pleasure to work with the team behind npm. They were incredible folks performing a difficult task and the engineering challenges they faced as a result of the very creative ways in which they were attacked was eye opening and impressive.

While we’re not focused on SCA, we do perform a different composition analysis of the applications we were tasked to help secure. With that information, I believe that at a minimum, we could alert on the highest/most critical packages your teams tell us they are concerned about and you could use our inventory (in a future state) to determine which of your applications are impacted.

Question 6 💭

The disconnect between developers and security teams is a well-known challenge. From your experience, what are the key factors that contribute to this gap, and how does DryRun's approach create better collaboration and understanding between these groups?

Answer 6 🎯

I do believe over time that the gap has gotten narrower and narrower with all the focus on empathy in many of the popular conference talks, podcasts, and thought leaders social media streams but… you’re right. In many places, there is still a large disconnect between the two. Our goal is to make our tool feel like a security buddy. It is not in the way, it is not meant to slow you down. It is meant to warn you, give you remediation options and guidance, and help you release software with confidence. If a tool can accomplish that task, it will go a long way in maintaining trust between the two groups and trust is everything.

I will say, not every piece of feedback should find its way prioritized on your roadmap. We look for patterns in requests/feedback and things that resonate with our shared experiences and understanding of the space.

Question 7 💭

Having launched a new product and startup, how do you uncover key learnings and build feedback into the product during the initial stages?

Answer 7 🎯

Oh my gosh, so many ways but the most important way is to ask people. Talk to them. Dig deeper. Ask people to elaborate on their answers. Listen, listen, and listen some more. We have many means of observability and cool technical ways we use data to improve our product but you can and should never get away from the core of your product which is the humans purchasing and using it.

I will say, not every piece of feedback should find its way prioritized on your roadmap. We look for patterns in requests/feedback and things that resonate with our shared experiences and understanding of the space. I also try not to take it personally because it is really sort of a “well, we tried it that way, but people are telling us it’d be better in this other way”. If we’re too attached, we won’t adapt the way we should.

Question 8 💭

Looking ahead, how do you see the landscape of application security evolving, especially with the rise of AI and machine learning? How is DryRun Security preparing to adapt or innovate in response to these trends?

Answer 8 🎯

Well we’ve seen AI used already in attack campaigns but alternatively, I think AppSec has a place in AI. We’ve already seen Huggingface have 1500 exposed API keys on GitHub where a malicious actor could use these keys to upload malicious LLMs. Speaking of malicious LLMs, we’ve seen that as well. I believe 9 valid and active malicious LLMs were discovered on Huggingface not long ago. So now you have supply chain attacks just in a different ecosystem. Not to mention, there are typical software flaws present in AI related systems. The most notable is prompt injection but there are many others and really they’re derivative of common AppSec vulnerabilities just manifesting in a slightly different way.

On the flip side, AI can be used to perform some really helpful application security tasks - things that deterministic scanning just can’t do. AST parsing has its place but we are advancing towards a new direction in how we analyze code bases using LLMs. Without going into specifics about how we did it at DryRun Security (sorry!), we’ve been able to prove it is incredibly useful in reviewing code. Having said that, you basically have to become (as much as one can be with this very new technology) an “expert” in the ecosystem and really, really understand application security as well as how to perform code reviews in order to do it. I would not say that “just plugging into OpenAI” will get you there today, as nice as that would be.

Question 9 💭

As we look to the future, what trends in cyber and cloud security do you believe will have the most significant impact on how developers and security professionals collaborate?

Answer 9 🎯

There really is no good future in software security without collaboration between these two groups and we’re seeing a shift in the right (left, hah) direction. I say that because this idea is seen in the “shift left” approach. But I think we’re now going even farther in recognizing that security and development truly is a continuous process and that there is no “left” in a continuous circle.

Now, with the advent of AI technology, we’re seeing it used to bridge that gap and enable real-time, cross team collaboration. For example, we’ve seen new solutions emerge to automate building threat models, remediating vulnerabilities, and products like co-pilot are advancing towards more secure software suggestions. At DryRun Security, we’re providing you with your own automated security engineer to help engineers understand risks and remediate them before they reach production. I’d argue that all of these methods are driving towards tighter coupling of security and engineering groups and enabling collaboration between them.

Question 10 💭

Navigating the cybersecurity world seems like a never-ending game of cat and mouse with hackers. From your years of experience, what's one of the most surprising or unconventional lessons you've learned about staying one step ahead?

As an individual, I may not be able to cover every possible way in which we could be exploited but by crowdsourcing, rewarding, and cultivating relationships with researchers we have a small army helping us stay a step ahead of attackers.

Answer 10 🎯

You know, one thing that really surprised me, at least at first, was how genuinely happy my former boss (Greg Ose) was when we received valid and important bug bounty submissions at GitHub. Remember, at that time, the bounty programs were fairly new. Initially I thought to myself, these submissions highlight our failures AND cost us a lot of money. Then, as I thought about it more, you start to realize that the researchers are presenting you with something a malicious actor could have exploited and finding complex vulnerability chains that most internal teams are unlikely to uncover due to time and resource constraints.

So what you are doing here with a bug bounty program is acknowledging this reality and being proactive about it. As an individual, I may not be able to cover every possible way in which we could be exploited but by crowdsourcing, rewarding, and cultivating relationships with researchers we have a small army helping us stay a step ahead of attackers.

Bug bounty programs completely make sense now, they’ve proven their value time and time again. But, at the time and at first, the idea was definitely unconventional to me.

‍