Whether you are triaging alerts or probing a full-blown incident, security operations center personnel must first ask questions, from “Is this a real alert” all the way to “How did the adversary exfiltrate data?”

Formulating questions and then applying what data the SOC has at its disposal to answer them are foundational skills for doing sound detection and response work. But there is a secret sauce to the process that may be less apparent to some.

Agatha Christie, the famous detective author and best-selling novelist of all time, wrote in one of her most popular works of fiction, “Murder at Vigarage”:  “Intuition is like reading a word without having to spell it out. A child can't do that because it has had so little experience. A grown-up person knows the word because they've seen it often before.”

Download "The Future of the SOC" Paper from Google Cloud and Deloitte

Of all the top traits the best investigators embody – curiosity, patience, persistence – perhaps the most important is one which many may mistakenly attribute to instinct: knowing when to persevere and when to pivot.

Yet for virtually all of them, deciding which path to traverse is less about intuition and more about data-driven proof points, something that can only be acquired through experience. In other words, what may look like a sixth sense is likely the product of far more habit than hunch.

The same truth lies in the SOC. To cement the instinct of when to stay on track and when to switch gears when tracing a threat, training is far and away the most obvious solution, especially during a time when your SOC success story remains very much about your people, despite the significant challenges they face, from alert volume to talent shortages.

Google Cloud Principal Security Strategist Dave Herrald recently joined the Cloud Security Podcast with Anton Chuvakin and Tim Peacock to share some wisdom about how to make training effective in the SOC, one big factor in keeping your team – especially earlier-career analysts – operating efficiently and feeling engaged. Here is some of his advice.

Make the Training Realistic

Creating as real-world scenarios as possible is a no-brainer when it comes to training in any domain. But the question is, how do you do it right, especially when there are so many variables that can limit a drill or exercise from coming across as sincere to the participants. How can you achieve this in an artificial environment?

1. Create a lab that represents, at a smaller scale, the environment you’re trying to protect

Data is integral to any SOC investigation, so you’ll want to make sure training replicates real-world threats in as realistic a manner as possible, from log sources and IP addresses to traffic patterns and indicators of compromise.

2. Procure legitimate vendor “gear” that represents what you have in your environment

It is best to practice on the tools you are actually using, such as SIEM, SOAR and threat intelligence. After all, these are the technologies that you will heavily rely on to help in your detection and response effort– and if you’re not familiar with using them, it can significantly slow you down.

3. Use a “golden image” from an endpoint

These are pre-configured virtual machine templates that represent what an end-user (one of the likeliest vectors to introduce something malicious into your environment) might be working on every day.

4. Purchase cloud services

With workloads migrating to the public cloud in droves, so is the magnitude rise of malicious attacks and user errors. Yet many SOC teams are struggling to implement effective cloud security controls, despite the technology’s allure. You will want to practice on what your organization’s tech spend is preaching.

5. Run real exploits (with proper isolation) and ensure telemetry you collect is what you encounter in real world

Generating “dummy” data that you program may be suitable in some instances (like triggering a simple rule) but when you’re trying to train personnel to pivot through related data sources, it becomes difficult to make duplicate data, such as IP addresses in multiple sources, match up.

Consider the Pros – and Cons – of Competition

Competition can be incredibly useful, especially because it can serve as a motivator and force the mind to consider all the things at play during an investigation, including cognitive biases. Yet leaders need to be aware of the downside of competition as well. For some, it can be a demotivator if they lack the confidence or even desire to be a “ninja” in the SOC. For others, it may be that the content of the contest doesn’t interest them because of their skill or career level.

Implement Scenarios That Embrace DEI

Security experts agree and studies have shown that a more diverse team – hailing from different backgrounds and bringing a wide variety of life experiences – makes an organization’s security posture stronger. A SOC team with diverse backgrounds brings distinctive perspectives, different ways of analyzing problems and novel approaches to finding solutions. This same equitable approach should apply to the education used to train SOC teams, as scenarios can easily become non-inclusive quagmires if organizers fail to consider DEI principles in the language and characters used in drills.

Learn how to fight cybercrime at Google speed with Chronicle Security analytics threat hunting & detection platform. Join the Chronos Challenge!