SOC: The Good, The Bad & The Ugly

My name is Rish Kumar Gopal Krishnan and uh uh I'm here to talk about the security operations center the good the bad and the ugly parts of it especially um so I started my uh career in IT help desk the the real plumbing of it and found my passion cyber security uh not through the show Mr. robot but it did have a pretty good influence on it. So uh and then uh so and then I got myself self-studied my way into the field once I found my passion and uh yeah I I got my promotion over there and the thing is that I learned a lot about the uh so there were there were a lot a

lot of things that went bad to my career and not not like especially to me but the company itself. So I kind of learned a lot from that experience. I have seen sock analysts that I mean sock team being handled poorly by the management. I have uh I have I felt this thing of losing clients because the performance were not that good. I have being part of active cyber incidents and uh also also help new clients integrate their way towards towards my company you know from ground up. So I've been through worse and all of this kind of put into slides. So these are not just slides. These are just experiences that I've gone through and uh thought I could

share with you all. So I'm currently studying masters. I took a break from my u work and do my masters in cyber security in university of war. Uh that's why the x not because I quit. So how this uh how this okay the top goes is I'm going to just talk about the basic the goods the good part of it which you probably would have heard outside the bad ones again all surface level stuff but then I talk about the ugly ones these are actual stories um that I have been part of I've heard from other analysts and other professionals and the takeaways from all of the stories that I'm going to be telling and uh sock team being in a sock

team is a very boring and a very bland life also demanding. So how do you level up there? And finally I'm going to introduce AI into it because why not? Because will that make any better for security analysts? Far as the good um I'm just going to be posting some of the comments. These are actual comments from uh just from Reddit. Um tough. So again first step I uh so this is L test of the cyber security is what they say and uh everyone every most of the cyber security professionals I've met have started their career as a soft analyst so most the entry- level roles are here so if anyone's transitioning into cyber security or starting now their career

pretty good place to start and the adrenaline as I said I've been part of uh cyber active cyber instance right So when you find that rogue PC or the evil user that's been spamming the whole network, that's a that'll give you a pretty good rush and not many jobs do that. So that's something to look out for. Blue team here you learn the uh the skill of team team work and uh you soon be you feel the our bad brotherhood. You soon be clicking viewers with people. It's a pretty good place. And the impact, this is not a job job anymore. You'll actually feel like you're part of a mission as um you'll actually be saving

companies thousands of pounds in some cases millions. And the bad you probably must have heard a lot of not so good things about Sandal because uh on the on the internet and stuff, which I wish I could say just don't listen to that, but unfortunately it's it's all true by the way. So is not a good place and like people get burned out, people quit stuff happens. Um some of these are like okay human filters uh most of the alerts uh in the uh I mean most of the time spent in the sock will be like just filtering noise. Um we'll be uh dealing with all sorts of uh alerts all from all different places and especially false

positives. I have an interesting take on these false cluster which I'll come back to later but um yeah the the solution to this is automated enrichment you can't you as a sock analyst cannot be doing uh responsible for this you'll be working with the engineers the sock manager to enrich so the alerts which you have on your dashboard is uh only real alerts not the ghosts shift handoff again this is something someone for who's like coming into the field just letting you know um this is something that every shift does at the end of their shift to do like a one kind of a brief meeting of what happened in the uh uh in their shift. This always

goes bad. I've seen this always going bad, especially during night shift, people, they they kind of after all working all of this stuff, they kind of they just want to close their laptop, throw it off the window, and they just want to sleep. But they kind of dump this whole uh information of what happened in the shift into the next shift people like in a drowsy, slurry voice doesn't always go good. So the uh the solution is they always do this report at the end of the >> Sorry, I was wondering like attackers like sometimes launch attacks at 6:59 a.m. to like >> Oh, yeah. This is like a strategy thing and stuff. >> Yes, it is. And some Yeah, especially I

mean we get attacked especially during holidays by the way. We we're kind of pretty chill with the holiday vibes and stuff, but that's when that [ __ ] happens. Oh, I've got a story too in the slide. I would be interested. But yeah, so my point is that should during handoff uh the shift there the whole shift doesn't do like a report they do do this at the end of the time they kind of this team lead kind of gets the report from all the other analysts all in one time one place and they dump it on the next step. That's not how it's supposed to do. Everyone should have their own personal report. like when you have to

create a case, when you solve a case or like uh escalated case or something, you have to create like a oneliner report of you did this with this and you kind of closed it this time. Something like that. This is uh something people still don't do with soft analyses by the way. And then false positives. Uh uh so false positives are not a thing. Uh again before people just get their hands on eggs and tomatoes and throw at me. Uh this is something that cyber professional I've uh I've spoken to cyber professionals and people say these to um managers and everyone false positives if you think about it every system account if it triggers an alarm for something

that is designed to do that is not a false positive that is a tuning problem. Again first I think uh John Strand famous influencer u cyber security influencer said this first and u and then people have started to realize this is a thing. So your tools are is doing what it's supposed to do. It's just letting noise in. The point is the the the work you need to do is to tune your tool as much as possible. The more uh alerts you it allows you in the more false positives you have and the more time your anal is going to waste. So there's a research that so there's 15 minutes that's been wasted for every analyst for every hour because

they're going to deal with this false positives. So probably the only solution is to tune your uh tool as much as possible. Me as a okay I have worked with my manager every time I get a false positive I I I go and talk to my manager saying like why is this a false positive? What do we do with it? just have this meeting. I know everyone wants to like do their job, just shut down the laptop and go, right? But this is something you need to step out of your field and just talk about it. Also, false positives I think um you shouldn't be um worried by this, you know, you should be terrified of false

positives is what the takeaway does here. NPC problems something uh we deal with this too a lot. So sometimes user when we do this uh fishing simulation exercises and the users they kind of create a P1 ticket because they received the email or they clicked on it and they just want to they just this kind of drives us crazy by the way because we deal with lots of actual alert like threats and stuff right and then these users they just create tickets and we need to the the whole process goes through like to close that ticket. So this is again something is very annoying. So again this is something we have to work with the engineers to make

sure that these gets auto closed by the way because this is no value for us. I mean we can't tell them to stop creating ticket. I mean that's it's better than not sending creating a ticket right. So again just auto closes as much as possible and send a report. I mean send a like a response or something just saying like dude this was just a just a test. Fine. and um anonymous. Sometimes you put all of this efforts into closing these tickets and solving problems and you don't get noticed much. This is something um many of the analysts go through and one pretty big reason for them to like quit. Uh what you do here is the only reason

the solution is you you kind of list your name in as many places as possible. If you're solving an incident which doesn't happen much but again whenever you do try to add your name in like specific actions you did right in in in the notes um also another create runbooks I am sure even till date not many I'm sure not all of the organizations will have a runbook for every other use cases so take your okay uh take off your or your sock hat and just like deal try to create like a methodology of how would you uh how would you uh attack I mean how would you kind of create solve this problem this alert and this is something

I have personally done a lot of times with the managers and then you kind of get to put your slap your name across that document and that document stays for a while and this is something you can get your management to notice get your brownie points and by ticketing. Uh sometimes uh you you see a ticket and uh that's mclassified or miscatategorized, right? And uh you you understand that is not a password reset but an account takeover at times like this even especially night shifts um it's not easy to categorize this I mean reroute that ticket you kind of have this battle with your ticketing system. I think this you can relate help desk or talk can relate to this. But you

got to when you see those kind of tickets, you need to change 12 mandatory fields and um you kind of do all these categories, subcategories, drop down menu options while you're fighting the UI, your attacker is just going to is just going to freeze your like just all your corporate network data just goes away. So again, this is something that I still see an issue with. I don't see anyone's um implementing here. So at times like this you need to have an emergency route where you don't need to they don't need to create I mean fill out these mandatory fields. Uh fast routing is what the solution to this is. So at times like this it's okay to not be the

perfect and just as close to ticket as much as possible. And I'm burnout by design. You know, all this dealing with the noise and and and the the stuff and their stress is going to just lead to burnout and quit. So the only the solution to this is they need to work with with your team to reduce as much as low noise value uh low noise alert as much as possible. So at least so the the ones you work with are like high value ones. And then uh yeah why stay or why leave uh 67% of the sock analysts they considered due to this this this distress uh I okay the time that I worked about two years I've seen 10

analysts quit because they have this dream of of of becoming cyber warrior and then when they the reality turns into this control C control V kind of machine because all you do is just copy each other in this nodes and you just slap on it and you kind of close it just change the IP address and stuff. I mean that's basically what sock analyst is but again that's you shouldn't leave that point. So analysts stay to evolve. You will soon see automation being come into the picture. I mean if you're working in as small to mediumsiz business, it will take a while because they don't do they don't u bring in automation that fast. But when it comes

you will have the u luxury of time and when you have you start to work on more niche alerts. You start to you will have time to study upskill yourself and uh you will finally you you see being yourself consulted by other teammates for some alerts and you see yourself evolving and this game finally starts to get fun and I'll move into the ugly parts of it. So these are stories I have personally gone through like I said um it will be uh it's something to look at. So plan was just panic. So I had this uh client who had this beautiful incident response playbook, right? Um 50 pages spiral bound. Beautiful. But then uh 3:00 a.m. come there was one

day it was 3:00 a.m. and uh we had a high critical alert and then do do you want to know how many people opened this 50page masterpiece? Zero, right? None. So what happens is everybody have their has their own runbook. That's what I see with so many companies and even people tell me this. So I have on that particular use case on that particular incident I have one I saw three analysts in a team and in a team's meeting just it was a micro way moving moving that way. It was just loud noises and stuff right and one analyst was immersed in logs. I don't know it was just trying to find evidence as much as possible. like

it was like that Titanic movie just finding the gem. The other analysts was he was just blocking IPS left and right. Um I don't know where he uh got the information from but but then there was one analyst um he was a still a dear friend of mine. We all have this pro AI person you know everyone and and he does AI and everything. So this guy he just uploaded the whole scenario and Chachi PD and he was like what would Mr. Robot do in this situation? When I heard about this, I was like, "Wow, this guy is so cool." And I'm like, but then this so and the manager on the other hand, he

was just calling people from the previous shift. He was like, "Did you see anything weird on your shift? Did you see anything before you log out?" So, this is what happens when panic takes place, right? And the only uh uh I'll give you the takeaways at the end of it, but this is why tabletop exercises exist. then failure to lead. Um so there was this junior analyst um in my team who saw this who spotted a data expiltration case. It was going on. He knew what to do from top to end and uh but he did not do any of those. He theoretically knew all the steps what to do. But what he did was he was just he just put

on his investigator hat and uh he was just going through logs as much as possible taking screenshots and he was collecting evidence and he just pasted in his email sent it to a manager and waiting for for his reply to become to be the leader to do something. Right? While all of this is going on the data was just flying out out the door. So at the end of the day before I mean when when the manager reached out to to to for the email responded it was a it turned out pretty bad actually. The client had to get involved. It was it was pretty messy. The point is that sometimes you need to take make a

decision to call people. The nominees analysts do this actually. they they kind of go into this silent mode and they let even it's okay even it's okay if if other person can talk. So I mean no one wants to be awoken at night um when from deep sleep and like to listen to someone say uh I think something might be wrong here you know but have these questions ready just what is actually going on what steps did you take and what's uh why are you calling them so that that that's the difference between uh uh an analyst who's trying to make a who's trying to get it right and a professional who owns the incident

tunnel vision. So this has happened in my u my friends companies. So over a weekend few of the analysts were working on these 50 lobby kiosks and uh so because they were generating like 5,000 alerts. So meanwhile there was this alert that just slipped in. This was uh this is for the patient's record database. This is important though. Healthcare and manufacturing are two of the 16 most critical infrastructure and the top two that's been that's targeted by ransomware threat actors till now till the you can you can literally just open up any weak news segment and you'll find two of these sectors pop up. So if you're clearing uh popups from lobby kiosks and um your

patient record database or your or your robotic manufacturing controller database gets hijacked, you lost the game. My point is know your crown jewels legal blind spot. Um so I was I was in a um I was in a meeting with the seuite executives. Um this is have this casual kind of meetings with the see what's up with everyone kind of thing and there was this question that popped out does does anyone know the difference I mean the connection between our cyber security team and legal and nobody knew the answer everyone was exchanging glances and the manager next to me was like just to say something don't be silent that would be the worst anything is fine and since he said

anything I was This is a trick question, isn't it? We don't have anything to do with legal is what I said. And the manager was like, "Dude, you could have said anything. Literally, just just don't talk to me." So that's fine. We didn't we didn't have exposure to legal. I mean, we had documents that mentions legal protocol, but we didn't we didn't get any exposure. My point is connect with your legal team before fire happens. Right. Finally, zombie reports. Okay. So, at one time we we found this critical unpassed vulnerability in our firewalls and uh we put that in a email summary, send it to the client. So we put that in a PDF, send it to the client, the client

put that in a folder. Six months passed by, no actions were taken. We didn't follow too. I mean, we didn't care. Our job is done. Um, but then their pentesting team found this vulnerability and then the client afterwards they came to know that we we found this way before. Did they were furious. Did they blame themselves? No, they blamed us because we didn't shout enough. So my point here is that don't just u put any of these important information in just documents and forget about it. Create a ticket operationalize these things. So my golden rule here is that if it ain't a ticket in Jira, it does not exist. Takeaways here. So what what did we learn from all of

these stories? Tabletops. Uh this is a pretty basic thing for like a sock team, but I still see I still hear from many of the sock analysts today that they do not practice this at all. Um you got to do this with your runbooks open, not from your mind. And if your map if your team successfully su uh contains a simulated breach but then don't they don't follow the the playbook you your team lost by so if your plan diverges from reality you update the process you you you need to do this at least quarterly you can even have fun with tabletops my my team found it was quite fun to do this we kind of do this

every month but do it at least quarterly Then um establish incident command. Again, this is basically like you need to know who's dealing with what and uh you don't want to go you don't want to find out very late that you need to like contact this person to do to to taking care of this, right? Then notification checklist. You don't want to be scrolling through a corporate directory at 3:00 a.m. figuring out uh again who takes care of what. So have like a small sheet of just like one page. Say if it's a P1, if it's a category one, you notify these people. If it's category 2, you notify these people. Just have like a sticky note or

whatever. Just have it be ready with that. Identify crown rules. We saw that uh lobby kiosk versus um with the patient database that a record database uh scenario right so this is something you need to work with your business stakehold I don't I know we socks do not have an exposure to all of these like a business stakeholders managers don't do that extra work kind of want to do that but I actually took uh a step in my team to speak with these business stakeholders to know to see if these particular this set of systems goes down will that bring the whole company down know those first so that we knows what what we can defend

and uh the best person is actually a CFO you have a meeting with him you're good it doesn't see from their perspective it doesn't really make much sense for us to be in a meeting with them but it can really enrich your your experience of being a soft catalyst ticket everything so like the zombie report I told you about before. Every lessons shouldn't be just just uh and it should shouldn't just end up in a email or a document. Always turn into tickets because those zombie reports are going to come back for your contracts because an analyst actually got fired at the time. It was it was pretty bad. So, uh then invite legal. I know legal team

don't want don't want to hang out with us nerds. They're all they're they're so cool though. But but you got to invite them. I mean it's not b it's no benefit for them but it's it can really enrich your um experience. Even even as simple as a tabletop exercise if you can bring a legal team to your tabletop it will really valuable. Um I mean so what we did is so there's you can there's different ways to like invite them. just drop them like a GDPR 72-hour notification list document to them and say it's an audit requirement. I see I seen pizza works just as fine. Just say you're buying lunch and you see that they're going to any they're going

to send out some legal person into our team that and uh re-educate the team. When you survive an incident again don't just put it in an email summary. do like a little video thing or or a brief bring the whole team in talk about how the threat actor got in like what happened what did you lose what's the lessons learned and do like a course presentation whatever but yeah educate the team and simple uh again talk team is going to be very hot very uh demanding it's it's not easy but you can make your job interesting Um, so something that I did uh to like pass time to learn more about is like follow cyber security news and

there are a lot of programs, a lot of YouTube channels out there. I follow Simply Cyber believe it's it's a pretty high energy kind of and then the whole stake on it. lots of knowledge bombs being dropped over there and gamify the network the the work have these little games between your team like uh say not just like CTF challenge or anything right like or like who creates who closes the ticket as fast as possible no no not that but like who can clear the cleanest ticket who can brief the client you know well something like that and uh upscale yourself again basic stuff just videos courses certifications whatever cuz that's why I did so when I was in

health test I just did all these certifications all free by the way and just posted on LinkedIn the management noticed me and I was like they gave me promotion that's how I got into the field by the way so don't just stay where you are at right now wherever you are with in terms of career just keep moving just just be updating yourself and step up whenever you get a new case alert issues even new client always be the the first to like step up because so in my case we had this new finance our first finance sector client come in and um so my manager was looking for the best people because it's a high intense

environment. I was the I was new to the team and I was the first to step up and I was like dude sign me up I'm doing for I'm up for this. I mean eventually we lost the client by the way because how do I put this? We sucked at it. We we suck pretty hard. So the point is it was a valuable lesson to me. I my me personally I learned a lot of stuff from a client from starting point to losing it was quite an experience for me to learn and uh that's how I am here where I am and finally again without mention of AI uh there's nothing we can do but will AI

makes things any better? The answer is no. No way. So team noise in 2026 is going to be worse, way worse because we're getting logs from all these different places. IoT logs, cloud streams, APIs. It's just too much now. And uh AI is not making anything better. And now the adversaries are are using the bad guys are now using adversarial AI to poison our baselines and even mimics our users. So if you are going into 2026 art even the next few years thinking that AI will make things better no you're just is going to be deafened finally uh so there's this um uh again fear that AI again taking over our jobs right and especially with the sock team

yes it will take some of the jobs but that's not how you look at the board though the the current entry level job is not tier one anymore. It's tier 1.5. It's tier one with advanced AI capabilities. So basically you will be competing to enter the field will be competing not be competing against AI but you'll be competing against someone who can wield AI as a weapon. So that's how you should see the the the job market now. And um and there's also this uh fear of AI exhaustion. Now every tool has a shiny AI tool there and uh your management will want you to push all the buttons as much as possible. But um remember so there's always

so five multiplied by zero is always going to be zero. So you definitely need a a human to take care of this to see if the AI or to make big decisions, final decisions right? Uh so always I always say this um with my team even the with the new people that's coming into my team don't let an incident go to waste. you always have you would have gone through. So I personally have uh gone through uh losing clients, losing people, getting scolded at not knowing anything in board meetings or or just just straight up verbal abuse. But I did not let that go to waste. I learned a lot of stuff from that experience. So that's what I say to

everyone. I think that's all for me. Uh, I'm done with it. Any questions? >> It sounds like, you know, every everyone's quitting, everyone's, you know, maybe getting fired doing stuff. I was wondering like to what degree does it take for a mistake to mean you lose your job and like if if to what degree is like everyone quitting and everyone you know losing their job in this area like but it also doesn't really align with like how many jobs in this area are available they're hiring you know >> again this this is the uh I think of all the cyber security jobs out there this is the uh this is a job I see a lot out

there kind than than any other because again you might not need much experience or technical knowledge to get into this but that's what they but once you get in you see all of this high level stuff knowing the seam tool store tool and then all these networking thing and then they kind of go into this job but then realize it's too hard but then they quit this kind of happens a lot so always again I always say foundations lay your foundations networking always know networking. Um they always understand that we don't need networking because we I'm not going to a networking kind of job like network engineer or whatever but cyber security that's a pretty basic

thing but and again there's no basic cyber security terms after the networking is done and uh again certifications anything I would love I I didn't pay anything to get the job by the way all I did was free courses and um you know with AI you can just simulate your stuff like what people do in everyday life. >> I think I should do it. >> The second part of the question like agree with it have to be like a mistake to be you know cooler over. >> Okay. Um so there was this okay this is something that kind of uh uh talks about your management not more than your uh because one of my uh

teammate analyzed he made a mistake and it was like my manager it was a pretty big mistake so he kind of contained uh a system during an incident and we weren't supposed to do that but it was again during a night shift time so it No one was there to call and everything. My manager took the decision not to throw him under the bus. He was it was he was he was trying to defend him in front of the client saying that yeah we as at the time that is a valid uh action to do that and he did what he did. So that if if it were another company if he did put him down the bus he would have quit

right away. And I have seen my friends quit because management was weren't good that good. And uh every time you do an action, you need to get an a if you need to get a a yes from a higher authority and then even if you do, if you will make a mistake and you still get blamed, yeah, you you have people quitting all the time. So >> So it's a lot of blame game. >> It's a lot of a blame game. If your m if your team is good enough to support you, whatever mistake you make, like I said, the containment system is not something an analyst is supposed to do. But again, at the time, he was the only analyst in

the entire ship. No L2 or anyone. But I still agree. He took the decision to make that. >> What would you have to do to get into trouble? Sorry. >> Um Okay. So, so in the in my case, okay, one of the stories I said, uh you know how some of the uh one one analyst was just blocking IPs left and right while an incident was going on. He didn't follow Facebook playbook. That was not a good decision because a lot of computers were were blocked and and there was a whole team that went down because of that. So again it is as an analyst you're supposed to follow table I mean your runbook playbook and then uh always

go with the flow. is at the flow with the runbook not you don't go by yourself it's not a oneman show the only the point of to take care is analyst is being a sock analyst is not a oneman show it's a team effort you follow the rules you follow your team what they say and you go with it so >> you know you follow the book more or less fine >> yes you follow the book as much as possible and also oh another thing if you drop something in AI because I' I've seen someone do that too they drop this whole everything the whole scenario including the IPs IOC's all of that stuff >> you're done.

>> Yeah. >> If someone finds it out but if you do it like but no one knows I mean I'm saying I can't people do that but yeah you're you're basically done. >> One did get fired because of this same. Yeah because someone ratted him out but >> yeah but you you enjoy you enjoy this. >> Oh yeah it's it's quite hard uh to live here. But again, good foundation. It'll give you now I'm moving on to GRC is uh from the from my sock. But but I'm pretty confident now more than ever to move into the strategy uh position because I've I've been in the trenches now. >> Do you think this is good time? Sorry,

I'm taking questions. >> Oh yeah. >> I had a similar kind of thing. we were share each other so I found with the amount of time that we're actually working at night but with all that obviously you're working how do you find time to then skill yourself up you know if I say to networking How do you find time to do that again considering things? >> I think uh my answer was passion. I think I I was working in health desk uh IT uh support for like a while. I got to a point where every day every call I pick was was a real um it was a hell for me. It was to a point and uh you kind of

get that burnout, right? I think uh I've seen so I didn't want to again I did lose a job by the way. So I had that motivation to push out of the u that place where I'm at and go into something meaningful valuable. So you find time actually I worked in night shifts and still found time to study upscale myself and when you follow like certain people in LinkedIn and they kind of motivate you to go forward and LinkedIn is a pretty good place for you to like motivation to study to upskill yourself. I had this uh I think when I when I started out uh there was network chuck and uh he he kind of did did this

networking challenge. You do the certificate in like 30 days kind of do it with me. Can you find stuff like this? I'm like I'm not alone. I have people with me like in like online. So I kind of did that and uh that kind of helped. >> Thank you. >> Yeah. Would you recommend security? It sounds like there's a lot of positions available >> and you know like you said too much. That's yeah that's you >> that's a yeah it's a myth but yeah more more people just get into this with I've seen people get into this with no idea of cyber security like I got into this with sec plus everything just the sec

plus but I've seen people getting know even like not basic networking stuff too but they kind of struggle their way into learning but yeah it's possible to get into this but I I completely uh recommend getting into sock First before getting to anywhere in cyber security, it's you should learn all that that stress handling and everything because cyber security at the end of the day is is just about saving people. Okay.