OktaFest – From Your Identity to Your Repository

let's play a game you are an incident responder you have been notified by OCTA that some suspicious activity is occurring in your environment you don't know what how or when in the same time you see spikes rising in gitlab dashboards you are a great incident responder so you imagine something like this a threat actor compromised account used SSO to access gitlab iterate over gitlab repositories to extract a token and return with that token back again to OCTA with elevated privileges now it's game over but how can you start investigating it before it's too late how could you detect it sooner how would you notice if someone used SSO without authenticating beforehand and how can you investigate

gitlab logs if you don't know how they look that's what we're here to talk about today we want to tell you a story of threat hunting in Octa and gitlab environments my name is Ariel Kalman and I'm a security researcher at mitiga this is my friend Gabby and he is also a security researcher at mea here is what we are going to talk about today we'll start with an overview of OCTA its core concept and the log structure it generates next we'll dive into an interesting in attack PA in Octa and how to detect it effectively moving on to gitlab we'll discuss different logs and various concepts related to its security we'll explore attack scenarios

on gitlab and how to hunt for potential security threats finally we'll wrap up everything with security remediations for both platforms before we begin let's clarify a fundamental concept an identity provider or IDP authentic indicates users and manages their access to applications and services by verifying credentials OCTA is a type of IDP and it allows users to access multiple applications using a single password one MFA device and unified security controls via SSO while it streamlines access and enhances security does it also create a single point of failure could compromising one user Grant access to m multiple applications not just one perhaps this is why despite OCTA being one of the most widely used identity providers globally it is also a prime

target for attackers over the past two years HTA has had experienced multiple significant breaches the most recent one occurred just four weeks ago many customers mistakenly treat OCTA as a comprehensive security solution however Brides like these highlighted the the need for a more proactive approach instead of solely relying on OCTA security controls you have to collect and analyze octal logs and actively engage in threat hunting but what do octal logs even look like let's make it simple in the next few slides we'll dive into octal loses we'll categorize the hundreds of event types into four main groups to simplify it we'll display the system log schema and highlight significant fields that Hunters must know and use finally we'll show you what

is not logged in Octa focusing on how the threat actors can enumerate your environment without being detected OCTA has nearly a thousand different event types sometimes it's hard to differentiate between two event types for example take a look at these two the first is application. policy do sign on. update the second is app policy do sign on. update the description for both update up sign on policy a bit confusing isn't it so apparently OCTA differentiates between app and application but they do not consistently stick to this distinction so to make it simple we divided all octave vent types into four groups first is authentication EVS where you can see user logins and logouts multiactor authentication events single

sign on application access and more the second group is management events where you can mainly see admin activity including policy modifications group alterations and Ro role assignments the third group is application events where you can see the actual integration between OCTA and different apps in this group you can see all what token generation and usage and also integration changes between OCTA and different apps the fourth group is security events which are somewhat mysterious these are security alerts generated by octal security product such as threat Insight because we don't know the logic behind them it's hard to trust them they are often spammy and not very informative but if you correlate them with anomalous user Behavior they might be

insightful now let's move on to the one and only OCT vent log this is the sole event log for OCTA where each operation is recorded it contains a variety of properties every log has like IP address user agent date and time but it also contain more interesting ones for example the actor property which is the entity that performs an action OCTA defines four types of entities user application IP address and system principle which represent an internal service of OCTA the second property is the target property which is the entity that the action was performed on OCTA defines dozens of different resource types including certificate MFA device user or group the third property is the debug

context property which contain one field called debug data that may have a variety of forensic information that can help you understand why an event is suspected to be suspicious when investigating an incident it's crucial to know that you can't see everything let's discuss what is not logged in OA focusing on how threat actors can enumerate your environment first all get requests to octa's API are not logged using octa's API a threat actor can enumerate your environment listing applications groups policies and more for example here you can see application enumeration by right actor me without being logged anywhere second some post requests to octa's API are not logged generally post requests that don't create an event type

are not logged in contrast to user creation or policy modification third all read access to octa's admin console is not logged it means that if a thread actor connected to your admin console you can't see is is read only activity now let's move on to the most exciting part of OCTA over the next few slides we'll dive into a complete attack p in Octa and how to detect each step for each step of the attack will present two detections and key forensic indicators that you can use in your environment the attack starts in Octa with user compromise and MFA bypass it then abuses SSO to access gitlab we'll skip gitlab part for now since Gabby

will cover that later finally we'll return back to OCTA with elevated Privileges and create persistence in a unique way let's dive into each step the first step is initial access the threat actor the threat actor needs to have a foothold in your environment often achieve by obtaining users password through methods like password spray or fishing attack the next step step is to bypass MFA we'll present two detections for it first is MFA fatigue this occurs when a threat actor repeatedly tries to log in to a real users account triggering push notifications after multiple denials the user may approve it to stop the notifications detect this by looking for multiple MFA failures followed by successful one the second detection is impossible

POS traveler this occurs when you detect two events from geographically distant places in a short time period for example if you see a login attempt from T Aviv followed by an MFA approval from Taran it may raise suspicion detect this by creating a g velocity function that will assess if the distance and time differences between events make sense the second step of the attack involves SSO once the threat actor obtained access to an account he can see the accessible applications by this user and can pivot using SSO detect this by creating a detection called new country SSO which detects an SSO connection made from a country you haven't seen before for this particular user to reduce false positive filter out

new users and common workplaces the second detection is malicious SSO while we haven't delved into OCTA threat Insight extensively it's a security product by OCTA that analyzes and create alerts for potentially malicious IP addresses although it may create some false positives combining its alerts with suspicious SSO connections reduces false positives significantly the final phase of the attack after which I will pass the torch to Gabby to continue the talk on gitlab involves a persistence mechanism that OCTA themselves have observed in the wild just four weeks ago assuming the threat actor obtained an admin token and now has elevated privileges how can they persist their privileges they can abuse octa's core settings course stands for cross origin resource sharing is a

security feature in web browsers that controls which web applications can interact with resources from different domains on the internet for example if the site apple.com tries to fetch cookies from the site google.com course determine whether the request is allowed ensuring that only trusted domains can interact with your site a threat actor can abuse octa's core settings to allow his malicious domain interact with OCTA API and bypass your detections for example here you can see me adding a malicious domain to OAS core settings and allow him to interact with OAS API now let's move on to talk on gitlab with Gabby thank you Ariel so officially our attacker has moved from OCTA to our gitlab environment but before we get

into the details and Concepts there's a bit more to to gitlab than you might think gitlab has more than 31 million users yes 31 million users who knew additionally they're used by over 100 th000 companies worldwide and most of their use cases are for company code meaning it's not only well dispersed it's also a prime goal for an attacker to access this with that being said let's understand what is gitlab gitlab is generally a One-Stop shop for all your development life cycle needs starting from code repository and and Version Control to cicd pipelines to managing tickets called issues kind of similar to jira an internal Wick and much more but for the sake of our talk

we'll focus on this that it's a repository so gitlab has a few different offerings it has three offerings one being a self-managed one being a SAS implementation and one being a dedicated one the self-managed is managed by our own infrastructure which gives us full control and high place for customization the SAS implementation is completely managed by gitlab.com which means we don't have that much room for customization but on the other hand we have a very low overhead for managing and the dedicated gitlab is managed by gitlab but it's considered a single tenant which means it's managed for a specific case which gives us some room for customization but it's usually used for compliance and data residency

in our talk we'll focus on the SAS implementation and the logs we could achieve through the rest API it's also key to note that the self-managed API uh self-managed gitlab has additional logs that we will not get through the SAS implementation so let's move on to some Concepts we have in gitlab although gitlab has a plethora of features and Concepts we'll focus on two four thread hunting actions one being groups and one being projects a group is essentially an organizational unit that could be represented by a company or a division so for example we could have a group being company X having a subgroup called division a under that we will have something that could be a that

could be a project a project could represent our code repository or some product or feature and what we could have here is again company X has subgroup a which has project one which is one product and project two which is the other product so with that being said we understood that gitlab is well dispersed its prime attacker and some of its key offerings let's move on to some of the logs we could find in the SAS rest API implementation and move on to our thread hunting process gitlab offers us two main logs that we could collect via rest API we have the events logs and we have the audit logs the events logs are known as

contribution events these are stuff like git operations when we're pushing code committing creating a new Branch Etc some important fields we could see from the logs yes I'm aware this looks horrible but this is just just to get it into your mind before we dive in or stuff like a time it was created an author which is the initiator of an operation an event name and a Target is what we did the operation on so let's dive into one of these cases here we could see that this dude did improve on this merge request which is our Target title and we could see that the target type was a merge request fairly simple but this is what kind of closes the

small story of what we could see in events logs but these are only some key Fields next we have something called the audit logs these are for GR group configuration and management of administration operations like creating new users new groups New repository and creating project access tokens some important fields we could see here can be similar to what we saw in the events logs but we also have an addition of an IP address here and here the event socalled is called custom message so let's dive deep into one of these cases in this case we could see that that dude created a project access token called besides 2024 as the target title and we could see as the entity path on which

project they created this token this kind of gives us a look of what we could look for and for you researchers or curious people out there try to understand why when we create a project access token it's considered a personal access token but I didn't mention what are project access tokens you're like what are you talking about so for that let's mentioned that axess tokens and gitlabs are essentially a string that allows us to connect with HTTP and git operations to our gitlab implementation without needing to authenticate we just need to show this token and we're able to operate so with that question in mind let's go on a hunt but before we start our hunt there's two things we want to

know to mention one a hunt is a assumed breach assumed breach kind of like method we're assuming someone's inside and we're actively looking for possible malicious activities and what we're going to show here today is something to spark your imagination we're looking for stuff that you might take from here but you would know your gitlab environment the best and therefore we trust you to know your own hunts but where did it all start we mentioned that our attacker abused their case in Octa and managed to Pivot all the way back to gitlab and in this case we could see that the attacker dude had a s group connection with our audit logs which indicates that they came here

through n SSO so our attackers right now in gitlab environment and arel mentioned previously that they want to have a key that they could connect back to OCTA so something they might decide they want to do is to download a full project gitlab allows us to have an archive download of full projects so stuff we could look for are event names that could say repository downloads started but enrichments could give could give us extra enhancements to ask oursel the right questions when we're hunting right is this user supposed to download these things when did this happen What IP did this come from and how many downloads are we expecting to see in our organization so our attacker has their

access token they're very happy but they want to keep a foothold in our gitlab environment so they have two things they might be doing the first is creating a per of access token as mentioned before an access token will allow us access back to our gitlab environment through rest apis without needing to authenticate so stuff we could look for here is again as mentioned before a personal access token but our custom message will tell us that we created a pro project access token which gives us a scope of an access token to the whole project and we might look for permissive API permissions or Scopes like writing apis creating Runners for cicd Etc Etc things we could enrich here are

expiration dates for example this won't be in the logs but if you have access to your system and you see an expiration date which doesn't usually stand with the standards of your company it might indicate malicious use how many tokens or project access tokens do we usually expect to have in our environment this is something you could also consider again IP location user agents are questions you can ask yourselves but with that being said let's have a a small chat about bot users gitlab's documentation states that whenever we create a project access token we're creating a representation of something called the bot user which is usually for internal processes and automation that will represent any

action done by this bot user uh by this token meaning that we will see something like project or group uncore idore bot something now why an attacker might want to use this is that every time we create a project access to token not only does it give us access to our gitlab environment it doesn't cost us a seat a seat in gitlab is basically an allowance of how many users we're allowed to have in our SAS implementation so if we don't want to alert any cases for additional seats a bot user will not alert it because it's not considered a seat so we could look for unusual spikes and Bot operations and unusual resources we know our environment we know when

they're supposed to occur and if an automation account what time is this happening why is my bot user accessing a bunch of things so with that being said the second method they might want to do for persistence is what I considered the old but gold adding another user that they might have access to giving them additional permissions we will look for something that we're adding to a user access o um kind of permissions such as owner or maintainer which are highly privileges but something you might want to filter out is a membership created event because maybe we are supposed to have a new owner or maintainer of our code additional things we might want to

see here are like organizational context how many administrators or owners maintainers are we supposed to have was this user supposed to be added supposed to give these permissions where did this come from and user agent is always a nice thing to have so we have the data we've downloaded we also have a way to get back into our gitlab environment but unfortunately something happened this morning the attacker woke up and they couldn't find any matching socks so that drove them crazy and they wanted to go full kamikazi and decided to make our project and our group completely public for the whole world to access because they need to have their socks ready so what we could look for here is a custom

message with the visibility or visibility level turning from private to public additional information we could look for here is how many of these projects or repositories are supposed to be public we could also look for when this happened where did it come from and can how many people are supposed to ask this that frequently and most important do we have matching socks for our attacker so some security remediations we have in Octa first review your MFA that everyone has MFA I mean do I really need to stress this out Additionally you have to check that your admin console as Ariel mentioned before that sometimes things aren't logged there we want it to be secure so we need to enforce MFA

additionally we could enforce Network policies which will limit the access to our OCTA environment but there is something of an early access feature that will allow us to also have Network policies with ssws tokens because previously you could have still login with a token even if you were on a limited network policy and finally review and possibly revoke your API tokens do you have any OCTA API tokens around that might lead people to get to start this whole process at the beginning now let's talk about the gitlab recommendations we have gitlab allows us to have a feature that's called secret detection enable it it searches your repository actively for codes for Keys passwords Etc we could

also enforce MFA on our gitlab implementation rather than just a user or password it's a feature for somehow you need to turn it on and we could restrict IP access based on network cidr ranges and by email domains to our environment and finally we also have the small tick that could prevent users from creating a project or a group access token to our environment it's a small tick but it could cause us um to be more protected from attackers losing their socks so what next tomorrow see if you're available to to have all your logs right for you right are you ready for investigating a possible case do you have all your logs ingested in a week from now now study

your logs check your patterns how does your OCTA environment look like what are you expecting to see in your gitlab stages and in a month from now actively conduct a hunt in your environment did our your OCTA users move from one system to another is there anything weird going on in your gitlab environment but if there's one thing I could have you all take away from here today is even though we can't always control or trust our security systems we can always hunt our security systems thank you very much we were Ariel and Gabrielle and this is our link to our